Container Orchestration

Introduction

Meta's data ingestion system, a critical component powering up-to-date snapshots of the social graph, has undergone a major transformation. This revamp aimed to enhance reliability at an unprecedented scale, moving from a legacy architecture to a new self-managed data warehouse service. The migration involved thousands of jobs and petabytes of data, requiring innovative solutions and robust strategies. In this article, we share the key factors that influenced our architectural decisions and the approaches that ensured a successful large-scale system migration.

The Scale of the Challenge

At Meta, the social graph is built on one of the world's largest MySQL deployments. Every day, the data ingestion system incrementally scrapes several petabytes of social graph data into the data warehouse. This data powers analytics, reporting, and downstream products used for decision-making, machine learning model training, and product development. As operations grew, the legacy system—comprising customer-owned pipelines that worked well at smaller scales—became unstable under strict data landing time requirements. The need for a new architecture was clear, but migrating at this scale posed immense challenges: ensuring seamless transitions for each job while managing the overall migration process.

Key Challenges

• Instability in legacy system: Increasingly strict latency requirements caused performance degradation.
• Scale of migration: Thousands of jobs had to be moved without disrupting operations.
• Data integrity: Ensuring zero data loss or corruption during the transition.

The Migration Strategy

To address these challenges, Meta established a clear migration lifecycle for each job. This lifecycle ensured data integrity and operational reliability throughout the process. The migration was not a single event but a phased approach with strict verification criteria at each stage.

Migration Lifecycle Overview

The lifecycle consisted of several steps, each requiring a job to meet defined success criteria before proceeding. These criteria focused on three main areas:

1. No data quality issues: The new system had to produce identical data as the old system. Verification included comparing row counts and checksums to ensure complete consistency.
2. No landing latency regression: The new system had to match or improve upon the landing latency of the legacy system. Any slowdown was unacceptable.
3. No resource utilization regression: Performance metrics such as CPU and memory usage had to remain stable or improve.

Jobs that passed these checks were gradually promoted through the lifecycle, with robust rollout and rollback controls in place. This allowed Meta to handle issues at the earliest possible stage and minimize impact on downstream systems.

Key Technical Solutions

Several solutions and strategies were instrumental in making the migration successful:

• Unified monitoring and alerting: A centralized system tracked the progress of each job, flagging anomalies immediately.
• Automated verification pipelines: Comparisons between old and new systems ran automatically to catch data discrepancies early.
• Graceful degradation protocols: If a job failed verification, it could be rolled back quickly to the legacy system without data loss.

Self-Managed Data Warehouse Service

The new architecture moved away from customer-owned pipelines to a simpler, self-managed data warehouse service that operated efficiently at hyperscale. This shift reduced complexity and improved maintainability, allowing Meta's engineering teams to focus on innovation rather than pipeline management.

Results and Lessons Learned

Meta successfully transitioned 100% of the workload to the new system and fully deprecated the legacy system. The migration process highlighted the importance of rigorous verification, phased rollouts, and strong communication between teams. Key takeaways include:

• Invest in automation: Automated checks significantly reduce human error and speed up migration.
• Plan for rollback: Every job should have a clear rollback path to prevent prolonged downtime.
• Monitor continuously: Real-time monitoring is essential to catch regressions before they affect users.

The migration of Meta's data ingestion system stands as a testament to the power of systematic planning and execution at scale. By sharing these strategies, we hope to help other organizations tackle similar large-scale system migrations.