The GitHub team recently shared their experience in seamlessly upgrading GitHub.com's underlying database to MySQL 8.0.
According to reports, GitHub uses MySQL to store a large amount of relational data, so it is not easy to upgrade the host cluster ( more than 1,200 MySQL hosts ) without affecting the website service level objective (SLO) . The team stated that in order to upgrade to MySQL 8.0, they spent more than a year planning, testing, and upgrading themselves, and required the collaboration of multiple teams within GitHub.
Overview of GitHub's MySQL infrastructure:
- Consists of more than 1,200 hosts, including Azure virtual machines and bare metal hosts in the data center
- Stores over 300 TB of data and processes 5.5 million queries per second across 50+ database clusters
- Each cluster is configured for high availability with a primary-secondary setup
- Partitioned storage data - Leverage horizontal and vertical sharding to scale MySQL Cluster, and use MySQL Cluster to store data for specific product areas. Horizontally sharded Vitess clusters are also provided for large-domain areas that outgrow single-master MySQL clusters.
- A vast ecosystem of tools, including Percona Toolkit, gh-ost, orchestrator, freno, and internal automation tools for operating host clusters
Because of the need to operate with two versions of MySQL, the tools and automation used internally by GitHub need to be able to handle mixed versions compatible and understand new, different, or deprecated syntax between 5.7 and 8.0 .
In order to meet availability standards, the GitHub team adopted a gradual upgrade strategy to meet the need for checkpoints and rollbacks throughout the process. Here is their upgrade plan:
- Step 1: Upgrade rolling replica
- Step 2: Upgrade the backup topology (replication topology)
- Step 3: Promote the MySQL 8.0 host to the primary cluster
- Step 4: Upgrade the internal-facing instance type
- Step 5: Clean up. After confirming that the cluster does not need to be rolled back and successfully upgraded to MySQL 8.0, delete the 5.7 server. Verification goes through at least a full 24-hour traffic cycle to ensure issues do not occur during peak traffic periods.
As for why they want to upgrade to MySQL 8.0, the GitHub team said it is mainly because the life cycle of MySQL 5.7 is coming to an end. In addition, after upgrading, you can get the latest security patches, bug fixes and performance-enhanced MySQL versions. They also hope to test and benefit from the new features in 8.0, including instant DDL, invisible indexes, and compressed bin logs.
See detailed technical details: https://github.blog/2023-12-07-upgrading-github-com-to-mysql-8-0/
Further reading