{"id":14,"date":"2015-06-02T00:00:00","date_gmt":"2015-06-02T07:00:00","guid":{"rendered":"https:\/\/github.test\/2015-06-02-deploying-branches-to-github-com\/"},"modified":"2024-06-17T08:31:24","modified_gmt":"2024-06-17T15:31:24","slug":"deploying-branches-to-github-com","status":"publish","type":"post","link":"https:\/\/github.blog\/engineering\/engineering-principles\/deploying-branches-to-github-com\/","title":{"rendered":"Deploying branches to GitHub.com"},"content":{"rendered":"

At GitHub, we use a variant of the Flow<\/a> pattern to deploy changes: new code is always deployed from a pull request branch, and merged only once it has been confirmed in production. master<\/code> is our stable release branch, so everything on master<\/code> is considered production-ready code. If a branch deploy ships bad code containing a bug or performance regression, it is rolled back by deploying the latest master<\/code> to production.<\/p>\n

Using this workflow, engineers at GitHub deploy changes to our website<\/a> several hundred times every week.<\/p>\n

\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/graph me 20150517..20150523 @github.deploys.total<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b>\n<\/div>\n<\/div>\n

All deployments happen in chat via Hubot commands<\/a>, which ensures that everyone in the company (from development to operations to support) has visibility into changes that are being pushed into production.<\/p>\n

\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to production<\/div>\n<\/div>\n
\n
tmm1 is deploying<\/span> github\/my-feature (deadbeef<\/span>) to production.
\nCheck out the haystack firehose<\/span> so you’re the first to know of any new exceptions.<\/div>\n
tmm1’s production deployment<\/span> of github\/my-feature (deadbeef) is done! (82s)<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1, make sure you watch for exceptions in haystack<\/span> and perf issues at graphme<\/span><\/div>\n<\/div>\n<\/div>\n

Deployments are reported to the GitHub API<\/a> and show up in the timeline on corresponding pull requests.<\/p>\n

\"deployments<\/p>\n

Recent deployments are also available through chat.<\/p>\n

\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deployed github<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
\n
2015-05-27 15:10:16 -0700 - rtomayko deployed<\/span> github\/regen-missing-merge-commits<\/span>(a987454a) to production (compare)<\/span>\n2015-05-27 13:37:05 -0700 - connors deployed<\/span> github\/master(ce83e8bb) to production (compare)<\/span>\n2015-05-27 13:29:32 -0700 - jakeboxer deployed<\/span> github\/org-migration-logged-out-500<\/span>(98451e5c) to production (compare)<\/span>\n2015-05-27 13:13:04 -0700 - grantr deployed<\/span> github\/add-at-to-code-tokenizer<\/span>(d4daff29) to production (compare)<\/span>\n2015-05-27 12:04:40 -0700 - jakeboxer deployed<\/span> github\/fix-accept-invitation-timeout<\/span>(f224305e) to production (compare)<\/span>\n2015-05-27 11:56:34 -0700 - jasonrudolph deployed<\/span> github\/api-fix-schema-doc-generation<\/span>(b2805955) to production (compare)<\/span>\n2015-05-27 11:35:02 -0700 - spraints deployed<\/span> github\/silence-yajl-errors<\/span>(3adc5bc7) to production (compare)<\/span><\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n
Over the years, we’ve built a number of deployment features into Hubot and Heaven (our Capistrano-based deployment API) to help streamline our process. Below are some of our favorites.<\/p>\n
Deploy Queues<\/span><\/a><\/h4>\n
During peak work hours, multiple developers are often trying to deploy their changes to production. To avoid confusion and give everyone a fair chance, we can ask Hubot to add us to the deployment queue.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/queue me for github<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: Ok, I added you to the queue for github. There are 3 people ahead of you.<\/div>\n<\/div>\n<\/div>\n
We can also check the status of the queue, to deploy directly if it’s empty or to find a less busy time if it’s looking particularly full.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/queue for github<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: The current queue for github: mastahyeti, aroben, bhuga<\/div>\n<\/div>\n<\/div>\n
We can also unqueue ourselves if something comes up and we have to step away from the computer.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/unqueue me for github<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: Ok, tmm1 isn’t in the github queue anymore.<\/div>\n<\/div>\n<\/div>\n
Deploy Guards<\/span><\/a><\/h4>\n
To ensure bad code cannot make it to production, Hubot won’t let us deploy a branch until continuous integration tests have run. This prevents trigger-finger deploys while CI is still running.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to production<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: Sorry, I couldn’t deploy github\/my-feature: github<\/span> and enterprise<\/span> are still building.<\/div>\n<\/div>\n<\/div>\n
Similarly, if CI completed but our branch failed some tests, Hubot will prevent us from deploying.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to production<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: Sorry, I couldn’t deploy github\/my-feature: github<\/span> and enterprise<\/span> failed to build.<\/div>\n<\/div>\n<\/div>\n
Since master<\/code> is our stable release branch, we want to ensure that any branches being deployed are caught up with the latest code in master<\/code>. Before proceeding with a deployment, Hubot will detect if our branch is behind and automatically merge in master<\/code> if required.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to production<\/div>\n<\/div>\n
\n
\nAuto-merged master into my-feature on deployment – Aman Gupta ( github\/github@feedface<\/span> )<\/div>\n
Build #2077391<\/span> (feedface) of github\/my-feature was successful (97s, queued 0s)<\/div>\n
Build #2077392<\/span> (feedface) of enterprise\/my-feature was successful (106s, queued 0s)<\/div>\n
tmm1 is deploying<\/span> github\/my-feature (feedface<\/span>) to production.<\/div>\n
tmm1’s production deployment<\/span> of github\/my-feature (feedface) is done! (100s)<\/div>\n<\/div>\n<\/div>\n
To ensure deployments are visible to the rest of the team, Hubot forces us to deploy from specific chat rooms.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to production<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: Sorry, github must be deployed from the appropriate room.<\/div>\n<\/div>\n<\/div>\n
In rare emergency situations, it is possible to override these guards using \/deploy!<\/code>.<\/p>\n
Deploy Locks<\/span><\/a><\/h4>\n
As soon as a branch is deployed, Hubot locks the environment so no other branches can be deployed to it. This prevents others from accidentally deploying while a developer is testing their branch.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to production<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: Sorry, github in production is locked by mastahyeti<\/div>\n<\/div>\n<\/div>\n
Once we’ve merged our branch, Hubot will automatically unlock the environment and let the next person in the queue know they can deploy.<\/p>\n
\n
\"hubot\"
\nHubot<\/b><\/p>\n
mastahyeti: it looks like you merged the “stronger-ssh-keys” branch into master, so I’ve unlocked github in production.<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: you’re up to deploy github!<\/div>\n<\/div>\n<\/div>\n
We can also manually unlock deploys to let someone else have a turn, when we decide not to merge our branch just yet.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/unlock github in prod<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: github in production is now unlocked.<\/div>\n<\/div>\n<\/div>\n
Finally, during outages, attacks, and other emergency situations, we can lock deployments manually to prevent changes while we investigate problems.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/lock github in production investigating deployment api 500s<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: github in production is now locked.<\/div>\n<\/div>\n<\/div>\n
Deploy Environments<\/span><\/a><\/h4>\n
In addition to the main production environment, we can deploy to staging servers that are only accessible by GitHub staff. This staging environment closely mirrors our production environment, including real-world datasets to ensure high-fidelity testing.<\/p>\n
To find out what environments are available for deployments, we can ask Hubot for a list and see which ones are currently unlocked.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/where can i deploy github<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
\n
Deployment status for github:\n--------------------------------------------------------------------------------\nproduction: locked 42 seconds ago by grantr: testing the code-search-docvalues branch\nlab: locked 15 minutes ago by tmm1: testing the git-repld-reconcile-full-resync branch\ngarage: unlocked\nspider-skull-island: locked 13 minutes ago by grantr: testing the code-search-docvalues branch\nmachine-room: locked 20 hours ago by haileys: testing the upgrade-route-syntax branch\nbranch-lab: always available\n\nThe queue for production has 1 person waiting.<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n
The lab, garage and other staging environments each replicate different aspects of production: frontend web workers, background job queues, CDN setup for assets, Git fileserver workers, etc. Depending on what part of the stack a branch touches, we can pick a matching staging environment to exercise the new code without affecting production user traffic.<\/p>\n
One of these environments is a special “branch lab” which does not require locking, because it sets up an isolated sandbox for each branch. This helps avoid deploy lock contention and lets developers and designers deploy experimental UI changes as shareable URLs they can send to others in the company for feedback.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/my-feature to branch-lab<\/div>\n<\/div>\n
\n
tmm1 is deploying<\/span> github\/my-feature (feedface<\/span>) to branch-lab.
\nReview the deployment checklist<\/span> before deploying to production.<\/div>\n
tmm1’s branch-lab deployment<\/span> of github\/my-feature (feedface) is done! (22s)<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
tmm1: your branch-lab is available at https:\/\/my-feature.branch.github.com<\/span><\/div>\n<\/div>\n<\/div>\n
The branch lab is implemented as a single staging server which runs one unicorn worker per branch. The branches deployed there can be listed via chat, and a branch can be deleted once it’s no longer being used. If the free memory on that server starts to run out, we automatically prune the oldest branches to free up some space.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/branch-lab list<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
\n
+------------------------------+----------------+----------------+--------------------------------------------------------+\n|                            branch-lab (9 branches, disk: 14G used + 91G free, mem: 3G free)                             |\n+------------------------------+----------------+----------------+--------------------------------------------------------+\n| octicons-playground          | aaronshekey    |    2 hours ago | https:\/\/octicons-playground.branch.github.com          |\n| move-audit-log               | amosie         |    5 hours ago | https:\/\/move-audit-log.branch.github.com               |\n| nh-audit-log-docs            | nickh          |    5 hours ago | https:\/\/nh-audit-log-docs.branch.github.com            |\n| show-lfs-even-if-over        | tclem          |    6 hours ago | https:\/\/show-lfs-even-if-over.branch.github.com        |\n| js-clone-selector            | dgraham        |    7 hours ago | https:\/\/js-clone-selector.branch.github.com            |\n| clear-user-session-for-oauth | ptoomey3       |    7 hours ago | https:\/\/clear-user-session-for-oauth.branch.github.com |\n| remove-wizard                | amosie         |   11 hours ago | https:\/\/remove-wizard.branch.github.com                |\n| mobile-file-finder           | tmm1           |   18 hours ago | https:\/\/mobile-file-finder.branch.github.com           |\n| ship-header-nav              | mdo            |   22 hours ago | https:\/\/ship-header-nav.branch.github.com              |\n+------------------------------+----------------+----------------+--------------------------------------------------------+<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n
We can also manually remove branches that we’re done testing, or have shipped to production already:<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/branch-lab remove my-feature<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
Removing my-feature from branch-lab<\/div>\n<\/div>\n<\/div>\n
Deploy Targets<\/span><\/a><\/h4>\n
Once a branch has passed automated tests, undergone code-review, and been verified in staging, it comes time to push it into production. Recall that GitHub engineers are not allowed to merge any pull request that has not yet been verified in production. Production traffic patterns and datasets often trigger edge-cases that expose bugs and performance issues which might not have been seen otherwise, and we want to ensure that our master<\/code> branch always represents our stable production release.<\/p>\n
To safely roll out a risky branch, we can ask Hubot to deploy it to a specific subset of servers within an environment. This limits the user impact of the change, and allows us to monitor for new exceptions or performance regressions coming from the servers that are running our branch.<\/p>\n
A change to the Rails version for example can be deployed to one or two frontend webservers, and if things look good we can continue to deploy it to more frontends. Similarly, an upgraded version of Git could be deployed to a handful of backend fileservers.<\/p>\n
\n
\"tmm1\"
\ntmm1<\/b><\/p>\n
\/deploy github\/rails-6-upgrade to production\/fe130,fe131<\/div>\n<\/div>\n
\"hubot\"<\/p>\n
tmm1 is deploying<\/span> github\/rails-6-upgrade (feedbeef<\/span>) to production (github-fe130-cp1-prd, github-fe131-cp1-prd).<\/div>\n
tmm1’s production deployment<\/span> of github\/rails-6-upgrade (feedbeef) is done! (46s)<\/div>\n<\/div>\n
\"hubot\"
\nHubot<\/b><\/p>\n
Exceptions have recently elevated on github (12 exceptions) in the last 3 minutes. tmm1 was the last person to deploy at 07:40 pm PDT (-0700). Care to check it out in haystack?<\/div>\n<\/div>\n<\/div>\n
Once we’ve gained confidence in our branch, we can deploy it to all of production and then merge it to unlock deployments for the next developer in the queue.<\/p>\n
Deploy Everything<\/span><\/a><\/h4>\n
Our deployment chatops and workflows work so well that we use them for everything in the company. If we want to add a DNS record, we make a pull request to github\/dns and use \/deploy dns<\/code>. If we want to add a monitoring alert for a new service, we make a pull request to github\/nagios and use \/deploy nagios<\/code>. If we want to install a new software package on a specific frontend, we use \/deploy puppet\/add-package-branch to prod\/fe142<\/code>. We even use similar workflows to ship new versions of our native desktop apps<\/a>.<\/p>\n
If you aren’t already, we highly recommend you try some of the techniques mentioned in this blog post. This workflow brings a ton of great benefits, including:<\/p>\n