The development and deployment pipeline
When responding to a security incident, you should make any code changes in private, so that you don’t accidentally disclose the vulnerability. To do this, follow the instructions for making a repo private.
Our development and deployment pipeline looks like this:
- Open a Pull Request (PR)
- Wait for Continuous Integration to pass
- Review your own changes
- Get someone to review your Pull Request
- Check for or implement a deploy freeze
- Merge your own Pull Request
- Deploy through each of the environments
Wait for Continuous Integration to pass
When a Pull Request (PR) is opened, it often triggers an automated job, which typically lints the code and runs the tests.
Read about Continuous Integration checks.
Review your own changes
As well as double-checking your code and commits, you may refer to the following:
Heroku App Review
Sometimes you may need to demo a running version of your change. All frontend applications and some publishing apps have Heroku Review Apps enabled, with a link near the bottom of each PR.
Branch Deploy Review
Sometimes you may need to deploy your change in Integration in order to test it works on real infrastructure. Refer to the manual deployments documentation.
Get someone to review your Pull Request
The GDS Way has guidelines on how to review code.
Only when the tests pass and the code has been approved the Pull Request can be merged, since we’ve configured GitHub to prevent merges otherwise.
Check for or implement a deploy freeze
In exceptional circumstances, we may wish to block or freeze deployments for a short period of time. This should be done by running a GitHub Action to toggle off continuous deployment, checking “Freeze deployments?” and adding an explanatory note to the application in the Release app.
Code freezes are now handled by toggling off Continuous deployments via a Github action named “Set automatic deploys” for the application. See here for government-frontend.
You can still deploy urgent changes manually using the deploy GitHub Action if necessary.
It is important to ensure people are aware of a code freeze
Checking “Freeze deployments?” on Release will add an “Automatic deployments disabled” label to the application. This label will be visible on the landing page of release and in applications page underneath the title. It provides visiblity of the code freeze to other developers who may check Release to view the status of deploys for that app.
Let people know on Slack
Send a message to #ott-core on slack with the @channel prefix (to ensure people who are offline are notified) and email govuk-tech-members@digital.cabinet-office.gov.uk. Your message should include the repo you are freezing, the reason why, and the expected duration. Follow up to let people know when the freeze is lifted.
When a deploy freeze is in effect, you should avoid merging any PRs. This is because your changes may block other, urgent changes related to the deploy freeze. Your changes will also remain undeployed for a long time.
Merge your own Pull Request
We’ve got guidelines on merging of Pull Requests.
Some applications have restrictions on who can merge PRs. If you are unable to merge your own PR, you should ask someone else to merge (and deploy) it for you.
Code that is merged to main is tested again on CI. This is because
the main branch may have changed since the tests last ran on the PR.
WARNING: some applications have Continuous Deployment enabled, which means the deployment process is fully automated. You should do any manual testing with a temporary, branch deployment before you merge.
Deployment
Teams are responsible for deploying their own work. We believe that regular releases minimise the risk of major problems and improve recovery time. The Technical 2nd Line team is responsible for providing access to deploy software for teams who can’t deploy it themselves.
Continuous Deployment
- Check the notes in the Release app to see if Continuous Deployment is enabled.
- If so, after merging, you should check the Release app to see if the deployment succeeds.
- If the latest release is not on Production within about 15 minutes, something went wrong:
- Refer to the deployment documentation for details of the deployment process so you can pinpoint where in the pipeline it failed.
- You can manually deploy your change if the automation fails e.g. due to a flakey Smokey test.
Manual Deployment
Wait for the release to deploy to Integration
Refer to the manual deployments documentation. You should verify your changes work in Integration before deploying downstream:
- Run a build of smoke tests in the environment you’re deploying to.
Our apps should always be in a state where main is deployable. You
should raise a PR to revert your changes if they cause a problem and
you’re unable to resolve that problem straight away.
Manually deploy to Staging, then Production
Deployments to these environments are manual and require production access. Go to the Release application and find the application you want to deploy, then select the release tag you want to deploy. Follow these rules:
- Deployments should generally take place between 9.30am and 5pm (4pm on Fridays), the core hours when most people are in the office.
- If there’s other people’s code to deploy, ask them whether they’re okay for the changes to go out.
- Announce in
#govuk-2ndline-techif you anticipate your release causing any issues. Stay around for a while just in case something goes wrong. - Check the Release app for a deploy note for the application, to see if there are any special instructions or reasons not to deploy. Individual app deploy freezes are usually announced here.
After a deployment:
- Check Sentry for any new errors.
- Check the Deployment Dashboard for any issues.
- Run a build of smoke tests in the environment you’re deploying to.