Resources

Release Time - The time, in hours, from a change set committed to mainline to be successfully running in production

Successfully running in production means that all pre-defined checks have been carried out successfully and no degradation in service has been detected for a defined period of time.

How we calculate release time

Calculating release time includes:

• Automated functional testing (pre- and post- deployment)
• Security/Compliance/non-functional testing
• Operational monitoring and alerting (including canary deployment)

Release time is calculated per change set.

Every commit to mainline is a separate change set.

In the example above:

• The time between each change set is 5 hours
• The wait time between from a new change set created to pipeline start is 1 hour
• The time for a successful pipeline execution is 3 hours

Factors that affect release time

• Batching commits - a number of commits may be batched into a single change set
• If a release candidate fails, further batching will occur for the next change set.
• Manual approvals increase release time
• Canary/incremental deployments increase the confidence that a deployment has not resulted in a failure (change failure). For the purposes of lead time calculation - the evaluation period during a canary deployment should, ideally, be excluded from lead time measurements. If changes are queued, canary release evaluation may impact the lead time for the subsequent change.
  • Identifying and catering for canary deployment may require detailed context of the customers pipeline configuration semantics.
  • A canary release would be considered successful if the new version is successfully deployed to 100% of users without issue for one evaluation period. An evaluation period is a period of monitoring before increasing access to the next group of users (e.g. 1 hour).

Considerations

• Successfully running in production
  • is a subjective term.
  • A failure may/may not be attributable to a deployment once root cause analysis has been conducted.
  • A failure may be the result of a latent bug in a previous release.
• We define a “successful” deployment:
  • In an automated pipeline, it is the final execution status of the pipeline (i.e. SUCCESS/FAIL) - this should be available in the event notification
  • In a manual process (including manual PVT), the final stage of the pipeline is a manual approval process to indicate that the release was successful.
  • If this process is not captured in an automated pipeline - we would recommend creating an automated pipeline with placeholder steps - the final being manual approval. The benefit would be improved confidence in metrics capture.