With permission, here is our general CI workflow document that my team drafted that governs our vision regarding how we do CI:
Continuous Integration Workflow
Abstract
We want to provide developers with the ability to make changes to project source code and to then automatically, transparently, and consistently subject all project source code to rigorous testing and quality assurance processes.
Further, we want the build artifacts resulting from source code changes to be quickly available to interested parties at a well-known location, but at a lower level of confidence in the those changes. These same artifacts, after a more time-consuming and rigorous testing process, are made available in another well-known "blessed" location.
Any projects which depend upon the newly built artifacts will detect changes to the "blessed" locations and begin their own workflow in much the same manner. This is recursive all the way up a dependency graph such that any low-level library can be certified against all projects that use it all the way up the chain within a relatively short period of time.
Objectives
- Define the basic structure of a continuous integration workflow.
- Provide a high degree of assurance regarding changes made to the source code such that a baseline of quality can be ensured within a few minutes and higher degrees of quality can be ensured shortly thereafter.
- Allow developers to explicitly "pull" shared libraries where high confidence has not yet been obtained.
- If modified source code is a shared library, ensure dependent projects only receive the binary at the highest level of confidence before kicking off their own integration workflow as a result of changes to a shared library.
- Allow projects to lock their dependencies to prevent # 4 from occurring.
Local/Developer "Private" Workflow
- Pull from "source code" repository on "mainline" server
- Make changes to source
- Execute developer "private" build script which:
- Runs all unit tests and fast quality metrics
- Commit to local "source code" repository
- Re-run steps 1-4 and rebase as necessary
- Push to "source code" repository on "mainline" server
Check-in Gate1 "Commit" Workflow (CI Server)
- Pull from "source code" repository" on "mainline" server
- Execute check-in gate "commit" build script which:
- Versions assemblies
- Compiles/AOP/ source index/ILMerge/obfuscation/etc.
- Runs quick/unit tests
Stamp Artifacts Workflow
- Executes "stamp" built script which:
- Tags local "source code" repository with version
- Commits compiled artifacts to local "artifact" repository (known location)
- Pushes tags to "source code" repository" on "mainline" server
- Pushes artifacts + tags to "artifact" repository on "mainline" server
Acceptance Test "Secondary Build" Workflow
- These tests can be platform specific and can be executed on multiple nodes simultaneously
- Pull the stamped artifacts from the "artifact" repository on the "mainline" server
- Execute any secondary/test scripts which:
- Runs unit and integration tests (with coverage)
- Runs source code quality metrics (complexity, lines of code, introspection, performance)
- Generates documentation
Bless Artifacts Workflow
- Once all previous tests have been accepted, the artifacts are now considered "blessed" and can be moved into the "artifact" repository on the "blessed" server. This particular script simply performs a push from its own repository to the "blessed" server (which is "origin" in git) and makes the binaries generally available as a release.
Recursive Workflow
Once the blessed artifacts have been published, any dependent project (via the master/trunk branch) can pull the update (via git submodules or svn:externals) and commence their own workflow, thus beginning the cycle all over again all the way up any project dependency chain to the leaf node projects. At any point if any of the dependent projects fail, we can easily follow the error back to either a bug in the library being modified or possibly the dependent project's usage of the library.
In addition, some projects may want to use a very specific version of a dependency. Through the use of tags (in git submodules or svn:externals) a dependent project can "lock" at a particular build of a dependency or a particular version of a dependency.
Note that git submodules are already "locked" at a particular revision and require another script to facilitate updating those modules to the latest revision as desired.
Developer Shortcuts
Oftentimes, a developer makes a change to a shared library because they need some immediate functionality in the library or they have identified a bug and want to quickly correct it so that they can move on. Git helps us in this scenario because the original developer who corrects the problem can simply "pull" from his local/committed repository into the dependent project. Further, other developers can quickly pull from the original developer's repository as necessary or they can pull the "artifact" repository on the "mainline" a minute or two after the bug fix is committed. In this way, we ensure that developer productivity is uninhibited while dependent projects always receive the highest quality build possible.
Additional Thoughts
There is a large debate about where to store build artifacts. Some teams prefer to use a shared network drive or other well-known location. While this works well for localized teams, it does present challenges for those that are developing with semi-connected laptops or elsewhere without local network or even internet connectivity. For this reason, we have opted to use Git repositories to store the build artifacts primarily because the repositories are cloned locally. Furthermore, because of Git's unique delta-based storage and compression mechanisms, large numbers of build artifacts can be stored with minimal increase in the size of the repository. Lastly, unlike most SCM systems, Git has the unique ability to remove history thus allowing us to remove old binaries if the repository becomes too large.
References
1. The Deployment Pipeline, Dave Farley, 2007.