About a year ago, I read Continuous Integration: Improving Software Quality and Reducing Risk. It's a great book that talks about the importance of making your CI server the hub of development.
One part of the book that was very insightful was the exact role of build scripts. Specifically, the build scripts should be able to execute on our developer machines as well as the CI server. Even though we were attempting to do this to some extent, there were some additional advantages that were obtained by going the rest of the way.
Choose Your Server
When we first started doing automated builds, we wrote scripts under CruiseControl which ran exclusively on the server. There was no possible way to run the scripts on the developer machine short of having a full-blown CC instance running on our dev machines. The XML scripts were error prone and bugging and tightly coupled us to a CruiseControl-specific implementation. Further, everything was lumped together into one monolithic build script which made us afraid to touch it for fear of breaking the build.
All CI products out there—CruiseControl, Cruise, TeamCity, Hudson, Continuum, etc. provide the ability to invoke activities, but I am not aware that they provide guidance stating that you should have CI-independent build scripts which are simply invoked through the CI server.
Separation of Concerns
To break our build server dependency, we moved away from CI-specific scripts and chose a "portable" solution—at first NAnt and finally rake. We then focused on writing build scripts that were broken down and focused on several different types of activities:
- Code compilation
- Static analysis/best practice compliance
- Testing in its various forms
- Source control actions (e.g. tagging)
By having smaller script activities, we could compose them into larger once as necessary. Further, it allowed us to narrowly focus our attention on a single concern rather than mixing code compilation with source control actions, for example.
In this way, we became CI server independent—we could pick whichever CI server implementation that we wanted because it would simply call a series of scripts. This also allowed us to have a private developer build to be performed locally prior to each commit (or "git push"). The local developer would be virtually assured that his source code changes would integrate because the developer build script was identical to the one that would be executed on the CI server.
Separate Workspaces
A best practice for build servers is to break down longer running tasks into shorter tasks. Usually you'll want to have some sort of fast-running "integration" task that compiles and does a few unit tests. This is a minimum threshold of quality that code must meet to be considered acceptable code. Beyond that, any number of additional long-running scripts may execute and perform a full range of testing against the code. Finally, at the end, some kind of "publish" is done to make the build artifacts generally available at a well-known location.
One big problem that we noticed was that in a local or developer environment, your workspace is "stable" in that you can run a particular build script and immediately be ready to run the next build script because the binaries are at a specific location. A CI server provides no such luxury.
A CI server almost always has a separate workspace (directory) for each "build project" that will be executed against a development project (i.e. a single C# project or solution). This presents a problem because the "Integrate" project has no way to push the compiled artifacts into the "Acceptance Testing", "Performance Testing", and "Code Metrics" projects.
Here we had to help our CI server a little bit. We simply modified our "Acceptance Testing" project to retrieve the build artifacts (using "wget") from the "Integration" project prior to kicking off the acceptance-testing related scripts.
Other than that, our CI server simply calls out to our build scripts. In this way, we maintain the ability to "hot swap" our build process at a moments notice.
And by the way, we're using Hudson.