With permission, here is our general CI workflow document that my team drafted that governs our vision regarding how we do CI:

Continuous Integration Workflow

Abstract

We want to provide developers with the ability to make changes to project source code and to then automatically, transparently, and consistently subject all project source code to rigorous testing and quality assurance processes.

Further, we want the build artifacts resulting from source code changes to be quickly available to interested parties at a well-known location, but at a lower level of confidence in the those changes. These same artifacts, after a more time-consuming and rigorous testing process, are made available in another well-known “blessed” location.

Any projects which depend upon the newly built artifacts will detect changes to the “blessed” locations and begin their own workflow in much the same manner. This is recursive all the way up a dependency graph such that any low-level library can be certified against all projects that use it all the way up the chain within a relatively short period of time.

Objectives

1. Define the basic structure of a continuous integration workflow.
2. Provide a high degree of assurance regarding changes made to the source code such that a baseline of quality can be ensured within a few minutes and higher degrees of quality can be ensured shortly thereafter.
3. Allow developers to explicitly “pull” shared libraries where high confidence has not yet been obtained.
4. If modified source code is a shared library, ensure dependent projects only receive the binary at the highest level of confidence before kicking off their own integration workflow as a result of changes to a shared library.
5. Allow projects to lock their dependencies to prevent #4 from occurring.

Local/Developer “Private” Workflow

1. Pull from “source code” repository on “mainline” server
2. Make changes to source
3. Execute developer “private” build script which:
1. Runs all unit tests and fast quality metrics
4. Commit to local “source code” repository
5. Re-run steps 1-4 and rebase as necessary
6. Push to “source code” repository on “mainline” server

Check-in Gate¹ “Commit” Workflow (CI Server)

1. Pull from “source code” repository” on “mainline” server
2. Execute check-in gate “commit” build script which:
1. Versions assemblies
2. Compiles/AOP/ source index/ILMerge/obfuscation/etc.
3. Runs quick/unit tests

Stamp Artifacts Workflow

1. Executes “stamp” built script which:
1. Tags local “source code” repository with version
2. Commits compiled artifacts to local “artifact” repository (known location)
3. Pushes tags to “source code” repository” on “mainline” server
4. Pushes artifacts + tags to “artifact” repository on “mainline” server

Acceptance Test “Secondary Build” Workflow

1. These tests can be platform specific and can be executed on multiple nodes simultaneously
2. Pull the stamped artifacts from the “artifact” repository on the “mainline” server
3. Execute any secondary/test scripts which:
1. Runs unit and integration tests (with coverage)
2. Runs source code quality metrics (complexity, lines of code, introspection, performance)
3. Generates documentation

Bless Artifacts Workflow

1. Once all previous tests have been accepted, the artifacts are now considered “blessed” and can be moved into the “artifact” repository on the “blessed” server. This particular script simply performs a push from its own repository to the “blessed” server (which is “origin” in git) and makes the binaries generally available as a release.

Recursive Workflow

Once the blessed artifacts have been published, any dependent project (via the master/trunk branch) can pull the update (via git submodules or svn:externals) and commence their own workflow, thus beginning the cycle all over again all the way up any project dependency chain to the leaf node projects. At any point if any of the dependent projects fail, we can easily follow the error back to either a bug in the library being modified or possibly the dependent project’s usage of the library.

In addition, some projects may want to use a very specific version of a dependency. Through the use of tags (in git submodules or svn:externals) a dependent project can “lock” at a particular build of a dependency or a particular version of a dependency.

Note that git submodules are already “locked” at a particular revision and require another script to facilitate updating those modules to the latest revision as desired.

Developer Shortcuts

Oftentimes, a developer makes a change to a shared library because they need some immediate functionality in the library or they have identified a bug and want to quickly correct it so that they can move on. Git helps us in this scenario because the original developer who corrects the problem can simply “pull” from his local/committed repository into the dependent project. Further, other developers can quickly pull from the original developer’s repository as necessary or they can pull the “artifact” repository on the “mainline” a minute or two after the bug fix is committed. In this way, we ensure that developer productivity is uninhibited while dependent projects always receive the highest quality build possible.

Additional Thoughts

There is a large debate about where to store build artifacts. Some teams prefer to use a shared network drive or other well-known location. While this works well for localized teams, it does present challenges for those that are developing with semi-connected laptops or elsewhere without local network or even internet connectivity. For this reason, we have opted to use Git repositories to store the build artifacts primarily because the repositories are cloned locally. Furthermore, because of Git’s unique delta-based storage and compression mechanisms, large numbers of build artifacts can be stored with minimal increase in the size of the repository. Lastly, unlike most SCM systems, Git has the unique ability to remove history thus allowing us to remove old binaries if the repository becomes too large.

References

1. The Deployment Pipeline, Dave Farley, 2007.

About a year ago, I read Continuous Integration: Improving Software Quality and Reducing Risk. It’s a great book that talks about the importance of making your CI server the hub of development.

One part of the book that was very insightful was the exact role of build scripts.  Specifically, the build scripts should be able to execute on our developer machines as well as the CI server.  Even though we were attempting to do this to some extent, there were some additional advantages that were obtained by going the rest of the way.

Choose Your Server

When we first started doing automated builds, we wrote scripts under CruiseControl which ran exclusively on the server.  There was no possible way to run the scripts on the developer machine short of having a full-blown CC instance running on our dev machines.  The XML scripts were error prone and bugging and tightly coupled us to a CruiseControl-specific implementation.  Further, everything was lumped together into one monolithic build script which made us afraid to touch it for fear of breaking the build.

All CI products out there—CruiseControl, Cruise, TeamCity, Hudson, Continuum, etc. provide the ability to invoke activities, but I am not aware that they provide guidance stating that you should have CI-independent build scripts which are simply invoked through the CI server.

Separation of Concerns

To break our build server dependency, we moved away from CI-specific scripts and chose a “portable” solution—at first NAnt and finally rake.  We then focused on writing build scripts that were broken down and focused on several different types of activities:

1. Code compilation
2. Static analysis/best practice compliance
3. Testing in its various forms
4. Source control actions (e.g. tagging)

By having smaller script activities, we could compose them into larger once as necessary.  Further, it allowed us to narrowly focus our attention on a single concern rather than mixing code compilation with source control actions, for example.

In this way, we became CI server independent—we could pick whichever CI server implementation that we wanted because it would simply call a series of scripts.  This also allowed us to have a private developer build to be performed locally prior to each commit (or “git push”).  The local developer would be virtually assured that his source code changes would integrate because the developer build script was identical to the one that would be executed on the CI server.

Separate Workspaces

A best practice for build servers is to break down longer running tasks into shorter tasks.  Usually you’ll want to have some sort of fast-running “integration” task that compiles and does a few unit tests.  This is a minimum threshold of quality that code must meet to be considered acceptable code.  Beyond that, any number of additional long-running scripts may execute and perform a full range of testing against the code.  Finally, at the end, some kind of “publish” is done to make the build artifacts generally available at a well-known location.

One big problem that we noticed was that in a local or developer environment, your workspace is “stable” in that you can run a particular build script and immediately be ready to run the next build script because the binaries are at a specific location.  A CI server provides no such luxury.

A CI server almost always has a separate workspace (directory) for each “build project” that will be executed against a development project (i.e. a single C# project or solution).  This presents a problem because the “Integrate” project has no way to push the compiled artifacts into the “Acceptance Testing”, “Performance Testing”, and “Code Metrics” projects.

Here we had to help our CI server a little bit.  We simply modified our “Acceptance Testing” project to retrieve the build artifacts (using “wget”) from the “Integration” project prior to kicking off the acceptance-testing related scripts.

Other than that, our CI server simply calls out to our build scripts.  In this way, we maintain the ability to “hot swap” our build process at a moments notice.

And by the way, we’re using Hudson.

I spent the better part of a day trying to get Gallio’s runner to execute NCover and run our unit and integration tests as part of our build process.  Our build server and all of our developer workstations run some variant of 64-bit Windows, e.g. Vista, 2008, 7, 2008 R2, etc.

At the command line, I would type the following:

Gallio.Echo.exe /runner:NCover PathToTestAssemblyMyApp.UnitTests.dll

Everything worked great except for NCover.  NCover.Console.exe would stall and timeout after 60 seconds and then die saying “Profiled process terminated. Profiler connection not established.”

I tried things in a 32-bit virtual machine which worked great, so the problem had to be 64-bit related.  I tried using the corflags 32bit flag trick, I tried explicitly specifying a 32-bit runner.    I ran the build scripts as admin on my local machine.  I even tried just running NCover.Console.exe directly specifying NUnit-Console-x86.exe but things still wouldn’t talk properly.  What was going on?!

I stumbled upon an article that solved my problem.  I simply ran: “regsvr32 coverlib.dll” from the command line in the NCoverv1.5.8 directory which registered a particular COM component.  That took care of the problem.  This old version of NCover wants to run with administrator privileges and requires registration of the COM component on the various machines that will run this coverage.  That makes it difficult to ensure machine independence, but we’ll live with it for the time being.

UPDATE: You should be able to provide the following argument to Gallio.Echo to get the COM component to register during execution and unregister after execution:

Gallio.Echo.exe /runner:NCover /rp:"NCoverArguments=’//reg’ " ….

In the very near future we will be upgrading to NCover 3, but before we do, we want to ensure that we’re receiving value from NCover’s current reports before we invest money into it.the latest version.

You gotta love NHibernate.  But sometimes there are little quirks the drive you nuts—at least until you understand how NHibernate handles things.  We ran into one such “hiccup” the other day.

Our objective was to have a parent object contain a list of child objects.  Simple enough.  Problem is, we didn’t want the child to know about or have a reference back to the parent.  But all of the simple example code that we found for NHibernate and Fluent NHibernate kept showing the bidirectional reference.

Inverse To The Rescue…Almost

The whole idea about the inverse attribute, specifically Inverse=true, is so that the parent “owns” the collection.  We thought this was enough.  Nope.  We kept getting issues where the database would throw an exception because NHibernate was trying to insert a null “parent ID” value into the child table.

After digging deeper, we came up with the following feature matrix related to the value:

Inverse=False means that:

1. Child object is inserted first (null parent id).
2. Parent object is inserted second.
3. Child is then updated with parent id.
4. Parent id in child table must be nullable.
5. Child doesn’t need a reference to the parent.

Inverse=True means that:

1. Parent inserted first.
2. Child inserted (with parent id value populated) second.
3. Parent id in child table can be (but need not be) nullable.
4. Child must have a reference to the parent.

Unfortunately, we couldn’t have it both ways.  We wanted the parent to have a reference to the child AND to have the parent id column in the child table to be “not null”.  In the end, we decided on Inverse=True.

DDD Guidance

When working with aggregates, the guidance is to access everything through the aggregate root.  So, when we create a child object, we simply call myParentObject.CreateChild() and we have the CreateChild method enforce the bidirectional reference.

Legacy Codebase

The solution above is for one of our legacy codebases.  Using newer CQS [Young] style approaches, we don’t really have to worry about this.

I try to avoid the debugger at all costs, but it does come in handy sometimes.  You’ll even catch Uncle Bob in the debugger on occasion.  When using a compiled language, you basically “lose” the source code once compiled.  Of course bytecode languages can be decompiled, but that’s beside the point.

We have a number of common library projects that we share between our applications.  Usually these common projects are very low-level infrastructure concerns, etc.  While these common projects are relatively mature, they do exhibit bugs from time to time.  Wouldn’t it be nice to step in to the source code of one of those projects while debugging your application?

Enter Microsoft Source Server. It has the responsibility of “indexing” your source code.  Specifically, it hooks the PDB files back to the source code, but enables Visual Studio to pull the source from a source control repository, such as Subversion or TFS.  That’s exactly what we need.  In fact, at my company, we’ve been using this very solution for several years with great success.  As part of our build process, we would index the PDB files to facilitate runtime debugging of common/shared projects when referenced in another project while using Subversion for source control.

Enter git.  Git is relatively new in the source control world—only about four years old as of this writing.  It’s even newer in the Windows world with very little tooling support, although that seems to be changing rapidly.  Because of this, there is no “Git adapter” to use with Microsoft Source Server.

I looked around the internet for anyone who had written anything related to Git and Microsoft Source Server.  I found one article, which turned out to be exactly what I needed.  Well, almost.  In the end, I completely re-wrote the implementation to be a little bit more “Clean Code” and to support distributed workflows better, even though I’m not a Perl programmer.  The end result was SourceServer-GitExtensions.

I have written a complete start-to-finish implementation guide on how to setup indexing with Git and how to get debugging working in Visual Studio.  The source code for the project can be found here.

We’ve been using Git for a little while now in a Windows environment and we have been quite pleased with it and the capabilities it affords us as part with its decentralized workflow.  We have a number of shared projects or “common libraries” that we use between projects.  Because of this, we are using Git’s submodules feature.

One small issue that we hit was when we would make changes to the code to the project in a submodule (for example, some scripting code in a submodule), we would want to commit and push those changes.  Whenever we would attempt to commit, we would get one of two messages:

$ git push origin master

Everything up-to-date

That’s frustrating.  Things definitely were not up-to-date.  We could do a “git log” and see the difference between the local repository and the remote repository.  Why was it saying things were “up-to-date”?

Other times, we’d get an even more frustrating error message:

$ git push origin master

To git@……..

! [rejected]      …. –> master (non-fast forward)

error failed to push …..

What kind of error message is that?  The part that made it confusing was that “non-fast forward” means that I’m not working at the head—basically I’m working off an older comment and trying to push that to the head.  That made no sense because a “git pull” wouldn’t give me anything new.

The Ah-Ha Moment

Run “git status” while in the submodule.  It will say “Not currently on any branch.”  There’s the problem.  If I’m not working on a branch (even master), Git has trouble.  To remedy this, run “git checkout master” before you do any work in the submodule.

Of course, this is all by-design: http://git.or.cz/gitwiki/GitSubmoduleTutorial

When you do “git submodule update” and pull all the submodules down, it doesn’t check them out on a particular branch.  According to the above article:

One major difference between "submodule update" and "submodule add" is that "update" checks out a specific commit, rather than the tip of a branch. It’s like checking out a tag: the head is detached, so you’re not working on a branch.

So that’s the solution—checkout a branch first (even if it’s master), do your work and push.