The error message that we’d get during our build was something like this:

The process cannot access the file ‘XYZ’ because it is being used by another process.

After digging into the problem for a few minutes I found the conflict.  Our builds currently use the “maxcpucount” switch which enables multi-processor builds such that projects can be built simultaneously provided they have separate project dependencies.  In essence, it’s a way to build “large” (10+ project) solutions more quickly.

The problem is that NuGet doesn’t work well with the maxcpucount switch when using package restore.  The reason for this is that msbuild will execute a build for each non-related project.  However, some of those projects may contain 3rd-party NuGet dependencies.  This causes NuGet to go out and start downloading each dependency from the NuGet server.  Following that, it will unzip the NuGet package in the $(SolutionDir)\packages\<package-version>\ directory.  If multiple projects reference the same NuGet package, e.g. Log4Net or something else similar, then multiple simultaneous builds may collide while attempting to download the nupkg file and/or when attempting to extract the contents of the NuGet package.

The current workaround for v1.6 is to not perform builds with the maxcpucount switch.

LogMeIn purchased the Hamachi source code a few years back and commercialized it.  They have a premium as well as a free version.  For our particular needs, we only need a “mesh” network which is part of the free version that allows the various nodes to talk to each other in a P2P fashion.  The installation was a piece of cake.  We didn’t even need to register and account.  We simply created a named network after installing the client on the first node and then joined the network after installing the client software on each additional node by supplying the network name/password.

The P2P networking client runs as a Windows service which means that it can be run without logging into the console—a must for servers.  One caveat that we ran into was that on one of our machines where we installed as the first node, because we didn’t want to do any kind of reboot, we had to restart MSMQ and SQL Server to get them to even recognize and listen on the Hamachi-based IP address (5.x.x.x).

All in all, I am very pleased with the solution.  It was simple and created a secure tunnel through the Internet without any fuss.  The best part is all of the traffic is direct/P2P which means there’s no proxying through LogMeIn servers which means that we don’t have to worry about bandwidth/latency issues.

At some point, if we decide to go with the paid solution, it’s only $20/server/year—a very reasonable price for such a capable solution.

Let’s take a “real-world” example and see how the presence of event sourcing could fundamentally change the investigation of a crime scene. Western cultures have a fascination with CSI-like mysteries and courtroom dramas. How many books, movies, and television shows are dedicated to the “whodunit” murder mystery? Sherlock Holmes is probably the original crime scene investigator while his modern-day counterpart, Shawn Spencer from Psych, is one of the most recent. This entire genre has one thing in common: all provide a final state and the protagonist has to re-create the series of events and motive that ultimately lead to the arrest of the perpetrator.  

Let’s suppose for a moment that in reality we could somehow overlay a stream of events into a crime scene.  How simple would it be for anyone, including a domain expert, to read through the stream of events to “see” exactly what happened?  (This is often referred to the “reveal” in whodunit-style murder mysteries where the audience is finally shown exactly what happened, how, and why.)  If anyone could read this stream of events it would be a trivial matter to understand why and how a given crime scene ended up the way it did and why a certain window was smashed or there were scratches on the wall, etc. Not only that, it would be easy to see when certain things occurred so that the “time of death” could be definitively established instead of a best-guess by the coroner.  Finally, because all of the motive is captured in the events, we could see into the perpetrator’s behavior as he was causing the events around him. All of these factors would ultimately be the “smoking gun” that points directly to the perpetrator each and every time.

The argument might be, “well that takes all the fun out of guessing what happened and figuring out whodunit.” Exactly. Guesswork can be dangerous and is, at best, guesswork. Granted, if we had event sourcing in a murder-mystery, that would sort of the defeat the point and would put poor old Scooby-Doo out of work.

As software developers, we have an amazing opportunity to capture all of the intention in our domains as a stream of events. This stream of events has amazing power to give insights into reasons why and how something changed. To me, this doesn’t sound like architectural gold plating, in fact, I would argue that it’s just the opposite–and this doesn’t even include all of the additional technical and business benefits derived from event sourcing such as true persistence ignorance, simplified backup strategies, multi-master deployments, independently scaled read models, etc. I will also be the first to say that you don’t need event sourcing everywhere.

I will also say that you shouldn’t dismiss the business and technical benefit that can be derived through its use with the simple wave of the hand merely because there’s some overhead to understanding the pattern.

New in the release (since the v3.0 beta release)

• Fixes in the SQL implementation that could cause concurrency exceptions when running multiple instances of the EventStore against a single backend storage.
• Better SQL paging support when working with large data sets.
• Added a CommittedHeaders property to stream to help determine which “aggregate type” created the stream.
• Pointing to the latest versions of RavenDB, Mongo, MySql Connector, Json.NET, and ServiceStack.
• Added full logging across the library.  Various logging implementations are supported including log4net and NLog.
• Added the ability to “upconvert” events as they are loaded from persistence.
• Added ability to share IDbConnection across multiple operations on a single thread using a “ConnectionScope” to avoid 2PC promotion when using TransactionScope.

Looking forward:

• .NET 4.0 only.  Sorry for those still using .NET 3.5 (CLR 2), it’s time to upgrade.
• As with most open source libraries, the docs are deficient.  Part of the next release is to get a few GitHub wiki pages going along with some good code examples.
• As mention in the 3.0 beta post, the essence and spirit of the CommonDomain project will be merged into the EventStore.
• More storage engine support will be forthcoming.
• Two areas that need a little bit of attention are the API surrounding snapshots as well as the concept of “GetFrom(DateTime start)”.  I’ll be looking at doing something more like “GetFrom(int index)” in the near future.

https://github.com/joliver/EventStore/downloads (contains all binaries and plugins for various persistence and serialization methods)

On NuGet, the following packages have been pushed.  Because this is a beta, not all of them are marked as “recommended” yet.

http://nuget.org/List/Packages/EventStore
http://nuget.org/List/Packages/EventStore.Logging.NLog
http://nuget.org/List/Packages/EventStore.Logging.Log4Net
http://nuget.org/List/Packages/EventStore.Serialization.Json
http://nuget.org/List/Packages/EventStore.Serialization.ServiceStack
http://nuget.org/List/Packages/EventStore.Persistence.RavenPersistence
http://nuget.org/List/Packages/EventStore.Persistence.MongoPersistence

Big Changes

• Paging: The SQL and Raven implementations now support paging automagically.  This is to help facilitate queries where large result sets could potentially be returned, e.g. an stream with hundreds or thousands of commits against it or returning all commits since a given point in time.  Page size is configurable with sensible defaults.  Mongo already supports paging under the hood automatically.
• Two-phase commit: Perhaps the biggest new feature is support for 2PC.  Because *so* many people asked for it, here it is.  There are a few big, fat issues to be aware of.  First and foremost, your storage engine must support 2PC.  That pretty much means SQL Server and Raven.  One issue with Raven is that you get one operation per 2PC.  This is due to the way Raven handles 2PC and doesn’t update its indexes on commits until *after* the full transaction completes.  In other words, you can’t query and get the stuff you just saved until after the transaction is complete.
• Most all dependencies are now automatically resolved from the NuGet gallery *at compile time*.
• The two message “dispatchers”—sync and async—are now called ‘”dispatch schedulers”, therefore the interface IDispatchCommits is now IScheduleDispatches.  The associated IPublishMessages (which never felt like a good fit), is now called IDispatchCommits.  The associated implementations have changed accordingly.
• Added logging throughout the library.  There’s a little bit left to do for logging in Raven and Mongo, but all activities can now be logged.  You can output the diagnostic messages to the console, trace window, log4net, or NLog.  This can be specified using the “LogTo” wireup overload.
• Added “SystemTime” abstraction to facilitate testing.
• Added support to publish packages to NuGet.
• For those using MySQL, I added an index that was missing—you may want to run this to add the index: CREATE INDEX IX_Commits_Stamp ON Commits (CommitStamp);
• The SQL implementation will now throw a “StorageUnavailableException” if storage is offline.
• A few of the interfaces and implementations surrounding SQL persistence have changed—this should only really affect you if you’re doing a custom SQL dialect.
• Raven can now be run be run in Azure without a config file.
• Fixed a few outstanding issues that had built up in the GitHub issue tracker.

Once this release solidifies, I’ll turn my attention to 3.1 (hopefully it’ll be a point release).  There are some exciting things coming in 3.1 and beyond:

• Merging in the "essence” of CommonDomain (which was spike code that worked a little too well) into EventStore but with a slightly different model that doesn’t violate the spirit and ultimate purpose of either library.
• Support for Amazon (SimpleDB/S3)
• Support for Windows Azure (Tables/Blobs)
• Support for CouchDB
• Support for Cassandra
• Support for Redis
• Support for Oracle
• Support for Sybase
• Support for a local, file system-based store.
• Support for Protocol Buffers and other serialization formats.

In the near future, perhaps after this release or one of the next ones, I will be dropping support for .NET 3.5 entirely.  There is a point at which I can’t hold this library back any longer from taking advantage of .NET 4.0-specific features.  I don’t expect any big showstoppers with this beta release.  Once I address any feedback I get and have used this in production on my own systems long enough, I plan to push a release by around the first or second week of October.

One of the things about Git that I never worried about too much was leveraging branches for various work items.  Instead, I’d often just do my work on my own local copy and use master.  Then, when I’d go to push, I’d first pull in changes, rebase, and push.  (There’s another great article on why rebase is bad.)  The thing about rebase and merge is that they creates a single “branch” that is long and all of the work that you performed for a particular work item or bug or whatever gets lost in the single master branch.  Enter “git merge –no-ff”.

The “no-ff” stands for “no fast forward”.  The whole idea is that each time you do something, you to keep its identity and know that it came as a result of a particular bug.  If you merge and you do so by fast forwarding, that context has been lost.  But by appending "–no-ff” onto the merge command, you’ll retain the context and the merge is maintained.  Of course the end result in terms of the source code is the same, but we know how we got there.  Kind of like event sourcing, huh?

If there’s anything you guys know about me, it’s that context is king. Without it, we’re lost because we can’t tell the forest from the trees.  That’s why from now on, I’m going to be doing lots of merging but without fast forwarding.

I had heard a lot about Ooma—especially how inexpensive it was which was very appealing because our needs were so minimal.  So on June 11th, I decided to take the leap and move to Ooma and I ordered the device on Amazon.com because it was $50 cheaper than through Ooma’s web store.  A few days later it arrived (gotta love Amazon Prime).  I plugged it in and set it up and within a few minutes I had a dial tone.  After doing a few tests over the following days, I decided to port my telephone number to Ooma.  Within a short time that was complete and everything was great.  Until it wasn’t.

On July 17, my Ooma device died.  Wouldn’t boot.  Nothing.  And that’s when the problems started.  From a technical side, it’s no big deal—just contact Ooma and have them send out another device which I did.  After contacting support, I got a response via email on July 21 asking me to confirm my shipping address so they could send a new device.  Okay, no big deal.  My family was on vacation for the next week and when I got home I expected a package to be waiting for me.  Except it wasn’t.

During this time all calls to the home phone were forwarded to my wife’s cell.  The latency was worse than abysmal.  It was literally a five-second lag which rendered conversations virtually impossible.

I finally called Ooma on July 28 to see what was happening.  They said they’d send a unit out as soon as possible.  It arrived on August 2.  It was then that I discovered that the power brick for the first one was bad—not the original device itself. The new device worked fine and I boxed up the original one and got it ready to send back.  The next day I received another Ooma device in the mail.  Weird.  I only asked for one.  Apparently they sent two because I had contacted them twice and they were somehow logged under different issues even though I provided an issue tracking number.

At this point my wife and I had a serious conversation about Ooma.  We had had enough.  The service, when it was working, was good enough, but it was totally useless during any internet/service outage because the cell phone delay was horrible.  Compounding the fact that it took about two weeks to get a new device to us—that’s worse than ridiculous.  On the flip side with Vonage, during any kind of internet outage when calls were forwarded to my wife’s cell she couldn’t tell that people didn’t just call her cell directly because there was no perceived latency.

About August 3, I called to cancel the service and to get a refund for the Ooma-brand device I purchased from Amazon.  I was informed that I would be unable to do so because the device was purchased through Amazon and I would need to seek a refund from them.  Amazon’s policy was 30 days so that wasn’t an option.   Furthermore, Amazon wasn’t the “at-fault” party in this.  Interestingly enough, Amazon also states that the serial numbers on the returns must match those shipped otherwise no refund would be issued.  Had I sent my original device back to Ooma, I’d be even more up a creek without a paddle.  The rep informed me that someone higher up would have to make the decision.  My wife received a phone call a few days later informing her that no refund would be issued and that we’d have to take the complaint up with Amazon.  #Epic Fail.

On August 11, I called again.  After explaining the entire situation again to another rep from their outsourced India-based telephone support, I was informed that someone else would contact me to discuss the matter further.  It’s been five days and I haven’t heard anything back from them.

Oh, and the phone has been unavailable since August 11 as well.  I sent back the replacement devices because the original device was actually working properly (with a new power brick).  I contacted them to re-associate the original device with the account and that was an exercise in pure frustration.  The complete and utter incompetence and inability of their support to do the most simple things beyond updating your email is astounding.  To date I have probably wasted upwards of seven hours on the telephone with them, which far outweighs any potential benefit the service was supposed to offer in cost savings.  I consider the small investment of time in this blog post to be a  public service to help those considering Ooma to make an informed decision.

I guess my beef with Ooma is four fold:

1. The customer “service” is by far the worst I have ever experienced.  AT&T, Comcast, Qwest, and the IRS are an absolute delight by comparison.
2. The investment of time it takes to resolve even the most simple of issues (like getting the device associated with your account) is astronomical and obliterates any potential cost benefits as compared to traditional PSTN or VOIP-based services.
3. When the internet is unavailable, the latency and delay experienced when the calls fails over to a cell phone makes the conversing impossible—thus defeating the purpose of a telephone altogether.
4. Ooma’s unwilling to budge on the idea of taking back a device they manufactured.

The last point deserves a bit of clarification.  With literally any other manufacturer or provider, it doesn’t matter the retail establishment through which a product was purchased, they will refund your money if you request it.  Oftentimes, a manufacturer will even have a 110% guarantee where they will refund more than the purchase price.  Ooma can’t seem to get this concept through their heads.  It’s almost as though they’ve disassociated themselves from the product they sell.  Ironically, they’re fully willing to send a replacement device when one fails, but when push comes to shove and a refund is requested, you’ve got to go through the original retail outlet.  As a manufacturer, I would expect them to stand behind their service and device regardless of where it was purchased—provided you can show proof of purchase.

In conclusion, I cannot recommend other telephone providers more highly compared to Ooma.  Every point of contact an interaction with them has been so excruciating and painful—such an epic fail—I am absolutely amazed they still exist.  Run away as fast as you can.  I’d go back to Vonage in a heartbeat.  And if you’re listening Ooma, which I doubt you are, you may want to start paying attention to Twitter.  There’s a good amount of chatter under the #ooma hashtag that you should be responding to.

My current plain is to actually run Asterisk on my Linux-based micro server at home and to use Vitelity or another SIP provider.  For those of you that have read to this point and that still have a home phone, what provider do you use?