Fret Over the Team, Not the Product

I’ve had a lot of jobs in the past where my manager has admonished me to be thinking about the product all the time, to check it at home to make sure it’s still up and running, and to be constantly vigilant.  Sometimes they hand me a cell-phone or a pager and say that I’m now “on call” and should be available in case anything goes wrong.  I’m a deep sleeper, so this led to me waking up over any noise, no matter how big or how small, terrified the pager was going off in the middle of the night, or getting out of the shower and rushing to the table to make sure I hadn’t been called (happened once… I forgot about the phone until I got to the office, and my boss was pulling his hair out wondering why I hadn’t picked up the phone).

At the time, it seemed like standard operating procedure, but looking back it seems mighty frantic and usually the problem was something small that someone else had done who could have diagnosed it immediately instead of the 30 minutes to an hour that it would take us.  Not to mention the stress and how unhappy it made engineers to be chained to the office more than we already were (btw, compensation for suddenly being “on call” was a pipe dream).

The product I’m working on now has almost full test coverage, and everytime we make a commit to the source code, our Cruise.rb continuous integration environment picks up the change and runs the full test suite, and if everything passes cuts us a tag to deploy from.  Before we commit we run the suite of tests that correspond to the area of the site we work on (our rspec tests are fast, but our Screw.Unit and Selenium test suites take some time to run).  So, we as a team, are expected to have a fully deployable product everytime we check-in code.  We use an automated build tool to do our deploys that kindly rolls things back for us if there is some major configuration problem.   Our builds normally are 10 minute affairs, kicked off from one command, and our usual procedure is to make sure the environment comes up and that we can log in with our test accounts and do a 5-minute spot check.  After that we’re done.  For all intents and purposes we forget about it until the next time we have to a build, and then we spend another 15-20 minutes worrying about it.

So we have a bunch of processes that help us keep our code clean, those mentioned above and others as well that we find to be very effective for our team.  The business people seem to be way too happy in my experience, and don’t stress about the product.  So, why should the engineering team?  And this brings me to the crux of my post.  There are lots of ways that you can ensure that your code and your releases aren’t going to blow up requiring you to come in and fix some errant code that you rushed to get in to the next release.  If you aren’t using them, you probably should be.  You don’t have to use ours, find the ones that work for your team.  Instead of worrying about your product, worry about your team.  Worry about the relationship you have with the other parts of your organization, and how they view you.  This has a natural tendency to make the team want to do the things that will increase the reliability of your product, because that is one of the ways you are directly represented to other parts of your organization.  Worrying about the perception of your team lets you make decisions that focus on the long term payoff, such as continuous integration, automated test suites, etc, etc instead of fixing this one bug and heaving a sigh of relief because everything is O.K. FOR RIGHT NOW.  It also has the natural tendency to make you look at how you are fulfilling your role in the organization and working to improve that.  Quality in the product, and your role within the organization, is just a side effect.

You also won’t be waking up in the middle of the night because the cat knocked over your books, terrified that the site is throwing errors, the office is burning, and all the other nasty things that are happening that managers seem to think come together as the perfect storm that only you can fix.