A Smooth Production Process

In my last post I painted a pretty Utopian picture of an imaginary delivery process, but it’s not that far from reality.  I’ve experienced a process pretty close to this on several different engagements.

How did we get here?

The first thing that went well is that your developers relied on a source and build management system to get them on the same page. Your process has captured all of the code and configuration, and associated them strongly with a particular deployment. There’s a one-to-one mapping between what’s on the server right now and some group of files that everybody knows exactly how to get to. When you say “Install build 432” people immediately know what commands to run to do that, no hand holding, no waiting in line. Five people can install the code exactly as fast as 2 people can do it, which means you can dog pile on the problem.

The second thing that happened is that you had a few developers with enough of a mental model of your system that they could begin forming theories about what’s wrong – even if they didn’t write the code.  This came from collaboration on the initial coding, transparency, and probably from code reviews.

The third thing that happened is that your developers were able to make a very small change to the code to affect a fix. In order to do this they need to be confident in their ability to refactor code. This is both a matter of code organization and practice.  Lots and lots of practice.  So much practice in fact that the fathers of Extreme Programming used to refer to this as ‘relentless refactoring’.  Do it until it’s boring, then keep doing it.

The fourth thing that happened is that the build process gave you a pretty good idea of how effective the change was.  You had a set of verification tools that always says ‘NO’ when something is wrong, and only says ‘YES’ when quite a lot of things are going right. Managers want confidence intervals on risky, last-minute code pushes, and confidence can be measured in test automation, eyeballs, and discipline.  I feel good, the tests look good, and 3 people checked my work.  Ship it.

The fifth thing that happened is a restatement of the first:  Once a new version exists, everyone needs to be able to get it, test it, and potentially deploy it. The possibility for human error has mostly been removed from this process.  Tooling handles most of it, and repetition handles most of the rest.

The biggest thing that happened is also the most valuable. Nobody panicked. Since everyone already knew what to do, people did their part as best they could and with a sense of urgency. At the end of the day people might be a little tired, but nobody was spent, or exhausted. No one person had to be the hero, and the load was shared. Tomorrow, real work will get done because people will have recovered.

The final thing that happens is that a plan is put in place to try to avoid repeating the problem that just occurred. No need to have this sort of emergency again if we can help it. The process is adjusted and estimates change to reflect any extra work. This extra work will be cheaper than having your whole team distracted by a fire drill while they were busy trying to perform delicate surgery on the code for your next release.

Why So Serious?

This sounds like a lot of discipline, but it’s no different than any emergency response team. People drill and drill so that even when they’re not expecting an emergency, even when they’re not at their best, even when several valuable team members are missing, they can react in a responsible and useful manner.

Perfect Storms Aren’t Rare

The funny thing about Releases is that they usually precede times of flux in the project. After a release, people start doing things they’ve put off for a while. Upgrading tools, planning major features, restructuring the project, going to the optometrist, visiting family, going on vacation, switching to a new project, or leaving the company.

The person who knows the piece of code might not be around. They might be elsewhere, or they may be gone. The tools you use today might be different than the old ones. It’s not enough to be able to upgrade, you have to be able to downgrade too. So everything has to be written down accurately somewhere, and several other people need to be familiar with your work.

It’s all about preparation.

Posted in Development | Tagged , | Comments Off on A Smooth Production Process

Production & Beyond

Since we’re working backward through the development process, the last step in software development is software that’s already deployed. So let’s start there.

When Production Software is going well, and I mean really well, something cool happens: Nothing much at all.

When a release goes off without a hitch, you can immediately turn your attention to the next release. People might relax a bit for a week, but then it’s right back into the action. They are focused and committed to the new features, not looking over their shoulder wondering when the next emergency will happen. With the major exception of any big technology upgrades that may be coming, their productivity more closely resembles the middle of the release than the typical start-of-milestone head scratch that usually occurs. People are being respectful and generous to each other, and good things are already happening.

Something Bad Happens

Even when things go a little sideways, your senior people know the drill. If the bug is severe, it may be all hands on deck, but the plan always looks pretty similar so the conversation is mercifully short. Now is the time for action, not recrimination. Your tools and version control are in good enough shape that several people can get the production version of the entire environment running on test rigs (or their own machine) within 15 minutes. A few minutes after that a set of repro steps are available, and while one or two people start debugging the current version, someone else starts installing older versions to see how old the bug is.

If the fix is trivial, it’s shared between a handful of devs and a few QA people to verify that there are no other new regressions. At the same time, test automation is running. The automated tests are checking the most tedious and difficult to replicate scenarios. At some point enough thumbs up are gathered, and the code is ready to go to production.

If however the fix eludes your best people, at some point the person inspecting older versions will chime in with which version was the one before the bug was introduced. Everyone knows what functionality was introduced since that build, so they can have a discussion of whether rolling back would make things worse or better. The Ops people know (roughly) how to reverse out the changes and get that version out onto the servers if it comes to that. Meanwhile your troubleshooters are looking at all of the commits that have happened since the last good build, trying to narrow down possibilities and form new theories. Once the light clicks on you can go back to the happy path.

Everyone knows more or less how long a redeploy takes, so once the announcement goes out that the fix – or workaround – is going live, everyone sees the light at the end of the tunnel. It may take 5 minutes or half an hour, but there are no more surprises because your build and deploy environment is repeatable.  A couple people babysit the deployment while everyone else gets back to working on the next release.

This may sound fantastical to some, or like a Tuesday to others.  Next up we’ll cover what qualities lead the team to this situation.


Posted in Development, Tools | Tagged , | Comments Off on Production & Beyond

Lean and Agile, in Reverse

I think we have been making a mistake in how we teach and ‘sell’ Agile and Lean processes to the general masses.  The industry has made tremendous strides and advances but the number of Late Adopters is still remarkably high.  I think we may need a different approach to convince those who still aren’t on board in 2014.  And I think the way to do it is to spend a lot more time talking about the end of the process and a lot less talking about the beginning.

In the late 90’s, Agile was a very new thing for some people, unheard of for most of the rest.  The Early Adopters really didn’t need that much convincing.  A little bit of ‘carrot’ and not much ‘stick’ and they were already onboard.  The biggest questions from these folks were about trying to figure out how to sell everybody else on it.

I had the opportunity then to hang out with some pretty sharp Early Adopters.  Both Bill Bohlman (an XP coach/manager) and later David Anderson (Mr Kanban) let me pepper them with questions and answered graciously.  Occasionally the insight flowed both ways.  I had a lot of peers who had a lot of good ideas and we talked about them endlessly.

Why?  We were trying to figure out how to get the people dragging their feet to be enthusiastic about participating in the process.  Trying to grease the wheels.  Sweeten the pot, whatever you want to call it.  The thing is that we honestly weren’t exactly sure what the process was ourselves, so that munged up the works a bit.

It’s 2014 now.  “Refactoring” by Martin Fowler turned 15 years old this summer.  XP won by dissapearing almost entirely.  Most Lean and Agile practitioners just Do XP without even really talking about it.  Many of us know what Agile success feels like and what it looks like.  For the people who still don’t believe, I think we need to talk more about what a well oiled project looks like and work backward from there to first principles.

Over the years I have convinced some, begged others, and browbeat more people than I care to say into just following Agile for a couple of release cycles.  After one release goes off without a hitch, the resistance always drops by about half.  After two releases, people spend their lunch talking about That Guy who still isn’t onboard.  On two projects I heard back six months after I was gone, some new manager tried to convince the team to go back to the old way of doing things, and the team flatly refused.  It’s a great feeling when you’ve given someone something they’re now willing to fight for.  Better than shipping code in my opinion.

On a couple projects where we ‘did Agile’ but with a long release cycle, we had quite a bit of trouble keeping people on task.  I now have a deeper appreciation of why the XP community was so very adamant about short release cycles.  If you don’t get to the end nobody sees what the end looks like.  But maybe we can show them without getting to the end first.  It’s something I spend a lot of my time thinking about lately, and will be posting about in installments.


Posted in Development | Tagged , | Comments Off on Lean and Agile, in Reverse

Branching Out

As a young developer, you probably got some good advice.  Assuredly one of those pieces of advice was to keep challenging yourself.  Branch out.  Learn new programming languages.  Use new libraries.  Don’t let yourself get too comfortable about what you know.

At first it works.  It really works.  But after a while, the insights get fewer, or smaller, or less frequent.  It gets harder to try.  Maybe you even give up.  Does that mean you’re an old fart now that you should take a corporate job and wait for that sweet sweet pension to kick in?

Or are you just overtrained?

There is a concept in sports called Overtraining. It is the long term consequence of thinking that you can get better at a physical skill simply by trying harder.  For a long time such a strategy will work, until it doesn’t, and then it becomes actively harmful.  When someone notices the harm they tell you that you’ve overtrain.

The first thing you do in this case is to slow down, and stop making things worse.  But some think the real antidote is Crosstraining (a fancy word for Doing Something Else).  This involves another complementary sport that uses your body differently.  A cyclist might ice skate, a swimmer might run, or ski.  The theory is that this builds resilience, reducing the chances of injury.  While that’s extremely important, I think it’s something bigger than that, and something we developers also suffer from.

It’s called “being too sure of yourself”.  A new sport is unfamiliar territory.  You’re not the best at it.  You aren’t even good at it.  In point of fact you suck.  Everyone knows and accepts this, especially you. Because of this, you are open to new ideas, new ways of thinking and feeling about the world.  And when you finally return to your comfort zone, you can’t help but take so e of this new information back with you.  And that makes you better.

In some circles this is known as beginner’s mind.  If you have forgotten what that feels like, then I suggest you start a new hobby.  The more uncomfortable the better.

It is amazing how much you can learn about software development by doing something seemingly unrelated.  For example, almost everything important I know I learned while riding a bicycle – an endurance activity requiring perseverance, planning, teamwork, and offering a great deal of time to think, improved blood flow to the brain, and endless stamina.  Just don’t overdo it like I did.

Maybe you’re not a sportsman.  Still plenty of other opportunities.  Juggling has a lot to teach about calmly reacting to deadlines – if you panic you’ve already lost, even if all the balls are still technically in the air.  In some respects cooking works the same way, but with less panic and more planning.  Gardening is an exercise in making plans that will never happen, and constructively accepting and dealing with the consequences of forces outside your control.  Want to get better at algorithms?  Learn the Rubik’s Cube, or a strategy game, like Go (hours to learn, a lifetime to master).

And if you’re horrible at these things?  All the more reason to stay with it.  Humility is another big one that most of us programmers could stand to work on.  You can enjoy doing things that you’re terrible at.  And being comfortable with being uncomfortable will make learning anything new a lot easier.

Posted in Development, Life | Tagged | Comments Off on Branching Out

Passive Testing


You get used to it. I don’t even see the code.

I have wanted a deskful of monitors since I first discovered that such a thing was technologically possible.  I gave up on that idea for a while, but I think it will finally happen.  I never would have guessed that this would be the way.

We all know the workstation market is shrinking, and that in the computer industry marginalized products become expensive products.  We as developers are already collectively moving to laptops, which don’t have the oomph of a workstation, even with the nearly-obligatory SSD.  Laptops are expensive, and can’t have many monitors attached.  In a work environment you’ll probably get one, possibly two.   Even if they offered you a third you wouldn’t be able to use it.

Divide and Conquer tactics have already becoming pretty common, in the form of Continuous Integration.  You get one or two wicked fast machines, you divide all your tests up into chunks that are likely to fail or pass in some pleasing order, and you run them on every commit.  With tools like ‘watch’ or the ‘auto-test’ button (Jetbrains), this is beginning to expand back to your desk, but I think we can do better.

If you do web apps, think about your code-build-test cycle.  It’s probably a bit like mine:

  1. Think about which page is most affected by the change I’m planning to make.
  2. Open it in a bunch of browsers.
  3. Make a change to my code.
  4. Start the unit tests.
  5. Alt-Tab between all the browsers
  6. Hit reload
  7. Poke at it
  8. Did I remember to reload?  Better poke it again to be sure
  9. Fix a quirk in IE a browser that’s having problems
  10. Rerun the tests
  11. Start a commit message
  12. Realize/suspect that I forgot a browser
  13. Repeat steps 5-10
  14. Realize I forgot to check if it works on the other page.
  15. Repeat steps 1-13 on other page
  16. Fix another IE browser issue
  17. Commit
  18. Fix a test I broke because I forgot to rerun the tests after step 16

There is only one word for this process: stupid.

If I could blame it on one person I would, but the fact of the matter is that every one of us has independently or with a little gentle prodding come to this very same development cycle.  We are all (and none) of us to blame.  It’s just how things are.  Since we understand that it sucks so very much, we write a bunch of test automation to make step 7 less arbitrary, error-prone and onerous, but what about all those other steps!?

If you think about it, that’s a awful lot of clicking, and a lot of time wasted on forgetfulness.  It’s an exceedingly interactive form of testing, and this is probably the same observation that underpins Lighttable.  In their tool you have your code running right next to your editor, and you can just glance over at it whenever you want.  In fact it’s kind of hard to avoid looking at it, which is the point.

It’s a really cool idea, but IDEs require extensive feature sets to reach parity with what we already use.  Namely, what will this piece of code look like in four different browsers? How would you event display that in Lighttable? Typically it’s taken tool vendors 6-10 years to reach the kind of feature parity that would make someone dump their existing IDE.  I’m impatient, so let’s try something else. Something more passive, and low-tech.

On top of the steps above, OOCSS actually makes this situation a lot worse, as I mentioned before, because ‘the most affected pages’ is less likely to be 1-3 and more likely to be 3-6 (I would be happy to explain this later, if anyone cares to see my notes).  But if we go with a more passive approach, all of this evaporates in a puff of tooling.

So here is my proposed model.

Earlier in the development process:

  1. Identify functional areas that are expected to ‘work the same’
  2. Generate a fake ‘portfolio’ page filled with page fragments representing all of these areas
  3. Populate it with as many corner cases as fit (long text, Unicode, escaped html, etc)
  4. Put a <meta> refresh header in the top of the fake page.
  5. Optionally add a button to disable the refresh.
  6. Tuck it away someplace in the project that will be omitted from the production builds
  7. Commit

Now when a new feature or bug fix needs to happen:

  1. Tweak the relevant portfolio page to express the behavior change.
  2. Open it in a bunch of browsers.
  3. Make a change to your code.
  4. Start the unit tests.
  5. Alt-Tab between all the browsers
  6. Poke at it
  7. Fix a quirk in IE a browser that’s having problems
  8. Rerun the tests
  9. Start a commit message
  10. Realize you forgot a browser
  11. Repeat steps 5-8
  12. Have enough energy left (hopefully) to remember to rerun the tests
  13. Commit

We’ve moved a big chunk of muscle memory out of the code-build-test loop.  This is a good thing.  An excellent thing in fact, because we are human and we make mistakes all the time.  However you may be thinking to yourself, “hey, he promised me The Matrix at the top of this post,” and I haven’t done anything about that yet.  So here goes:

Most of the time you don’t even need step 6 (ie, poke at it).  You can just visually verify that everything still looks good.  So all you have to do is arrange your browsers so you can glance at them all without lifting a finger – passively.

If you plug your tablet in and set it not to go to sleep, then you can test that as well, and now you have 4+ screens in front of you.  If you can find a Windows box or another laptop, or an older tablet (and who among us doesn’t have those now?) it goes up from there, until you run out of desks, jacks, or outlets.

If anyone tries this out, or has already come to this same conclusion, I’d love to get some feedback on your experiences.


I’m trying to get a pull request into Jasmine to add a refresh button for your JS tests, I’m using this to short-circuit more of the code-build-test cycle. If you like and use Jasmine, please give my pull request an upvote, or try it yourself and let me know if it works for you.

Posted in Development, Tools | Tagged , | Comments Off on Passive Testing

Things I wish I’d known about OOCSS a year ago

One of my favorite interview questions to ask is this:  If you could go back and do your last project over, what would you do differently?  Here is my answer.

18 months ago I went to my UEX lead and my boss with a pitch:  Use Object Oriented CSS to build our new web application. They agreed it sounded like a good idea, so we went forward.  Ultimately that was the right decision I think, and for many of the reasons that Nicole Sullivan will volunteer.

If you don’t know what it is, OOCSS is a form of DRY for front ends.  Websites have repeated patterns in them.  Some of them overt, some subtle.  With a single CSS file for the site, and the same HTML fragment to represent the same concept, you only have to get it working right one time, and then people who don’t live and breathe front ends can be productive while changing the site.  You just compose the existing rules and things happen.

That’s the theory at least.  In practice things are not that rosy.  It’s hard to avoid turning OOCSS into widgets, and every new piece of UI has a tendency to turn into a renegotiation of what the phrase “look the same” means.  The widgets look just a touch different every time you display them.  Over here it’s big.  Over here make it narrow.   Over there add a little bit of extra information.  Over here make it show extra information.  Can’t you just user a bigger icon over in this spot?

It’s not that you can’t do these things, but the Old Way made it easy answer these questions.  What it didn’t answer was “Why don’t they look the same in these two spots?”  OOCSS means that mostly you solve a problem once and it stays solved.  Mostly.

In practice you get regressions.  Different teams fix different bugs, and they break things other people just got done fixing.  It leads to some big frustrations and some hard feelings.

So How Do You Fix It?

You can’t expect people to look in 12 places to make sure they haven’t broken anything.  And you can’t expect people to make things look identical in 12 places.  You can try, but I assure you it won’t work.  So what do you do?

Put all 12 ‘places’ in one spot.

It’s so simple that it seems stupid, in that way that makes the brain slide over it when brainstorming for a solution.  If a repeated meme has subtle variations for different occasions, just manufacture a situation where they all appear contiguous to each other.

Build a fake page with fake data.  Don’t rely on fake data on a real page, that’s too diffuse and hard to maintain.  Build a proper fake ‘portfolio’ page, and express the full range of patterns.  Short text, long text, missing bits, etc.  Put it where only the developers and testers can find it, and expect them to refer to it when modifying the CSS.

Push Back, With Examples

The idea was sound.  Making a million ‘exceptions to the rules’ completely erodes that idea, in ways that are quickly worse than having done nothing at all.  Some people need a picture to explain to them that they’ve become unreasonable.  The thing is that sample pages you just wrote is a powerful demonstration of whether things have gotten ridiculous.

Repeated HTML is better than repeated CSS

One declaration in CSS and one in HTML is too DRY for the real world.  It gets very hard for the developers to understand, and templating engines don’t like deeply nested templates anyway.  Denormalize your HTML when it seems right to do so, but keep a single set of CSS rules.

I’m currently writing some extremely simple tools to facilitate doing just these things.  I’ll keep you posted when I have something code and more screenshots to show for it.

Posted in Development | Tagged | 1 Comment

Go First

I have spent most of my life believing in the power of good leadership.  I was convinced that people were following my boss and I because we were leading.  Now I think it’s simply that most people are just uncomfortable going first, of being the only one.

Any time there is big change in the world, there are a bunch of people who are quick to say that they have always felt that the ‘new way’ was the right thing all along, and they knew it.  Usually we just laugh at this cliche and just call them poseurs, or other less savory names.

But what if they’re right?  Everybody has a million ideas flit through their heads every day.   All the things they want to do.  All the things they should do.  They don’t have time for a tenth of it.  So they ignore most of it and concentrate on things like keeping their children clean, and staying employed.  Much of the rest is dismissed out of hand, because if you stick your neck out, people might make fun of you.  Maybe yell at you.  Maybe fire you.  Too risky.  Take the safe option instead.

But then one day you see a guy minding his own business.  He looks happy.  He doesn’t have a care in the world.  But he’s doing one of those million things you wish you could do.  And nobody is laughing at him, nobody is yelling, and he appears to be employed.  Huh.  Maybe you could do it too.  Was he leading?  Was he serving as an example?  Was he living his values?  No.  He just didn’t wait for permission.  He just did it.

They say a hero is someone who is afraid to do the right thing but does it anyway.  When they interview these people, they almost always say the same thing, “Look, nobody else was doing anything so I just did it.  I had to.  I didn’t think about it.” Often he or she is embarrassed or even dismissive of the attention.  He wasn’t being special or making a statement.  He certainly wasn’t ‘leading’ anyone.  He just did it.

He just went first. That’s all.  And maybe that’s everything.

Posted in Uncategorized | Comments Off on Go First