Home > Open source, Openstack, Ubuntu, Ubuntu Server > Making more solid OpenStack releases

Making more solid OpenStack releases

January 18, 2012 Leave a comment Go to comments

As we pass the middle of the Essex development cycle, questions about the solidity of this release start to pop up. After all, the previous releases were far from stellar, and with more people betting their business on OpenStack we can’t really afford another half-baked release.

Common thinking (mostly coming from years of traditional software development experience) is that we shouldn’t release until it’s ready, or good enough, and calls early for pushing back the release dates. This assumes the issue is incidental: that we underestimated the time it would take our finite team of internal developers working on bugs to reach a sufficient level of quality.

OpenStack, being an open source project produced by a large community, works differently. We have a near-infinite supply of developers. The issue is, unfortunately, more structural than incidental. The lack of solidity for a release comes from:

  • Lack of focus on generic bugfixes. Developers should work on fixing bugs. Not just the ones they filed or the ones blocking them in their feature-adding frenzy. Fixing identified, targeted, known issues. The bugtracker is full of them, but they don’t get attention.
  • Not enough automated testing to efficiently catch regressions. Even if everyone was working on bug fixes, if half your fixes end up creating a set of regressions, then there is no end to it.
  • Lack of bug triaging resources. Only a few people work on confirming, triaging and prioritizing the flow of incoming bugs. So the bugs that need the most attention are lost in the noise.

For the Diablo cycle, we had less than a handful of people focused on generic bugfixing. The rest of our 150+ authors were busy working on something else. Pushing back the release for a week, a month or a year won’t help OpenStack solidity if the focus doesn’t switch. And if our focus switches, then there will be no need for a costly release delay.

Acting now to make Essex a success

During the Essex cycle, our Project Technical Leads have done their share of the work by using a very early milestone for their feature freeze. Keystone, Glance and Nova will freeze at Essex-3, giving us 10 weeks for bugfixing work (compared to the 4 weeks we had for Diablo). Now we need to take advantage of that long period and really switch our mindset away from feature development and towards generic bug fixing.

Next week we’ll hit feature freeze, so now is the time to switch.  If we could:

  • have some more developers working on increasing our integration and unit test coverage
  • have the rest of the developers really working on generic bug fixing
  • have very active core reviewers that get more anal-retentive as we get closer to release, to avoid introducing regressions that would not be caught by our automated tests

…then I bet that it will lead to a stronger release than any delaying of the release could give you. Note that we’ll also have a bug squashing day on February 2 that will hopefully help us getting on top of old, deprecated and easy fixes, and give us a clear set of targets for the rest of the cycle.

That’s on our ability to switch our focus that hinges the quality of future OpenStack releases. That’s on what we’ll be judged. The world awaits, and the time is now.

  1. Leandro
    January 20, 2012 at 02:35

    Excellent post, I agree with you 100% about that main Openstack problem is the lack of stability. Openstack developers are very smart people and do an excellent work, but develop process is so “fast and furious” that many bugs are introduced. By the way, documentation needs to be improved too. However, I love Openstack: is very featured, very powerful and a truly open project.

    Thank you very much for this post.

    Best regards,

  1. January 20, 2012 at 22:45
  2. February 8, 2012 at 19:32

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s