Home > Open source, Openstack > The next step for OpenStack

The next step for OpenStack

September 28, 2011 Leave a comment Go to comments

Just after a release, discovery of significant bugs always revives discussion around the need for maintenance branches or point releases. Those discussions, however, are not solving the root cause for the issue, but merely try to do damage control on the consequences.

The root cause for presence of significant bugs in a given release is not the presence or absence of maintenance branches. It’s not about the choice of time-based cycles, or the length of it. It’s about lack of focus on testing and fixing the release deliverables. If only a few people work on that, while all the others are busy adding new features in trunk, delaying your release by one or more weeks won’t change anything.

From tactical to strategic contributions

OpenStack is one of the few open source projects where development is truly shared across multiple companies. The trick is, most companies involved so far are doing what I call tactical contributions: adding a feature that they care about, fix bugs that affect them. Tactical contributions are great to expand a project scope, community and mindshare, however they add technical debt. Companies involved need to move to what I call strategic contributions: funding development resources that care about the end result, the release deliverables, the absence of bugs, the coherence of the features.

The obvious comparison point is the Linux kernel. The reason why it’s successful, despite lots of companies only involved in tactical contributions, is that at its core it has a strong group of key developers whose primary allegiance goes to the Linux kernel itself, no matter what company they happen to work for. Those companies understood the necessity of funding strategic contributions.

Currently, especially in Nova, it’s quite difficult to get merge proposals reviewed, random bugs fixed, integration tests contributed, or holes in scope covered. That’s because most groups are focused on their own objectives, rather than the common project objectives. That’s the mindset we need to change now, and that’s the only thing that can give us better releases.

The cost of strategic contributions

The problem with strategic contributions is that they are typically more costly than tactical contributions, which have a more obvious return on investment. Accepting to have developers on payroll “fixing what needs to be fixed”, or giving 30% free time to all your developers so that they can work on project objectives rather than only your own is not that easy. But OpenStack has now proven that it’s here to stay, lots of companies have now bet their strategy on it, so I think the time is now.

If we don’t adjust, OpenStack in general (and Nova in particular) will crumble under the technical debt of tactical contributions, and everyone involved will lose. We might need to adjust governance to encourage other companies to invest long-term in project-centered resources. We’ll need to set up open, multi-company workgroups (like the recently-setup QA team) to clearly show that it’s a common effort. It won’t happen in a day, but if we don’t change our mindset now, no matter how we adjust the release cycle, Essex deliverables will be of the same quality as Diablo.

Categories: Open source, Openstack
  1. September 28, 2011 at 15:56

    Don’t forget the community cost of “stabilising phases” of development. Mozilla, when they started looking, were surprised to discover that after code freezes in their trunk branch, they lost developers – not just for the period of the freeze, but forever. Can’t find the reference to the presentation/article right now, I’m afraid 😦

    • Thierry Carrez
      October 1, 2011 at 13:03

      That doesn’t surprise me. For Cactus we had a long freeze, for Diablo we had an “always-open” trunk with a short soft feature freeze. In both cases it didn’t really attract more people towards the QA/bugfixing space. Like I said, we need more a mindset change than a cycle adjustment.

  2. September 28, 2011 at 16:08

    +1, As individual companies, we have interest in our “tactical” contributions, but the larger strategic investment we each are making in OpenStack critically depends on our ability as a community to release a production-ready system that serves our customers. This will require us all to see the real value in “fixing what needs to be fixed”, reviewing code, and contributing to the overall QA and documentation efforts. This is the only way to help ensure the overall success of OpenStack.

  3. October 2, 2011 at 00:50

    Great summary Thierry! I agree wholeheartedly with you on this. We’ll be dedicating some resources to the OpenStack continue integration project for this very reason.

  4. October 2, 2011 at 15:31

    Couldn’t agree with you more, Thierry. The OpenStack QA team (now 28 members) is a good example of this cross-company type of team (and mindset!) that needs to be embraced by the OpenStack contributor community at large.

    I’m hoping that this new QA team will be enabled to take charge of efforts like cross-project integration testing, stress and capacity testing, and driving a philosophy that “if it ain’t working the way the spec says it should, it ain’t workin!”.

  5. October 4, 2011 at 01:00

    I’ve been on this project for about two weeks and this lack of drive for major refactoring is one of the first things I have noticed. We need to start a discussion about both the architecture of the nova codebase, and the code quality within it. I’ve had several conversations with my coworkers about wanting to make some major refactorings to the data model and the architecture of how we access the data stores.

    One of the big problems in making these changes is other to buy into the improvements. Any major change will cause merge, and understanding conflicts. Including contention about the ‘right way’ and what merits will come from making the codebase easier to understand, and therefor easier to change without introducing unintended side effects.

    • Thierry Carrez
      October 4, 2011 at 02:26

      Re: One of the big problems in making these changes is other to buy into the improvements
      That’s why having face-to-face summits every 6 months is so important: so that we can get easy buy-in to complex changes.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s