Bug Triage day results

How did the OpenStack Bug Triage day we organized yesterday go ? Did organizing an event make a difference ? Here are the results !

Nova has more bugs than all the other core projects combined, and the most slack to clean up. We went from 237 “New” bugs at the beginning of the day and down to 42, a completion rate of 82%. In the mean time we managed to close permanently 86 open bugs over a total of 627:

So the BugTriage day definitely made a difference ! Congrats to all the participants ! It leaves our bug tracker in a lot better shape, and created a momentum around bug triaging and having an up-to-date database of known issues.

The success is even more obvious on smaller projects, with Glance, Keystone and Quantum all managing to complete all BugTriage tasks in the day ! See for example the results for Quantum:

See you all for our next BugDay… which will most likely be a Bug Squashing Day (close as many bugs as possible) shortly after folsom-2.

OpenStack BugTriage day

Tomorrow, Thursday June 7th, the OpenStack community will run a BugTriage day. Why are we doing this ? What are we going to do ? How can you participate ?

Bug tracking is an essential part of our development processes. Well-maintained bug lists help us know the current state of our projects better, define bugfixing priorities, and identify milestone and release-critical issues.

The trick is, the bug lists can quickly get unusable if they are not well-maintained. Most of our core projects managed to keep their bug lists relevant and current, but the largest ones (Nova and Swift) allowed some pile-up in the recent months… and that creates a vicious circle: as the bug tracker becomes less relevant, bugs gets even less attention, and things get worse.

BugTriage days are a category of BugDays specifically designed to break this vicious circle. They were discussed at the Folsom Design Summit as a way to improve our bug triaging practice. The idea is to concentrate efforts, for one day, in making our bug tracker relevant again, and start a virtuous circle of maintenance instead of of a vicious circle of abandonment.

How are we going to achieve that ? The BugTriage page on the wiki describes a set of triaging tasks that we should complete. Task one, for example, is about confirming incoming, untouched “New” bugs. The goal is to complete as many tasks as possible. Participants will gather in the #openstack-bugday IRC channel on Freenode. It starts as soon as it’s Thursday somewhere in the world, and will last as long as it’s still Thursday somewhere. We will track the results of our efforts live on pretty graphs, to quantify how well we do.

So please join us tomorrow in that long-overdue Spring cleaning effort, which will go a long way into making Folsom an awesome OpenStack release ! You can read more about the whole event here.

 

A community maturing

A few days after an intense and fruitful OpenStack Design summit, I just recovered enough from jet lag to deliver my impressions in written form. We put a lot of smart people into rooms to discuss various subjects hastily defined while we were busy releasing Essex… and the magic worked again: open collaboration between developers from competing companies, strong but always polite technical discussions, lots of decisions, teams of developers with common interests forming, duplication of effort avoided…

It’s clear that the format (mostly inherited from Ubuntu’s Developers Summits) works very well in our open innovation project: everybody comes with a plan that is open to modifications and the developers are empowered with decision making.  This makes the design summit sessions very appealing to developers, turning them into advocates of our development model in their companies, removing any barriers to contribution that could be left. Being part of the OpenStack community is just pleasant !

However this edition was a bit different from previous ones. There were a lot of signs that our community is maturing. With OpenStack growing, developers can no longer follow every session and give their opinion on every subject: they have to pick their fights, and trust the other developers to come up with the right design in sessions they can’t attend. So sessions had a lot less advice-giving people and a lot more people actually signing up to do work. The topics were much more deployers-oriented and much less about changing to the latest shiny stuff. Even less glamorous sessions like bug triaging, documentation, internationalization or stable branch maintenance saw a lot a participants present, and signing up to help.

People realized that OpenStack is here to stay, and that strategic contributions are necessary for it to reach the final stages of its long-term world domination plans. When did that switch happen ? A graph recently published in the community newsletter shows the change happening a few months into Essex:

As you can see, people used to care about fixing bugs in spikes around release times. But starting around November, 2011, we see the bugfixes curve starting to follow the bugreporting curve more closely.

After the Diablo release I advocated for companies to put their money where their mouth is and start contributing strategically to OpenStack. I’m happy to see that it happened during the Essex cycle, and that the awesome Design Summit we just had confirms that trend.

OpenStack Folsom Design Summit

In a few days the OpenStack developer community will gather in the heart of San Francisco for three days of brainstorming and discussions around the next release cycle of OpenStack projects, code-named “Folsom”.

The Design Summit is a key moment for our open innovation community. This is not a conference with speakers. This is not where a closed developer group announces to the public the changes they intend to push to their private “open source” project. We design, discuss and make decisions at the summit as a community. It’s quite uncommon, and that’s what makes us different.

Our (elected) PTLs have final say in case of unsolvable conflicts, but generally consensus is reached in those face-to-face meetings much more easily than on mailing-lists. That’s why this is a critical moment, and we need to make the best use of this short time together. Connect with other people interested to solve the same issues, avoid duplication of work, and collaborate with developers from all those different companies on making OpenStack awesome.

We have great brainstorming topics for those three days. Most tracks already have a tentative schedule posted at
http://folsomdesignsummit2012.sched.org/
, although it’s still subject to scheduling changes. If you have a new idea for a session, it’s too late to get in the official tracks, but we provide an Unconference room for talks that could not fit in the tracks, last-minute ideas and continuation of discussions. And since we like to talk about random stuff that matters to us, we will also have 5-min lightning talks every day after lunch.

Session leads should take the time to view Jim Plamondon’s training video, it’s a great introduction on how to make the most of a session you lead. I hope to meet all of you in person next week !

 

Ask not what OpenStack can do for you…

Over the last months I’ve seen more and more tweets and news articles using the formulation “OpenStack should”, as in “OpenStack should support Amazon APIs since it’s the de-facto standard”. I think there is a fundamental misconception there and I’d like to address it.

As a quick aside (and contrary to what the twittersphere sometimes report), it should be noted that OpenStack Nova always supported the Amazon EC2 API, and that OpenStack Swift grew an Amazon S3 compatibility layer last year. That said, I’ll be the first to admit that one could rightfully claim that the AWS API support in OpenStack is in less better shape than the OpenStack API support. But the reason behind it is not some “OpenStack strategy”, it’s a reflection of the participating companies focus.

OpenStack is a true Open Innovation project. It’s a collaboration ground where multiple companies are free to invest development resources to care about the stuff that is important to them. It’s an influence game where you need to donate developers to play: OpenStack is the playing field, not the players that push the ball.

Red Hat cared about QPID support, they fielded developers to make it happen in OpenStack. EC2 API support is originally in Nova because NASA cared about it. Then with the increase of Rackspace’s influence on the project, the OpenStack API grew faster. Now with Canonical (and others) interest, Amazon’s API support is getting better. Ultimately, code talks, and you can make things happen. That’s what makes OpenStack so appealing but also so confusing to the industry.

As “OpenStack”, we need to make sure the playing field is level (and hopefully the Foundation will be set up soon enough to address that) and that the code is modular and welcoming. But it’s up to the participating companies, which throw development resources at the project, to invest in what’s important for them or their customers. And maintain it over the long run.

So whenever you say “OpenStack should”, ask yourself if you shouldn’t really be saying… [Rackspace, Cisco, HP, IBM, Red Hat...] should. Ask not what OpenStack can do for you. Ask what you can do for OpenStack.

OpenStack Essex: the last mile

At the time I’m writing this,  we have final release candidates published for all the components that make up OpenStack 2012.1, codenamed “Essex”:

• OpenStack Compute (Nova), at RC3
• OpenStack Image Service (Glance), at RC3
• OpenStack Identity (Keystone), at RC2
• OpenStack Dashboard (Horizon), at RC2
• OpenStack Storage (Swift) at version 1.4.8

Unless a critical, last-minute regression is found today in these proposed packages, they should make up the official OpenStack 2012.1 release tomorrow ! Please check out those tarballs for a last check, and don’t hesitate to ping us on IRC (#openstack-dev @ Freenode) or file bugs (tagged essec-rc-potential) if you think you can convince us to reroll.

Those six months have been a long ride, with 139 features added and 1650 bugs fixed, but this is the last mile.

The road to OpenStack Essex release candidates

Since the beginning of March, OpenStack developers are focusing on testing and bugfixes, with the objective of producing a release candidate for each project.

Regular readers of my blog know I’m not the last one to complain when I feel developers don’t care about the release and don’t participate to that critical sequence of the cycle. I’m happy to report that the engagement of developers around making Essex good is overwhelming. Driven by technical leads that understand the challenges, significant groups of old and new developers are participating, testing, assigning themselves to bugs reported by others and fixing them in record time. As project-focused groups, I think we are quickly maturing.

So, how far are we from Essex today ? The final release is set to April 5th, which means each core project needs to produce its final release candidate before then. As of today no project did produce a release candidate yet.

Swift is expected to release its final Essex version (1.4.8) on March 22. This version will be included in OpenStack Essex release, unless a critical regression is detected in it.

Keystone (which every other project depends on) is still struggling with 9 RC bugs, including some key decisions to be made on configuration handling. This is the hot spot right now: if you have free cycles, please consider asking Joe Heck (heckj on IRC) how you can best help the Keystone crew. I’d really like to have an RC1 for that project by Thursday next week.

Glance also looks still a bit far away, with 5 RC bugs listed and slow progress on them. I still hope Glance can be ready by Tuesday next week though.

Nova is looking quite good, with 2 RC bugs left on the list. The trick with Nova is that regressions and critical issues can hide in dark corners of its 120k lines of code, so the focus is really on finding the remaining issues, filing and targeting them. I expect we’ll be able to publish RC1 on Monday or Tuesday next week.

Horizon is almost ready (3 RC bugs left), waiting for a few fixes to land in other projects before it can issue its RC1. It should be out early next week.

Once all projects release their RC1, the hunt for the overlooked release-critical issue will be on. It will be time to put the proposed release to the test. In order to limit potential last-minute regressions, only showstoppers will warrant a respin of the release candidate, other bugs will be listed in the Known Bugs section of the release notes. You can use the “essex-rc-potential” tag to mark bugs that you think should be fixed before we release Essex.

Let’s all make it rock !

Open development, releases and quality

Every 6 months, as a cycle ends and we prepare the next, we look back at our release model and try to see how we can improve it. My opinion is that we need (once more) to evolve it, and here is why.

Objectives and past evolutions

Our main objective is to produce stable and usable software. Our secondary objective is let stable new features reach the hands of our users in a timely manner. Our tertiary objective is to empower developers to work efficiently on the features and improvements of tomorrow. In simple solutions, those three objectives are not really compatible, and in the past we tried to make adjustments without acknowledging the fundamental incompatibility between those objectives.

The Austin-Cactus system was a 3-month cycle with various freezes. The issues with this system were that it took too long to get features in the hands of testers and users, developers had trouble working during the frozen periods, and not so many people actually helped during the QA period.

For Diablo, we decided to switch to a 6-month cycle with monthly milestones. Those milestones were supposed to address the “get features into the hands of users early” issue. To empower developers, we introduced an almost-always-open trunk: we only had 2 weeks of feature freeze before the Essex branch opened (and coexisted with the Diablo release branch). The problem was the resulting quality was not good, since we accumulated 6 months’ worth of features and only had 2 weeks’ worth of QA on them.

So for Essex, there was a decision that each project would decide where to place its feature freeze and its first release candidate (which would be when the Folsom branch opens). Most of them decided to use the essex-3 milestone as a soft feature freeze and essex-4 as a hard one, which reintroduced the “closed trunk” issue. And the work needed to stabilize 6 months’ worth of features is still daunting. And we saw lots of feature freeze exceptions for last-minute “almost there” features that can’t afford to wait another 6 months.

So what’s the solution ? Is there a good way to (at the same time) reduce the pain of “missing a release”, have always-open development branches, and get higher quality ?

The kernel model

The only way to reduce the pain of missing a release is to have shorter time-based releases. The only way to have always-open development branches without sacrificing release quality is to separate them completely from release branches. Where do those truisms lead us ?

Another famous open innovation project has already been there, and that’s the Linux kernel. Development happens in various always-open topic branches, and regularly a merge window opens to propose stable features for inclusion in the mainline kernel.

If we manage to efficiently separate OpenStack in topics, we could adopt the same model. We could have several topic branches where core developers on a specific area can collaborate and review code affecting that area. We could have frequent releases (every 6-8 weeks ?), and for each release we could have a “merge window” where stuff from topic branches can be proposed for release, if deemed stable enough. Between the moment the merge window closes and the moment the final release is cut, various release candidates can be produced, on which serious QA can be unleashed without blocking other developers.

This solves all objectives. If a feature is not ready yet (according to the team maintaining the topic branch or according to release management), it can bake until the next merge window, which is not far away. Regular releases ensure that improvements reach the users in a timely manner. Development branches are always open. And with a reasonable amount of new code in every release, QA work is facilitated, theoretically resulting in higher release quality.

Additional benefits

Splitting development into topics has an additional benefit: smaller groups developing an expertise on a specific area make better reviewers on that specific area than a random nova-core developer that can’t be an expert in all things Nova. I’d say that each topic should be small enough to be manageable with a team of 6 core reviewers at the maximum. Of course, nothing prevents anyone from being active on multiple topics if they wish.

More frequent releases also allows you to set themes for each release. It’s difficult to refuse some feature in a 6-month cycle, but it’s easy to delay a feature to the next 6-week cycle. In the same way some kernel releases introduce large architectural changes and some others are more geared towards performance improvements or stability, we could also define themes for every release — after all the next one is not so far away.

Challenges

There are a few issues with this model obviously. Code needs to be componentized enough so that merge pains can be limited. Changes from every release need to be merged back into topic branches. Code needs to be clearly separated into a set of relevant topics, each with its own set of core reviewers maintaining the topic branch. The release team must be staffed enough to be able to review proposed code for stability. Bug fixes need to easily end up in the release branch. And how do those “releases” match the 6-month period between summits ? What means “Folsom” in that respect ?

I hope we can use the following weeks to discuss the devil in the details of this possible evolution, and be ready to take a decision when the time comes for us to gather at the Folsom Design summit.

FOSDEM 2012 feedback

I’m back from Brussels, where happened the coldest FOSDEM ever. It started on Friday night with the traditional beer event. Since the Delirium was a bit small to host those thousands of frozen geeks, the FOSDEM organizers had enlisted the whole block as approved bars !

On the Saturday, I spent most of my time in the Cloud and Virtualization devroom, which I escaped only to see Simon Phipps announce the new membership-based OSI, and Paolo Bonzini talking about the KVM ecosystem (in a not technical enough way, IMO). My own OpenStack talk was made a bit difficult due to the absence of mike to cover the 550-seat Chavanne auditorium… but the next talks got one. The highlight of the day was Ryan Lane’s “infrastructure as an open source project” presentation, about how Wikimedia Labs uses Git, Gerrit, Jenkins and OpenStack to handle its infrastructure like a contributor-driven open source project. The day ended with a good and frank discussion between OpenStack developers, with upstream projects and downstream distributions.

On Sunday I tried to hop between devrooms, but in a lot of cases the room was full and I couldn’t enter, so I spent more time in the hallway track. I enjoyed Soren’s talk about using more prediction algorithms (instead of simple thresholds) in monitoring systems, introducing his Surveilr project. The highlight of the day was Dan Berrangé’s  talk about using libvirt to run sandboxed applications, using virt-sandbox. There are quite a few interesting uses for this, and the performance penalty sounds more than acceptable.

Overall it was a great pleasure for me to attend FOSDEM this year. Congratulations to the organizers again. I’ll be back next year, hopefully it will be warmer

About collaboration

In recent years, as open source becomes more ubiquitous, I’ve seen a new breed of participants appearing. They push their code to GitHub like you would wear a very visible good behavior marketing badge. They copy code from multiple open source projects, modify it, but don’t contribute back their changes to upstream. They seem to consider open source as a trendy all-you-can-eat buffet combined with a cool marketing gimmick.

In my opinion, this is not what open source is about. I see open source, and more generally open innovation (which adds open design, open development and open community), as a solution for the future. The world is facing economical and ecological limits: it needs to stop designing for obsolescence, produce smarter, reduce duplication of effort, and fix the rift between consumers and producers. Open innovation encourages synergy and collaboration. It reduces waste. It enables consumers to be producers again. That’s a noble goal, but without convergence, we can’t succeed.

The behavior of these new participants goes against that. I call this the GitHub effect: you encourage access to the code, forking and fragmentation, while you should encourage convergence and collaboration on a key repository. And like having a “packaging made from recyclable materials” sign on your product doesn’t make it environment-friendly, just publishing your own code somewhere under an open source license doesn’t really make it open.

On the extreme fringe of that movement, we also see the line with closed source blurring. Building your own closed product on top of open source technology, and/or abusing the word “Open” to imply that all you do is open source, using the uncertainty to reap easy marketing benefits. I’ve even seen a currently-closed-source project being featured as an open source project to watch in 2012. We probably need to start playing harder, denounce fake participants and celebrate good ones.

Some people tell me my view goes against making money with open source. That might be true for easy, short-term money. But I don’t think you need to abuse open source to make money out of it. The long-term benefits of open innovation are obvious, and like for green businesses, good behavior and long-term profit go well together. Let’s all make sure we encourage collaboration and promote the good behavior, and hopefully we’ll fix this.