Back from a (almost) entirely-offline week vacation, a lot of news were waiting for me. A full book was written. OpenStack projects graduated. An Ubuntu rolling release model was considered. But what grabbed my attention was the announcement of UDS moving to a virtual event. And every 3 months. And over two days. And next week.
As someone who attended all UDSes (but one) since Prague in May 2008, as a Canonical employee then as an upstream developer, that was quite a shock. We all have fond memories and anecdotes of stuff that happened during those Ubuntu developer summits.
What those summits do
For those who never attended one, UDS (and the OpenStack Design Summits that were modeled after them) achieve a lot of goals for a community of open source developers:
- Celebrate recent release, motivate all your developer community for the next 6 months
- Brainstorm early ideas on complex topics, identify key stakeholders to include in further design discussion
- Present an implementation plan for a proposed feature and get feedback from the rest of the community before starting to work on it
- Reduce duplication of effort by getting everyone working on the same type of issues in the same room and around the same beers for a few days
- Meet in informal settings people you usually only interact with online, to get to know them and reduce friction that can build up after too many heated threads
This all sounds very valuable. So why did Canonical decide to suppress UDSes as we knew them, while they were arguably part of their successful community development model ?
Who killed UDS
The reason is that UDS is a very costly event, and it was becoming more and more useless. A lot of Ubuntu development happens within Canonical those days, and UDS sessions gradually shifted from being brainstorming sessions between equal community members to being a formal communication of upcoming features/plans to gather immediate feedback (point  above). There were not so many brainstorming design sessions anymore (point  above, very difficult to do in a virtual setting), with design happening more and more behind Canonical curtains. There is less need to reduce duplication of effort (point  above), with less non-Canonical people starting to implement new things.
Therefore it makes sense to replace it with a less-costly, purely-virtual communication exercise that still perfectly fills point , with the added benefits of running it more often (updating everyone else on status more often), and improving accessibility for remote participants. If you add to the mix a move to rolling releases, it almost makes perfect sense. The problem is, they also get rid of points  and . This will result in a even less motivated developer community, with more tension between Canonical employees and non-Canonical community members.
I’m not convinced that’s the right move. I for one will certainly regret them. But I think I understand the move in light of Canonical’s recent strategy.
What about OpenStack Design Summits ?
Some people have been asking me if OpenStack should move to a similar model. My answer is definitely not.
When Rick Clark imported the UDS model from Ubuntu to OpenStack, it was to fulfill one of the 4 Opens we pledged: Open Design. In OpenStack Design Summits, we openly debate how features should be designed, and empower the developers in the room to make those design decisions. Point  above is therefore essential. In OpenStack we also have a lot of different development groups working in parallel, and making sure we don’t duplicate effort is key to limit friction and make the best use of our resources. So we can’t just pass on point . With more than 200 different developers authoring changes every month, the OpenStack development community is way past Dunbar’s number. Thread after thread, some resentment can build up over time between opposed developers. Get them to informally talk in person over a coffee or a beer, and most issues will be settled. Point  therefore lets us keep a healthy developer community. And finally, with about 20k changes committed per year, OpenStack developers are pretty busy. Having a week to celebrate and recharge motivation batteries every 6 months doesn’t sound like a bad thing. So we’d like to keep point .
So for OpenStack it definitely makes sense to keep our Design Summits the way they are. Running them as a track within the OpenStack Summit allows us to fund them, since there is so much momentum around OpenStack and so many people interested in attending those. We need to keep improving the remote participation options to include developers that unfortunately cannot join us. We need to keep doing it in different locations over the world to foster local participation. But meeting in person every 6 months is an integral part of our success, and we’ll keep doing it.
Next stop is in Portland, from April 15 to April 18. Join us !
Mark’s recent blogpost on Raring community skunkworks got me thinking. I agree it would be unfair to spin this story as Canonical/Ubuntu switching to closed development. I also agree that (as the damage control messaging was quick to point out) inviting some members of the community to participate in closed development projects is actually a step towards more openness rather than a step backwards.
That said, it certainly is making the “closed development” option more official and organized, which is not a step in the right direction in my opinion. It reinforces it as a perfectly valid option, while I would really like it to be an exception for corner cases. So at this point, it may be useful to insist a bit on the benefits of open development, and why dropping them might not be that good of an idea.
Open Development is a transparent way of developing software, where source code, bugs, patches, code reviews, design discussions, meetings happen in the open and are accessible by everyone. “Open Source” is a prerequisite of open development, but you can certainly do open source without doing open development: that’s what I call the Android model and what others call Open behind walls model. You can go further than open development by also doing “Open Design”: letting an open community of equals discuss and define the future features your project will implement, rather than restricting that privilege to a closed group of “core developers”.
Open Development allows you to “release early, release often” and get the testing, QA, feedback of (all) your users. This is actually a good thing, not a bad thing. That feedback will help you catch corner cases, consider issues that you didn’t predict, get outside patches. More importantly, Open Development helps lowering the barrier of entry for contributors to your project. It blurs the line between consumers and producers of the software (no more “us vs. them” mentality), resulting in a much more engaged community. Inviting select individuals to have early access to features before they are unveiled sounds more like a proprietary model beta testing program to me. It won’t give you the amount of direct feedback and variety of contributors that open development gives you. Is the trade-off worth it ?
How much as I dislike the Android model, I understand that the ability for Google to give some select OEMs a bit of advance has some value. Reading Mark’s post though, it seems that the main benefits for Ubuntu are in avoiding early exposure of immature code and get more splash PR effect at release time. I personally think that short-term, the drop in QA due to reduced feedback will offset those benefits, and long-term, the resulting drop in community engagement will also make this a bad trade-off.
In OpenStack, we founded the project on the Four Opens: Open Source, Open Development, Open Design and Open Community. This early decision is what made OpenStack so successful as a community, not the “cloud” hype. Open Development made us very friendly to new developers wanting to participate, and once they experienced Open Design (as exemplified in our Design Summits) they were sold and turned into advocates of our model and our project within their employing companies. Open Development was really instrumental to OpenStack growth and adoption.
In summary, I think Open Development is good because you end up producing better software with a larger and more engaged community of contributors, and if you want to drop that advantage, you better have a very good reason.
At the time I’m writing this, we have final release candidates published for all the components that make up OpenStack 2012.1, codenamed “Essex”:
- OpenStack Compute (Nova), at RC3
- OpenStack Image Service (Glance), at RC3
- OpenStack Identity (Keystone), at RC2
- OpenStack Dashboard (Horizon), at RC2
- OpenStack Storage (Swift) at version 1.4.8
Unless a critical, last-minute regression is found today in these proposed packages, they should make up the official OpenStack 2012.1 release tomorrow ! Please check out those tarballs for a last check, and don’t hesitate to ping us on IRC (#openstack-dev @ Freenode) or file bugs (tagged essec-rc-potential) if you think you can convince us to reroll.
Those six months have been a long ride, with 139 features added and 1650 bugs fixed, but this is the last mile.
I’m back from Brussels, where happened the coldest FOSDEM ever. It started on Friday night with the traditional beer event. Since the Delirium was a bit small to host those thousands of frozen geeks, the FOSDEM organizers had enlisted the whole block as approved bars !
On the Saturday, I spent most of my time in the Cloud and Virtualization devroom, which I escaped only to see Simon Phipps announce the new membership-based OSI, and Paolo Bonzini talking about the KVM ecosystem (in a not technical enough way, IMO). My own OpenStack talk was made a bit difficult due to the absence of mike to cover the 550-seat Chavanne auditorium… but the next talks got one. The highlight of the day was Ryan Lane’s “infrastructure as an open source project” presentation, about how Wikimedia Labs uses Git, Gerrit, Jenkins and OpenStack to handle its infrastructure like a contributor-driven open source project. The day ended with a good and frank discussion between OpenStack developers, with upstream projects and downstream distributions.
On Sunday I tried to hop between devrooms, but in a lot of cases the room was full and I couldn’t enter, so I spent more time in the hallway track. I enjoyed Soren’s talk about using more prediction algorithms (instead of simple thresholds) in monitoring systems, introducing his Surveilr project. The highlight of the day was Dan Berrangé’s talk about using libvirt to run sandboxed applications, using virt-sandbox. There are quite a few interesting uses for this, and the performance penalty sounds more than acceptable.
Overall it was a great pleasure for me to attend FOSDEM this year. Congratulations to the organizers again. I’ll be back next year, hopefully it will be warmer
In recent years, as open source becomes more ubiquitous, I’ve seen a new breed of participants appearing. They push their code to GitHub like you would wear a very visible good behavior marketing badge. They copy code from multiple open source projects, modify it, but don’t contribute back their changes to upstream. They seem to consider open source as a trendy all-you-can-eat buffet combined with a cool marketing gimmick.
In my opinion, this is not what open source is about. I see open source, and more generally open innovation (which adds open design, open development and open community), as a solution for the future. The world is facing economical and ecological limits: it needs to stop designing for obsolescence, produce smarter, reduce duplication of effort, and fix the rift between consumers and producers. Open innovation encourages synergy and collaboration. It reduces waste. It enables consumers to be producers again. That’s a noble goal, but without convergence, we can’t succeed.
The behavior of these new participants goes against that. I call this the GitHub effect: you encourage access to the code, forking and fragmentation, while you should encourage convergence and collaboration on a key repository. And like having a “packaging made from recyclable materials” sign on your product doesn’t make it environment-friendly, just publishing your own code somewhere under an open source license doesn’t really make it open.
On the extreme fringe of that movement, we also see the line with closed source blurring. Building your own closed product on top of open source technology, and/or abusing the word “Open” to imply that all you do is open source, using the uncertainty to reap easy marketing benefits. I’ve even seen a currently-closed-source project being featured as an open source project to watch in 2012. We probably need to start playing harder, denounce fake participants and celebrate good ones.
Some people tell me my view goes against making money with open source. That might be true for easy, short-term money. But I don’t think you need to abuse open source to make money out of it. The long-term benefits of open innovation are obvious, and like for green businesses, good behavior and long-term profit go well together. Let’s all make sure we encourage collaboration and promote the good behavior, and hopefully we’ll fix this.
Next week, the European free and open source software developers will converge to Brussels for FOSDEM. We took this opportunity to apply for an OpenStack developers gathering in the Virtualization and Cloud devroom.
At 6pm on Saturday (last session of the day), in the Chavanne room, we will have a one-hour town hall meeting. If you’re an existing OpenStack contributor, a developer considering to join us, an upstream project developer, a downstream distribution packager, or just curious about OpenStack, you’re welcome to join us ! I’ll be there, Stefano Maffulli (our community manager) will be there, and several OpenStack core developers will be there.
We’ll openly discuss issues and solutions about integration with upstream projects, packaging, governance, development processes, community or release cycles. In particular, we’ll have a distribution panel where every OpenStack distribution will be able to explain how they support OpenStack and discuss what we can improve to make things better for them.
And at the end of the session we can informally continue the discussion around fine Belgian beers or their famous Carbonade !
As we pass the middle of the Essex development cycle, questions about the solidity of this release start to pop up. After all, the previous releases were far from stellar, and with more people betting their business on OpenStack we can’t really afford another half-baked release.
Common thinking (mostly coming from years of traditional software development experience) is that we shouldn’t release until it’s ready, or good enough, and calls early for pushing back the release dates. This assumes the issue is incidental: that we underestimated the time it would take our finite team of internal developers working on bugs to reach a sufficient level of quality.
OpenStack, being an open source project produced by a large community, works differently. We have a near-infinite supply of developers. The issue is, unfortunately, more structural than incidental. The lack of solidity for a release comes from:
- Lack of focus on generic bugfixes. Developers should work on fixing bugs. Not just the ones they filed or the ones blocking them in their feature-adding frenzy. Fixing identified, targeted, known issues. The bugtracker is full of them, but they don’t get attention.
- Not enough automated testing to efficiently catch regressions. Even if everyone was working on bug fixes, if half your fixes end up creating a set of regressions, then there is no end to it.
- Lack of bug triaging resources. Only a few people work on confirming, triaging and prioritizing the flow of incoming bugs. So the bugs that need the most attention are lost in the noise.
For the Diablo cycle, we had less than a handful of people focused on generic bugfixing. The rest of our 150+ authors were busy working on something else. Pushing back the release for a week, a month or a year won’t help OpenStack solidity if the focus doesn’t switch. And if our focus switches, then there will be no need for a costly release delay.
Acting now to make Essex a success
During the Essex cycle, our Project Technical Leads have done their share of the work by using a very early milestone for their feature freeze. Keystone, Glance and Nova will freeze at Essex-3, giving us 10 weeks for bugfixing work (compared to the 4 weeks we had for Diablo). Now we need to take advantage of that long period and really switch our mindset away from feature development and towards generic bug fixing.
Next week we’ll hit feature freeze, so now is the time to switch. If we could:
- have some more developers working on increasing our integration and unit test coverage
- have the rest of the developers really working on generic bug fixing
- have very active core reviewers that get more anal-retentive as we get closer to release, to avoid introducing regressions that would not be caught by our automated tests
…then I bet that it will lead to a stronger release than any delaying of the release could give you. Note that we’ll also have a bug squashing day on February 2 that will hopefully help us getting on top of old, deprecated and easy fixes, and give us a clear set of targets for the rest of the cycle.
That’s on our ability to switch our focus that hinges the quality of future OpenStack releases. That’s on what we’ll be judged. The world awaits, and the time is now.