Yesterday we entered the Icehouse development cycle Feature Freeze. But with the incredible growth of the OpenStack development community (508 different contributors over the last 30 days, including 101 new ones !), I hear a lot of questions about it. I’ve explained it on various forums in the past, but I figured it couldn’t hurt to write something a bit more definitive about it.
Those are valid questions. Why freeze features ? That sounds very anti-agile. Isn’t our test-centric development model supposed to protect us from regressions anyway ? Let’s start with what feature freeze is not. Feature freeze should only affect the integrated OpenStack release. If you don’t release (i.e. if you don’t special-case certain moments in the development), then feature freezing makes little sense. It’s also not a way to punish people who failed to meet a deadline. There are multiple reasons that a given feature will miss a deadline, and most of those are not the fault of the original author of the feature. We do time-based releases, so some features and some developers will necessarily be caught on the wrong side of the fence at some point and need to wait for the next boat. It’s an artifact of open innovation projects.
Feature freeze (also known as “FF”) is, obviously, about stopping adding new features. You may think of it as artificially blocking your progress, but this has a different effect on other people:
- As was evidenced by the Icehouse cycle, good code reviewers are a scarce resource. The first effect of feature freeze is that it limits the quantity of code reviews and make them all about bugfixes. This lets reviewers concentrate on getting as many bugfixes in as possible before the “release”. It also helps developers spend time on bugfixes. As long as they can work on features, their natural inclination (or their employer orders) might conflict with the project interest at this time in the cycle, which is to make that point in time we call the “release” as bug-free as possible.
- From a QA perspective, stopping the addition of features means you can spend useful time testing “in real life” how OpenStack behaves. There is only so much our automated testing will catch. And it’s highly frustrating to spend time testing software that constantly changes under you.
- QA is not the only group that needs to catch up. For the documentation team, the I18N team, feature freeze is essential. It’s difficult to write documentation if you don’t know what will be in the end product. It’s frustrating to translate strings that are removed or changed the next day.
- And then you have all the downstream consumers of the release that can use time to prepare it. Packagers need software that doesn’t constantly change and add dependencies, so that they can prepare packages for OpenStack projects that are released as close to our release date as possible. The marketing team needs time to look into what was produced over the cycle and arrange it in key messages to communicate to the outside world at release time.
- Finally, for release management, feature freeze is a tool to reduce risk. The end goal is to avoid introducing an embarassing regression just before release. By gradually limiting the impact of what we accept in the release branch (using feature freeze, but also using the RC dance that comes next), we try our best to prevent that.
For all these groups, it’s critical that we stop adding features, changing behavior, adding new configuration options, or changing translatable strings as early as possible. Of course, it’s a trade-off. There might be things that are essential to the success of the release, or things that are obviously risk-limited. That’s why we have an exception process: the Feature Freeze exceptions (“FFEs”).
Feature freeze exceptions may be granted by the PTL (with the friendly but strong advice from the release management team). The idea is to weigh the raw benefit of having that feature in the release, against the complexity of the code that is being brought in, its risk of causing a regression, and how deep we are in feature freeze already. A self-contained change that is ready to merge a few days after feature freeze is a lot more likely to get an exception than a refactoring of a key layer that still needs some significant work to land. It also depends on how many exceptions were already granted on that project, because at some point adding anything more just causes too much disruption.
It’s a difficult call to make, and the release management team is here to help the PTLs make it. If your feature gets denied, don’t take it personally. As you saw there are a large number of factors involved. Our common goal is to raise the quality of the end release, and every feature freeze exception we grant is a step away from that. We just can’t take that many steps back and still guaranteeing we’ll win the race.
Open innovation vs. proprietary innovation
For companies, there are two ways to develop open source projects. The first one is to keep design and innovation inside your corporate borders, and only accept peripheral contributions. In that case you produce open source software, but everything else resembles traditional software development: you set the goals and roadmap for your product, and organize your development activity to meet those goals, using Agile or waterfall methodologies.
The second one is what we call open innovation: build a common and level playing field for contributions from anywhere, under the auspices of an independent body (foundation or other). In that case you don’t really have a roadmap: what ends up in the software is what the contributors manage to push through a maintainers trust tree (think: the Linux kernel) or a drastic code review / CI gate (think: OpenStack). Products or services are generally built on top of those projects and let the various participants differentiate on top of the common platform.
Now, while I heavily prefer the second option (which I find much closer to the ideals of free software), I recognize that both options are valid and both are open source. The first one ends up attracting far less contributions, but it works quite well for niche, specialized products that require some specific know-how and where focused product design gives you an edge. But the second works better to reach universal adoption and total world domination.
A tragedy of the commons
The dilemma of open innovation is that it’s a natural tragedy of the commons. You need strategic contributions to keep the project afloat: people working on project infrastructure, QA, security response, documentation, bugfixing, release management which do not directly contribute to your employer baseline as much as a tactical contribution (like a driver to interface with your hardware) would. Some companies contribute those necessary resources, while some others just get the benefits of monetizing products or services on top of the platform without contributing their fair share. The risk, of course, is that the strategic contributor gets tired of paying for the free rider.
Open innovation is a living ecosystem, a society. Like all societies, it has its parasites, its defectors, those which don’t live by the rules. And like all societies, it actually needs a certain amount of defectors, as it makes the society stronger and more able to evolve. The trick is to keep the amount of parasites down to a tolerable level. In our world, this is usually done by increasing the difficulty or the cost of defecting, while reducing the drawbacks or the cost of cooperating.
Keeping our society healthy
In his book Liars and Outliers, Bruce Schneier details the various knobs a society can play with to adjust the number of defectors. There are moral pressures, reputational pressures, institutional pressures and security pressures. In open innovation projects, moral pressures and security pressures don’t work that well, so we usually use a combination of institutional pressures (licensing, trademark rules) and reputational pressures (praising contributors, shaming free riders) to keep defectors to an acceptable level.
Those are challenges that are fully understood and regularly applied in the Linux kernel project. For OpenStack, the meteoritic growth of the project (and the expertise land-grab that came with it) protected us from the effects of the open innovation dilemma so far. But the Technical Committee shall keep an eye on this dilemma and be ready to adjust the knobs if it starts becoming more of a problem. Because at some point, it will.
StoryBoard is a project I started a few months ago. We have been running into a number of issues with Launchpad (inability to have blueprints spanning multiple code bases, inability to have flexible project group views, inability to use non-Launchpad OpenID for login…), and were investigating replacements. I was tired to explain why those alternatives wouldn’t work for our task tracking, so I started to describe the features we needed, and ended up writing a proof-of-concept to show a practical example.
That proof-of-concept was sufficiently compelling that the Infrastructure team decided we should follow the path of writing our own tool. To be useful, task tracking for complex projects has to precisely match your workflow. And the POC proved that it wasn’t particularly difficult to write. Then people from HP, Mirantis and RedHat joined this effort.
My Django-based proof-of-concept had a definite last-century feel to it, though. We wanted a complete REST API to cover automation and scripting needs, and multiple clients on top of that. Time was ripe for doing things properly and start building a team effort around this. Time was ripe for… the StoryBoard sprint.
We gathered in Brussels for two days in advance of FOSDEM, in a meeting room sponsored by the OpenStack Foundation (thanks!). On day 2 we were 12 people in the room, which was more than we expected !
Colette helped us craft a mission statement and structure our goals. Michael presented an architecture (static JS client on top of OpenStack-like REST service) that we blessed. Jaromir started to draw wireframes. Sergey, Ruslan and Nikita fought uncooperative consulates and traveled at night to be present on day 2. We also confirmed a number of other technology choices (Bootstrap, AngularJS…). We discussed the basic model, bikeshedded over StoryBoard vs. Storyboard and service URLs. We got a lot covered, had very few breaks, ate nice food and drank nice beer. But more importantly, we built a strong set of shared understandings which should help us make progress as a united team going forward.
We have automated testing and continuous deployment set up now, and once the initial basic functionality is up (MVP0) we should iterate fast. The Infrastructure program is expected to be the first to dogfood this, and the goal is to have something interesting to present to other programs by the Atlanta summit. To participate or learn more about StoryBoard, please join us on #storyboard on Freenode IRC, or at our weekly meeting.
Looking at our recently-concluded icehouse-2 development timeframe, we landed far less features and bugfixes than we wanted and expected. That created concerns about us losing our velocity, so I run a little analysis to confirm or deny that feeling.
Velocity loss ?
If we compare icehouse to the havana cycle and focus on implemented blueprints (not the best metric), it is pretty obvious that icehouse-2 was disappointing:
Using the first milestone as a baseline (growth of 10% expected), we should have been at 110 blueprints, so we are at 45% of the expected results. That said, looking at bugs gives a slightly different picture:
The first milestone baseline again gives a 10% expected growth, which means the target was 715 bugs… but we “only” fixed 650 bugs (like in havana-2). So on the bugfixes front, we are at 91% of the expected result.
Comparing with grizzly
But havana is not really the cycle we should compare icehouse with. We should compare with another cycle where the end-of-year holidays hit during the -2 milestone development… so grizzly. Let’s look at the number of commits (ignoring merges), for a number of projects that have been around since then. Here are the results for nova:
nova grizzly-1: 549 commits
nova grizzly-2: 465 commits
nova icehouse-1: 548 commits
nova icehouse-2: 282 commits
Again using the -1 milestone as a baseline for expected growth (here +0%), nova in icehouse-2 ended up at 61% of the expected number of commits. The results are similar for neutron:
neutron grizzly-1: 155 commits
neutron grizzly-2: 128 commits
neutron icehouse-1: 203 commits
neutron icehouse-2: 110 commits
Considering the -1 milestones gives an expected growth in commits between grizzly and icehouse of +31%. Icehouse-2 is at 66% of expected result. So not good but not catastrophic either. What about cinder ?
cinder grizzly-1: 86 commits
cinder grizzly-2: 54 commits
cinder icehouse-1: 175 commits
cinder icehouse-2: 119 commits
Now that’s interesting… Expected cinder growth between grizzly and icehouse is +103%. Icehouse-2 scores at 108% of the expected, grizzly-based result.
keystone grizzly-1: 95 commits
keystone grizzly-2: 42 commits
keystone icehouse-1: 116 commits
keystone icehouse-2: 106 commits
That’s even more apparent with keystone, which had a quite disastrous grizzly-2: expected growth is +22%, Icehouse-2 is at 207% of the expected result. Same for Glance:
glance grizzly-1: 100 commits
glance grizzly-2: 38 commits
glance icehouse-1: 98 commits
glance icehouse-2: 89 commits
Here we expect 2% less commits, so based on grizzly-2 we should have had 37 commits… icehouse-2 here is at 240% !
In summary, while it is quite obvious that we delivered far less than we wanted to, due to the holidays and the recent gate issues, from a velocity perspective icehouse-2 is far from being disastrous if you compare it to the last development cycle where the holidays happened at the same time in the cycle. Smaller projects in particular have handled that period significantly better than last year.
We just need to integrate the fact that the October – April cycle includes a holiday period that will reduce our velocity… and lower our expectations as a result.
Every year, free and open source developers from all over Europe and beyond converge in cold Brussels for a week-end of talks, hacking and beer. OpenStack will be present !
We have a number of devroom and lightning talks already scheduled:
Saturday 12:20 in Chavanne (Virtualization and IaaS devroom)
Autoscaling best practices
Marc Cluet will look into autoscaling using Heat and Ceilometer as examples.
Saturday 13:00 in Chavanne (Virtualization and IaaS devroom)
Network Function Virtualization and Network Service Insertion and Chaining
Balaji Padnala will present NFV and how to deploy it using OpenStack and OpenFlow Controller.
Saturday 13:40 in Chavanne (Virtualization and IaaS devroom)
oVirt and OpenStack Storage (present and future)
Federico Simoncelli will cover integration between oVirt and Glance/Cinder for storage needs.
Saturday 15:00 in Chavanne (Virtualization and IaaS devroom)
Why, Where, What and How to contribute to OpenStack
I will go through a practical introduction to OpenStack development and explain why you should contribute if you haven’t already.
Saturday 16:20 in Chavanne (Virtualization and IaaS devroom)
Hypervisor Breakouts – Virtualization Vulnerabilities and OpenStack Exploitation
Rob Clark will explore this class of interesting vulnerabilities from an OpenStack perspective.
Saturday 17:40 in Chavanne (Virtualization and IaaS devroom)
oVirt applying Nova scheduler concepts for data center virtualization
Gilad Chaplik will present how oVirt could reuse OpenStack Nova scheduling concepts.
Sunday 10:00 in U.218A (Testing and automation devroom)
Preventing craziness: a deep dive into OpenStack testing automation
Me again, in a technical exploration on the OpenStack gating system and its unique challenges.
Sunday 13:40 in Chavanne (Virtualization and IaaS devroom)
Tunnels as a Connectivity and Segregation Solution for Virtualized Networks
Join Assaf Muller for an architectural, developer oriented overview of (GRE and VXLAN) tunnels in OpenStack Networking.
Sunday 16:20 in Chavanne (Virtualization and IaaS devroom)
Bring your virtualized networking stack to the next level
Mike Kolesnik will look into integration opportunities between oVirt and OpenStack Neutron.
Sunday 17:00 in Ferrer (Lightning talks room)
Putting the PaaS in OpenStack
Dirk Diane Mueller will give us an update on cross-community collaboration between OpenStack, Solum, Docker and OpenShift.
Sunday 17:20 in Ferrer (Lightning talks room)
Your Complete Open Source Cloud
Dave Neary should explain how to mix OpenStack with oVirt, OpenShift and Gluster to build a complete private cloud.
We’ll also have a booth manned by OpenStack community volunteers ! I hope to see you all there.
Just back from an amazing week at the OpenStack Summit in Hong-Kong, I would like to share a number of discussions we had (mainly on the release management track) and mention a few things I learned there.
First of all, Hong-Kong is a unique city. Skyscrapers built on vertiginous slopes, crazy population density, awesome restaurants, shops everywhere… Everything is clean and convenient (think: Octopus cards), even as it grows extremely fast. Everyone should go there at least one time in their lives !
On the Icehouse Design Summit side, the collaboration magic happened again. I should be used to it by now, but it is still amazing to build this level playing field for open design, fill it with smart people and see them make so much progress over 4 days. We can still improve, though: for example I’ll make sure we get whiteboards in every room for the next time :). As was mentioned in the feedback session, we are considering staggering the design summit and the conference (to let technical people participate to the latter), set time aside to discuss cross-project issues, and set up per-project space so that collaboration can continue even if there is no scheduled “session” going on.
I have been mostly involved in release management sessions. We discussed the Icehouse release schedule, with a proposed release date of April 17, and the possibility to have a pre-designated “off” week between release and the J design summit. We discussed changes in the format of the weekly project/release status meeting, where we should move per-project status updates off-meeting to be able to focus on cross-project issues instead. During this cycle we should also work on streamlining library release announcements. For stable branch maintenance, we decided to officially drop support for version n-2 by feature freeze (rather than at release time), which reflects more accurately what ended up being done during the past cycles. The security support is now aligned to stable branch support, which should make sure the vulnerability management team (VMT) doesn’t end up having to maintain old stable branches that are already abandoned by the stable branch maintainers. Finally, the VMT should review the projects from all official programs to come up with a clear list of what projects are actually security-supported and which aren’t.
Apart from the release management program, I’m involved in two pet projects: Rootwrap and StoryBoard. Rootwrap should be split from the oslo-incubator into its own package early in the Icehouse cycle, and its usage in Nova, Cinder and Neutron should be reviewed to result in incremental strengthening. StoryBoard (our next-generation task tracker) generated a lot of interest at the summit, I expect a lot of progress will be made in the near future. Its architecture might be overhauled from the current POC, so stay tuned.
Finally, it was great meeting everyone again. Our PTLs and Technical Committee members are a bunch of awesome folks, this open source project is in great hands. More generally, it seems that we not only designed a new way of building software, we also created a network of individuals and companies interested in that kind of open collaboration. That network explains why it is so easy for people to jump from one company to another, while continuing to do the exact same work for the OpenStack project itself. And for developers, I think it’s a great place to be in: if you haven’t already, you should definitely consider joining us.
When we changed the Technical Committee membership model to an all-directly-elected model a few months ago, we proposed we would enable detailed ballot reporting in order to be able to test alternative algorithms and run various analysis over the data set. As an official for this election, here is my analysis of the results, hoping it will help in the current discussion on a potential evolution of the Foundation individual members voting system.
In the OpenStack technical elections, we always used the Condorcet method (with the Schulze completion method), as implemented by Cornell’s CIVS public voting system. In a Condorcet vote, you rank your choices in order of preference (it’s OK to rank multiple choices at the same level). To calculate the results, you simulate 1:1 contests between all candidates in the set. If someone wins all such contests, he is the Condorcet winner for the set. The completion method is used to determine the winner when there is no clear Condorcet winner. Most completion methods can result in ties, which then need to be broken in a fair way.
One thing we can analyze is the spread of the rankings for any given candidate:
On that graph the bubbles on the left represent the number of high rankings for a given candidate (bubbles on the right represent low rankings). When multiple candidates are given the same rank, we average their ranking (that explains all those large bubbles in the middle of the spectrum). A loved-or-hated candidate would have large bubbles at each end of the spectrum, while a consensus candidate would not.
Looking at the graph we can see how Condorcet favors consensus candidates (Doug Hellmann, James E. Blair, John Griffith) over less-consensual ones (Chris Behrens, Sergey Lukjanov, Boris Pavlovic).
Proportional Condorcet ?
Condorcet indeed favors consensus candidates (and “natural” 1:1 election winners). It is not designed to represent factions in a proportional way, like STV is. There is an experimental proportional representation option in CIVS software though, and after some ballot conversion we can run the same ballots and see what it would give.
I set up a test election and the results are here. The winning 11 would have included Sergey Lukjanov instead of John Griffith, giving representation to a less-consensual candidate. That happens even if a clear majority of voters prefers John to Sergey (John defeats Sergey in the 1:1 Condorcet comparison by 154-76).
It’s not better or worse, it’s just different… We’ll probably have a discussion at the Technical Committee to see whether we should enable this experimental variant, or if we prefer to test it over a few more elections.
Partisan voting ?
Another analysis we can run is to determine if there was any corporate-driven voting. We can look at the ballots and see how many of the ballots consistently placed all the candidates from a given company above any other candidate.
7.8% of ballots placed the 2 Mirantis candidates above any other. 5.2% placed the 2 IBM candidates above any other. At the other end of the spectrum, 0.8% of ballots placed all 5 Red Hat candidates above any other, and 1.1% of the ballots placed all 4 Rackspace candidates above any other. We can conclude that partisan voting was limited, and that Condorcet’s preference for consensus candidates further limited its impact.
What about STV ?
STV is another ranked-choice election method, which favors proportional representation. Like the “proportional representation” CIVS option described above, it may result in natural Condorcet winners to lose against more factional candidates.
I would have loved to run the same ballots through STV and compare the results. Unfortunately STV requires strict ranking of candidates in an order of preference. I tried converting the ballots and randomly breaking similar rankings, but the end results vary extremely depending on that randomness, so we can’t really analyze the results in any useful way.
Run your own analysis !
That’s it for me, but you can run your own analysis by playing with the CSV ballot file yourself ! Download it here, and share the results of your analysis if you find anything interesting !
Yesterday, as the final conclusion of the 6-month “Havana” development cycle, we released the latest version of OpenStack, the 2013.2 version. It’s now composed of 9 integrated components, which saw the completion of more than 400 feature blueprints and the fixing of more than 3000 reported bugs.
As always, it’s been an interesting week for me. Not as busy as you’d think, but lots of hours, day or night, spent frantically checking dashboards, collecting input from our fearful technical leads, waiting for a disaster to happen and pushing the right buttons at the right moment, finally aligning the stars and releasing everything on time. Yes, I use checklists to make sure I don’t overlook anything:
Even if I have plenty of free time between those key hours, I can’t concentrate on getting anything else done, or get something else started (like a blog post). This is why, one day after release, I can finally spend some time looking back on the last stage of the Havana development cycle and see how well we performed. Here is the graph showing the number of release-critical bugs logged against various components (those observing the common feature freeze) as we make progress towards the final release date:
Personally I think we were a bit late, with RC1s globally landing around October 3 and RC2s still being published around October 15. I prefer when we can switch to “respin only for major regressions and upgrade issues” mode at least a week before final release, not two days before final release. Looking at the graph, we can see where we failed: it took us 11 days to get a grip on the RC bug count (either by fixing the right issues, or by stopping adding new ones, or by refining the list and dropping non-critical stuff). Part of this delay is due to stress recovery after a rather eventful feature freeze. Part of it is lack of prioritization and focus on the right bugs. The rest of the graph pretty much looks like the Grizzly one. We were just at least one week too late.
We’ll explore ways to improve on that during the Icehouse Design Summit in Hong-Kong. One solution might be to add a week between feature freeze and final release. Another solution would be to filter what gets targeted to the last milestone to reduce the amount of features that land late in the cycle, to reduce FeatureFreeze trauma. If you want to be part of the discussion, join us all in Hong-Kong in 18 days !
Over the last 3 years, the technical governance of the OpenStack open source project evolved a lot, and most recently last Tuesday. As an elected member of that governance body since April 2011, I witnessed that evolution from within and helped in drafting the various models over time. Now seems like a good time to look back in history, and clear a few misconceptions about the OpenStack project governance along the way.
The project was originally created by Rackspace in July 2010 and seeded with code from NASA (Nova) and Rackspace (Swift). At that point an initial project governance was set up. There was an Advisory Board (which was never really created), the OpenStack Architecture Board, and technical committees for each subproject, each lead by a Technical Lead. The OpenStack Architecture Board had 5 members appointed by Rackspace and 4 elected by the community, with 1-year to 3-year (!) terms. The technical leads for the subprojects were appointed by Rackspace.
By the end of the year 2010 the Architecture Board was renamed Project Oversight Committee (POC), but its structure didn’t change. While it left room for community input, the POC was rightfully seen as fully controlled by Rackspace, which was a blocker to deeper involvement for a lot of the big players in the industry.
It was a danger for the open source project as well, as the number of contributors external to Rackspace grew. As countless examples prove, when the leadership of an open source project is not seen as representative of its contributors, you face the risk of a revolt, a fork of the code and seeing your contributors leave for a more meritocratic and representative alternative.
In March 2011, a significant change was introduced to address this perceived risk. Technical leads for the 3 projects (Nova, Swift, and Glance at that point) would from now on be directly elected by their contributors and called Project Technical Leads (PTLs). The POC was replaced by the Project Policy Board (PPB), which had 4 seats appointed by Rackspace, 3 seats for the above PTLs, and 5 seats directly-elected by all the contributors of the project. By spring 2012 we grew to 6 projects and therefore the PPB had 15 members.
This was definitely an improvement, but it was not perfect. Most importantly, the governance model itself was still owned by Rackspace, which could potentially change it and displace the PPB if it was ever unhappy with it. This concern was still preventing OpenStack from reaching the next adoption step. In October 2011, Rackspace therefore announced that they would set up an independent Foundation. By the summer of 2012 that move was completed and Rackspace had transfered the control over the governance of the OpenStack project to the OpenStack Foundation.
At that point the governance was split into two bodies. The first one is the Board of Directors for the Foundation itself, which is responsible for promoting OpenStack, protecting its trademark, and deciding where to best spend the Foundation’s sponsors money to empower future development of OpenStack.
The second body was the successor to the PPB, the entity that would govern the open source project itself. A critical piece in the transition was the need to preserve and improve the independence of the technical meritocracy. The bylaws of the Foundation therefore instituted the Technical Committee, a successor for the PPB that would be self-governed, and would no longer have appointed members (or any pay-to-play members). The Technical Committee would be completely elected by the active technical contributors: a seat for each elected PTL, plus 5 directly-elected seats.
The TC started out in September 2012 as an 11-member committee, but with the addition of 3 new projects (and the creation of a special seat for Oslo), it grew to 15 members in April 2013, with the perspective to grow to 18 members in Fall 2013 if all projects applying for incubation recently get finally accepted. With the introduction of the “integrated” project concept (separate from the “core” project concept), we faced the addition of even more projects in the future and committee bloat would inevitably ensue. That created a potential for resistance to the addition of “small” projects or the splitting of existing projects (which make sense technically but should not be worth adding yet another TC seat).
Another issue was the ever-increasing representation of “vertical” functions (project-specific PTLs elected by each project contributors) vs. general people elected by all contributors. In the original PPB mix, there were 3 “vertical” seats for 5 general seats, which was a nice mix to get specific expertise but overall having a cross-project view. With the growth in the number of projects, in the current TC we had 10 “vertical” seats for 5 general seats. Time was ripe for a reboot.
Various models were considered and discussed, and while everyone agreed on the need to change, no model was unanimously seen as perfect. In the end, simplicity won and we picked a model with 13 directly-elected members, which will be put in place at the Fall 2013 elections.
Power to the active contributors
This new model is a direct, representative model, where if you recently authored a change for an OpenStack project, you get one vote, and a chance every 6 months to choose new people to represent you. This model is pretty flexible and should allow for further growth of the project.
Few open source projects use such a direct governance model. In Apache projects for example (often cited as a model of openness and meritocracy), the oversight committee equivalent to OpenStack’s TC would be the PMC. In most cases, PMC membership is self-sustaining: existing PMC members ultimately decide, through discussions and votes on the private PMC list, who the new PMC members should be. In contrast, in OpenStack the recently-active contributors end up being in direct control of who their leaders are, and can replace the Technical Committee members if they feel like they are not relevant or representing them anymore. Oh, and the TC doesn’t use a private list: all our meetings are public and our discussions are archived.
As far as open source projects governance models go, this is as open, meritocratic, transparent and direct as it gets.
The beginning of a new release cycle is as good as any moment to question why we actually go through the hassle of producing OpenStack releases. Twice per year, on a precise date we announce 6 months in advance, we bless and publish source code tarballs of the various integrated projects in OpenStack. Every week we have a meeting that tracks our progress toward this common goal. Why ?
Releases vs. Continuous deployment
The question is particularly valid if you take into account the type of software that we produce. We don’t really expect cloud infrastructure providers to religiously download our source code tarballs every 6 months and run from that. For the largest installations, running much closer to the master branch and continuously deploy the latest changes is a really sound strategy. We invested a lot of effort in our gating systems and QA automated testing to make sure the master branch is always runnable. We’ll discuss at the OpenStack Summit next week how to improve CD support in OpenStack. We backport bugfixes to the stable branches post-release. So why do we continue to single out a few commits and publish them as “the release” ?
The need for cycles
The value is not really in releases. It is in release cycles.
Producing OpenStack involves the work of a large number of people. While most of those people are paid to participate in OpenStack development, as far as the OpenStack project goes, we don’t manage them. We can’t ask them to work on a specific area, or to respect a given deadline, or to spend that extra hour to finalize something. The main trick we use to align everyone and make us all part of the same community is to have a cycle. We have regular milestones that we ask contributors to target their features to. We have a feature freeze to encourage people to switch their mindset to bugfixing. We have weekly meetings to track progress, communicate where we are and motivate us to go that extra mile. The common rhythm is what makes us all play in the same team. The “release” itself is just the natural conclusion of that common effort.
A reference point in time
Singling out a point in time has a number of other benefits. It’s easier to work on documentation if you group your features into a coherent set (we actually considered shortening our cycles in the past, and the main blocker was our capacity to produce good documentation often enough). It’s easier to communicate about OpenStack progress and new features if you do it periodically rather than continuously. It’s easier to have Design Summits every 6 months if you create a common brainstorm / implementation / integration cycle. The releases also serve as reference points for API deprecation rules, for stable release maintenance, for security backports.
If you’re purely focused on the software consumption part, it’s easy to underestimate the value of release cycles. They actually are one of the main reasons for the pace of development and success of OpenStack so far.
The path forward
We need release cycles… do we need release deliverables ? Do we actually need to bless and publish a set of source code tarballs ? My personal view on that is: if there is no additional cost in producing releases, why not continue to do them ? With the release tooling we have today, blessing and publishing a few tarballs is as simple as pushing a tag, running a script and sending an email. And I like how this formally concludes the development cycle to start the stable maintenance period.
But what about Continuous Deployment ? Well, the fact that we produce releases shouldn’t at all affect our ability to continuously deploy OpenStack. The master branch should always be in good shape, and we definitely should have the necessary features in place to fully support CD. We can have both. So we should have both.