August was very busy for OpenStack Nova and Glance developers, and the culmination of those efforts is the delivery of the final feature milestone of the Diablo development cycle: diablo-4.
Glance gained final integration with the Keystone common authentication system, support for sharing images between groups of tenants, a new notification system and i18n. Twelve feature blueprints were completed in Nova, including final Keystone integration, the long-awaited capacity to boot from volumes, a configuration drive to pass information to instances, integration points for Quantum, KVM block migration support, as well as several improvements to the OpenStack API.
Diablo-4 is mostly feature-complete: a few blueprints for standalone features were granted an exception and will land post-diablo-4, like volume types or virtual storage arrays in Nova, or like SSL support in Glance.
Now we race towards the release branch point (September 8th) which is when the Diablo release branch will start to diverge from a newly-open Essex development branch. The focus is on testing, bug fixing and consistency… up until September 22, the Diablo release day.
How to control what gets into your open source project code ? The classic model, inherited from pre-DVCS days, is to have a set of “committers” that are trusted with direct access while the vast majority of project “contributors” must kindly ask them to sponsor their patches. You can find that model in a lot of projects, including most Linux distributions. This model doesn’t scale that well — even trusted individuals are error-prone, nobody should escape peer review. But the main issue is the binary nature of the committer power: it divides your community (us vs. them) and does not really encourage contribution.
The solution to this is to implement a gated trunk with a code review system like GitHub pull requests or Launchpad branch merge proposals. Your “committers” become “core developers” that have a casting vote on whether the proposal should be merged. Everyone goes through the peer review process, and the peer review process is open for everyone: your “contributors” become “developers” that can comment too. You reduce the risk of human error and the community is much healthier, but some issues remain: your core developers can still (wittingly or unwittingly) evade peer review, and the final merge process is human and error-prone.
The solution is to add more automation, and not trust humans with direct repository access anymore. An “automated gated trunk” bot can watch for reviews and when a set of pre-defined rules are met (human approvals, testsuites passed, etc.), trigger the trunk merge automatically. This removes human error from the process, and effectively turns your “core developers” into “reviewers”. This last aspect makes for a very healthy development community: there is no elite group anymore, just a developer subgroup with additional review duties.
In OpenStack, we used Tarmac in conjunction with Launchpad/bzr code review in our first attempt to implement this. As we considered migration to git, the lack of support for tracking formal approvals in GitHub code review prevented the implementation of a complex automated gated trunk on top of GitHub, so we deployed Gerrit. I was a bit resisting the addition of a new tool to our toolset mix, but the incredible Monty Taylor and Jim Blair did a great integration job, and I realize now that this gives us a lot more flexibility and room for future evolution. For example I like that some tests can be run when the change is proposed, rather than only after the change is approved (which results in superfluous review roundtrips).
At the end of the day, gated trunk automation helps in having a welcoming, non-elitist (and lazy) developer community. I wish more projects, especially distributions, would adopt it.
No rest for the OpenStack developers, today saw the release of the July development efforts for Nova and Glance: the Diablo-3 milestone.
With a bit more than 100 trunk commits over the month, Nova gained support for multiple NICs, FlatDHCP network mode now support a high-availability option (read more about it here), instances can be migrated and system usage notifications were added to the notification framework. Network code was also refactored in order to facilitate integration with the new networking projects, and countless fixes were made in OpenStack API 1.1 support.
We have one more milestone left (diablo-4) before the final 2011.3 release… still a lot to do !
About a month ago I commented on the features delivered in the diablo-1 milestone. Last week we released the diablo-2 milestone for your testing and feature evaluation pleasure.
Most of the changes to Glance were made under the hood. In particular the new WSGI code from Nova was ported to Glance, and images collections can now be sorted by a subset of the image model attributes. Most of the groundwork to support Keystone authentication was done, but that should only be available in diablo-3 !
Those same initial Keystone integration steps were also done for Nova, along with plenty of other features. We now support distributed scheduling for complex deployments, together with a new instance referencing model. Was also added during this timeframe: support for floating IPs (in OpenStack API), a basic mechanism for pushing notifications out to interested parties, global firewall rules, and an instance type extra specs table that can be used in a capabilities-aware scheduler. More invisible to the user, we completed efforts to standardize error codes and refactored the OpenStack API serialization mechanism.
And there is plenty more coming up in diablo-3… scheduled for release on July 28th.
There was a bit of discussion lately on feature-based vs. time-based release schedules in OpenStack, and here are my thoughts on it. In a feature-based release cycle, you release when a given set of features is implemented, while in a time-based release cycle, you release at a predetermined date, with whatever is ready at the time.
Release early, release often
One the basic principles in open source (and agile) is to release early and release often. This allows fast iterations, which avoid the classic drawbacks of waterfall development. If you push that logic to the extreme, you can release at every commit: that is what continuous deployment is about. Continuous deployment is great for web services, where there is only one deployment of the software and it runs the latest version.
OpenStack projects actually provide builds (and packages) for every commit made to development trunk, but we don’t call them releases. For software that has multiple deployers, having “releases” that combine a reasonable amount of new features and bugfixes is more appropriate. Hence the temptation of doing feature-based releases: release often, whenever the next significant feature is ready.
Frequent feature-based releases
The main argument of supporters of frequent feature-based releases is that time-based cycles are too long, so they delay the time it takes for a given feature to be available to the public. But time-based isn’t about “a long time”. It’s about “a predetermined amount of time”. You can make that “predetermined amount of time” as small as needed…
Supporters of feature-based releases say that time-based releases are good for distributions, since those have limited bearing on the release cycles of their individual subcomponents. I’d argue that time-based releases are always better, for anyone that wants to do open development in a community.
Time-based releases as a community enabler
If you work with a developer community rather than with a single-company development group, the project doesn’t have full control over its developers, but just limited influence. Doing feature-based releases is therefore risky, since you have no idea how long it will take to have a given feature implemented. It’s better to have frequent time-based releases (or milestones), that regularly delivers to a wider audience what happens to be implemented at a given, predetermined date.
If you work with an open source community rather than with a single-company product team, you want to help the different separate stakeholders to synchronize. Pre-announced release dates allow everyone (developers, testers, distributions, users, marketers, press…) to be on the same line, following the same cadence, responding to the same rhythm. It might be convenient for developers to release “whenever it makes sense”, but the wider community benefits from having predictable release dates.
It’s no wonder that most large open source development communities switched from feature-based releases to time-based releases: it’s about the only way to “release early, release often” with a large community. And since we want the OpenStack community to be as open and as large as possible, we should definitely continue to do time-based releases, and to announce the schedule as early as we can.
There are multiple available delivery channels available to install OpenStack packages on Ubuntu.
First of all, starting with 11.04 (Natty), packages for OpenStack Nova, Swift and Glance are available directly in Ubuntu’s official universe repository. This contains the latest release available at the time of Ubuntu release: 2011.2 “Cactus” in 11.04. If you don’t use Ubuntu 11.04, or if you want a more recent version, you’ll need to enable one of our specific PPAs, please read on.
OpenStack release PPA
If you want to run 2011.2 on 10.04 LTS (Lucid) or 10.10 (Maverick) you can use the ppa:openstack-release/2011.2 PPA. Enabling it is as simple as running:
$ sudo apt-get install python-software-properties $ sudo add-apt-repository ppa:openstack-release/2011.2 $ sudo apt-get update
Starting with the Diablo cycle, we do a coordinated OpenStack release every 6 months. That said, Swift releases stable versions more often. You can get the latest Swift release through the ppa:swift-core/release PPA. Just replace the PPA name in the above example.
Also starting with the Diablo cycle, Nova and Glance deliver a development milestone every 4 weeks. If you want to test the latest features, you can enable those PPAs: ppa:nova-core/milestone or ppa:glance-core/milestone.
PPAs for testers and developers
Just before we deliver one of these intermediary releases or milestones, we use a specific “milestone-proposed” PPA to do final QA on release candidates. Enabling this one and reporting issues will help us in delivering high-quality milestones. Just enable ppa:nova-core/milestone-proposed, ppa:glance-core/milestone-proposed or ppa:swift-core/milestone-proposed.
Finally, for all projects and for every code commit, we generate a package in the trunk PPA. If you’re a developer, or like living on the bleeding edge, you should enable those: ppa:nova-core/trunk, ppa:glance-core/trunk or ppa:swift-core/trunk.
I hope this will help you select the best delivery channel for your use case, depending on whether you’re deploying, evaluating, helping in QA or actively developing. For future reference, the list of PPAs is maintained on the wiki.
Back at the OpenStack Design Summit in Santa Clara, we decided to switch from a 3-month cycle to a 6-month coordinated release cycle, with more frequent milestones delivery in the middle.
Lately we have been busy adapting the release processes to match the delivery of the first milestones. Swift 1.4.0 was released last Tuesday, and today sees the release of the diablo-1 milestone for Nova and Glance.
What should you expect from diablo-1, just 4 weeks after the design summit ? In this short timeframe lots of features have been worked on, and the developers managed to land quite a few of them in time for diablo-1.
Glance’s API was improved to support filtering of /images and /images/detail results and limiting and paging of results. This made support of API versioning necessary. It also grew a new disk format (“iso”) that should ultimately allow to boot ISOs directly in Nova.
On Nova’s side, the most notable addition is support for snapshotting and cloning volumes with the EC2 API. The XenServer plugin now supports Open vSwitch, and pause and suspend capabilities have been added to the KVM hypervisor.
Now keep your seatbelt fastened, because diablo-2 is set to release on June 30th.