Blog moved to ttx.re !

This will be the last post to “Seeing the fnords”. I decided to embrace progress, static websites, shiny DNS domains and free github page hosting, and moved this blog to http://ttx.re.

The new blog is called ttx:reloaded. Most posts from this blog have been imported. I won’t be posting here anymore, so please update your links!

Analysis of April 2014 TC election

Some people asked me to analyze the results of the recent TC election in the same way I ran the analysis for the previous one. I finally found the time to do it, here are the results.

The April 2014 election was set up to renew 7 of the 13 TC members. We had 17 candidates, you can find the official results of the election here.

Condorcet spread

This graph shows how each candidate was ranked. The first bubble on the left represents the number of people that have placed that candidate as their only first choice. The last bubble on the right represents the number of people that have placed this candidate as their only last choice. If multiple candidates are ranked at the same level, we average their “score”.

We can see that once again the Condorcet algorithm preferred a consensual candidate (Devananda) over less-consensual ones (Julien, Sergey). It’s also interesting to compare the spread between Michael, Jim and Mark.

Proportional Condorcet

At the previous election, running the same ballot with CIVS so-called “proportional mode” option altered the result. This time, the proportional mode returns the exact same set of winners.

Partisan voting

The goal of this analysis is to detect blocks of voters, who consistently place a set of candidates above anyone else. I slightly modified my script to reveal the most popular pairs: calculate how many people place the same two people above anyone else, and try to detect bias in the most popular pairs.

The most popular pair was Vish/Thierry (with 5.58% of voters placing us above anyone else). We could call that the old-timers party. The second most popular pair was Jay/Sergey (the Mirantis party) with 5.13%. 4.24% of the voters placed the Julien/Thierry pair on top of anything else: that could reveal an influential French party ! The Foundation party pair (Jim/Thierry) was preferred by 3.34%, same result as the Foodie party (Mark/Thierry). The Jay/Vish pair was preferred by 3.12% of the voters (a variant of the old-timers party). It’s also worth noting that 2.45% of the voters favored the Flavio/Steven pair (best score of the RedHat pairs), 1.56% favored Michael/JohnG (best score of the Rackspace pairs) and 1.11% favored Joe/Devananda (best score of the HP pairs).

While this analysis shows some corporate bias in the vote, it’s worth noting that (a) it’s extremely limited, (b) it’s actually decreasing from the previous election, (c) it did not affect the result in any significant way and (d) it’s comparable to other bias (old timers, French people).

Run your own analysis

That’s all I had to share. But don’t blindly trust me, you can run your own analysis by downloading the anonymized ballot !

F/OSS project governance models

Various governance models exist for free and open source software projects. Most of those happen naturally, some of them are chosen… Which one is the best ? Is there a best ? How could we judge the best ? Like any ecosystem, I’d postulate that F/OSS project communities should have long-term survival as their main goal: the ability to continue operation as the same community over time, without fracture of fork.

Dictatorship

The “benevolent dictator for life” model usually happens naturally. The project is often originally the brain child of a single, talented individual, who retains final say over everything that happens to their project. This usually works very well: the person is naturally respected in the community. It also naturally allows for opinionated design, and people who sign up to the project can’t ignore what they sign up for.

The main issue with that setup is that it’s not replicable, it can’t be dictated. It either happens naturally, or it will never happen. You don’t choose someone to become your dictator-for-life after the fact. Any attempt to do so would fail to get enough legitimacy and natural respect to make it last. The second issue with that setup is that it’s not durable. If the dictator stops being active in the community, their opinion is not as much respected anymore (especially by new contributors), which usually triggers a painful fork or governance model switch (that’s what happened in Gentoo). Even in the rare cases where the original dictator manages to retain interest and respect in the project, it’s inherently brittle: the “natural” dictator can’t really be replaced in case something bad happens. Succession is always dirty. So from a long-term survival standpoint, this model is not that great.

Aristocracy

Aristocracy is used to solve the perceived drawbacks of the dictator-for-life model. Instead of focusing on one person, let’s have a group of people in control of the project, and let that group self-select successors in the wider pool of contributors. That’s the role of “committers” in certain projects, and it’s also how Apache project management committees (PMCs) usually work. It also works quite well, with self-selection usually ensuring that the members share enough common culture to reach consensus on most decisions.

The drawback here is obviously the self-selection bias. Aristocracies all fall after getting more and more disconnected from the people they control, and revolution happens. Open source aristocracies are no different: they fall after gradually growing disconnected from their project contributors base. Whenever contributors to an open source project feel like their leaders are no longer representative of the contributors or relevant to the present of the project, this disconnect happens. In mild cases, people just go contribute somewhere else, and in difficult cases this usually triggers a fork.

Direct democracy / Anarchy

The obvious way to solve that disconnect is to give the power directly to the contributors. Direct democracy projects give ultimate power to all the contributors. Anarchy projects let contributors do whatever they want. Debian is an interesting mix of the two: developers vote on general resolutions, but maintainers also have a lot of control on their packages.

While these models have a certain appeal, those projects usually have a hard time taking necessary decisions that affect the whole project, so they tend to linger not taking any critical decision. It’s also a model that is difficult to evolve: when you try to add new layers on top of it, they are never really accepted by the contributors base.

Representative democracy

That leaves us with representative democracy. You regularly designate a small group of people and trust them to make the right decisions for the governance of the project. It can happen in cases where there was no natural dictator at the beginning of the project. It’s different from aristocracy in that they are chosen by the contributors base and regularly renewed — ensuring that they are always seen as a fair representation of the contributors to the project. It’s more efficient than direct democracy or anarchy in making clear and opinionated decisions.

Now it’s far from perfect. As Churchill famously said, it’s the worst form of government, except all those other forms that have been tried from time to time. It also only works if the elected people are seen as legitimate and representative, so it requires good participation levels in elections. So here is my plea: the OpenStack Technical Committee, which oversees the development of the OpenStack open source project as a whole, is being partially renewed this week. If you’re an OpenStack contributor, please vote: this will ensure that elected people have the legitimacy necessary for making the decisions that need to be made, and increase the health of the project.

 

Upcoming changes to Design Summit format

Since the very beginning of OpenStack we fulfilled our Open Design promise by organizing a developer gathering open to all OpenStack contributors at the beginning of all our development cycles, called the Design Summit. Those events have proven to be an essential part of OpenStack success and growth.

Design Summits are a set of discussion sessions under the auspices of a given OpenStack program. There are no formal presentations or speakers, just open discussions around a given development theme. The elected Program Technical Leads are responsible for picking a set of discussion topics, and they take suggestions from the rest of the community on our session suggestion website at summit.openstack.org.

Improvements

One of the last sessions at the Icehouse Design Summit in Hong-Kong was about the Design Summit format, and how we should improve on it. Several issues were reported during that session, like:

• the inability for technical people to attend (or present at) the rest of the OpenStack Summit
• the (in)visibility of the Unconference track, making it difficult for nascent projects to attract interested contributors
• the difficulty to have cross-project discussions in a schedule strictly organized around project-specific topics
• the difficulty for incubated projects to continue to collaborate during the week, outside of the limited scheduled slots allocated to them

I’m happy to report that we acted on that feedback and will implement a number of changes in the upcoming Juno Design Summit in Atlanta in May.

First, we started staggering the Design Summit from the rest of the OpenStack Summit. The main event starts on Monday and ends on Thursday, while the Juno Design Summit starts on Tuesday and ends on Friday. This should allow our key technical assets to attend some of the conference and maybe present there.

Second, the Unconference track is abandoned. It will be replaced by several initiatives. Part of the Unconference was traditionally used by open source projects related to OpenStack to present themselves and recruit contributors. We’ll have an Other projects track at the Design Summit to cover for those. This will be limited to one session per project, but those will appear on the official schedule. If you are an open source project related to OpenStack and would like one of those slots, please head to the session suggestion site !

Another classic use of the Unconference was ad-hoc continuation of discussions that started in scheduled sessions, or coverage of lesser topics that couldn’t find a place in scheduled sessions. To cover for that, we’ll set up a roundtable for each program (complete with paperboard) to serve as a rallying point for contributors around that program. This designated space (codenamed project pod) can be used to have additional discussions and continue collaboration outside of the limited scheduled sessions.

Last but not least, we’ll dedicate the first day of the summit (Tuesday) to cross-project workshops. During that day, no other integrated project sessions will be running, which should facilitate presence of key stakeholders. We’ll be able to discuss OpenStack-wide goals, convergence, integration and other cross-project issues.

We hope that those changes will let us make the most of those 4 days all together. At the end of the Juno Design Summit we’ll discuss whether those changes were an improvement and whether we should do them again for the K design summit in Paris in November. See you all in Atlanta in 8 weeks!

 

Why we do Feature freeze

Yesterday we entered the Icehouse development cycle Feature Freeze. But with the incredible growth of the OpenStack development community (508 different contributors over the last 30 days, including 101 new ones !), I hear a lot of questions about it. I’ve explained it on various forums in the past, but I figured it couldn’t hurt to write something a bit more definitive about it.

Why

Those are valid questions. Why freeze features ? That sounds very anti-agile. Isn’t our test-centric development model supposed to protect us from regressions anyway ? Let’s start with what feature freeze is not. Feature freeze should only affect the integrated OpenStack release. If you don’t release (i.e. if you don’t special-case certain moments in the development), then feature freezing makes little sense. It’s also not a way to punish people who failed to meet a deadline. There are multiple reasons that a given feature will miss a deadline, and most of those are not the fault of the original author of the feature. We do time-based releases, so some features and some developers will necessarily be caught on the wrong side of the fence at some point and need to wait for the next boat. It’s an artifact of open innovation projects.

Feature freeze (also known as “FF”) is, obviously, about stopping adding new features. You may think of it as artificially blocking your progress, but this has a different effect on other people:

• As was evidenced by the Icehouse cycle, good code reviewers are a scarce resource. The first effect of feature freeze is that it limits the quantity of code reviews and make them all about bugfixes. This lets reviewers concentrate on getting as many bugfixes in as possible before the “release”. It also helps developers spend time on bugfixes. As long as they can work on features, their natural inclination (or their employer orders) might conflict with the project interest at this time in the cycle, which is to make that point in time we call the “release” as bug-free as possible.

• From a QA perspective, stopping the addition of features means you can spend useful time testing “in real life” how OpenStack behaves. There is only so much our automated testing will catch. And it’s highly frustrating to spend time testing software that constantly changes under you.

• QA is not the only group that needs to catch up. For the documentation team, the I18N team, feature freeze is essential. It’s difficult to write documentation if you don’t know what will be in the end product. It’s frustrating to translate strings that are removed or changed the next day.

• And then you have all the downstream consumers of the release that can use time to prepare it. Packagers need software that doesn’t constantly change and add dependencies, so that they can prepare packages for OpenStack projects that are released as close to our release date as possible. The marketing team needs time to look into what was produced over the cycle and arrange it in key messages to communicate to the outside world at release time.

• Finally, for release management, feature freeze is a tool to reduce risk. The end goal is to avoid introducing an embarassing regression just before release. By gradually limiting the impact of what we accept in the release branch (using feature freeze, but also using the RC dance that comes next), we try our best to prevent that.

Exceptions

For all these groups, it’s critical that we stop adding features, changing behavior, adding new configuration options, or changing translatable strings as early as possible. Of course, it’s a trade-off. There might be things that are essential to the success of the release, or things that are obviously risk-limited. That’s why we have an exception process: the Feature Freeze exceptions (“FFEs”).

Feature freeze exceptions may be granted by the PTL (with the friendly but strong advice from the release management team). The idea is to weigh the raw benefit of having that feature in the release, against the complexity of the code that is being brought in, its risk of causing a regression, and how deep we are in feature freeze already. A self-contained change that is ready to merge a few days after feature freeze is a lot more likely to get an exception than a refactoring of a key layer that still needs some significant work to land. It also depends on how many exceptions were already granted on that project, because at some point adding anything more just causes too much disruption.

It’s a difficult call to make, and the release management team is here to help the PTLs make it. If your feature gets denied, don’t take it personally. As you saw there are a large number of factors involved. Our common goal is to raise the quality of the end release, and every feature freeze exception we grant is a step away from that. We just can’t take that many steps back and still guaranteeing we’ll win the race.

 

The dilemma of open innovation

Open innovation vs. proprietary innovation

For companies, there are two ways to develop open source projects. The first one is to keep design and innovation inside your corporate borders, and only accept peripheral contributions. In that case you produce open source software, but everything else resembles traditional software development: you set the goals and roadmap for your product, and organize your development activity to meet those goals, using Agile or waterfall methodologies.

The second one is what we call open innovation: build a common and level playing field for contributions from anywhere, under the auspices of an independent body (foundation or other). In that case you don’t really have a roadmap: what ends up in the software is what the contributors manage to push through a maintainers trust tree (think: the Linux kernel) or a drastic code review / CI gate (think: OpenStack). Products or services are generally built on top of those projects and let the various participants differentiate on top of the common platform.

Now, while I heavily prefer the second option (which I find much closer to the ideals of free software), I recognize that both options are valid and both are open source. The first one ends up attracting far less contributions, but it works quite well for niche, specialized products that require some specific know-how and where focused product design gives you an edge. But the second works better to reach universal adoption and total world domination.

A tragedy of the commons

The dilemma of open innovation is that it’s a natural tragedy of the commons. You need strategic contributions to keep the project afloat: people working on project infrastructure, QA, security response, documentation, bugfixing, release management which do not directly contribute to your employer baseline as much as a tactical contribution (like a driver to interface with your hardware) would. Some companies contribute those necessary resources, while some others just get the benefits of monetizing products or services on top of the platform without contributing their fair share. The risk, of course, is that the strategic contributor gets tired of paying for the free rider.

Open innovation is a living ecosystem, a society. Like all societies, it has its parasites, its defectors, those which don’t live by the rules. And like all societies, it actually needs a certain amount of defectors, as it makes the society stronger and more able to evolve. The trick is to keep the amount of parasites down to a tolerable level. In our world, this is usually done by increasing the difficulty or the cost of defecting, while reducing the drawbacks or the cost of cooperating.

Keeping our society healthy

In his book Liars and Outliers, Bruce Schneier details the various knobs a society can play with to adjust the number of defectors. There are moral pressures, reputational pressures, institutional pressures and security pressures. In open innovation projects, moral pressures and security pressures don’t work that well, so we usually use a combination of institutional pressures (licensing, trademark rules) and reputational pressures (praising contributors, shaming free riders) to keep defectors to an acceptable level.

Those are challenges that are fully understood and regularly applied in the Linux kernel project. For OpenStack, the meteoritic growth of the project (and the expertise land-grab that came with it) protected us from the effects of the open innovation dilemma so far. But the Technical Committee shall keep an eye on this dilemma and be ready to adjust the knobs if it starts becoming more of a problem. Because at some point, it will.

StoryBoard sprint in Brussels

StoryBoard is a project I started a few months ago. We have been running into a number of issues with Launchpad (inability to have blueprints spanning multiple code bases, inability to have flexible project group views, inability to use non-Launchpad OpenID for login…), and were investigating replacements. I was tired to explain why those alternatives wouldn’t work for our task tracking, so I started to describe the features we needed, and ended up writing a proof-of-concept to show a practical example.

That proof-of-concept was sufficiently compelling that the Infrastructure team decided we should follow the path of writing our own tool. To be useful, task tracking for complex projects has to precisely match your workflow. And the POC proved that it wasn’t particularly difficult to write. Then people from HP, Mirantis and RedHat joined this effort.

My Django-based proof-of-concept had a definite last-century feel to it, though. We wanted a complete REST API to cover automation and scripting needs, and multiple clients on top of that. Time was ripe for doing things properly and start building a team effort around this. Time was ripe for… the StoryBoard sprint.

We gathered in Brussels for two days in advance of FOSDEM, in a meeting room sponsored by the OpenStack Foundation (thanks!). On day 2 we were 12 people in the room, which was more than we expected !

Colette helped us craft a mission statement and structure our goals. Michael presented an architecture (static JS client on top of OpenStack-like REST service) that we blessed. Jaromir started to draw wireframes. Sergey, Ruslan and Nikita fought uncooperative consulates and traveled at night to be present on day 2. We also confirmed a number of other technology choices (Bootstrap, AngularJS…). We discussed the basic model, bikeshedded over StoryBoard vs. Storyboard and service URLs. We got a lot covered, had very few breaks, ate nice food and drank nice beer. But more importantly, we built a strong set of shared understandings which should help us make progress as a united team going forward.

We have automated testing and continuous deployment set up now, and once the initial basic functionality is up (MVP0) we should iterate fast. The Infrastructure program is expected to be the first to dogfood this, and the goal is to have something interesting to present to other programs by the Atlanta summit. To participate or learn more about StoryBoard, please join us on #storyboard on Freenode IRC, or at our weekly meeting.

Icehouse-2 velocity analysis

Looking at our recently-concluded icehouse-2 development timeframe, we landed far less features and bugfixes than we wanted and expected. That created concerns about us losing our velocity, so I run a little analysis to confirm or deny that feeling.

Velocity loss ?

If we compare icehouse to the havana cycle and focus on implemented blueprints (not the best metric), it is pretty obvious that icehouse-2 was disappointing:

havana-1: 63
havana-2: 100
icehouse-1: 69
icehouse-2: 50

Using the first milestone as a baseline (growth of 10% expected), we should have been at 110 blueprints, so we are at 45% of the expected results. That said, looking at bugs gives a slightly different picture:

havana-1: 671
havana-2: 650
icehouse-1: 738
icehouse-2: 650

The first milestone baseline again gives a 10% expected growth, which means the target was 715 bugs… but we “only” fixed 650 bugs (like in havana-2). So on the bugfixes front, we are at 91% of the expected result.

Comparing with grizzly

But havana is not really the cycle we should compare icehouse with. We should compare with another cycle where the end-of-year holidays hit during the -2 milestone development… so grizzly. Let’s look at the number of commits (ignoring merges), for a number of projects that have been around since then. Here are the results for nova:

nova grizzly-1: 549 commits
nova grizzly-2: 465 commits
nova icehouse-1: 548 commits
nova icehouse-2: 282 commits

Again using the -1 milestone as a baseline for expected growth (here +0%), nova in icehouse-2 ended up at 61% of the expected number of commits. The results are similar for neutron:

neutron grizzly-1: 155 commits
neutron grizzly-2: 128 commits
neutron icehouse-1: 203 commits
neutron icehouse-2: 110 commits

Considering the -1 milestones gives an expected growth in commits between grizzly and icehouse of +31%. Icehouse-2 is at 66% of expected result. So not good but not catastrophic either. What about cinder ?

cinder grizzly-1: 86 commits
cinder grizzly-2: 54 commits
cinder icehouse-1: 175 commits
cinder icehouse-2: 119 commits

Now that’s interesting… Expected cinder growth between grizzly and icehouse is +103%. Icehouse-2 scores at 108% of the expected, grizzly-based result.

keystone grizzly-1: 95 commits
keystone grizzly-2: 42 commits
keystone icehouse-1: 116 commits
keystone icehouse-2: 106 commits

That’s even more apparent with keystone, which had a quite disastrous grizzly-2: expected growth is +22%, Icehouse-2 is at 207% of the expected result. Same for Glance:

glance grizzly-1: 100 commits
glance grizzly-2: 38 commits
glance icehouse-1: 98 commits
glance icehouse-2: 89 commits

Here we expect 2% less commits, so based on grizzly-2 we should have had 37 commits… icehouse-2 here is at 240% !

In summary, while it is quite obvious that we delivered far less than we wanted to, due to the holidays and the recent gate issues, from a velocity perspective icehouse-2 is far from being disastrous if you compare it to the last development cycle where the holidays happened at the same time in the cycle. Smaller projects in particular have handled that period significantly better than last year.

We just need to integrate the fact that the October – April cycle includes a holiday period that will reduce our velocity… and lower our expectations as a result.

OpenStack @ FOSDEM ’14

Every year, free and open source developers from all over Europe and beyond converge in cold Brussels for a week-end of talks, hacking and beer. OpenStack will be present !

We have a number of devroom and lightning talks already scheduled:

Saturday 12:20 in Chavanne (Virtualization and IaaS devroom)
Autoscaling best practices
Marc Cluet will look into autoscaling using Heat and Ceilometer as examples.

Saturday 13:00 in Chavanne (Virtualization and IaaS devroom)
Network Function Virtualization and Network Service Insertion and Chaining
Balaji Padnala will present NFV and how to deploy it using OpenStack and OpenFlow Controller.

Saturday 13:40 in Chavanne (Virtualization and IaaS devroom)
oVirt and OpenStack Storage (present and future)
Federico Simoncelli will cover integration between oVirt and Glance/Cinder for storage needs.

Saturday 15:00 in Chavanne (Virtualization and IaaS devroom)
Why, Where, What and How to contribute to OpenStack
I will go through a practical introduction to OpenStack development and explain why you should contribute if you haven’t already.

Saturday 16:20 in Chavanne (Virtualization and IaaS devroom)
Hypervisor Breakouts – Virtualization Vulnerabilities and OpenStack Exploitation
Rob Clark will explore this class of interesting vulnerabilities from an OpenStack perspective.

Saturday 17:40 in Chavanne (Virtualization and IaaS devroom)
oVirt applying Nova scheduler concepts for data center virtualization
Gilad Chaplik will present how oVirt could reuse OpenStack Nova scheduling concepts.

Sunday 10:00 in U.218A (Testing and automation devroom)
Preventing craziness: a deep dive into OpenStack testing automation
Me again, in a technical exploration on the OpenStack gating system and its unique challenges.

Sunday 13:40 in Chavanne (Virtualization and IaaS devroom)
Tunnels as a Connectivity and Segregation Solution for Virtualized Networks
Join Assaf Muller for an architectural, developer oriented overview of (GRE and VXLAN) tunnels in OpenStack Networking.

Sunday 16:20 in Chavanne (Virtualization and IaaS devroom)
Bring your virtualized networking stack to the next level
Mike Kolesnik will look into integration opportunities between oVirt and OpenStack Neutron.

Sunday 17:00 in Ferrer (Lightning talks room)
Putting the PaaS in OpenStack
Dirk Diane Mueller will give us an update on cross-community collaboration between OpenStack, Solum, Docker and OpenShift.

Sunday 17:20 in Ferrer (Lightning talks room)
Your Complete Open Source Cloud
Dave Neary should explain how to mix OpenStack with oVirt, OpenShift and Gluster to build a complete private cloud.

We’ll also have a booth manned by OpenStack community volunteers ! I hope to see you all there.

Lessons and outcomes from the Hong-Kong summit

Just back from an amazing week at the OpenStack Summit in Hong-Kong, I would like to share a number of discussions we had (mainly on the release management track) and mention a few things I learned there.

First of all, Hong-Kong is a unique city. Skyscrapers built on vertiginous slopes, crazy population density, awesome restaurants, shops everywhere… Everything is clean and convenient (think: Octopus cards), even as it grows extremely fast. Everyone should go there at least one time in their lives !

On the Icehouse Design Summit side, the collaboration magic happened again. I should be used to it by now, but it is still amazing to build this level playing field for open design, fill it with smart people and see them make so much progress over 4 days. We can still improve, though: for example I’ll make sure we get whiteboards in every room for the next time :). As was mentioned in the feedback session, we are considering staggering the design summit and the conference (to let technical people participate to the latter), set time aside to discuss cross-project issues, and set up per-project space so that collaboration can continue even if there is no scheduled “session” going on.

I have been mostly involved in release management sessions. We discussed the Icehouse release schedule, with a proposed release date of April 17, and the possibility to have a pre-designated “off” week between release and the J design summit. We discussed changes in the format of the weekly project/release status meeting, where we should move per-project status updates off-meeting to be able to focus on cross-project issues instead. During this cycle we should also work on streamlining library release announcements. For stable branch maintenance, we decided to officially drop support for version n-2 by feature freeze (rather than at release time), which reflects more accurately what ended up being done during the past cycles. The security support is now aligned to stable branch support, which should make sure the vulnerability management team (VMT) doesn’t end up having to maintain old stable branches that are already abandoned by the stable branch maintainers. Finally, the VMT should review the projects from all official programs to come up with a clear list of what projects are actually security-supported and which aren’t.

Apart from the release management program, I’m involved in two pet projects: Rootwrap and StoryBoard. Rootwrap should be split from the oslo-incubator into its own package early in the Icehouse cycle, and its usage in Nova, Cinder and Neutron should be reviewed to result in incremental strengthening. StoryBoard (our next-generation task tracker) generated a lot of interest at the summit, I expect a lot of progress will be made in the near future. Its architecture might be overhauled from the current POC, so stay tuned.

Finally, it was great meeting everyone again. Our PTLs and Technical Committee members are a bunch of awesome folks, this open source project is in great hands. More generally, it seems that we not only designed a new way of building software, we also created a network of individuals and companies interested in that kind of open collaboration. That network explains why it is so easy for people to jump from one company to another, while continuing to do the exact same work for the OpenStack project itself. And for developers, I think it’s a great place to be in: if you haven’t already, you should definitely consider joining us.