UDS Lucid

This week, Dallas hosts the Ubuntu Developer Summit for the Lucid Lynx release. This is the key moment where we define what will be done for Ubuntu 10.04 LTS, and discuss how it will be done. There will be plenty of interesting sessions in all the tracks, and sometimes I wish I could attend two sessions at the same time. In the server track, Monday will have a special focus on applications, Tuesday on cloud images, Wednesday on Ubuntu Enterprise Cloud, Thursday on infrastructure sessions, and Friday on testing.

I’ll just highlight one session that I think represent the QA focus of this release: “Server usability papercuts”. The 100 papercuts project focused on common annoyances, easy-to-fix bugs that hinder the user experience in Ubuntu Desktop. It was a great idea, and I think a great success. Several members of the server team think this can be transposed to the Ubuntu Server product as well. Some server packages may be counter-intuitive, suffer from bad default configurations or force the systems administrator to stupid repetitive tasks. Some other packages might not work well together, while it is good practice to use them together. We could identify some of those and fix them, to improve the general server experience for this release. Interested ? Join us today, 15:00 Dallas time !

Run your own Ubuntu Enterprise Cloud, part 3

In part 1 and part 2 of this series, we saw how to set up a minimal cloud infrastructure and bundle a basic image (and test it). In this final article, we’ll play with our cloud from an end-user perspective.

Setting up the web UI

First of all, before accepting end users, as the administrator of the cloud you will have to setup a few things on the web UI. Using your favorite browser, you should:

• Open https://cloudcontroller:8443/
• Log in using the default user/password: admin/admin
• Change the default password, setup the cloud admin email address
• Logout

Setting up the cloud client

We’ll use Ubuntu 9.10 beta for this purpose, as it includes all the needed packages, and it’s so great ! You will have to install the following packages:

$ sudo apt-get install euca2ools unzip
 

Registering on UEC, getting credentials

As the end-user, fire up your favorite browser and:

• Open https://cloudcontroller:8443/
• Click “Apply” and enter your end user details

If you set up the email correctly on your cloud controller, it should send an email to the cloud admin address asking him to approve that request. Follow the instructions on that email to approve the account as the admin.

You should then get an email at the end user email address asking you to confirm the account request. Follow the instructions on that email, then you can log in on the web UI:

• Open https://cloudcontroller:8443/
• Login using your end user username and password
• Click “Download Credentials” in the “Credentials” tab
• Note the EMI reference you can use on the “Images” tab

Starting up an instance

You should unzip the credentials zipfile you just downloaded, then source the eucarc file and test the connection:

$ unzip euca2-enduser-x509.zip
$ . eucarc
$ euca-describe-availability-zones verbose
 

Setup a SSH key and allow connection to the SSH port:

$ euca-add-keypair enduserkey > enduserkey.priv
$ chmod 0600 enduserkey.priv
$ euca-authorize default -P tcp -p 22 -s 0.0.0.0/0
 

Then starting up an instance is just a matter of passing the right EMI and type:

$ euca-run-instances -k enduserkey emi-XXXXXXXX -t c1.medium
 

Enjoy !

Run your own Ubuntu Enterprise Cloud, part 2

In part 1 of this series, we saw how to install the cloud infrastructure. In this article, we’ll bundle and upload an EMI (Eucalyptus Machine Image), based on Ubuntu Server 9.10 Beta, and validate that we can run an instance of it.

Download required elements

Go to the cloud/cluster controller and download the required items.

For a 64-bit image:

$ URL="http://uec-images.ubuntu.com/releases/karmic"
$ wget -O image.gz $URL/beta/ubuntu-uec-karmic-amd64.img.gz
$ wget -O vmlinuz $URL/beta/ubuntu-uec-karmic-amd64-vmlinuz-2.6.31-11-server
$ wget -O initrd $URL/beta/ubuntu-uec-karmic-amd64-initrd.img-2.6.31-11-server
 

For a 32-bit image:

$ URL="http://uec-images.ubuntu.com/releases/karmic"
$ wget -O image.gz $URL/beta/ubuntu-uec-karmic-i386.img.gz
$ wget -O vmlinuz $URL/beta/ubuntu-uec-karmic-i386-vmlinuz-2.6.31-11-generic-pae
$ wget -O initrd $URL/beta/ubuntu-uec-karmic-i386-initrd.img-2.6.31-11-generic-pae
 

Bundle the EMI

First you should unpack and resize your image to the desired size, lets say 4Gb. This can take a very long time (15 minutes !) on slow disks as you unpack 10Gb-worth of image space:

$ zcat -f image.gz | cp --sparse=always /dev/stdin image
$ e2fsck -f image
$ resize2fs image 4G
$ truncate --size=4G image
 

Then bundle and upload the kernel:

$ . eucarc
$ euca-bundle-image -i vmlinuz --kernel true
$ euca-upload-bundle -b ueckernel -m /tmp/vmlinuz.manifest.xml
$ euca-register ueckernel/vmlinuz.manifest.xml
IMAGE     eki-KKKKKKKK
 

Take note of the EKI reference, you’ll need it later. Then bundle, upload and register the ramdisk:

$ euca-bundle-image -i initrd --ramdisk true
$ euca-upload-bundle -b uecramdisk -m /tmp/initrd.manifest.xml
$ euca-register uecramdisk/initrd.manifest.xml
IMAGE     eri-RRRRRRRR
 

Take note of the ERI reference. Finally, bundle the image with the kernel and ramdisk, upload and register:

$ euca-bundle-image -i image --kernel eki-KKKKKKKK --ramdisk eri-RRRRRRRR
$ euca-upload-bundle -b uecimage -m /tmp/image.manifest.xml
$ euca-register uecimage/image.manifest.xml
IMAGE     emi-XXXXXXXX
 

Bundling will also take a lot of time ! Take note of your EMI reference.

Start an instance of your EMI

In order to access your instance using SSH, you’ll need to setup a few one-time things (create a SSH key and authorize access to port 22 of your instances):

$ euca-add-keypair mykey > mykey.priv
$ chmod 0600 mykey.priv
$ euca-authorize default -P tcp -p 22 -s 0.0.0.0/0
 

Now it’s time to start your instance !

$ euca-run-instances -k mykey emi-XXXXXXXX -t c1.medium
 

The “c1.medium” VM type is sufficient by default to run a 4Gb instance. You should take note of the i-YYYYYYYY reference that is displayed on your INSTANCE line. The first time you start an EMI, it can take some time (like 10 minutes) to move from “pending” state to “running”, depending on size. You can use the following command to automatically watch the output of euca-describe-instances, every 5 seconds:

$ watch -n 5 euca-describe-instances
 

Take note of the first ZZZ.ZZZ.ZZZ.ZZZ IP address mentioned in the output of the command. When the instance is “running”, ctrl-C to exit watch, then:

$ ssh -i mykey.priv ubuntu@ZZZ.ZZZ.ZZZ.ZZZ
 

You are in ! When you’re done playing with your instance, just run the following command on the cloud/cluster controller.

$ euca-terminate-instances i-YYYYYYYY
 

In the third and last part of this series of articles, we’ll talk about how to run instances from another workstation, as a cloud “customer”.

Run your own Ubuntu Enterprise Cloud, part 1

Ubuntu Enterprise Cloud is the product, powered by Eucalyptus, that allows you to easily run your own Amazon-EC2-like private cloud. It’s a lot simpler than you’d think. With the recent Ubuntu Server 9.10 beta release, you are now able to easily deploy that infrastructure from the CD installer.

Prerequisites

To deploy a minimal cloud infrastructure, you’ll need at least two dedicated systems. One will hold the cloud controller (clc), the cluster controller (cc), walrus (the S3-like storage service) and the storage controller (sc). This one needs fast disks and a reasonably fast processor. The other system(s) are node controllers (nc) that will actually run the instances. These ones need CPUs with VT extensions, lots of CPU cores, lots of RAM, and fast disks. For both, 64-bit support is highly recommended.

Installing the cloud/cluster controller

Download the 9.10 Server beta ISO. When you boot, select “Ubuntu Enterprise Cloud install”. When asked whether you want a “Cluster” or a “Node” install, select “Cluster”. It will ask two other cloud-specific questions during the course of the install:

1. Name of your cluster: pick any name you want
2. List of IP addresses on the LAN that the cloud can allocate to instances: enter a list of space-separated unused IP addresses on your LAN.

When it reboots, run the following to get the latest eucalyptus package and reboot:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo reboot
 

Installing node controllers

The node controller install is even simpler. Just make sure that you are connected to the network on which the cloud/cluster controller is already running. Take the same ISO, select “Ubuntu Enterprise Cloud install”. It should detect the Cluster and preselect “Node” install for you. That’s all.

It is also recommended to update to the latest 9.10 status:

$ sudo apt-get update
$ sudo apt-get upgrade
 

Connect your node controllers to the cloud

After all nodes are installed, you need to return to the cloud/controller and run the following command to make it “discover” your newly-installed nodes.

$ sudo euca_conf --no-rsync --discover-nodes
 

Confirm all the nodes it finds, and you are done. To check that your private cloud infrastructure is ready to serve, you need to retrieve admin credentials and run euca-describe-availability-zones command. Run the following on your cloud/cluster controller:

$ sudo euca_conf --get-credentials mycreds.zip
$ unzip mycreds.zip
$ . eucarc
$ euca-describe-availability-zones verbose
 

This last command returns a description of the capabilities of your cloud cluster, how many instances of each type you could run on it, for example:

AVAILABILITYZONE   myowncloud                 192.168.1.1
AVAILABILITYZONE   |- vm types                free / max   cpu   ram  disk
AVAILABILITYZONE   |- m1.small                0004 / 0004   1    128     2
AVAILABILITYZONE   |- c1.medium               0004 / 0004   1    256     5
AVAILABILITYZONE   |- m1.large                0002 / 0002   2    512    10
AVAILABILITYZONE   |- m1.xlarge               0002 / 0002   2   1024    20
AVAILABILITYZONE   |- c1.xlarge               0001 / 0001   4   2048    20
 

In part 2 of this series, we’ll cover bundling your first EMI (Eucalyptus Machine Image), based on Ubuntu Server 9.10 Beta. We’ll test it by starting an instance of it. Stay tuned !

On burnout and technical management

Jono posted recently the slidedeck for his famous 12 stages of burnout presentation. I highly recommend this presentation, especially to technical teams working from home.

I think we are especially vulnerable to burnout, with limited social interactions and sporadic discussions with our peers and managers. It’s quite easy to fall into the trap of the first two stages, trying to prove yourself and work harder. And from there we are vulnerable to falling into the spiral of the next ten stages.

This highlights one important role of managers of technical teams: to protect ourselves from this outcome. You shouldn’t have to prove yourself if your manager makes you confident you’re in the right place and you earned your position. You shouldn’t have to work harder if your work output is closely monitored and realistic goals have been set for you.

Technical managers have lots of duties. They must build their team, define objectives, ensure that goals are reached, protect their team from vertical and horizontal organizational hazards… But keeping their team in shape is one of their most important duties, and detecting and avoiding burnout in their team is an important part of it.

Your family and your peers can watch your back and help you recover from it. But a good manager should save you from it.

Join the Ubuntu Java Team meetings

The incredible Emmet Hikory (persia) wrote last week that the new season was open for Java Team meetings.

If you are interested in the state of Java packages in Ubuntu, want to participate in Java software packaging, or are just curious, please feel free to join us on Thursdays, 0900 UTC, on Freenode’s #ubuntu-meeting channel ! Let’s shape the future of Ubuntu Java together.

All you ever wanted to know about Java library packaging

I’ll be giving a packaging training session this Thursday at 12:00 UTC about Java libraries.

• If you are interested in Java and want to learn how to properly package Java libraries for Debian/Ubuntu…
• If you don’t want to touch Java but are curious to know what makes Java library packaging different from other types of packaging…
• If you already know this stuff inside out but want to share some tips and tricks…

Then please join us on Freenode’s #ubuntu-classroom !

Best UDS so far ?

UDS Karmic in Barcelona is shaping up to be a great Ubuntu Developer Summit. The venue looks gorgeous, Barcelona is a great city to spend evenings in, and the schedule is packed with great sessions, twice as many as usual in the Server track.

So if you live nearby and consider joining, you should definitely do. You can learn a lot, we can learn from you, you get to meet all those great Ubuntu developers in real… and dendrobates will offer a beer to you if you are an Ubuntu Server team member.

How to find Java classes in the distribution

Anyone working on packaging Java software could tell you it’s sometimes difficult to determine in which package a given class might be available. It’s because classes are packaged in .jar files, which are in turn packaged in packages. Being able to tell which .jar files are provided in which packages is not enough, since sometimes the .jar name doesn’t convey any useful information on its contents.

To address that problem, I generated java-Contents files that map out which classes are in which jar, for every package that contain .jar files. For bonus points, I added in which repository (main, universe, multiverse) the package was published.

So the java-Contents file contains one line for each class in each .jar file, with the following format:

packagename repository jarfile javaclass
 

You can download java-Contents files at http://people.ubuntu.com/~ttx/java-Contents. Here is a direct link to the Jaunty one.

If for example, you get a failed compilation complaining about being unable to import a org.xmlpull.mxp1.MXParserFactory class, you can tell which package you should Build-Depend on using the following query:

$ zgrep org.xmlpull.mxp1.MXParserFactory jaunty-java-Contents.gz
jspwiki universe /usr/share/jspwiki/WEB-INF/lib/sandler.jar org.xmlpull.mxp1.MXParserFactory
libxpp3-java main /usr/share/java/xpp3-1.1.3.4.O.jar org.xmlpull.mxp1.MXParserFactory
 

This query, for example, will tell you which packages provide javax.activation.* classes:

$ zgrep javax.activation jaunty-java-Contents.gz | \
  awk '{ print $1 " (" $2 ")" }' | uniq
ec2-api-tools (multiverse)
gcc-snapshot (universe)
glassfish-activation (universe)
glassfish-javaee (universe)
glassfishv2 (multiverse)
jspwiki (universe)
libgeronimo-activation-1.1-spec-java (universe)
libgnujaf-java (main)
openjdk-6-jre-headless (main)
roxen4 (universe)
sun-java6-bin (multiverse)
 

Those two examples just show how much redundancy we currently have in our Java packages. But now, we have one more tool to solve such problems. For those interested, the script used to generate those maps is maintained in a bzr branch here.

The server GUI dilemma

Among the many comments on my last article there were calls for a graphical administration interface, i.e. providing by default a desktop environment on the server (like Microsoft or Redhat do) or some web-based admin application. This is an often required feature, and so far we stood firm on the desktop environment part, and did not really invest in integrating a specific web-based admin tool by default.

The reason behind this is that we think the command-line interface is great. We want everyone to use it. CLI is good for you: it allows easier reproduction (I can remember snapshotting every tab of Windows GUI tools to document the changes necessary on a Windows server), it allows scripting. SSH and screen are great. That’s the way things should be done. Most of all, it allows to keep the server stack focused on what it has to do (serve) rather than spending resources running a desktop environment. That’s key to performance and security, and fits well in a growing virtual machines world.

Why do people want admin GUIs ? I hope it’s not to open Firefox on the server console. I guess it’s not to have multiple windows. To get notifications without having to check the logs for events ? That makes sense, but we can still address that in the CLI, as evidenced by Dustin Kirkland’s work on screen-profiles. Status dashboards ? I agree with that. Colorful graphical reports do indeed convey better information than a text-only report. But I think what people are really after is an easier learning curve.

The learning curve for Ubuntu Server is quite steep. Even if most packages come with sane defaults, setting them up usually requires reading a good HOWTO, if you can find one. And for customizing what the server will really do, in most cases our Server Guide won’t go deep enough in the subject, while the upstream documentation might go too much in details.

I think the missing link is discoverability. GUI admin tools allow to play around with the interface and discover what it can do. And it’s true that it’s quite easy to setup Microsoft SQL Server by just exploring the GUI, right clicking on some node items on its admin interface. But it’s also true that when this interface will show its limits those same users will probably have to call Microsoft consultants for help.

Our challenge here is to provide discoverability while not shielding away the users from the CLI tools. Have an easier learning curve while still teaching the user the right way. How can it be done ? For example by providing a task-oriented CLI tool that will show you what you should have executed in the CLI to get the same result. Next time, type this directly. To automate that, just put that line in a script. That’s what SMIT did in AIX. Trying to educate the user while providing an accessible admin tool.

You might reply that you don’t really care, you don’t want to learn stuff, you just want an appliance with limited predetermined choices on a web interface that “just works”. I see value in that, and I wouldn’t prevent such interfaces from running on top of Ubuntu Server. But making it the default, primary way of administering Ubuntu Server ? No. I prefer the server platform to empower our users, rather than limiting their options.