devstack in a lxd container

When doing Openstack development, it's hard to avoid using devstack to build and maintain a development environment. While devstack is a neat tool, it can't possibly handle every single case and environment and OS and network configuration. At least, it's very difficult, and requires no end of (sometimes archaic) configuration settings. This is all to say that I appreciate those that work on devstack, but I often find myself futzing more with devstack than actually writing code, and I find this less than optimal.

As I've been working an nova-lxd I continue to run into weird networking issues, mostly related to neutron bridges and openvswitch. I got into a weird case where devstack's stack.sh script couldn't divine what interfaces need be connected to openvswitch bridge, etc. This meant that I could fire up nova instances, but I could never access them, because the routing never worked.

While most other developers turn to virtual machines, it seemed a little counterproductive that I'm working on the LXD hypervisor for Openstack Nova. Naturally, I turned to trying devstack in a lxd container.

(I'm using Ubuntu 16.04; varying mileage and all that)

$ lxc launch ubuntu:16.04 devstack -c security.privileged=true -c security.nesting=true
$ lxc exec devstack -- apt-get install -y avahi-daemon

In this case, you need security.nesting to fire up lxd instances (which is required for me, as I'm developing nova-lxd). I'm not sure whether I need security.privileged here, but it certainly made things a little easier for me. I install avahi-daemon so I can do ssh devstack.local without remembering ip addresses.

At this point, I created an account in the container that is identical to my current username (which, as you may infer, is rockstar). You could create a stack user if you're one of those people that creates a specific devstack user.

$ user=`whoami`
$ lxc exec devstack -- adduser --disabled-password --gecos "" $user

...and to make things easier in my development environment, I added my home directory into the container.

$ lxc config device add devstack homedir disk source=/home/$user path=/home/$user

And now, the devstack stuff.

$ git clone https://git.openstack.org/openstack-dev/devstack

At this point, you've got a container with a real eth0, instead of some sort of ens0lmnopwftbbq random device name. Here's my local.conf:

[[local|localrc]]

HOST_IP=10.193.59.173 # set this to your IP
FLAT_INTERFACE=eth0

ADMIN_PASSWORD=password
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD

ENABLED_SERVICES=rabbit,mysql,key
ENABLED_SERVICES+=,g-api,g-reg
ENABLED_SERVICES+=,n-cpu,n-api,n-crt,n-obj,n-cond,n-sch,n-novnc,n-cauth
ENABLED_SERVICES+=,neutron,q-svc,q-agt,q-dhcp,q-meta,q-l3

disable_service n-net

enable_plugin nova-lxd https://git.openstack.org/openstack/nova-lxd

We explicitly disable nova-network (it doesn't really work with nova-lxd). I don't use horizon at all, and cinder requires some extra settings container configuration, as it requires lvm (I don't know the specifics here; I may explore it later).

Once you're all configured, run ./stack.sh and watch the world burn (for a bit). Once complete, if you're like me and are running nova-lxd, you can fire up a nova instance with the following.

$ OS_USERNAME=admin
$ OS_PASSWORD=password
$ OS_PROJECT_NAME=demo
$ OS_AUTH_URL=http://10.193.59.173/identity
$
$ wget http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz
$ openstack image create --disk-format raw --container-format bare --file xenial-server-cloudimg-amd64-root.tar.gz xenial
$ openstack server create --image xenial --flavor m1.tiny test-container