Free software has been my career for a long time -- nothing else since 1999 -- and it continues to be a happy surprise each time I find a way to continue that streak.
The latest is that I'm being funded for a couple of years to work part-time on git-annex. The funding comes from the DataLad project, which was recently awarded a grant by the National Science Foundation. DataLad folks (at Dartmouth College and at Magdeburg University in Germany) are working on providing easy access to scientific data (particularly neuroimaging). So git-annex will actually be used for science!
I'm being funded for around 30 hours of work each month, to do general work on the git-annex core (not on the webapp or assistant). That includes bugfixes and some improvements that are wanted for DataLad, but are all themselves generally useful. (see issue list)
This is enough to get by on, at least in my current living situation. It would be great if I could find some funding for my other work time -- but it's also wonderful to have the flexibility to spend time on whatever other interesting projects I might want to.
I left Debian. I don't really have a lot to say about why, but I do want to clear one thing up right away. It's not about systemd.
As far as systemd goes, I agree with my friend John Goerzen:
I promise you – 18 years from now, it will not matter what init Debian chose in 2014. It will probably barely matter in 3 years.
And with Jonathan Corbet:
However things turn out, if it becomes clear that there is a better solution than systemd available, we will be able to move to it.
I have no problem with trying out a piece of Free Software, that might have abrasive authors, all kinds of technical warts, a debatable design, scope creep etc. None of that stopped me from giving Linux a try in 1995, and I'm glad I jumped in with both feet.
It's important to be unafraid to make a decision, try it out, and if it doesn't work, be unafraid to iterate, rethink, or throw a bad choice out. That's how progress happens. Free Software empowers us to do this.
Debian used to be a lot better at that than it is now. This seems to have less to do with the size of the project, and more to do with the project having aged, ossified, and become comfortable with increasing layers of complexity around how it makes decisions. To the point that I no longer feel I can understand the decision-making process at all ... or at least, that I'd rather be spending those scarce brain cycles on understanding something equally hard but more useful, like category theory.
It's been a long time since Debian was my main focus; I feel much more useful when I'm working in a small nimble project, making fast and loose decisions and iterating on them. Recent events brought it to a head, but this is not a new feeling. I've been less and less involved in Debian since 2007, when I dropped maintaining any packages I wasn't the upstream author of, and took a year of mostly ignoring the larger project.
Now I've made the shift from being a Debian developer to being an upstream author of stuff in Debian (and other distros). It seems best to make a clean break rather than hang around and risk being sucked back in.
My mailbox has been amazing over the past week by the way. I've heard from so many friends, and it's been very sad but also beautiful.
Propellor has supported docker containers for a "long" time, and it works great. This week I've worked on adding more container support.
docker containers (revisited)
The syntax for docker containers has changed slightly. Here's how it looks now:
example :: Host
example = host "example.com"
& Docker.docked webserverContainer
webserverContainer :: Docker.Container
webserverContainer = Docker.container "webserver" (Docker.latestImage "joeyh/debian-stable")
& os (System (Debian (Stable "wheezy")) "amd64")
& Docker.publish "80:80"
& Apt.serviceInstalledRunning "apache2"
& alias "www.example.com"
That makes example.com have a web server in a docker container, as you'd expect, and when propellor is used to deploy the DNS server it'll automatically make www.example.com point to the host (or hosts!) where this container is docked.
I use docker a lot, but I have drank little of the Docker KoolAid. I'm not keen on using random blobs created by random third parties using either unreproducible methods, or the weirdly underpowered dockerfiles. (As for vast complicated collections of containers that each run one program and talk to one another etc ... I'll wait and see.)
That's why propellor runs inside the docker container and deploys whatever configuration I tell it to, in a way that's both replicatable later and lets me use the full power of Haskell.
Which turns out to be useful when moving on from docker containers to something else...
systemd-nspawn containers
Propellor now supports containers using systemd-nspawn. It looks a lot like the docker example.
example :: Host
example = host "example.com"
& Systemd.persistentJournal
& Systemd.nspawned webserverContainer
webserverContainer :: Systemd.Container
webserverContainer = Systemd.container "webserver" chroot
& Apt.serviceInstalledRunning "apache2"
& alias "www.example.com"
where
chroot = Chroot.debootstrapped (System (Debian Unstable) "amd64") Debootstrap.MinBase
Notice how I specified the Debian Unstable chroot that forms the basis of this container. Propellor sets up the container by running debootstrap, boots it up using systemd-nspawn, and then runs inside the container to provision it.
Unlike docker containers, systemd-nspawn containers use systemd as their
init, and it all integrates rather beautifully. You can see the container
listed in systemctl status, including the services running inside it,
use journalctl to examine its logs, etc.
But no, systemd is the devil, and docker is too trendy...
chroots
Propellor now also supports deploying good old chroots. It looks a lot like the other containers. Rather than repeat myself a third time, and because we don't really run webservers inside chroots much, here's a slightly different example.
example :: Host
example = host "mylaptop"
& Chroot.provisioned (buildDepChroot "git-annex")
buildDepChroot :: Apt.Package -> Chroot.Chroot
buildDepChroot pkg = Chroot.debootstrapped system Debootstrap.BuildD dir
& Apt.buildDep pkg
where
dir = /srv/chroot/builddep/"++pkg
system = System (Debian Unstable) "amd64"
Again this uses debootstrap to build the chroot, and then it runs propellor inside the chroot to provision it (btw without bothering to install propellor there, thanks to the magic of bind mounts and completely linux distribution-independent packaging).
In fact, the systemd-nspawn container code reuses the chroot code, and so turns out to be really rather simple. 132 lines for the chroot support, and 167 lines for the systemd support (which goes somewhat beyond the nspawn containers shown above).
Which leads to the hardest part of all this...
debootstrap
Making a propellor property for debootstrap should be easy. And it was, for Debian systems. However, I have crazy plans that involve running propellor on non-Debian systems, to debootstrap something, and installing debootstrap on an arbitrary linux system is ... too hard.
In the end, I needed 253 lines of code to do it, which is barely one magnitude less code than the size of debootstrap itself. I won't go into the ugly details, but this could be made a lot easier if debootstrap catered more to being used outside of Debian.
closing
Docker and systemd-nspawn have different strengths and weaknesses, and there are sure to be more container systems to come. I'm pleased that Propellor can add support for a new container system in a few hundred lines of code, and that it abstracts away all the unimportant differences between these systems.
PS
Seems likely that systemd-nspawn containers can be nested to any depth. So, here's a new kind of fork bomb!
infinitelyNestedContainer :: Systemd.Container
infinitelyNestedContainer = Systemd.container "evil-systemd"
(Chroot.debootstrapped (System (Debian Unstable) "amd64") Debootstrap.MinBase)
& Systemd.nspawned infinitelyNestedContainer
Strongly typed purely functional container deployment can only protect us against a certian subset of all badly thought out systems. ;)
Note that the above was written in 2014 and some syntatix details have changed. See the documentation for Propellor.Property.Chroot, Propellor.Property.Debootstrap, Propellor.Property.Docker, Propellor.Property.Systemd for current examples.