For years I've recorded solar panel data by hand. Filled two notebooks with columns of figures. My new charge controller, an EPsolar Tracer-BN, finally let me automate it.
 |
You can explore my home power data here: http://house.joeyh.name/
(click and drag to zoom)
The web interface loads the RRD files into a web browser using javascriptRRD. I wrote a haskell program that drives the epsolar-tracer python library to poll for data, and stores it in RRD files. Could have used collectd or something, but the interface to the charge controller is currently a bit flakey and I have to be careful about retries and polling frequencies. Also I wanted full control over how much data is stored in the RRD files.
Three years ago, I realized that propellor (my configuration management system that is configured using haskell) could be used as an installer for Debian (or other versions of Linux). In propellor is d-i 2.0, I guessed it would take "a month and adding a few thousand lines of code".
I've now taken that month, and written that code, and I presented the result at DebConf yesterday. I demoed propellor building a live Debian installation image, and then handed it off to a volenteer from the audience to play with its visual user interface and perform the installation. The whole demo took around 20 minutes, and ended with a standard Debian desktop installation. (Video)
The core idea is to reuse the same configuration management system for several different purposes.
- Building a bootable disk image that can be used as both a live system and as an OS installer.
- Running on that live system, to install the target system. Which can just involve copying the live system to the target disk and then letting the configuration management system make the necessary changes to get from the live system configuration to the target system configuration.
- To support such things as headless arm boards, building customized images tuned for the target board and use case, that can then simply be copied to the board to install.
- Optionally, running on the installed system later, to further customize it. Starting from the same configuration that produced the installed system in the first place.
There can be enormous code reuse here, and improvements made for one of those will often benefit all the rest as well.
Once everything is handled by configuration management, all user interface requirements become just a matter of editing the configuration. Including:
- A user interface that runs on the live system and gets whatever input is needed to install to the target system. This is really just a config editor underneath. I built a prototype gamified interface that's as minimal as such an interface could get.
- With a regular text editor, of course. This is the equivalent of preseeding in d-i, giving advanced users full control over the system that gets built. Unlike with preseeding, users have the full power of a configuration management system, so can specify precisely the system they want installed.
- A separate user interface for customizing disk images, for arm boards and similar use cases. This would run on a server, or on the user's own laptop.
That's the gist of it. Configuration management reused for installation and image building, and multiple editor interfaces to make it widely usable.
I was glad, sitting in to a BoF session before my talk, that several people in Debian are already thinking along similar lines. And if Debian wanted to take this work and run with it, I'd be glad to assist as propellor's maintainer. But the idea is more important than the code and I hope my elaboration of it helps point a way if not the way.
While what I've built installs Debian, little of it is Debian-specific. It would probably be easy to port it to Arch Linux, which propellor already supports. There are Linux-specific parts, so porting to FreeBSD would be harder, but propellor knows, at the type level which OSs properties support, which will ease porting.
GuixSD and NixOS already use configuration management for installation, and were part of my inspiration. I've extended what they do in some ways (in other ways they remain far ahead).
The code is here. And here are some links to more details about what I built, and ideas encountered along the way:
- In Functional Reactive Propellor I found a way to express the commonalities and differences between the installer's configuration and the target system's configuration. This lets the installer disk image be copied to the target and the minimum work be done to convert it into the desired target system.
- In disk partitioning nitty gritty I tackled configuring the partition table of the target system. I extended a DSL propellor already used for partitioning disk images. The result is basically partman in 1/100th the lines of code.
- In end in sight I found a way to make propellor
build disk images super fast, so a new 5 gb disk image can be ready in 30
seconds, a quarter of the time that it takes to write a 5 gb file with
dd. - For completeness sake, I also devblogged about unfortunately needing a progress bar, picking the disk to install to and installing grub, dependency yak shaving, high bandwidth propellor hacking and finishing touches