I recently spent a little time adding support for isohybrid images to d-i. isohybrid is a tricksy boot sector hack that allows copying a CD image direct to a USB stick and booting it.

I used it in anger for the first time yesterday, and it worked so well compared with the old method of booting d-i from USB sticks that I feel I need to blog about it so others can enjoy the fun..

The documentation on the web site has not caught up yet, but this is already usable with current d-i daily builds. For now only in the netboot mini.iso. That can be written directly to a USB stick or USB disk to make it boot d-i.

cat mini.iso > /dev/sdX

The netboot mini.iso is so named because it's functionally identical to netbooting d-i. Except without the need to set up a tftp server. Besides being a 16mb download that will fit on even your oldest USB stick, another advantage to using the mini.iso is that you don't even need to use a separate USB stick to boot the installer. You can write the mini.iso directly to the same disk that d-i will be installing to! (Particularly handy for installing debian to USB disks.)

(This is also very handy for d-i development and testing, since a mini.iso can be written out to a USB key about as fast as a test machine will post..)

adding firmware

I also arranged for there to be a second partition on the mini.iso. The firmware partition is most easily accessed by unplugging and replugging the drive after writing the mini.iso to it -- then it will show up as /dev/sdX2 (or "Firmware" in a GUI). If you need to add firmware to d-i, you can just mount it and extract the firmware tarball into it. d-i will then automatically find, use, and install the firmware.

isohybrid with other Debian CD images

All this will also work with other larger Debian CD images, but we're not yet building the larger images with isohybrid, due to issues with jidgo. However, you can just run isohybrid yourself (from the syslinux package) on the CD image you download, and if it's got a recent version of d-i on it, d-i will be able to find and use the hybrid CD image after booting.

That doesn't add the firmware partition to the image. If you need to do that, you could use this script.

I should probably have researched the batteries in this old solar-powered house before I installed a charge controller. I had assumed that the 24 6-volt batteries in the house's battery bay were standard lead-acid deep cycle batteries. It wasn't until I stumbled over a receipt from 1997 that I learned that the batteries are really NiCads. Specifically Saft model STM5-180.

Thirty of these were purchased in 1997 from TVA in Chatanooga, for $350 total. An amazing price, since a single deep cycle battery is in that ballpark, new. These batteries were used, they had been in a "bus" -- perhaps it was a 1995 Chrysler TEVan, or maybe one of the pilot electric buses running there in the late 90's. They were probably not in this '79 VW TVA Bus. All mentions of these batteries I can find involve electric vehicles -- it's unusual for them to be used to power a house.

Being vented NiCads probably accounts for these batteries' long useful life -- surely at least 15 years. Still, with only twenty-four good ones left, they are probably toward the end of their lifespan and need to be taken care of in order to last.

Once I realized they were NiCads, I knew the charge controller was charging them wrong. NiCads like to be charged at a higher voltage than the 13 or so volts used for lead acid. I would have liked to charge them at 16 volts, but that would feed back through the house wiring, and could fry 12 volt stuff. Checking ratings, 15 volts seemed the highest voltage I could risk.

Coming back a week later, I found the batteries charged up to 13.4 volts. And they are now working great. Through several cloudy days, we had all the power we needed. And when the sun was out fully, I sometimes saw the solar panels charging the batteries at 125 watts -- fully half of the panels' rated capacity, and much better than before. The batteries start each evening at 13.4 volts, and only drop to 13.3 by morning. We started using electric lights more, and then just leaving them on all evening, and the nslu2 online all night, and the batteries remained at the magic 13.3 in the morning. This is because NiCad batteries have a near-constant voltage until they are perhaps 30% discharged. In other words, I had been charging them less than half full before.

I have figured out how to combine the two banks the batteries are in into one large bank, and once I get the cables to do that, I hope to have battery capacity to get through up to a week of solid clouds in midwinter.

battery box and solar panel

I've finally started writing my first real program in Haskell. I've been learning Haskell since 2007. That is to say, I've been reading about it, over and over.

Progress learning has been annoyingly slow because I have not been able to commit to using it in a real program. Because when I'm writing a real program I want the program now, and I can code in other languages ten to one hundred times faster than I can currently code in Haskell. Nasty viscious cycle. Probably best to avoid getting really good at specific programming languages to avoid it, in hindsight.

Writing a real program in Haskell is important because I mostly learn by doing real things. Exercises don't work well for me. Up till now the only real thing I did in Haskell was write a 250 line xmonad config file in it. Which really helped a lot, but mostly only with beating some of its syntax into my head.

I have a few hundred lines of git-annex done. I don't need this urgently so I am willing to spend a week writing it instead of dashing off a perl version tonight.

I seem to have had bad luck on the parts I tackled first, and run into some swampier parts of the Haskell platform. I hope these problems are behind me, they sure made the first 8 hours fun.

time

First thing I needed was a data structure with a time stamp. There are multiple modules to handle time, mostly seemingly incompatible, deprecated, and/or possibly broken.

Data.DateTime seems to be the current choice, but I don't entirely trust it. Check this weird thing out, it thinks that 0 seconds and 1 second after the Unix epoch are the same second.

Data.DateTime> map (toSeconds . fromSeconds) [-1,0,1,2,100,1000,100000,100000,1000000,100000000,100000000,1000000000,10000000000]
[-2,0,0,1,99,999,99999,99999,999999,99999999,99999999,1000000000,10000000000]

Rounding error?

(Now it's not all bad... it does support dates right up until the heat death of the universe, on 32 bit even. No Y2038 bug here.)

locking

Huh, I read Real_World_Haskell twice and never noticed that it seems to omit POSIX file locking. Which in my Real World is a necessity.

Probably because it's so hard in Haskell, with multiple gotchas.

First there's the lazy IO problem, which means that if you explicitly close a file, code that read from it may not have really run yet. I ended up needing this scariness from System.IO.Strict, which is sadly not packaged in Debian or part of the Haskell Platform:

hGetContentsStrict h  = hGetContents h >>= \s -> length s `seq` return s

Then there's the Haskell's interface for fcntl locking, which exposes rather more of fcntl locking than I like to think about, being spoiled by perl's 2-parameter interface to it.

waitToSetLock lockfd (ReadLock, AbsoluteSeek, 0, 0)

And then there's that lockfd, which is not a regalar file handle, but a fd number, which has to be obtained by calling handleToFd.

And then there's the crazy interface that has handleToFd close the input handle! Which is problimatic if you were going to use the file after you locked it..

Neil asks about removing debconf old values when replacing a package with a different one.

Usually, you shouldn't need to worry about this; when the old package is finally purged all its debconf values will be cleaned up. In case that doesn't happen for some reason, you do not need to include the templates in the new package in order to get them removed. Actually, due to debconf's reference counting, that shouldn't work anyway.

What you need is debconf-communicate(1). Specifically, something like this in a maintainer script of the new package, before it sources debconf's confmodule:

echo purge | debconf-communicate oldpackage

(Updated: Fixed incorrect syntax!)

Meh, databases add such complexity.. If I were redesigning debconf today, it would use flat files on disk and those files would be removed by dpkg on purge.

I brought my cat Leo to the house in the hollow a week ago. He settled in quickly; this was his 7th move. He seems both more kittenish, and more predatory out here. This was a trial run, I'll bring him back to town soon for a few weeks. Since early August, I've for one reason and another been away every other week, and I got tired of worrying about the cat home alone.

With a cat, and with electricity seemingly sorted, it's getting homelike here. As the days shorten, I'm being power wasteful with computers and radio and lights, and the batteries still never drop below 13.5 volts. Bodes well for winter. All that remains to be done is refrigeration that does not involve hauling ice, and more water. Winter will help with both.

Still remains to be seen if I will continue bouncing back and forth, or if there will be an inflection point.

I've regained a tranquility that had been eluding me for ages. It's been a lot of work, but it seems worth it.

(git-annex announcement soon!)

First there was mr, then etckeeper. Now git-annex completes my triptych of software tools that use git in ways that were never intended.

Git's original tagline was "the stupid content tracker". Git-annex inverts that, by managing files with git, without checking their contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, checksumming time, or disk space.

I've been working on git-annex for two weeks, and have been trusting my data to it for a week. Not a little data, but terabytes spread amoung multiple archival drives, file servers, laptops, and portable USB drives. In situations too complex for unison or rsync to be of much use. Applying git to this problem in an unconventional way has simplified things enormously.

At this point it should be very usable, and solid. A package will be uploaded to Debian shortly; or git clone it.

Three thousand lines of code is not a huge program, but it is enough to get a pretty good feel for a language. Now that I've completed my first real Haskell program I feel that I've gotten over several of the humps in the learning curve and am starting to get a good feel for it.

Actually, I've written closer to five thousand lines, since there were several big refactorings. One was when I stopped manually threading my program state around and added a StateT monad. I did know from the beginning I would need one, but it seemed easier and a better learning exercise to let the program start out with a vesigial tail and gills before growing up into a modern Haskell program. (I suppose it's still written in baby-Haskell, really..)

Another refactoring came when I realized I needed to use a custom data type, not String, to represent keys. That was a great experience in type-based refactoring. Being able to keep typing ':make' and landing on the next bit of code that needed fixing was great, and simply adding that one type exposed several non-obvious bugs.

I found myself writing code that is much more solid and reusable than normally comes easily. And yet it's also very malleable. Actually, pulling out better data types and abstractions can get a bit addictive.

When I realized that I had a similar three-stage control flow being used for each of git-annex's subcommands, and factored that control flow out into a function that used the 3 data types below, I felt I'd gone down that rabbit hole perhaps far enough for now.

type SubCmdStart = String -> Annex (Maybe SubCmdPerform)
type SubCmdPerform = Annex (Maybe SubCmdCleanup)
type SubCmdCleanup = Annex Bool

(That will allow for some nice parallelism later though, and removed dozens of lines of code, so was worth it.)

Since git-annex is a very Real World Haskell type program, there is a lot of impure code in it. I could probably do better at factoring out more pure code. I count 117 impure functions, and only 37 pure.

Anyhow, from my perspective of a long-time perl programmer, some other random impressions..

• ghc --make is handy, but every time it spits out a new 13 mb executable I can feel my laptop's SSD groan!
• It was surpisingly easy to get into nasty situations with recursive dependencies between the 19 haskell modules I wrote. Sometimes solving them was really messy. I lost hours to this. More time than I've lost to the problem in all other languages combined over 15 years. It's not clear to me if it was due to the overall design of my program, or if Haskell's types tend to encourage this problem. Or if there's some simple "please let me have recursive dependencies" switch to ghc that I missed..
• I'm used to being able to use man to get at mutiple books worth of detailed documentation for perl, and work easily offline or with limited bandwidth. With Haskell, I spend much more time searching online for documentation than I an comfortable with (although Hoogle is pretty neat). And the haddock-produced documentation is often pretty sketchy. The saving grace is that the source to any library function is a click away, and tends to be very readable.
• I'm used to being able to use pretty much any Unix syscall by name from perl: mkdir, chmod, rename, etc. In Haskell, there is a Windows smell to the names, like createDirectoryIfMissing and setPermissions. And there are pointless distinctions like renameFile vs renameDirectory. These long names are not memorable and I have to look them up every time. Most of POSIX is available, but it's scattered amoung many disparate libraries, and I can't find an interface for sysconf(3) at all. There is a certian temptation, that I am so far resisting, to make a library for C/perl refugees that exports the sane Unix names for everything.
• Anything involving the IO monad, or probably most monads, has a certian level of syntactic clumsiness about it. Compare:

    if ($flag{foo} && length $l = <>) {
  

  vs

    foo <- getFlag "foo"
    l <- getLine
    if (foo && not $ null l)
        then do
  

  When writing lots of impure code, that got old, and while I could use ifM, or make up some other similar thing, its syntax would also be somewhat clumsy.
• The fixity levels for a lot of stuff seems a bit off. I too often found myself writing error $ "foo: " ++ (show bar) or return $ Just $ ... (Still a lot better than Scheme thanks to $!)
• I've leveled up a couple times now, but this particular video game seems to have more levels going up and up, forever. Can't even see the top from here!

Today I forgetfully left my USB drive at home when I took off for a night at the yurt. This has happened before and generally involves staring at an scp for rather too long, probably getting low on battery power in the process, and is followed by not quite having files in the places I expect and forgetting to delete them from my small laptop SSD later. With git-annex those annoyances can go away.

So I wanted a couple of gigabytes of files that were left at home behind a slightly slow link. Rather than scp them to my laptop for several hours, I just did a quick ssh into a server, and browsed through my stuff there, running git annex get $foo & on files and directories I wanted. Then went to eat dinner and play board games while the content was scped across the slow link from home.

Now I can quickly download it all to my laptop from that fast server (named "finch"), like this:

git pull
git annex move . --from finch

That removes the data from the server too, freeing its space back up for next time. And the files are in their regular places on my laptop, automatically. And I can simply git annex drop . when I need that space back without worrying about losing any data.

(Hmm, this feels a little bit like it did to switch from doing everything by hand with dpkg to using apt-get..)