2020 hindsight

(Someone stumbled upon my 2010 decade retrospective post and suggested I write a followup...)

This has been a big decade for me.

Ten years ago, I'd been in an increasingly stale job for several years too long. I was tired of living in the city, and had a yurt as a weekend relief valve. I had the feeling a big change was coming.

Four months on and I quit my job, despite the ongoing financial crisis making prospects poor for other employment, especially work on free software.

I tried to start a business, Branchable, with liw, based on my earlier ikiwiki project, but it never really took off. However, I'm proud it's still serving the users it did find, 10 years later.

Then, through luck and connections, I found a patch of land in a blank spot in the map with the most absurd rent ever ($5/acre/month). It had a house on it, no running water, barely solar power, a phone line, no cell service or internet, total privacy.

This proved very inspiring. Once again I was hauling water, chopping wood, poking at web pages on the other end of a dialup modem. Just like it was 2000 again. Now I was also hacking by lantern-light until the ancient batteries got so depleted I could hear the voltage regulator crackle with every surge of CPU activity.

I had wanted to learn Haskell, but could never concentrate on it enough. I learned me some Haskell and wrote git-annex, my first real world Haskell program, to help me deal with shuttling data back and forth from civilization on sneakernet.

After two idyllic years of depleting savings, I did a Kickstarter for git-annex and raised not much, but I was now living on very little, so that was a nice windfall. I went full crowdfunding for a couple of years. After a while, I started getting contracting work, supplementing the croudfunding, as git-annex found use in science and education. Both have continued ever since, amazingly.

I was free to do whatever I wanted to. A lot of that was git-annex, with some Debian, and some smaller projects, too many to list here.

Then, mid-decade, I left the Debian project. I'm still sad, still miss everybody, but I also think, had I not been so free, I would not have been able to leave it. It had driven most of my career before this point. I was lucky to be able to leave Debian. 💧

Adding to the stress of that, my patch of countryside was being sold out from under me. I considered moving to some city, but the income that's freeing here would be barely getting by there. Instead, I bought the place, using git-annex income, plus a crucial loan from a wonderful friend.

That changed how I dealt with being offgrid. Before it was an interesting constraint, something to adapt to, an added texture to life. Now it's all of those and also a source of inspiration and learning. How to install solar panels on a roof. How to wire things to code. Circuit design. Plumbing. Ditch digging. With my offgrid fridge project, things are feeling interdisciplinary in ways my work has not been before.

From here at its end, this decade feels both inevitable and highly unlikely. Now I feel.. comfortable. Settled. Surely older. More unsure of myself than ever really, nearly everything is more complicated than I used to think it was. Maybe a little stuck? But not really.

I'm planting fruit trees, something says I will be here to enjoy them. But times are getting beyond interesting. Anything could be around the corner.

announcing the filepath-bytestring haskell library

filepath-bytestring is a drop-in replacement for the standard haskell filepath library, that operates on RawFilePath rather than FilePath.

The benefit, of course, is speed. "foo" </> "bar" is around 25% faster with the new library. dropTrailingPathSeparator is 120% faster. But the real speed benefits probably come when a program is able to input filepaths as ByteStrings, manipulate them, and operate on the files, all without using String.

It's extensively tested, not only does it run all the same doctests that the filepath library does, but each function is quickchecked to behave the same as the equivilant function from filepath.

While I implemented almost everything, I did leave off some functions that operate on PATH, which seem unlikely to be useful, and the complicated normalise and stuff that uses it.

This work was sponsored by Jake Vosloo on Patron.

counterpoint on Yahoo Groups archiving

Yahoo Groups being shut down and the data deleted is a big enough story that it's being talked about on the radio. The typical presentation is that they're deleting these mailing list archives, and blocking attempts to save them and so huge amount of things will be lost from the historical record.

That's a common story these days, but it's not entirely accurate in this case. These are mailing lists, so they're not necessarily only archived by Yahoo. Anyone who subscribed to a mailing list may have archived it. I've been on a couple of those mailing lists, and the emails I kept from them are already archived rather well (10+ copies). I probably didn't keep every email, and I probably won't be exhuming those emails to add them to some large archive.org collection of Yahoo Groups. But multiply all the people who subscribed to these lists by all the traffic to them, by the probability that people keep copies of mailing list mails, and there's a huge, well-distributed archive of Yahoo Groups out there.

That ensures some of it will survive in the historical record, probably enough to satisfy a historian.

Probably even after Gmail and the other cloud mail services shut down and delete all their archives.

Previously: I am ArchiveTeam (but not speaking for them above)

left handed scissors

They return my hand's grasp, smotheringly close. Was this how it was meant to feel, in a classroom cutting multi-colored construction paper? Not a pain to be gotten through, but comfort, closeness, togetherness. Their design now feels aggressively overdone, broad curve just so around the thumb, as if they might tighten and snap it off. Only too large index finger's knuckle, chafing, provides some relief, some reminder that I shouldn't run.

(Thanks, liw.)

Posted
how I maybe didn't burn out

Last week I found myself in the uncomfortable position of many users strongly disagreeing with a decision I had made about git-annex. It felt much like a pile-on of them vs me, strong language was being deployed, and it was starting to get mentioned in multiple places on the website, in ways I felt would lead to fear, uncertainty, and doubt when other users saw it.

It did not help that I had dental implant surgery recently, and was still more in recovery than I knew when I finally tackled looking at this long thread. So it hit hard.

I've been involved in software projects that have a sometimes adversarial relationship with their users. At times, Debian has been one. I don't know if it is today, but I remember being on #debian and #debian-devel, or debian-user@lists and debian-devel@lists, and seeing two almost entirely diverged communities who were interacting only begrudgingly and with friction.

I don't want that in any of my projects now. My perspective on the history of git-annex is that most of the best developments have come after I made a not great decision or a mistake and got user feedback, and we talked it over and found a way to improve things, leading to a better result than would have been possible without the original mistake, much how a broken bone heals stronger. So this felt wrong, wrong, wrong.

Part of the problem with this discussion was that, though I'd tried to explain the constraints that led to the design decision -- which I'd made well over three years ago -- not everyone was able to follow that or engage with it constructively. Largely, I think because git-annex has a lot more users now, with a wider set of viewpoints. (Which is generally why Debian has to split off user discussions of course.) The users are more fearful of change than earlier adopters tended to be, and have more to lose if git-annex stops meeting their use case. They're invested in it, and so defensive of how they want it to work.

It also doesn't help that, surgery aside, I lack time to keep up with every discussion about git-annex now, if I'm going to also develop it. Just looking at the website tends to eat an entire day with maybe a couple bug fixes and some support answers being the only productive result. So, I have stepped back from following the git-annex website at all, for now. (I'll eventually start looking at parts of it again I'm sure.)

I did find enough value in the thread that I was able to develop a fix that should meet everyone's needs, as I now understand them (released in version 7.20191024). I actually come away with entirely new use cases; I did not realize that some users would perhaps use git-annex for a single large file in a repository otherwise kept entirely in git. Or quite how many users mix storing files in git and git-annex, which I have always seen as somewhat of an exception aside from the odd dotfile.

So the open questions are: How do I keep up with discussion and support traffic now; can I find someone to provide lower-level support and filtering or something? (Good news is, some funding could probably be arranged.) How do I prevent the git-annex community fracturing along users/developer lines as it grows, given that I don't want to work within such a fractured community?

I've been working on git-annex for 9 years this week. Have I avoided burning out? Probably, but maybe too early to tell. I think that being able to ask these questions is a good thing. I'd appreciate hearing from anyone who has grappled with these issues in their own software communities.

Project 62 Valencia Floor Lamp review

From Target, this brass finish floor lamp evokes 60's modernism, updated for the mid-Anthropocene with a touch plate switch.

The integrated microcontroller consumes a mere 2.2 watts while the lamp is turned off, in order to allow you to turn the lamp on with a stylish flick. With a 5 watt LED bulb (sold separately), the lamp will total a mere 7.2 watts while on, making it extremely energy efficient. While off, the lamp consumes a mere 19 kilowatt-hours per year.

clamp multimeter reading 0.02 amps AC, connected to a small circuit board with a yellow capacitor, a coil, and a heat sinked IC visible lamp from rear; a small round rocker switch has been added to the top of its half-globe shade

Though the lamp shade at first appears perhaps flimsy, while you are drilling a hole in it to add a physical switch, you will discover metal, though not brass all the way through. Indeed, this lamp should last for generations, should the planet continue to support human life for that long.

As an additional bonus, the small plastic project box that comes free in this lamp will delight any electrical enthusiast. As will the approximately 1 hour conversion process to delete the touch switch phantom load. The 2 cubic foot of syrofoam packaging is less delightful.

Two allen screws attach the pole to the base; one was missing in my lamp. Also, while the base is very heavily weighted, the lamp still rocks a bit when using the aftermarket switch. So I am forced to give it a mere 4 out of 5 stars.

front view of lit lamp beside a bookcase

Posted
turing complete version numbers

A quick standard for when you want to embed an arbitrary program in the version number of your program.

2   increment the data pointer (to point to the next cell to the right).
3   decrement the data pointer (to point to the next cell to the left).
+   increment (increase by one) the byte at the data pointer.
-   decrement (decrease by one) the byte at the data pointer.
.   output the byte at the data pointer.
4   accept one byte of input, storing its value in the byte at the data pointer.
6   if the byte at the data pointer is zero, then instead of moving the instruction pointer forward to the next command, jump it forward to the command after the matching 9 command.
9   if the byte at the data pointer is nonzero, then instead of moving the instruction pointer forward to the next command, jump it back to the command after the matching 6 command. 

This is simply Brainfuck with operators that are legal in (Debian) version numbers kept as-is, and some numbers replacing the rest.

Note that all other operators are ignored as usual. In particular, 1 and 8 are ignored, which make it easy to build version number programs that compare properly with past versions. And in some cases, adding 1 or 8 will be needed to make a particular program be a properly formatted version number.

For example, an infinite loop version number is:

1+69

A nice short hello world is:

1+6-6336+6-8-1-29-6333999222-92-1.1-1-1-8.2.8.2.3333-1.3+1.22222.2.33.3-1.1

Licensing: Yes, there should also be a way to embed a license in a version ... Oh, I mean to say, the Wikipedia excerpt above is CC-BY-SA, and the hello world is based on https://esolangs.org/wiki/Hello_world_program_in_esoteric_languages

Previously: a brainfuck monad

how to detect chef

If you want your program to detect when it's being run by chef, here's one way to do that.

sleep 1 while $ENV{PATH} =~ m#chef[^:]+/bin#;

This works because Chef's shell_out adds Gem.bindir to PATH, which is something like /opt/chefdk/embedded/bin.

You may want to delete the "sleep", which will make it run faster.

Would I or anyone ever really do this? Chef Inc's management seems determined to test the question, don't they.

releasing two haskell libraries in one day: libmodbus and git-lfs

The first library is a libmodbus binding in haskell.

There are a couple of other haskell modbus libraries, but none that support serial communication out of the box. I've been using a python library to talk to my solar charge controller, but it is not great at dealing with the slightly flakey interface. The libmodbus C library has features that make it more robust, and it also supports fast batched reads.

So a haskell interface to it seemed worth starting while I was doing laundry, and then for some reason it seemed worth writing a whole bunch more FFIs that I may never use, so it covers libmodbus fairly extensively. 660 lines of code all told.

Writing a good binding to a C library has art to it. I've seen ones that are so close you feel you're writing C and not haskell. On the other hand, some are so far removed from the underlying library that its documentation does not carry over at all.

I tried to strike a balance. Same function names so the extensive libmodbus documentation is easy to refer to while using it, but plenty of haskell data types so you won't mix up the parity with the stop bits.

And while it uses a mutable vector under the hood as the buffer for the FFI interface, so it can be just as fast as the C library, I also made functions for reading stuff like registers and coils be polymorphic so easier data types can be used at the expense of a bit of extra allocation.

The big win in this haskell binding is that you can leverage all the nice haskell libraries for dealing with binary data to parse the modbus data, rather than the ad-hoc integer and float conversion stuff from the C library.

For example, the Epever solar charge controller has its own slightly nonstandard way to represent 16 bit and 32 bit floats. Using the binary library to parse its registers in applicative style came out quite nice:

data Epever = Epever
    { pv_array_voltage :: Float
    , pv_array_current :: Float
    , pv_array_power :: Float
    , battery_voltage :: Float
    } deriving (Show)

getEpever :: Get Epever
getEpever = Epever
    <$> epeverfloat  -- register 0x3100
    <*> epeverfloat  -- register 0x3101
    <*> epeverfloat2 -- register 0x3102 (low) and 0x3103 (high)
    <*> epeverfloat  -- register 0x3104
 where
    epeverfloat = decimals 2 <$> getWord16host
    epeverfloat2 = do
        l <- getWord16host
        h <- getWord16host
        return (decimals 2 (l + h*2^16))
    decimals n v = fromIntegral v / (10^n)

The second library is a git-lfs implementation in pure Haskell.

Emphasis on the pure -- there is not a scrap of IO code in this library, just 400+ lines of data types, parsing, and serialization.

I wrote it a couple weeks ago so git-annex can store files in a git-lfs remote. I've also used it as a git-lfs server, mostly while exploring interesting edge cases of git-lfs.


This work was sponsored by Jake Vosloo on Patreon.

custom type checker errors for propellor

Since propellor is configured by writing Haskell, type errors are an important part of its interface. As more type level machinery has been added to propellor, it's become more common for type errors to refer to hard to understand constraints. And sometimes simple mistakes in a propellor config result in the type checker getting confused and spewing an error that is thousands of lines of gobbledygook.

Yesterday's release of the new type-errors library got me excited to improve propellor's type errors.

Most of the early wins came from using ghc's TypeError class, not the new library. I wanted custom type errors that were able to talk about problems with Property targets, like these:

    • ensureProperty inner Property is missing support for: 
    FreeBSD

    • This use of tightenTargets would widen, not narrow, adding: 
        ArchLinux + FreeBSD

    • Cannot combine properties:
        Property FreeBSD
        Property HasInfo + Debian + Buntish + ArchLinux

So I wrote a type-level pretty-printer for propellor's MetaType lists. One interesting thing about it is that it rewrites types such as Targeting OSDebian back to the Debian type alias that the user expects to see.

To generate the first error message above, I used the pretty-printer like this:

(TypeError
    ('Text "ensureProperty inner Property is missing support for: "
        ':$$: PrettyPrintMetaTypes (Difference (Targets outer) (Targets inner))
    )
)

Often a property constructor in propellor gets a new argument added to it. A propellor config that has not been updated to include the new argument used to result in this kind of enormous and useless error message:

    • Couldn't match type ‘Propellor.Types.MetaTypes.CheckCombinable
                             (Propellor.Types.MetaTypes.Concat
                                (Propellor.Types.MetaTypes.NonTargets y0)
                                (Data.Type.Bool.If
                                   (Propellor.Types.MetaTypes.Elem
                                      ('Propellor.Types.MetaTypes.Targeting 'OSDebian)
                                      (Propellor.Types.MetaTypes.Targets y0))
                                   ('Propellor.Types.MetaTypes.Targeting 'OSDebian
                                      : Data.Type.Bool.If
                                          (Propellor.Types.MetaTypes.Elem
                                             ('Propellor.Types.MetaTypes.Targeting 'OSBuntish)
    -- many, many lines elided
    • In the first argument of ‘(&)’, namely
        ‘props & osDebian Unstable’

The type-errors library was a big help. It's able to detect when the type checker gets "stuck" reducing a type function, and is going to dump it all out to the user. And you can replace that with a custom type error, like this one:

    • Cannot combine properties:
        Property <unknown>
        Property HasInfo + Debian + Buntish + ArchLinux + FreeBSD
        (Property <unknown> is often caused by applying a Property constructor to the wrong number of arguments.)
    • In the first argument of ‘(&)’, namely
        ‘props & osDebian Unstable’

Detecting when the type checker is "stuck" also let me add some custom type errors to handle cases where type inference has failed:

    • ensureProperty outer Property type is not able to be inferred here.
      Consider adding a type annotation.
    • When checking the inferred type
        writeConfig :: forall (outer :: [Propellor.Types.MetaTypes.MetaType]) t.

    • Unable to infer desired Property type in this use of tightenTargets.
      Consider adding a type annotation.

Unfortunately, the use of TypeError caused one problem. When too many arguments are passed to a property constructor that's being combined with other properties, ghc used to give its usual error message about too many arguments, but now it gives the custom "Cannot combine properties" type error, which is not as useful.

Seems likely that's a ghc bug but I need a better test case to make progress on that front. Anyway, I decided I can live with this problem for now, to get all the other nice custom type errors.

The only other known problem with propellor's type errors is that, when there is a long list of properties being combined together, a single problem can result in a cascade of many errors. Sometimes that also causes ghc to use a lot of memory. While custom error messages don't help with this, at least the error cascade is nicer and individual messages are not as long.

Propellor 5.9.0 has all the custom type error messages discussed here. If you see a hard to understand error message when using it, get in touch and let's see if we can make it better.


This was sponsored by Jake Vosloo and Trenton Cronholm on Patreon.