Withrawing github-backup

I am no longer maintaining github-backup. I'll contine hosting its website and git repo for the time being, but it needs a new maintainer if it's going to survive.

I don't really think it needs to survive. If the farce of youtube-dl being removed from github, thus losing access to all its issues and pull requests, taught us anything, it's that having that happen does not make many people reconsider their dependence on github. (Not even youtube-dl it turns out, which is back on there.) Clearly people don't generally have any interest in backing that stuff up.

As far as the git repositories on Github, they are getting archived very effectively by softwareheritage.org which vaccumes up all git repositories from Github. Which points to a problem, because the same can't be said for git repositories not hosted on Github. There's a form to submit them but the submissions often get hung up needing manual review, and it doesn't seem to pull in new commits actively if at all, based on the few git repositories I've had archived there so far.

That seems like something it might be worth building some software to manage. But it's also just another case of Github's mass bending reality around it; the average Github user doesn't care about this and still gets archived; the average self-hosting git user may care about this slightly more, but most won't get archived, even if that software did get built.

how to publish git repos that cannot be republished to github

So here's an interesting thing. Certain commit hashes are rapidly heading toward being illegal on Github.

So, if you clone a git repo from somewhere else, you had better be wary of pushing it to Github. Because if it happened to contain one of those hashes, that could get you banned from Github. Which, as we know, is your resume.

Now here's another interesting thing. It's entirely possible for me to add one of those commit hashes to any of my repos, which of course, I self host. I can do it without adding any of the content which Github/Microsoft, as a RIAA member, wishes to suppress.

When you clone the my repo, here's how it looks:

# git log
commit 1fff890c0980a72d669aaffe9b13a7a077c33ecf (HEAD -> master, origin/master, origin/HEAD)
Author: Joey Hess <joeyh@joeyh.name>
Date:   Mon Nov 2 18:29:17 2020 -0400

    remove submodule

commit 8864d5c1182dccdd1cfc9ee6e5d694ae3c70e7af
Author: Joey Hess <joeyh@joeyh.name>
Date:   Mon Nov 2 18:29:00 2020 -0400

# git ls-tree HEAD^
160000 commit b5[redacted cuz DMCA+Nov 3 = too much]    back up your cat videos with this
100644 blob 45b983be36b73c0788dc9cbcb76cbb80fc7bb057    hello

I did this by adding a submodule in one commit, without committing the .gitmodules file, and them removing the submodule in a subsequent commit.

What would then happen if you cloned my git repo and pushed it to Github?

The next person to complain at me about my not having published one of my git repos to Github, and how annoying it is that they have to clone it from somewhere else in order to push their own fork of it to Github, and how no, I would not be perpertuating Github's monopolism in doing so, and anyway, Github's monopoloy is not so bad actually ...

printf "Enter the url of the illegal repo, Citizen: "
read wha
git submodule add "$wha" wha
git rm .gitmodules
git commit -m wha
git rm wha
git commit -m wha
comically bad shipping estimates and middlemen

My inverter has unfortunately died, and I wanted to replace it with the same model. Ideally before I lose the contents of the fridge. It's a 24v inverter, which is not at all as easy to find a replacement for as a 12v inverter would be.

Somehow Walmart was the only retailer that had it available with a delivery estimate: Just 2 days.

It's the second day now, with no indication they've shipped it. I noticed the "sold and shipped by Zoro", so went and found it on that website.

So, the reality is it ships direct from China via container ship. As does every product from Zoro, which all show as 2 day delivery on Walmart's website.

I don't think this is a pandemic thing. I think it's a trying to compete with Amazon and failing thing.

My other comically bad shipping estimate this pandemic was from Amazon though. There was a run this summer on Kayaks, because social distancing is great on the water. I found a high quality inflatable kayak.

Amazon said "only 2 left in stock" and promised delivery in 1 week. One week later, it had not shipped, and they updated the delivery estimate forward 1 week. A week after that, ditto.

Eventually I bought a new model from the same manufacturer, Advanced Elements. Unfortunately, that kayak exploded the second time I inflated it, due to a manufacturing defect.

So I got in touch with Advanced Elements and they offered a replacement. I asked if, instead, they maybe still had any of the older model of kayak I had tried to order. They checked their warehouse, and found "the last one" in a corner somewhere.

No shipping estimate was provided. It arrived in 3 days.

Mr Process's wild ride

When a unix process is running in a directory, and that directory gets renamed, the process is taken on a ride to a new location in the filesystem. Suddenly, any "../" paths it might be using point to new, and unexpected locations.

This can be a source of interesting behavior, and also of security holes.

Suppose root is poking around in ~user/foo/bar/ and decides to vim ../../etc/conffile

If the user notices this process is running, they can mv ~/foo/bar /tmp and when vim saves the file, it will write to /tmp/bar/../../etc/conffile AKA /etc/conffile.

(Vim does warn that the file has changed while it was being edited. Other editors may not. Or root may be feeling especially BoFH and decide to overwrite the user's changes to their file. Or the rename could perhaps be carefully timed to avoid vim's overwrite protection.)

Or, suppose root, in the same place, decides to archive ../../etc with tar, and then delete it:

tar cf etc.tar ../../etc; rm -rf ../../etc

Now the user has some time to take root's shell on a ride, before the rm starts ... and make it delete all of /etc!

Anyone know if this class of security hole has a name?

bracketing and async exceptions in haskell

I've been digging into async exceptions in haskell, and getting more and more concerned. In particular, bracket seems to be often used in ways that are not async exception safe. I've found multiple libraries with problems.

Here's an example:

withTempFile a = bracket setup cleanup a
    setup = openTempFile "/tmp" "tmpfile"
    cleanup (name, h) = do
        hClose h
        removeFile name

This looks reasonably good, it makes sure to clean up after itself even when the action throws an exception.

But, in fact that code can leave stale temp files lying around. If the thread receives an async exception when hClose is running, it will be interrupted before the file is removed.

We normally think of bracket as masking exceptions, but it doesn't prevent async exceptions in all cases. See Control.Exception on "interruptible operations", which can receive async exceptions even when other exceptions are masked.

It's a bit surprising, but hClose is such an interruptable operation, because it flushes the write buffer. The only way to know is to read the code.

It can be quite hard to determine if an operation is interruptable, since it can come down to whether it retries a STM transaction, or uses a MVar that is not always full. I've been auditing libraries and I often have to look at code several dependencies away, and even then may not be sure if a library has this problem.

  • process's withCreateProcess could fail to wait on the process, leaving a zombie. Might also leak file descriptors?

  • http-client's withResponse might fail to close a network connection. (If a MVar happened to be empty when it's called.)

    Worth noting that there are plenty of examples of using http-client to eg, race downloading two urls and cancel the slower download. Which is just the kind of use of an async exception that could cause a problem.

  • persistent's withSqlPool and withSqlConn might fail to clean up, when used with persistent-postgresql. (If another thread is using the connection and so a MVar over in postgresql-simple is empty.)

  • concurrent-output has some locking code that is not async exception safe. (My library, so I've fixed part of it, and hope to fix the rest.)

So far, around half of the libraries I've looked at, that use bracket or onException or the like probably have this problem.

What can libraries do?

  • Document whether these things are async exception safe. Or perhaps there should be an expectation that "withFoo" always is, but if so the Haskell comminity has some work ahead of it.

  • Use finally. Good mostly in simple situations; more complicated things would be hard to write this way.

    hClose h `finally` removeFile name

  • Use uninterruptibleMask, but it's a big hammer and is often not the right tool for the job. If the operation takes a while to run, the program will not respond to ctrl-c during that time.

  • May be better to run the actions in worker threads, to insulate them from receiving any async exceptions.

    bracketInsulated :: IO a -> (a -> IO b) -> (a -> IO c) -> IO c
    bracketInsulated a b = bracket
      (uninterruptibleMask $ \u -> async (u a) >>= u . wait)
      (\v -> uninterruptibleMask $ \u -> async (u (b v)) >>= u . wait)
    (Note use of uninterruptibleMask here in case async itself does an interruptable operation. My first version got that wrong.. This is hard!)

My impression of the state of things now is that you should be very cautious using race or cancel or withAsync or the like, unless the thread is small and easy to audit for these problems. Kind of a shame, since I had wanted to be able to cancel a thread that is big and sprawling and uses all the libraries mentioned above.

This work was sponsored by Jake Vosloo and Graham Spencer on Patreon.


Lemon is one of my things. I homegrow meyer lemons and mostly eat them whole. My mom makes me lemon meringue pie on my birthday. I thought I knew how much work that must be.

Gorgeous whole lemon-meringue pie

Well, that was harder than anticipated, and so worth it. Glad my mom was there on jitsi to give moral support while I hand whisked the egg whites and cursed.

I also got a homemade mask whose quarantimer expired just in time.

But all I really want want for my birthday, this April 11th 2020, is for the coronavirus to have peaked today. I mean, having a pandemic peak on your birthday is sour, but it's better than the alternative.

Please give me that gift. Stay home. Even when some are saying it's over, watch the graphs. Don't go visit even just one person, even on their birthday. I think you can do it.


Joey in a rainbow tie-die mask,
holding a thumb up.

Lemon-meringue pie slice

solar powered waterfall controlled by a GPIO port

This waterfall is beside my yard. When it's running, I know my water tanks are full and the spring is not dry.

Also it's computer controlled, for times when I don't want to hear it. I'll also use the computer control later on to avoid running the pump excessively and wearing it out, and for some safety features like not running when the water is frozen.

This is a whole hillside of pipes, water tanks, pumps, solar panels, all controlled by a GPIO port. Easy enough; the pump controller has a float switch input and the GPIO drives a 4n35 optoisolator to open or close that circuit. Hard part will be burying all the cable to the pump. And then all the landscaping around the waterfall.

There's a bit of lag to turning it on and off. It can take over an hour for it to start flowing, and around half an hour to stop. The water level has to get high enough in the water tanks to overcome some airlocks and complicated hydrodynamic flow stuff. Then when it stops, all that excess water has to drain back down.

Anyway, enjoy my soothing afternoon project and/or massive rube goldberg machine, I certainly am.

DIN distractions

My offgrid house has an industrial automation panel.

A row of electrical devices, mounted on a
metal rail. Many wires neatly extend from it above and below,
disappearing into wire gutters.

I started building this in February, before covid-19 was impacting us here, when lots of mail orders were no big problem, and getting an unusual 3D-printed DIN rail bracket for a SSD was just a couple clicks.

I finished a month later, deep into social isolation and quarentine, scrounging around the house for scrap wire, scavenging screws from unused stuff and cutting them to size, and hoping I would not end up in a "need just one more part that I can't get" situation.

It got rather elaborate, and working on it was often a welcome distraction from the news when I couldn't concentrate on my usual work. I'm posting this now because people sometimes tell me they like hearing about my offfgrid stuff, and perhaps you could use a distraction too.

The panel has my house's computer on it, as well as both AC and DC power distribution, breakers, and switching. Since the house is offgrid, the panel is designed to let every non-essential power drain be turned off, from my offgrid fridge to the 20 terabytes of offline storage to the inverter and satellite dish, the spring pump for my gravity flow water system, and even the power outlet by the kitchen sink.

Saving power is part of why I'm using old-school relays and stuff and not IOT devices, the other reason is of course: IOT devices are horrible dystopian e-waste. I'm taking the utopian Star Trek approach, where I can command "full power to the vacuum cleaner!"

Two circuit boards, connected by numerous
ribbon cables, and clearly hand-soldered. The smaller board is suspended
above the larger. An electrical schematic, of moderate complexity.

At the core of the panel, next to the cubietruck arm board, is a custom IO daughterboard. Designed and built by hand to fit into a DIN mount case, it uses every GPIO pin on the cubietruck's main GPIO header. Making this board took 40+ hours, and was about half the project. It got pretty tight in there.

This was my first foray into DIN rail mount, and it really is industrial lego -- a whole universe of parts that all fit together and are immensely flexible. Often priced more than seems reasonable for a little bit of plastic and metal, until you look at the spec sheets and the ratings. (Total cost for my panel was $400.) It's odd that it's not more used outside its niche -- I came of age in the Bay Area, surrounded by rack mount equipment, but no DIN mount equipment. Hacking the hardware in a rack is unusual, but DIN invites hacking.

Admittedly, this is a second system kind of project, replacing some unsightly shelves full of gear and wires everywhere with something kind of overdone. But should be worth it in the long run as new gear gets clipped into place and it evolves for changing needs.

Also, wire gutters, where have you been all my life?

A cramped utility room with an entire wall
covered with electronic gear, including the DIN rail, which is surrounded by
wire gutters Detail of a wire gutter with the cover removed.
Numerous large and small wires run along it and exit here and there.

Finally, if you'd like to know what everything on the DIN rail is, from left to right: Ground block, 24v DC disconnect, fridge GFI, spare GFI, USB hub switch, computer switch, +24v block, -24v block, IO daughterboard, 1tb SSD, arm board, modem, 3 USB hubs, 5 relays, AC hot block, AC neutral block, DC-DC power converters, humidity sensor.

Full width of DIN rail.

quarantimer: a coronovirus quarantine timer for your things

I am trying to avoid bringing coronovirus into my house on anything, and I also don't want to sterilize a lot of stuff. (Tedious and easy to make a mistake.) Currently it seems that the best approach is to leave stuff to sit undisturbed someplace safe for long enough for the virus to degrade away.

Following that policy, I've quickly ended up with a porch full of stuff in different stages of quarantine, and I am quickly losing track of how long things have been in quarantine. If you have the same problem, here is a solution:


Open it on your mobile device, and you can take photos of each thing, select the kind of surfaces it has, and it will track the quarantine time for you. You can share the link to other devices or other people to collaborate.

I anticipate the javascript and css will improve, but it's good enough for now. I will provide this website until the crisis is over. Of course, it's free software and you can also host your own.

If this seems useful, please tell your friends and family about it.

Be well!

This is made possible by my supporters on Patreon, particularly Jake Vosloo.

arduino-copilot combinators

My framework for programming Arduinos in Haskell has two major improvements this week. It's feeling like I'm laying the keystone on this project. It's all about the combinators now.

Sketch combinators

Consider this arduino-copilot program, that does something unless a pause button is pushed:

paused <- input pin3
pin4 =: foo @: not paused
v <- input a1
pin5 =: bar v @: sometimes && not paused

The pause button has to be checked everywhere, and there's a risk of forgetting to check it, resulting in unexpected behavior. It would be nice to be able to factor that out somehow. Also, notice that it inputs from a1 all the time, but won't use that input when pause is pushed. It would be nice to be able to avoid that unnecessary work.

The new whenB combinator solves all of that:

paused <- input pin3
whenB (not paused) $ do
    pin4 =: foo
    v <- input a1
    pin5 =: bar v @: sometimes

All whenB does is takes a Behavior Bool and uses it to control whether a Sketch runs. It was not easy to implement, given the constraints of Copilot DSL, but it's working. And once I had whenB, I was able to leverage RebindableSyntax to allow if then else expressions to choose between Sketches, as well as between Streams.

Now it's easy to start by writing a Sketch that describes a simple behavior, like turnRight or goForward, and glue those together in a straightforward way to make a more complex Sketch, like a line-following robot:

ll <- leftLineSensed
rl <- rightLineSensed
if ll && rl
    then stop
    else if ll
        then turnLeft
        else if rl
            then turnRight
            else goForward

(Full line following robot example here)

TypedBehavior combinators

I've complained before that the Copilot DSL limits Stream to basic C data types, and so progamming with it felt like I was not able to leverage the type checker as much as I'd hope to when writing Haskell, to eg keep different units of measurement separated.

Well, I found a way around that problem. All it needed was phantom types, and some combinators to lift Copilot DSL expressions.

For example, a Sketch that controls a hot water heater certainly wants to indicate clearly that temperatures are in C not F, and PSI is another important unit. So define some empty types for those units:

data PSI
data Celsius

Using those as the phantom type parameters for TypedBehavior, some important values can be defined:

maxSafePSI :: TypedBehavior PSI Float
maxSafePSI = TypedBehavior (constant 45)

maxWaterTemp :: TypedBehavior Celsius Float
maxWaterTemp = TypedBehavior (constant 35)

And functions like this to convert raw ADC readings into our units:

adcToCelsius :: Behavior Float -> TypedBehavior Celsius Float
adcToCelsius v = TypedBehavior $ v * (constant 200 / constant 1024)

And then we can make functions that take these TypedBehaviors and run Copilot DSL expressions on the Stream contained within them, producing Behaviors suitable for being connected up to pins:

isSafePSI :: TypedBehavior PSI Float -> Behavior Bool
isSafePSI p = liftB2 (<) p maxSafePSI

isSafeTemp :: TypedBehavior Celsius Float -> Behavior Bool
isSafeTemp t = liftB2 (<) t maxSafePSI

(Full water heater example here)

BTW, did you notice the mistake on the last line of code above? No worries; the type checker will, so it will blow up at compile time, and not at runtime.

    • Couldn't match type ‘PSI’ with ‘Celsius’
      Expected type: TypedBehavior Celsius Float
        Actual type: TypedBehavior PSI Float

The liftB2 combinator was all I needed to add to support that. There's also a liftB, and there could be liftB3 etc. (Could it be generalized to a single lift function that supports multiple arities? I don't know yet.) It would be good to have more types than just phantom types; I particularly miss Maybe; but this does go a long way.

So you can have a good amount of type safety while using Copilot to program your Arduino, and you can mix both FRP style and imperative style as you like. Enjoy!

This work was sponsored by Trenton Cronholm and Jake Vosloo on Patreon.