hacking water

From water insecurity to offgrid, solar pumped, gravity flow 1000 gallons of running water.

I enjoy hauling water by hand, which is why doing it for 8 years was not really a problem. But water insecurity is; the spring has been drying up for longer periods in the fall, and the cisterns have barely been large enough to get through.

And if I'm going to add storage, it ought to be above the house, so it can gravity flow. And I have this 100 watt array of 20 year old solar panels sitting unused after my solar upgrade. And a couple of pumps for a pressure tank system that was not working when I moved in. And I stumbled across an odd little flat spot halfway up the hillside. And there's an exposed copper pipe next to the house's retaining wall; email to Africa establishes that it goes down and through the wall and connects into the plumbing.

a solar
panel with a large impact crater; the glass has cracked into thousands of
peices but is still hanging together barely perceptable flat spot on tree
covered hillside a copper pipe sheathed in
black plastic curves out of the ground next to a wall

So I have an old system that doesn't do what I want. Let's hack the system..

(This took a year to research and put together, including learning a lot about plumbing.)

Run a cable from the old solar panels 75 feet over to the spring. Repurpose an old cooler as a pumphouse, to keep the rain off the Shurflow pump, and with the opening facing so it directs noise away from living areas. Add a Shurflow 902-200 linear current booster to control the pump.

red cooler attached to a tree with a
pump in it. water is streaming out of one of the two pipes attached to it circuit board with terminals
labeled PUMP, PV, HIGH, LOW, GND, FLOAT SWITCH 50 gallon water barrel
perched on a hillside with some hoses connected to it

Run a temporary pipe up to the logging road, and verify that the pump can just manage to push the water up there.

Sidetrack into a week spent cleaning out and re-sealing the spring's settling tank. This was yak shaving, but it was going to fail. Build a custom ladder because regular ladders are too wide to fit into it. Flashback to my tightest squeezes from caving. Yuurgh.

very narrow concrete
water tank with its concrete lid opened and a rough wooden ladder sticking
out of it interior of water tank
drained with muck covering the bottom and plaster flaking from walls interior of water
tank with walls bright and new, water level sensors and pipe to pump

Install water level sensors in the settling tank, cut a hole for pipe, connect to pumphouse.

Now how to bury 250 feet of PEX pipe a foot deep up a steep hillside covered in rock piles and trees that you don't want to cut down to make way for equipment? Research every possibility, and pick the one that involves a repurposed linemans's tool resembling a medieval axe.

hillside strewn in large rocks
with trees wherever there are not rocks just unboxed
trenching tool looks like a large black metal spatula 30 feet of very narrow trench
comes out of the woods and along the side of the house past the satellite
internet dish lines drawn
over a photo of the hillside show the pipe's curving route to the top

Dig 100 feet of 1 inch wide trench in a single afternoon by hand. Zeno in on the rest of the 300 foot run. Gain ability to bury underground cables without raising a sweat as an accidental superpower. Arms ache for a full month afterwards.

Connect it all up with a temporary water barrel, and it works! Gravity flow yields 30 PSI!

Pressure-test the copper pipe going into the house to make sure it's not leaking behind the retaining wall. Fix all the old leaky plumbing and fixtures in the house.

water pressure guage connected to
PEX pipe coming out of trench and connecting to copper pipe that goes into
house modern restaurant-style sprung arched
faucet with water flowing into the kitchen sink

Clear a 6 foot wide path through the woods up the hill and roll up two 550 gallon Norwesco water tanks. Haul 650 pounds of sand up the hill, by hand, one 5 gallon bucket at a time. Level and prepare two 6 foot diameter pads.

Joey standing in front of a black 4x4
pickup truck with a large white 550 gallon water tank on its side in the
bed and arching high above Joey from the back as he rolls water
tank up a forested hill six foot circle marked off with rope
and filled with sand; a water tank is in the background and tools are
strewn around the cramped worksite

Build a buried manifold with valves turned by water meter key. Include a fire hose outlet just in case.

in a hole in
the ground between two water tanks is a complex assembly of blue pipes and
brass fittings, with several valves close-up of old cracked solar panel the two tanks installed, high on a hillside

Begin filling the tanks, unsure how long it will take as the pump balances available sunlight and spring flow.

the most important missing unicode extension

Unicode allows you to write any symbol in use in the world, and if one is somehow missing from Unicode, it'll get added in a future version of the specification. Right?

Here's a symbol we all know well: ￿
Aka, "box with 4 small F's in it".

But that's not the only such symbol, there was a whole class of them in common use in the early 2000's, so common that they were called "tofu" when grouped together.

Modern unicode systems cannot display those historical characters!

Clearly this needs to be fixed with an extension to the Unicode standard. I propose TOFU, a combining character that makes whatever it's combined with be displayed as a font fallback box glyph containing its Unicode code point.

80 percent

I added dh to debhelper a decade ago, and now Debian is considering making use of dh mandatory. Not being part of Debian anymore, I'm in the position of needing to point out something important about it anyway. So this post is less about pointing in a specific direction as giving a different angle to think about things.

debhelper was intentionally designed as a 100% solution for simplifying building Debian packages. Any package it's used with gets simplified and streamlined and made less a bother to maintain. The way debhelper succeeds at 100% is not by doing everything, but by being usable in little pieces, that build up to a larger, more consistent whole, but that can just as well be used sparingly.

dh was intentionally not designed to be a 100% solution, because it is not a collection of little pieces, but a framework. I first built an 80% solution, which is the canned sequences of commands it runs plus things like dh_auto_build that guess at how to build any software. Then I iterated to get closer to 100%. The main iteration was override targets in the debian/rules file, to let commands be skipped or run out of order or with options. That closed dh's gap by a further 80%.

So, dh is probably somewhere around a 96% solution now. It may have crept closer still to 100%, but it seems likely there is still a gap, because it was never intended to completely close the gap.

Starting at 100% and incrementally approaching 100% are very different design choices. The end results can look very similar, since in both cases it can appear that nearly everyone has settled on doing things in the same way. I feel though, that the underlying difference is important.

PS: It's perhaps worth re-reading the original debhelper email and see how much my original problems with debstd would also apply to dh if its use were mandatory!

git-annex and funding update

git-annex v7 was released this fall, the culmination of a long effort to add some important new features to git-annex. Rather than go into details about it here, see this LWN article comparing and contrasting git-annex with git lfs.

For three years my work on git-annex had major support from Dartmouth's DataLad project, pushing it into use in the sciences, and driving large improvements in git-annex's API, concurrency support, etc. But that relied on government funding which has been drying up. Increasingly I have been relying on croudfunding from git-annex's users.

Now I'm entering a new phase, where DataLad users may also want to support git-annex. So far, McGill's NeuroHub project has committed to supporting its development (funded by the Canada First Research Excellence Fund, for the Healthy Brains for Healthy Lives initiative), but I hope others will too. A diversity of funding sources is best.

A survey of git-annex users is now underway, the first in three years. If you've not already, please participate in it to help direct my newly funded work.

fridge 0.2

My offgrid, solar powered, zero-battery-use fridge has sucessfully made it through spring, summer, fall, and more than enough winter.

I've proven that it works. I've not gotten food poisoning, though I did lose half a gallon of milk on one super rainy week. I have piles of data, and a whole wiki documenting how I built it. I've developed 3 thousand lines of control software. It purrs along without any assistance.

Fridge0 consists of a standard chest freezer, an added thermal mass, an inverter, and computer control. It ties into the typical offfgrid system of a solar charge controller, battery bank, and photovoltaic panels.

This isn't going to solve global warming or anything, but it does seem much less expensive than traditional offgrid fridge systems, and it ties in with thinking on renewable power such as Low Tech magazine's Redefining Energy Security "To improve energy security, we need to make infrastructures less reliable."

I mostly wanted to share the wiki, in case someone wants to build something like this. And to post some data. Here's the summer and fall temperature data.

(More on temperature ranges here.)

I want to be upfront that this is not guaranteed to work in every situation. Here's that time that the milk spoiled. A tropical storm was involved. Most of the time milk stays good 2 to 3 weeks in my fridge.

Some things I might get around to doing eventually:

  • Using a supercapacitor to provide power while shutting down on loss of solar power, instead of the current few minutes of use of batteries.
  • Also running a freezer, dividing up solar power between them.
  • A self-contained build with its own solar panels and electronics, instead of the current build that uses my house's server and solar panels.
  • A full BOM or kit, just add solar panels and chest freezer to quickly build your own.

I probably won't be devoting much time to this in the upcoming year, but if anyone wants to build one I'm happy to help you.

effective bug tracking illustrated with AT&T

I'm pleased to have teamed up with AT&T to bring you this illustrated guide to effective bug tracking.

telephone pole with phone box spewing wires, and several obviously cut cables attaches

The original issue description was "noise / static on line", and as we can see, AT&T have very effectively closed the ticket: There is no longer any noise, of any kind, on the phone line.

No electrons == no noise, so this is the absolute simplest and most effective fix possible. Always start with the simplest such fix, and be sure to close the problem ticket immediately on fixing. Do not followup with the issue reporter, or contact them in any way to explain how the issue was resolved.

telephone pole with phone wire wrapped down it and extending across the ground

While in the guts of the system fixing such a bug report, you'll probably see something that could be improved by some light refactoring. It's always a good idea to do that right away, because refactoring can often just solves an issue on its own somehow. (Never use your own issue tracking system to report issues to yourself to deal with later, because that would just be bonkers.)

But don't go overboard with refactoring. As we see here, when AT&T decided to run a new line between two poles involved in my bug report, they simply ran it along the ground next to my neighbor's barn. A few festive loops and bows prevent any possible damage by tractor. Can always refactor more later.

phone cable tied to pole

The only other information included in my bug report was "house at end of loong driveway". AT&T helpfully limited the size of the field to something smaller than 1 (old-style) tweet, to prevent some long brain dump being put in there.

You don't want to hear that I've lived here for 7 years and the buried line has never been clean but's been getting a bit more noisy lately, or that I noticed signs of water ingress at two of the junction boxes, or that it got much much worse after a recent snow storm, to the point that I was answering the phone by yelling "my phone line is broken" down the line consumed with static.

cartoon of room 641A, red lines on a
screen connect 'nowden', Applebaum, Hess and 'unar' above a 'land line inactive'.
Speech bubble: My Holidaze came early this year!

Design your bug tracking system to not let the user really communicate with you. You know what's wrong better than them.

And certianly don't try to reproduce the circumstances of the bug report. No need to visit my house and check the outside line when you've already identified and clearly fixed the problem at the pole.

My second bug report is "no dial tone" with access information "on porch end of long driveway". With that, I seem to be trying to solicit some kind of contact outside the bug tracking system. That is never a good idea though, and AT&T should instruct their linemen to avoid any possible contact with the user, or any attempts to convey information outside the issue tracking system.

laminated handwritten note pinned to
phone pole with one red and one green pin. reads: Buried phone line was cut
by last lineman -- please repair. House 500 ft up driveway has no dialtone.
Santa, all I want for Xmas is a dialtone!

AT&T's issue tracking system reports "Service Restore Date: 12/25/2018 at 12:00 AM" but perhaps they'll provide more effective issue tracking tips for me to share with you. Watch this space.

censored Amazon review of Sandisk Ultra 32GB Micro SDHC Card

★ counterfeits in amazon pipeline

The 32 gb card I bought here at Amazon turned out to be fake. Within days I was getting read errors, even though the card was still mostly empty.

The logo is noticably blurry compared with a 32 gb card purchased elsewhere. Also, the color of the grey half of the card is subtly wrong, and the lettering is subtly wrong.

Amazon apparently has counterfiet stock in their pipeline, google "amazon counterfiet" for more.

You will not find this review on Sandisk Ultra 32GB Micro SDHC UHS-I Card with Adapter - 98MB/s U1 A1 - SDSQUAR-032G-GN6MA because it was rejected. As far as I can tell my review violates none of Amazon's posted guidelines. But it's specific about how to tell this card is counterfeit, and it mentions a real and ongoing issue that Amazon clearly wants to cover up.

usb drives with no phantom load

For a long time I've not had any network attached storage at home, because it's offgrid and power budget didn't allow it. But now I have 16 terabytes of network attached storage, that uses no power at all when it's not in use, and automatically spins up on demand.

I used a USB hub with per-port power control. But even with a USB drive's port powered down, there's a parasitic draw of around 3 watts per drive. Not a lot, but with 4 drives that's more power wasted than leaving a couple of ceiling lights on all the time. So I put all the equipment behind a relay too, so it can be fully powered down.

I'm using systemd for automounting the drives, and have it configured to power a drive's USB port on and off as needed using uhubctl. This was kind of tricky to work out how to do, but it works very well.

Here's the mount unit for a drive, media-joey-passport.mount:


That's on port 4 of the USB hub, the startech-hub-port-4.service unit file is this:

Description=Startech usb hub port 4
ExecStart=/usr/sbin/uhubctl -a on -p 4 ; /bin/sleep 20
ExecStop=/usr/sbin/uhubctl -a off -p 4

The combination of PartOf with Requires and After in these units makes systemd start the port 4 service before mounting the drive, and stop it after unmounting. This was the hardest part to work out.

The sleep 20 is a bit unfortunate, it seems that it can take a few seconds for the drive to power up enough for the kernel to see it, and so without that the mount can fail, leaving the drive powered on indefinitely. Seems there ought to be a way to declare an additional dependency and avoid needing that sleep? Update: See my comment below for a better way.

Finally, the automount unit for the drive, media-joey-passport.automount:

Description=Automount passport

The TimeoutIdleSec makes it unmount after around 5 minutes of not being used, and then its USB port gets powered off.

I decided to not automate the relay as part of the above, instead I typically turn it on for 5 hours or so, and use the storage whenever I want during that window. One advantage to that is cron jobs can't spin up the drives in the early morning hours.