cubietruck temperature sensor

I wanted to use 1-wire temperature sensors (DS18B20) with my Cubietruck board, running Debian. The only page I could find documenting this is for the sunxi kernel, not the mainline kernel Debian uses. After a couple of hours of research I got it working, so here goes.

wiring

First you need to pick a GPIO pin to use for the 1-wire signal. The Cubietruck's GPIO pins are documented here, and I chose to use pin PG8. Other pins should work as well, although I originally tried to use PB17 and could not get it to work for an unknown reason. I also tried to use PB18 but there was a conflict with something else trying to use that same pin. To find a free pin, cat /sys/kernel/debug/pinctrl/1c20800.pinctrl/pinmux-pins and look for a line like: "pin 200 (PG8): (MUX UNCLAIMED) (GPIO UNCLAIMED)"

Now wire the DS18B20 sensor up. With its flat side facing you, the left pin goes to ground, the center pin to PG8 (or whatever GPIO pin you selected), and the right pin goes to 3.3V. Don't forget to connect the necessary 4.7K ohm resistor between the center and right pins.

You can find plenty of videos showing how to wire up the DS18B20 on youtube, which typically also involve a quick config change to a Raspberry Pi running Raspbian to get it to see the sensor. With Debian it's unfortunately quite a lot more complicated, and so this blog post got kind of long.

configuration

We need to get the kernel to enable the GPIO pin. This seems like a really easy thing, but this is where it gets really annoying and painful.

You have to edit the Cubietruck's device tree. So apt-get source linux and in there edit arch/arm/boot/dts/sun7i-a20-cubietruck.dts

In the root section ('/'), near the top, add this:

    onewire_device {
       compatible = "w1-gpio";
       gpios = <&pio 6 8 GPIO_ACTIVE_HIGH>; /* PG8 */
       pinctrl-names = "default";
       pinctrl-0 = <&my_w1_pin>;
    };

In the '&pio` section, add this:

    my_w1_pin: my_w1_pin@0 {
         allwinner,pins = "PG8";
         allwinner,function = "gpio_in";
    };

Note that if you used a different pin than PG8 you'll need to change that. The "pio 6 8" means letter G, pin 8. The 6 is because G is the 7th letter of the alphabet. I don't know where this is documented; I reverse engineered it from another example. Why this can't be hex, or octal, or symbolic names or anything sane, I don't know.

Now you'll need to compile the dts file into a dtb file. One way is to configure the kernel and use its Makefile; I avoided that by first sudo apt-get install device-tree-compiler and then running, in the top of the linux source tree:

cpp -nostdinc -I include -undef -x assembler-with-cpp \
    ./arch/arm/boot/dts/sun7i-a20-cubietruck.dts | \
    dtc -O dtb -b 0 -o sun7i-a20-cubietruck.dtb -

You'll need to install that into /etc/flash-kernel/dtbs/sun7i-a20-cubietruck.dtb on the cubietruck. Then run flash-kernel to finish installing it.

use

Now reboot, and if all went well, it'll come up and the GPIO pin will finally be turned on:

# grep PG8 /sys/kernel/debug/pinctrl/1c20800.pinctrl/pinmux-pins
pin 200 (PG8): onewire_device 1c20800.pinctrl:200 function gpio_in group PG8

And if you picked a GPIO pin that works and got the sensor wired up correctly, in /sys/bus/w1/devices/ there should be a subdirectory for the sensor, using its unique ID. Here I have two sensors connected, which 1-wire makes easy to do, just hang them all off the same wire.. er wires.

root@honeybee:/sys/bus/w1/devices> ls
28-000008290227@  28-000008645973@  w1_bus_master1@
root@honeybee:/sys/bus/w1/devices> cat *-*/w1_slave
f6 00 4b 46 7f ff 0a 10 d6 : crc=d6 YES
f6 00 4b 46 7f ff 0a 10 d6 t=15375
f6 00 4b 46 7f ff 0a 10 d6 : crc=d6 YES
f6 00 4b 46 7f ff 0a 10 d6 t=15375

So, it's 15.37 Celsius in my house. I need to go feed the fire, this took too long to get set up.

future work

Are you done at this point? I fear not entirely, because what happens when there's a kernel upgrade? If the device tree has changed in some way in the new kernel, you might need to update the modified device tree file. Or it might not boot properly or not work in some way.

With Raspbian, you don't need to modify the device tree. Instead it has support for device tree overlay files, which add some entries to the main device tree. The distribution includes a bunch of useful overlays, including one that enables GPIO pins. The Raspberry Pi's bootloader takes care of merging the main device tree and the selected overlays.

There are u-boot patches to do such merging, or the merging could be done before reboot (by flash-kernel perhaps), but apparently Debian's device tree files are built without phandle based referencing needed for that to work. (See http://elektranox.org/2017/05/0020-dt-overlays/)

There's also a kernel patch to let overlays be loaded on the fly using configfs. It seems to have been around for several years without being merged, for whatever reason, but would avoid this problem nicely if it ever did get merged.

Posted
Spectre question

Could ASLR be used to prevent the Spectre attack?

The way Spectre mitigations are shaping up, it's going to require modification of every program that deals with sensitive data, inserting serialization instructions in the right places. Or programs can be compiled with all branch prediction disabled, with more of a speed hit.

Either way, that's going to be piecemeal and error-prone. We'll be stuck with a new class of vulnerabilities for a long time. Perhaps good news for the security industry, but it's going to become as tediously bad as buffer overflows for the rest of us.

Also, so far the mitigations being developed for Spectre only cover branching, but the Spectre paper also suggests the attack can be used in the absence of branches to eg determine the contents of registers, as long as the attacker knows the address of suitable instructions to leverage.

So I wonder if a broader mitigation can be developed, if only to provide some defense in depth from Spectre.

The paper on Spectre has this to say about ASLR:

The algorithm that tracks and matches jump histories appears to use only the low bits of the virtual address (which are further reduced by simple hash function). As a result, an adversary does not need to be able to even execute code at any of the memory addresses containing the victim’s branch instruction. ASLR can also be compensated, since upper bits are ignored and bits 15..0 do not appear to be randomized with ASLR in Win32 or Win64.

I think ASLR on Linux also doesn't currently randomize those bits of the address; cat /proc/self/maps shows the lower 3 bytes always 0. But, could it be made to randomize more of the address, and so guard against Spectre?

(Then we'd only have to get all binaries built PIE, which is easily audited for and distributions are well on their way toward already. Indeed, most executables on my laptop are already built PIE, with the notable exception of /bin/bash and all haskell programs.)

I don't have an answer to this question, but am very curious about thoughts from anyone who knows about ASLR and low level CPU details. I have not been able to find any discussion of it aside from the above part of the Spectre paper.

two holiday stories

Two stories of something nice coming out of something not-so-nice for the holidays.

Story the first: The Gift That Kept on Giving

I have a Patreon account that is a significant chunk of my funding to do what I do. Patreon has really pissed off a lot of people this week, and people are leaving it in droves. My Patreon funding is down 25%.

This is an opportunity for Liberapay, which is run by a nonprofit, and avoids Patreon's excessive fees, and is free software to boot. So now I have a Liberapay account and have diversified my sustainable funding some more, although only half of the people I lost from Patreon have moved over. A few others have found other ways to donate to me, including snail mail and Paypal, and others I'll just lose. Thanks, Patreon..

Yesterday I realized I should check if anyone had decided to send me Bitcoin. Bitcoin donations are weird because noone ever tells me that they made them. Also because it's never clear if the motive is to get me invested in bitcoin or send me some money. I prefer not to be invested in risky currency speculation, preferring risks like "write free software without any clear way to get paid for it", so I always cash it out immediately.

I have not used bitcoin for a long time. I could see a long time ago that its development community was unhealthy, that there was going to be a messy fork and I didn't want the drama of that. My bitcoin wallet files were long deleted. Checking my address online, I saw that in fact two people had reacted to Patreon by sending a little bit of bitcoin to me.

I checked some old notes to find the recovery seeds, and restored "hot wallet" and "cold wallet", not sure which was my public incoming wallet. Neither was, and after some concerned scrambling, I found the gpg locked file in a hidden basement subdirectory that let me access my public incoming wallet.

What of the other two wallets? "Hot wallet" was empty. But "cold wallet" turned out to be some long forgotten wallet, and yes, this is now a story about "some long forgotten bitcoin wallet" -- you know where this is going right?

Yeah, well it didn't have a life changing amount of bitcoin in it, but it had a little almost-dust from a long-ago bitcoin donor, which at current crazy bitcoin prices, is enough that I may need to fill out a tax form now that I've sold it. And so I will be having a happy holidays, no matter how the Patreon implosion goes. But for sustainable funding going forward, I do hope that Liberapay works out.

Story the second: "a lil' positive end note does wonders"

I added this to the end of git-annex's bug report template on a whim two years ago:

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

That prompt turned out to be much more successful than I had expected, and so I want to pass the gift of the idea on to you. Consider adding something like that to your project's bug report template.

It really works: I'll see a bug report be lost and confused and discouraged, and keep reading to make sure I see whatever nice thing there might be at the end. It's not just about meaningless politeness either, it's about getting an impression about whether the user is having any success at all, and how experienced they are in general, which is important in understanding where a bug report is coming from.

I've learned more from it than I have from most other interactions with git-annex users, including the git-annex user surveys. Out of 217 bug reports that used this template, 182 answered the question. Here are some of my favorite answers.

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

  • I do! I wouldn't even have my job, if it wasn't for git-annex. ;-)

  • Yeah, it works great! If not for it I would not have noticed this data corruption until it was too late.

  • Indeed. All my stuff (around 3.5 terabytes) is stored in git-annex with at least three copies of each file on different disks and locations, spread over various hard disks, memory sticks, servers and you name it. Unused disk space is a waste, so I fill everything up to the brim with extra copies.

    In other words, Git-Annex and I are very happy together, and I'd like to marry it. And because you are the father, I hereby respectfully ask for your blessing.

  • Yes, git with git annex has revolutionised my scientific project file organisation and thats why I want to improve it.

  • <3 <3 <3

  • We use git-annex for our open-source FreeSurfer software and find very helpful indeed. Thank you. https://surfer.nmr.mgh.harvard.edu/

  • Yes I have! I've used it manage lots of video editing disks before, and am now migrating several slightly different copies of 15TB sized documentary footage from random USB3 disks and LTO tapes to a RAID server with BTRFS.

  • Oh yeah! This software is awesome. After getting used to having "dummy" shortcuts to content I don't currently have, with the simple ability to get/drop that content, I can't believe I haven't seen this anywhere before. If there is anything more impressive than this software, it's the support it has had from Joey over all this time. I'd have pulled my hair out long ago. :P

  • kinda

  • Yep, works apart from the few tests that fail.

  • Not yet, but I'm excited to make it work!

  • Roses are red
    Violets are blue
    git-annex is awesome
    and so are you
    ;-)
    But bloody hell, it's hard to get this thing to build.

  • git-annex is awesome, I lean on it heavily nearly every single day.

  • I have a couple of repositories atm, one with my music, another that backs up our family pictures for the family and uses Amazon S3 as a backup.

  • Yes! It's by far one of my favorite apps! it works very well on my laptop, on my home file server, and on my internal storage on my Android phone :)

  • Yes! I've been using git-annex quite a bit over the past year, for everything from my music collection to my personal files. Using it for a not-for-profit too. Even trying to get some Mac and Windows users to use it for our HOA's files.

  • I use git-annex for everything. I've got 10 repositories and around 2.5TB of data in those repos which in turn is synced all over the place. It's excellent.

  • Really nice tool. Thanks Joey!

  • Git-annex rocks !!!!

  • I'd love to say I have. You'll hear my shout of joy when I do.

  • Mixed bag, works when it works, but I've had quite a few "unexplained" happenings. Perservering for now, hoping me reporting bugs will see things improve...

  • Yes !!! I'm moving all my files into my annex. It is very robust; whenever something is wrong there is always some other copy somewhere that can be used.

  • Yes! git annex has been enormously helpful. Thanks so much for this tool.

  • Oh yes! I love git-annex :) I've written the hubiC special remote for git-annex, the zsh completion, contributed to the crowdfunding campaigns, and I'm a supporter on Patreon :)

  • Yes, managing 30000 files, on operating systems other than Windows though...

  • Of course ;) All the time

  • I trust Git Annex to keep hundreds of GB of data safe, and it has never failed me - despite my best efforts

  • Oh yeah, I am still discovering this powerfull git annex tool. In fact, collegues and I are forming a group during the process to exchange about different use cases, encountered problems and help each other.

  • I love the metadata functionality so much that I wrote a gui for metadata operations and discovered this bug.

  • Sure, it works marvels :-) Also what I was trying to do is perhaps not by the book...

  • Oh, yes. It rules. :) One of the most important programs I use because I have all my valuable stuff in it. My files have never been safer.

  • I'm an extremely regular user of git-annex on OS X and Linux, for several years, using it as a podcatcher and to manage most of my "large file" media. It's one of those "couldn't live without" tools. Thanks for writing it.

  • Yes, I've been using git annex for I think a year and a half now, on several repositories. It works pretty well. I have a total of around 315GB and 23K annexed keys across them (counting each annex only once, even though they're cloned on a bunch of machines).

  • I only find (what I think are) bugs because I use it and I use it because I like it. I like it because it works (except for when I find actual bugs :]).

  • I'm new to git-annex and immediately astonished by its unique strength.

  • As mentioned before, I am very, very happy with git-annex :-) Discovery of 2015 for me.

  • git-annex is great and revolutionized my file organization and backup structure (if they were even existing before)

  • That’s just a little hiccup in, up to now, various months of merry use! ;-)

  • Yes. Love it. Donated. Have been using it for years. Recommend it and get(/force) my collaborators to use it. ;-)

  • git-annex is an essential building block in my digital life style!

  • Well, git-annex is wonderful!

A lil' positive end note turned into a big one, eh? :)

new old thing

closeup of red knothole, purple-pink and
yellow wood grain

This branch came from a cedar tree overhanging my driveway.

It was fun to bust this open and shape it with hammer and chisels. My dad once recommended learning to chisel before learning any power tools for wood working.. so I suppose this is a start.

roughly split wood branch

Some tung oil and drilling later, and I'm very pleased to have a nice place to hang my cast iron.

slightly curved wood beam with cast iron
pots hung from it

Posted
Happy Thanksgiving

After thanksgiving at my sister's, I got the fun of a big family visit without the turkey day stress. We ate lemons and stuffed three people inside a tree.

Posted
custom ARM disk image generation with propellor

Following up on propelling disk images, Propellor can now build custom ARM disk images for a variety of different ARM boards. The disk image build can run on a powerful laptop or server, so it's super fast and easy compared with manually installing Debian on an ARM board.

Here's a simple propellor config for a Olimex LIME board, with ssh access and a root password:

lime :: Host
lime = host "lime.example.com" $ props
    & osDebian Unstable ARMHF
    & Machine.olimex_A10_OLinuXino_LIME
    & hasPartition (partition EXT4 `mountedAt` "/" `setSize` MegaBytes 8192)
        & hasPassword (User "root")
        & Ssh.installed
    & Ssh.permitRootLogin (RootLogin True)

To make a disk image for that board, I only have to add this property to my laptop:

& imageBuiltFor lime
    (RawDiskImage "/srv/lime.img")
    (Debootstrapped mempty)

Propellor knows what kernel to install and how to make the image bootable for a bunch of ARM boards, including the Olimex LIME, the SheevaPlug, Banana Pi, and CubieTruck.

To build the disk image targeting ARM, propellor uses qemu. So it's helpful that, after the first build, propellor incrementally updates disk images, quite quickly and efficiently.

Once the board has the image installed, you can run propellor on it to further maintain it, and if there's a hardware problem, you can quickly replace it with an updated image.

computer tower that I will be maintaining with propellor

It's fairly simple to teach propellor about other ARM boards, so it should be quite easy to keep propellor knowing about all ARM boards supported by Debian (and other distros). Here's how I taught it about the Olimex LIME:

olimex_A10_OLinuXino_LIME :: Property (HasInfo + DebianLike)
olimex_A10_OLinuXino_LIME = FlashKernel.installed "Olimex A10-OLinuXino-LIME"
    `requires` sunixi "A10-OLinuXino-Lime"
    `requires` armmp

My home server is a CubieTruck which serves as a wireless access point, solar panel data collector, and git-annex autobuilder. It's deployed from a disk image built by propellor, using this config. I've been involved with building disk image for ARM boards for a long time -- it was part of my job for five years -- and this is the first time I've been entirely happy with the process.

stupid long route

There's an old net story from the 80's, which I can't find right now, but is about two computers, 10 feet apart, having a ridiculously long network route between them, packets traveling into other states or countries and back, when they could have flowed over a short cable.

Ever since I read that, I've been collecting my own ridiculously long routes. ssh bouncing from country to country, making letters I type travel all the way around the world until they echo back on my screen. Tasting the latency that's one of the only ways we can viscerally understand just how big a tangle of wires humanity has built.

Yesterday, I surpassed all that, and I did it in a way that hearkens right back to the original story. I had two computers, 20 feet apart, I wanted one to talk to the other, and the route between the two ended up traveling not around the Earth, but almost the distance to the Moon.

I was rebuilding my home's access point, and ran into a annoying bug that prevented it from listening to wifi. I knew it was still connected over ethernet to the satellite receiver.

I connected my laptop to the satellite receiver over wifi. But, I didn't know the IP address to reach the access point. Then I remembered I had set it up so incoming ssh to the satellite receiver was directed to the access point.

So, I sshed to a computer in New Jersey. And from there I sshed to my access point. And the latency was amazing. Because, every time I pressed a key:

  • It was sent to a satellite in geosynchrous orbit, 22250 miles high.
  • Which beamed it back to a ground station in Texas, another 22250 miles.
  • Which routed it over cable to New Jersey to my server there.
  • Which bounced it back to a Texas-size dish, which zapped it back to orbit, another 22250 miles.
  • And the satellite transmitted it back in the general direction of my house, another 22250 miles.
  • So my keystroke finally reached the access point. But then it had to show me it had received it. So that whole process happened again in reverse, adding another 89000 miles travel total.
  • And finally, after 178000 and change miles of data transfer, the letter I'd typed a full second ago appeared on my screen.

Not bad for a lazy solution to a problem that could have been solved by walking across the room, eh?

Previously: roundtrip latency from a cabin with dialup in 2011

Posted
extending Scuttlebutt with Annah

This post has it all. Flotillas of sailboats, peer-to-peer wikis, games, and de-frogging. But, I need to start by talking about some tech you may not have heard of yet...

  • Scuttlebutt is way for friends to share feeds of content-addressed messages, peer-to-peer. Most Scuttlebutt clients currently look something like facebook, but there are also github clones, chess games, etc. Many private encrypted conversations going on. All entirely decentralized.
    (My scuttlebutt feed can be viewed here)

  • Annah is a purely functional, strongly typed language. Its design allows individual atoms of the language to be put in content-addressed storage, right down to data types. So the value True and a hash of the definition of what True is can both be treated the same by Annah's compiler.
    (Not to be confused with my sister, Anna, or part of the Debian Installer with the same name that I wrote long ago.)

So, how could these be combined together, and what might the result look like?

Well, I could start by posting a Scuttlebutt message that defines what True is. And another Scuttlebutt message defining False. And then, another Scuttlebutt message to define the AND function, which would link to my messages for True and False. Continue this until I've built up enough Annah code to write some almost useful programs.

Annah can't do any IO on its own (though it can model IO similarly to how Haskell does), so for programs to be actually useful, there needs to be Scuttlebutt client support. The way typing works in Annah, a program's type can be expressed as a Scuttlebutt link. So a Scuttlebutt client that wants to run Annah programs of a particular type can pick out programs that link to that type, and will know what type of data the program consumes and produces.

Here are a few ideas of what could be built, with fairly simple client-side support for different types of Annah programs...

  • Shared dashboards. Boats in a flotilla are communicating via Scuttlebutt, and want to share a map of their planned courses. Coders collaborating via Scuttlebutt want to see an overview of the state of their project.

    For this, the Scuttlebutt client needs a way to run a selected Annah program of type Dashboard, and display its output like a Scuttlebutt message, in a dashboard window. The dashboard message gets updated whenever other Scuttlebutt messages come in. The Annah program picks out the messages it's interested in, and generates the dashboard message.

    So, send a message updating your boat's position, and everyone sees it update on the map. Send a message with updated weather forecasts as they're received, and everyone can see the storm developing. Send another message updating a waypoint to avoid the storm, and steady as you go...

    The coders, meanwhile, probably tweak their dashboard's code every day. As they add git-ssb repos, they make the dashboard display an overview of their bugs. They get CI systems hooked in and feeding messages to Scuttlebutt, and make the dashboard go green or red. They make the dashboard A-B test itself to pick the right shade of red. And so on...

    The dashboard program is stored in Scuttlebutt so everyone is on the same page, and the most recent version of it posted by a team member gets used. (Just have the old version of the program notice when there's a newer version, and run that one..)

    (Also could be used in disaster response scenarios, where the data and visualization tools get built up on the fly in response to local needs, and are shared peer-to-peer in areas without internet.)

  • Smart hyperlinks. When a hyperlink in a Scuttlebutt message points to a Annah program, optionally with some Annah data, clicking on it can run the program and display the messages that the program generates.

    This is the most basic way a Scuttlebutt client could support Annah programs, and it could be used for tons of stuff. A few examples:

    • Hiding spoilers. Click on the link and it'll display a spoiler about a book/movie.
    • A link to whatever I was talking about one year ago today. That opens different messages as time goes by. Put it in your Scuttlebutt profile or something. (Requires a way for Annah to get the current date, which it normally has no way of accessing.)
    • Choose your own adventure or twine style games. Click on the link and the program starts the game, displaying links to choose between, and so on.
    • Links to custom views. For example, a link could lead to a combination of messages from several different, related channels. Or could filter messages in some way.
  • Collaborative filtering. Suppose I don't want to see frog-related memes in my Scuttlebutt client. I can write a Annah program that calculates a message's frogginess, and outputs a Filtered Message. It can leave a message unchanged, or filter it out, or perhaps minimize its display. I publish the Annah program on my feed, and tell my Scuttlebutt client to filter all messages through it before displaying them to me.

    I published the program in my Scuttlebutt feed, and so my friends can use it too. They can build other filtering functions for other stuff (such an an excess of orange in photos), and integrate my frog filter into their filter program by simply composing the two.

    If I like their filter, I can switch my client to using it. Or not. Filtering is thus subjective, like Scuttlebutt, and the subjectivity is expressed by picking the filter you want to use, or developing a better one.

  • Wiki pages. Scuttlebutt is built on immutable append-only logs; it doesn't have editable wiki pages. But they can be built on top using Annah.

    A smart link to a wiki page is a reference to the Annah program that renders it. Of course being a wiki, there will be more smart links on the wiki page going to other wiki pages, and so on.

    The wiki page includes a smart link to edit it. The editor needs basic form support in the Scuttlebutt client; when the edited wiki page is posted, the Annah program diffs it against the previous version and generates an Edit which gets posted to the user's feed. Rendering the page is just a matter of finding the Edit messages for it from people who are allowed to edit it, and combining them.

    Anyone can fork a wiki page by posting an Edit to their feed. And can then post a smart link to their fork of the page.

    And anyone can merge other forks into their wiki page (this posts a control message that makes the Annah program implementing the wiki accept those forks' Edit messages). Or grant other users permission to edit the wiki page (another control message). Or grant other users permissions to grant other users permissions.

    There are lots of different ways you might want your wiki to work. No one wiki implementation, but lots of Annah programs. Others can interact with your wiki using the program you picked, or fork it and even switch the program used. Subjectivity again.

  • User-defined board games. The Scuttlebutt client finds Scuttlebutt messages containing Annah programs of type Game, and generates a tab with a list of available games.

    The players of a particular game all experience the same game interface, because the code for it is part of their shared Scuttlebutt message pool, and the code to use gets agreed on at the start of a game.

    To play a game, the Scuttlebutt client runs the Annah program, which generates a description of the current contents of the game board.

    So, for chess, use Annah to define a ChessMove data type, and the Annah program takes the feeds of the two players, looks for messages containing a ChessMove, and builds up a description of the chess board.

    As well as the pieces on the game board, the game board description includes Annah functions that get called when the user moves a game piece. That generates a new ChessMove which gets recorded in the user's Scuttlebutt feed.

    This could support a wide variety of board games. If you don't mind the possibility that your opponent might cheat by peeking at the random seed, even games involving things like random card shuffles and dice rolls could be built. Also there can be games like Core Wars where the gamers themselves write Annah programs to run inside the game.

    Variants of games can be developed by modifying and reusing game programs. For example, timed chess is just the chess program with an added check on move time, and time clock display.

  • Decentralized chat bots. Chat bots are all the rage (or were a few months ago, tech fads move fast), but in a decentralized system like Scuttlebutt, a bot running on a server somewhere would be a ugly point of centralization. Instead, write a Annah program for the bot.

    To launch the bot, publish a message in your own personal Scuttlebutt feed that contains the bot's program, and a nonce.

    The user's Scuttlebutt client takes care of the rest. It looks for messages with bot programs, and runs the bot's program. This generates or updates a Scuttlebutt message feed for the bot.

    The bot's program signs the messages in its feed using a private key that's generated by combining the user's public key, and the bot's nonce. So, the bot has one feed per user it talks to, with deterministic content, which avoids a problem with forking a Scuttlebutt feed.

    The bot-generated messages can be stored in the Scuttlebutt database like any other messages and replicated around. The bot appears as if it were a Scuttlebutt user. But you can have conversations with it while you're offline.

    (The careful reader may have noticed that deeply private messages sent to the bot can be decrypted by anyone! This bot thing is probably a bad idea really, but maybe the bot fad is over anyway. We can only hope. It's important that there be at least one bad idea in this list..)

This kind of extensibility in a peer-to-peer system is exciting! With these new systems, we can consider lessons from the world wide web and replicate some of the good parts, while avoiding the bad. Javascript has been both good and bad for the web. The extensibility is great, and yet it's a neverending security and privacy nightmare, and it ties web pages ever more tightly to programs hidden away on servers. I believe that Annah combined with Scuttlebutt will comprehensively avoid those problems. Shall we build it?


This exploration was sponsored by Jake Vosloo on Patreon.

Posted
unifying OS installation and configuration management

Three years ago, I realized that propellor (my configuration management system that is configured using haskell) could be used as an installer for Debian (or other versions of Linux). In propellor is d-i 2.0, I guessed it would take "a month and adding a few thousand lines of code".

I've now taken that month, and written that code, and I presented the result at DebConf yesterday. I demoed propellor building a live Debian installation image, and then handed it off to a volenteer from the audience to play with its visual user interface and perform the installation. The whole demo took around 20 minutes, and ended with a standard Debian desktop installation. (Video)

The core idea is to reuse the same configuration management system for several different purposes.

  1. Building a bootable disk image that can be used as both a live system and as an OS installer.
  2. Running on that live system, to install the target system. Which can just involve copying the live system to the target disk and then letting the configuration management system make the necessary changes to get from the live system configuration to the target system configuration.
  3. To support such things as headless arm boards, building customized images tuned for the target board and use case, that can then simply be copied to the board to install.
  4. Optionally, running on the installed system later, to further customize it. Starting from the same configuration that produced the installed system in the first place.

There can be enormous code reuse here, and improvements made for one of those will often benefit all the rest as well.

Once everything is handled by configuration management, all user interface requirements become just a matter of editing the configuration. Including:

  • A user interface that runs on the live system and gets whatever input is needed to install to the target system. This is really just a config editor underneath. I built a prototype gamified interface that's as minimal as such an interface could get.
  • With a regular text editor, of course. This is the equivalent of preseeding in d-i, giving advanced users full control over the system that gets built. Unlike with preseeding, users have the full power of a configuration management system, so can specify precisely the system they want installed.
  • A separate user interface for customizing disk images, for arm boards and similar use cases. This would run on a server, or on the user's own laptop.

That's the gist of it. Configuration management reused for installation and image building, and multiple editor interfaces to make it widely usable.

I was glad, sitting in to a BoF session before my talk, that several people in Debian are already thinking along similar lines. And if Debian wanted to take this work and run with it, I'd be glad to assist as propellor's maintainer. But the idea is more important than the code and I hope my elaboration of it helps point a way if not the way.

While what I've built installs Debian, little of it is Debian-specific. It would probably be easy to port it to Arch Linux, which propellor already supports. There are Linux-specific parts, so porting to FreeBSD would be harder, but propellor knows, at the type level which OSs properties support, which will ease porting.

GuixSD and NixOS already use configuration management for installation, and were part of my inspiration. I've extended what they do in some ways (in other ways they remain far ahead).


The code is here. And here are some links to more details about what I built, and ideas encountered along the way:

home power monitoring

For years I've recorded solar panel data by hand. Filled two notebooks with columns of figures. My new charge controller, an EPsolar Tracer-BN, finally let me automate it.

morning activity; by 8 am the sun is still behind the hill but, 16 watts are being produced, and by 11:30 am, the battery bank is full

You can explore my home power data here: http://homepower.joeyh.name/
(click and drag to zoom)

The web interface loads the RRD files into a web browser using javascriptRRD. I wrote a haskell program that drives the epsolar-tracer python library to poll for data, and stores it in RRD files. Could have used collectd or something, but the interface to the charge controller is currently a bit flakey and I have to be careful about retries and polling frequencies. Also I wanted full control over how much data is stored in the RRD files.

Full source code

Posted