I took the train from Charlottesville to New York on Saturday. With this year's DebConf in the USA, I finally have a chance to experience the conference from a local perspective, without jetlag, or a language barrier. I'm still working on that. Part of me thinks I am in a foreign country here. :)
Actually, the neighboorhood around Columbia University is one I have ties to. Here is my Mom's old apartment, a few short blocks from the main DebConf venue.
And here is family friend Ruth Shereff's memorial garden in the Broadway median.
DebConf has been outstanding, and very exhausting. Lars and I have been spending a lot of time here putting the final touches on something to be announced soon.
Time to introduce what I've been working on for these past too many months..
Branchable is an Ikiwiki hosting site,
making it very easy to create blogs and wikis. The name "Branchable"
was chosen because with its git backend, and
ikiwiki-hosting frontend,
sites hosted there can be easily branched by their owner (or by anyone, if
the owner allows).
The ability to take your data and configuration and go home, as well as everything being free software, means Branchable follows the vision of the Franklin Street Statement on Freedom and Network Services, like identi.ca and last.fm. This is very important to me and to my cofounder, Lars Wirzenius.
Branchable has a one month free trial, so if you've been meaning to do something with Ikiwiki, I hope you'll sign up.
Also, we are offering free hosting for Free Software projects that sign up while Branchable is in beta.
 |
In a few hours I will be catching the train for home. It's been a great DebConf, and a thrilling visit to New York. I'm especially happy with how well my CUT BoF went, and that we launched Branchable.
As usual, DebConf is inspiring. I'm leaving with a scary new todo list:
- Form CUT team, design document, begin work on first cut release.
- 3.0 (git) source should default to a depth-1 shallow clone that includes only tags of released versions of the package.
- Add a sd foreign replica: git. So git commit messages can manipulate bug info.
- Deploy monkeysphere for all my ssh servers, and file UI bugs.
- Integrate monkeysphere into Branchable.
- Join monkeysphere project?
- Integrate libravatar into ikiwiki.
- Watch 20-some hours of recorded talks I missed!
 |
Following up on my Flattr FOSS post, the most interesting thing about having my projects on Flattr has been watching their relative popularities in the control panel. It looks something like this, which is done using javascript, so you might need to follow the link to my blog to see it properly.
I've been watching debhelper and moreutils keep pace with each other. This is interesting, because moreutils has been highlighted by Flattr FOSS, and debhelper has not. Still, the general level of interest in moreutils continues to suprise me -- and some more utils are lined up to go into it soon!
Popcon tells me that alien still has more users than anything except for debhelper; easily 5 to 10 times the users of moreutils or pristine-tar. So alien's low numbers in Flattr are also surprising. I think it might be explained by more developer types than user types following Flattr FOSS right now. Perhaps it will change now that Flattr does not need invites to sign up?
Long before I took the train up to New York for DebConf10, I was sure that Eben Moglen's talk, the followup to his earlier, excellent "Freedom in the Cloud", would be a highlight of the conference. Even though it was scheduled for an ugly 9 am slot the morning after that other certain highlight, the wine and cheese party. I was not disappointed.
The amazing DebConf video team has already put up the video of "How We Can Be the Silver Lining of the Cloud". Embedded below for those with html5. If you watch one video from DebConf, this should be it.
(Note that the audio stops cutting in and out after he switches mics a few minutes in.)
In this talk, Eben goes far past the earlier "Freedom in the Cloud" talk, changing the focus from scraping Facebook, to laying down a concrete, detailed vision of the "FreedomBox", and how it could be a game changer.
After the first standing ovation I can remember at a DebConf, given to the first speaker in a suit I can remember at a DebConf, many of us ended up in this huddle in the hall.
Which turned into an impromptu BoF in a nearby room, which led to this wiki page, which was followed by seemingly everyone buzzing about it for the rest of the conference, and a second BoF that probably exceeded its room's occupancy limit.
Which led to ... ??
I will spend the next several months trying to find other things we can do to help make this suddenly happen. And believe me, to the rest of the world, this will happen very suddenly. We will disrupt a lot of games we would like to have disrupted.
-- Eben Moglen
My thoughts:
- Focusing on this is a tactical retreat from promoting free software on eg, desktops and cell phones. But Eben makes a good case for doing so in order to disrupt many things that are blocking free software elsewhere.
- Clearly there is much interest in Debian about helping with the FreedomBox. Many in the project have done a lot of work on supporting the target hardware, and supporting users who already do this kind of thing. If the software stack for the FreedomBox already existed, we'd be packaging and integrating it. Since it mostly doesn't, many of us are interested in developing parts of it.
- Eben seems to be looking for a technical project lead. Until someone steps up to provide some clarity and decision making, we're all sorta fumbling around the edges, in the dark. At least with the same goal in our heads. In other words, FreedomBox needs an "upstream", it can't just be done in Debian. This doesn't mean that someone (or a small group) from Debian couldn't become that project lead.
An old house, theoretically empty, but still filled with the cruft of its history. Here an old classroom-style pencil sharpener mounted on a wall. In a kitchen cabinet, a plastic bag of dried fish of unknown provenance. Old books. Somewhere, a moldering, out of tune piano. How does this make you feel? Maybe depressed, or urgently needing to wipe it all clean and repaint?
Myself, as long as the previous inhabitants are people whose heads I don't mind getting into, I like this environment. I will carve out my own part of it, clean and reorganise that, but generally leave the rest as-is.
It's weird that, if we were talking not about houses, but software, I'd probably bulldoze it down, compost it, grow trees from it, and start over.
This house has the added complication of a solar power system that has accreted broken parts and workarounds over the years. So far I don't understand the original design, let alone what it developed into!
I'm spending a few days here now just to try to learn how it behaves -- how long can I run my laptop and forgetfully leave lights on before the battery bank reaches unsafe levels? Am I even measuring the level of the right battery bank? (Why does this incoming solar power display still read 21, in the middle of the night?)
I haven't been in this situation since spring of 2001. And then I only had to contend with no running water, and dialup. Now I also have to deal with limited power. Interesting times.
This LWN article is perhaps a bit confused, but it does point out a truth: On modern hardware, with fast, local storage, and large, often static files, rsync is often unncessarily slow.
That's probably true to some extent when using rsync across a LAN, but it's especially true when using it to copy files locally. rsync runs as a client/server pair, and both processes MD5 checksum the files as they are being transferred. That's nice, but it's slow too.
Funny thing about rsync is that probably 50% of uses of it don't involve the core feature that it was written to provide: Updating files by transferring differences. Which, other than ensuring data validity, is the only reason it needs slow checksums. Instead, rsync is often chosen because of all the other awesome features that were glommed onto it over the past several decades.
Compared with rsync, cp is pretty laughable -- it can't even exclude files
from being copied by patterns. And unlike rsync, cp command lines are not
often developed by repeated trial and error -- cp does not recover well
from being ctrl-c'd in the middle, while rsync does. These kinds of things
make a lot of us reach for rsync first, even if the situation does
not involve incremental file changes. In most any situation, one of
rsync's 120-some options is sure to be just what you need...
So lots of scripts use rsync to synchronise directories, but the amount of speedup obtained by using the rsync algorithm is often low. (Correction: rsync turns off the delta-transfer algorythm by default for local to local transfers. It still does md5 checksums however.) Since rsync is reasonably fast, we generally don't care that it does these checksums that probably on average slow it down. But if larger files, like videos, are involved, this starts to change.. When rsync is run on a typical home NAS, with a slow (arm) CPU, the picture changes entirely -- now the checksum overhead is unbearable, while the IO overhead is minimal.
So, here's local-rsync. It takes all the same options as rsync, with the caveat that SOURCE and DEST must be the first two options, and must be local directories.
% local-rsync huge/ /media/usb/huge/ -a -v --exclude '*~' --delete --max-delete=100
It speeds up rsync in these types of situations, by querying it to find what files need to be updated, and updating them the brute force way, with cp. At the end, rsync is run, to take care of the non-brute force stuff (like deletions and file permissions).
On a fast CPU, local-rsync will probably speed up rsync of large, static files by a factor of 2. On a slow CPU, local-rsync is so fast, and rsync so slow, that I have not bothered to benchmark. :)
This is just a hack (easily replaced by a --no-checksum option in rsync, of course). But I think it illustrates some interesting things about how a program's underlying assumptions about its environment can change over time, and how free software programs can accrete value until the original differentiating reason for their existence is not the most important thing about them anymore.
When Lars and I started work on
Branchable, the
Ikiwiki-based wiki and blog hosting service that we first discussed a year ago,
and launched two weeks ago, we didn't know what software to use to build it.
Probably, we assumed, we'd need a relational database. (Or maybe Couchdb, we wondered?) To manage the sysadmin jobs of keeping a slew of Ikiwiki sites running, we investigated using Puppet or Chef. (We worried that scaling Puppet or Chef to handle an arbitrary number of ikiwiki sites could be a problem.) For the user interface frontend, we assumed we'd be writing code in some kind of web application framework.
Nearly all of that conventional wisdom turned out to be false. Instead,
all the frontend code for Branchable runs as part of Ikiwiki.
www.branchable.com is just another Branchable site, though one with some
custom plugins that can make other sites and display a user's control
panel. Other plugins handle things like the
DNS setup interface.
Those plugins are part of the first public site hosted on Branchable,
ikiwiki-hosting.branchable.com.
That wiki's git repository holds all the code, as well as the wiki for
documentation and bug tracking. Also included in ikiwiki-hosting are some
backend programs, like ikisite, which handles the high-level site
management (creation, deletion, branching, etc), and is what we used
instead of Chef. (Though we may use Chef for server deployment later.)
Rather than use a relational database, we did out own little noSQL thing, by using a special, locked-down wiki as our database. That wiki is so special-purpose that it is not even accessible via the web! But it's managed by Branchable just like any other site. Deciding to not use a database was a watershed moment, because at that point we were fully dogfooding everything for Branchable. It's worked out well so far, possibly because our database needs are minimal.
We have over a dozen other Ikiwiki sites in Branchable that we use either internally, or for other special purposes. Like this parked site that is branched every day to refresh sandbox.branchable.com.
So, no database, no Puppet, lots of dogfooding. Did we use a web app framework? Not really. The closest we came is when developing Branchable highlighted that Ikiwiki needed a more user-friendly openid login interface, so I went off and adapted one, and of course added it into Ikiwiki proper. And that uses the JQuery framework. (I might write sometime about all the other Ikiwiki improvements that Branchable spurred.)
We've done a lot of eating our own dogfood at Branchable, but I think it's all been pragmatic, not due to Not Invented Here syndrome. And the result is that Branchable is increasingly flexible in what its sites can do, and runs on a very simple architecture, with no odd special-cases for database servers, etc. It's just a bunch of Ikiwiki sites that work together.
 |
This year at the beach we're being lazy and instead of camping, have rented a beach house. It's less intense but has its perks. :)
Update: Added an ocracode for this trip:
OBX1.1 P7/3 L7 S67+++(Pawley Island beach house)d++++ U10 T0f0b0 R1t
Bn---b---m--- F+++u+++ SC-s---g3 H++f0i0 V+++s---m0 E++r++++
Swarmnation is a neat game, but I'm not sure if it's for the reasons its authors intended. It may be interesting mostly because of its bugs. Read on for spoilers and the story that developed from an apparently abstract time waster game.
When I first saw the game, I saw a grid of squares moving around. I was obviously one: A (Blue) Square. There seemed to be no pattern to the movements. So, it seemed the game represented random passers-in-the-night on the net, unable to communicate except by dashing back and forth.
Then after a few minutes, a geometric shape was highlighted in yellow from out of the mass of squares. Woah! I'd been missing something here. It turned out my display was too small, and I had not noticed that the gameboard scolled over to show a tetris-style "next shape" display. Which the other squares had been busily trying to make before time ran out.
Now all their movements made sense. Now with a shared goal, we could communicate. Some of us were trolls and blocked shapes from forming. A few of us became leaders, boldly taking that center position in the hollow-square-with-spot-in-the-middle shape. Most of us herded into place as soon as a shape began to form, and stayed there, frantically hoping our neighbors would also conform and keep property values high. We were playing the game. We were accumulating scores.
We split into two groups, both playing the game, and then some of us defected from one group to the other, which seemed to be doing better at making a particular shape, and there was no reason to go back to that first group, I felt strongly that I was part of the second, better, group.
Then, as I was getting bored and feeling the neighbors all around stifling, I noticed my square highlighted yellow for a shape that I was not currently part of. Oh, this must be a bug I thought. I hacked around, and got it to happen again. I thought maybe it was just being a bit fuzzy in accepting shapes, but no, it turned out to be more interesting.
The game didn't care if we stayed in the shape. Just being part of the shape for an instant was enough. With this realization, whole new ways to play the game opened up. Dash left and right, near a forming shape, and you'll probably, for an instant, be part of it. Hang back and have your fun, spot an instant to plug a hole, and then get out and let someone else also take part. Try to form the same shape in both groups. And so on.
So a grid of squares hooked up to keyboards on the net has let me watch the invention of politics, rebel from conformity and hack the system, in an hour? That's what games are about!