"Hey Joey, we got a telephone!" Anna's voice mail about getting a phone line. After many months of trying, this epic struggle concludes, and listen to the happiness in her voice. Yay Anna!

My father and Ed Counts made this short film, and Mark has rescued it from videotape so I can put it on the web. (If Ed Counts doesn't want this on the web, I'll have to take it down.)

Watching it closely, I've found many more scenes from my childhood than I remembered. Of course there's the trail up to the cleared spot, and that trestle down near Natural Tunnel, but I also found the house I grew up in, a view from the hill above the house, our barn, and even Ownee and Silas's house.

I remember running and jumping down the hill as Ed Counts made sketches and studies for the film. The closest I've ever come to a lucid dream is one where I can jump up, tense the right part of my mind, and not come back down. Thanks, Dad..

discussion

What better place to be on a cold snowy day than lying on my side in a crawl space under my porch, in a muddy pool of water, getting sprayed on my glasses until I could hardly see. Yes, I had a plumbing disaster today, though I was able to fix it without waiting for a plumber or wasting hours of running water. Seems that somehow, whether influenced by the cold or what I don't know, an "T" on the pipe going into my water heater started leaking out of an unused fixture. I couldn't find anything that could have been blocking that before, so how it suddenly started gushing water I don't know. It was easy enough to screw in a pipe plug and fix it.

It's amazing how closely the process of fixing plumbing parallels fixing a software bug.

• At first there's some weird problem -- my hot water heater seemed to be making a loud hissing sound.
• Various things are tried to work around the symptoms -- I turned off water to the heater, and turned off its pilot but the problem persisted.
• Experts are pulled in and incoherent problem reports are given to them -- I called my dad and a plumber, and blathered to both at various points, with speculation and bad observations.
• Other parts of the system are observed to try to narrow down the problem -- I watched my water meter spin (and stop, and spin.. weird..), listened to pipes to figure out where the noise was coming from, and eventually discovered the sound of a waterfall underneath my floor.
• When the actual cause of the problem is finally identified, the actual process of fixing it is often amazingly trivial after all this -- and so I found myself in my crawl space, looking at water streaming out of the pipe, and screwing a plug into the hole. A 1 line of code type fix..
• Fixes sometimes just fix the symptoms -- why did that thing suddenly start leaking? Did it have to do with the low temperatures? Until I know the answer to that, I can't consider this "bug" definitively fixed.

Update: It was probably an old pressure relief valve that failed somehow.

This graph by itself is enough to justify the cost of buying a UPS for me. I guess those voltage dips at night are neighbors' heating systems.

discussion

Today wiki.debian.org was briefly running on a full disk, and this MoinMoin based wiki failed pretty spectacularly:

• Every page view failed with a huge python backtrace.
• Saving a page that was being edited before it ran out of disk lost the edit and possibly corrupted the page that was being edited. Not 100% sure as I wasn't the one who experienced that.

We're used to some unix programs not dealing with being out of disk space very well. But IMHO wikis should be in the same class as package management systems, revision control systems, or databases: running out of disk should be something they handle gracefully without data loss and with graceful degredation when possible.

Course I wrote a wiki engine too, and it didn't fare much better. Possibly worse, although I tested it much harder than MoinMoin:

• Viewing pages? No problem! (They're static..)
• Its internal state files could be truncated if the disk went full at exactly the wrong time. This would need a rebuild of the wiki to fix.
• It could write truncated or empty html files if changes were made while the disk was full. Again a wiki rebuild would fix this.
• It could save truncated source files if the user edited a page, and it might be possible (though unlikely) for those to be committed to the backend revision control system. If so you'd have to revert back to an old version during recovery; no data should be lost unless the backend revision control system stinks.
• The user database could delete itself if a user tried to get an account while the disk was full (I'd like to thank perl's Storable module for this admirable behavior.)

Of course I've fixed all these problems, and it seems quite robust now on randomly failing disks. If a page edit fails, it will go as far as to bring you back to the edit form and let you try again.

I'm interested in how well other wiki engines stack up.. Also, if anyone has some hints for good ways to write regression tests involving full disks, I haven't found an approach I like yet.

discussion

Not happy with my plumbing. I got up this noon to find that I didn't have any hot water pressure. My washer's cold water spigot was also dry, and that's on the same pipe as the hot water heater, and so it seemed to be related to the earlier problems I'd had with that same pipe.

Perhaps it froze? But last night was not a record low.. And it was still not working at 1 pm. But it's working fine now at 3:30. Hrm.

I hope my plumber has an idea of how to debug this. If it were me, I'd have to wait and get some more data on whether it's reproducible after another cold night, or try to put in a preventative fix like improving the insulation on that pipe.

What will it be like to be able to google yourself for everything that's hit the net about you ever since you were born?

I don't know, but I'm having an amusing time leaving little nuggets hidden on the net for someone to find in a dozen years.

Does linux's flock() implementation atomically downgrade exclusive locks to shared, or is it one of the ones that does not do this atomically and can be raced? The source has an old comment that says it's not atomic, but then seems to do the lock removal and addition inside the big kernel lock, which would seem to make it atomic. But I'm not much of a kernel hacker..

I've been in a good mood toward the end of this week as the stuff I've gotten done has reached that critical mass that makes me feel I'm accomplishing something.

First, I got a nice new domain for ikiwiki, with an irc channel for good measure, and there's also work on a Google Summer of code application for ikiwiki afoot.

Then there was rather a lot of useful feedback and patches on ikiwiki, and a few nights ago a very tricky race condition was discovered. Batting around possible fixes for it on irc for several hours was pretty useful, it's a lot easier to think about locking, deadlock, races, etc, when there are some other minds to share the thinking. I eventually came up with a fix that seems ok, around 5 am that morning. Yes, my sleep schedule has been suffering..

On Thursday I took a digression into RL to fix some cladding that was pulled away from the eves of my house by a badly attached cable. I ended up wandering around Lowe's for over an hour -- when did that become such a fun thing to do? Oh yeah, when I bought a house.. Some kids were playing hide and seek; maybe a bit dangerous there but it has to be the funnest possible place to do it. Anyway, amoung other things I bought a collapsable ladder and had an interesting time on it 2 stories up while avoiding the nearby electric wire. Also fixed a broken latch on my wardrobe.

Concurrent with all this, in $DAY_JOB, I've been working on the armel port of Debian, including getting d-i to work with it. Since this is an unofficial port, that is not gpg signed currently, this also gave me a chance to make d-i capable of dealing with an unsigned repository. Earlier this evening, I almost completed my first armel install on an nslu2, and I expect to release images by Monday.

Finally, AJ reminded me about an item that'd been on my TODO list for far too long; some way for James to deal with gpg keyrings that doesn't treat them as binary blobs. I'd been meaning to work on that for oh, 2 years? 3? And I must have been thinking about it in the background a bit, because I fairly quickly came up with this gpg changesets idea, which is basically a dedicated revision control system, and a data format that's not too horrible (though it could be much nicer with support in gpg). Quickly followed by a whole working sample implementation that I hope James, or someone, will find useful.

Why you won't be voting for me for DPL this year. Or, my plans for Debian stuff over the next year (or three).

Most of these plans embody larger themes and directions that I feel Debian should take. I am a strong beliver that working code and getting stuff done is the best way to work toward larger goals, so I won't bother to try to explain the larger goals.

constantly usable testing

This is so important that it has its own page.

This is my first public posting of that proposal, and it should not be considered finished. I plan to finish it, and get to work on it.

website

www.debian.org is deprecated, but this is not obvious to either users or developers. That's because it's not formally deprecated, we just treat it like it is: We don't use it for anything new. Instead we have wikis, planet, times.debian.net, etc, etc, etc, etc.

Unfortunatly that has created a bit of a mess for users, who are still often stuck trying to use www.debian.org, or who can't find new stuff that is on other sites. A solution could be creating an alternate starting point for users, to help them learn about Debian, install it, and point them to docs and all the other useful sites once they're more involved and using Debian. I hope to find time to work on and promote such a thing.

I plan to continue moving stuff that needs to be maintained actively and grow off of www.debian.org to the wiki (as I've done with some of d-i stuff, for example), and encourage others to do so as well. Anything under /devel/ on www.debian.org is an especially good candidate for the wiki.

double the archive sync frequency again

Now that we've switched to updating the archive twice a day, and dealt with the (minor) issues caused by it, it should be easier to increase it to a frequency that is more likely to cause noticible turnaround time improvements and thus speed up some development. As it is, many people sleep through one or the other archive update. So I support doubling it to four times a day, or more (six seems better).

automated whole system testing

Automated whole system testing is important because Debian needs to keep working in so many obscure and annoying to test situations. Does d-i work on m68k? Does the desktop task install properly and work? Does debootstrap install a usable system on every architecture? If we rely on user reports, things can fall through the cracks entirely or not be noticed for too long.

I've been running a d-i lab for a few years, which tries to automatically test installs on 6 architectures. However, between the current kernel not supporting several of my test machines due to hardware and software bugs, and the general time sync of keeping a rack of aging and obscure test boxes working, this has not been very useful lately.

I feel that the solution is emulation: It should be possible to run Debian under emulation for most architectures. I hope to get an absurdly powerful server donated, and switch my test lab over to running in emulation on it.

developer (virtual) machines

The other benefit of emulation is that everything needed for testing stuff under emulation can be easily distributed to others. Getting developer accessible machines for certian architectures has been a problem in Debian. Emulated systems can sidestep this problem.

All that's needed is a collection of documents and/or bootable images that developers can use to set up virtual machines for as many architectures as possible.

We have steps in this direction. Hercules contains documentation for how to install in it using d-i. There is a kernel flavor and d-i boot images for mips on qemu.

Having to ssh out to a DSA adminned box to deal with a buld failure on $ARCH, and having to wait for build deps to be installed should begin to feel antequated and quaint. It is.

find the next nslu2

Debian's support for the nslu2 is an excellent development. It opens up using Debian for a whole new set of tasks, that need low-powered, semi-embedded, commodity hardware, that before were the province of special-purpose embedded distros. It also blew the popcon numbers for arm through the roof.

We need to find the next nslu2-like target and get to work on it. I'm hoping that it will be a wireless router, because that's the only thing I'm not running Debian on now. :-) But it could also be a cell phone, or PDA..