I wonder if I've missed the deadline to apply for travel funding for DebConf8. I didn't see a mail about a deadline; perhaps I missed one but I also don't see one in the list archives or on the website.
OTOH, I do see talk in the list archives in the end of April about an upcoming meeting to allocate funding.
So, confused, but as usual also ambivilant about expecting to be funded for transcontinental travel, especially as I'm too lazy lately to legitimise it by presenting at the conference.
Out hiking and wading yesterday, and again today. Then the board meeting, amending documents older than I am, then another walk to recuperate from that (though it really went quite well). The trilliums are mostly down to pink, still some white ones by the spring, quite gorgeous.
Unlike at Anna's, the overriding impression is lush and green and blooming. Maybe that valley is just a very special place. Still, this spring feels fragile, as if rain is much more important than it should be.
Oh, and nettle soup is yummy. I take back all my unkind thoughts about stinging nettles over the years.
Yay, perl 5.10 at last. I've been working on fixing various breakage related to it all day. But to make up for that, I benchmarked ikiwiki. Thanks mostly to the trie optimisation of string alternations, ikiwiki is nearly twice as fast with perl 5.10 as it was with 5.8.
When I look at how people are using the net, I see a spectrum...
At the low end, there are users who browse, and maybe post stuff to sites like flickr or youtube or wikipedia. When I hang out with these people, I'm struck by them being often quite smart, savvy, capable (and young), and wasting a lot of their time and inginuity fitting what they want to do into these narrow and (mostly) corporate-controlled and censored channels. And being limited by it in ways that they're not fully aware of. If you're in this group, please, please consider finding a way up the spectrum.
At the high end are tecchies like me, we have at least one and often multiple servers, sometimes even in different countries/continents. We can write and run our own software with full control. This excellent advogato article calls it the "sovereign" level. We're not exactly kings when you look at who controls things from DNS to the internet backbone, but we're as close as it's practical to be without running your own wires. And we're statistically vanishingly few these days.
So the middle is more and more interesting. For one thing, being king is expensive. So some of us move down a level, to virtual servers with xen or the like. Maybe we can't build our own kernel on our server anymore, but we don't have to worry about maintaining spinning disks and fans. And some people move up to this level too. One path is learning about running linux on a personal machine and then using an easy and cheap provider like slicehost. But still, users at this level are rare.
The other interesting level (and the one I've not explored much myself) is a step up from the low end, where you have some form of inexpensive shared hosting, but can at least run your own code. This level seems quite a mess, there is no standardisation, everything has to be set up by hand, unless you use prepackaged control panel type things that probably take away most of the empowerment available at this level. A lot of people reach this level, but it's still fractions of a percent.
So there seems quite a hump up from the lowest end of the empowerment spectrum to using shared hosting. How to encourage people over that hump is an interesting problem.
I've been playing with using some power tools from the top, sovereign level down in these murky shared hosting depths. Decided to see what kind of stuff I could accomplish for $5. Although it ended up costing only 2 cents for hosting so far.
Amazon and google's hosting servives look interesting, but need things to be designed explcitly for them. And those companies are too big already. NearlyFreeSpeech.net is more flexible, funky, and has a cheap pay-as-you-go pricing that's ideal for little things that will only use a few dollars of bandwidth.
So, I got ikiwiki working there (and documented how), along with a git backend, so my wiki's sources can be cloned to elsewhere for backups and development.
Some things I hope to do later include:
- Allowing ikiwiki aggregation to be kicked off via the web. There's no cron jobs on these systems so you have to use something like webcron. (Which I dreamed up last night, but am happy to see someone else already implemented.)
- Implementing Madduck's idea to trigger ikiwiki mirrors via the web. So you can set up a cluster of mirrored wikis on disparate shared hosting.
- Using Amazon s3 to store wiki pages and handle file serving.
Done some interesting stuff in ikiwiki this evening..
Maybe you want to set up a mirror of a wiki. It's easy enough to do with an ikiwiki that's backed by git since you can just clone its repository and set up the mirror. But how to know when there's an update of the origin wiki, to update your mirror? I've added a plugin that allows you to edit a page on the origin wiki, and ask it to ping your wiki. And another plugin that your wiki can use to listen for pings and update itself, pulling down the changes from version control.
Nice thing about this is that any ikiwiki wiki that publishes its revision
control, and enables the pinger plugin, can then be mirrored by anyone,
with no coordination needed with its admin. Even multilevel mirror networks
are possible to set up. (The astute may notice that loops are also
possible.. but they will will be broken after 1 cycle.)
But this doesn't only allow mirroring. If you're using distributed version control, it also allows branching of a wiki. Just mirror as usual, but then make changes to the mirror, and don't send them back to the origin. Instant branch, that will be kept up-to-date with changes made to the origin. (Unless there's a conflict, that would need to be manually resolved, obviously.)
Wouldn't it be nice if you could git clone git://wikipedia.org/ or git://wiki.debian.org/ and go off and make it into something you're really happy with? Only thing standing in the way is that neither site uses ikiwiki. For now, you'll have to settle with cloning and branching git://git.ikiwiki.info/ :-)
Technical details here.
Continuing on with my plans to make ikiwiki more appealing to users without a dedicated server, this evening I've written an ikiwiki plugin that makes it use Amazon S3.
So, it's possible to publish a blog or other static website, built using ikiwiki, without needing your own web server at all. Ikiwiki builds a website and uploads it to Amazon, which then handles the web serving for you.
If you want a traditional wiki that people can edit online, you can still serve the pages out of S3, but you will need to find a "real" web server to host the ikiwiki CGI that handles the page editing. It'll then inject modified files into S3 as necessary.
Amazon EC2 would be the obvious choice for where to run the "real" web server, but probably not the easiest one to set up. In my experiments, I've been running the ikiwiki CGI on nearlyfreespeech, and serving the rest of the wiki out of S3. Since page edits are relatively rare, I estimate this approach will cost a dollar or so a year for the CGI hosting (most of it paying for disk storage). The Amazon S3 hosting of course depends on number of hits and storage size. And presumably it will scale very well, and be very competatively priced, if you believe Amazon's marketing. :-)
I'm loving that the design decisions I made about ikiwiki at the very beginning -- that it would use static web pages, and would be backed by a real revision control system, is now letting it be deployed in these interesting ways that I did not begin to envision back then!
"your download link will arrive momentarily at the email address you submitted. to ensure the email does not get marked as spam, please instruct your email client to accept mail from nin.com."
I wonder how many spambots are busy blasting off zillions of spams with headers forged to be from nin.com as we speak? Best of all, the spams can be customised, you know that many of the people getting them will enjoy this music.
(Sometimes I wish I could take advantage of these money-making opportunities as they present themselves to me..)
- 1 am, cautiuosly feeling my way through a construction site with a dim flashlight, weird bits of metal on concrete, sticking to the center as there are no guardrails. Suprised by a spot in the middle where the stars are bright and clear.
- Crossing the other way one day, since a train parked for an hour blocking the road. It's nearly done except for the approaches, but I still nervously look for police cars. View of Holston mountain.
- Twilight yesterday, threatening to storm, and suddenly I realise there are other pedestrians on the bridge ahead of me. Crossing over from illicit to legit. Kids walking down the center line. Cerimonial tent in the park.
- Evening today and I only bring myself to walk over it with traffic whizzing by because I know this is the last crossing that will be special.
- (A few days from now, in a car. Up and over. Another boring, ugly concrete bridge.)
Really, really windy day. There was a tornado watch until 3, which I didn't think much about until I was up on a ridge overlooping Slagle Hollow at 3:30, in one of the strongest winds I've experienced, blowing small branches past me. And lost. That could have not turned out well, but I made the right guesses at the turns, and it gave me the incentive to power-hike for an hour to get down off the hill and back to Roosterfront.
Before that I found a beautiful long hollow full of mossy deadfalls and trilliums, atonishingly close to the well-traveled trails.
So, Charter will be spying on every web page their customers view, and collating and using this information in a Big Brotherly fashion. But it's for advertising, so it's a good thing, right?
Yeah, right. I signed up for fiber to the home from my local ISP today.
$charter_customers--. Oddly, besides not being a privacy nightmare, my
local ISP is cheaper too. Who'd have thought.
Anyone using http://kitenet.net/wifi wireless or on the network here: You should assume Charter is spying on you for now. (BTES will install their connection on Monday.)
There's something about certian very visually information dense console apps that makes certian bits of my neural network that have been quietly developing for years very happy. rtorrent is the best example of this ever (excluding the unix shell as a whole). It starts out a utter incomprehensible mess, and I can just feel my brain reconfiguring over several hours so it can understand it at a glance.
Chunks seen: [C/A/D 39/10/2.98] X downloaded X missing X qu
Peer list 0 7465767364 6546566456 7576655464 3676833757 5654376467
50 5565765774 4785647665 4565464565 5777557464 4865465737
Info 100 4435567665 4547636475 6756567736 5665345373 6356566756
150 5454855548 6766756846 4476585655 5566676764 5665754652
File list 200 3674466375 7474666747 8766756866 5646667647 6677676353
250 5557754755 6755756775 4476686458 6544656687 5777557555
Tracker list 300 5567666653 7675564686 6263555577 6467766443 8635577643
350 7586655656 7765373543 4454677664 6654665666 4344666556
Chunks seen 400 6565665554 7656544673 7636765776 3567556727 6466735257
450 6344246566 7364539664 5366744446 6577564574 3563656833
Transfer list 500 5765366475 6685663564 7466756575 5567636754 4425467555
This is the same part of my brain that is happy out in the woods following along faint game trails. It's a pity that we've been taught to think of this kind of UI as "hard" and thus "bad".
12.xxx.75.xx 0.3 0.0 10.2 l/ci/ci 0/0 21Azureus
67.xxx.58.xxx 0.0 0.0 0.0 l/cn/ci 0/0 100uTorrent
* 67.xxx.222.xxx 0.0 0.1 214.2 r/Ui/ci 0/2 88 617Az
24.xxx.89.xx 0.2 0.1 26.5 R/Ui/ui 0/2 30 695uT
65.xx.113.xxx 0.0 0.0 0.0 l/cn/ci 0/0 100Azureus
24.xxx.38.xxx 0.0 0.0 0.0 L/cn/ci 0/0 100Azureus
Does this make your neural networks light up and say "more plz"?
Transfer list: [Size 13]
Peer list 620 [P: 2 F: 0]AAAAaAAAAbbBBBbbbbaC...DdCc.....
59 [P: 2 F: 0]EEEEEeeFff......................
Info 141 [P: 2 F: 0]GGGGGGGGgGGg.GG.................
618 [P: 2 F: 0]HHHHHHHHHHHHHHHHHHHHHhhhh.......
File list 696 [P: 2 F: 0]IIIIIIIIIIIIIIIIIIIIIJJIIIIIijjI
26 [P: 2 F: 0]VVii............................
Tracker list 56 [P: 2 F: 0]WWWWWWWWWWWWWWWWWWWWWWWWWIWwIwIw
93 [P: 2 F: 0]NNNNNLLLLNNNLNNNNUNNNtXXPPtUUYYP
Chunks seen 313 [P: 2 F: 0]zzww............................
96 [P: 2 F: 0]NzZZUNNXXNNNRRPPZZZZPxXNNnZZXzXP
Transfer list 633 [P: 2 F: 0]ZZZz..Zz........................
694 [P: 2 F: 0]YYYYYyyy........................
109 [P: 2 F: 0]pzzppxxpznnnppuuRRuuxzzrrnn.....
I was reading about fiber to the home, but got interrupted by the fiber installation man.
BTES got here less than 24 hours after I signed up, so I got to see a fiber optic splice done in the field this morning, 1st time for me.
I think they're using Active Ethernet. I will so not miss my crappy cable modem.
A while ago I wrote down a goal on a whim, without really understanding how important it was:
Completely change everything I'm working on in Debian at least once per 5 years.
This goal has really grown on me. And I've seen a good way to accomplish it. I've decided to stop being involved in packaging third party software for Debian.
There's a bunch of reasons, but the overarching one is that I've been doing that for 13 years, and I don't feel it's a very valuable way for me to spend my time any more. I think I've done a pretty good job, but there is not much room for growth. And it's not as if I spend much time on it. In Steve's survey I estimated I spend about 1/30th of my Debian work time on packaging. I won't gain a lot of time by not doing that work, but I'll lose one mental burden, and slim down my areas of responsibility.
I'll take care of passing packages on to appropriate teams, etc, over the next month. This decision leaves me able to continue to work on teams like d-i and the CD team.
I'll also continue to maintain my own software in Debian. Using a distribution as an incubator for new software, with ready-made community, bug tracking, and of course distribution channels, is something that has been a big benefit to my software, and I think also to Debian, and I plan to continue exploring that space.
I'm excited! This is going to be interesting..
While I'm mentioning changes, I stopped using Evil Spying Charter Communicatons yesterday and switched over to BTES fiber.
I'm using my thecus as an internet gateway since it has two ethernet ports. This is the first time since college that my internet connection has been a simple ethernet jack with no "modem". Quite nice. :-)
Trying to resist the urge to put a packet analizer on the house's other ethernet uplink jack, which is used for the new power meter they installed.
I'm reading USENET again.
<shock>
strftime's time format does not support fractional seconds, but a lot of people want to do that, based on a google search. It's been implemented as an extension to several programs.
Each implementation I found uses a different conversion specification, and in each case the character chosen was already used by something else in the standard strftime. The only unused letters are: %f, %i %J %K %L %q %Q %w (and perhaps some punctuation). Of these, "%f" has a good mnemonic for fractional seconds. But I hestiate to poach from such a small space.
Looking at glibc's extensions to the strftime format suggested a different approach. I suggest using "%.S" for fractional seconds. There's a good mnemoic and this seems unlikely to be used for anything else. This could be extended to include an optional precision ("%.4S").
It would also be nice to have a standard way of showing microseconds, nanoseconds, and milliseconds too. "%uS", "%nS", and "%mS" would be nice to use; unfortunatly, "%u", "%n", and "%m" are already used in strftime (so are "%U, "%N", and "%M"). I don't have a solution for these, but they're probably generally less useful than fractional seconds anyway.
Update: I neglected to mention the possibility of using the O modifier.
Dirk Eddelbuettel points out that R uses "%OS" for fractional seconds.
As compromises go, R has chosen a pretty good one. But SuSv3 actually
defines the O modifier as causing the locale's alternative numeric symbols
to be used, with an example of Roman Numerals. So fractional seconds is
probably not quite what "%OS" is intended to do, unless you're in the locale
for very fast living people. :-)
In current versions of glibc, the alt_digits field used by the O modifier
is only present for a few locales: fa_IR, ja_JP, and or_IN. Still,
there are plenty of other
numeral systems out there,
so I still favor making up a new modifier like "%.S", rather than trading off
localisation for precision.
The way .ssh/authorized_keys is typically used is not secure. Because
using it securely is hard, and dumping in passwordless ssh keys is easy. I
spent about 5 hours today locking down my authorized_keys.
rsync
The only easy case is a rsync of a single directory. Follow these good instructions.
If you need to rsync multiple separate directories, it's easy to find
several
documents involving a
validate-rsync.sh. Do not use, it is insecure -- it allows rsync to be
run with any parameters. Including parameters that allow the remote system
to rsync in a new ~/.ssh/authorized_keys. Oops. (You can probably also
trick validate-rsync.sh into running other arbitrary commands.) To be
secure, you have to check the rsync parameters against some form of
whitelist.
git
Locking down git should be easy. Use git-shell.
Unless you also want to be able to log in to an interactive shell on the
same account. To do that, you'll need to generate a separate ssh key for
git. Both git servers I use are named git.*, so I set it up like this in
~/.ssh/config on the client:
Host git.*
IdentityFile ~/.ssh/id_rsa.git
Then on the server, I added the key to authorized_keys, prefixed with this:
command="perl -e 'exec qw(git-shell -c), $ENV{SSH_ORIGINAL_COMMAND}'"
(I also tried the simpler command="git-shell -c $SSH_ORIGINAL_COMMAND"; but
it didn't work with alioth's old version of openssh, and I didn't want
to worry about exposing SSH_ORIGINAL_COMMAND to the shell.)
Before arriving at that, I took a detour to using gitosis. It can make it impressively easy to manage locked down git over ssh, but you have to fit how it does things. Including repository layout, and a dedicated account. So I spent several hours changing my git repo layout. In the end, I decided gitosis was too complicated and limiting for what I needed, but I do recommend checking it out.
svn
Like git, this is fairly easy, since svn provides svnserve. The svnbook
documents how to use it in .authroized_keys. Basically, just use
command="svnserve -t"
A similar approach as used for git can be used here if you want to have a dedicated ssh key that causes svnserve to be run.
d-i daily builds
d-i has a d-i-unpack-helper that can be put in authorized_keys.
unison
Can probably be locked down similarly to rsync, but I haven't tried yet.
duplicity
The only way seems to not use duplicity over ssh at all, and instead use a different transport.
too hard
What if you want to allow locked down git, and a rsync, and maybe unison too, all to the same account? You'll end up with some nasty mismash of all of the above, and there are plenty more things using ssh as a transport that need other techniques to lock them down.
Until this becomes easier, a majority will just dump passwordless ssh keys
in ~/.ssh/authorized_keys, creating exploitable trust paths that don't
need to exist.