This is my discussion blog. The way it works is that when any pages in this wiki have a discussion page created for them, the discussion pages show up below. Also, any comments on my blog posts also show up here.

Re: Common prefix and suffix

@tanmail sorry, you're entirely wrong.

Once the common prefix and the collision data have been fed into SHA1, both the "good" and "bad" SHA1 calculations have the same internal state. So, any common suffix after that will result in the same SHA1.

If you don't believe me (or the paper that says that same thing in a more mathy way), you can easily verify this with the published SHA1 collision files.

Comment by joey
Re: comment 5

@nomeata, I'm sure it would be harder to limit the collision to printable characters. How hard I don't know (even after reading the paper).

AIUI, they're using a SAT solver; it might be that it could be modified to remove solutions that use unprintable characters, and it might only have to work a "little" harder.

Or it might take the obvious worst case of needing to run the collision generator repeatedly until it finally finds a suitable collision.

Comment by joey
comment 5
Getting a collision that has a certain form (valid ASCII, no \, for example) is, AFAIK, much harder than a general collision. I wanted to use a MD5 collision to break Haskell’s type safety via Typeable, but failed because of that. Also, would the prefix have to have a certain length?
Comment by nomeata
Common prefix and suffix


I do not think your idea is applicable as it is, because what you are using here is two files, with a variable part (the first line of ASCII art), a common prefix (everything before that line), but also a common suffix (everything after that line). If your variable part is designed to that the concatenation of the common prefix and that part gives the same hash with the two versions, that will not be the case for the concatenation of the common prefix, that part and the common suffix. Therefore, the two versions of the complete files with not have the same hash, only their first three lines do, but nobody verifies files integrity by taking the hash of its first three lines…

It may be possible to mount such an attack, but I think it would be quite more complicated…

Comment by
comment 3

As far as I know, you do not need a 192 byte prefix, it should be able to be any length you choose. "192" does not appear in the paper. The 192 bytes that appear before the good/bad chunk in the SHA1 colliding pdfs consists of the regular header of a pdf document. A prefix consisting of some C code would work just as well.

To make files that result in colliding git blobs, one only has to add at the front of the above prefix eg "blob 5000\0". The SHA1 collision generation attack then proceeds as normal. Once you have its results, you just strip that blob prefix from them, pad them up to the selected size of 5000 (eg by adding more common C code at the end), to get files that can be checked into git and will yeild colliding blobs. The need for the file size to remain fixed is the only real complication here.

Comment by joey

Hello Joey,

neat post. As far as I can tell, you also need to get the SHA calculation in the proper state, though For example, just the two different string parts will not compare equal, you need the 192 bytes of prefix, too. Also, you cannot prefix the blobs with a common prefix and get the same result. So I would think that there is still significant "cryptographic" effort needed to apply this to git.

Best regards


Comment by
comment 1
misterbonnie             7*yLN#!NOK
misterbonnie             mFMC^&*FMM'
misterbonnie             | *'     _\
misterbonnie             |  =|MM|\MM  but i like
misterbonnie              \V    c_)|   putting ascii art
misterbonnie               |\   __ /   in my code :(
misterbonnie               |    __/
misterbonnie               \     |
misterbonnie                \ 
Comment by joey
comment 3

@Joachim I was aware of haskell-tor when I built this. I chose not to use it for a bunch of reasons including, it does not suppport tor hidden services, it's not been peer reviewed, tor has a much more active development community that's working on a next generation of hidden services amoung other things.

Comment by joey
Using haskell-tor


instead of relying on a system installation of tor, have you considered using tor as a library, maybe using the bindings provided by Galois:

You might also be interested in how this Tor-Hidden-Service based instant messenger does it:


Comment by mail
comment 5

Flatpak and snap don't seem to make it any easier to deal with this than docker does. Probably harder if your app has any unusual dependencies, since if the Flatpack runtime does not include a dependency, you have to bundle the dependency with the app.

Compiler developers should not need to worry about including their compiler in a runtime.

Flatpack might be slightly less of a security nightmare than docker is, due to more finegrained sandboxing, and since the app runtimes can be upgraded.

It seems unlikely they're going to take off though. Docker has already nearly fully occupied their space, and instead of competing with it, flatpak and snap are competing with one-another.

Comment by joey
tmux's API and ii

i have a similar concern in a project i'm working on now, which is to run a bunch of tests in one window and showing system logs in another window for convenience. i thought i would use tmux for this because the commandline interface is simple enough to be used like an API. my previous approach to this was to use a multi-interface design with simple functions like "alert", "prompt" or "log" that would send stuff to different places (terminal or gtk windows) depending on the environment, a bit like what debconf is doing, with all the crap that implies (generally: a suboptimal interface everywhere).

as for an irc client, you may be curious to try out ii: it's a simple FIFO and filesystem-based irc client that implements basically no user interface... a little nuts, but sounds like it would be the backend to the UI you are describing.

Comment by anarcat

this is awesome. :) i have used this in weird cases (which, unfortunately, now include debian jessie ;) and bundle at least one of those binaries next to my git-annex archives...

it seems there's a new player in this problem space - did you look at flatpak? the haskell folks have started to consider it...

Comment by anarcat
I2P is fit for anonymous filesharing by design

Trying out I2P is what I would recommend here, because especially if you are interested in filesharing or anonymous mail-services, this would probably be more cost-efficient, it is said to exist inside the FeeBSD ports-collection. So from my perspective TOR is nice for anonymous web-surfing, no idea about onion-services. Some big sites closed their services to TOR-users unfortunately, probably because of DDoS-attacks, some even don't accept connections from filtering proxies anymore. I2P is still in beta-status AFAIK, but probably one day it is going to be fit for real productive use and match up TOR. I2P is a different category, in so far as even the devolpers are anonymous and known by their pseudonyms only.

Comment by andreas.glaeser
climate change

2016 was a record-year globally, as far as I know, even warmer in average than 2015. Now, upon break of the new year, there is even snow and ice again in Berlin, and it is supposed to stay this way for three more days at least with temperatures below zero. This is not bad weather in my opinion, but more lovable weather indeed, because far less people are out there to get in your way. Sometimes bad weather is good weather in this sense, sometimes cool weather is good weather.

Comment by andreas.glaeser
Multi Pty programs
Voltron is a good example of this kind of program I think. I haven't actually used it, but it's a debugger interface that lets you spwawn new windows by using multiple ptys.
Comment by bburky
comment 5

The C API could look like this:

int openpty_managed(const char *addr,
    int *amaster, int *aslave, char *name,
    const struct termios *termp,
    const struct winsize *winp);

pid_t execvpepty_managed(const char *addr,
    const struct winsize *winp,
    const char *file, char *const argv[], char *const envp[]);

Compare with openpty(3) and forkpty(3).

Here addr is the address of the pty, eg "", and any parent address is automatically prefixed to it.

The implementation of these would use the socket environment variable to send requests to the pty manager. Mm, fd passing..

The winsize parameter lets a preferred window size be selected (to open an xterm with that size, or tile a buffer at that size).

It would probably be a good idea to add a parameter to both API calls for additional hints. For example, a pty manager might be able to lay out one buffer to the left of an other one, if provided a hint to do so. Or a pty might be non-interactive, and so a pty manager should not let the focus move to it. This needs to be something that's extensible. Could be as simple as char *const managehints[]

Comment by joey
comment 4

@josh interesting thoughts..

A pty-management library does seem like a good idea. There could be different pty-manager commands that open xterms, open screen buffers, etc. So the user runs something like:

in-xterms irc-client

in-screen irc-client

And irc-client could also be run in any shell that was started inside a pty-manager, talking to the outer pty-manager.

When the irc-client opens a pty, in-xterms would need to use something like xterm -e connect-pty to open a new xterm connected back to that pty. But when the irc-client runs the mail-client, in-xterms can just run xterm -e mail-client and doesn't need to relay.

(Leaving aside for the momemnt the problem that xterm -e does not propigate the exit status of the program it runs to its caller..)

While connect-pty would have some copying and syscall overhead (probably similar to screen), if a program found that excessive, it could run a child process, which would be connected up directly to the terminal (in the in-xterms case). That also provides a workaround for the singleton termnal in curses libraries.

As well as an environment variable containing the socket of the pty-manager, there could be one containing the address of the pty. If an irc-client opens a new pty to run a mail client, the mail client would see an address of eg "irc-client.mail-client", and it can use that address when it opens a new pty for an editor, so the editor gets an address of eg "irc-client.mail-client.editor". The pty manager can see a whole tree:

irc-client.mail-client.message.happy new year

Such human-friendly names in a tree would be useful for interactively managing ptys and could also inform tiling layouts. It also lets the pty manager set the default window title/screen buffer name to something reasonable.

Comment by joey
comment 3

Trying st with my ad-hoc benchmark, it managed to scroll around 4 thousand lines without noticable flicker. But I don't think st has a scroll buffer. Running screen in it to buffer the scrollback slowed it down to around 1-2 thousand lines, so comprable to vte.

Comment by joey
comment 2

I like the idea of unifying terminals, editors, IRC clients, mail clients, and other things that need multiple windows (whether tiled, tabbed, or just switched between). I don't know that I'd want to use terminal scrollback for that (I prefer editor-style or screen-style buffers with scroll positions), but many aspects of all those programs overlap. I can see why people use Emacs for everything, and I hope to see vim/neovim go in the same direction now that both have introduced coprocess support.

Editor buffers work nicely as a generalization, but they don't fit perfectly. For instance, embedding a shell in an editor needs to disable some of the editing functions for part but not all of the buffer. You want to use the editor to edit the next command line, but treat the prompt before it as read-only. (But you can use editor functions to search and copy text.)

I'd love to have a "libtmux" or "libscreen" that applications can embed. But I'd also like those applications to cooperate within a single thing that behaves like screen/tmux/vim, not to each have their own. What would such an abstraction look like?

What about a multi-pty-management library that can delegate to an outer multi-pty-management application. Imagine a library that lets you manage a pile of independent terminal-like buffers, and manage them like screen/tmux within a normal containing terminal. However, that library can also recognize when run inside an application that can handle such buffers itself (such as via an environment variable providing a communication socket). In that case, it'll contact that application, and delegate pty management to that application. The application can then logically group the buffers from each application together, and provide various functions for navigation, tiling, tabbing, switching, scrolling, searching, etc.

Comment by josh