moreutils is a growing collection of the unix tools that nobody thought to write long ago when unix was young.

It began when I blogged:

I'm a fan of the unix tools philosophy, but I sometimes wonder if there's much room for new tools to be added to that toolbox. I've always wanted to come up with my own general-purpose new unix tool.

Well, after lots of feedback documented in the many followups (1 2 3) in my blog, I've concluded:

Maybe the problem isn't that no-one is writing them, or that the unix toolspace is covered except for specialised tools, but that the most basic tools fall through the cracks and are never noticed by people who could benefit from them.

And so the moreutils collection was born, to stop these programs from falling through the cracks.

What's included

Probably the most general purpose tool in moreutils so far is sponge(1), which lets you do things like this:

% sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

There are lots more listed below, and I'm always interested to add more to the collection, as long as they're suitably general-purpose, and don't duplicate other well-known tools.

  • chronic: runs a command quietly unless it fails
  • combine: combine the lines in two files using boolean operations
  • ifdata: get network interface info without parsing ifconfig output
  • ifne: run a program if the standard input is not empty
  • isutf8: check if a file or standard input is utf-8
  • lckdo: execute a program with a lock held
  • mispipe: pipe two commands, returning the exit status of the first
  • parallel: run multiple jobs at once
  • pee: tee standard input to pipes
  • sponge: soak up standard input and write to a file
  • ts: timestamp standard input
  • vidir: edit a directory in your text editor
  • vipe: insert a text editor into a pipe
  • zrun: automatically uncompress arguments to command

Download

A Debian package as well as the source tarball for moreutils can be downloaded from packages.debian.org, or using apt. It's also in Ubuntu, and, I hear in several other Linux distributions.

The git repository can be cloned from git://git.kitenet.net/moreutils

News

version 0.54

moreutils 0.54 released with these changes

  • vidir: Fix bug in handling filenames that share a common prefix when renaming. Closes: #654326 Thanks, Kushal Kumaran
Posted
version 0.53

moreutils 0.53 released with these changes

  • Orphaned the Debian package.
Posted
version 0.51

moreutils 0.51 released with these changes

  • Fix ifdata to build on OS X. Thanks, Peter Eisentraut Closes: #575294
  • use docbook-xsl for building man pages. Thanks, Peter Eisentraut for extensive patch set! Closes: #728831
  • Fix ifdata, ifne, parallel to build on illumos. Thanks, Gabriele Giacone Closes: #733736
Posted

TODO

  • pee should support non-blocking i/o to write to the pipes to allow concurrent processing of the data by the programs. Alternatively, switch to fountain http://hea-www.cfa.harvard.edu/~dj/tmp/fountain-1.0.2.tar.gz.

    Alternatively, make sponge buffer to stdout if no file is given, and use it to buffer the data from pee. Although this will be less efficient and will not work as well for very large streams unless sponge avoids buffering the whole contents in memory in this case.

Tools under consideration

Here are some that are under consideration but have not yet been included. Feel free to suggest others. I also welcome feedback on which of these to include.

  • dirempty/exists

    It's too hard to tell if a directory is empty in shell. Also, while test -e works ok for a single file, it fails if you want to see if a wildcard matches anything.

    Suggested in bug #385069, see bug for my comments.

  • cattail

    Allows catting a file that's still changing (ie, being downloaded) to a program. The new bits of the file will continue to be fed to the program until the download is done.

    Submitted by Justin Azoff, with code. However, it has to use heuristics to guess when the download (or whatever) is done. The current heuristic, 10 seconds w/o growth, wouldn't work very well for me on dialup.

    Using inotify is probably the best approach. fsniper is a less general-purpose tool that uses inotify to detect when a file is closed.

  • phoenix

    Respawns a process unless a user really wants to quit. Suggested in bug #382428

    Doesn't seem general enough.

  • haschanged

    Run it once to store a file's hash, and the second time it'll check whether the file has changed. http://blog.steve.org.uk/the_traffic_is_waiting_outside.html

  • tmp

    puts stdin into a temp file and passes it to the specified program

    ex: zcat file.bmp.gz | tmp zxgv

    Alternative: pip

  • connect

    connect 'cmd1' op 'cmd2' ... -- connects fd's of commands together, etc

    • In the same spirit as 'pee', but much more powerful.
    • If done very simply, this is handy for writing coprocesses as pipelines that need to communicate back and forth.
    • You can do SOME of this with a great shell, like bash or zsh; you can do almost all the rest with a bunch of mkfifo commands plus simple redirection, but with added complexity and a lot of manual steps.
    • This command could be even more powerful if you gave it essentially a "netlist" of fd's to connect. I'm sure the command line syntax could be improved, but you get the idea. Very very complex example just to illustrate: connect 'cmd1' '<> #0:3>4' 'cmd2' '3>' \ 'cmd3' '3<>3 #0:0>' 'cmd4' '3>#1:5'
      * specs specify connections between adjacent cmds
      * qualified specs (w/ '#') allow more complex connections
      * Some sane defaults, but can be overridden
        * stdin goes to first process that doesn't redirect it
        * stdout comes from everyone that doesn't redirect it
        * stderr comes from everyone that doesn't redirect it
      * cmd1's stdout -> cmd2's stdin
      * cmd2's stdout -> cmd1's stdin
      * fd3           -> cmd2's fd4
      * cmd2's fd3    -> cmd3's stdin
      * cmd3's fd3    -> cmd4's fd3
      * cmd4's fd3    -> cmd3's fd3
      * stdin         -> cmd4
      * cmd4's fd3    -> cmd1's fd5
      * stdout <- all w/o redirected stdout (in this case, cmd3)
      * stderr <- all w/o redirected stderr (in this case, all)
      
    • If you think this is a good idea, let me know. I have a basic connect command, but it only does two commands. However, I'll be happy to code this up if there is interest. (In fact, I think I may anyway, so I don't keep doing stuff like this ad-hoc all the time). -- from Wesley J. Landaker

    • Should be possible to roll mispipe up into this by adding a way to flag which command(s) exit status to return.

Rejected tools

(Some of these rejections may be reconsidered later.)

  • add

    adds up numbers from stdin

    Already available in numutils. RFP bug filed.

  • todist

    inputs a list of numbers and outputs their distribution, a value and how many time it occurs in the input http://baruch.ev-en.org/files/todist

    More suitable for numutils, which can probably do it already. RFP bug filed.

  • tostats

    inputs a list of numbers and outputs some statistics about the numbers: average, stddev, min, max, mid point http://baruch.ev-en.org/files/tostats

    More suitable for numutils, which can probably do it already. RFP bug filed.

  • unsort

    Randomise the lines of a file. Perfect candidate, but bogosort and rl (from the randomize-lines package) already do it.

    http://savannah.nongnu.org/projects/shuffle/ is a similar thing, which its author describes as "almost coreutil ready, but its memory bound, a big nono". (Apparently coreutils 6 has a shouf and sort --random-sort.)

  • mime

    determines the mime type of a file using the gnome mine database

    The File::MimeInfo perl module has a mimetype that works like this, and uses the freedesktop.org mime database, same as GNOME.

    file -bi can do this too.

  • srename

    Applies a sed pattern to a list of files to rename them. Rejected because perl has a rename program that works nearly identically.