Seems Debian is talking about why they are unable to package whole categories of modern software, such as anything using npm. It's good they're having a conversation about that, and I want to give a broader perspective.
Lars Wirzenius's blog post about it explains the problem well from the Debian perspective. In short: The granularity at which software is built has fundamentally changed. It's now typical for hundreds of small libraries to be used by any application, often pegged to specific versions. Language-specific tools manage all the resulting complexity automatically, but distributions can't muster the manpower to package a fraction of this stuff.
Lars lists some ideas for incremental improvements, but the space within which a Linux distribution exists has changed, and that calls not for incremental changes, but for a fundamental rethink from the ground up. Whether Debian is capable of making such fundamental changes at this point in its lifecycle is up to its developers to decide.
Perhaps other distributions are dealing with the problem better? One way to evaluate this is to look at how a given programming language community feels about a distribution's handling of their libraries. Do they generally see the distribution as a road block that must be worked around, or is the distribution a useful part of their workflow? Do they want their stuff included in the distribution, or does that seem like a lot of pointless bother?
I can only speak about the Haskell community. While there are some exceptions, it generally is not interested in Debian containing Haskell packages, and indeed system-wide installations of Haskell packages can be an active problem for development. This is despite Debian having done a much better job at packaging a lot of Haskell libraries than it has at say, npm libraries. Debian still only packages one version of anything, and there is lag and complex process involved, and so friction with the Haskell community.
On the other hand, there is a distribution that the Haskell community broadly does like, and that's Nix. A subset of the Haskell community uses Nix to manage and deploy Haskell software, and there's generally a good impression of it. Nix seems to be doing something right, that Debian is not doing.
It seems that Nix also has pretty good support for working with npm packages, including ingesting a whole dependency chain into the package manager with a single command, and thousands of npm libraries included in the distribution I don't know how the npm community feels about Nix, but my guess is they like it better than Debian.
Nix is a radical rethink of the distribution model. And it's jettisoned a lot of things that Debian does, like manually packaging software, or extreme license vetting. It's interesting that Guix, which uses the same technologies as Nix, but seems in many ways more Debian-like with its care about licensing etc, has also been unable to manage npm packaging. This suggests to me that at least some of the things that Nix has jettisoned need to be jettisoned in order to succeed in the new distribution space.
But. Nix is not really exploding in popularity from what I can see. It seems to have settled into a niche of its own, and is perhaps expanding here and there, but not rapidly. It's insignificant compared with things like Docker, that also radically rethink the distribution model.
We could easily end up with some nightmare of lithification, as described by Robert "r0ml" Lefkowitz in his Linux.conf.au talk. Endlessly copied and compacted layers of code, contained or in the cloud. Programmer-archeologists right out of a Vinge SF novel.
r0ml suggests that we assume that's where things are going (or indeed where they already are outside little hermetic worlds like Debian), and focus on solving technical problems, like deployment of modifications of cloud apps, that prevent users from exercising software freedoms.
In a way, r0ml's ideas are what led me to thinking about extending Scuttlebutt with Annah, and indeed if you squint at that right, it's an idea for a radically different kind of distribution.
Well, that's all I have. No answers of course.
One of the killer features of Debian is that it does not just package one version of anything. C libraries are usually packaged so that the package name contains soname version, so that multiple versions are coinstallable and dependencies just work. When things break (and they do), it's usually easy to keep an old version for years while the rest of the system is updated regularly. This usually applies to libs only, not their -dev packages, but sometimes there are multiple versions of those as well, and even non-libraries, e.g python3.5 & python3.6. Obviously this increases maintenance costs, but from an end user (me) point of view, it's absolutely worth it. (Thanks to everyone involved!)
Now for Haskell this a bit more difficult because of code inlining. To make things coinstallable, I'd suggest reversing the current package namimg practice: instead of having "libghc-mtl-dev" as name and "libghc-mtl-dev-2.2.1-93d32" as Provides, do it the other way round. Perhaps even "libghc8.0-mtl-dev-..." Admittedly this would poison the package namespace and slow apt down considerably. Also, unless it's all automated, the manual labor needed to maintain multiple versions of everything would be unbearable.
I guess if there was an apt repository that contained such packages and I could ask it to install from a specific version of Stackage LTS, I'd use it instead of stack immediately. I'm quite annoyed by stack's ignorance of disk space. As if these young people never ever had to uninstall anything.
I believe the popularity of NixPkgs has multiplied over the past few years, e.g. look at "contributors per month" (60 -> 250 in the past four years). That shows rapid growth of the number of people who "only" send a few changes (per month), which is IMHO a plausible indication of being an common active user.
I actually think that too rapid growth would be detrimental, as quite some things need to change in the organization of such a project to handle the growth (e.g. just manpower for issues and PRs), and such changes tend work better if given sufficient time.
BTW, NixPkgs also strives to only have one version per package, except for cases with good-enough reason to do otherwise. It really helps maintenance, debugging, etc. So it's a kind of strange situation: technically Nix makes multiple versions easier, but tries to avoid using this :-)