after spending +3 years getting experienced and reading a lot of the source code (getting past the documentation/learning-curve problem), I can say there are at least two fundamental flaws, with nixpkgs moreso than nixos:
The messy mono-repo was never desgined to be searchable. It might feel searchable to a newby, but many teams have dumped tons of effort and slapped together lots of hacks to make niche-package-versions even halfway searchable. Devbox is doing a good job of fixing this, but its not there yet.
The monorepo design is unmaintainable/unscalable from a package maintainer standpoint. There’s a ton of contributor burnout, there’s no automated quality control on packages, to outsiders its not obvious how to report package-specific issues or how to edit/fix/contribute to a single package, and instead of 30min to publish a hello-world npm or cargo package, users need to make a PR on the core, and get it approved. Meaning publishing a hello-world package would get rejected anyways. The good news is flakes hub is fixing this, and I’m really excited for that.
The good is;
The pros still outweigh the cons. My projects from 3 years ago that I haven’t touched still work (0 bitrot) first try, no manual install/setup needed.
people have put a ton of effort into nix. Its truely amazing how many things work in nix despite how absurdly difficult it is to get things working with nix
it is pretty much as reproducable as it can be and nothing else is even remotely close.
The monorepo design is unmaintainable/unscalable from a package maintainer standpoint. There’s a ton of contributor burnout, there’s no real quality control on packages
There is contributor burnout and perhaps some quality issues here and there but the monorepo isn’t the reason for that. In fact, it’d be a lot worse if we had hundreds of smaller repos instead as wide-reaching changes would become basically impossible with our current manpower.
There have been calls to fragment Nixpkgs for years but they’ve almost always been struck down because none of what was suggested would improve anything about our current maintenance issues.
to outsiders its not obvious how to report package-specific issues
You simply open an issue on GitHub. I don’t know how it could be any more clear.
I just checked and we don’t explicitly document this in CONTRIBUTING.md but I don’t think we should need to. It’s just too obvious IMO.
You actually bring up a very good point that I missed; another problem, maybe the largest problem with nix, is the culture issue.
A comment like “what do you mean nixOS doesn’t have good documentation??? we have tons of good documentation!” feels very similar to reading what you just said about “You could simply open an issue on Github, I dont know how it could be clearer”. Neither of those are a question. They’re not asking “how can we do better? (I’m sorry you, as a new user, had a hard time)”. Those comments are statements, and they say “you [person struggling to report a package-specific issue] are the problem; our system doesn’t need to change. YOU need to change”.
This culture issue has real consequences. It’s the reason Determinate Systems has a separately maintained nixpkgs-installer script, and separately maintained documentation. Its the reason I also have my own independently maintained nixpkg-installer and nixpkg-uninstaller script.
_
Maybe you haven’t had our problem because you’re really skilled and familiar with nix/github/whatever. Maybe you haven’t run into our problem because your use-case doesn’t overlap with ours. I am glad you don’t run into the problem.
But that doesn’t somehow make the problem not-a-problem.
Adopt a legacy codebase that is massive and requires EXACTLY ruby 2.6.0-rc1. On ubuntu, using rbenv, it takes a newby 30 sec to list all the available versions, and 30 sec to install ruby 2.6.0-rc1.
Even excluding versions, try searching for the extremely-commonly-needed “Core Foundation” package on search.nixos.org. I assure you the package does exist on nixpkgs, I’ve been using it for years. As a newby had to spend WEEKS looking through the source code and learning nix-lang quirks just to find it.
_
But more importantly please ignore those details and look at the bigger picture; we are on the same team. I’m not insulting or ignoring the massive accomplishments nix team has made. They (maybe you as well) are giants that have moved moutains and accomplished things I wouldve considered basically impossible. I want to help the core devs have LESS work. I want to have productive discussions about the trade-offs of federated vs monorepo, searchability, documentation improvements, installer scripts, etc.
But we can’t.
Not until the discussion starts with “I agree there’s a problem” instead of “there is no problem other than YOUR lack of skill”
Yup, figuring out how to install gcc 10 was a pain. Turns out I needed stdenvgcc10, but how would I have known that without reading like the entirety of nix pills? I don’t want to read a manual for 2 hours to figure out how to do something basic. Not to mention nix pills is SO dense that as someone who can’t learn from textbooks, its impossible.
they say “you [person struggling to report a package-specific issue] are the problem; our system doesn’t need to change. YOU need to change”.
I have not met anyone in the Nix community who’s opposed to fixing actual systematic issues. I just highly doubt that the discoverability of github issue tracking in particular is a problem that Nixpkgs is in any way responsible for.
I too wouldn’t be entirely opposed to adding i.e. a little section on how to report issues but when you’ve read half a README, then opened CONTRIBUTING.md and read through that, you really should have discovered the “issues” tab to report your, well, issue by then.
It’s the reason Determinate Systems has a separately maintained nixpkgs-installer script, and separately maintained documentation.
No. The reason they have separate instances of those is that they allow a green-field approach to things. “Move fast and break things” is great for development but you can’t do that when the entire ecosystem relies on the things you might be breaking.
The installer is a great counter example actually. If someone wanted to replace the regular installer with the detsys installer right now, the greatest opposition they’d likely face is “hey, let’s be careful to not break users’ setups, does (niche feature) still work?”.
I wouldn’t be surprised to see it replace the current official installer within the next year.
Adopt a legacy codebase that is massive and requires EXACTLY ruby 2.6.0-rc1. On ubuntu, using rbenv, it takes a newby 30 sec to list all the available versions, and 30 sec to install ruby 2.6.0-rc1.
Nixpkgs has never has supported that version and does not support using multiple versions of Nixpkgs either (not even the currently maintained branches). You can try to and it’ll probably (perhaps even likely) work but it’s not “intended” to and nobody will want to deal with issues you might encounter with that.
We regularly kick out packages that have stupid version requirements like that for a reason. Eventhough we could technically have an infinite amount of versions of any package we choose not to because it’s a maintenance burden we cannot support.
The “proper” way of handling an issue like that (I’m sorry to say but depending on some old specific version is actually an issue of the dependant) is to “vendor” the dependency; copying its expression out of the Nixpkgs tree and maintaining it yourself.
try searching for the extremely-commonly-needed “Core Framework” package on search.nixos.org
According to repology, a package under than name exists in no repository and it knows about a damn lot of repositories:
I don’t know which “Core Framework” you are referring to either.
But more importantly please ignore those details and look at the bigger picture; we are on the same team. I’m not insulting or ignoring the massive accomplishments nix team has made. They (maybe you as well) are giants that have moved moutains and accomplished things I wouldve considered basically impossible. I want to help the core devs have LESS work. I want to have productive discussions about the trade-offs of federated vs monorepo, searchability, documentation improvements, installer scripts, etc.
But we can’t.
Not until the discussion starts with “I agree there’s a problem” instead of “there is no problem other than YOUR lack of skill”
I’ve yet to see a “bigger picture”-issue described in what you wrote.
At this point I’m not sure whether we’re talking about the same Nix community anymore. We have a lot of those “big picture” issues in the Nix community and we’re aware of them. What we need the most help with is fixing them, not finding them.
I have not met anyone in the Nix community who’s opposed to fixing actual systematic issues
True, there is no one actually broadly and openly opposing change willingly or based on conviction. I think the main inhibitor to change in the nix (-pkgs / -OS) communities stems from fear. The situation is already pretty unpleasant, I think we can all agree. Contrary to that, Nix* has gained a lot of momentum in the last couple of years. Fundamental change would affect a lot of people and can easily turn the situation from unpleasant to unbearable. So people are not easily convinced to take a chance on ideas and just “try it out for a release or two” but rather die on the hills of their own personal arguments pro or against something. This ultimately leads to people blocking each other in and even though there is a will and a need to change, there is no change or it is coming very, very slowly.
My personal fear is that the shift from unpleasant to unbearable will happen not because of wrong action but because of fearful inaction.
If many people report a problem, the correct response is never “no, you’re wrong, that’s not a problem”. There’s no technical aspect to it, that is just common dececency when talking to another human.
That’s the culture issue with Nix.
It doesn’t mean an issue will never be “wont-fix” or that a issue won’t genuinely be a skill-issue. It means it’s not hard to say “I’m sorry, I disagree with adding support for that feature” or “here’s a tutorial for [skill issue]” instead of saying “no, you’re wrong, what you reported is not a problem”. When you mention editing the CONTRIBUTING.md you’re not offering help, you’re just setting up a punchline for an “[but even that wouldn’t help this IDIOT]”. There’s no sincerity, no sympathy or attempt to understand.
I don’t know how to explain it any more clearly… you’re not convincing anyone to help you on those “big issues” with words like “you really should have discovered the issues tab”.
It’s actually not. If you search for go you get the go package, but if you search for go1.21, no go package will even appear on the first page. The package you are looking for is somewhere on page 7 or 8.
How do you see these solutions by third-party? Wouldn’t that likely lead to go far away from the community and do commercialization and proprietary solutions?
I worry some about devbox being closed source with their package indexing soltuion. I talked with them on a video call and they seem like great guys. And they said they’d pull the indexing part out of their private repo and make it open source so I could help work on it. They openly talked about their design and have blog posts about it too. But that was 2 maybe 3 months ago and I still haven’t heard back. I should maybe ping them. The real downside is they’ve got an unusual amount of “lock in”. Devbox is not a tool that enhances nix, its just a tool that uses nix under the hood. However they do solve the ease-of-use problem which is a really big deal (they make some tradeoffs though).
Flakes hub, by Determinate Systems, I have absolutely 0 concern about. They’re truely just enhancing nix, and even if they dissapeared the packages already on flakeshub would still effectively work because they’re distrubuted. Flakeshub is just a registry for standarized searching of flakes by individual people. Publishing is built on top of github actions which I’m not the biggest fan of. But there are ways of running github actions locally.
I have to agree on those two cons. Contributing to nixpkgs is a major pain point for me. I don’t mind having to get PRs reviewed, but the speed and the lack of consistency have really put me of from doing that again.
Quality control is also a major pain point. Nixpks supports a bunch of platforms and architectures, but anything other than tier 1 (x86_64-linux, gcc + glibc ) can and will just randomly break because maintainers and contributors don’t usually care for anything that’s not right in front of their eyeballs.
Personally, I avoid touching the monorepo like the plague and stick to putting everything into flakes if possible.
Agreed, flakes are the way. Its just making them searchable hasn’t been easy/realistic until recently.
Also I feel like flakes are kind of tainted by always pulling in nixpkgs as a massive dependency chain. I think there should be a standard library separate from the packages, and nixpkgs.lib is 80% pure functions. So I’ve been working on making a “lib” flake that
Is 100% pure (no stdenv)
is versioned
doesnt link/depend on all of nixpkgs
I know it still won’t be practical to avoid depending on nixpkgs, but I think its a step towards breaking up nixpkg into organized/managable chunks.
I actually think I have seen this being discussed, but does not seem to be at the RFC level yet. Probably drowned in all of the flakes vs monorepo discussions.
after spending +3 years getting experienced and reading a lot of the source code (getting past the documentation/learning-curve problem), I can say there are at least two fundamental flaws, with nixpkgs moreso than nixos:
(Edit: Added 0)
The culture issue. See thread below
The messy mono-repo was never desgined to be searchable. It might feel searchable to a newby, but many teams have dumped tons of effort and slapped together lots of hacks to make niche-package-versions even halfway searchable. Devbox is doing a good job of fixing this, but its not there yet.
The monorepo design is unmaintainable/unscalable from a package maintainer standpoint. There’s a ton of contributor burnout, there’s no automated quality control on packages, to outsiders its not obvious how to report package-specific issues or how to edit/fix/contribute to a single package, and instead of 30min to publish a hello-world npm or cargo package, users need to make a PR on the core, and get it approved. Meaning publishing a hello-world package would get rejected anyways. The good news is flakes hub is fixing this, and I’m really excited for that.
The good is;
I’m not sure what you mean by “searchable”?
https://search.nixos.org/packages exists and it’s generally pretty damn good.
There is contributor burnout and perhaps some quality issues here and there but the monorepo isn’t the reason for that. In fact, it’d be a lot worse if we had hundreds of smaller repos instead as wide-reaching changes would become basically impossible with our current manpower.
There have been calls to fragment Nixpkgs for years but they’ve almost always been struck down because none of what was suggested would improve anything about our current maintenance issues.
You simply open an issue on GitHub. I don’t know how it could be any more clear.
I just checked and we don’t explicitly document this in CONTRIBUTING.md but I don’t think we should need to. It’s just too obvious IMO.
https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md#how-to-create-pull-requests
FlakeHub fixes none of the maintenance issues of Nixpkgs.
You actually bring up a very good point that I missed; another problem, maybe the largest problem with nix, is the culture issue.
A comment like “what do you mean nixOS doesn’t have good documentation??? we have tons of good documentation!” feels very similar to reading what you just said about “You could simply open an issue on Github, I dont know how it could be clearer”. Neither of those are a question. They’re not asking “how can we do better? (I’m sorry you, as a new user, had a hard time)”. Those comments are statements, and they say “you [person struggling to report a package-specific issue] are the problem; our system doesn’t need to change. YOU need to change”.
This culture issue has real consequences. It’s the reason Determinate Systems has a separately maintained nixpkgs-installer script, and separately maintained documentation. Its the reason I also have my own independently maintained nixpkg-installer and nixpkg-uninstaller script. _
Maybe you haven’t had our problem because you’re really skilled and familiar with nix/github/whatever. Maybe you haven’t run into our problem because your use-case doesn’t overlap with ours. I am glad you don’t run into the problem. But that doesn’t somehow make the problem not-a-problem.
Adopt a legacy codebase that is massive and requires EXACTLY ruby
2.6.0-rc1. On ubuntu, using rbenv, it takes a newby 30 sec to list all the available versions, and 30 sec to install ruby2.6.0-rc1.Try finding that version on search.nixos.org.
Even excluding versions, try searching for the extremely-commonly-needed “Core Foundation” package on search.nixos.org. I assure you the package does exist on nixpkgs, I’ve been using it for years. As a newby had to spend WEEKS looking through the source code and learning nix-lang quirks just to find it.
_
But more importantly please ignore those details and look at the bigger picture; we are on the same team. I’m not insulting or ignoring the massive accomplishments nix team has made. They (maybe you as well) are giants that have moved moutains and accomplished things I wouldve considered basically impossible. I want to help the core devs have LESS work. I want to have productive discussions about the trade-offs of federated vs monorepo, searchability, documentation improvements, installer scripts, etc.
But we can’t.
Not until the discussion starts with “I agree there’s a problem” instead of “there is no problem other than YOUR lack of skill”
Yup, figuring out how to install gcc 10 was a pain. Turns out I needed stdenvgcc10, but how would I have known that without reading like the entirety of nix pills? I don’t want to read a manual for 2 hours to figure out how to do something basic. Not to mention nix pills is SO dense that as someone who can’t learn from textbooks, its impossible.
I have not met anyone in the Nix community who’s opposed to fixing actual systematic issues. I just highly doubt that the discoverability of github issue tracking in particular is a problem that Nixpkgs is in any way responsible for.
I too wouldn’t be entirely opposed to adding i.e. a little section on how to report issues but when you’ve read half a README, then opened CONTRIBUTING.md and read through that, you really should have discovered the “issues” tab to report your, well, issue by then.
No. The reason they have separate instances of those is that they allow a green-field approach to things. “Move fast and break things” is great for development but you can’t do that when the entire ecosystem relies on the things you might be breaking.
The installer is a great counter example actually. If someone wanted to replace the regular installer with the detsys installer right now, the greatest opposition they’d likely face is “hey, let’s be careful to not break users’ setups, does (niche feature) still work?”.
I wouldn’t be surprised to see it replace the current official installer within the next year.
Nixpkgs has never has supported that version and does not support using multiple versions of Nixpkgs either (not even the currently maintained branches). You can try to and it’ll probably (perhaps even likely) work but it’s not “intended” to and nobody will want to deal with issues you might encounter with that.
We regularly kick out packages that have stupid version requirements like that for a reason. Eventhough we could technically have an infinite amount of versions of any package we choose not to because it’s a maintenance burden we cannot support.
The “proper” way of handling an issue like that (I’m sorry to say but depending on some old specific version is actually an issue of the dependant) is to “vendor” the dependency; copying its expression out of the Nixpkgs tree and maintaining it yourself.
According to repology, a package under than name exists in no repository and it knows about a damn lot of repositories:
https://repology.org/projects/?search=Core+Framework&maintainer=&category=&inrepo=¬inrepo=&repos=&families=&repos_newest=&families_newest=
I don’t know which “Core Framework” you are referring to either.
At this point I’m not sure whether we’re talking about the same Nix community anymore. We have a lot of those “big picture” issues in the Nix community and we’re aware of them. What we need the most help with is fixing them, not finding them.
True, there is no one actually broadly and openly opposing change willingly or based on conviction. I think the main inhibitor to change in the nix (-pkgs / -OS) communities stems from fear. The situation is already pretty unpleasant, I think we can all agree. Contrary to that, Nix* has gained a lot of momentum in the last couple of years. Fundamental change would affect a lot of people and can easily turn the situation from unpleasant to unbearable. So people are not easily convinced to take a chance on ideas and just “try it out for a release or two” but rather die on the hills of their own personal arguments pro or against something. This ultimately leads to people blocking each other in and even though there is a will and a need to change, there is no change or it is coming very, very slowly. My personal fear is that the shift from unpleasant to unbearable will happen not because of wrong action but because of fearful inaction.
If many people report a problem, the correct response is never “no, you’re wrong, that’s not a problem”. There’s no technical aspect to it, that is just common dececency when talking to another human.
That’s the culture issue with Nix.
It doesn’t mean an issue will never be “wont-fix” or that a issue won’t genuinely be a skill-issue. It means it’s not hard to say “I’m sorry, I disagree with adding support for that feature” or “here’s a tutorial for [skill issue]” instead of saying “no, you’re wrong, what you reported is not a problem”. When you mention editing the CONTRIBUTING.md you’re not offering help, you’re just setting up a punchline for an “[but even that wouldn’t help this IDIOT]”. There’s no sincerity, no sympathy or attempt to understand.
I don’t know how to explain it any more clearly… you’re not convincing anyone to help you on those “big issues” with words like “you really should have discovered the issues tab”.
It’s actually not. If you search for
goyou get the go package, but if you search forgo 1.21, no go package will even appear on the first page. The package you are looking for is somewhere on page 7 or 8.Indeed, that looks like a bug. The version field is there but it doesn’t appear to be searched.
Could you open an issue on https://github.com/NixOS/nixos-search?
How do you see these solutions by third-party? Wouldn’t that likely lead to go far away from the community and do commercialization and proprietary solutions?
I worry some about devbox being closed source with their package indexing soltuion. I talked with them on a video call and they seem like great guys. And they said they’d pull the indexing part out of their private repo and make it open source so I could help work on it. They openly talked about their design and have blog posts about it too. But that was 2 maybe 3 months ago and I still haven’t heard back. I should maybe ping them. The real downside is they’ve got an unusual amount of “lock in”. Devbox is not a tool that enhances nix, its just a tool that uses nix under the hood. However they do solve the ease-of-use problem which is a really big deal (they make some tradeoffs though).
Flakes hub, by Determinate Systems, I have absolutely 0 concern about. They’re truely just enhancing nix, and even if they dissapeared the packages already on flakeshub would still effectively work because they’re distrubuted. Flakeshub is just a registry for standarized searching of flakes by individual people. Publishing is built on top of github actions which I’m not the biggest fan of. But there are ways of running github actions locally.
I have to agree on those two cons. Contributing to nixpkgs is a major pain point for me. I don’t mind having to get PRs reviewed, but the speed and the lack of consistency have really put me of from doing that again. Quality control is also a major pain point. Nixpks supports a bunch of platforms and architectures, but anything other than tier 1 (
x86_64-linux,gcc+glibc) can and will just randomly break because maintainers and contributors don’t usually care for anything that’s not right in front of their eyeballs.Personally, I avoid touching the monorepo like the plague and stick to putting everything into flakes if possible.
Agreed, flakes are the way. Its just making them searchable hasn’t been easy/realistic until recently.
Also I feel like flakes are kind of tainted by always pulling in nixpkgs as a massive dependency chain. I think there should be a standard library separate from the packages, and nixpkgs.lib is 80% pure functions. So I’ve been working on making a “lib” flake that
I know it still won’t be practical to avoid depending on nixpkgs, but I think its a step towards breaking up nixpkg into organized/managable chunks.
I actually think I have seen this being discussed, but does not seem to be at the RFC level yet. Probably drowned in all of the flakes vs monorepo discussions.
One prerequisite for this that’s in the making are flake versions: https://github.com/NixOS/rfcs/pull/158
Oh cool! I didn’t know about that. Maybe I can work eith the other person who is doing that.