I have 2 systems running Nix, and I'm afraid to touch them. I've already broken both of them enough that I had to reinstall from scratch in the past (yes yes - it's supposed to be impossible I know), and now I've forgotten most of it. In theory, Nix is idempotent and deterministic, but the problem is "deterministic in what way?" Unless you intimately understand what every dependent part is doing, you're going to get strange results and absolutely bizarre and unhelpful errors (or far more likely: nothing at all, with no feedback). Nix feels more like alchemy than science. Like trying to get random Lisp packages to play nice together.

Documentation is just plain AWFUL (as in: complete and technically accurate, but maddeningly obtuse), and tutorials only get you part of the way. The moment you step off the 80% path, you're in for a world of hurt, because the underlying components are just not built to support anything else. Sure, you can always "build your own", but this requires years of experiential knowledge and layers upon layers of frustration that I just don't want to deal with anymore (which is also why I left Gentoo all those years ago). And woe unto you if you want to use a more modern version than the distribution supports!

The strength of Docker is the chaos itself. You can easily build pretty much anything, without needing much more than a cursory understanding of the shell and your distro's package manager. Or you can mix and match whatever the hell you want! When things break, it's MUCH easier to diagnose and fix the problems because all of the tooling has been around for decades, which makes it mature enough to handle edge cases (and breakage is almost ALWAYS about edge cases).

Nix is more like Emacs: It can do absolutely anything if you have the patience for it and the deep, arcane knowledge to keep it from exploding in a brilliant flash of octarine. You either go full-in and drink the kool aid, or you keep it at arm's length - smiling and nodding as you back slowly towards the door whenever an enthusiast speaks.

So in theory Nix would be perfect. But it's not, because it's so different. Get a tool from a vendor => won't work on Nix. Get an error => impossible to quickly find a solution on the web.

Anyway, out of that frustration I've funded https://www.stablebuild.com. Deterministic builds w/ Docker, but with containers built on Ubuntu, Debian or Alpine. Currently consists of an immutable Docker Hub pull-through cache, full daily copies of the Ubuntu/Debian/Alpine package registries, full daily copies of most popular PPAs, daily copies of the PyPI index (we do a lot of ML), and arbitrary immutable file/URL cache.

So far it's been the best of both worlds in my day job: easy to write, easy to debug, wide software compatibility, and we have seen 0 issues due to non-determinism in containers that we moved over to StableBuild in my day job.

I've work many years on bare metal. We did (by requirement) acceptance tests, so we did need deterministic builds, before such thing had even a name, or at least before it was mentioned as much as nowadays.

Redhat has a lot of tooling around versioning of mirrors, channels, releases, updates, etc. But I'm so old that even foreman and spacewalk didn't exist, redhat satellite was out of the budget, and the project was migrating from the first versions of CentOS to Debian.

What I did was simply use DNS + Vhosts (dev, stage, prod + versions) for our own package mirrors, and bash+rsync (and of course, raid+backups), with both, CentOS and Debian (and our project packages).

So we had repos like prod/v1.1.0, stage/v1.1.0, dev/v1.1.0, dev/v2.0.0, dev/2.0.1, etc Allowing us to rebuild things without praying, backport bug fixings with confidence, etc

Feels old and simple, however I think it was the same problem/issue that people gets now (re)building containers.

If you need to be able to produce the same output from the same input, you need the same input.

BTW about stablebuild: nice project!

If you want reproducibility, the first step is to copy everything to a storage you control. Luckily, this is pretty cheap nowdays

Any source of that claim?

> or when you depend on some packages installed through apt - as you'll get the latest version (impossible to pin those).

Well... please re-read my previous comment - we do Java thing so we use any JDK base image and then we slap our distribution on top of it (which are mostly fixed-version jars).

Of course if you are after perfection and require additional packages then you can install it via dpgk or somesuch but... do you really need that? What about security implications?

Any tag like ubuntu:20.04 -> this tag gets overwritten every time there's a new release (which is very often)

https://hub.docker.com/r/nvidia/cuda -> these get removed (see e.g. https://stackoverflow.com/questions/73513439/on-what-conditi...)

Besides, if you really need utmost stability you can use image digest instead of tag and you will always get exactly the same image...

Very nice project!

not necessarily, eg snapshot.debian.org

> pinning on hashes is architecture dependent

can't you pin the multi-arch manifest instead?

I still like StableBuild for protection against package deletion, and mirroring non-pinnable deps

Huh, I have never had this issue with apt (Debian/Ubuntu) but frequently with apk/Alpine: The package's latest version this week gets deleted next week.

Here's a related discussion about reproducible builds by the Docker people, where they provide some more details: https://github.com/docker-library/official-images/issues/160...

Free …

Number of Users 1

Number of Users 15GB

Is that a mistake or if not can you explain please?

https://www.stablebuild.com/pricing

Free => Access to all functionality, 1 user, 15GB traffic/month, 1GB of storage for files/URLs. $0

Pro => Unlimited users, 500GB traffic included (overage fees apply), 1TB of storage included. $199/mo

Enterprise => Unlimited users, 2,000GB traffic included (overage fees apply), 3TB of storage included, SAML/SSO. $499/mo

But I also agree with all the flaws of Nix people are pointing out here.

Documentation is often just plain erroneous, especially for the new CLI and flakes, not even edge cases. I remember spending some time trying to understand why nix develop doesn't work like described and how to make it work like it should. I feel like nobody ever actually used it for its intended purpose. Turns out that by default it doesn't just drop you into the build-time environment like the docs claim (hermetically sealed with stdenv scripts available), it's not sealed by default and the commandline options have confusing naming, you need to fish out the knowledge from the sources to make it work. Plenty of little things like this.

>In theory, Nix is idempotent and deterministic

I surely wish they talked more about edge cases that break reproducibility. Things like floating point code being sensitive to the order of operations with state potentially leaking from OS preemption, and all that. Which might be obvious, but not saying obvious things explicitly is how you get people shoot themselves in the foot.

That’s profoundly cursed and also something that doesn’t happen, to my knowledge. Unless the kernel programmer screwed up, an x86-64 FPU is perfectly virtualizable (and I expect an AArch64 FPU too, I just haven’t tried). So it doesn’t matter where preemtion happens.

(What did happen with x87 is that it likes to compute things in more precision than you requested, depending on how it’s configured—normally determined by the OS ABI. Yet variable spills usually happened in the declared precision, so you got different results depending on the particulars of the compiler’s register allocator. But that’s still a far cry from depending on preemption of all things, and anyway don’t use x87.

Floating-point computation does depend on associativity, in that nearestfp(nearestfp(a+b)+c) is not the same as nearestfp(a+nearestfp(b+c)), but the sane default state is that the compiler will reproduce the source code as written, without reassociating things behind your back.)

We don't need to be looking for such contrived examples actually, nixpkgs track the packages that fail to reproduce for much more trivial reasons. There aren't many of them, but they exist:

https://github.com/NixOS/nixpkgs/issues?q=is%3Aopen+is%3Aiss...

Less than a couple of thousand packages are reproduced. Nobody has even attempted to rebuild the entirety of the nixpkgs repository and I'd make a decent wager on it being close to impossible.

https://grahamc.com/blog/erase-your-darlings/

Inside my user account, I don’t bother. I don’t like Home Manager.

https://elis.nu/blog/2020/05/nixos-tmpfs-as-root/

That way it doesn't even need to bother with resetting to ZFS snapshots, instead it just wipes root on shutdown and reconstructs it in RAM on reboot.

Then, optionally, with some extra work you can put /home in tmpfs too:

https://elis.nu/blog/2020/06/nixos-tmpfs-as-home/

That setup uses Home Manager, so maybe it's not for you, but worth mentioning if we're talking about making all state declarative and reproducible. You have to use the Impermanence module and set up some soft links to permanent home folders on different drive or partition. But for making all state on the system reproducible and declarative, this is the best way afaik.

That said I'll just mention that ZFS support on NixOS is like nothing else I've seen in Linux. ZFS is like a first-class citizen on NixOS, painless to configure and usually just works like any other filesystem.

https://old.reddit.com/r/NixOS/comments/ops0n0/big_shoutout_...

Nix Doc are horrible but I've found that ChatGPT4 is awesome at troubleshooting Nix issues.

I feel like 90% of the time I run into Nix issues, it's because I decided to do something "Not the Nix way."

That has been the case for as long as I can remember. I gave up on Nix about 5 years ago because of it, and apparently not much has changed on that front since then..

I've been a nixos user for years and I generally had the opposite problem: the latest of the package you want is not available but hey here's a version from months ago - or just build it yourself (which is not hard, oftentimes updates work fine with no build change, you just point at a different version).

Also rebuilding everything at every update take forever (I had a few nix-shells with ai dependencies that would take hours to upgrade).

I love the concept of nix but I'm back to Arch, binary bleeding edge packages and AUR for less supported stuff.

Could you mention a bit about how they broke? I'm curious to see how that state looks, as from my perspective switching to a previous configuration seems to cover everything.

That depends whether you are okay with chaos.

It appears that you are, so it is suitable tool for you. Choose the right tool for the right job.

---

Docker is a poor choice for people who are interested in deterministic/reproducible builds.

I’ll use docker, I like docker, but I can see the point of how it’s not necessarily advantageous if stability is your main goal.

I think that might have changed with the release of flox: https://flox.dev, it’s basically seamless (and that’s not surprising coming from DE Shaw).

Nix doesn’t really make sense without flakes and nix-command, those things are documented as experimental and defaulted off. The documentation story is getting better, but it’s not there. nixlang is pretty cool once you learn it well, but it’s never going to be an acceptable barrier to entry at the mainstream. It’s not really the package manager it’s advertised as, nix-env -iA foo is basically never what you want. It’s completely unsurprising that it’s still a secret weapon of companies with an appetite for bleeding-edge shit that requires in-house expertise.

flox addresses all of this for the “try it out and immediately have a better time” barrier.

Nix/NixOS or something like it is going to send Docker to the same dustbin Subversion is in now, but that’s not going to happen until it has the GitHub moment, and then it’ll happen all at once.

Most of the complaints with Nix in this thread are technically false, but eminently understandable and more importantly (repeat after me Nix folks): it’s never the users fault.

I’m aware that I’m part of a shrinking cohort who ever knew a world without git/GitHub, so I know this probably sounds crazy to a large part of the readership, but listen to Linus explaining to a room full of people who passed the early Google hiring bar why they should care about a tool they feel is too complicated for them:

https://youtu.be/MjIPv8a0hU8?si=QC0UnHXRdMpp2tI4

If you drop me an email at [email protected] (I also just joined your slack) I’d love to give my feedback on this or that nitpick.

But overall, well done friends, very very nice stuff.

This is true for git, but not so true yet for Nix, so I'm not sure a GitHub-like moment helps.

But anywhere else you just brew/apt/rpm install whatever and nix develop —impure, which is easier than most intermediate git stuff and plenty of beginner stuff. git and Nix are almost the same data structure if you start poking around in .git or /nix/store, I might not understand what you mean without examples.

But all my guesses about what you might mean are addressed well by flox.

Nix is absolutely the best tool for what I want to achieve but it has those dark forsaken corners that just suck your soul out dry.

I love it but sometimes it feels like being a Morty on Rick’s adventure to the compilerland.

Things are straight forward on Linux. You build your binary, place in a docker container and you are done. The nix code will also be straight forward. If you can build your code, then creating a container is just one more operation away.

Unfortunately docker requires Linux binary and you are on Mac. So the docker desktop actually runs a Linux VM and performs all operations on it, abstracting this away from you.

Nix doesn't do that and you have two options:

1. Do cross compilation, the problem is that for this to work you need to be able to cross compile down to glibc, the problem is that while this will work for most community used dependencies you might get some package where the author didn't put effort making sure it cross compile. To make things worse the Hydra that populates standard caches that nix uses, doesn't do cross compile builds, so you will run into lengthy processes that might potentially end with a failure.

2. You can have a Linux builder, that you add to your Mac and configure to send build jobs for x86_64-linux to that builder. Now you could have a physical box, create a VM, or even have a NixOS docker container (after all docker ion Mac runs inside of the VM).

The #1 seems like the proper way, while #2 is more of a practical way.

I think you are running into issues, because you're likely trying #1, and that requires a lot of experience not only with Nix, but also with cross compiling. I wish Nix's Hydra would also build Darwin to Linux cross compilation as that would not only provide caches, but also help making sure the cross compilation doesn't break, but that would also increase costs for them.

I think you should try the #2 solution.

Edit: looks like there might have been an official solution to this problem: https://ryantm.github.io/nixpkgs/builders/special/darwin-bui... I haven't used it yet.

I'm using `clang` from `pkgs.pkgsCross.musl64.llvmPackages_latest.stdenv` to cross-compile Rust binaries from ARM macos to `x86_64-unknown-linux-musl`. It _works_, but every time I update my `flake.nix` it rebuilds *the entire LLVM toolchain*. On an M2 air, that takes something like 4 hours. It's incredibly frustrating and makes me wary of updating my dependencies or my flake file.

The alternative is to switch to dockerized builds but:

1) That adds a fairly heavyweight requirement to the build process

2) All the headache of writing dockerfiles with careful cache layering

3) Most importantly, feels like admitting defeat.

I pass the output of each stage of a toolchain as a dependency to the next stage. By chaining the stages, changes made to a single stage only require a rebuild of each succeeding stage. The final stage is the default of the flake, so you can easy get the complete package.

In addition, I can debug along the toolchain by entering a single stage env with nix develop

Not sure if this is the most optimal way, but it appears to work in modularizing the rebuild.(using 23.11)

This sort of twenty-minute adventure?

The rougher part is the speed of it; it's a one-two punch between the hardware & the fact that Docker has to emulate a Linux VM.

If something does break, rollbacks are free and an integral part of Nix.

Before someone asks. I've been using macOS for a long time. I've never seen remnants like this from a program. Sure, there are often directories left in ~/Library/Application Support/, but this was more than that. Unfortunately, I didn't write down the details, but I ran across the bits in at least 3-4 places.

- ~/.orbstack

- Docker context that points to OrbStack (for CLI)

- "source ~/.orbstack/shell/init.zsh" in .zprofile/bash_profile (to add CLI tools to PATH)

- ~/.ssh/config (for convenient SSH to OrbStack's Linux machines)

- Symlinks to CLI tools in ~/.local/bin, ~/bin, or /usr/local/bin depending on what's available (to add CLI tools to existing shells on first install — only one of these is used, not all)

- Standard macOS paths (~/Library/{Application Support, Preferences, Caches, HTTPStorages, Saved Application State, WebKit})

- Keychain items (for secure storage)

- ~/OrbStack (empty dir for mounting shared files)

- /Library/PrivilegedHelperTools (to create symlinks for compatibility)

Not sure what the best solution is for people who don't use Homebrew to uninstall it. I've never liked separate uninstaller apps, and it's not possible to detect removal from /Applications when the app isn't running.

And cough since we’re at with - did you consider Nixpkgs distribution?

I’m slowly moving deeper and deeper into ecosystem and use Home Manager for utilities that I use often (and use nix shell/nix run for one offs). Some packages are strictly GUI and while they aren’t handled flawlessly (self-updaters) it’s nice to have them on a single list.

Yet based on your list it’s a definitely a nixventure…

And yet, you are the only(?) one with that knowledge, so the alternative seems to be replying to HN threads with a curated list of things that a user must now open iTerm2 and handle by themselves. Something, unless I'm mistaken, that computers are really good at doing (Gatekeeper and privilege elevation nonsense aside)

Even just linking to the zap portion of your brew cask could go a long way since it would be the most succinct manifest if I correctly understand what it does

But unfortunately cross compiling quickly broke when I started doing mild customization (and one reasons I’m doing this is a complex setup that’s very sensitive to version changes).

In the end solution was to “simply” get darwin.linux-builder up but that pulled a lot of weight behind it.

It works, but it’s not the first time I spent my time on nix-ventures.

Why is docker bad at this? In order to enjoy the caching benefit, each time you build a docker image you want it to output as much existing layers as possible. So running apt-get install python3 today should result in the exact same layer as yesterday, if there are no new updates. But this requires the all the files to be exactly the same, including the metadata like creation time. As docker layers are cached by hashing the files.

Now, Nix already does storing dependencies by hash. So the layers will always be the same with the same version and same configuration.

The Dockerfile format imposes a hierarchical relationship between layers. This quickly becomes very annoying, since dependencies usually form dependency graphs, not dependency trees.

Alternative tools, like nix (probably bazel too), are not bound in the same way. They can achieve fine grained caching by mapping their dependency graph to docker layers, which is something that can not be expressed with a Dockerfile.

The final result need not be. You can build a bunch of things then merge the results in a final stage without any hierarchy (this is "COPY --link" in a Dockerfile).

With nix, re-usability is very high. It's a function that is very baked in at very low levels of it's design. This comes with up front complexity but getting to these reusable layers is basically forced.

Docker is very simple and often touts reusable layers, but in practice is not. Unless you tackle that complexity.

Making reproducible and reusable content takes effort. Other tools are not designed for that. As a result the getting to the same state requires a similar amount of complexity. Worse, with docker you can never be sure that you actually succeeded in your goal of reproducibility.

An analogy could be rust. Rust has up front complexity, but tackling that complexity gives confidence that memory safety and concurrency primitives are done correctly. It's not that C _can't_ achieve the same runtime safety, it's just requires a lot more skill to do correctly; and even then memory exploits are reported on a near daily basses for very popular and widely used libraries.

Complex problems are complex. And sooner or later you'll need to face that complexity.

What you are describing is chaining a bunch of commands together. Yes, this forms a dependency chain stored in separate layers and is part of the cache chain.

Nix suffers the exact same problems with reproducibility. The thing it provides is the toolchain of dependencies that are reproducible. Docker does not provide your dependencies.

If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results.

EDIT: Fixed a sentence I apparently got distracted while writing and didn't complete (about layer caching).

> The thing it provides is the toolchain of dependencies that are reproducible. [...] If the inputs change then so does the output. If the output itself is not reproducible (like, say an artifact with a build-time embedded in it) then you have something that is inherently not reproducible and two people trying to build the same exact nix package will have different results.

There are no guarantees they are reproducible. The only guarantees Nix gives you is that the build environment is the same which allows you to make some claims about the system behaving the same way. But they are certainly no guarantees about artifacts being bit-by-bit identical.

The content address of a docker image is a json blob (referencing other objects).

The content address of a Dockerfile "RUN" command is the content address of what came before it and the command being run.

In the case of Nix it's addressed by the input. Not the content of the build. It's an important distinction and one Nix also makes.

https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3...

But doing this is going to give you a slight headace as most of the package repository in Nix is not checked for reproducible builds and there are no way to guarantee the hashes are actually static.

We are saying the same thing here, I'm just trying to point out this is exactly how docker build works, but rather it is more about what you are willing to put into your docker build.

Docker layers are completely independent from each other. A docker layer is a sha256 sum of that layer. Separately there is an image manifest, which is also fetched with a sha256 of that manifest, states the order of the layers and at runtime those layers are stacked up on each other.

With docker, there is no explicit dependency chain. A layer is just a tarball or some JSON. Some tooling can take advantage of this fact. How nix builds docker images takes advantage of this.

Nix on the other hand, the output hash is not tied to the output hash, because the output hash is irrelevant to how it was produced. You also cannot know in advance the output hash of something. IE. If I do say "echo foo > bar.txt", I cannot know the sha256 sum of bar.txt until the code runs. But before the code runs I can know the hash all the inputs that will create bar.txt.

This fundamental difference means two builds, executing the same code can share the outputs. Provided that the build environment is trust worthy.

While docker build can/does output OCI images, that is only an output. How that output comes to be is not the output itself, same as the nix side the article is talking about.

I see the confusion now. The nix image builders OCI layer's contents are _only_ nix store paths which _do_ include the input.

Nix store paths are are guaranteed to never overlap each other, and such the order of layering them in the docker manifest does not matter. But the docker layers are just tar balls of nix store paths. Each layer in the image has no dependence on previous or future layers at all. It is just one or more nix store paths.

When said caching, I meant on the nodes that run the containers. With Nix you can also update only one layer, while keeping the other layers the same.

And sorry for gradle2nix, I'm working on an improvement that's less of a hack.

Don't be. Thanks for your work. Excited to learn about the improvement. Can you tell more about what you have in mind?

It looked something like this: https://gist.github.com/takeda/17b6b645ad4758d5aaf472b84447b...

So what I did was:

- link everything with musl

- compile python and disable all packages that I didn't use in my application

- trim boto3/botocore, to remove all stuff I did not use, that sucker on it's own is over 100MB

The thing is what you need to understand is that the packages are primarily targeting the NixOS operating system, where in normal situation you have plenty of disk space, and you rather want all features to be available (because why not?). So you end up with bunch of dependencies, that you don't need. Alpine image for example was designed to be for docker, so the goal with all packages is to disable extra bells and whistles.

This is why your result is bigger.

To build a small image you will need to use override and disable all that unnecessary shit. Look at zulu for example:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...

you add alsa, fontconfig (probably comes with entire X11), freetype, xorg (oh, nvm fontconfig, it's added explicitly), cups, gtk, cairo and ffmpeg)

Notice how your friend carefully extracts and places only needed files in the container, while you just bundle the entire zulu package with all of its dependencies in your project.

Edit: tadfisher seems to be more familiar with it than me, so I would start with that advice and modify code so it only includes a single jdk. Then things that I mentioned could cut the size of jdk further.

Edit2: noticed another comment from tadfisher about openjdk_headless, so things might be even simpler than I thought.

- it contains information how to actually build the application

- how to set up a dev environment

- how to build application with musl

- how to build application with glibc

- how to build python with only with expat, libffi, openssl, zlib packages

- how to take botocore and patch it up to only have cloudformation, dynamodb, ec2, elbv2, ssm, sso, sts clients

Try to get all of that into a single Dockerfile and see how complicated mess you end up with.

The actual docker configuration is here:

https://gist.github.com/takeda/17b6b645ad4758d5aaf472b84447b...

It might be still confusing to you at first, as you're used to list of incremental steps how to get to the final result, while this description instead is declarative (you're describing not the steps to do, but what the final image should be).

It's basically comparing bash script with bunch of "aws" CLI invocations to a terraform or cloudformation file.

I wouldn't expect a layman to be able to grok that file. That's fine though - it's not for laymen.

> I wouldn't expect a layman to be able to grok that file. That's fine though - it's not for laymen.

This is the kind of comment that makes me want to stay far, far away from Nix and the Nix "community".

https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...

What that file is doing, is building a package, and it essentially is a combination of what Makefile and what RPM spec file does.

I don't know if you're familiar with those tools, but if you aren't it takes some time to know them enough to understand what is happening. So why would be different here?

I actually pointed out that mkDerivation is something helpful to learn - that's one thing I wish someone made me sit and learn when I first got exposed to Nix. It unlocks a lot.

I agree with your assertion regarding the language though. I think nix-lang makes it harder to get into Nix.

https://discourse.nixos.org/t/how-to-create-a-docker-image-w...

Wonder if there is a good term for this. I have been jokingly referring to this as 'headed' and headless as 'beheaded'.

I've switched to using a headless OpenJDK build from Nixpkgs as a baseline instead of Zulu, to remove all the unnecessary dependencies on GUI libraries. Then I've used pkgs.jre_minimal to produce a custom minimal JRE with jlink.

The image size now comes out to 161MB, which is slightly larger than the demo_jlink image. This is because it actually includes all the modules required to run the application, resulting in a ~90MB JRE. The jdeps invocation in Dockerfile_jlink fails to detect all the modules, so that JRE is only built with java.base. Building my minimal JRE with only java.base brings the JRE size down to about 50MB, the resulting (broken) container image is 117MB according to Podman.

I've also removed the erroneous copyToRoot from your call to dockerTools.buildImage, which resulted in copying the app into the image a second time while the use of string context in config.Cmd would have already sufficed.

I've also switched to dockerTools.buildLayeredImage, which puts each individual store path into its own image layer, which is great for space scalability due to dependency sharing between multiple container images, but won't have an impact for this single-image experiment.

This is mostly a JRE size optimization challenge. The full list of dependencies and their respective size is as follows:

  /nix/store/v27dxnsw0cb7f4l1i3s44knc7y9sw688-zlib-1.3                            125.6K
  /nix/store/j6n6ky7pidajcc3aaisd5qpni1w1rmya-xgcc-12.3.0-libgcc                  139.1K
  /nix/store/l0ydz31lwa97zickpsxj2vmprcigh1m4-gcc-12.3.0-libgcc                   139.1K
  /nix/store/a3n1vq6fxkpk5jv4wmqa1kpd3jzqhml9-libidn2-2.3.4                       350.4K
  /nix/store/s5ka5vdlp4izan3nfny194yzqw3y4d1z-lcms2-2.15                          445.3K
  /nix/store/a5l3w6hiprvsz7c46jv938iij41v57k6-libjpeg-turbo-2.1.5.1                 1.6M
  /nix/store/r9h133c9m8f6jnlsqzwf89zg9w0w78s8-bash-5.2-p15                          1.6M
  /nix/store/3dfyf6lyg6rvlslvik5116pnjbv57sn0-libunistring-1.1                      1.8M
  /nix/store/a3zlvnswi1p8cg7i9w4lpnvaankc7dxx-gcc-12.3.0-lib                        7.5M
  /nix/store/657b81mfpbdz09m4sk4r9i1c86pm0i8f-app-1.0.0                            19.0M
  /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44                        28.8M
  /nix/store/b1fhkmscb0vff63xl8ypp4nsc7sd96np-openjdk-headless-minimal-jre-21+35   91.4M

https://github.com/max-privatevoid/hackernews-docker-challen...

I would be very interested to know where the difference is; is nix including things it doesn't need to? Is the non-nix build not including things it should?

As stated above, I havent done any trimming on the resulting image, so There's too many stuff in the image.

If you're already using Docker but want to gradually adopt Nix, there is an alternative approach outlined by this talk: https://youtu.be/l17oRkhgqHE. Instead of migrating both the configuration AND container building to Nix straight away, you can keep the Dockerfile to build the nix configuration.

The biggest downside is that you don't take advantage of layers at all, but the upside is that you can gradually adapt your Dockerfiles, and reuse any Docker infrastructure or automation you already use.

[1] https://mitchellh.com/writing/nix-with-dockerfiles

But... is a lot of that irreproducibiity (apologies for that word) because there's no guarantee one of the docker layers will be available?

And... does Nix have some guarantee to the end of the universe that package versions will stay in the repository?

But to briefly answer your specific questions: Docker files are commonly not reproducible because they contain arbitary stateful commands like `apt-get update`, `curl`, etc. For a layer with these kinds of commands to be reproducible you would need a mechanism to version and verify the result.

Nix provides such a mechanism, and a community package repository with versioned dependancies between packages. These are defined in a domain specific language called Nix (text files) and kept into a git repository. This should be familiar if you've used a package manager with lock files before.

You can guarentee the package version will stay in the repository by pinning your build to an exact commit hash in the repository.

and the dx is still pretty bad IMO.

for example, i prefer devbox DX just because i can add pkg like this `devbox add [email protected]`.

also, looking at 120 lines flake.nix. it's not exactly "easier"

https://github.com/Xe/douglas-adams-quotes/blob/main/flake.n...

1. Defines a Go binary (i.e., how to build it)

2. Defines a Docker image that uses said Go binary as an entry point

3. Defines a NixOS module that creates a systemd service that runs the Go binary (only relevant on NixOS)

4. Defines a NixOS test for the module that ensures that the NixOS module actually creates a systemd service that runs the Go binary as expected. The NixOS test framework is actually quite impressive - tests run in a QEMU VM that also runs NixOS :)

Note that only (1) and (2) are relevant to the linked article (+ some of the surrounding boilerplate).

[0] https://std.divnix.com

It doesn't add kitschy terms like "Cell" and "growOn", it's just a way to standardize attribute names according to where things live in the filesystem.

So in their repo, /foo/bar/baz.nix has the attribute path depot.foo.bar.baz

I will say that to understand how it works you need to have a solid grasp of the nix language. But once established I think it's a pattern that noobs could learn in a very quick and superficial way.

[0]: https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md

[0] https://code.tvl.fyi/about/tvix

I've even used Nix a little bit (though early days, before flakes) and it makes absolutely no sense to me.

Then this std background of "We bring clarity, clear mental picture, manageable Nix projects throughout the lifecycle." appealed a lot.

If the target audience is limited to those already highly experienced (and yet frustrated) with Nix then I guess it's fine.

(PS. This cell breakdown reminded me a bit of Atomic Design for front-end UI to make that easier.)

I don't think most people's flakes are more complex than the alternative (which would be, I don't know, a bunch of different script, maybe some ansible playbooks?) but it is a bit daunting when all that complexity is wrangled into a single abstraction.

Because it certainly looks like the things you listed are separate/orthogonal and should be in separate modules/files.

Having many years of Java experience this is the reason why I stick to Maven (not moving to Gradle) - it is opinionated and strongly encourages fine grained modularisation.

Having said that - Java is in general IDE targeted language and once you have an IDE - many small files is not an issue anymore.

Nix absolutely allows that, to the point where I'm surprised that the linked example doesn't separate them. Most of my flakes have a flake.nix that's just a thin wrapper around a block that looks like

    devShell = import ./shell.nix { inherit pkgs; };
    defaultPackage = import ./default.nix { inherit pkgs; };
    ...

I am aware of flake-parts that are supposed to offer that but the ecosystem of flake-parts is on the smaller side of things...

You can pull in arbitrary code from pretty much anywhere as an input

https://ryantm.github.io/nixpkgs/languages-frameworks/maven/

nix is a build tool a similae way that docker is a build tool. You define build scripts that call the tools you already use. The major difference is that a docker file gives you no way tl be sure you can reproduce that build in the future. A flake on the other hand gives you a high degree of trust.

What we have today is that there is a multitude of dependency managers and - what's worse - all of them are _also_ build tools targeted at specific language.

Nix has a unique position because it is not language specific. Where it is lacking is missing standards and reusable libraries that would simplify common tasks.

I am comparing to Maven because I am looking for multi-platform Maven alternative. There is Bazel but its dependency management is non-existent. There is Buck2 which is great in theory but the lack of ecosystem makes it a non-starter.

Nix is the only contender in the space that offers almost everything and has a chance to become a de-facto standard of software delivery thanks to this.

What's missing though is... easy to use canned solutions similar to maven plugins.

EDIT: grammar

Nix has composable, reusable, and shareable functions. Nix is a full, be awkward, programming language. You'll find functions and flakes for nearly everything you might want to do. An example of one that is more plugin like is sops-nix.

Though have never used maven or maven plugins, I may be missing your overall point.

Want a packaged spring boot application? There is a plugin for that. Want to package it in a container image? Just add a plugin dependency. Want to build an RPM or deb package? Add another two plugin dependencies. All various artifacts are going to be uploaded to a repository and made available as dependencies.

Missing a specific plugin? You can easily implement it in Java as a module and use it inside your project (and expose it as a standalone artifact as well).

I can’t find anything similar in Nix ecosystem.

Having a language allowing for this is not the same as having solutions already available.

For example, `dockerTools` provides different functions like creating an image and other things. There is a module of fetchers, functions that retrieve source files from GitHub and other sources.

But I don't think there are many language-specific functions, like the ones you are describing. I can't think of any, except for the ones that build a Nix package from a language-specific package.

https://flake.parts/

or

https://github.com/nix-community/flakelight

Their aim is to create an ecosystem of reusable Nix libraries. But it is tiny.

[0]: https://flakehub.com/flakes

Writing a Maven plugin (ie. a reusable piece of configuration management logic) is easy because you can use any library from the vast Java ecosystem (just add a library as a dependency of your plugin).

Doing the same in Nix requires recreating these libraries in Nix.

Looks like Maven might simply be the right choice...

This example is a "let's do a trivial thing the way you'd do a big serious thing". If you wanted to treat it as a one-off, I'm sure you could cut it down to 40 lines or so.

Hrmmmm...this is such a TOUGH decision.

cramming everything into one RUN line to save space in a layer.

I really wish instead of:

  RUN foo && \
      bar && \
      bletch

  LAYER
  RUN foo
  RUN bar
  RUN bletch
  LAYER

maybe even during development you could do:

  docker build --ignore-layer .

  docker build .

[0]: https://docs.docker.com/reference/dockerfile/#shell-form

[1]: https://www.docker.com/blog/introduction-to-heredocs-in-dock...

[2]: https://docs.podman.io/en/latest/markdown/podman-build.1.htm...

Docker should address it one day, but until then, we just have to do it in real world scenarios.

It's not always best or most efficient to squash everything into one giant layer.

That binary + config file are effectively as close as we're going to get to a "flat pack" on linux.

Im not sure what is the forest and where are the trees but this example shows exactly what we have lost sight of.

One issue that really bugged me was to build multi-arch images, it actually wants to execute stuff as the other architecture, and only supports using qemu with hardware virtualization for that. My build machine (and workstation) is a VM, so I don't have that. I do have binfmt-misc, though, so if you just happened to fork and exec the arm64 "mkdir" to "mkdir /tmp", it would have worked. Of course, this implementation is a travesty when docker layers are just tar files, and you can make the directory like this:

    echo "tmp uid=0 gid=0 time=0 mode=0755 type=dir" | bsdtar -cf - @-

Every time I try nix, I feel like it's just a few months away from being something I'd use regularly. nixpkgs has a lot of packages, everything you could ever want. They all install OK onto my workstation. But "I need bash, python, build-essential, and Bazel" doesn't seem like something they're targeting the docker image builder at. I guess people just want to put their go binary in a docker image and ... you don't need nix for that. Pull distroless, stick your application in a tar file, and there's your container. (I personally use `rules_oci` with Bazel... but that's all it does behind the scenes. It just has some smarts about knowing how to build different binaries for different architectures and assembling and image index yaml file to push to your registry.)

You should be able to cross compile binaries for other architectures without actually running them. As long as the package's build files support it of course.

> and only supports using qemu with hardware virtualization for that

That doesn't sound right. You can use qemu for architectures that be only software emulated too.

The minimal example is discussed here:

https://discourse.nixos.org/t/how-do-i-get-a-shell-nix-with-...

I don't want to say it should be as simple as using pkgCross (https://nix.dev/tutorials/cross-compilation.html), but... are some specific issues with the usual process that you're running into?

  >  I spent a half a day or so relatively recently trying to build our CI base image with Nix (at the recommendation of our infra team), but it was huge, and some stuff didn't work because of linking issues.

[1] https://hub.docker.com/r/nixos/nix/tags

I think Nix really needs some articles outlining how to play well and smoothly transition from an existing system piece by piece.

I never saw went being used so im a bit confused where that came into play for you

Frequently it's not packaged or some version was but the upstream build system changed an you have to do it yourself.

If you don't either benefit significantly from Nix's upsides (which most projects really don't), or you don't enjoy this and do it for fun, it's not a productive use for your time. If you can convince your employer to pay you to shave that yak, it's a net waste.

Write your pipelines and more in languages you already use. It unlocks the advanced features of BuildKit, like the layer graph and advanced caching.

nit: this post makes a point about deterministic builds and then uses "latest" for source image tags, which is not deterministic. I've always appreciated Kelsey's comment that "latest" is not a version

They also recently released their "github actions" replacement https://news.ycombinator.com/item?id=39550431> but holy hell their documentation is just aggressively bad

Anything in particular you'd like to see improved in the docs?

[1] https://nix.dev/tutorials/nixos/building-and-running-docker-...

What you linked is the equivalent of:

FROM scratch

COPY hello-world /hello-world

Of course that's small (hello-world is statically linked). Try to add coreutils (or any other small package) and you'll see what I mean. In my experience the size of a Docker image built with some nix packages is greater than the Debian counterpart. I don't know why though.

However I'll give you that it could be smaller in more complex examples. For example glibcLocales is for all locales which is quite chunky but your application only needs one locale.

Default is none (i.e. like "FROM scratch" in a Dockerfile); you can specify a baseImage if needed, but I haven't had to yet. It works by copying parts of the nix store into the image as needed, but see also below.

> wouldn’t the image size be large if the glibc is copied over again

The original Nix docker-tools buildImage did suffer from poor reuse of common dependencies. Docker already has a way to reuse parts of images (e.g. if you build 7 images where the first N lines of a Dockerfile are the same, the 7 images will use a shared store for the results of running the first N lines). There are several backends for Docker storage that accomplish this in various ways (e.g. FS overlays, tricks with ZFS/btrfs snapshots).

Nix docker-tools now has a "buildLayeredImage" that uses this ability of Docker to share much of the storage for the dependencies, so if you build several images that all rely on glibc, you only pay the cost of storing glibc in docker once.

Docker build are deterministic and easily reproductible, you use a tagged image, that's it, it set in stone.

The 0.01% of Dockerfile that don't work, what does it even means, what does not work?

The other thing is about that buildGoModule module so now you need somehow a third party tool to use or build Go in a Docker image, when using a Dockerfile you just use regular Go commands such as go build and you know exactly what is going on and what args you use to build the binary.

As for the thing about using Ubuntu 18 which is out of date and not finding it, most orgs have docker image cache especially since docker hub closed access to large downloads, but more importantly there is a reason that's its not there anymore, it's not secure to use it, it's like wanting to use the JVM 6, you should not use something that is out of date security wise.

But you can achieve the same result if you use similar techniques with a bash script.

Just create a snapshot of the OS repo, so apt/dnf/opkg/ etc will all reproduce the same results.

Make sure _any_ scripts you call don't make web requests. If they do you have the validate the checksums of everything downloaded.

Have no way to be sure that npm/pip/cargo's package build scripts are not actually pulling down arbitrary content at build time.

You seem to be comparing 2 different things.

There are still ways that package will not be fully reproducible, for example if it uses rand() during build, Nix doesn't patch that, but stuff like that is fortunately not common.

Using this approach you get something reproducible. Using apt-get in a docker file is an antipattern

We have 2-3 service updates a day from a dozen engineers working asynchronously — and we allow non-critical packages to float their versions. I’d say that successfully applies a fix/security patch/etc far, far more often than it breaks things.

Presumably we’re trying to minimize developer effort or maximize system reliability — and in my experience, having fresh packages does both.

So what’s the harm, precisely?

Regardless.... I can give a few reasons that it matters, off the top of my head:

1) Debugging: It can make debugging more difficult because you can't trace your dependencies back to the source files they came from. To make it concrete, imagine debugging a stack trace but once you trace it into the code for your dependency, the line numbers don't seem to make any sense.

2) Compliance: It's extremely difficult to audit what version of what dependency was running in what environment at what time

3) Update Reliability: If you are depending on mutable Docker tags or floating dependency installation within your Dockerfile, you may be surprised to discover that it's extremely inconsistent when dependency updates actually get picked up, as it is on the whim of Docker caching. Using a system that always does proper pinning makes it more deterministic as to when updates will roll out.

4) Large Version Drift: If you only work on a given project infrequently, you may be surprised to find that the difference between the cached versions of your mutably-referenced dependencies and the actual latest has gotten MUCH bigger than you expected. And there may be no way to make any fixes (even critical bugfixes) while staying on known-working dependencies.

Nix is the toolchain, which of course has its advantages.

Both are mostly running well-controlled shell commands and hashing their outputs, while tightly controlling what's visible to what processes in terms of the filesystem. The difference is that nix is just enough better at it that it's practical to rebase the whole ecosystem on top of it (what you refer to as a "toolchain") whereas Docker is slightly too limited to do this.

nix provides the toolchain of reproducible artifacts... and then uses that toolchain to build a graph of content addressable inputs in order to produce a content addressable output.

So yes they are very different, but not in the way you are describing. Using nix, just like using docker, cannot guarantee a reproducible output. Reproducible outputs are dependent on inputs. If your inputs change (and inputs can even be a build timestamp you inject into a binary) then so does your output.

This is achieved by building inside of a chroot, with blocked network access etc. Only the dependencies that are explicitly listed in the derivation are available.

It doesn't break as you scale. If you don't need that, then keep using Docker. (Personally, for me "scale" starts at "3 PC's in the home", so I eventually switched all of them to NixOS. I don't have time to babysit these computers.)

> Docker build are deterministic and easily reproductible

No, they definitely aren't. You don't really want to go down this rabbit hole, because at the end you realize Nix is still the simplest and most mature solution.

Building an image using nix solves many problems regarding not only reproducible environments that can be tested outside a container but also fully horizontal dependency management where each dependency gets a layer that's not stacked on one another like a typical apt/npm/cargo/pip command. And I don't have to reverse engineer the world just to see what files changed in the filesystem since everything has its place and has a systematic BOM.

And that all relies on discipline. Just like using a dynamically typed programming language can in theory have no type errors at run time, if you are careful enough.

    FROM base-image@e70197813aa3b7c86586e6ecbbf0e18d2643dfc8a788aac79e8c906b9e2b0785
    RUN pkg install foo=1.2.3 bar=2.3.4
    RUN git clone https://some/source.git && cd source && git checkout f8b02f5809843d97553a1df02997a5896ba3c1c6
    RUN gcc --reproducible-flags source/foo.c -o foo

And then there's the Dockerfiles that just `RUN git clone repo.git` and run with whatever happened to be in the latest commit at the moment...

The only reason nix isn't dog slow is that it has really strong caching so it doesn't have to build everything from source.

You can have as many "FROM"'s as you want. "FROM scratch" along with "ADD" is also valid (for non-image dependencies).

From there you do not need to copy things, you can mount the reference into another stage directly.

Also, this is the Dockerfile format, the underlying build API's are _far_ more powerful than what Dockerfile exposes.

A Docker FROM is essentially the equivalent of a dependency in nix... but each RUN only has access to the stuff that comes from the FROM directly above it plus content that has been COPY-ed across (and COPY-ing destroys the ability to share data with the source of the COPY). For Docker to have a similar power to nix at building Docker images, you would need to be be able to union together an arbitrary number of FROM sources to create a composed filesystem.

People use the Dockerfile format because it is accessible. You can still use "docker build" with whatever format you want, or drive it completely via API where you have the full power of the system.