warning: download buffer is full; consider increasing the 'download-buffer-size' setting (won't work)

[roberth]

Describe the bug

I'm getting this warning during substitution and/or remote builds.
These machines have gigabit connections.

I'd need buffers of hundreds of megabytes to avoid this warning when the store paths are big.
(Now imagine a well connected small machine without swap)

Steps To Reproduce

nix-build -A nixosTests.nixos-rebuild-target-host

after a GC or on a fresh "generation" of Nixpkgs

Expected behavior

When the download buffer is full, continue downloading at the rate of the consumer, and don't complain about it.

Limit the buffer size to avoid jittery consumption, which is known to cause connections to drop when the pauses are too long.
I've already outlined this as a better solution in the PR that introduced this problem,

• Increase download buffer size and improve tarball import logging #11171

nix-env --version output

Additional context

Priorities

Add 👍 to issues you find important.

[roberth]

Also occurs when copying between local stores, with nix store repair --extra-substituters /some/path?trusted=true for example; probably also nix copy --to /some/path etc.

[malikwirin]

So what is the solution to no longer get this warning?

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-13-nix-team-meeting-minutes-202/57281/1

[boozedog]

i'm getting this too

[l0b0]

So what is the solution to no longer get this warning?

Looks like the default used to be 1 mebibyte until Nix 2.24. Since then you can also set download-buffer-size. That said, I still see this warning in CI with the current default of 67108864 (64 mebibytes).

Bumping to 500 MB seems to work for my builds.

[imincik]

Just hit this issue as well on my machine which is using the remote builder

buffer is full; consider increasing the 'download-buffer-size' setting
warning: download buffer is full; consider increasing the 'download-buffer-size' setting

[apholden]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-12-13-nix-team-meeting-minutes-202/57281/1

I just got this for the first time while running nix-shell. I followed the link to NixOS Discourse, and all I found was a link back to this issue. Could someone explain what I did wrong?

[davychhouk]

So what is the solution to no longer get this warning?

Looks like the default used to be 1 mebibyte until Nix 2.24. Since then you can also set download-buffer-size. That said, I still see this warning in CI with the current default of 67108864 (64 mebibytes).

Bumping to 500 MB seems to work for my builds.

Yep, it seems like it's going away after bumping it to 500MB.
You could do this by adding nix config in your configuration.nix file:

nix.settings.download-buffer-size = 524288000;

Then if you apply, you should be able to see the nix.conf (It should be at /etc/nix/nix.conf) updated with the download-buffer-size set to 524288000.

Then there should be no warning anymore.