archive builds on zfs /tank #4
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This currently doesn't work very well... https://github.com/NixOS/hydra/issues/796
Hydra is serving off the ZFS tank now, and things are mostly working, including the download links. So we now have 13TB of space to archive builds :)
Two issues remain:
nix copythe whole/nix/storeinto/tank/hydraso that old builds including dependencies are still accessible, which is quite unwieldy (especially considering the slowness and excessive memory consumption ofnix copy) and also wastes a few dozen gigabytes. This would also allow serving builds that did not go through Hydra. And, less importantly, the system nix store is on NVMe SSD which is faster than the cheap spinning rust of the ZFS tank.There is still some unexplained and potentially major breakage.
Seems to affect those store entries that have been copied with
nix copyand not with hydra-queue-runner...Disabled file:// store for now.
Involved files in the Hydra store:
This corruption still eludes me.
Could please you attach a dump of the HTTP response which contains the actual corruption for the .nar.xz in hydra-bug.tar? Maybe a diff of these is going to hint at what is going on.
What did you try?
Does spawning an Hydra instance with the files I posted in a
file://store_uri fail to reproduce the problem?That is exactly what I did. Then I put them through Hydra/http and they decompress just fine.
I don't have a clue where this is being corrupted. There is no mangling of the .nar.xz payload. We have already tried opening the file in raw mode.
Please share a
curl -D- ...nar.xz > dumpso that I can investigate the type of corruption.I couldn't reproduce the issue with the original path - it now works normally. But now
/nix/store/b433sdzddsafbr7mzpjycys0bxhxzby4-openocd-mlabs-0.10.0is affected. Hydra is serving an emptynar.xzfile.Nothing reported in
journalctl -xe -u hydra-server.serviceMh it actually sends empty files on any
/nar/xxxURL... what kind of error behavior is that.@astro you should be able to reproduce the "return empty file on error" problem and fix it. Just try to access
http://hydra/nar/bogusfilewith your patch applied.Without your patch I correctly get a 404, so it's related.
Ok, I added some error handling for that case to PR #11.
Also, I ran fetches for all files in my binary cache store. They decompressed and
nix-store --restorewithout error.Now there are errors such as
for things that actually exist.
What are you doing to cause such a path to be requested? Both filetypes (
/*.narinfoand/nar/*.nar.xz) have only a hash in their filename. Is this a client error?Access patterns look like this for me (running
nix-env -i /nix/store/b433sdzddsafbr7mzpjycys0bxhxzby4-openocd-mlabs-0.10.0, running nginx in the middle):nix-shellon a regularshell.nixfile to install ARTIQ. The error message is as printed by Nix, I do not know if it corresponds to the actual URL.New server arrived with root-on-ZFS.