Add offline installer #12

Merged
sb10q merged 1 commits from esavkin/artiq-extrapkg:2289-offline-installer into master 2024-08-17 17:37:24 +08:00
Member
No description provided.
sb10q reviewed 2024-04-30 10:30:25 +08:00
flake.nix Outdated
@ -538,0 +597,4 @@
msys2.wine-msys2
];
};
installer = pkgs.mkShell {
Owner

msys2-
Or just merge it with the above one.

msys2- Or just merge it with the above one.
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:31:51 +08:00
@ -0,0 +445,4 @@
})
(pkgs.fetchurl {
url = "https://msys2.m-labs.hk/artiq-beta/mingw-w64-x86_64-python-sipyco-1.6-1-any.pkg.tar.zst";
Owner

I told you at least four times that this had to use the packages built here in the same flake and not download them from Hydra.

I told you at least four times that this had to use the packages built here in the same flake and not download them from Hydra.
sb10q reviewed 2024-04-30 10:34:08 +08:00
@ -0,0 +1,102 @@
#
Owner

My script didn't need another copy of this script which is mostly downloaded from GitHub. Why does yours do?

My script didn't need another copy of this script which is mostly downloaded from GitHub. Why does yours do?
sb10q reviewed 2024-04-30 10:35:57 +08:00
@ -0,0 +7,4 @@
mkdir -p $MSYS2DIR/var/lib/pacman $MSYS2DIR/msys/etc
curl -L https://repo.msys2.org/msys/x86_64/pacman-mirrors-20240210-1-any.pkg.tar.zst | tar xvf - -C $MSYS2DIR --zstd
curl -L https://raw.githubusercontent.com/msys2/MSYS2-packages/master/pacman/pacman.conf | sed -e "s|/etc/pacman.d|$MSYS2DIR/etc/pacman.d|g" -e "s|SigLevel = Required|SigLevel = Never|g" > $MSYS2DIR/etc/pacman.conf
Owner

Why do you need another copy of that file if you're downloading and editing it here?

Why do you need another copy of that file if you're downloading and editing it here?
sb10q reviewed 2024-04-30 10:36:32 +08:00
flake.nix Outdated
@ -526,0 +571,4 @@
-p "qt-ifw/packages" \
-c "qt-ifw/config/config.xml" \
--offline-only \
"msys2-x86_64.exe" \
Owner

Give it a better name to differentiate it from the original MSYS2 installer.

Give it a better name to differentiate it from the original MSYS2 installer.
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:37:32 +08:00
flake.nix Outdated
@ -526,0 +562,4 @@
export HOME=`mktemp -d`
cp -r $src/installer/qt-ifw qt-ifw
chmod -R 700 qt-ifw
find qt-ifw \( -name "package.xml" -or -name "config.xml" \) -exec sed -i "s|@DATE@|$(date +'%Y-%m-%d')|g" "{}" \;
Owner

find? There are only two files and the paths are known, no?

find? There are only two files and the paths are known, no?
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:37:41 +08:00
flake.nix Outdated
@ -526,0 +561,4 @@
buildPhase = ''
export HOME=`mktemp -d`
cp -r $src/installer/qt-ifw qt-ifw
chmod -R 700 qt-ifw
Owner

Why?

Why?
Author
Member

So that 7z can place the archive in this directory

So that 7z can place the archive in this directory
Owner

Needs detailed explanation in comments.

Needs detailed explanation in comments.
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:40:01 +08:00
README.md Outdated
@ -8,0 +14,4 @@
nix build .#hydraJobs.msys2-offline-installer
```
Update packages if needed:
Owner

This should only update the dependencies of the packages that are built here.
The packages built here should be injected into the list as another step so they can be updated as part of nix flake update, and also so this repos can work without Hydra.

This should only update the dependencies of the packages that are built here. The packages built here should be injected into the list as another step so they can be updated as part of ``nix flake update``, and also so this repos can work without Hydra.
sb10q reviewed 2024-04-30 10:41:10 +08:00
@ -0,0 +1,37 @@
#!/usr/bin/env bash
Owner

Can the code duplication with my script be avoided?

Can the code duplication with my script be avoided?
sb10q reviewed 2024-05-02 09:51:46 +08:00
flake.nix Outdated
@ -526,0 +536,4 @@
installPhase = (pkgs.lib.strings.concatStringsSep "\n" (["mkdir $out"] ++ (map (p: "tar xvf ${p} -C $out") srcs)));
};
msys2_image = pkgs.stdenvNoCC.mkDerivation rec {
Owner

image?

image?
sb10q reviewed 2024-05-02 09:56:24 +08:00
flake.nix Outdated
@ -538,0 +601,4 @@
fakeroot
p7zip
msys2.wine-msys2
wine64
Owner

Is there a difference between wine64 and wineWowPackages.stable that was used everywhere else before your patch?

In any case you need to be consistent.

Is there a difference between wine64 and wineWowPackages.stable that was used everywhere else before your patch? In any case you need to be consistent.
sb10q reviewed 2024-05-02 09:58:34 +08:00
@ -0,0 +37,4 @@
component.addOperation("CreateShortcut", "@TargetDir@/mingw32.exe", "@StartMenuDir@/MSYS2 MINGW32.lnk", "iconPath=@TargetDir@/mingw32.exe");
component.addOperation("CreateShortcut", "@TargetDir@/mingw64.exe", "@StartMenuDir@/MSYS2 MINGW64.lnk", "iconPath=@TargetDir@/mingw64.exe");
component.addOperation("CreateShortcut", "@TargetDir@/ucrt64.exe", "@StartMenuDir@/MSYS2 UCRT64.lnk", "iconPath=@TargetDir@/ucrt64.exe");
component.addOperation("CreateShortcut", "@TargetDir@/clang64.exe", "@StartMenuDir@/MSYS2 CLANG64.lnk", "iconPath=@TargetDir@/clang64.exe");
Owner

Can the non-mingw64 stuff be removed entirely?
As far as I can tell in the majority of cases it just causes software bloat and user confusion.

Can the non-mingw64 stuff be removed entirely? As far as I can tell in the majority of cases it just causes software bloat and user confusion.
sb10q reviewed 2024-05-02 10:03:03 +08:00
@ -24,1 +20,3 @@
echo "]" >> msys2_packages.nix
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf \
--cachedir $MSYS2DIR/msys/cache --sync --print-format "%l %n" \
mingw-w64-x86_64-python\
Owner

Obviously the code duplication with https://git.m-labs.hk/M-Labs/artiq-extrapkg/src/branch/master/flake.nix#L445-L465 has to be addressed.

I have already told you many times that code duplication is bad. Please pay attention.

Put this list into a separate file and have Nix and the shell script read it.

https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFile

Obviously the code duplication with https://git.m-labs.hk/M-Labs/artiq-extrapkg/src/branch/master/flake.nix#L445-L465 has to be addressed. I have already told you many times that code duplication is bad. Please pay attention. Put this list into a separate file and have Nix and the shell script read it. https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFile
sb10q reviewed 2024-05-02 10:04:15 +08:00
@ -25,0 +37,4 @@
mingw-w64-x86_64-lld \
mingw-w64-x86_64-python-aiohttp \
mingw-w64-x86_64-python-idna \
mingw-w64-x86_64-python-yarl \
Owner

Where are idna and yarl coming from?
Pacman should be able to find such dependencies by itself, no?

Where are idna and yarl coming from? Pacman should be able to find such dependencies by itself, no?
esavkin force-pushed 2289-offline-installer from ce6a1e8975 to 14ecd12286 2024-05-02 15:53:46 +08:00 Compare
sb10q reviewed 2024-05-02 18:57:04 +08:00
flake.nix Outdated
@ -19,0 +22,4 @@
openocdDependenciesFile = ./msys2/openocd_deps.txt;
pythonparserDependenciesFile = ./msys2/pythonparser_deps.txt;
# reads filename and return list of non-empty lines from it
nonEmptyLinesFileReader = (filename: builtins.filter (x: builtins.lessThan 0 (builtins.length (pkgs.lib.strings.stringToCharacters x)))
Owner

Silly name.

Silly name.
Owner

You removed the .lnk files for the non-mingw64 stuff. But, is the corresponding data also removed, i.e. is the bloat issue resolved?

You removed the .lnk files for the non-mingw64 stuff. But, is the corresponding data also removed, i.e. is the bloat issue resolved?
esavkin force-pushed 2289-offline-installer from cacd9e685c to f49d816ea2 2024-05-03 17:24:15 +08:00 Compare
sb10q reviewed 2024-05-03 19:35:08 +08:00
@ -0,0 +1,2 @@
mingw-w64-x86_64-openocd
Owner

pythonparser_deps does not contain pythonparser
artiq_deps does not contain artiq
openocd_deps contains openocd
Why?

pythonparser_deps does not contain pythonparser artiq_deps does not contain artiq openocd_deps contains openocd Why?
Owner

Maybe it should just not be called _deps ?

Maybe it should just not be called _deps ?
Owner

Or rather, just merge it into artiq_deps.txt. They are always used together.

Or rather, just merge it into artiq_deps.txt. They are always used together.
sb10q reviewed 2024-05-03 19:37:04 +08:00
flake.nix Outdated
@ -19,0 +22,4 @@
openocdDependenciesFile = ./msys2/openocd_deps.txt;
pythonparserDependenciesFile = ./msys2/pythonparser_deps.txt;
# reads filename and return list of non-empty lines from it
readPackagesFromFile = (filename: builtins.filter (x: builtins.lessThan 0 (builtins.length (pkgs.lib.strings.stringToCharacters x)))
Owner

readStringListFromFile ? Nothing tied to packages.

readStringListFromFile ? Nothing tied to packages.
sb10q reviewed 2024-05-03 19:42:44 +08:00
flake.nix Outdated
@ -526,0 +527,4 @@
installPhase = (pkgs.lib.strings.concatStringsSep "\n" (["mkdir $out"] ++ (map (p: "tar xvf ${p} -C $out") srcs)));
};
msys2_artiq_image = pkgs.stdenvNoCC.mkDerivation rec {
Owner

For the second time, you need to rename this. It's not an image and I can't even find a single person using "image" in this context on the web.

For the second time, you need to rename this. It's not an image and I can't even find a single person using "image" in this context on the web.
Owner

Maybe "environment" ?

Maybe "environment" ?
sb10q reviewed 2024-05-03 19:47:49 +08:00
@ -4,21 +4,40 @@ set -e
cd $(dirname $0)
MSYS2DIR=`pwd`/msys2
rm -rf $MSYS2DIR
Owner

Remove. It's not in my script because leaving it there is very useful when debugging.

Remove. It's not in my script because leaving it there is very useful when debugging.
sb10q reviewed 2024-05-03 19:48:03 +08:00
@ -4,21 +4,40 @@ set -e
cd $(dirname $0)
MSYS2DIR=`pwd`/msys2
rm -rf $MSYS2DIR
Owner

Remove. It's not in my script because leaving it there is very useful when debugging.

Remove. It's not in my script because leaving it there is very useful when debugging.
sb10q reviewed 2024-05-03 19:49:24 +08:00
@ -13,1 +15,3 @@
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf --cachedir $MSYS2DIR/msys/cache -Sp mingw-w64-x86_64-cc mingw-w64-x86_64-python3.11 mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-setuptools > $MSYS2DIR/packages.txt
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf \
--cachedir $MSYS2DIR/msys/cache --sync --print-format "%l %n" \
mingw-w64-x86_64-cc mingw-w64-x86_64-python3.11 \
Owner

mingw-w64-x86_64-python3.11 vs. mingw-w64-x86_64-python in your files should make alarm bells go off.

mingw-w64-x86_64-python3.11 vs. mingw-w64-x86_64-python in your files should make alarm bells go off.
sb10q reviewed 2024-05-03 19:50:20 +08:00
@ -12,2 +14,3 @@
fakeroot pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf -Syy
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf --cachedir $MSYS2DIR/msys/cache -Sp mingw-w64-x86_64-cc mingw-w64-x86_64-python3.11 mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-setuptools > $MSYS2DIR/packages.txt
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf \
--cachedir $MSYS2DIR/msys/cache --sync --print-format "%l %n" \
Owner

I never needed --sync or --print-format. Why did you add them?

I never needed --sync or --print-format. Why did you add them?
Author
Member

--sync is same as -S https://archlinux.org/pacman/pacman.8.html, print format is used to separate package name from its full name, which may contain tilde.

--sync is same as -S https://archlinux.org/pacman/pacman.8.html, print format is used to separate package name from its full name, which may contain tilde.
sb10q reviewed 2024-05-03 19:51:35 +08:00
flake.nix Outdated
@ -526,0 +517,4 @@
msys2-offline-installer = pkgs.stdenv.mkDerivation {
name = "msys2-installer";
buildInputs = with pkgs; [ curl pacman fakeroot p7zip wine64 ];
Owner

Please pay attention.

Please pay attention.
Owner

There's a bunch of messages like:

msys2_artiq_image> could not change the root directory (Operation not permitted)
msys2_artiq_image> error: command failed to execute correctly

Are they harmless?

There's a bunch of messages like: ``` msys2_artiq_image> could not change the root directory (Operation not permitted) msys2_artiq_image> error: command failed to execute correctly ``` Are they harmless?
Owner

Why did the binary size increase from 200-something MB to 381MB?

Why did the binary size increase from 200-something MB to 381MB?
Owner

Include the ARTIQ version in the file name.

Include the ARTIQ version in the file name.
Author
Member

Why did the binary size increase from 200-something MB to 381MB?

I added LLVM as a dependency, otherwise artiq_run/compile fails because there are no LLVM tools

> Why did the binary size increase from 200-something MB to 381MB? I added LLVM as a dependency, otherwise artiq_run/compile fails because there are no LLVM tools
Owner

This needs to be in a separate PR which is independent from this one, as I have already told you several times.

This needs to be in a separate PR which is independent from this one, as I have already told you several times.
esavkin force-pushed 2289-offline-installer from f49d816ea2 to 9a818cc70b 2024-05-06 15:39:21 +08:00 Compare
esavkin force-pushed 2289-offline-installer from 03c9f705ca to b65402644c 2024-05-06 15:54:37 +08:00 Compare
esavkin force-pushed 2289-offline-installer from b65402644c to 471027d336 2024-05-07 12:04:50 +08:00 Compare
Author
Member

Are they harmless?

Yes, they are not producing errors afterwards and do not influence the artiq.
Though it would be advised to run pacman -Syu (upgrade) two times after install.

> Are they harmless? Yes, they are not producing errors afterwards and do not influence the artiq. Though it would be advised to run `pacman -Syu` (upgrade) two times after install.
Owner

Though it would be advised to run pacman -Syu (upgrade) two times after install.

This defeats the purpose of an offline installer.

> Though it would be advised to run pacman -Syu (upgrade) two times after install. This defeats the purpose of an *offline* installer.
Author
Member

This defeats the purpose of an offline installer.

Not really, it works totally fine without these updates. Just the base image is slightly outdated, and there is no good way to install base packages from repos under Linux/nix environment.

> This defeats the purpose of an *offline* installer. Not really, it works totally fine without these updates. Just the base image is slightly outdated, and there is no good way to install base packages from repos under Linux/nix environment.
esavkin force-pushed 2289-offline-installer from 471027d336 to 196f0b163a 2024-05-07 13:11:25 +08:00 Compare
sb10q merged commit 196f0b163a into master 2024-05-07 17:42:51 +08:00
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/artiq-extrapkg#12
No description provided.