Add offline installer #12

Merged
sb10q merged 2 commits from esavkin/artiq-extrapkg:2289-offline-installer into master 2024-05-07 17:42:51 +08:00
Collaborator
There is no content yet.
esavkin added 1 commit 2024-04-30 10:19:30 +08:00
0a55b036d9 Add offline installer hydra job
Signed-off-by: Egor Savkin <es@m-labs.hk>
sb10q reviewed 2024-04-30 10:30:25 +08:00
flake.nix Outdated
@ -538,0 +597,4 @@
msys2.wine-msys2
];
};
installer = pkgs.mkShell {

msys2-
Or just merge it with the above one.

msys2- Or just merge it with the above one.
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:31:51 +08:00
@ -0,0 +445,4 @@
})
(pkgs.fetchurl {
url = "https://msys2.m-labs.hk/artiq-beta/mingw-w64-x86_64-python-sipyco-1.6-1-any.pkg.tar.zst";

I told you at least four times that this had to use the packages built here in the same flake and not download them from Hydra.

I told you at least four times that this had to use the packages built here in the same flake and not download them from Hydra.
sb10q reviewed 2024-04-30 10:34:08 +08:00
@ -0,0 +1,102 @@
#

My script didn't need another copy of this script which is mostly downloaded from GitHub. Why does yours do?

My script didn't need another copy of this script which is mostly downloaded from GitHub. Why does yours do?
sb10q reviewed 2024-04-30 10:35:57 +08:00
@ -0,0 +7,4 @@
mkdir -p $MSYS2DIR/var/lib/pacman $MSYS2DIR/msys/etc
curl -L https://repo.msys2.org/msys/x86_64/pacman-mirrors-20240210-1-any.pkg.tar.zst | tar xvf - -C $MSYS2DIR --zstd
curl -L https://raw.githubusercontent.com/msys2/MSYS2-packages/master/pacman/pacman.conf | sed -e "s|/etc/pacman.d|$MSYS2DIR/etc/pacman.d|g" -e "s|SigLevel = Required|SigLevel = Never|g" > $MSYS2DIR/etc/pacman.conf

Why do you need another copy of that file if you're downloading and editing it here?

Why do you need another copy of that file if you're downloading and editing it here?
sb10q reviewed 2024-04-30 10:36:32 +08:00
flake.nix Outdated
@ -526,0 +571,4 @@
-p "qt-ifw/packages" \
-c "qt-ifw/config/config.xml" \
--offline-only \
"msys2-x86_64.exe" \

Give it a better name to differentiate it from the original MSYS2 installer.

Give it a better name to differentiate it from the original MSYS2 installer.
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:37:32 +08:00
flake.nix Outdated
@ -526,0 +562,4 @@
export HOME=`mktemp -d`
cp -r $src/installer/qt-ifw qt-ifw
chmod -R 700 qt-ifw
find qt-ifw \( -name "package.xml" -or -name "config.xml" \) -exec sed -i "s|@DATE@|$(date +'%Y-%m-%d')|g" "{}" \;

find? There are only two files and the paths are known, no?

find? There are only two files and the paths are known, no?
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:37:41 +08:00
flake.nix Outdated
@ -526,0 +561,4 @@
buildPhase = ''
export HOME=`mktemp -d`
cp -r $src/installer/qt-ifw qt-ifw
chmod -R 700 qt-ifw

Why?

Why?
Poster
Collaborator

So that 7z can place the archive in this directory

So that 7z can place the archive in this directory

Needs detailed explanation in comments.

Needs detailed explanation in comments.
sb10q marked this conversation as resolved
sb10q reviewed 2024-04-30 10:40:01 +08:00
README.md Outdated
@ -8,0 +14,4 @@
nix build .#hydraJobs.msys2-offline-installer
```
Update packages if needed:

This should only update the dependencies of the packages that are built here.
The packages built here should be injected into the list as another step so they can be updated as part of nix flake update, and also so this repos can work without Hydra.

This should only update the dependencies of the packages that are built here. The packages built here should be injected into the list as another step so they can be updated as part of ``nix flake update``, and also so this repos can work without Hydra.
sb10q reviewed 2024-04-30 10:41:10 +08:00
@ -0,0 +1,37 @@
#!/usr/bin/env bash

Can the code duplication with my script be avoided?

Can the code duplication with my script be avoided?
esavkin added 1 commit 2024-04-30 17:38:52 +08:00
b281f4d727 Reuse existing packages
Signed-off-by: Egor Savkin <es@m-labs.hk>
sb10q reviewed 2024-05-02 09:51:46 +08:00
flake.nix Outdated
@ -526,0 +536,4 @@
installPhase = (pkgs.lib.strings.concatStringsSep "\n" (["mkdir $out"] ++ (map (p: "tar xvf ${p} -C $out") srcs)));
};
msys2_image = pkgs.stdenvNoCC.mkDerivation rec {

image?

image?
sb10q reviewed 2024-05-02 09:56:24 +08:00
flake.nix Outdated
@ -538,0 +601,4 @@
fakeroot
p7zip
msys2.wine-msys2
wine64

Is there a difference between wine64 and wineWowPackages.stable that was used everywhere else before your patch?

In any case you need to be consistent.

Is there a difference between wine64 and wineWowPackages.stable that was used everywhere else before your patch? In any case you need to be consistent.
sb10q reviewed 2024-05-02 09:58:34 +08:00
@ -0,0 +37,4 @@
component.addOperation("CreateShortcut", "@TargetDir@/mingw32.exe", "@StartMenuDir@/MSYS2 MINGW32.lnk", "iconPath=@TargetDir@/mingw32.exe");
component.addOperation("CreateShortcut", "@TargetDir@/mingw64.exe", "@StartMenuDir@/MSYS2 MINGW64.lnk", "iconPath=@TargetDir@/mingw64.exe");
component.addOperation("CreateShortcut", "@TargetDir@/ucrt64.exe", "@StartMenuDir@/MSYS2 UCRT64.lnk", "iconPath=@TargetDir@/ucrt64.exe");
component.addOperation("CreateShortcut", "@TargetDir@/clang64.exe", "@StartMenuDir@/MSYS2 CLANG64.lnk", "iconPath=@TargetDir@/clang64.exe");

Can the non-mingw64 stuff be removed entirely?
As far as I can tell in the majority of cases it just causes software bloat and user confusion.

Can the non-mingw64 stuff be removed entirely? As far as I can tell in the majority of cases it just causes software bloat and user confusion.
sb10q reviewed 2024-05-02 10:03:03 +08:00
@ -24,1 +20,3 @@
echo "]" >> msys2_packages.nix
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf \
--cachedir $MSYS2DIR/msys/cache --sync --print-format "%l %n" \
mingw-w64-x86_64-python\

Obviously the code duplication with https://git.m-labs.hk/M-Labs/artiq-extrapkg/src/branch/master/flake.nix#L445-L465 has to be addressed.

I have already told you many times that code duplication is bad. Please pay attention.

Put this list into a separate file and have Nix and the shell script read it.

https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFile

Obviously the code duplication with https://git.m-labs.hk/M-Labs/artiq-extrapkg/src/branch/master/flake.nix#L445-L465 has to be addressed. I have already told you many times that code duplication is bad. Please pay attention. Put this list into a separate file and have Nix and the shell script read it. https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFile
sb10q reviewed 2024-05-02 10:04:15 +08:00
@ -25,0 +37,4 @@
mingw-w64-x86_64-lld \
mingw-w64-x86_64-python-aiohttp \
mingw-w64-x86_64-python-idna \
mingw-w64-x86_64-python-yarl \

Where are idna and yarl coming from?
Pacman should be able to find such dependencies by itself, no?

Where are idna and yarl coming from? Pacman should be able to find such dependencies by itself, no?
esavkin added 1 commit 2024-05-02 12:51:00 +08:00
esavkin added 1 commit 2024-05-02 13:39:47 +08:00
ce6a1e8975 Fix missing llvm tools
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin force-pushed 2289-offline-installer from ce6a1e8975 to 14ecd12286 2024-05-02 15:53:46 +08:00 Compare
sb10q reviewed 2024-05-02 18:57:04 +08:00
flake.nix Outdated
@ -19,0 +22,4 @@
openocdDependenciesFile = ./msys2/openocd_deps.txt;
pythonparserDependenciesFile = ./msys2/pythonparser_deps.txt;
# reads filename and return list of non-empty lines from it
nonEmptyLinesFileReader = (filename: builtins.filter (x: builtins.lessThan 0 (builtins.length (pkgs.lib.strings.stringToCharacters x)))

Silly name.

Silly name.

You removed the .lnk files for the non-mingw64 stuff. But, is the corresponding data also removed, i.e. is the bloat issue resolved?

You removed the .lnk files for the non-mingw64 stuff. But, is the corresponding data also removed, i.e. is the bloat issue resolved?
esavkin added 1 commit 2024-05-03 17:23:06 +08:00
cacd9e685c Leave only msys (base) and mingw64 repos and files
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin force-pushed 2289-offline-installer from cacd9e685c to f49d816ea2 2024-05-03 17:24:15 +08:00 Compare
sb10q reviewed 2024-05-03 19:35:08 +08:00
@ -0,0 +1,2 @@
mingw-w64-x86_64-openocd

pythonparser_deps does not contain pythonparser
artiq_deps does not contain artiq
openocd_deps contains openocd
Why?

pythonparser_deps does not contain pythonparser artiq_deps does not contain artiq openocd_deps contains openocd Why?

Maybe it should just not be called _deps ?

Maybe it should just not be called _deps ?

Or rather, just merge it into artiq_deps.txt. They are always used together.

Or rather, just merge it into artiq_deps.txt. They are always used together.
sb10q reviewed 2024-05-03 19:37:04 +08:00
flake.nix Outdated
@ -19,0 +22,4 @@
openocdDependenciesFile = ./msys2/openocd_deps.txt;
pythonparserDependenciesFile = ./msys2/pythonparser_deps.txt;
# reads filename and return list of non-empty lines from it
readPackagesFromFile = (filename: builtins.filter (x: builtins.lessThan 0 (builtins.length (pkgs.lib.strings.stringToCharacters x)))

readStringListFromFile ? Nothing tied to packages.

readStringListFromFile ? Nothing tied to packages.
sb10q reviewed 2024-05-03 19:42:44 +08:00
flake.nix Outdated
@ -526,0 +527,4 @@
installPhase = (pkgs.lib.strings.concatStringsSep "\n" (["mkdir $out"] ++ (map (p: "tar xvf ${p} -C $out") srcs)));
};
msys2_artiq_image = pkgs.stdenvNoCC.mkDerivation rec {

For the second time, you need to rename this. It's not an image and I can't even find a single person using "image" in this context on the web.

For the second time, you need to rename this. It's not an image and I can't even find a single person using "image" in this context on the web.

Maybe "environment" ?

Maybe "environment" ?
sb10q reviewed 2024-05-03 19:47:49 +08:00
@ -4,21 +4,40 @@ set -e
cd $(dirname $0)
MSYS2DIR=`pwd`/msys2
rm -rf $MSYS2DIR

Remove. It's not in my script because leaving it there is very useful when debugging.

Remove. It's not in my script because leaving it there is very useful when debugging.
sb10q reviewed 2024-05-03 19:48:03 +08:00
@ -4,21 +4,40 @@ set -e
cd $(dirname $0)
MSYS2DIR=`pwd`/msys2
rm -rf $MSYS2DIR

Remove. It's not in my script because leaving it there is very useful when debugging.

Remove. It's not in my script because leaving it there is very useful when debugging.
sb10q reviewed 2024-05-03 19:49:24 +08:00
@ -13,1 +15,3 @@
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf --cachedir $MSYS2DIR/msys/cache -Sp mingw-w64-x86_64-cc mingw-w64-x86_64-python3.11 mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-setuptools > $MSYS2DIR/packages.txt
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf \
--cachedir $MSYS2DIR/msys/cache --sync --print-format "%l %n" \
mingw-w64-x86_64-cc mingw-w64-x86_64-python3.11 \

mingw-w64-x86_64-python3.11 vs. mingw-w64-x86_64-python in your files should make alarm bells go off.

mingw-w64-x86_64-python3.11 vs. mingw-w64-x86_64-python in your files should make alarm bells go off.
sb10q reviewed 2024-05-03 19:50:20 +08:00
@ -12,2 +14,3 @@
fakeroot pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf -Syy
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf --cachedir $MSYS2DIR/msys/cache -Sp mingw-w64-x86_64-cc mingw-w64-x86_64-python3.11 mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-setuptools > $MSYS2DIR/packages.txt
pacman --root $MSYS2DIR --config $MSYS2DIR/etc/pacman.conf \
--cachedir $MSYS2DIR/msys/cache --sync --print-format "%l %n" \

I never needed --sync or --print-format. Why did you add them?

I never needed --sync or --print-format. Why did you add them?
Poster
Collaborator

--sync is same as -S https://archlinux.org/pacman/pacman.8.html, print format is used to separate package name from its full name, which may contain tilde.

--sync is same as -S https://archlinux.org/pacman/pacman.8.html, print format is used to separate package name from its full name, which may contain tilde.
sb10q reviewed 2024-05-03 19:51:35 +08:00
flake.nix Outdated
@ -526,0 +517,4 @@
msys2-offline-installer = pkgs.stdenv.mkDerivation {
name = "msys2-installer";
buildInputs = with pkgs; [ curl pacman fakeroot p7zip wine64 ];

Please pay attention.

Please pay attention.

There's a bunch of messages like:

msys2_artiq_image> could not change the root directory (Operation not permitted)
msys2_artiq_image> error: command failed to execute correctly

Are they harmless?

There's a bunch of messages like: ``` msys2_artiq_image> could not change the root directory (Operation not permitted) msys2_artiq_image> error: command failed to execute correctly ``` Are they harmless?

Why did the binary size increase from 200-something MB to 381MB?

Why did the binary size increase from 200-something MB to 381MB?

Include the ARTIQ version in the file name.

Include the ARTIQ version in the file name.
Poster
Collaborator

Why did the binary size increase from 200-something MB to 381MB?

I added LLVM as a dependency, otherwise artiq_run/compile fails because there are no LLVM tools

> Why did the binary size increase from 200-something MB to 381MB? I added LLVM as a dependency, otherwise artiq_run/compile fails because there are no LLVM tools

This needs to be in a separate PR which is independent from this one, as I have already told you several times.

This needs to be in a separate PR which is independent from this one, as I have already told you several times.
esavkin force-pushed 2289-offline-installer from f49d816ea2 to 9a818cc70b 2024-05-06 15:39:21 +08:00 Compare
esavkin added 13 commits 2024-05-06 15:47:36 +08:00
esavkin force-pushed 2289-offline-installer from 03c9f705ca to b65402644c 2024-05-06 15:54:37 +08:00 Compare
esavkin force-pushed 2289-offline-installer from b65402644c to 471027d336 2024-05-07 12:04:50 +08:00 Compare
Poster
Collaborator

Are they harmless?

Yes, they are not producing errors afterwards and do not influence the artiq.
Though it would be advised to run pacman -Syu (upgrade) two times after install.

> Are they harmless? Yes, they are not producing errors afterwards and do not influence the artiq. Though it would be advised to run `pacman -Syu` (upgrade) two times after install.

Though it would be advised to run pacman -Syu (upgrade) two times after install.

This defeats the purpose of an offline installer.

> Though it would be advised to run pacman -Syu (upgrade) two times after install. This defeats the purpose of an *offline* installer.
Poster
Collaborator

This defeats the purpose of an offline installer.

Not really, it works totally fine without these updates. Just the base image is slightly outdated, and there is no good way to install base packages from repos under Linux/nix environment.

> This defeats the purpose of an *offline* installer. Not really, it works totally fine without these updates. Just the base image is slightly outdated, and there is no good way to install base packages from repos under Linux/nix environment.
esavkin force-pushed 2289-offline-installer from 471027d336 to 196f0b163a 2024-05-07 13:11:25 +08:00 Compare
sb10q merged commit 196f0b163a into master 2024-05-07 17:42:51 +08:00
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/artiq-extrapkg#12
There is no content yet.