LLVM Linker (ld.lld) Invocation Fails on Compilation with MSYS2 #1611
esavkin
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Bug Report
One-Line Summary
When attempting to compile and run an experiment on ARTIQ 8 Beta, the compilation fails with the worker exception
Exception: ld.lld invocation failed.Issue Details
I've been hesitating to submit this issue as I'm still on the fence about whether or not this is a local configuration issue, but I've run out of avenues to try to diagnose what's going on. I have also gone back and attempted to replicate this problem using conda. If I install a conda environment and set it up with the same hardware, this behavior does not occur, and the object file for the experiment is compiled and runs on the core.
Steps to Reproduce
pacmanto installmingw-w64-x86_64-artiqper instructions on the manual-beta page.artiq_masterandartiq_dashboardin MINGW64.Expected Behavior
Experiment object file is compiled, assembled, and loaded onto the core for running by the scheduler.
Actual (undesired) Behavior
worker returns an exception and quits out before finishing compilation.
Your System (omit irrelevant parts)
EK-Z7-ZC706-G)List of Managed Packages
I was giving this another shot after a thought this afternoon. Sadly I didn't get it working, but I did get another more verbose error that might help? Nothing has changed appreciably between the state of the system reported above and this output:
Exception: ld.lld invocation failed output:
Did you do that?
No, I have not. After looking into LLVM's reporting guidelines, I originally didn't think I'd be able to produce a minimal test case for them and debug due to not knowing how the ARTIQ compilation pipeline worked. After spending a bit more time today, I was able to directly invoke
in
/tmpusing MINGW64. In attempting to compile the same experiment with ARTIQ again, I received the same invocation error. From what I can tell, I do believe this is a problem specifically related to ARTIQ and perhaps how it interacts with the MSYS2 environment.Does it crash then?
Why?
It did not. As far as I can tell, the output file was generated successfully.
As I can explicitly run the linker with the same files in the MSYS2 environment without issue, and I can compile the code successfully on a conda-managed environment using the same version of ARTIQ.
@thomasfire ping
It does not fail on ld.lld now, but fails on llvm tools when I tried to use artiq_run, because there is no such dependency
Will be added with M-Labs/artiq-extrapkg#12
This has nothing to do with the offline installer so please fix it separately.
@thomasfire What is the status of this Issue?
Can be closed since llvm tools is a dependency now and shipped with installer.
We've experienced this fail today on a fresh MSYS2 installation and ARTIQ installed according to https://m-labs.hk/artiq/manual-beta/installing.html#installing-via-msys2-windows Will this installation path be available or will be deprecated in favor of the installer?
Yes. @thomasfire what did you test?
artiq_compilecompiles file after following https://m-labs.hk/artiq/manual-beta/installing.html#installing-via-msys2-windowsI tried to use above instruction on my script. I received following result:
I see, seems this bug is floating and reproduces only a few times, while others may be successful
I don't think so. Pacman is quite deterministic.
Pacman yes, but the problem is compilation process is not
Blergh. I'm so tired of LLVM.
What happens if you run lld manually multiple times with the same inputs?
Also happens to be different on different windows versions, for example on (tiny) windows 11 it failed only once after reboot, which feels like it got timed out because of loading something...
However on windows 10 it fails once in a few times
Yeah, it is different
In @martafafara case it also happens to work from time to time, but time after boot does not seem to be relevant.
FWIW in our case it would never succeed when compiling through the dashboard (if it matters, as we never tried
artiq_compiledirectly), but it worked the few times we tried to directly invokeld.lldwith the same inputs. We did not do more than a few repeated tests of the latter.I just checked lld from clang64 repo on the same file, and it seems like only the mingw64 version occasionally fails. The lld package in conda is built with MSVC2019, which may explain the differences in behavior.
Also the LLVM itself doesn't really support mingw on windows https://llvm.org/docs/GettingStarted.html#hardware , so I doubt its LLVM's fault.
I found your results really optimistic. In my case I receive more
ld.llderrors then successful compilations. Running any script requires at least few tries, so limiting it to "occasionally" will be really helpful.@martafafara We are experimenting with clang64 version of ARTIQ, would you mind checking if it works on your OS, since it seems you have it being reproduced more often? The installer is built from https://git.m-labs.hk/esavkin/artiq-extrapkg/src/branch/2316-move-to-clang . I tried to run
artiq_runa few times andartiq_compilea ~20 times each on win10 and win11 VM with latest updates, and no segfault happened.Just as a reminder, it is a standalone installer, and artiq should work out of the box, though installation size and time increased quite significantly (compared to base MSYS2 installer, but similar in case of installing all the packages).
Unfortunately I'm rather the user of Artiq than developer, so I need help of more experienced colleagues. The operation will be done asap
Packages are moved to clang64 environment, they can be installed with
pacman -S mingw-w64-clang-x86_64-artiqnow, or with a few clicks offline installer https://nixbld.m-labs.hk/job/artiq/extra-beta/msys2-offline-installer/latest . No ARTIQ developer experience required for installing it 😊Using ARTIQ requires you to do some programming e.g. to write kernels. It is misleading to advertise ARTIQ as a "no developer experience required" control system in any way.
Partially true, but programming is not exactly developing, and developing is not only programming. Based on my experience with analytics'/data scientists' code, there are major differences in the ways we work. In similar way, based on helpdesk, using ARTIQ is far from developing it, and developing is far from using it.
Of coarse some knowledge and programming skills are required, but from my point of view Artiq is just a tool. Complicated, but still only the tool that I'm using. The knowledge how it is exactly constructed is for sure fascinating but not necessary.
I followed the offline installer and as for now for 5 scripts runs I've received no errors at all. What a relief... Thank you!
@thomasfire I have just run in to the same llvm issue on a new windows/MSYS2 install, I had
ld.lld invocation failederrors every time I tried to compile with the originalmingw64packagesI've just reinstalled things using your new
mingw-w64-clang-x86_64-artiqpackage, and I can now compile without error 🙂I can also submit through the dashboard without error (at least simple test Experiments).
Thanks for the fix!