M-Labs/nac3
M-Labs
/
nac3
8
0
Fork 10
Code Issues 99 Pull Requests 21 Releases Wiki Activity

compilation of empty kernel emits undefined llvm.stackrestore.p0i8 #443

New Issue
Open
opened 2024-07-03 13:55:45 +08:00 by sb10q · 10 comments
sb10q commented 2024-07-03 13:55:45 +08:00
Owner
Copy Link

Replace blink_forever.py with:

from artiq.experiment import *
from artiq.coredevice.core import Core

@nac3
class BlinkForever(EnvExperiment):
    core: KernelInvariant[Core]

    def build(self):
        self.setattr_device("core")

    @kernel
    def run(self):
        pass

$ artiq_run repository/blink_forever.py
...
artiq.coredevice.comm_kernel.LoadError: cannot load kernel: symbol lookup error: llvm.stackrestore.p0i8

$ llvm-readelf -a repository/blink_forever.elf
...
     2: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND llvm.stackrestore.p0i8
Replace blink_forever.py with: ``` from artiq.experiment import * from artiq.coredevice.core import Core @nac3 class BlinkForever(EnvExperiment): core: KernelInvariant[Core] def build(self): self.setattr_device("core") @kernel def run(self): pass ``` ``` $ artiq_run repository/blink_forever.py ... artiq.coredevice.comm_kernel.LoadError: cannot load kernel: symbol lookup error: llvm.stackrestore.p0i8 ``` ``` $ llvm-readelf -a repository/blink_forever.elf ... 2: 00000000 0 NOTYPE GLOBAL DEFAULT UND llvm.stackrestore.p0i8 ```
sb10q added the
high-priority
 label 2024-07-03 13:55:45 +08:00
lyken was assigned by sb10q 2024-07-03 13:55:45 +08:00
sb10q commented 2024-07-03 14:00:06 +08:00
Author
Owner
Copy Link

Also occurs in other situations, but the empty kernel seems to be the simplest repro.

Also occurs in other situations, but the empty kernel seems to be the simplest repro.
derppening commented 2024-07-04 15:19:55 +08:00
Collaborator
Copy Link

Try to partially revert ad4832dc for all intrinsic functions that take a pointer as its input parameter. This has been reported upstream here: https://github.com/TheDan64/inkwell/issues/496.

Try to partially revert ad4832dc for all intrinsic functions that take a pointer as its input parameter. This has been reported upstream here: https://github.com/TheDan64/inkwell/issues/496.
lyken commented 2024-07-04 16:08:57 +08:00
Collaborator
Copy Link

Some discoveries (on d6565feed3):

Creating nac3artiq/demo/demo2.py with the following:

from min_artiq import *


@nac3
class Demo:
    core: KernelInvariant[Core]
    led0: KernelInvariant[TTLOut]
    led1: KernelInvariant[TTLOut]

    def __init__(self):
        self.core = Core()
        self.led0 = TTLOut(self.core, 18)
        self.led1 = TTLOut(self.core, 19)

    @kernel
    def run(self):
        self.core.reset()
        while True:
            pass


if __name__ == "__main__":
    Demo().run()

and do cargo build && python3 demo2.py && llvm-readelf -s module.elf in nac3artiq/demo/ reveals that llvm.stackrestore.p0i8 is sometimes there and sometimes not, it is non-deterministic.

With llvm.stackrestore:

   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 00000124     0 NOTYPE  GLOBAL DEFAULT     1 .LBB0_18
     2: 000001cc     0 NOTYPE  GLOBAL DEFAULT     1 .LBB0_20
     3: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_init
     4: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_get_counter
     5: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND now
     6: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rpc_send
     7: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND llvm.stackrestore.p0i8
     8: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rpc_recv
     9: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_output
    10: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND __nac3_personality
    11: 000000d4     0 FUNC    GLOBAL DEFAULT     1 __modinit__
    12: 0000069c     0 NOTYPE  GLOBAL DEFAULT    12 __bss_start
    13: 0000069c     0 NOTYPE  GLOBAL DEFAULT    12 _end
    14: 00001000     0 NOTYPE  GLOBAL DEFAULT    12 _sstack_guard

Without llvm.stackrestore:

   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 00000130     0 NOTYPE  GLOBAL DEFAULT     1 .LBB1_18
     2: 000001a0     0 NOTYPE  GLOBAL DEFAULT     1 .LBB1_19
     3: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_init
     4: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_get_counter
     5: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND now
     6: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_output
     7: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND __nac3_personality
     8: 000000d4     0 FUNC    GLOBAL DEFAULT     1 __modinit__
     9: 000005e4     0 NOTYPE  GLOBAL DEFAULT    11 __bss_start
    10: 000005e4     0 NOTYPE  GLOBAL DEFAULT    11 _end
    11: 00001000     0 NOTYPE  GLOBAL DEFAULT    11 _sstack_guard
Some discoveries (on https://git.m-labs.hk/M-Labs/nac3/commit/d6565feed349182eafa4d372f95fc3dc043859a6): Creating `nac3artiq/demo/demo2.py` with the following: ```python from min_artiq import * @nac3 class Demo: core: KernelInvariant[Core] led0: KernelInvariant[TTLOut] led1: KernelInvariant[TTLOut] def __init__(self): self.core = Core() self.led0 = TTLOut(self.core, 18) self.led1 = TTLOut(self.core, 19) @kernel def run(self): self.core.reset() while True: pass if __name__ == "__main__": Demo().run() ``` and do `cargo build && python3 demo2.py && llvm-readelf -s module.elf` in `nac3artiq/demo/` reveals that `llvm.stackrestore.p0i8` is sometimes there and sometimes not, it is non-deterministic. With `llvm.stackrestore`: ``` Num: Value Size Type Bind Vis Ndx Name 0: 00000000 0 NOTYPE LOCAL DEFAULT UND 1: 00000124 0 NOTYPE GLOBAL DEFAULT 1 .LBB0_18 2: 000001cc 0 NOTYPE GLOBAL DEFAULT 1 .LBB0_20 3: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_init 4: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_get_counter 5: 00000000 0 NOTYPE GLOBAL DEFAULT UND now 6: 00000000 0 NOTYPE GLOBAL DEFAULT UND rpc_send 7: 00000000 0 NOTYPE GLOBAL DEFAULT UND llvm.stackrestore.p0i8 8: 00000000 0 NOTYPE GLOBAL DEFAULT UND rpc_recv 9: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_output 10: 00000000 0 NOTYPE GLOBAL DEFAULT UND __nac3_personality 11: 000000d4 0 FUNC GLOBAL DEFAULT 1 __modinit__ 12: 0000069c 0 NOTYPE GLOBAL DEFAULT 12 __bss_start 13: 0000069c 0 NOTYPE GLOBAL DEFAULT 12 _end 14: 00001000 0 NOTYPE GLOBAL DEFAULT 12 _sstack_guard ``` Without `llvm.stackrestore`: ``` Num: Value Size Type Bind Vis Ndx Name 0: 00000000 0 NOTYPE LOCAL DEFAULT UND 1: 00000130 0 NOTYPE GLOBAL DEFAULT 1 .LBB1_18 2: 000001a0 0 NOTYPE GLOBAL DEFAULT 1 .LBB1_19 3: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_init 4: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_get_counter 5: 00000000 0 NOTYPE GLOBAL DEFAULT UND now 6: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_output 7: 00000000 0 NOTYPE GLOBAL DEFAULT UND __nac3_personality 8: 000000d4 0 FUNC GLOBAL DEFAULT 1 __modinit__ 9: 000005e4 0 NOTYPE GLOBAL DEFAULT 11 __bss_start 10: 000005e4 0 NOTYPE GLOBAL DEFAULT 11 _end 11: 00001000 0 NOTYPE GLOBAL DEFAULT 11 _sstack_guard ```
lyken commented 2024-07-04 16:21:02 +08:00
Collaborator
Copy Link

With https://git.m-labs.hk/M-Labs/nac3/commit/ad4832dc reverted (as suggested by @derppening) on top of d6565feed3, and running the same steps above. The orphaned llvm.stackrestore.p0i8 issue is eliminated but the output of readelf -s still reveals some form of non-determinism within the code generator in that sometimes rpc_send and rpc_recv might or might not be added to the symbol table in this particular example.

With rpc_send and rpc_recv:

   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 00000124     0 NOTYPE  GLOBAL DEFAULT     1 .LBB0_18
     2: 000001c4     0 NOTYPE  GLOBAL DEFAULT     1 .LBB0_20
     3: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_init
     4: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_get_counter
     5: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND now
     6: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rpc_send
     7: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rpc_recv
     8: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_output
     9: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND __nac3_personality
    10: 000000d4     0 FUNC    GLOBAL DEFAULT     1 __modinit__
    11: 00000654     0 NOTYPE  GLOBAL DEFAULT    12 __bss_start
    12: 00000654     0 NOTYPE  GLOBAL DEFAULT    12 _end
    13: 00001000     0 NOTYPE  GLOBAL DEFAULT    12 _sstack_guard

Without rpc_send and rpc_recv:

   Num:    Value  Size Type    Bind   Vis       Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT   UND 
     1: 00000130     0 NOTYPE  GLOBAL DEFAULT     1 .LBB1_18
     2: 000001a0     0 NOTYPE  GLOBAL DEFAULT     1 .LBB1_19
     3: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_init
     4: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_get_counter
     5: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND now
     6: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND rtio_output
     7: 00000000     0 NOTYPE  GLOBAL DEFAULT   UND __nac3_personality
     8: 000000d4     0 FUNC    GLOBAL DEFAULT     1 __modinit__
     9: 000005e4     0 NOTYPE  GLOBAL DEFAULT    11 __bss_start
    10: 000005e4     0 NOTYPE  GLOBAL DEFAULT    11 _end
    11: 00001000     0 NOTYPE  GLOBAL DEFAULT    11 _sstack_guard
With https://git.m-labs.hk/M-Labs/nac3/commit/ad4832dc reverted (as suggested by @derppening) on top of https://git.m-labs.hk/M-Labs/nac3/commit/d6565feed349182eafa4d372f95fc3dc043859a6, and running the same steps above. The orphaned `llvm.stackrestore.p0i8` issue is eliminated but the output of `readelf -s` still reveals some form of non-determinism within the code generator in that sometimes `rpc_send` and `rpc_recv` might or might not be added to the symbol table in this particular example. With `rpc_send` and `rpc_recv`: ``` Num: Value Size Type Bind Vis Ndx Name 0: 00000000 0 NOTYPE LOCAL DEFAULT UND 1: 00000124 0 NOTYPE GLOBAL DEFAULT 1 .LBB0_18 2: 000001c4 0 NOTYPE GLOBAL DEFAULT 1 .LBB0_20 3: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_init 4: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_get_counter 5: 00000000 0 NOTYPE GLOBAL DEFAULT UND now 6: 00000000 0 NOTYPE GLOBAL DEFAULT UND rpc_send 7: 00000000 0 NOTYPE GLOBAL DEFAULT UND rpc_recv 8: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_output 9: 00000000 0 NOTYPE GLOBAL DEFAULT UND __nac3_personality 10: 000000d4 0 FUNC GLOBAL DEFAULT 1 __modinit__ 11: 00000654 0 NOTYPE GLOBAL DEFAULT 12 __bss_start 12: 00000654 0 NOTYPE GLOBAL DEFAULT 12 _end 13: 00001000 0 NOTYPE GLOBAL DEFAULT 12 _sstack_guard ``` Without `rpc_send` and `rpc_recv`: ``` Num: Value Size Type Bind Vis Ndx Name 0: 00000000 0 NOTYPE LOCAL DEFAULT UND 1: 00000130 0 NOTYPE GLOBAL DEFAULT 1 .LBB1_18 2: 000001a0 0 NOTYPE GLOBAL DEFAULT 1 .LBB1_19 3: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_init 4: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_get_counter 5: 00000000 0 NOTYPE GLOBAL DEFAULT UND now 6: 00000000 0 NOTYPE GLOBAL DEFAULT UND rtio_output 7: 00000000 0 NOTYPE GLOBAL DEFAULT UND __nac3_personality 8: 000000d4 0 FUNC GLOBAL DEFAULT 1 __modinit__ 9: 000005e4 0 NOTYPE GLOBAL DEFAULT 11 __bss_start 10: 000005e4 0 NOTYPE GLOBAL DEFAULT 11 _end 11: 00001000 0 NOTYPE GLOBAL DEFAULT 11 _sstack_guard ```
lyken commented 2024-07-04 16:50:37 +08:00
Collaborator
Copy Link

and, as expected, the non-determinism would (or at least seems to) go away when the code generation stage is run only with a single thread. More specifically, doing this in nac3artiq/src/lib.rs:

// let num_threads = if is_multithreaded() { 4 } else { 1 };
let num_threads = 1; // test
let thread_names: Vec<String> = (0..num_threads).map(|_| "main".to_string()).collect();
let threads: Vec<_> = thread_names
    .iter()
    .map(|s| Box::new(ArtiqCodeGenerator::new(s.to_string(), size_t, self.time_fns)))
    .collect();
and, as expected, the non-determinism would (or at least seems to) go away when the code generation stage is run only with a single thread. More specifically, doing this in `nac3artiq/src/lib.rs`: ```rust // let num_threads = if is_multithreaded() { 4 } else { 1 }; let num_threads = 1; // test let thread_names: Vec<String> = (0..num_threads).map(|_| "main".to_string()).collect(); let threads: Vec<_> = thread_names .iter() .map(|s| Box::new(ArtiqCodeGenerator::new(s.to_string(), size_t, self.time_fns))) .collect(); ```
sb10q removed the
high-priority
 label 2024-07-12 19:06:32 +08:00
lyken commented 2024-08-16 16:41:49 +08:00
Collaborator
Copy Link

Closing, the llvm.stackrestore.p0i8 problem is fixed by #446.

About non-deterministic object outputs, see #7.

Closing, the `llvm.stackrestore.p0i8` problem is fixed by https://git.m-labs.hk/M-Labs/nac3/pulls/446. About non-deterministic object outputs, see https://git.m-labs.hk/M-Labs/nac3/issues/7.
lyken closed this issue 2024-08-16 16:41:50 +08:00
sb10q commented 2024-08-16 17:22:58 +08:00
Author
Owner
Copy Link

It is not fixed, it is worked around.

It is not fixed, it is worked around.
sb10q reopened this issue 2024-08-16 17:23:00 +08:00
lyken commented 2024-08-16 17:32:17 +08:00
Collaborator
Copy Link

I think the compilation of kernels part is fixed?

The issue is only caused by Inkwell emitting llvm.stackrestore.p0i8 due to a bug in Inkwell itself https://github.com/TheDan64/inkwell/issues/496. #446 is a workaround on call_stackrestore to make Inkwell spit out the name llvm.stackrestore correctly.

Compilation of certain kernels yielding llvm.stackrestore.p0i8 randomly is probably due to indeterministic codegen worker outputs and LLVM sometimes couldn't pair up llvm.stackstore with a llvm.stackrestore (wrongly named llvm.stackrestore.p0i8).

I think the compilation of kernels part is fixed? The issue is only caused by Inkwell emitting `llvm.stackrestore.p0i8` due to a bug in Inkwell itself https://github.com/TheDan64/inkwell/issues/496. https://git.m-labs.hk/M-Labs/nac3/pulls/446 is a workaround on `call_stackrestore` to make Inkwell spit out the name `llvm.stackrestore` correctly. Compilation of certain kernels yielding `llvm.stackrestore.p0i8` randomly is probably due to indeterministic codegen worker outputs and LLVM sometimes couldn't pair up `llvm.stackstore` with a `llvm.stackrestore` (wrongly named `llvm.stackrestore.p0i8`).
sb10q commented 2024-08-16 17:33:04 +08:00
Author
Owner
Copy Link

When there are no more "probably" and "workaround" then we can close this Issue.

When there are no more "probably" and "workaround" then we can close this Issue.
lyken commented 2024-08-16 17:37:46 +08:00
Collaborator
Copy Link

Sure

Sure
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
misc/impl-tracert
ndstrides-neo-next
ndstrides-14-end
ndstrides-13-tests
ndstrides-12-builtins
ndstrides-11-matmul
ndstrides-10-ops
ndstrides-9-subassign
ndstrides-8-transpose
ndstrides-7-broadcasting
ndstrides-6-reshape
ndstrides-5-miscfuncs
ndstrides-4-nparray
ndstrides-3-indexing
ndstrides-2-basic
ndstrides-1-model
ndstrides
refactor_composer
misc/update-pyo3
misc/mold
ndstrides-1-model-lifetime
ndstrides-enhance-irrt
ndstrides-intro
issue-313
refactor-primstore
numpy-anyall
enhance/cargo-test-standalone-demos
issue-136
rpc-obj-as-param
iterators
escape-analysis
refactor_anto
Labels
Clear labels   
future-extension

   
high-priority

   
low-priority

   
UROP0622
No Label
future-extension
high-priority
low-priority
UROP0622
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
lyken
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/nac3#443
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?