Changes to kernel source while experiment running changes kernel #401

New Issue

sb10q · 2026-01-18T18:34:41+08:00

r-srinivas commented

2026-01-18 18:34:41 +08:00

Migrated from GitHub: #416

Using 1.0 rc3 on Windows. If I have an experiment that's running, if I make any changes to it while it's running and save the file it changes the experiment while it's running. I think it would be better if changes were only effected when the experiment is restarted. Basically a running experiment should be unaffected by changes in the python script.

I was running the following experiment

from artiq.experiment import *

class BD_Scan_Test(EnvExperiment):

    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl1")
        self.setattr_device("ttl2")
        self.setattr_device("dds0")
        self.setattr_device("scheduler")
        #  self.setattr_device("ttl6")
        self.setattr_argument("frequency_scan", Scannable(
            default=LinearScan(100, 200, 100)))
        self.setattr_argument("repetitions", NumberValue(100, min=10, max=100))

    def run(self):
        for i, f in enumerate(self.frequency_scan):
            self.core.comm.close()
            self.scheduler.pause()
            self.dds_scan(f)

    @kernel
    def dds_scan(self, frequency):
        self.core.break_realtime()
        self.dds0.set(frequency*MHz)
        print(frequency)

And was just turning the print on and off.

> **Migrated from GitHub:** [#416](https://github.com/m-labs/artiq/issues/416) --- Using 1.0 rc3 on Windows. If I have an experiment that's running, if I make any changes to it while it's running and save the file it changes the experiment while it's running. I think it would be better if changes were only effected when the experiment is restarted. Basically a running experiment should be unaffected by changes in the python script. I was running the following experiment ``` python from artiq.experiment import * class BD_Scan_Test(EnvExperiment): def build(self): self.setattr_device("core") self.setattr_device("ttl1") self.setattr_device("ttl2") self.setattr_device("dds0") self.setattr_device("scheduler") # self.setattr_device("ttl6") self.setattr_argument("frequency_scan", Scannable( default=LinearScan(100, 200, 100))) self.setattr_argument("repetitions", NumberValue(100, min=10, max=100)) def run(self): for i, f in enumerate(self.frequency_scan): self.core.comm.close() self.scheduler.pause() self.dds_scan(f) @kernel def dds_scan(self, frequency): self.core.break_realtime() self.dds0.set(frequency*MHz) print(frequency) ``` And was just turning the print on and off.

sb10q added this to the 2.0 milestone 2026-01-18 18:34:41 +08:00

sb10q added the area:compiler prio:2-normal labels 2026-01-18 18:34:41 +08:00

sb10q self-assigned this 2026-01-18 18:34:41 +08:00

sb10q closed this issue

2026-01-18 18:34:41 +08:00

sb10q commented

2026-01-18 18:34:41 +08:00

That's a Python problem...

import inspect
import time


def foo():
    print("hello1")  # run and change this line

time.sleep(2)
print(inspect.getsource(foo))

That's a Python problem... ``` python import inspect import time def foo(): print("hello1") # run and change this line time.sleep(2) print(inspect.getsource(foo)) ```

whitequark commented

2026-01-18 18:34:41 +08:00

Yes, Python caches lines inside inspect.getsource.

Yes, Python caches lines inside `inspect.getsource`.

sb10q commented

2026-01-18 18:34:41 +08:00

No, it does not, which is the problem.

sb10q commented

2026-01-18 18:34:42 +08:00

https://bugs.python.org/issue1218234

whitequark commented

2026-01-18 18:34:42 +08:00

Oh, so they broke it in 3.5...

sb10q commented

2026-01-18 18:34:42 +08:00

We could do this, but it's ugly and not thread-safe.

--- a/artiq/compiler/embedding.py
+++ b/artiq/compiler/embedding.py
@@ -492,6 +492,21 @@ class TypedtreeHasher(algorithm.Visitor):
             fields = fields + node._types
         return hash(tuple(freeze(getattr(node, field_name)) for field_name in fields))

+
+# HACK
+# We want the copy in cache, so that experiments keep running
+# the same kernel source as when they were started, even when
+# the source file is modified on disk.
+# Unfortunately, the cache invalidation is deep inside
+# the inspect module and there is no clean API to disable it.
+def getsource_in_cache(obj):
+    checkcache = linecache.checkcache
+    linecache.checkcache = lambda x: None
+    source_code = inspect.getsource(obj)
+    linecache.checkcache = checkcache
+    return source_code
+
+
 class Stitcher:
     def __init__(self, core, dmgr, engine=None):
         self.core = core
@@ -582,7 +597,7 @@ class Stitcher:

         # Extract function source.
         embedded_function = function.artiq_embedded.function
-        source_code = inspect.getsource(embedded_function)
+        source_code = getsource_in_cache(embedded_function)
         filename = embedded_function.__code__.co_filename
         module_name = embedded_function.__globals__['__name__']
         first_line = embedded_function.__code__.co_firstlineno

We could do this, but it's ugly and not thread-safe. ``` --- a/artiq/compiler/embedding.py +++ b/artiq/compiler/embedding.py @@ -492,6 +492,21 @@ class TypedtreeHasher(algorithm.Visitor): fields = fields + node._types return hash(tuple(freeze(getattr(node, field_name)) for field_name in fields)) + +# HACK +# We want the copy in cache, so that experiments keep running +# the same kernel source as when they were started, even when +# the source file is modified on disk. +# Unfortunately, the cache invalidation is deep inside +# the inspect module and there is no clean API to disable it. +def getsource_in_cache(obj): + checkcache = linecache.checkcache + linecache.checkcache = lambda x: None + source_code = inspect.getsource(obj) + linecache.checkcache = checkcache + return source_code + + class Stitcher: def __init__(self, core, dmgr, engine=None): self.core = core @@ -582,7 +597,7 @@ class Stitcher: # Extract function source. embedded_function = function.artiq_embedded.function - source_code = inspect.getsource(embedded_function) + source_code = getsource_in_cache(embedded_function) filename = embedded_function.__code__.co_filename module_name = embedded_function.__globals__['__name__'] first_line = embedded_function.__code__.co_firstlineno ```

whitequark commented

2026-01-18 18:34:42 +08:00

Huh?? You could simply cache it yourself instead, it is actually easier to implement. But I'm not convinced that this is the desired behavior yet. It seems like we ought to cache all files during startup or something, not cache on first use.

sb10q commented

2026-01-18 18:34:42 +08:00

Huh?? You could simply cache it yourself instead

How? inspect will call checkcache which will look for changes on the disk.

> Huh?? You could simply cache it yourself instead How? `inspect` will call `checkcache` which will look for changes on the disk.

whitequark commented

2026-01-18 18:34:42 +08:00

Do not call inspect.getsource every time, use a dict from functions to sources. That would be equivalent to your snippet above.

Do not call `inspect.getsource` every time, use a dict from functions to sources. That would be equivalent to your snippet above.

sb10q commented

2026-01-18 18:34:42 +08:00

Ah right, I thought linecache was smarter than that.

Ah right, I thought `linecache` was smarter than that.

sb10q commented

2026-01-18 18:34:42 +08:00

We can hook imports, keep in-memory copies of imported sources, and ship our modified version of inspect that uses those copies instead of linecache. With this strategy, the typical cost is ~15MB of bytes objects per worker, and ~45ms of startup slow-down per experiment.

Options to cut that down are:

an ad-hoc list of modules that won't contain kernels - ugly.
loading the user code only - tricky to detect reliably (e.g. what if someone is developing an ARTIQ library package and modifies it?).
loading the whole file upon first invokation of a kernel from it - rather fast and non-bloated, but only marginal improvement (e.g. if the file is modified before the first kernel is executed, the problem will occur again).

import sys

# typical imports from worker
import time
import os
import logging
import traceback
from collections import OrderedDict

import h5py

import artiq
from artiq.protocols import pipe_ipc, pyon
from artiq.protocols.packed_exceptions import raise_packed_exc
from artiq.tools import multiline_log_config, file_import
from artiq.master.worker_db import DeviceManager, DatasetManager
from artiq.language.environment import (is_experiment, TraceArgumentManager,
                                        ProcessArgumentManager)
from artiq.language.core import set_watchdog_factory, TerminationRequested
from artiq.coredevice.core import CompileError, host_only, _render_diagnostic
from artiq import __version__ as artiq_version
import artiq.experiment
import artiq.coredevice.core
#


t1 = time.time()
total = 0
for mod in sys.modules.values():
   fn = getattr(mod, "__file__", None)
   if fn is not None:
        with open(fn, "rb") as f:
            total += len(f.read())

t2 = time.time()

print(total, t2-t1)

We can hook imports, keep in-memory copies of imported sources, and ship our modified version of `inspect` that uses those copies instead of `linecache`. With this strategy, the typical cost is ~15MB of bytes objects per worker, and ~45ms of startup slow-down per experiment. Options to cut that down are: - an ad-hoc list of modules that won't contain kernels - ugly. - loading the user code only - tricky to detect reliably (e.g. what if someone is developing an ARTIQ library package and modifies it?). - loading the whole file upon first invokation of a kernel from it - rather fast and non-bloated, but only marginal improvement (e.g. if the file is modified before the first kernel is executed, the problem will occur again). ``` python import sys # typical imports from worker import time import os import logging import traceback from collections import OrderedDict import h5py import artiq from artiq.protocols import pipe_ipc, pyon from artiq.protocols.packed_exceptions import raise_packed_exc from artiq.tools import multiline_log_config, file_import from artiq.master.worker_db import DeviceManager, DatasetManager from artiq.language.environment import (is_experiment, TraceArgumentManager, ProcessArgumentManager) from artiq.language.core import set_watchdog_factory, TerminationRequested from artiq.coredevice.core import CompileError, host_only, _render_diagnostic from artiq import __version__ as artiq_version import artiq.experiment import artiq.coredevice.core # t1 = time.time() total = 0 for mod in sys.modules.values(): fn = getattr(mod, "__file__", None) if fn is not None: with open(fn, "rb") as f: total += len(f.read()) t2 = time.time() print(total, t2-t1) ```

sb10q commented

2026-01-18 18:34:42 +08:00

FYI, Chrome does this on Unix:
http://neugierig.org/software/chromium/notes/2011/08/zygote.html
but this cannot be applied here because text editors (at least Sublime Text which I tested) do not do file replacements like package managers.

That works as long as nobody overwrites the contents of the file we have open; thankfully, package updates write a new file and rename it over the old name, leaving our open copy the only remaining reference to the old file

FYI, Chrome does this on Unix: http://neugierig.org/software/chromium/notes/2011/08/zygote.html but this cannot be applied here because text editors (at least Sublime Text which I tested) do not do file replacements like package managers. > That works as long as nobody overwrites the contents of the file we have open; thankfully, package updates write a new file and rename it over the old name, leaving our open copy the only remaining reference to the old file

whitequark commented

2026-01-18 18:34:42 +08:00

This solution sounds good to me.

jordens commented

2026-01-18 18:34:43 +08:00

ack

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/artiq#401