drtio: add InjectionRequest to expects_response

Signed-off-by: Jonathan Coates <jonathan.coates@oxionics.com>

README.md

@@ -4,60 +4,102 @@ ARTIQ on Zynq
 How to use
 ----------
 
-1. Install the ARTIQ version that corresponds to the artiq-zynq version you are targeting.
+1. [Install ARTIQ](https://m-labs.hk/artiq/manual/installing.html). Get the corresponding version to the ``artiq-zynq`` version you are targeting.
-2. To obtain firmware binaries, select the latest successful build on [Hydra](https://nixbld.m-labs.hk/) for the targeted artiq-zynq version, or use AFWS. If using Hydra, search for the job named ``<board>-<variant>-sd`` (for example: ``zc706-nist_clock-sd`` or ``zc706-nist_qc2-sd``).
+2. To obtain firmware binaries, use AFWS or build your own; see [the ARTIQ manual](https://m-labs.hk/artiq/manual/building_developing.html) for detailed instructions or skip to "Development" below. ZC706 variants only can also be downloaded from latest successful build on [Hydra](https://nixbld.m-labs.hk/).
-3. Place the ``boot.bin`` file, obtained from Hydra's "binary distribution" download link or from AFWS, at the root of a FAT-formatted SD card.
+3. Place ``boot.bin`` file at the root ``/`` of a FAT-formatted SD card.
-4. Optionally, create a ``config.txt`` configuration file at the root of the SD card containing ``key=value`` pairs on each line. Use the ``ip``, ``ip6`` and ``mac`` keys to respectively set the IPv4, IPv6 and MAC address of the board. Configuring an IPv6 address is entirely optional. If these keys are not found, the firmware will use default values that may or may not be compatible with your network.
+4. Optionally, create a ``config.txt`` configuration file containing ``key=value`` pairs on each line and place it at the root of the SD card. See below for valid keys. The ``ip``, ``ip6`` and ``mac`` keys can be used to set networking information. If these keys are not found, the firmware will use default values which may or may not be compatible with your network.
-5. Insert the SD card into the board and set up the board to boot from the SD card. For the ZC706, this is achieved by placing the large DIP switch SW11 in the 00110 position.
+5. Insert the SD card into the board and set the board to boot from the SD card. For ZC706, this is achieved by placing the large DIP switch SW11 into the 00110 position. On Kasli-SoC, place the BOOT MODE switches to SD.
-6. Power up the board. After the firmware starts successfully, it should respond to ping at its IP addresses, and boot messages can be observed from its UART at 115200bps.
+6. Power up the board. After successful boot the firmware should respond to ping at its IP addresses. Boot output can be observed from UART at 115200bps 8-N-1.
-7. Create and use an ARTIQ device database as usual, but set ``"target": "cortexa9"`` in the arguments of the core device.
+7. Create and use an ARTIQ device database as usual.
 
 Configuration
 -------------
 
-Configuring the device is done using the ``config.txt`` text file at the root of the SD card, plus the contents of the ``config`` folder. When searching for a configuration key, the firmware first looks for a file named ``/config/[key].bin`` and, if it exists, returns the contents of that file. If not, it looks into ``/config.txt``, which contains a list of ``key=value`` pairs, one per line. The ``config`` folder allows configuration values that consist in binary data, such as the startup kernel.
+Configuring the device is done using the ``config.txt`` text file at the root of the SD card plus optionally a ``config`` folder. When searching for a configuration key, the firmware first looks for a file named ``/config/[key].bin`` and, if it exists, returns the contents of that file. If not, it looks into ``/config.txt``, which should contain a list of ``key=value`` pairs, one per line. ``config.txt`` should be used for most keys but the ``config`` folder allows for setting configuration values which consist of binary data, such as the startup kernel.
 
-The following configuration keys are available:
+The following configuration keys are available among others:
 
 - ``mac``: Ethernet MAC address.
 - ``ip``: IPv4 address.
 - ``ip6``: IPv6 address.
-- ``startup``: startup kernel in ELF format (as produced by ``artiq_compile``).
+- ``idle_kernel``: idle kernel in ELF format (as produced by ``artiq_compile``).
+- ``startup_kernel``: startup kernel in ELF format (as produced by ``artiq_compile``).
 - ``rtio_clock``: source of RTIO clock; valid values are ``ext0_bypass`` and ``int_125``.
-- ``boot``: SD card "boot.bin" file, for replacing the boot firmware/gateware. Write only.
 
-Configurations can be read/written/removed via ``artiq_coremgmt``. Config erase is
+See [ARTIQ manual](https://m-labs.hk/artiq/manual-beta/core_device.html#configuration-storage) for full list. Configurations can be read/written/removed with ``artiq_coremgmt``. Config erase is not implemented, as it isn't particularly useful.
-not implemented as it seems not very useful.
+
+For convenience, the ``boot`` key can be used with ``artiq_coremgmt`` and a ``boot.bin`` file to replace firmware/gateware in a running system. This key is read-only. When loading ``boot.bin`` onto the SD card directly, place it at the root and not in the ``config`` folder.
 
 Development instructions
 ------------------------
 
-ARTIQ on Zynq is packaged using the [Nix](https://nixos.org) Flakes system. Install Nix 2.8+ and enable flakes by adding ``experimental-features = nix-command flakes`` to ``nix.conf`` (e.g. ``~/.config/nix/nix.conf``).
+ARTIQ on Zynq is packaged using [Nix](https://nixos.org) Flakes. Install Nix 2.8+ and enable flakes by adding ``experimental-features = nix-command flakes`` to ``nix.conf`` (e.g. ``~/.config/nix/nix.conf``).
 
-Pure build with Nix and execution on a remote JTAG server:
+**Pure build with Nix:**
 
 ```shell
-nix build .#zc706-nist_clock-jtag  # or zc706-nist_qc2-jtag or zc706-nist_clock_satellite-jtag etc.
+nix build .#zc706-nist_clock-jtag  # or zc706-nist_qc2-jtag or zc706-nist_clock-sd or etc
-./remote_run.sh
 ```
 
-Impure incremental build and execution on a remote JTAG server:
+Run ``nix flake show`` to see all valid build targets. Targets suffixed with ``-jtag`` produce separate firmware and gateware files, intended for use in booting via JTAG server/Ethernet, e.g. ``./remote_run.sh -i`` with a remote JTAG server. Targets suffixed with ``-sd`` will produce ``boot.bin`` file suitable for SD card boot. ``-firmware`` and ``-gateware`` respectively build firmware and gateware only.
+
+The Kasli-SoC target requires a system description file as input. See ARTIQ manual for exact instructions or use incremental build.
+
+**Impure incremental build:**
+
+For boards with fixed variants, i.e. ZC706, etc. :
 
 ```shell
 nix develop
 cd src
-gateware/zc706.py -g ../build/gateware -V <variant> # build gateware
+gateware/<board>.py -g ../build/gateware -V <variant> # gateware
-make GWARGS="-V <variant>" <runtime/satman>    # build firmware
+make GWARGS="-V <variant>" <runtime/satman>    # firmware
-cd ..
+```
-./remote_run.sh -i
+
+For boards with system descriptions, i.e. Kasli-SoC, etc. :
+
+```shell
+nix develop
+cd src
+gateware/<board>.py -g ../build/gateware <description.json> # gateware
+make TARGET=<board> GWARGS="path/to/description.json" <runtime/satman> # firmware
+```
+
+``szl.elf`` can be obtained with:
+
+```shell
+nix build git+https://git.m-labs.hk/m-labs/zynq-rs#<board>-szl
+```
+
+To generate ``boot.bin`` use ``mkbootimage``, e.g.:
+
+```shell
+echo "the_ROM_image:
+    {
+        [bootloader]result/szl.elf
+        gateware/top.bit
+        firmware/armv7-none-eabihf/release/<runtime/satman>
+    }
+    EOF" >> boot.bif
+mkbootimage boot.bif boot.bin
 ```
 
 Notes:
 
 - The impure build process is also compatible with non-Nix systems.
-- When calling make, you need to specify both the variant and firmware type.
 - Firmware type must be either ``runtime`` for DRTIO-less or DRTIO master variants, or ``satman`` for DRTIO satellite.
-- If the board is connected to the local machine, use the ``local_run.sh`` script.
+- If the board is connected to the local machine by JTAG, use the ``local_run.sh`` script.
+- A known Xilinx hardware bug prevents repeatedly loading the bootloader over JTAG without a POR reset. If booting over JTAG, install a jumper on ``PS_POR_B`` and use the POR reset script [here](https://git.m-labs.hk/M-Labs/zynq-rs/src/branch/master/kasli_soc_por.py).
+
+Pre-Commit Hooks
+----------------
+
+You are strongly recommended to use the provided pre-commit hooks to automatically reformat files and check for non-optimal Rust/C/C++ practices. Run `pre-commit install` to install the hook and `pre-commit` will automatically run `cargo fmt`, `cargo clippy`, and `clang-format` for you.
+
+Several things to note:
+
+- If `cargo fmt`, `cargo clippy`, or `clang-format` returns an error, the pre-commit hook will fail. You should fix all errors before trying to commit again.
+- If `cargo fmt` or `clang-format` reformats some files, the pre-commit hook will also fail. You should review the changes and, if satisfied, try to commit again.
 
 License
 -------

examples/ebaz4205/device_db.py

@@ -0,0 +1,78 @@
+core_addr = "192.168.1.57"
+
+device_db = {
+    "core": {
+        "type": "local",
+        "module": "artiq.coredevice.core",
+        "class": "Core",
+        "arguments": {
+            "host": core_addr,
+            "ref_period": 1e-9,
+            "target": "cortexa9",
+        },
+    },
+    "core_log": {
+        "type": "controller",
+        "host": "::1",
+        "port": 1068,
+        "command": "aqctl_corelog -p {port} --bind {bind} " + core_addr,
+    },
+    "core_moninj": {
+        "type": "controller",
+        "host": "::1",
+        "port_proxy": 1383,
+        "port": 1384,
+        "command": "aqctl_moninj_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
+        + core_addr,
+    },
+    "core_analyzer": {
+        "type": "controller",
+        "host": "::1",
+        "port_proxy": 1385,
+        "port": 1386,
+        "command": "aqctl_coreanalyzer_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
+        + core_addr,
+    },
+    "core_cache": {
+        "type": "local",
+        "module": "artiq.coredevice.cache",
+        "class": "CoreCache",
+    },
+    "core_dma": {"type": "local", "module": "artiq.coredevice.dma", "class": "CoreDMA"},
+    "led0": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 0},
+    },
+    "led1": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 1},
+    },
+}
+
+# TTLs starting at RTIO channel 2, ending at RTIO channel 15
+for i in range(2, 16):
+    device_db["ttl" + str(i)] = {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLInOut",
+        "arguments": {"channel": i},
+    }
+
+device_db.update(
+    spi0={
+        "type": "local",
+        "module": "artiq.coredevice.spi2",
+        "class": "SPIMaster",
+        "arguments": {"channel": 16},
+    },
+    dds0={
+        "type": "local",
+        "module": "artiq.coredevice.ad9834",
+        "class": "AD9834",
+        "arguments": {"spi_device": "spi0"},
+    },
+)

flake.lock

@@ -11,11 +11,11 @@
         "src-pythonparser": "src-pythonparser"
       },
       "locked": {
-        "lastModified": 1727679538,
+        "lastModified": 1732066716,
-        "narHash": "sha256-5ICdQG0cFpzmUK4np0PpWtxjMnTPoyglpcB31GyNikw=",
+        "narHash": "sha256-krjvt9+RccnAxSEZcFhRpjA2S3CoqE4MSa1JUg421b4=",
         "ref": "refs/heads/master",
-        "rev": "21cc0f7273aeec2e831eca04ace95a2681acc03c",
+        "rev": "270a417a28b516d36983779a1adb6d33a3c55a4a",
-        "revCount": 9019,
+        "revCount": 9102,
         "type": "git",
         "url": "https://github.com/m-labs/artiq.git"
       },
@@ -68,45 +68,13 @@
         "type": "github"
       }
     },
-    "mozilla-overlay": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1704373101,
-        "narHash": "sha256-+gi59LRWRQmwROrmE1E2b3mtocwueCQqZ60CwLG+gbg=",
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "rev": "9b11a87c0cc54e308fa83aac5b4ee1816d5418a2",
-        "type": "github"
-      },
-      "original": {
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "type": "github"
-      }
-    },
-    "mozilla-overlay_2": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1704373101,
-        "narHash": "sha256-+gi59LRWRQmwROrmE1E2b3mtocwueCQqZ60CwLG+gbg=",
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "rev": "9b11a87c0cc54e308fa83aac5b4ee1816d5418a2",
-        "type": "github"
-      },
-      "original": {
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "type": "github"
-      }
-    },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1727348695,
+        "lastModified": 1731319897,
-        "narHash": "sha256-J+PeFKSDV+pHL7ukkfpVzCOO7mBSrrpJ3svwBFABbhI=",
+        "narHash": "sha256-PbABj4tnbWFMfBp6OcUK5iGy1QY+/Z96ZcLpooIbuEI=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "1925c603f17fc89f4c8f6bf6f631a802ad85d784",
+        "rev": "dc460ec76cbff0e66e269457d7b728432263166c",
         "type": "github"
       },
       "original": {
@@ -119,7 +87,6 @@
     "root": {
       "inputs": {
         "artiq": "artiq",
-        "mozilla-overlay": "mozilla-overlay",
         "zynq-rs": "zynq-rs"
       }
     },
@@ -145,6 +112,28 @@
         "type": "github"
       }
     },
+    "rust-overlay_2": {
+      "inputs": {
+        "nixpkgs": [
+          "zynq-rs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1719454714,
+        "narHash": "sha256-MojqG0lyUINkEk0b3kM2drsU5vyaF8DFZe/FAlZVOGs=",
+        "owner": "oxalica",
+        "repo": "rust-overlay",
+        "rev": "d1c527659cf076ecc4b96a91c702d080b213801e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "oxalica",
+        "ref": "snapshot/2024-08-01",
+        "repo": "rust-overlay",
+        "type": "github"
+      }
+    },
     "sipyco": {
       "inputs": {
         "nixpkgs": [
@@ -153,11 +142,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1724921939,
+        "lastModified": 1728371104,
-        "narHash": "sha256-/S5iip1LHLiCP2VY7PwClDteP9ZMRZvzzKR1LZuV3fs=",
+        "narHash": "sha256-PPnAyDedUQ7Og/Cby9x5OT9wMkNGTP8GS53V6N/dk4w=",
         "owner": "m-labs",
         "repo": "sipyco",
-        "rev": "32ddd78ff3641b75054793ea0d5681c951766754",
+        "rev": "094a6cd63ffa980ef63698920170e50dc9ba77fd",
         "type": "github"
       },
       "original": {
@@ -185,11 +174,11 @@
     "src-misoc": {
       "flake": false,
       "locked": {
-        "lastModified": 1715647536,
+        "lastModified": 1729234629,
-        "narHash": "sha256-q+USDcaKHABwW56Jzq8u94iGPWlyLXMyVt0j/Gyg+IE=",
+        "narHash": "sha256-TLsTCXV5AC2xh+bS7EhBVBKqdqIU3eKrnlWcFF9LtAM=",
         "ref": "refs/heads/master",
-        "rev": "fea9de558c730bc394a5936094ae95bb9d6fa726",
+        "rev": "6085a312bca26adeca6584e37d08c8ba2e1d6e38",
-        "revCount": 2455,
+        "revCount": 2460,
         "submodules": true,
         "type": "git",
         "url": "https://github.com/m-labs/misoc.git"
@@ -233,18 +222,18 @@
     },
     "zynq-rs": {
       "inputs": {
-        "mozilla-overlay": "mozilla-overlay_2",
         "nixpkgs": [
           "artiq",
           "nixpkgs"
-        ]
+        ],
+        "rust-overlay": "rust-overlay_2"
       },
       "locked": {
-        "lastModified": 1727676910,
+        "lastModified": 1731749494,
-        "narHash": "sha256-8/4uxiGp8y5mN8l78U+pvvHjOxdlyf648B9BOuPYVrQ=",
+        "narHash": "sha256-WGigAhvVCGN5YZ1dHPyvoqAh47W1Gtph036O1aKFlLE=",
         "ref": "refs/heads/master",
-        "rev": "5ef30165540aa0749e8f3cbe74b43f8a34a3d59b",
+        "rev": "12975de2e110d7948bf47b768559f727d0abc8fc",
-        "revCount": 650,
+        "revCount": 655,
         "type": "git",
         "url": "https://git.m-labs.hk/m-labs/zynq-rs"
       },

flake.nix

@@ -2,13 +2,12 @@
   description = "ARTIQ port to the Zynq-7000 platform";
 
   inputs.artiq.url = git+https://github.com/m-labs/artiq.git;
-  inputs.mozilla-overlay = { url = github:mozilla/nixpkgs-mozilla; flake = false; };
   inputs.zynq-rs.url = git+https://git.m-labs.hk/m-labs/zynq-rs;
   inputs.zynq-rs.inputs.nixpkgs.follows = "artiq/nixpkgs";
 
-  outputs = { self, mozilla-overlay, zynq-rs, artiq }:
+  outputs = { self, zynq-rs, artiq }:
   let
-    pkgs = import artiq.inputs.nixpkgs { system = "x86_64-linux"; overlays = [ (import mozilla-overlay) ]; };
+    pkgs = import artiq.inputs.nixpkgs { system = "x86_64-linux"; overlays = [ (import zynq-rs.inputs.rust-overlay) ]; };
     zynqpkgs = zynq-rs.packages.x86_64-linux;
     artiqpkgs = artiq.packages.x86_64-linux;
     llvmPackages_11 = zynq-rs.llvmPackages_11;
@@ -364,7 +363,8 @@
       (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_satellite_100mhz"; }) //
       (board-package-set { target = "kasli_soc"; variant = "demo"; json = ./demo.json; }) //
       (board-package-set { target = "kasli_soc"; variant = "master"; json = ./kasli-soc-master.json; }) //
-      (board-package-set { target = "kasli_soc"; variant = "satellite"; json = ./kasli-soc-satellite.json; });
+      (board-package-set { target = "kasli_soc"; variant = "satellite"; json = ./kasli-soc-satellite.json; }) //
+      (board-package-set { target = "ebaz4205"; variant = "base"; });
 
     hydraJobs = packages.x86_64-linux // { inherit zc706-hitl-tests; inherit gateware-sim; inherit fmt-check; };
 
@@ -384,6 +384,7 @@
         artiqpkgs.artiq
         artiqpkgs.vivado
         binutils-arm
+        pre-commit
       ];
       XARGO_RUST_SRC = "${rust}/lib/rustlib/src/rust/library";
       CLANG_EXTRA_INCLUDE_DIR = "${llvmPackages_11.clang-unwrapped.lib}/lib/clang/11.1.0/include";

src/.clang-format

@@ -0,0 +1,32 @@
+BasedOnStyle: LLVM
+
+Language: Cpp
+Standard: Cpp11
+
+AccessModifierOffset: -1
+AlignEscapedNewlines: Left
+AlwaysBreakAfterReturnType: None
+AlwaysBreakTemplateDeclarations: Yes
+AllowAllParametersOfDeclarationOnNextLine: false
+AllowShortFunctionsOnASingleLine: Inline
+BinPackParameters: false
+BreakBeforeBinaryOperators: NonAssignment
+BreakBeforeTernaryOperators: true
+BreakConstructorInitializers: AfterColon
+BreakInheritanceList: AfterColon
+ColumnLimit: 120
+ConstructorInitializerAllOnOneLineOrOnePerLine: true
+ContinuationIndentWidth: 4
+DerivePointerAlignment: false
+IndentCaseLabels: true
+IndentPPDirectives: None
+IndentWidth: 4
+MaxEmptyLinesToKeep: 1
+PointerAlignment: Left
+ReflowComments: true
+SortIncludes: false
+SortUsingDeclarations: true
+SpaceAfterTemplateKeyword: false
+SpacesBeforeTrailingComments: 2
+TabWidth: 4
+UseTab: Never

src/.clippy.toml

@@ -0,0 +1 @@
+doc-valid-idents = ["CPython", "NumPy", ".."]

src/.pre-commit-config.yaml

@@ -0,0 +1,32 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+
+default_stages: [commit]
+
+repos:
+  - repo: local
+    hooks:
+      - id: cargo-fmt
+        name: artiq-zynq cargo format
+        entry: nix
+        language: system
+        types: [file, rust]
+        pass_filenames: false
+        description: Runs cargo fmt on the codebase.
+        args: [develop, -c, cargo, fmt, --manifest-path, src/Cargo.toml, --all]
+      - id: cargo-clippy
+        name: artiq-zynq cargo clippy
+        entry: nix
+        language: system
+        types: [file, rust]
+        pass_filenames: false
+        description: Runs cargo clippy on the codebase.
+        args: [develop, -c, cargo, clippy, --manifest-path, src/Cargo.toml, --tests]
+  - repo: https://github.com/pre-commit/mirrors-clang-format
+    rev: v19.1.0
+    hooks:
+      - id: clang-format
+        name: artiq-zynq clang-format
+        description: Runs clang-format on the codebase.
+        files: \.(cpp|h|hpp|c)$
+        args: [-style=file, -fallback-style=none, -assume-filename=src/.clang-format]

src/Cargo.lock

@@ -559,7 +559,9 @@ name = "satman"
 version = "0.0.0"
 dependencies = [
  "build_zynq",
+ "byteorder",
  "core_io",
+ "crc",
  "cslice",
  "embedded-hal",
  "io",

src/gateware/ebaz4205.py

@@ -1,23 +1,19 @@
 #!/usr/bin/env python
 
 import argparse
-from operator import itemgetter
 
+import analyzer
+import dma
+from artiq.gateware import rtio
+from artiq.gateware.rtio.phy import spi2, ttl_simple
+from artiq.gateware.rtio.xilinx_clocking import fix_serdes_timing_path
+from config import write_csr_file, write_mem_file, write_rustc_cfg_file
 from migen import *
+from migen.build.generic_platform import IOStandard, Misc, Pins, Subsignal
 from migen.build.platforms import ebaz4205
-from migen.build.generic_platform import Pins, Subsignal, IOStandard, Misc
 from migen_axi.integration.soc_core import SoCCore
 from misoc.interconnect.csr import *
 
-from artiq.gateware import rtio
-from artiq.gateware.rtio.phy import ttl_simple
-from artiq.gateware.rtio.xilinx_clocking import fix_serdes_timing_path
-
-import dma
-import analyzer
-
-from config import write_csr_file, write_mem_file, write_rustc_cfg_file
-
 _ps = [
     (
         "ps",
@@ -83,6 +79,28 @@ _i2c = [
     )
 ]
 
+_spi = [
+    (
+        "spi",
+        0,
+        Subsignal("clk", Pins("V20")),
+        Subsignal("mosi", Pins("U20")),
+        Subsignal("cs_n", Pins("P19")),
+        IOStandard("LVCMOS33"),
+    )
+]
+
+
+# Connector DATA1
+def _create_ttl():
+    _ttl = []
+
+    for idx, elem in enumerate([x for x in range(5, 21) if x not in (10, 12)]):
+        _ttl.append(
+            ("ttl", idx, Pins("DATA1:DATA1-{}".format(elem)), IOStandard("LVCMOS33")),
+        )
+    return _ttl
+
 
 class EBAZ4205(SoCCore):
     def __init__(self, rtio_clk=125e6, acpki=False):
@@ -97,6 +115,8 @@ class EBAZ4205(SoCCore):
         platform.add_extension(_ps)
         platform.add_extension(_ddr)
         platform.add_extension(_i2c)
+        platform.add_extension(_spi)
+        platform.add_extension(_create_ttl())
 
         gmii = platform.request("gmii")
         platform.add_period_constraint(gmii.rx_clk, 10)
@@ -171,6 +191,18 @@ class EBAZ4205(SoCCore):
             phy = ttl_simple.Output(user_led)
             self.submodules += phy
             self.rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        for i in range(14):
+            print("TTL at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            ttl = self.platform.request("ttl", i)
+            phy = ttl_simple.InOut(ttl)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        print("SPI at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+        spi_phy = spi2.SPIMaster(platform.request("spi"))
+        self.submodules += spi_phy
+        self.rtio_channels.append(rtio.Channel.from_phy(spi_phy, ififo_depth=4))
         self.config["RTIO_LOG_CHANNEL"] = len(self.rtio_channels)
         self.rtio_channels.append(rtio.LogChannel())
 
@@ -212,6 +244,14 @@ class EBAZ4205(SoCCore):
         self.csr_devices.append("rtio_analyzer")
 
 
+class BASE(EBAZ4205):
+    def __init__(self, rtio_clk, acpki):
+        EBAZ4205.__init__(self, rtio_clk, acpki)
+
+
+VARIANTS = {cls.__name__.lower(): cls for cls in [BASE]}
+
+
 def main():
     parser = argparse.ArgumentParser(
         description="ARTIQ port to the EBAZ4205 control card of Ebit E9+ BTC miner"
@@ -232,11 +272,25 @@ def main():
     )
     parser.add_argument("--rtio-clk", default=125e6, help="RTIO Clock Frequency (Hz)")
     parser.add_argument(
-        "--acpki", default=False, action="store_true", help="enable ACPKI"
+        "-V",
+        "--variant",
+        default="base",
+        help="variant: " "[acpki_]base" "(default: %(default)s)",
     )
     args = parser.parse_args()
 
-    soc = EBAZ4205(rtio_clk=int(args.rtio_clk), acpki=args.acpki)
+    rtio_clk = int(args.rtio_clk)
+    variant = args.variant.lower()
+    acpki = variant.startswith("acpki_")
+    if acpki:
+        variant = variant[6:]
+
+    try:
+        cls = VARIANTS[variant]
+    except KeyError:
+        raise SystemExit("Invalid variant (-V/--variant)")
+
+    soc = cls(rtio_clk=rtio_clk, acpki=acpki)
     soc.finalize()
 
     if args.r is not None:

src/libboard_artiq/src/drtio_eem.rs

@@ -185,6 +185,24 @@ unsafe fn align_comma(timer: &mut GlobalTimer) {
     }
 }
 
+pub unsafe fn align_wordslip(timer: &mut GlobalTimer, trx_no: u8) -> bool {
+    pl::csr::eem_transceiver::transceiver_sel_write(trx_no);
+
+    for slip in 0..=1 {
+        pl::csr::eem_transceiver::wordslip_write(slip as u8);
+        timer.delay_us(1);
+        pl::csr::eem_transceiver::comma_align_reset_write(1);
+        timer.delay_us(100);
+
+        if pl::csr::eem_transceiver::comma_read() == 1 {
+            debug!("comma alignment completed with {} wordslip", slip);
+            return true;
+        }
+    }
+
+    false
+}
+
 pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
     for trx_no in 0..pl::csr::CONFIG_EEM_DRTIO_COUNT {
         unsafe {
@@ -222,7 +240,6 @@ pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
 
         unsafe {
             align_comma(timer);
-            pl::csr::eem_transceiver::rx_ready_write(1);
         }
     }
 }

src/libboard_artiq/src/drtioaux_proto.rs

@@ -255,6 +255,7 @@ pub enum Packet {
         destination: u8,
         id: u32,
         run: bool,
+        timestamp: u64,
     },
     SubkernelLoadRunReply {
         destination: u8,
@@ -287,6 +288,77 @@ pub enum Packet {
     SubkernelMessageAck {
         destination: u8,
     },
+
+    CoreMgmtGetLogRequest {
+        destination: u8,
+        clear: bool,
+    },
+    CoreMgmtClearLogRequest {
+        destination: u8,
+    },
+    CoreMgmtSetLogLevelRequest {
+        destination: u8,
+        log_level: u8,
+    },
+    CoreMgmtSetUartLogLevelRequest {
+        destination: u8,
+        log_level: u8,
+    },
+    CoreMgmtConfigReadRequest {
+        destination: u8,
+        length: u16,
+        key: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigReadContinue {
+        destination: u8,
+    },
+    CoreMgmtConfigWriteRequest {
+        destination: u8,
+        last: bool,
+        length: u16,
+        data: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigRemoveRequest {
+        destination: u8,
+        length: u16,
+        key: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigEraseRequest {
+        destination: u8,
+    },
+    CoreMgmtRebootRequest {
+        destination: u8,
+    },
+    CoreMgmtAllocatorDebugRequest {
+        destination: u8,
+    },
+    CoreMgmtFlashRequest {
+        destination: u8,
+        payload_length: u32,
+    },
+    CoreMgmtFlashAddDataRequest {
+        destination: u8,
+        last: bool,
+        length: u16,
+        data: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtDropLinkAck {
+        destination: u8,
+    },
+    CoreMgmtDropLink,
+    CoreMgmtGetLogReply {
+        last: bool,
+        length: u16,
+        data: [u8; SAT_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigReadReply {
+        last: bool,
+        length: u16,
+        value: [u8; SAT_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtReply {
+        succeeded: bool,
+    },
 }
 
 impl Packet {
@@ -514,6 +586,7 @@ impl Packet {
                 destination: reader.read_u8()?,
                 id: reader.read_u32()?,
                 run: reader.read_bool()?,
+                timestamp: reader.read_u64()?,
             },
             0xc5 => Packet::SubkernelLoadRunReply {
                 destination: reader.read_u8()?,
@@ -563,6 +636,115 @@ impl Packet {
                 destination: reader.read_u8()?,
             },
 
+            0xd0 => Packet::CoreMgmtGetLogRequest {
+                destination: reader.read_u8()?,
+                clear: reader.read_bool()?,
+            },
+            0xd1 => Packet::CoreMgmtClearLogRequest {
+                destination: reader.read_u8()?,
+            },
+            0xd2 => Packet::CoreMgmtSetLogLevelRequest {
+                destination: reader.read_u8()?,
+                log_level: reader.read_u8()?,
+            },
+            0xd3 => Packet::CoreMgmtSetUartLogLevelRequest {
+                destination: reader.read_u8()?,
+                log_level: reader.read_u8()?,
+            },
+            0xd4 => {
+                let destination = reader.read_u8()?;
+                let length = reader.read_u16()?;
+                let mut key: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut key[0..length as usize])?;
+                Packet::CoreMgmtConfigReadRequest {
+                    destination: destination,
+                    length: length,
+                    key: key,
+                }
+            }
+            0xd5 => Packet::CoreMgmtConfigReadContinue {
+                destination: reader.read_u8()?,
+            },
+            0xd6 => {
+                let destination = reader.read_u8()?;
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut data: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut data[0..length as usize])?;
+                Packet::CoreMgmtConfigWriteRequest {
+                    destination: destination,
+                    last: last,
+                    length: length,
+                    data: data,
+                }
+            }
+            0xd7 => {
+                let destination = reader.read_u8()?;
+                let length = reader.read_u16()?;
+                let mut key: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut key[0..length as usize])?;
+                Packet::CoreMgmtConfigRemoveRequest {
+                    destination: destination,
+                    length: length,
+                    key: key,
+                }
+            }
+            0xd8 => Packet::CoreMgmtConfigEraseRequest {
+                destination: reader.read_u8()?,
+            },
+            0xd9 => Packet::CoreMgmtRebootRequest {
+                destination: reader.read_u8()?,
+            },
+            0xda => Packet::CoreMgmtAllocatorDebugRequest {
+                destination: reader.read_u8()?,
+            },
+            0xdb => Packet::CoreMgmtFlashRequest {
+                destination: reader.read_u8()?,
+                payload_length: reader.read_u32()?,
+            },
+            0xdc => {
+                let destination = reader.read_u8()?;
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut data: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut data[0..length as usize])?;
+                Packet::CoreMgmtFlashAddDataRequest {
+                    destination: destination,
+                    last: last,
+                    length: length,
+                    data: data,
+                }
+            }
+            0xdd => Packet::CoreMgmtDropLinkAck {
+                destination: reader.read_u8()?,
+            },
+            0xde => Packet::CoreMgmtDropLink,
+            0xdf => {
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut data: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut data[0..length as usize])?;
+                Packet::CoreMgmtGetLogReply {
+                    last: last,
+                    length: length,
+                    data: data,
+                }
+            }
+            0xe0 => {
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut value: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut value[0..length as usize])?;
+                Packet::CoreMgmtConfigReadReply {
+                    last: last,
+                    length: length,
+                    value: value,
+                }
+            }
+            0xe1 => Packet::CoreMgmtReply {
+                succeeded: reader.read_bool()?,
+            },
+
             ty => return Err(Error::UnknownPacket(ty)),
         })
     }
@@ -877,12 +1059,14 @@ impl Packet {
                 destination,
                 id,
                 run,
+                timestamp,
             } => {
                 writer.write_u8(0xc4)?;
                 writer.write_u8(source)?;
                 writer.write_u8(destination)?;
                 writer.write_u32(id)?;
                 writer.write_bool(run)?;
+                writer.write_u64(timestamp)?;
             }
             Packet::SubkernelLoadRunReply { destination, succeeded } => {
                 writer.write_u8(0xc5)?;
@@ -938,6 +1122,115 @@ impl Packet {
                 writer.write_u8(0xcc)?;
                 writer.write_u8(destination)?;
             }
+
+            Packet::CoreMgmtGetLogRequest { destination, clear } => {
+                writer.write_u8(0xd0)?;
+                writer.write_u8(destination)?;
+                writer.write_bool(clear)?;
+            }
+            Packet::CoreMgmtClearLogRequest { destination } => {
+                writer.write_u8(0xd1)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtSetLogLevelRequest { destination, log_level } => {
+                writer.write_u8(0xd2)?;
+                writer.write_u8(destination)?;
+                writer.write_u8(log_level)?;
+            }
+            Packet::CoreMgmtSetUartLogLevelRequest { destination, log_level } => {
+                writer.write_u8(0xd3)?;
+                writer.write_u8(destination)?;
+                writer.write_u8(log_level)?;
+            }
+            Packet::CoreMgmtConfigReadRequest {
+                destination,
+                length,
+                key,
+            } => {
+                writer.write_u8(0xd4)?;
+                writer.write_u8(destination)?;
+                writer.write_u16(length)?;
+                writer.write_all(&key[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigReadContinue { destination } => {
+                writer.write_u8(0xd5)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtConfigWriteRequest {
+                destination,
+                last,
+                length,
+                data,
+            } => {
+                writer.write_u8(0xd6)?;
+                writer.write_u8(destination)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&data[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigRemoveRequest {
+                destination,
+                length,
+                key,
+            } => {
+                writer.write_u8(0xd7)?;
+                writer.write_u8(destination)?;
+                writer.write_u16(length)?;
+                writer.write_all(&key[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigEraseRequest { destination } => {
+                writer.write_u8(0xd8)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtRebootRequest { destination } => {
+                writer.write_u8(0xd9)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtAllocatorDebugRequest { destination } => {
+                writer.write_u8(0xda)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtFlashRequest {
+                destination,
+                payload_length,
+            } => {
+                writer.write_u8(0xdb)?;
+                writer.write_u8(destination)?;
+                writer.write_u32(payload_length)?;
+            }
+            Packet::CoreMgmtFlashAddDataRequest {
+                destination,
+                last,
+                length,
+                data,
+            } => {
+                writer.write_u8(0xdc)?;
+                writer.write_u8(destination)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&data[..length as usize])?;
+            }
+            Packet::CoreMgmtDropLinkAck { destination } => {
+                writer.write_u8(0xdd)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtDropLink => writer.write_u8(0xde)?,
+            Packet::CoreMgmtGetLogReply { last, length, data } => {
+                writer.write_u8(0xdf)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&data[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigReadReply { last, length, value } => {
+                writer.write_u8(0xe0)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&value[0..length as usize])?;
+            }
+            Packet::CoreMgmtReply { succeeded } => {
+                writer.write_u8(0xe1)?;
+                writer.write_bool(succeeded)?;
+            }
         }
         Ok(())
     }
@@ -974,7 +1267,9 @@ impl Packet {
             | Packet::SubkernelLoadRunReply { .. }
             | Packet::SubkernelMessageAck { .. }
             | Packet::DmaPlaybackStatus { .. }
-            | Packet::SubkernelFinished { .. } => false,
+            | Packet::SubkernelFinished { .. }
+            | Packet::CoreMgmtDropLinkAck { .. }
+            | Packet::InjectionRequest { .. } => false,
             _ => true,
         }
     }

src/libdwarf/src/eh.rs

@@ -85,10 +85,7 @@ unsafe fn get_ttype_entry(
         encoding | DW_EH_PE_pcrel,
         ttype_base,
     )
-    .map(|v| match v {
+    .map(|v| (v != ttype_base).then(|| v as *const u8))
-        ttype_base => None,
-        ttype_entry => Some(ttype_entry as *const u8),
-    })
 }
 
 pub unsafe fn find_eh_action(

src/libksupport/src/kernel/mod.rs

@@ -81,6 +81,7 @@ pub enum Message {
         id: u32,
         destination: u8,
         run: bool,
+        timestamp: u64,
     },
     #[cfg(has_drtio)]
     SubkernelLoadRunReply {

src/libksupport/src/kernel/subkernel.rs

@@ -3,7 +3,7 @@ use alloc::vec::Vec;
 use cslice::CSlice;
 
 use super::{Message, SubkernelStatus, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0};
-use crate::{artiq_raise, eh_artiq, rpc::send_args};
+use crate::{artiq_raise, eh_artiq, rpc::send_args, rtio::now_mu};
 
 pub extern "C" fn load_run(id: u32, destination: u8, run: bool) {
     unsafe {
@@ -14,6 +14,7 @@ pub extern "C" fn load_run(id: u32, destination: u8, run: bool) {
                 id: id,
                 destination: destination,
                 run: run,
+                timestamp: now_mu() as u64,
             });
     }
     match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {

src/runtime/Cargo.toml.tpl

@@ -21,6 +21,7 @@ cslice = "0.3"
 log = "0.4"
 embedded-hal = "0.2"
 core_io = { version = "0.1", features = ["collections"] }
+crc = { version = "1.7", default-features = false }
 byteorder = { version = "1.3", default-features = false }
 void = { version = "1", default-features = false }
 futures = { version = "0.3", default-features = false, features = ["async-await"] }

src/runtime/src/comms.rs

@@ -405,8 +405,9 @@ async fn handle_run_kernel(
                 id,
                 destination: _,
                 run,
+                timestamp,
             } => {
-                let succeeded = match subkernel::load(aux_mutex, routing_table, timer, id, run).await {
+                let succeeded = match subkernel::load(aux_mutex, routing_table, timer, id, run, timestamp).await {
                     Ok(()) => true,
                     Err(e) => {
                         error!("Error loading subkernel: {:?}", e);
@@ -784,7 +785,15 @@ pub fn main(timer: GlobalTimer, cfg: Config) {
 
     let cfg = Rc::new(cfg);
     let restart_idle = Rc::new(Semaphore::new(1, 1));
-    mgmt::start(cfg.clone(), restart_idle.clone());
+    mgmt::start(
+        cfg.clone(),
+        restart_idle.clone(),
+        Some(mgmt::DrtioContext(
+            aux_mutex.clone(),
+            drtio_routing_table.clone(),
+            timer,
+        )),
+    );
 
     task::spawn(async move {
         let connection = Rc::new(Semaphore::new(1, 1));
@@ -910,7 +919,7 @@ pub fn soft_panic_main(timer: GlobalTimer, cfg: Config) -> ! {
     Sockets::init(32);
 
     let dummy = Rc::new(Semaphore::new(0, 1));
-    mgmt::start(Rc::new(cfg), dummy);
+    mgmt::start(Rc::new(cfg), dummy, None);
 
     // getting eth settings disables the LED as it resets GPIO
     // need to re-enable it here

src/runtime/src/rtio_clocking.rs

@@ -12,9 +12,9 @@ use libboard_artiq::si549;
 use libboard_zynq::i2c::I2c;
 use libboard_zynq::timer::GlobalTimer;
 use libconfig::Config;
-use log::warn;
+use log::{info, warn};
-#[cfg(not(feature = "target_ebaz4205"))]
+#[cfg(feature = "target_ebaz4205")]
-use log::info;
+use {libboard_zynq::slcr, libregister::RegisterRW};
 
 #[derive(Debug, PartialEq, Copy, Clone)]
 #[allow(non_camel_case_types)]
@@ -410,6 +410,38 @@ fn get_si549_setting(clk: RtioClock) -> si549::FrequencySetting {
     }
 }
 
+#[cfg(feature = "target_ebaz4205")]
+fn set_fclk0_freq(clk: RtioClock, cfg: &Config) {
+    let io_pll_freq: u32 = 1_000_000_000; // Hardcoded in zynq-rs
+    let mut target_freq = 0;
+    let mut divisor0 = 1u8;
+
+    match clk {
+        RtioClock::Int_100 => {
+            target_freq = 100_000_000;
+            divisor0 = 10;
+        }
+        RtioClock::Int_125 => {
+            target_freq = 125_000_000;
+            divisor0 = 8;
+        }
+        _ => {
+            warn!("Unsupported RTIO Clock: '{:?}'", clk);
+            return;
+        }
+    }
+
+    slcr::RegisterBlock::unlocked(|slcr| {
+        slcr.fpga0_clk_ctrl.modify(|_, w| w.divisor0(divisor0));
+    });
+
+    info!(
+        "Set FCLK0 to {:.2} MHz (target: {} MHz).",
+        io_pll_freq as f64 / divisor0 as f64,
+        target_freq / 1_000_000
+    );
+}
+
 pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
     let clk = get_rtio_clock_cfg(cfg);
     #[cfg(has_si5324)]
@@ -436,6 +468,16 @@ pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
     #[cfg(not(any(has_drtio, feature = "target_ebaz4205")))]
     init_rtio(timer);
 
+    #[cfg(feature = "target_ebaz4205")]
+    {
+        match clk {
+            RtioClock::Int_100 | RtioClock::Int_125 => {
+                set_fclk0_freq(clk, cfg);
+            }
+            _ => {} // Not set for external clocks
+        }
+    }
+
     #[cfg(all(has_si549, has_wrpll))]
     {
         // SYS CLK switch will reset CSRs that are used by WRPLL

src/runtime/src/rtio_mgt.rs

@@ -13,9 +13,13 @@ pub mod drtio {
     use core::fmt;
 
     use embedded_hal::blocking::delay::DelayMs;
+    #[cfg(has_drtio_eem)]
+    use embedded_hal::blocking::delay::DelayUs;
     use ksupport::{resolve_channel_name, ASYNC_ERROR_BUSY, ASYNC_ERROR_COLLISION, ASYNC_ERROR_SEQUENCE_ERROR,
                    SEEN_ASYNC_ERRORS};
     use libasync::{delay, task};
+    #[cfg(has_drtio_eem)]
+    use libboard_artiq::drtio_eem;
     use libboard_artiq::{drtioaux::Error as DrtioError,
                          drtioaux_async,
                          drtioaux_async::Packet,
@@ -26,6 +30,10 @@ pub mod drtio {
     use super::*;
     use crate::{analyzer::remote_analyzer::RemoteBuffer, rtio_dma::remote_dma, subkernel};
 
+    #[cfg(has_drtio_eem)]
+    const DRTIO_EEM_LINKNOS: core::ops::Range<usize> =
+        (csr::DRTIO.len() - csr::CONFIG_EEM_DRTIO_COUNT as usize)..csr::DRTIO.len();
+
     #[derive(Debug, PartialEq, Eq, Clone, Copy)]
     pub enum Error {
         Timeout,
@@ -76,8 +84,18 @@ pub mod drtio {
         });
     }
 
-    async fn link_rx_up(linkno: u8) -> bool {
+    async fn link_rx_up(linkno: u8, _timer: &mut GlobalTimer) -> bool {
         let linkno = linkno as usize;
+        #[cfg(has_drtio_eem)]
+        if DRTIO_EEM_LINKNOS.contains(&linkno) {
+            let eem_trx_no = linkno - DRTIO_EEM_LINKNOS.start;
+            unsafe {
+                csr::eem_transceiver::transceiver_sel_write(eem_trx_no as u8);
+                csr::eem_transceiver::comma_align_reset_write(1);
+            }
+            _timer.delay_us(100);
+            return unsafe { csr::eem_transceiver::comma_read() == 1 };
+        }
         unsafe { (csr::DRTIO[linkno].rx_up_read)() == 1 }
     }
 
@@ -152,8 +170,8 @@ pub mod drtio {
         }
     }
 
-    async fn recv_aux_timeout(linkno: u8, timeout: u64, timer: GlobalTimer) -> Result<Packet, Error> {
+    async fn recv_aux_timeout(linkno: u8, timeout: u64, mut timer: GlobalTimer) -> Result<Packet, Error> {
-        if !link_rx_up(linkno).await {
+        if !link_rx_up(linkno, &mut timer).await {
             return Err(Error::LinkDown);
         }
         match drtioaux_async::recv_timeout(linkno, Some(timeout), timer).await {
@@ -168,9 +186,9 @@ pub mod drtio {
         linkno: u8,
         routing_table: &RoutingTable,
         request: &Packet,
-        timer: GlobalTimer,
+        mut timer: GlobalTimer,
     ) -> Result<Packet, Error> {
-        if !link_rx_up(linkno).await {
+        if !link_rx_up(linkno, &mut timer).await {
             return Err(Error::LinkDown);
         }
         let _lock = aux_mutex.async_lock().await;
@@ -194,11 +212,11 @@ pub mod drtio {
         aux_mutex: &Rc<Mutex<bool>>,
         linkno: u8,
         routing_table: &RoutingTable,
-        timer: GlobalTimer,
+        mut timer: GlobalTimer,
     ) -> u32 {
         let mut count = 0;
         loop {
-            if !link_rx_up(linkno).await {
+            if !link_rx_up(linkno, &mut timer).await {
                 return 0;
             }
             count += 1;
@@ -462,7 +480,7 @@ pub mod drtio {
         aux_mutex: &Rc<Mutex<bool>>,
         routing_table: &RoutingTable,
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
-        timer: GlobalTimer,
+        mut timer: GlobalTimer,
     ) {
         let mut up_links = [false; csr::DRTIO.len()];
         loop {
@@ -470,16 +488,35 @@ pub mod drtio {
                 let linkno = linkno as u8;
                 if up_links[linkno as usize] {
                     /* link was previously up */
-                    if link_rx_up(linkno).await {
+                    if link_rx_up(linkno, &mut timer).await {
                         process_unsolicited_aux(aux_mutex, linkno, routing_table).await;
                         process_local_errors(linkno).await;
                     } else {
                         info!("[LINK#{}] link is down", linkno);
                         up_links[linkno as usize] = false;
+
+                        #[cfg(has_drtio_eem)]
+                        if DRTIO_EEM_LINKNOS.contains(&(linkno as usize)) {
+                            unsafe {
+                                csr::eem_transceiver::rx_ready_write(0);
+                            }
+                            while !matches!(drtioaux_async::recv(linkno).await, Ok(None)) {}
+                        }
                     }
                 } else {
                     /* link was previously down */
-                    if link_rx_up(linkno).await {
+                    #[cfg(has_drtio_eem)]
+                    if DRTIO_EEM_LINKNOS.contains(&(linkno as usize)) {
+                        let eem_trx_no = linkno - DRTIO_EEM_LINKNOS.start as u8;
+                        if !unsafe { drtio_eem::align_wordslip(&mut timer, eem_trx_no) } {
+                            continue;
+                        }
+                        unsafe {
+                            csr::eem_transceiver::rx_ready_write(1);
+                        }
+                    }
+
+                    if link_rx_up(linkno, &mut timer).await {
                         info!("[LINK#{}] link RX became up, pinging", linkno);
                         let ping_count = ping_remote(aux_mutex, linkno, routing_table, timer).await;
                         if ping_count > 0 {
@@ -523,7 +560,7 @@ pub mod drtio {
 
         for linkno in 0..csr::DRTIO.len() {
             let linkno = linkno as u8;
-            if task::block_on(link_rx_up(linkno)) {
+            if task::block_on(link_rx_up(linkno, &mut timer)) {
                 let reply = task::block_on(aux_transact(
                     &aux_mutex,
                     linkno,
@@ -540,7 +577,7 @@ pub mod drtio {
         }
     }
 
-    async fn partition_data<PacketF, HandlerF>(
+    pub async fn partition_data<PacketF, HandlerF>(
         linkno: u8,
         aux_mutex: &Rc<Mutex<bool>>,
         routing_table: &RoutingTable,
@@ -792,6 +829,7 @@ pub mod drtio {
         id: u32,
         destination: u8,
         run: bool,
+        timestamp: u64,
     ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let reply = aux_transact(
@@ -803,6 +841,7 @@ pub mod drtio {
                 source: 0,
                 destination: destination,
                 run: run,
+                timestamp,
             },
             timer,
         )

src/runtime/src/subkernel.rs

@@ -100,12 +100,22 @@ pub async fn load(
     timer: GlobalTimer,
     id: u32,
     run: bool,
+    timestamp: u64,
 ) -> Result<(), Error> {
     if let Some(subkernel) = SUBKERNELS.async_lock().await.get_mut(&id) {
         if subkernel.state != SubkernelState::Uploaded {
             return Err(Error::IncorrectState);
         }
-        drtio::subkernel_load(aux_mutex, routing_table, timer, id, subkernel.destination, run).await?;
+        drtio::subkernel_load(
+            aux_mutex,
+            routing_table,
+            timer,
+            id,
+            subkernel.destination,
+            run,
+            timestamp,
+        )
+        .await?;
         if run {
             subkernel.state = SubkernelState::Running;
         }

src/satman/Cargo.toml.tpl

@@ -15,7 +15,9 @@ build_zynq = { path = "../libbuild_zynq" }
 
 [dependencies]
 log = { version = "0.4", default-features = false }
+byteorder = { version = "1.3", default-features = false }
 core_io = { version = "0.1", features = ["collections"] }
+crc = { version = "1.7", default-features = false }
 cslice = "0.3"
 embedded-hal = "0.2"
 

src/satman/src/main.rs

@@ -4,7 +4,9 @@
 
 #[macro_use]
 extern crate log;
+extern crate byteorder;
 extern crate core_io;
+extern crate crc;
 extern crate cslice;
 extern crate embedded_hal;
 
@@ -38,16 +40,18 @@ use libboard_artiq::{drtio_routing, drtioaux,
                      pl::csr};
 #[cfg(feature = "target_kasli_soc")]
 use libboard_zynq::error_led::ErrorLED;
-use libboard_zynq::{i2c::I2c, print, println, time::Milliseconds, timer::GlobalTimer};
+use libboard_zynq::{i2c::I2c, print, println, slcr, time::Milliseconds, timer::GlobalTimer};
 use libconfig::Config;
 use libcortex_a9::{l2c::enable_l2_cache, regs::MPIDR};
 use libregister::RegisterR;
 use libsupport_zynq::{exception_vectors, ram};
+use mgmt::Manager as CoreManager;
 use routing::Router;
 use subkernel::Manager as KernelManager;
 
 mod analyzer;
 mod dma;
+mod mgmt;
 mod repeater;
 mod routing;
 mod subkernel;
@@ -149,6 +153,7 @@ fn process_aux_packet(
     dma_manager: &mut DmaManager,
     analyzer: &mut Analyzer,
     kernel_manager: &mut KernelManager,
+    core_manager: &mut CoreManager,
     router: &mut Router,
 ) -> Result<(), drtioaux::Error> {
     // In the code below, *_chan_sel_write takes an u8 if there are fewer than 256 channels,
@@ -826,6 +831,7 @@ fn process_aux_packet(
             destination: _destination,
             id,
             run,
+            timestamp,
         } => {
             forward!(
                 router,
@@ -844,7 +850,7 @@ fn process_aux_packet(
                     // cannot run kernel while DDMA is running
                     succeeded = false;
                 } else {
-                    succeeded |= kernel_manager.run(source, id).is_ok();
+                    succeeded |= kernel_manager.run(source, id, timestamp).is_ok();
                 }
             }
             router.send(
@@ -1010,6 +1016,335 @@ fn process_aux_packet(
             }
             Ok(())
         }
+        drtioaux::Packet::CoreMgmtGetLogRequest {
+            destination: _destination,
+            clear,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+            let mut data_slice = [0; SAT_PAYLOAD_MAX_SIZE];
+            let meta = core_manager.log_get_slice(&mut data_slice, clear);
+            drtioaux::send(
+                0,
+                &drtioaux::Packet::CoreMgmtGetLogReply {
+                    last: meta.status.is_last(),
+                    length: meta.len as u16,
+                    data: data_slice,
+                },
+            )
+        }
+        drtioaux::Packet::CoreMgmtClearLogRequest {
+            destination: _destination,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+            mgmt::clear_log();
+            drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: true })
+        }
+        drtioaux::Packet::CoreMgmtSetLogLevelRequest {
+            destination: _destination,
+            log_level,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            if let Ok(level_filter) = mgmt::byte_to_level_filter(log_level) {
+                info!("Changing log level to {}", level_filter);
+                log::set_max_level(level_filter);
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: true })
+            } else {
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+            }
+        }
+        drtioaux::Packet::CoreMgmtSetUartLogLevelRequest {
+            destination: _destination,
+            log_level,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            if let Ok(level_filter) = mgmt::byte_to_level_filter(log_level) {
+                info!("Changing UART log level to {}", level_filter);
+                unsafe {
+                    logger::BufferLogger::get_logger()
+                        .as_ref()
+                        .unwrap()
+                        .set_uart_log_level(level_filter);
+                }
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: true })
+            } else {
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+            }
+        }
+        drtioaux::Packet::CoreMgmtConfigReadRequest {
+            destination: _destination,
+            length,
+            key,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            let mut value_slice = [0; SAT_PAYLOAD_MAX_SIZE];
+
+            let key_slice = &key[..length as usize];
+            if !key_slice.is_ascii() {
+                error!("invalid key");
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+            } else {
+                let key = core::str::from_utf8(key_slice).unwrap();
+                if core_manager.fetch_config_value(key).is_ok() {
+                    let meta = core_manager.get_config_value_slice(&mut value_slice);
+                    drtioaux::send(
+                        0,
+                        &drtioaux::Packet::CoreMgmtConfigReadReply {
+                            last: meta.status.is_last(),
+                            length: meta.len as u16,
+                            value: value_slice,
+                        },
+                    )
+                } else {
+                    drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+                }
+            }
+        }
+        drtioaux::Packet::CoreMgmtConfigReadContinue {
+            destination: _destination,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            let mut value_slice = [0; SAT_PAYLOAD_MAX_SIZE];
+            let meta = core_manager.get_config_value_slice(&mut value_slice);
+            drtioaux::send(
+                0,
+                &drtioaux::Packet::CoreMgmtConfigReadReply {
+                    last: meta.status.is_last(),
+                    length: meta.len as u16,
+                    value: value_slice,
+                },
+            )
+        }
+        drtioaux::Packet::CoreMgmtConfigWriteRequest {
+            destination: _destination,
+            last,
+            length,
+            data,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            core_manager.add_config_data(&data, length as usize);
+
+            let mut succeeded = true;
+            if last {
+                succeeded = core_manager.write_config().is_ok();
+                core_manager.clear_config_data();
+            }
+
+            drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded })
+        }
+        drtioaux::Packet::CoreMgmtConfigRemoveRequest {
+            destination: _destination,
+            length,
+            key,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            let key_slice = &key[..length as usize];
+            if !key_slice.is_ascii() {
+                error!("invalid key");
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+            } else {
+                let key = core::str::from_utf8(key_slice).unwrap();
+                let succeeded = core_manager.remove_config(key).is_ok();
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded })
+            }
+        }
+        drtioaux::Packet::CoreMgmtConfigEraseRequest {
+            destination: _destination,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            error!("config erase not supported on zynq device");
+            drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+        }
+        drtioaux::Packet::CoreMgmtRebootRequest {
+            destination: _destination,
+        } => {
+            info!("received reboot request");
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: true })?;
+            info!("reboot imminent");
+            slcr::reboot();
+
+            unreachable!();
+        }
+        drtioaux::Packet::CoreMgmtAllocatorDebugRequest {
+            destination: _destination,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            error!("debug allocator not supported on zynq device");
+            drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: false })
+        }
+        drtioaux::Packet::CoreMgmtFlashRequest {
+            destination: _destination,
+            payload_length,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            core_manager.allocate_image_buffer(payload_length as usize);
+            drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: true })
+        }
+        drtioaux::Packet::CoreMgmtFlashAddDataRequest {
+            destination: _destination,
+            last,
+            length,
+            data,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            core_manager.add_image_data(&data, length as usize);
+
+            if last {
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtDropLink)
+            } else {
+                drtioaux::send(0, &drtioaux::Packet::CoreMgmtReply { succeeded: true })
+            }
+        }
+        drtioaux::Packet::CoreMgmtDropLinkAck {
+            destination: _destination,
+        } => {
+            forward!(
+                router,
+                _routing_table,
+                _destination,
+                *rank,
+                *self_destination,
+                _repeaters,
+                &packet,
+                timer
+            );
+
+            unsafe {
+                csr::gt_drtio::txenable_write(0);
+            }
+            core_manager.write_image();
+            info!("reboot imminent");
+            slcr::reboot();
+            Ok(())
+        }
 
         p => {
             warn!("received unexpected aux packet: {:?}", p);
@@ -1028,6 +1363,7 @@ fn process_aux_packets(
     dma_manager: &mut DmaManager,
     analyzer: &mut Analyzer,
     kernel_manager: &mut KernelManager,
+    core_manager: &mut CoreManager,
     router: &mut Router,
 ) {
     let result = drtioaux::recv(0).and_then(|packet| {
@@ -1043,6 +1379,7 @@ fn process_aux_packets(
                 dma_manager,
                 analyzer,
                 kernel_manager,
+                core_manager,
                 router,
             )
         } else {
@@ -1239,7 +1576,7 @@ pub extern "C" fn main_core0() -> i32 {
     #[cfg(has_si549)]
     si549::helper_setup(&mut timer, &SI549_SETTINGS).expect("cannot initialize helper Si549");
 
-    let cfg = match Config::new() {
+    let mut cfg = match Config::new() {
         Ok(cfg) => cfg,
         Err(err) => {
             warn!("config initialization failed: {}", err);
@@ -1314,6 +1651,7 @@ pub extern "C" fn main_core0() -> i32 {
         let mut dma_manager = DmaManager::new();
         let mut analyzer = Analyzer::new();
         let mut kernel_manager = KernelManager::new(&mut control);
+        let mut core_manager = CoreManager::new(&mut cfg);
 
         drtioaux::reset(0);
         drtiosat_reset(false);
@@ -1331,6 +1669,7 @@ pub extern "C" fn main_core0() -> i32 {
                 &mut dma_manager,
                 &mut analyzer,
                 &mut kernel_manager,
+                &mut core_manager,
                 &mut router,
             );
             #[allow(unused_mut)]

src/satman/src/mgmt.rs

@@ -0,0 +1,149 @@
+use alloc::vec::Vec;
+
+use byteorder::{ByteOrder, NativeEndian};
+use crc::crc32;
+use io::{ProtoRead, ProtoWrite};
+use libboard_artiq::{drtioaux_proto::SAT_PAYLOAD_MAX_SIZE,
+                     logger::{BufferLogger, LogBufferRef}};
+use libconfig::Config;
+use log::{debug, error, info, warn, LevelFilter};
+
+use crate::routing::{SliceMeta, Sliceable};
+
+type Result<T> = core::result::Result<T, ()>;
+
+pub fn byte_to_level_filter(level_byte: u8) -> Result<LevelFilter> {
+    Ok(match level_byte {
+        0 => LevelFilter::Off,
+        1 => LevelFilter::Error,
+        2 => LevelFilter::Warn,
+        3 => LevelFilter::Info,
+        4 => LevelFilter::Debug,
+        5 => LevelFilter::Trace,
+        lv => {
+            error!("unknown log level: {}", lv);
+            return Err(());
+        }
+    })
+}
+
+fn get_logger_buffer() -> LogBufferRef<'static> {
+    let logger = unsafe { BufferLogger::get_logger().as_mut().unwrap() };
+    loop {
+        if let Some(buffer_ref) = logger.buffer() {
+            return buffer_ref;
+        }
+    }
+}
+
+pub fn clear_log() {
+    let mut buffer = get_logger_buffer();
+    buffer.clear();
+}
+
+pub struct Manager<'a> {
+    cfg: &'a mut Config,
+    last_log: Sliceable,
+    config_payload: Vec<u8>,
+    last_value: Sliceable,
+    image_payload: Vec<u8>,
+}
+
+impl<'a> Manager<'_> {
+    pub fn new(cfg: &mut Config) -> Manager {
+        Manager {
+            cfg: cfg,
+            last_log: Sliceable::new(0, Vec::new()),
+            config_payload: Vec::new(),
+            last_value: Sliceable::new(0, Vec::new()),
+            image_payload: Vec::new(),
+        }
+    }
+
+    pub fn log_get_slice(&mut self, data_slice: &mut [u8; SAT_PAYLOAD_MAX_SIZE], consume: bool) -> SliceMeta {
+        // Populate buffer if depleted
+        if self.last_log.at_end() {
+            let mut buffer = get_logger_buffer();
+            self.last_log.extend(buffer.extract().as_bytes());
+            if consume {
+                buffer.clear();
+            }
+        }
+
+        self.last_log.get_slice_satellite(data_slice)
+    }
+
+    pub fn fetch_config_value(&mut self, key: &str) -> Result<()> {
+        self.cfg
+            .read(&key)
+            .map(|value| {
+                debug!("got value");
+                self.last_value = Sliceable::new(0, value)
+            })
+            .map_err(|_| warn!("read error: no such key"))
+    }
+
+    pub fn get_config_value_slice(&mut self, data_slice: &mut [u8; SAT_PAYLOAD_MAX_SIZE]) -> SliceMeta {
+        self.last_value.get_slice_satellite(data_slice)
+    }
+
+    pub fn add_config_data(&mut self, data: &[u8], data_len: usize) {
+        self.config_payload.write_all(&data[..data_len]).unwrap();
+    }
+
+    pub fn clear_config_data(&mut self) {
+        self.config_payload.clear();
+    }
+
+    pub fn write_config(&mut self) -> Result<()> {
+        let mut payload = &self.config_payload[..];
+        let key = payload.read_string().map_err(|_err| error!("error on reading key"))?;
+        debug!("write key: {}", key);
+        let value = payload.read_bytes().unwrap();
+
+        self.cfg
+            .write(&key, value)
+            .map(|()| debug!("write success"))
+            .map_err(|err| error!("failed to write: {:?}", err))
+    }
+
+    pub fn remove_config(&mut self, key: &str) -> Result<()> {
+        debug!("erase key: {}", key);
+        self.cfg
+            .remove(&key)
+            .map(|()| debug!("erase success"))
+            .map_err(|err| warn!("failed to erase: {:?}", err))
+    }
+
+    pub fn allocate_image_buffer(&mut self, image_size: usize) {
+        self.image_payload = Vec::with_capacity(image_size);
+    }
+
+    pub fn add_image_data(&mut self, data: &[u8], data_len: usize) {
+        self.image_payload.extend(&data[..data_len]);
+    }
+
+    pub fn write_image(&self) {
+        let mut image = self.image_payload.clone();
+        let image_ref = &image[..];
+        let bin_len = image.len() - 4;
+
+        let (image_ref, expected_crc) = {
+            let (image_ref, crc_slice) = image_ref.split_at(bin_len);
+            (image_ref, NativeEndian::read_u32(crc_slice))
+        };
+
+        let actual_crc = crc32::checksum_ieee(image_ref);
+
+        if actual_crc == expected_crc {
+            info!("CRC passed. Writing boot image to SD card...");
+            image.truncate(bin_len);
+            self.cfg.write("boot", image).expect("failed to write boot image");
+        } else {
+            panic!(
+                "CRC failed, images have not been written to flash.\n(actual {:08x}, expected {:08x})",
+                actual_crc, expected_crc
+            );
+        }
+    }
+}

src/satman/src/routing.rs

@@ -4,7 +4,7 @@ use core::cmp::min;
 #[cfg(has_drtio_routing)]
 use libboard_artiq::pl::csr;
 use libboard_artiq::{drtio_routing, drtioaux,
-                     drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE}};
+                     drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE, SAT_PAYLOAD_MAX_SIZE}};
 
 pub struct SliceMeta {
     pub destination: u8,
@@ -58,6 +58,7 @@ impl Sliceable {
     }
 
     get_slice_fn!(get_slice_master, MASTER_PAYLOAD_MAX_SIZE);
+    get_slice_fn!(get_slice_satellite, SAT_PAYLOAD_MAX_SIZE);
 }
 
 // Packets from downstream (further satellites) are received and routed appropriately.

src/satman/src/subkernel.rs

@@ -8,7 +8,7 @@ use core_io::{Error as IoError, Write};
 use cslice::AsCSlice;
 use dma::{Error as DmaError, Manager as DmaManager};
 use io::{Cursor, ProtoWrite};
-use ksupport::{eh_artiq, kernel, rpc};
+use ksupport::{eh_artiq, kernel, rpc, rtio};
 use libboard_artiq::{drtio_routing::RoutingTable,
                      drtioaux,
                      drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE},
@@ -349,7 +349,7 @@ impl<'a> Manager<'_> {
         }
     }
 
-    pub fn run(&mut self, source: u8, id: u32) -> Result<(), Error> {
+    pub fn run(&mut self, source: u8, id: u32, timestamp: u64) -> Result<(), Error> {
         if self.session.kernel_state != KernelState::Loaded || self.session.id != id {
             self.load(id)?;
         }
@@ -359,6 +359,7 @@ impl<'a> Manager<'_> {
             csr::cri_con::selected_write(2);
         }
 
+        rtio::at_mu(timestamp as i64);
         self.control.tx.send(kernel::Message::StartRequest);
         Ok(())
     }
@@ -812,6 +813,7 @@ impl<'a> Manager<'_> {
                 id,
                 destination: sk_destination,
                 run,
+                timestamp,
             } => {
                 self.session.kernel_state = KernelState::SubkernelAwaitLoad;
                 router.route(
@@ -820,6 +822,7 @@ impl<'a> Manager<'_> {
                         destination: sk_destination,
                         id: id,
                         run: run,
+                        timestamp,
                     },
                     routing_table,
                     rank,