cargo fmt

Signed-off-by: Jonathan Coates <jonathan.coates@oxionics.com>

README.md

@@ -4,65 +4,107 @@ ARTIQ on Zynq
 How to use
 ----------
 
-1. Install the ARTIQ version that corresponds to the artiq-zynq version you are targeting.
+1. [Install ARTIQ](https://m-labs.hk/artiq/manual/installing.html). Get the corresponding version to the ``artiq-zynq`` version you are targeting.
-2. To obtain firmware binaries, select the latest successful build on [Hydra](https://nixbld.m-labs.hk/) for the targeted artiq-zynq version, or use AFWS. If using Hydra, search for the job named ``<board>-<variant>-sd`` (for example: ``zc706-nist_clock-sd`` or ``zc706-nist_qc2-sd``).
+2. To obtain firmware binaries, use AFWS or build your own; see [the ARTIQ manual](https://m-labs.hk/artiq/manual/building_developing.html) for detailed instructions or skip to "Development" below. ZC706 variants only can also be downloaded from latest successful build on [Hydra](https://nixbld.m-labs.hk/).
-3. Place the ``boot.bin`` file, obtained from Hydra's "binary distribution" download link or from AFWS, at the root of a FAT-formatted SD card.
+3. Place ``boot.bin`` file at the root ``/`` of a FAT-formatted SD card.
-4. Optionally, create a ``config.txt`` configuration file at the root of the SD card containing ``key=value`` pairs on each line. Use the ``ip``, ``ip6`` and ``mac`` keys to respectively set the IPv4, IPv6 and MAC address of the board. Configuring an IPv6 address is entirely optional. If these keys are not found, the firmware will use default values that may or may not be compatible with your network.
+4. Optionally, create a ``config.txt`` configuration file containing ``key=value`` pairs on each line and place it at the root of the SD card. See below for valid keys. The ``ip``, ``ip6`` and ``mac`` keys can be used to set networking information. If these keys are not found, the firmware will use default values which may or may not be compatible with your network.
-5. Insert the SD card into the board and set up the board to boot from the SD card. For the ZC706, this is achieved by placing the large DIP switch SW11 in the 00110 position.
+5. Insert the SD card into the board and set the board to boot from the SD card. For ZC706, this is achieved by placing the large DIP switch SW11 into the 00110 position. On Kasli-SoC, place the BOOT MODE switches to SD.
-6. Power up the board. After the firmware starts successfully, it should respond to ping at its IP addresses, and boot messages can be observed from its UART at 115200bps.
+6. Power up the board. After successful boot the firmware should respond to ping at its IP addresses. Boot output can be observed from UART at 115200bps 8-N-1.
-7. Create and use an ARTIQ device database as usual, but set ``"target": "cortexa9"`` in the arguments of the core device.
+7. Create and use an ARTIQ device database as usual.
 
 Configuration
 -------------
 
-Configuring the device is done using the ``config.txt`` text file at the root of the SD card, plus the contents of the ``config`` folder. When searching for a configuration key, the firmware first looks for a file named ``/config/[key].bin`` and, if it exists, returns the contents of that file. If not, it looks into ``/config.txt``, which contains a list of ``key=value`` pairs, one per line. The ``config`` folder allows configuration values that consist in binary data, such as the startup kernel.
+Configuring the device is done using the ``config.txt`` text file at the root of the SD card plus optionally a ``config`` folder. When searching for a configuration key, the firmware first looks for a file named ``/config/[key].bin`` and, if it exists, returns the contents of that file. If not, it looks into ``/config.txt``, which should contain a list of ``key=value`` pairs, one per line. ``config.txt`` should be used for most keys but the ``config`` folder allows for setting configuration values which consist of binary data, such as the startup kernel.
 
-The following configuration keys are available:
+The following configuration keys are available among others:
 
 - ``mac``: Ethernet MAC address.
 - ``ip``: IPv4 address.
 - ``ip6``: IPv6 address.
-- ``startup``: startup kernel in ELF format (as produced by ``artiq_compile``).
+- ``idle_kernel``: idle kernel in ELF format (as produced by ``artiq_compile``).
+- ``startup_kernel``: startup kernel in ELF format (as produced by ``artiq_compile``).
 - ``rtio_clock``: source of RTIO clock; valid values are ``ext0_bypass`` and ``int_125``.
-- ``boot``: SD card "boot.bin" file, for replacing the boot firmware/gateware. Write only.
 
-Configurations can be read/written/removed via ``artiq_coremgmt``. Config erase is
+See [ARTIQ manual](https://m-labs.hk/artiq/manual-beta/core_device.html#configuration-storage) for full list. Configurations can be read/written/removed with ``artiq_coremgmt``. Config erase is not implemented, as it isn't particularly useful.
-not implemented as it seems not very useful.
+
+For convenience, the ``boot`` key can be used with ``artiq_coremgmt`` and a ``boot.bin`` file to replace firmware/gateware in a running system. This key is read-only. When loading ``boot.bin`` onto the SD card directly, place it at the root and not in the ``config`` folder.
 
 Development instructions
 ------------------------
 
-ARTIQ on Zynq is packaged using the [Nix](https://nixos.org) Flakes system. Install Nix 2.8+ and enable flakes by adding ``experimental-features = nix-command flakes`` to ``nix.conf`` (e.g. ``~/.config/nix/nix.conf``).
+ARTIQ on Zynq is packaged using [Nix](https://nixos.org) Flakes. Install Nix 2.8+ and enable flakes by adding ``experimental-features = nix-command flakes`` to ``nix.conf`` (e.g. ``~/.config/nix/nix.conf``).
 
-Pure build with Nix and execution on a remote JTAG server:
+**Pure build with Nix:**
 
 ```shell
-nix build .#zc706-nist_clock-jtag  # or zc706-nist_qc2-jtag or zc706-nist_clock_satellite-jtag etc.
+nix build .#zc706-nist_clock-jtag  # or zc706-nist_qc2-jtag or zc706-nist_clock-sd or etc
-./remote_run.sh
 ```
 
-Impure incremental build and execution on a remote JTAG server:
+Run ``nix flake show`` to see all valid build targets. Targets suffixed with ``-jtag`` produce separate firmware and gateware files, intended for use in booting via JTAG server/Ethernet, e.g. ``./remote_run.sh -i`` with a remote JTAG server. Targets suffixed with ``-sd`` will produce ``boot.bin`` file suitable for SD card boot. ``-firmware`` and ``-gateware`` respectively build firmware and gateware only.
+
+The Kasli-SoC target requires a system description file as input. See ARTIQ manual for exact instructions or use incremental build.
+
+**Impure incremental build:**
+
+For boards with fixed variants, i.e. ZC706, etc. :
 
 ```shell
 nix develop
 cd src
-gateware/zc706.py -g ../build/gateware -V <variant> # build gateware
+gateware/<board>.py -g ../build/gateware -V <variant> # gateware
-make GWARGS="-V <variant>" <runtime/satman>    # build firmware
+make GWARGS="-V <variant>" <runtime/satman>    # firmware
-cd ..
+```
-./remote_run.sh -i
+
+For boards with system descriptions, i.e. Kasli-SoC, etc. :
+
+```shell
+nix develop
+cd src
+gateware/<board>.py -g ../build/gateware <description.json> # gateware
+make TARGET=<board> GWARGS="path/to/description.json" <runtime/satman> # firmware
+```
+
+``szl.elf`` can be obtained with:
+
+```shell
+nix build git+https://git.m-labs.hk/m-labs/zynq-rs#<board>-szl
+```
+
+To generate ``boot.bin`` use ``mkbootimage``, e.g.:
+
+```shell
+echo "the_ROM_image:
+    {
+        [bootloader]result/szl.elf
+        gateware/top.bit
+        firmware/armv7-none-eabihf/release/<runtime/satman>
+    }
+    EOF" >> boot.bif
+mkbootimage boot.bif boot.bin
 ```
 
 Notes:
 
 - The impure build process is also compatible with non-Nix systems.
-- When calling make, you need to specify both the variant and firmware type.
 - Firmware type must be either ``runtime`` for DRTIO-less or DRTIO master variants, or ``satman`` for DRTIO satellite.
-- If the board is connected to the local machine, use the ``local_run.sh`` script.
+- If the board is connected to the local machine by JTAG, use the ``local_run.sh`` script.
+- A known Xilinx hardware bug prevents repeatedly loading the bootloader over JTAG without a POR reset. If booting over JTAG, install a jumper on ``PS_POR_B`` and use the POR reset script [here](https://git.m-labs.hk/M-Labs/zynq-rs/src/branch/master/kasli_soc_por.py).
+
+Pre-Commit Hooks
+----------------
+
+You are strongly recommended to use the provided pre-commit hooks to automatically reformat files and check for non-optimal Rust/C/C++ practices. Run `pre-commit install` to install the hook and `pre-commit` will automatically run `cargo fmt`, `cargo clippy`, and `clang-format` for you.
+
+Several things to note:
+
+- If `cargo fmt`, `cargo clippy`, or `clang-format` returns an error, the pre-commit hook will fail. You should fix all errors before trying to commit again.
+- If `cargo fmt` or `clang-format` reformats some files, the pre-commit hook will also fail. You should review the changes and, if satisfied, try to commit again.
 
 License
 -------
 
-Copyright (C) 2019-2023 M-Labs Limited.
+Copyright (C) 2019-2024 M-Labs Limited.
 
 ARTIQ is free software: you can redistribute it and/or modify
 it under the terms of the GNU Lesser General Public License as published by

examples/ebaz4205/device_db.py

@@ -0,0 +1,78 @@
+core_addr = "192.168.1.57"
+
+device_db = {
+    "core": {
+        "type": "local",
+        "module": "artiq.coredevice.core",
+        "class": "Core",
+        "arguments": {
+            "host": core_addr,
+            "ref_period": 1e-9,
+            "target": "cortexa9",
+        },
+    },
+    "core_log": {
+        "type": "controller",
+        "host": "::1",
+        "port": 1068,
+        "command": "aqctl_corelog -p {port} --bind {bind} " + core_addr,
+    },
+    "core_moninj": {
+        "type": "controller",
+        "host": "::1",
+        "port_proxy": 1383,
+        "port": 1384,
+        "command": "aqctl_moninj_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
+        + core_addr,
+    },
+    "core_analyzer": {
+        "type": "controller",
+        "host": "::1",
+        "port_proxy": 1385,
+        "port": 1386,
+        "command": "aqctl_coreanalyzer_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
+        + core_addr,
+    },
+    "core_cache": {
+        "type": "local",
+        "module": "artiq.coredevice.cache",
+        "class": "CoreCache",
+    },
+    "core_dma": {"type": "local", "module": "artiq.coredevice.dma", "class": "CoreDMA"},
+    "led0": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 0},
+    },
+    "led1": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 1},
+    },
+}
+
+# TTLs starting at RTIO channel 2, ending at RTIO channel 15
+for i in range(2, 16):
+    device_db["ttl" + str(i)] = {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLInOut",
+        "arguments": {"channel": i},
+    }
+
+device_db.update(
+    spi0={
+        "type": "local",
+        "module": "artiq.coredevice.spi2",
+        "class": "SPIMaster",
+        "arguments": {"channel": 16},
+    },
+    dds0={
+        "type": "local",
+        "module": "artiq.coredevice.ad9834",
+        "class": "AD9834",
+        "arguments": {"spi_device": "spi0"},
+    },
+)

flake.lock

@@ -3,19 +3,19 @@
     "artiq": {
       "inputs": {
         "artiq-comtools": "artiq-comtools",
-        "mozilla-overlay": "mozilla-overlay",
         "nixpkgs": "nixpkgs",
+        "rust-overlay": "rust-overlay",
         "sipyco": "sipyco",
         "src-migen": "src-migen",
         "src-misoc": "src-misoc",
         "src-pythonparser": "src-pythonparser"
       },
       "locked": {
-        "lastModified": 1706785107,
+        "lastModified": 1734418848,
-        "narHash": "sha256-Uj72tqigiOCdewSSBBMg6zUpVKhwjAo1HeLJgvyZ3oc=",
+        "narHash": "sha256-FiK84edtdmpJ3FUA58XAUmDDp4oVPgupmf1CcgJ6rC0=",
         "ref": "refs/heads/master",
-        "rev": "3aaa7e04f26a495e8847e47424bfc16d76d82bf8",
+        "rev": "366bb0fc59dbe4d5f544908b0f3e31f8eb19f7c1",
-        "revCount": 8672,
+        "revCount": 9121,
         "type": "git",
         "url": "https://github.com/m-labs/artiq.git"
       },
@@ -37,11 +37,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1701573753,
+        "lastModified": 1720768567,
-        "narHash": "sha256-vhEtXjb9AM6/HnsgfVmhJQeqQ9JqysUm7iWNzTIbexs=",
+        "narHash": "sha256-3VoK7o5MtHtbHLrc6Pv+eQWFtaz5Gd/YWyV5TD3c5Ss=",
         "owner": "m-labs",
         "repo": "artiq-comtools",
-        "rev": "199bdabf4de49cb7ada8a4ac7133008e0f8434b7",
+        "rev": "f93570d8f2ed5a3cfb3e1c16ab00f2540551e994",
         "type": "github"
       },
       "original": {
@@ -55,11 +55,11 @@
         "systems": "systems"
       },
       "locked": {
-        "lastModified": 1694529238,
+        "lastModified": 1710146030,
-        "narHash": "sha256-zsNZZGTGnMOf9YpHKJqMSsa0dXbfmxeoJ7xHlrt+xmY=",
+        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
         "owner": "numtide",
         "repo": "flake-utils",
-        "rev": "ff7b65b44d01cf9ba6a71320833626af21126384",
+        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
         "type": "github"
       },
       "original": {
@@ -68,66 +68,18 @@
         "type": "github"
       }
     },
-    "mozilla-overlay": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1704373101,
-        "narHash": "sha256-+gi59LRWRQmwROrmE1E2b3mtocwueCQqZ60CwLG+gbg=",
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "rev": "9b11a87c0cc54e308fa83aac5b4ee1816d5418a2",
-        "type": "github"
-      },
-      "original": {
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "type": "github"
-      }
-    },
-    "mozilla-overlay_2": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1704373101,
-        "narHash": "sha256-+gi59LRWRQmwROrmE1E2b3mtocwueCQqZ60CwLG+gbg=",
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "rev": "9b11a87c0cc54e308fa83aac5b4ee1816d5418a2",
-        "type": "github"
-      },
-      "original": {
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "type": "github"
-      }
-    },
-    "mozilla-overlay_3": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1695805681,
-        "narHash": "sha256-1ElPLD8eFfnuIk0G52HGGpRtQZ4QPCjChRlEOfkZ5ro=",
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "rev": "6eabade97bc28d707a8b9d82ad13ef143836736e",
-        "type": "github"
-      },
-      "original": {
-        "owner": "mozilla",
-        "repo": "nixpkgs-mozilla",
-        "type": "github"
-      }
-    },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1706515015,
+        "lastModified": 1733940404,
-        "narHash": "sha256-eFfY5A7wlYy3jD/75lx6IJRueg4noE+jowl0a8lIlVo=",
+        "narHash": "sha256-Pj39hSoUA86ZePPF/UXiYHHM7hMIkios8TYG29kQT4g=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "f4a8d6d5324c327dcc2d863eb7f3cc06ad630df4",
+        "rev": "5d67ea6b4b63378b9c13be21e2ec9d1afc921713",
         "type": "github"
       },
       "original": {
         "owner": "NixOS",
-        "ref": "nixos-23.11",
+        "ref": "nixos-unstable",
         "repo": "nixpkgs",
         "type": "github"
       }
@@ -135,10 +87,53 @@
     "root": {
       "inputs": {
         "artiq": "artiq",
-        "mozilla-overlay": "mozilla-overlay_2",
         "zynq-rs": "zynq-rs"
       }
     },
+    "rust-overlay": {
+      "inputs": {
+        "nixpkgs": [
+          "artiq",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1719454714,
+        "narHash": "sha256-MojqG0lyUINkEk0b3kM2drsU5vyaF8DFZe/FAlZVOGs=",
+        "owner": "oxalica",
+        "repo": "rust-overlay",
+        "rev": "d1c527659cf076ecc4b96a91c702d080b213801e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "oxalica",
+        "ref": "snapshot/2024-08-01",
+        "repo": "rust-overlay",
+        "type": "github"
+      }
+    },
+    "rust-overlay_2": {
+      "inputs": {
+        "nixpkgs": [
+          "zynq-rs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1719454714,
+        "narHash": "sha256-MojqG0lyUINkEk0b3kM2drsU5vyaF8DFZe/FAlZVOGs=",
+        "owner": "oxalica",
+        "repo": "rust-overlay",
+        "rev": "d1c527659cf076ecc4b96a91c702d080b213801e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "oxalica",
+        "ref": "snapshot/2024-08-01",
+        "repo": "rust-overlay",
+        "type": "github"
+      }
+    },
     "sipyco": {
       "inputs": {
         "nixpkgs": [
@@ -147,11 +142,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1701572254,
+        "lastModified": 1734267097,
-        "narHash": "sha256-ixq8dlpyOytDr+d/OmW8v1Ioy9V2G2ibOlNj8GFDSq4=",
+        "narHash": "sha256-aWg7XDiOlWnkXfDbKrBn9ITR46/JXfndvYHxFJ1vN78=",
         "owner": "m-labs",
         "repo": "sipyco",
-        "rev": "cceac0df537887135f99aa6b1bdd82853f16b4d6",
+        "rev": "430978ada3fefe32de01f1b884b3031e48aaef96",
         "type": "github"
       },
       "original": {
@@ -163,11 +158,11 @@
     "src-migen": {
       "flake": false,
       "locked": {
-        "lastModified": 1702942348,
+        "lastModified": 1727677091,
-        "narHash": "sha256-gKIfHZxsv+jcgDFRW9mPqmwqbZXuRvXefkZcSFjOGHw=",
+        "narHash": "sha256-Zg3SQnTwMM/VkOGKogbPyuCC2NhLy8HB2SPEUWWNgCU=",
         "owner": "m-labs",
         "repo": "migen",
-        "rev": "50934ad10a87ade47219b796535978b9bdf24023",
+        "rev": "c19ae9f8ae162ffe2d310a92bfce53ac2a821bc8",
         "type": "github"
       },
       "original": {
@@ -179,11 +174,11 @@
     "src-misoc": {
       "flake": false,
       "locked": {
-        "lastModified": 1699352904,
+        "lastModified": 1729234629,
-        "narHash": "sha256-SglyTmXOPv8jJOjwAjJrj/WhAkItQfUbvKfUqrynwRg=",
+        "narHash": "sha256-TLsTCXV5AC2xh+bS7EhBVBKqdqIU3eKrnlWcFF9LtAM=",
         "ref": "refs/heads/master",
-        "rev": "a53859f2167c31ab5225b6c09f30cf05527b94f4",
+        "rev": "6085a312bca26adeca6584e37d08c8ba2e1d6e38",
-        "revCount": 2452,
+        "revCount": 2460,
         "submodules": true,
         "type": "git",
         "url": "https://github.com/m-labs/misoc.git"
@@ -227,18 +222,18 @@
     },
     "zynq-rs": {
       "inputs": {
-        "mozilla-overlay": "mozilla-overlay_3",
         "nixpkgs": [
           "artiq",
           "nixpkgs"
-        ]
+        ],
+        "rust-overlay": "rust-overlay_2"
       },
       "locked": {
-        "lastModified": 1702982463,
+        "lastModified": 1734668221,
-        "narHash": "sha256-jKR3drE2rsTaYGEgIdv5kUo2LOb1JyIb4tJhVuCXTTc=",
+        "narHash": "sha256-X0U2yPmlsD3VLBZQyfWv8qw04Qn0qFWIONJUPPigB0U=",
         "ref": "refs/heads/master",
-        "rev": "4168eb63a7e846863331ae4e656cfd82a867cca8",
+        "rev": "213529cf7a50aa1b2d9ffdf575e3e38202ff9bd6",
-        "revCount": 636,
+        "revCount": 666,
         "type": "git",
         "url": "https://git.m-labs.hk/m-labs/zynq-rs"
       },

flake.nix

@@ -2,26 +2,26 @@
   description = "ARTIQ port to the Zynq-7000 platform";
 
   inputs.artiq.url = git+https://github.com/m-labs/artiq.git;
-  inputs.mozilla-overlay = { url = github:mozilla/nixpkgs-mozilla; flake = false; };
   inputs.zynq-rs.url = git+https://git.m-labs.hk/m-labs/zynq-rs;
   inputs.zynq-rs.inputs.nixpkgs.follows = "artiq/nixpkgs";
 
-  outputs = { self, mozilla-overlay, zynq-rs, artiq }:
+  outputs = { self, zynq-rs, artiq }:
   let
-    pkgs = import artiq.inputs.nixpkgs { system = "x86_64-linux"; overlays = [ (import mozilla-overlay) ]; };
+    pkgs = import artiq.inputs.nixpkgs { system = "x86_64-linux"; overlays = [ (import zynq-rs.inputs.rust-overlay) ]; };
     zynqpkgs = zynq-rs.packages.x86_64-linux;
     artiqpkgs = artiq.packages.x86_64-linux;
+    zynqRev = self.sourceInfo.rev or "unknown";
 
     rust = zynq-rs.rust;
     rustPlatform = zynq-rs.rustPlatform;
 
     fastnumbers = pkgs.python3Packages.buildPythonPackage rec {
       pname = "fastnumbers";
-      version = "2.2.1";
+      version = "5.1.0";
 
       src = pkgs.python3Packages.fetchPypi {
         inherit pname version;
-        sha256 = "0j15i54p7nri6hkzn1wal9pxri4pgql01wgjccig6ar0v5jjbvsy";
+        sha256 = "sha256-4JLTP4uVwxcaL7NOV57+DFSwKQ3X+W/6onYkN2AdkKc=";
       };
     };
 
@@ -74,7 +74,7 @@
 
       propagatedBuildInputs = with pkgs.python3Packages; [ setuptools click numpy toolz jinja2 ramda artiqpkgs.migen artiqpkgs.misoc ];
 
-      checkInputs = with pkgs.python3Packages; [ pytest-runner pytestCheckHook pytest-timeout ];
+      checkInputs = with pkgs.python3Packages; [ pytestCheckHook pytest-timeout ];
 
       # migen/misoc version checks are broken with pyproject for some reason
       postPatch = ''
@@ -113,7 +113,7 @@
       "nist_clock_satellite" "nist_qc2_satellite" "acpki_nist_clock_satellite" "acpki_nist_qc2_satellite" 
       "nist_clock_satellite_100mhz" "nist_qc2_satellite_100mhz" "acpki_nist_clock_satellite_100mhz" "acpki_nist_qc2_satellite_100mhz"
     ];
-    build = { target, variant, json ? null }: let
+    board-package-set = { target, variant, json ? null }: let
       szl = zynqpkgs."${target}-szl";
       fsbl = zynqpkgs."${target}-fsbl";
       fwtype = if builtins.elem variant sat_variants then "satman" else "runtime";
@@ -125,6 +125,9 @@
           lockFile = src/Cargo.lock;
           outputHashes = {
               "tar-no-std-0.1.8" = "sha256-xm17108v4smXOqxdLvHl9CxTCJslmeogjm4Y87IXFuM=";
+              "nalgebra-0.32.6" = "sha256-ZbQQZbM3A5cJ4QbujtUxkrI0/qGlI4UzfahtyQnvMZA=";
+              "core_io-0.1.0" = "sha256-0HINFWRiJx8pjMgUOL/CS336ih7SENSRh3Kah9LPRrw=";
+              "fatfs-0.3.6" = "sha256-Nz9hCq/1YgSXF8ltJ5ZawV0Hc8WV44KNK0tJdVnNb4U=";
             };
         };
 
@@ -132,12 +135,13 @@
           pkgs.gnumake
           (pkgs.python3.withPackages(ps: [ ps.jsonschema artiqpkgs.migen migen-axi artiqpkgs.misoc artiqpkgs.artiq ]))
           zynqpkgs.cargo-xbuild
-          pkgs.llvmPackages_9.llvm
+          pkgs.llvmPackages_13.llvm
-          pkgs.llvmPackages_9.clang-unwrapped
+          pkgs.llvmPackages_13.clang-unwrapped
         ];
         buildPhase = ''
+          export ZYNQ_REV=${zynqRev}
           export XARGO_RUST_SRC="${rust}/lib/rustlib/src/rust/library"
-          export CLANG_EXTRA_INCLUDE_DIR="${pkgs.llvmPackages_9.clang-unwrapped.lib}/lib/clang/9.0.1/include"
+          export CLANG_EXTRA_INCLUDE_DIR="${pkgs.llvmPackages_13.clang-unwrapped.lib}/lib/clang/13.0.1/include"
           export CARGO_HOME=$(mktemp -d cargo-home.XXX)
           export ZYNQ_RS=${zynq-rs}
           make TARGET=${target} GWARGS="${if json == null then "-V ${variant}" else json}" ${fwtype}
@@ -163,6 +167,7 @@
           ];
         }
         ''
+          export ZYNQ_REV=${zynqRev}
           python ${./src/gateware}/${target}.py -g build ${if json == null then "-V ${variant}" else json}
           mkdir -p $out $out/nix-support
           cp build/top.bit $out
@@ -273,7 +278,7 @@
     };
 
     # for hitl-tests
-    zc706-nist_qc2 = (build { target = "zc706"; variant = "nist_qc2"; });
+    zc706-nist_qc2 = (board-package-set { target = "zc706"; variant = "nist_qc2"; });
     zc706-hitl-tests = pkgs.stdenv.mkDerivation {
       name = "zc706-hitl-tests";
 
@@ -340,29 +345,30 @@
       {
         inherit fastnumbers artiq-netboot ramda migen-axi binutils-arm;
       } //
-      (build { target = "zc706"; variant = "nist_clock"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock"; }) //
-      (build { target = "zc706"; variant = "nist_clock_master"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_master"; }) //
-      (build { target = "zc706"; variant = "nist_clock_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_master_100mhz"; }) //
-      (build { target = "zc706"; variant = "nist_clock_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_satellite"; }) //
-      (build { target = "zc706"; variant = "nist_clock_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_satellite_100mhz"; }) //
-      (build { target = "zc706"; variant = "nist_qc2"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2"; }) //
-      (build { target = "zc706"; variant = "nist_qc2_master"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_master"; }) //
-      (build { target = "zc706"; variant = "nist_qc2_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_master_100mhz"; }) //
-      (build { target = "zc706"; variant = "nist_qc2_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_satellite"; }) //
-      (build { target = "zc706"; variant = "nist_qc2_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_satellite_100mhz"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_clock"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_clock_master"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_master"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_clock_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_master_100mhz"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_clock_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_satellite"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_clock_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_satellite_100mhz"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_qc2"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_qc2_master"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_master"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_qc2_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_master_100mhz"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_qc2_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_satellite"; }) //
-      (build { target = "zc706"; variant = "acpki_nist_qc2_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_satellite_100mhz"; }) //
-      (build { target = "kasli_soc"; variant = "demo"; json = ./demo.json; }) //
+      (board-package-set { target = "kasli_soc"; variant = "demo"; json = ./demo.json; }) //
-      (build { target = "kasli_soc"; variant = "master"; json = ./kasli-soc-master.json; }) //
+      (board-package-set { target = "kasli_soc"; variant = "master"; json = ./kasli-soc-master.json; }) //
-      (build { target = "kasli_soc"; variant = "satellite"; json = ./kasli-soc-satellite.json; });
+      (board-package-set { target = "kasli_soc"; variant = "satellite"; json = ./kasli-soc-satellite.json; }) //
+      (board-package-set { target = "ebaz4205"; variant = "base"; });
 
     hydraJobs = packages.x86_64-linux // { inherit zc706-hitl-tests; inherit gateware-sim; inherit fmt-check; };
 
@@ -370,8 +376,8 @@
       name = "artiq-zynq-dev-shell";
       buildInputs = with pkgs; [
         rust
-        llvmPackages_9.llvm
+        llvmPackages_13.llvm
-        llvmPackages_9.clang-unwrapped
+        llvmPackages_13.clang-unwrapped
         gnumake
         cacert
         zynqpkgs.cargo-xbuild
@@ -382,15 +388,17 @@
         artiqpkgs.artiq
         artiqpkgs.vivado
         binutils-arm
+        pre-commit
       ];
+      ZYNQ_REV="${zynqRev}";
       XARGO_RUST_SRC = "${rust}/lib/rustlib/src/rust/library";
-      CLANG_EXTRA_INCLUDE_DIR = "${pkgs.llvmPackages_9.clang-unwrapped.lib}/lib/clang/9.0.1/include";
+      CLANG_EXTRA_INCLUDE_DIR = "${pkgs.llvmPackages_13.clang-unwrapped.lib}/lib/clang/13.0.1/include";
       ZYNQ_RS = "${zynq-rs}";
       OPENOCD_ZYNQ = "${zynq-rs}/openocd";
       SZL = "${zynqpkgs.szl}";
     };
 
-    makeArtiqZynqPackage = build;
+    makeArtiqZynqPackage = board-package-set;
 
   };
 }

src/.clang-format

@@ -0,0 +1,32 @@
+BasedOnStyle: LLVM
+
+Language: Cpp
+Standard: Cpp11
+
+AccessModifierOffset: -1
+AlignEscapedNewlines: Left
+AlwaysBreakAfterReturnType: None
+AlwaysBreakTemplateDeclarations: Yes
+AllowAllParametersOfDeclarationOnNextLine: false
+AllowShortFunctionsOnASingleLine: Inline
+BinPackParameters: false
+BreakBeforeBinaryOperators: NonAssignment
+BreakBeforeTernaryOperators: true
+BreakConstructorInitializers: AfterColon
+BreakInheritanceList: AfterColon
+ColumnLimit: 120
+ConstructorInitializerAllOnOneLineOrOnePerLine: true
+ContinuationIndentWidth: 4
+DerivePointerAlignment: false
+IndentCaseLabels: true
+IndentPPDirectives: None
+IndentWidth: 4
+MaxEmptyLinesToKeep: 1
+PointerAlignment: Left
+ReflowComments: true
+SortIncludes: false
+SortUsingDeclarations: true
+SpaceAfterTemplateKeyword: false
+SpacesBeforeTrailingComments: 2
+TabWidth: 4
+UseTab: Never

src/.clippy.toml

@@ -0,0 +1 @@
+doc-valid-idents = ["CPython", "NumPy", ".."]

src/.pre-commit-config.yaml

@@ -0,0 +1,32 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+
+default_stages: [commit]
+
+repos:
+  - repo: local
+    hooks:
+      - id: cargo-fmt
+        name: artiq-zynq cargo format
+        entry: nix
+        language: system
+        types: [file, rust]
+        pass_filenames: false
+        description: Runs cargo fmt on the codebase.
+        args: [develop, -c, cargo, fmt, --manifest-path, src/Cargo.toml, --all]
+      - id: cargo-clippy
+        name: artiq-zynq cargo clippy
+        entry: nix
+        language: system
+        types: [file, rust]
+        pass_filenames: false
+        description: Runs cargo clippy on the codebase.
+        args: [develop, -c, cargo, clippy, --manifest-path, src/Cargo.toml, --tests]
+  - repo: https://github.com/pre-commit/mirrors-clang-format
+    rev: v19.1.0
+    hooks:
+      - id: clang-format
+        name: artiq-zynq clang-format
+        description: Runs clang-format on the codebase.
+        files: \.(cpp|h|hpp|c)$
+        args: [-style=file, -fallback-style=none, -assume-filename=src/.clang-format]

src/Cargo.lock

@@ -2,6 +2,15 @@
 # It is not intended for manual editing.
 version = 3
 
+[[package]]
+name = "approx"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cab112f0a86d568ea0e627cc1d6be74a1e9cd55214684db5561995f6dad897c6"
+dependencies = [
+ "num-traits",
+]
+
 [[package]]
 name = "arrayvec"
 version = "0.7.4"
@@ -49,9 +58,9 @@ version = "0.0.0"
 
 [[package]]
 name = "byteorder"
-version = "1.4.3"
+version = "1.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
+checksum = "60f0b0d4c0a382d2734228fd12b5a6b5dac185c60e938026fd31b265b94f9bd2"
 
 [[package]]
 name = "cc"
@@ -73,18 +82,14 @@ checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
 
 [[package]]
 name = "compiler_builtins"
-version = "0.1.39"
+version = "0.1.49"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3748f82c7d366a0b4950257d19db685d4958d2fa27c6d164a3f069fec42b748b"
+checksum = "20b1438ef42c655665a8ab2c1c6d605a305f031d38d9be689ddfef41a20f3aa2"
 
 [[package]]
 name = "core_io"
-version = "0.1.20210325"
+version = "0.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
+source = "git+https://git.m-labs.hk/M-Labs/rs-core_io.git?rev=e9d3edf027#e9d3edf0272502b0dd6c26e8a4869c2912657615"
-checksum = "97f8932064288cc79feb4d343a399d353a6f6f001e586ece47fe518a9e8507df"
-dependencies = [
- "rustc_version",
-]
 
 [[package]]
 name = "crc"
@@ -132,9 +137,8 @@ dependencies = [
 
 [[package]]
 name = "fatfs"
-version = "0.3.5"
+version = "0.3.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
+source = "git+https://git.m-labs.hk/M-Labs/rust-fatfs.git?rev=4b5e420084#4b5e420084fd1c4a9c105680b687523909b6469c"
-checksum = "e18f80a87439240dac45d927fd8f8081b6f1e34c03e97271189fa8a8c2e96c8f"
 dependencies = [
  "bitflags",
  "byteorder",
@@ -246,6 +250,7 @@ dependencies = [
  "libsupport_zynq",
  "log",
  "log_buffer",
+ "nalgebra",
  "nb 0.1.3",
  "unwind",
  "vcell",
@@ -363,9 +368,9 @@ checksum = "822add9edb1860698b79522510da17bef885171f75aa395cff099d770c609c24"
 
 [[package]]
 name = "log"
-version = "0.4.17"
+version = "0.4.14"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
+checksum = "51b9bbe6c47d51fc3e1a9b945965946b4c44142ab8792c50835a980d362c2710"
 dependencies = [
  "cfg-if 1.0.0",
 ]
@@ -382,6 +387,19 @@ version = "0.7.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c75de51135344a4f8ed3cfe2720dc27736f7711989703a0b43aadf3753c55577"
 
+[[package]]
+name = "nalgebra"
+version = "0.32.6"
+source = "git+https://git.m-labs.hk/M-Labs/nalgebra.git?rev=ad42410ab0#ad42410ab0abb014229e3ff6bc6ccd39ca92d5d1"
+dependencies = [
+ "approx",
+ "num-complex",
+ "num-rational",
+ "num-traits",
+ "simba",
+ "typenum",
+]
+
 [[package]]
 name = "nb"
 version = "0.1.3"
@@ -397,6 +415,15 @@ version = "1.0.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "546c37ac5d9e56f55e73b677106873d9d9f5190605e41a856503623648488cae"
 
+[[package]]
+name = "num-complex"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "26873667bbbb7c5182d4a37c1add32cdf09f841af72da53318fdb81543c15085"
+dependencies = [
+ "num-traits",
+]
+
 [[package]]
 name = "num-derive"
 version = "0.3.3"
@@ -408,6 +435,26 @@ dependencies = [
  "syn",
 ]
 
+[[package]]
+name = "num-integer"
+version = "0.1.46"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f"
+dependencies = [
+ "num-traits",
+]
+
+[[package]]
+name = "num-rational"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d41702bd167c2df5520b384281bc111a4b5efcf7fbc4c9c222c815b07e0a6a6a"
+dependencies = [
+ "autocfg",
+ "num-integer",
+ "num-traits",
+]
+
 [[package]]
 name = "num-traits"
 version = "0.2.15"
@@ -415,8 +462,15 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd"
 dependencies = [
  "autocfg",
+ "libm",
 ]
 
+[[package]]
+name = "paste"
+version = "1.0.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
+
 [[package]]
 name = "pin-project-lite"
 version = "0.2.9"
@@ -461,6 +515,7 @@ dependencies = [
  "build_zynq",
  "byteorder",
  "core_io",
+ "crc",
  "cslice",
  "dwarf",
  "dyld",
@@ -486,21 +541,14 @@ dependencies = [
  "void",
 ]
 
-[[package]]
-name = "rustc_version"
-version = "0.1.7"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c5f5376ea5e30ce23c03eb77cbe4962b988deead10910c372b226388b594c084"
-dependencies = [
- "semver",
-]
-
 [[package]]
 name = "satman"
 version = "0.0.0"
 dependencies = [
  "build_zynq",
+ "byteorder",
  "core_io",
+ "crc",
  "cslice",
  "embedded-hal",
  "io",
@@ -518,10 +566,16 @@ dependencies = [
 ]
 
 [[package]]
-name = "semver"
+name = "simba"
-version = "0.1.20"
+version = "0.8.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d4f410fedcf71af0345d7607d246e7ad15faaadd49d240ee3b24e5dc21a820ac"
+checksum = "50582927ed6f77e4ac020c057f37a268fc6aebc29225050365aacbb9deeeddc4"
+dependencies = [
+ "approx",
+ "num-complex",
+ "num-traits",
+ "paste",
+]
 
 [[package]]
 name = "smoltcp"
@@ -555,6 +609,12 @@ dependencies = [
  "log",
 ]
 
+[[package]]
+name = "typenum"
+version = "1.17.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825"
+
 [[package]]
 name = "unicode-ident"
 version = "1.0.5"

src/armv7-none-eabihf.json

@@ -1,12 +1,4 @@
 {
-  "abi-blacklist": [
-    "stdcall",
-    "fastcall",
-    "vectorcall",
-    "thiscall",
-    "win64",
-    "sysv64"
-  ],
   "arch": "arm",
   "data-layout": "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64",
   "emit-debug-gdb-scripts": false,
@@ -21,7 +13,6 @@
   "os": "none",
   "panic-strategy": "abort",
   "requires-uwtable": true,
-  "force-unwind-tables": "yes",
   "relocation-model": "static",
   "target-c-int-width": "32",
   "target-endian": "little",

src/gateware/config.py

@@ -1,5 +1,15 @@
+import os
+from artiq._version import get_version
 from misoc.integration import cpu_interface
 
+
+def generate_ident(variant):
+    return "{}+{};{}".format(
+        get_version().split(".")[0],
+        os.getenv("ZYNQ_REV", default="unknown")[:8],
+        variant,
+    )
+
 def write_csr_file(soc, filename):
     with open(filename, "w") as f:
         f.write(cpu_interface.get_csr_rust(

src/gateware/ddmtd.py

@@ -0,0 +1,119 @@
+from migen import *
+from migen.genlib.cdc import PulseSynchronizer, MultiReg
+from misoc.interconnect.csr import *
+
+
+class DDMTDSampler(Module):
+    def __init__(self, cd_ref, main_clk_se):
+        self.ref_beating = Signal()
+        self.main_beating = Signal()
+
+        # # #
+
+        ref_clk = Signal()
+        self.specials +=[
+            # ISERDESE2 can only be driven from fabric via IDELAYE2 (see UG471)
+            Instance("IDELAYE2",
+                    p_DELAY_SRC="DATAIN",
+                    p_HIGH_PERFORMANCE_MODE="TRUE",
+                    p_REFCLK_FREQUENCY=208.3,   # REFCLK frequency from IDELAYCTRL
+                    p_IDELAY_VALUE=0,
+
+                    i_DATAIN=cd_ref.clk,
+
+                    o_DATAOUT=ref_clk
+            ),
+            Instance("ISERDESE2",
+                    p_IOBDELAY="IFD",   # use DDLY as input
+                    p_DATA_RATE="SDR",
+                    p_DATA_WIDTH=2,     # min is 2
+                    p_NUM_CE=1,
+
+                    i_DDLY=ref_clk,
+                    i_CE1=1,
+                    i_CLK=ClockSignal("helper"),
+                    i_CLKDIV=ClockSignal("helper"),
+
+                    o_Q1=self.ref_beating
+            ),
+            Instance("ISERDESE2",
+                    p_DATA_RATE="SDR",
+                    p_DATA_WIDTH=2,     # min is 2
+                    p_NUM_CE=1,
+
+                    i_D=main_clk_se,
+                    i_CE1=1,
+                    i_CLK=ClockSignal("helper"),
+                    i_CLKDIV=ClockSignal("helper"),
+
+                    o_Q1=self.main_beating,
+            ),
+        ]
+
+
+class DDMTDDeglitcherMedianEdge(Module):
+    def __init__(self, counter, input_signal, stable_0_period=100, stable_1_period=100):
+        self.tag = Signal(len(counter))
+        self.detect = Signal()
+
+        stable_0_counter = Signal(reset=stable_0_period - 1, max=stable_0_period)
+        stable_1_counter = Signal(reset=stable_1_period - 1, max=stable_1_period)
+
+        # # #
+
+        # Based on CERN's median edge deglitcher FSM
+        # https://white-rabbit.web.cern.ch/documents/Precise_time_and_frequency_transfer_in_a_White_Rabbit_network.pdf (p.72)
+        fsm = ClockDomainsRenamer("helper")(FSM(reset_state="WAIT_STABLE_0"))
+        self.submodules += fsm
+
+        fsm.act("WAIT_STABLE_0",
+            If(stable_0_counter != 0,
+                NextValue(stable_0_counter, stable_0_counter - 1)
+            ).Else(
+                NextValue(stable_0_counter, stable_0_period - 1),
+                NextState("WAIT_EDGE")
+            ),
+            If(input_signal,
+                NextValue(stable_0_counter, stable_0_period - 1)
+            ),
+        )
+        fsm.act("WAIT_EDGE",
+            If(input_signal,
+                NextValue(self.tag, counter),
+                NextState("GOT_EDGE")
+            )
+        )
+        fsm.act("GOT_EDGE",
+            If(stable_1_counter != 0,
+                NextValue(stable_1_counter, stable_1_counter - 1)
+            ).Else(
+                NextValue(stable_1_counter, stable_1_period - 1),
+                self.detect.eq(1),
+                NextState("WAIT_STABLE_0")
+            ),
+            If(~input_signal,
+                NextValue(self.tag, self.tag + 1),
+                NextValue(stable_1_counter, stable_1_period - 1)
+            ),
+        )
+        
+
+class DDMTD(Module):
+    def __init__(self, counter, input_signal):
+
+        # in helper clock domain
+        self.h_tag = Signal(len(counter))
+        self.h_tag_update = Signal()
+
+        # # #
+
+        deglitcher = DDMTDDeglitcherMedianEdge(counter, input_signal)
+        self.submodules += deglitcher
+
+        self.sync.helper += [
+            self.h_tag_update.eq(0),
+            If(deglitcher.detect,
+                self.h_tag_update.eq(1),
+                self.h_tag.eq(deglitcher.tag)
+               )
+        ]

src/gateware/drtio_aux_controller.py

@@ -1,12 +1,12 @@
 """Auxiliary controller, common to satellite and master"""
 
-from artiq.gateware.drtio.aux_controller import Transmitter, Receiver
+from artiq.gateware.drtio.aux_controller import (max_packet, aux_buffer_count,
+    Transmitter, Receiver)
 from migen.fhdl.simplify import FullMemoryWE
 from misoc.interconnect.csr import *
 from migen_axi.interconnect.sram import SRAM
 from migen_axi.interconnect import axi
 
-max_packet = 1024
 
 class _DRTIOAuxControllerBase(Module):
     def __init__(self, link_layer):
@@ -27,12 +27,12 @@ class DRTIOAuxControllerAxi(_DRTIOAuxControllerBase):
         tx_sdram_if = SRAM(self.transmitter.mem, read_only=False)
         rx_sdram_if = SRAM(self.receiver.mem, read_only=True)
         aw_decoder = axi.AddressDecoder(self.bus.aw,
-            [(lambda a: a[log2_int(max_packet)] == 0, tx_sdram_if.bus.aw),
+            [(lambda a: a[log2_int(max_packet*aux_buffer_count)] == 0, tx_sdram_if.bus.aw),
-             (lambda a: a[log2_int(max_packet)] == 1, rx_sdram_if.bus.aw)],
+             (lambda a: a[log2_int(max_packet*aux_buffer_count)] == 1, rx_sdram_if.bus.aw)],
             register=True)
         ar_decoder = axi.AddressDecoder(self.bus.ar,
-            [(lambda a: a[log2_int(max_packet)] == 0, tx_sdram_if.bus.ar),
+            [(lambda a: a[log2_int(max_packet*aux_buffer_count)] == 0, tx_sdram_if.bus.ar),
-             (lambda a: a[log2_int(max_packet)] == 1, rx_sdram_if.bus.ar)],
+             (lambda a: a[log2_int(max_packet*aux_buffer_count)] == 1, rx_sdram_if.bus.ar)],
             register=True)
         # unlike wb, axi address decoder only connects ar/aw lanes,
         # the rest must also be connected!
@@ -82,4 +82,4 @@ class DRTIOAuxControllerBare(_DRTIOAuxControllerBase):
         return self.receiver.mem.get_port(write_capable=False)
 
     def get_mem_size(self):
-        return max_packet
+        return max_packet*aux_buffer_count

src/gateware/ebaz4205.py

@@ -0,0 +1,307 @@
+#!/usr/bin/env python
+
+import argparse
+
+import analyzer
+import dma
+from artiq.gateware import rtio
+from artiq.gateware.rtio.phy import spi2, ttl_simple
+from artiq.gateware.rtio.xilinx_clocking import fix_serdes_timing_path
+from config import generate_ident, write_csr_file, write_mem_file, write_rustc_cfg_file
+from migen import *
+from migen.build.generic_platform import IOStandard, Misc, Pins, Subsignal
+from migen.build.platforms import ebaz4205
+from migen_axi.integration.soc_core import SoCCore
+from misoc.interconnect.csr import *
+
+_ps = [
+    (
+        "ps",
+        0,
+        Subsignal("clk", Pins("E7"), IOStandard("LVCMOS33"), Misc("SLEW=FAST")),
+        Subsignal("por_b", Pins("C7"), IOStandard("LVCMOS33"), Misc("SLEW=FAST")),
+        Subsignal("srst_b", Pins("B10"), IOStandard("LVCMOS18"), Misc("SLEW=FAST")),
+    )
+]
+
+_ddr = [
+    (
+        "ddr",
+        0,
+        Subsignal(
+            "a",
+            Pins("N2 K2 M3 K3 M4 L1 L4 K4 K1 J4 F5 G4 E4 D4 F4"),
+            IOStandard("SSTL15"),
+        ),
+        Subsignal("ba", Pins("L5 R4 J5"), IOStandard("SSTL15")),
+        Subsignal("cas_n", Pins("P5"), IOStandard("SSTL15")),
+        Subsignal("cke", Pins("N3"), IOStandard("SSTL15")),
+        Subsignal("cs_n", Pins("N1"), IOStandard("SSTL15")),
+        Subsignal("ck_n", Pins("M2"), IOStandard("DIFF_SSTL15"), Misc("SLEW=FAST")),
+        Subsignal("ck_p", Pins("L2"), IOStandard("DIFF_SSTL15"), Misc("SLEW=FAST")),
+        # Pins "T1 Y1" not connected
+        Subsignal("dm", Pins("A1 F1"), IOStandard("SSTL15_T_DCI"), Misc("SLEW=FAST")),
+        Subsignal(
+            "dq",
+            Pins("C3 B3 A2 A4 D3 D1 C1 E1 E2 E3 G3 H3 J3 H2 H1 J1"),
+            # Pins "P1 P3 R3 R1 T4 U4 U2 U3 V1 Y3 W1 Y4 Y2 W3 V2 V3" not connected
+            IOStandard("SSTL15_T_DCI"),
+            Misc("SLEW=FAST"),
+        ),
+        Subsignal(
+            "dqs_n",
+            Pins("B2 F2"),  # Pins "T2 W4" not connected
+            IOStandard("DIFF_SSTL15_T_DCI"),
+            Misc("SLEW=FAST"),
+        ),
+        Subsignal(
+            "dqs_p",
+            Pins("C2 G2"),  # Pins "R2 W5" not connected
+            IOStandard("DIFF_SSTL15_T_DCI"),
+            Misc("SLEW=FAST"),
+        ),
+        Subsignal("vrn", Pins("G5"), IOStandard("SSTL15_T_DCI"), Misc("SLEW=FAST")),
+        Subsignal("vrp", Pins("H5"), IOStandard("SSTL15_T_DCI"), Misc("SLEW=FAST")),
+        Subsignal("drst_n", Pins("B4"), IOStandard("SSTL15"), Misc("SLEW=FAST")),
+        Subsignal("odt", Pins("N5"), IOStandard("SSTL15")),
+        Subsignal("ras_n", Pins("P4"), IOStandard("SSTL15")),
+        Subsignal("we_n", Pins("M5"), IOStandard("SSTL15")),
+    )
+]
+
+# Connector J3
+_i2c = [
+    (
+        "i2c",
+        0,
+        Subsignal("scl", Pins("U12"), IOStandard("LVCMOS33")),
+        Subsignal("sda", Pins("V13"), IOStandard("LVCMOS33")),
+    )
+]
+
+_spi = [
+    (
+        "spi",
+        0,
+        Subsignal("clk", Pins("V20")),
+        Subsignal("mosi", Pins("U20")),
+        Subsignal("cs_n", Pins("P19")),
+        IOStandard("LVCMOS33"),
+    )
+]
+
+
+# Connector DATA1
+def _create_ttl():
+    _ttl = []
+
+    for idx, elem in enumerate([x for x in range(5, 21) if x not in (10, 12)]):
+        _ttl.append(
+            ("ttl", idx, Pins("DATA1:DATA1-{}".format(elem)), IOStandard("LVCMOS33")),
+        )
+    return _ttl
+
+
+class EBAZ4205(SoCCore):
+    def __init__(self, rtio_clk=125e6, acpki=False):
+        self.acpki = acpki
+
+        platform = ebaz4205.Platform()
+        platform.toolchain.bitstream_commands.extend(
+            [
+                "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
+            ]
+        )
+        platform.add_extension(_ps)
+        platform.add_extension(_ddr)
+        platform.add_extension(_i2c)
+        platform.add_extension(_spi)
+        platform.add_extension(_create_ttl())
+
+        gmii = platform.request("gmii")
+        platform.add_period_constraint(gmii.rx_clk, 10)
+        platform.add_period_constraint(gmii.tx_clk, 10)
+        platform.add_platform_command(
+            "set_property CLOCK_DEDICATED_ROUTE FALSE [get_nets gmii_tx_clk_IBUF]"
+        )
+
+        ident = generate_ident(self.__class__.__name__)
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident)
+        fix_serdes_timing_path(platform)
+        self.config["RTIO_FREQUENCY"] = str(rtio_clk / 1e6)
+        platform.add_period_constraint(self.ps7.cd_sys.clk, 10)
+
+        self.comb += [
+            self.ps7.enet0.enet.gmii.tx_clk.eq(gmii.tx_clk),
+            self.ps7.enet0.enet.gmii.rx_clk.eq(gmii.rx_clk),
+        ]
+        self.clock_domains.cd_eth_rx = ClockDomain(reset_less=False)
+        self.clock_domains.cd_eth_tx = ClockDomain(reset_less=False)
+        self.comb += [
+            ClockSignal("eth_rx").eq(gmii.rx_clk),
+            ClockSignal("eth_tx").eq(gmii.tx_clk),
+        ]
+        self.sync.eth_tx += [
+            gmii.txd.eq(self.ps7.enet0.enet.gmii.txd),
+            gmii.tx_en.eq(self.ps7.enet0.enet.gmii.tx_en),
+        ]
+        self.sync.eth_rx += [
+            self.ps7.enet0.enet.gmii.rxd.eq(gmii.rxd),
+            self.ps7.enet0.enet.gmii.rx_dv.eq(gmii.rx_dv),
+        ]
+
+        # MDIO
+        mdio = platform.request("mdio")
+        self.comb += mdio.mdc.eq(self.ps7.enet0.enet.mdio.mdc)
+        self.specials += Instance(
+            "IOBUF",
+            i_I=self.ps7.enet0.enet.mdio.o,
+            io_IO=mdio.mdio,
+            o_O=self.ps7.enet0.enet.mdio.i,
+            i_T=~self.ps7.enet0.enet.mdio.t_n,
+        )
+
+        # I2C
+        i2c = self.platform.request("i2c")
+        self.specials += [
+            # SCL
+            Instance(
+                "IOBUF",
+                i_I=self.ps7.i2c0.scl.o,
+                io_IO=i2c.scl,
+                o_O=self.ps7.i2c0.scl.i,
+                i_T=~self.ps7.i2c0.scl.t_n,
+            ),
+            # SDA
+            Instance(
+                "IOBUF",
+                i_I=self.ps7.i2c0.sda.o,
+                io_IO=i2c.sda,
+                o_O=self.ps7.i2c0.sda.i,
+                i_T=~self.ps7.i2c0.sda.t_n,
+            ),
+        ]
+
+        self.rtio_channels = []
+        for i in (0, 1):
+            print("USER LED at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            user_led = self.platform.request("user_led", i)
+            phy = ttl_simple.Output(user_led)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        for i in range(14):
+            print("TTL at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            ttl = self.platform.request("ttl", i)
+            phy = ttl_simple.InOut(ttl)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        print("SPI at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+        spi_phy = spi2.SPIMaster(platform.request("spi"))
+        self.submodules += spi_phy
+        self.rtio_channels.append(rtio.Channel.from_phy(spi_phy, ififo_depth=4))
+        self.config["RTIO_LOG_CHANNEL"] = len(self.rtio_channels)
+        self.rtio_channels.append(rtio.LogChannel())
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+        self.submodules.rtio_core = rtio.Core(self.rtio_tsc, self.rtio_channels)
+        self.csr_devices.append("rtio_core")
+        if self.acpki:
+            import acpki
+
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(
+                self.rtio_tsc,
+                bus=self.ps7.s_axi_acp,
+                user=self.ps7.s_axi_acp_user,
+                evento=self.ps7.event.o,
+            )
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.rtio.cri, self.rtio_dma.cri],
+            [self.rtio_core.cri],
+            enable_routing=True,
+        )
+        self.csr_devices.append("cri_con")
+
+        self.submodules.rtio_moninj = rtio.MonInj(self.rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(
+            self.rtio_tsc, self.rtio_core.cri, self.ps7.s_axi_hp1
+        )
+        self.csr_devices.append("rtio_analyzer")
+
+
+class BASE(EBAZ4205):
+    def __init__(self, rtio_clk, acpki):
+        EBAZ4205.__init__(self, rtio_clk, acpki)
+
+
+VARIANTS = {cls.__name__.lower(): cls for cls in [BASE]}
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="ARTIQ port to the EBAZ4205 control card of Ebit E9+ BTC miner"
+    )
+    parser.add_argument(
+        "-r", default=None, help="build Rust interface into the specified file"
+    )
+    parser.add_argument(
+        "-m", default=None, help="build Rust memory interface into the specified file"
+    )
+    parser.add_argument(
+        "-c",
+        default=None,
+        help="build Rust compiler configuration into the specified file",
+    )
+    parser.add_argument(
+        "-g", default=None, help="build gateware into the specified directory"
+    )
+    parser.add_argument("--rtio-clk", default=125e6, help="RTIO Clock Frequency (Hz)")
+    parser.add_argument(
+        "-V",
+        "--variant",
+        default="base",
+        help="variant: " "[acpki_]base" "(default: %(default)s)",
+    )
+    args = parser.parse_args()
+
+    rtio_clk = int(args.rtio_clk)
+    variant = args.variant.lower()
+    acpki = variant.startswith("acpki_")
+    if acpki:
+        variant = variant[6:]
+
+    try:
+        cls = VARIANTS[variant]
+    except KeyError:
+        raise SystemExit("Invalid variant (-V/--variant)")
+
+    soc = cls(rtio_clk=rtio_clk, acpki=acpki)
+    soc.finalize()
+
+    if args.r is not None:
+        write_csr_file(soc, args.r)
+    if args.m is not None:
+        write_mem_file(soc, args.m)
+    if args.c is not None:
+        write_rustc_cfg_file(soc, args.c)
+    if args.g is not None:
+        soc.build(build_dir=args.g)
+
+
+if __name__ == "__main__":
+    main()

src/gateware/kasli_soc.py

@@ -20,13 +20,14 @@ from artiq.gateware.drtio.transceiver import gtx_7series, eem_serdes
 from artiq.gateware.drtio.siphaser import SiPhaser7Series
 from artiq.gateware.drtio.rx_synchronizer import XilinxRXSynchronizer
 from artiq.gateware.drtio import *
+from artiq.gateware.wrpll import wrpll
 
 import dma
 import analyzer
-import acpki
+import acpki as acpki_lib
 import drtio_aux_controller
 import zynq_clocking
-from config import write_csr_file, write_mem_file, write_rustc_cfg_file
+from config import generate_ident, write_csr_file, write_mem_file, write_rustc_cfg_file
 
 eem_iostandard_dict = {
      0: "LVDS_25",
@@ -108,24 +109,18 @@ class GenericStandalone(SoCCore):
     def __init__(self, description, acpki=False):
         self.acpki = acpki
         clk_freq = description["rtio_frequency"]
+        with_wrpll = description["enable_wrpll"]
 
         platform = kasli_soc.Platform()
         platform.toolchain.bitstream_commands.extend([
             "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
         ])
-        ident = description["variant"]
+        ident = generate_ident(description["variant"])
         if self.acpki:
             ident = "acpki_" + ident
         SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
 
         self.config["HW_REV"] = description["hw_rev"]
-        
-
-        self.submodules += SMAClkinForward(self.platform)
-
-        self.config["HAS_SI5324"] = None
-        self.config["SI5324_SOFT_RESET"] = None
-
         clk_synth = platform.request("cdr_clk_clean_fabric")
         clk_synth_se = Signal()
         clk_synth_se_buf = Signal()
@@ -140,6 +135,7 @@ class GenericStandalone(SoCCore):
         ]
         fix_serdes_timing_path(platform)
         self.submodules.bootstrap = GTPBootstrapClock(self.platform, clk_freq)
+        self.config["RTIO_FREQUENCY"] = str(clk_freq/1e6)
         self.config["CLOCK_FREQUENCY"] = int(clk_freq)
 
         self.submodules.sys_crg = zynq_clocking.SYSCRG(self.platform, self.ps7, clk_synth_se_buf)
@@ -149,6 +145,23 @@ class GenericStandalone(SoCCore):
         self.crg = self.ps7 # HACK for eem_7series to find the clock
         self.crg.cd_sys = self.sys_crg.cd_sys
 
+        if with_wrpll:
+            self.submodules.wrpll_refclk = wrpll.FrequencyMultiplier(platform.request("sma_clkin"))
+            self.submodules.wrpll = wrpll.WRPLL(
+                platform=self.platform,
+                cd_ref=self.wrpll_refclk.cd_ref,
+                main_clk_se=clk_synth_se)
+            self.csr_devices.append("wrpll_refclk")
+            self.csr_devices.append("wrpll")
+            self.comb += self.ps7.core.core0.nfiq.eq(self.wrpll.ev.irq)
+            self.config["HAS_SI549"] = None
+            self.config["WRPLL_REF_CLK"] = "SMA_CLKIN"
+        else:
+            self.submodules += SMAClkinForward(self.platform)
+            self.config["HAS_SI5324"] = None
+            self.config["SI5324_SOFT_RESET"] = None
+
+
         self.rtio_channels = []
         has_grabber = any(peripheral["type"] == "grabber" for peripheral in description["peripherals"])
         if has_grabber:
@@ -171,10 +184,10 @@ class GenericStandalone(SoCCore):
 
         if self.acpki:
             self.config["KI_IMPL"] = "acp"
-            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+            self.submodules.rtio = acpki_lib.KernelInitiator(self.rtio_tsc,
-                                                         bus=self.ps7.s_axi_acp,
+                                                             bus=self.ps7.s_axi_acp,
-                                                         user=self.ps7.s_axi_acp_user,
+                                                             user=self.ps7.s_axi_acp_user,
-                                                         evento=self.ps7.event.o)
+                                                             evento=self.ps7.event.o)
             self.csr_devices.append("rtio")
         else:
             self.config["KI_IMPL"] = "csr"
@@ -207,6 +220,7 @@ class GenericStandalone(SoCCore):
 class GenericMaster(SoCCore):
     def __init__(self, description, acpki=False):
         clk_freq = description["rtio_frequency"]
+        with_wrpll = description["enable_wrpll"]
 
         has_drtio_over_eem = any(peripheral["type"] == "shuttler" for peripheral in description["peripherals"])
         self.acpki = acpki
@@ -215,15 +229,13 @@ class GenericMaster(SoCCore):
         platform.toolchain.bitstream_commands.extend([
             "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
         ])
-        ident = description["variant"]
+        ident = generate_ident(description["variant"])
         if self.acpki:
             ident = "acpki_" + ident
         SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
 
         self.config["HW_REV"] = description["hw_rev"]
 
-        self.submodules += SMAClkinForward(self.platform)
-
         data_pads = [platform.request("sfp", i) for i in range(4)]
 
         self.submodules.gt_drtio = gtx_7series.GTX(
@@ -231,6 +243,7 @@ class GenericMaster(SoCCore):
             pads=data_pads,
             clk_freq=clk_freq)
         self.csr_devices.append("gt_drtio")
+        self.config["RTIO_FREQUENCY"] = str(clk_freq/1e6)
         self.config["CLOCK_FREQUENCY"] = int(clk_freq)
 
         txout_buf = Signal()
@@ -257,8 +270,25 @@ class GenericMaster(SoCCore):
         self.comb += ext_async_rst.eq(self.sys_crg.clk_sw_fsm.o_clk_sw & ~gtx0.tx_init.done)
         self.specials += MultiReg(self.sys_crg.clk_sw_fsm.o_clk_sw & self.sys_crg.mmcm_locked, self.gt_drtio.clk_path_ready, odomain="bootstrap")
 
-        self.config["HAS_SI5324"] = None
+        if with_wrpll:
-        self.config["SI5324_SOFT_RESET"] = None
+            clk_synth = platform.request("cdr_clk_clean_fabric")
+            clk_synth_se = Signal()
+            platform.add_period_constraint(clk_synth.p, 8.0)
+            self.specials += Instance("IBUFGDS", p_DIFF_TERM="TRUE", p_IBUF_LOW_PWR="FALSE", i_I=clk_synth.p, i_IB=clk_synth.n, o_O=clk_synth_se)
+            self.submodules.wrpll_refclk = wrpll.FrequencyMultiplier(platform.request("sma_clkin"))
+            self.submodules.wrpll = wrpll.WRPLL(
+                platform=self.platform,
+                cd_ref=self.wrpll_refclk.cd_ref,
+                main_clk_se=clk_synth_se)
+            self.csr_devices.append("wrpll_refclk")
+            self.csr_devices.append("wrpll")
+            self.comb += self.ps7.core.core0.nfiq.eq(self.wrpll.ev.irq)
+            self.config["HAS_SI549"] = None
+            self.config["WRPLL_REF_CLK"] = "SMA_CLKIN"
+        else:
+            self.submodules += SMAClkinForward(self.platform)
+            self.config["HAS_SI5324"] = None
+            self.config["SI5324_SOFT_RESET"] = None
 
         self.rtio_channels = []
         has_grabber = any(peripheral["type"] == "grabber" for peripheral in description["peripherals"])
@@ -319,10 +349,10 @@ class GenericMaster(SoCCore):
 
         if self.acpki:
             self.config["KI_IMPL"] = "acp"
-            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+            self.submodules.rtio = acpki_lib.KernelInitiator(self.rtio_tsc,
-                                                         bus=self.ps7.s_axi_acp,
+                                                             bus=self.ps7.s_axi_acp,
-                                                         user=self.ps7.s_axi_acp_user,
+                                                             user=self.ps7.s_axi_acp_user,
-                                                         evento=self.ps7.event.o)
+                                                             evento=self.ps7.event.o)
             self.csr_devices.append("rtio")
         else:
             self.config["KI_IMPL"] = "csr"
@@ -400,6 +430,7 @@ class GenericMaster(SoCCore):
 class GenericSatellite(SoCCore):
     def __init__(self, description, acpki=False):
         clk_freq = description["rtio_frequency"]
+        with_wrpll = description["enable_wrpll"]
 
         self.acpki = acpki
 
@@ -407,7 +438,7 @@ class GenericSatellite(SoCCore):
         platform.toolchain.bitstream_commands.extend([
             "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
         ])
-        ident = description["variant"]
+        ident = generate_ident(description["variant"])
         if self.acpki:
             ident = "acpki_" + ident
         SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
@@ -513,10 +544,10 @@ class GenericSatellite(SoCCore):
 
         if self.acpki:
             self.config["KI_IMPL"] = "acp"
-            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+            self.submodules.rtio = acpki_lib.KernelInitiator(self.rtio_tsc,
-                                                         bus=self.ps7.s_axi_acp,
+                                                             bus=self.ps7.s_axi_acp,
-                                                         user=self.ps7.s_axi_acp_user,
+                                                             user=self.ps7.s_axi_acp_user,
-                                                         evento=self.ps7.event.o)
+                                                             evento=self.ps7.event.o)
             self.csr_devices.append("rtio")
         else:
             self.config["KI_IMPL"] = "csr"
@@ -529,7 +560,10 @@ class GenericSatellite(SoCCore):
         self.submodules.local_io = SyncRTIO(
             self.rtio_tsc, self.rtio_channels, lane_count=description["sed_lanes"]
         )
-        self.comb += self.drtiosat.async_errors.eq(self.local_io.async_errors)
+        self.comb += [
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
 
         self.submodules.cri_con = rtio.CRIInterconnectShared(
             [self.drtiosat.cri, self.rtio_dma.cri, self.rtio.cri],
@@ -551,14 +585,30 @@ class GenericSatellite(SoCCore):
         self.config["RTIO_FREQUENCY"] = str(clk_freq/1e6)
         self.config["CLOCK_FREQUENCY"] = int(clk_freq)
 
-        self.submodules.siphaser = SiPhaser7Series(
+        if with_wrpll:
-            si5324_clkin=platform.request("cdr_clk"),
+            clk_synth = platform.request("cdr_clk_clean_fabric")
-            rx_synchronizer=self.rx_synchronizer,
+            clk_synth_se = Signal()
-            ultrascale=False,
+            platform.add_period_constraint(clk_synth.p, 8.0)
-            rtio_clk_freq=self.gt_drtio.rtio_clk_freq)
+            self.specials += Instance("IBUFGDS", p_DIFF_TERM="TRUE", p_IBUF_LOW_PWR="FALSE", i_I=clk_synth.p, i_IB=clk_synth.n, o_O=clk_synth_se)
-        self.csr_devices.append("siphaser")
+            self.submodules.wrpll = wrpll.WRPLL(
-        self.config["HAS_SI5324"] = None
+                platform=self.platform,
-        self.config["SI5324_SOFT_RESET"] = None
+                cd_ref=self.gt_drtio.cd_rtio_rx0,
+                main_clk_se=clk_synth_se)
+            self.submodules.wrpll_skewtester = wrpll.SkewTester(self.rx_synchronizer)
+            self.csr_devices.append("wrpll_skewtester")
+            self.csr_devices.append("wrpll")
+            self.comb += self.ps7.core.core0.nfiq.eq(self.wrpll.ev.irq)
+            self.config["HAS_SI549"] = None
+            self.config["WRPLL_REF_CLK"] = "GT_CDR"
+        else:
+            self.submodules.siphaser = SiPhaser7Series(
+                si5324_clkin=platform.request("cdr_clk"),
+                rx_synchronizer=self.rx_synchronizer,
+                ultrascale=False,
+                rtio_clk_freq=self.gt_drtio.rtio_clk_freq)
+            self.csr_devices.append("siphaser")
+            self.config["HAS_SI5324"] = None
+            self.config["SI5324_SOFT_RESET"] = None
 
         gtx0 = self.gt_drtio.gtxs[0]
         platform.add_false_path_constraints(

src/gateware/zc706.py

@@ -25,7 +25,7 @@ import analyzer
 import acpki
 import drtio_aux_controller
 import zynq_clocking
-from config import write_csr_file, write_mem_file, write_rustc_cfg_file
+from config import generate_ident, write_csr_file, write_mem_file, write_rustc_cfg_file
 
 class SMAClkinForward(Module):
     def __init__(self, platform):
@@ -130,7 +130,7 @@ class ZC706(SoCCore):
         platform = zc706.Platform()
         prepare_zc706_platform(platform)
 
-        ident = self.__class__.__name__
+        ident = generate_ident(self.__class__.__name__)
         if self.acpki:
             ident = "acpki_" + ident
         SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
@@ -203,7 +203,7 @@ class _MasterBase(SoCCore):
 
         platform = zc706.Platform()
         prepare_zc706_platform(platform)
-        ident = self.__class__.__name__
+        ident = generate_ident(self.__class__.__name__)
         if self.acpki:
             ident = "acpki_" + ident
         SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
@@ -344,7 +344,7 @@ class _SatelliteBase(SoCCore):
 
         platform = zc706.Platform()
         prepare_zc706_platform(platform)
-        ident = self.__class__.__name__
+        ident = generate_ident(self.__class__.__name__)
         if self.acpki:
             ident = "acpki_" + ident
         SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
@@ -487,6 +487,10 @@ class _SatelliteBase(SoCCore):
         self.csr_devices.append("rtio_dma")
 
         self.submodules.local_io = SyncRTIO(self.rtio_tsc, rtio_channels)
+        self.comb += [
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
         self.submodules.cri_con = rtio.CRIInterconnectShared(
             [self.drtiosat.cri, self.rtio_dma.cri, self.rtio.cri],
             [self.local_io.cri] + self.drtio_cri,

src/libboard_artiq/Cargo.toml.tpl

@@ -10,6 +10,8 @@ name = "libboard_artiq"
 [features]
 target_zc706 = ["libboard_zynq/target_zc706", "libconfig/target_zc706"]
 target_kasli_soc = ["libboard_zynq/target_kasli_soc", "libconfig/target_kasli_soc"]
+target_ebaz4205 = ["libboard_zynq/target_ebaz4205", "libconfig/target_ebaz4205"]
+calibrate_wrpll_skew = []
 
 [build-dependencies]
 build_zynq = { path = "../libbuild_zynq" }
@@ -18,7 +20,7 @@ build_zynq = { path = "../libbuild_zynq" }
 log = "0.4"
 log_buffer = { version = "1.2" }
 crc = { version = "1.7", default-features = false }
-core_io = { version = "0.1", features = ["collections"] }
+core_io = { git = "https://git.m-labs.hk/M-Labs/rs-core_io.git", rev = "e9d3edf027", features = ["collections"] }
 embedded-hal = "0.2"
 nb = "1.0"
 void = { version = "1", default-features = false }

src/libboard_artiq/src/drtio_eem.rs

@@ -185,6 +185,24 @@ unsafe fn align_comma(timer: &mut GlobalTimer) {
     }
 }
 
+pub unsafe fn align_wordslip(timer: &mut GlobalTimer, trx_no: u8) -> bool {
+    pl::csr::eem_transceiver::transceiver_sel_write(trx_no);
+
+    for slip in 0..=1 {
+        pl::csr::eem_transceiver::wordslip_write(slip as u8);
+        timer.delay_us(1);
+        pl::csr::eem_transceiver::comma_align_reset_write(1);
+        timer.delay_us(100);
+
+        if pl::csr::eem_transceiver::comma_read() == 1 {
+            debug!("comma alignment completed with {} wordslip", slip);
+            return true;
+        }
+    }
+
+    false
+}
+
 pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
     for trx_no in 0..pl::csr::CONFIG_EEM_DRTIO_COUNT {
         unsafe {
@@ -222,7 +240,6 @@ pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
 
         unsafe {
             align_comma(timer);
-            pl::csr::eem_transceiver::rx_ready_write(1);
         }
     }
 }

src/libboard_artiq/src/drtioaux.rs

@@ -1,11 +1,12 @@
+use core::slice;
+
 use core_io::{Error as IoError, ErrorKind as IoErrorKind};
 use crc;
 use io::{proto::{ProtoRead, ProtoWrite},
          Cursor};
 use libboard_zynq::{time::Milliseconds, timer::GlobalTimer};
-use libcortex_a9::asm::dmb;
 
-pub use crate::drtioaux_proto::Packet;
+pub use crate::drtioaux_proto::{Packet, MAX_PACKET};
 use crate::{drtioaux_proto::Error as ProtocolError, mem::mem::DRTIOAUX_MEM, pl::csr::DRTIOAUX};
 
 #[derive(Debug)]
@@ -34,6 +35,15 @@ impl From<IoError> for Error {
     }
 }
 
+pub fn copy_work_buffer(src: *mut u32, dst: *mut u32, len: isize) {
+    // fix for artiq-zynq#344
+    unsafe {
+        for i in 0..(len / 4) {
+            *dst.offset(i) = *src.offset(i);
+        }
+    }
+}
+
 pub fn reset(linkno: u8) {
     let linkno = linkno as usize;
     unsafe {
@@ -56,30 +66,14 @@ pub fn has_rx_error(linkno: u8) -> bool {
     }
 }
 
-pub fn copy_work_buffer(src: *mut u32, dst: *mut u32, len: isize) {
-    // AXI writes must be 4-byte aligned (drtio proto doesn't care for that),
-    // and AXI burst reads/writes are not implemented yet in gateware
-    // thus the need for a work buffer for transmitting and copying it over
-    unsafe {
-        for i in 0..(len / 4) {
-            *dst.offset(i) = *src.offset(i);
-            //data memory barrier to prevent bursts
-            dmb();
-        }
-    }
-}
-
 fn receive<F, T>(linkno: u8, f: F) -> Result<Option<T>, Error>
 where F: FnOnce(&[u8]) -> Result<T, Error> {
     let linkidx = linkno as usize;
     unsafe {
         if (DRTIOAUX[linkidx].aux_rx_present_read)() == 1 {
-            let ptr = (DRTIOAUX_MEM[linkidx].base + DRTIOAUX_MEM[linkidx].size / 2) as *mut u32;
+            let read_ptr = (DRTIOAUX[linkidx].aux_read_pointer_read)() as usize;
-            let len = (DRTIOAUX[linkidx].aux_rx_length_read)() as usize;
+            let ptr = (DRTIOAUX_MEM[linkidx].base + DRTIOAUX_MEM[linkidx].size / 2 + read_ptr * 0x400) as *mut u32;
-            // work buffer to accomodate axi burst reads
+            let result = f(slice::from_raw_parts(ptr as *mut u8, 0x400 as usize));
-            let mut buf: [u8; 1024] = [0; 1024];
-            copy_work_buffer(ptr, buf.as_mut_ptr() as *mut u32, len as isize);
-            let result = f(&buf[0..len]);
             (DRTIOAUX[linkidx].aux_rx_present_write)(1);
             Ok(Some(result?))
         } else {
@@ -100,15 +94,15 @@ pub fn recv(linkno: u8) -> Result<Option<Packet>, Error> {
 
         let mut reader = Cursor::new(buffer);
 
-        let checksum_at = buffer.len() - 4;
+        let packet = Packet::read_from(&mut reader)?;
+        let padding = (12 - (reader.position() % 8)) % 8;
+        let checksum_at = reader.position() + padding;
         let checksum = crc::crc32::checksum_ieee(&reader.get_ref()[0..checksum_at]);
         reader.set_position(checksum_at);
         if reader.read_u32()? != checksum {
             return Err(Error::CorruptedPacket);
         }
-        reader.set_position(0);
+        Ok(packet)
-
-        Ok(Packet::read_from(&mut reader)?)
     })
 }
 
@@ -130,10 +124,8 @@ where F: FnOnce(&mut [u8]) -> Result<usize, Error> {
     unsafe {
         while (DRTIOAUX[linkno].aux_tx_read)() != 0 {}
         let ptr = DRTIOAUX_MEM[linkno].base as *mut u32;
-        let len = DRTIOAUX_MEM[linkno].size / 2;
+        let mut buf: [u8; MAX_PACKET] = [0; MAX_PACKET];
-        // work buffer, works with unaligned mem access
+        let len = f(&mut buf)?;
-        let mut buf: [u8; 1024] = [0; 1024];
-        let len = f(&mut buf[0..len])?;
         copy_work_buffer(buf.as_mut_ptr() as *mut u32, ptr, len as isize);
         (DRTIOAUX[linkno].aux_tx_length_write)(len as u16);
         (DRTIOAUX[linkno].aux_tx_write)(1);

src/libboard_artiq/src/drtioaux_async.rs

@@ -1,3 +1,5 @@
+use core::slice;
+
 use core_io::{Error as IoError, ErrorKind as IoErrorKind};
 use crc;
 use io::{proto::{ProtoRead, ProtoWrite},
@@ -7,7 +9,7 @@ use libboard_zynq::{time::Milliseconds, timer::GlobalTimer};
 use nb;
 use void::Void;
 
-pub use crate::drtioaux_proto::Packet;
+pub use crate::drtioaux_proto::{Packet, MAX_PACKET};
 use crate::{drtioaux::{copy_work_buffer, has_rx_error, Error},
             mem::mem::DRTIOAUX_MEM,
             pl::csr::DRTIOAUX};
@@ -38,12 +40,9 @@ where F: FnOnce(&[u8]) -> Result<T, Error> {
     let linkidx = linkno as usize;
     unsafe {
         if (DRTIOAUX[linkidx].aux_rx_present_read)() == 1 {
-            let ptr = (DRTIOAUX_MEM[linkidx].base + DRTIOAUX_MEM[linkidx].size / 2) as *mut u32;
+            let read_ptr = (DRTIOAUX[linkidx].aux_read_pointer_read)() as usize;
-            let len = (DRTIOAUX[linkidx].aux_rx_length_read)() as usize;
+            let ptr = (DRTIOAUX_MEM[linkidx].base + DRTIOAUX_MEM[linkidx].size / 2 + read_ptr * 0x400) as *mut u32;
-            // work buffer to accomodate axi burst reads
+            let result = f(slice::from_raw_parts(ptr as *mut u8, 0x400 as usize));
-            let mut buf: [u8; 1024] = [0; 1024];
-            copy_work_buffer(ptr, buf.as_mut_ptr() as *mut u32, len as isize);
-            let result = f(&buf[0..len]);
             (DRTIOAUX[linkidx].aux_rx_present_write)(1);
             Ok(Some(result?))
         } else {
@@ -64,15 +63,15 @@ pub async fn recv(linkno: u8) -> Result<Option<Packet>, Error> {
 
         let mut reader = Cursor::new(buffer);
 
-        let checksum_at = buffer.len() - 4;
+        let packet = Packet::read_from(&mut reader)?;
+        let padding = (12 - (reader.position() % 8)) % 8;
+        let checksum_at = reader.position() + padding;
         let checksum = crc::crc32::checksum_ieee(&reader.get_ref()[0..checksum_at]);
         reader.set_position(checksum_at);
         if reader.read_u32()? != checksum {
             return Err(Error::CorruptedPacket);
         }
-        reader.set_position(0);
+        Ok(packet)
-
-        Ok(Packet::read_from(&mut reader)?)
     })
     .await
 }
@@ -103,10 +102,8 @@ where F: FnOnce(&mut [u8]) -> Result<usize, Error> {
     unsafe {
         let _ = block_async!(tx_ready(linkno)).await;
         let ptr = DRTIOAUX_MEM[linkno].base as *mut u32;
-        let len = DRTIOAUX_MEM[linkno].size / 2;
+        let mut buf: [u8; MAX_PACKET] = [0; MAX_PACKET];
-        // work buffer, works with unaligned mem access
+        let len = f(&mut buf)?;
-        let mut buf: [u8; 1024] = [0; 1024];
-        let len = f(&mut buf[0..len])?;
         copy_work_buffer(buf.as_mut_ptr() as *mut u32, ptr, len as isize);
         (DRTIOAUX[linkno].aux_tx_length_write)(len as u16);
         (DRTIOAUX[linkno].aux_tx_write)(1);

src/libboard_artiq/src/drtioaux_proto.rs

@@ -1,9 +1,12 @@
 use core_io::{Error as IoError, Read, Write};
 use io::proto::{ProtoRead, ProtoWrite};
 
+pub const MAX_PACKET: usize = 1024;
+
 // maximum size of arbitrary payloads
 // used by satellite -> master analyzer, subkernel exceptions
-pub const SAT_PAYLOAD_MAX_SIZE: usize  = /*max size*/512 - /*CRC*/4 - /*packet ID*/1 - /*last*/1 - /*length*/2;
+pub const SAT_PAYLOAD_MAX_SIZE: usize = /*max size*/
+    MAX_PACKET - /*CRC*/4 - /*packet ID*/1 - /*last*/1 - /*length*/2;
 // used by DDMA, subkernel program data (need to provide extra ID and destination)
 pub const MASTER_PAYLOAD_MAX_SIZE: usize = SAT_PAYLOAD_MAX_SIZE - /*source*/1 - /*destination*/1 - /*ID*/4;
 
@@ -89,8 +92,6 @@ pub enum Packet {
     RoutingSetRank {
         rank: u8,
     },
-    RoutingRetrievePackets,
-    RoutingNoPackets,
     RoutingAck,
 
     MonitorRequest {
@@ -255,6 +256,7 @@ pub enum Packet {
         destination: u8,
         id: u32,
         run: bool,
+        timestamp: u64,
     },
     SubkernelLoadRunReply {
         destination: u8,
@@ -267,12 +269,14 @@ pub enum Packet {
         exception_src: u8,
     },
     SubkernelExceptionRequest {
+        source: u8,
         destination: u8,
     },
     SubkernelException {
+        destination: u8,
         last: bool,
         length: u16,
-        data: [u8; SAT_PAYLOAD_MAX_SIZE],
+        data: [u8; MASTER_PAYLOAD_MAX_SIZE],
     },
     SubkernelMessage {
         source: u8,
@@ -285,6 +289,77 @@ pub enum Packet {
     SubkernelMessageAck {
         destination: u8,
     },
+
+    CoreMgmtGetLogRequest {
+        destination: u8,
+        clear: bool,
+    },
+    CoreMgmtClearLogRequest {
+        destination: u8,
+    },
+    CoreMgmtSetLogLevelRequest {
+        destination: u8,
+        log_level: u8,
+    },
+    CoreMgmtSetUartLogLevelRequest {
+        destination: u8,
+        log_level: u8,
+    },
+    CoreMgmtConfigReadRequest {
+        destination: u8,
+        length: u16,
+        key: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigReadContinue {
+        destination: u8,
+    },
+    CoreMgmtConfigWriteRequest {
+        destination: u8,
+        last: bool,
+        length: u16,
+        data: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigRemoveRequest {
+        destination: u8,
+        length: u16,
+        key: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigEraseRequest {
+        destination: u8,
+    },
+    CoreMgmtRebootRequest {
+        destination: u8,
+    },
+    CoreMgmtAllocatorDebugRequest {
+        destination: u8,
+    },
+    CoreMgmtFlashRequest {
+        destination: u8,
+        payload_length: u32,
+    },
+    CoreMgmtFlashAddDataRequest {
+        destination: u8,
+        last: bool,
+        length: u16,
+        data: [u8; MASTER_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtDropLinkAck {
+        destination: u8,
+    },
+    CoreMgmtDropLink,
+    CoreMgmtGetLogReply {
+        last: bool,
+        length: u16,
+        data: [u8; SAT_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtConfigReadReply {
+        last: bool,
+        length: u16,
+        value: [u8; SAT_PAYLOAD_MAX_SIZE],
+    },
+    CoreMgmtReply {
+        succeeded: bool,
+    },
 }
 
 impl Packet {
@@ -325,8 +400,6 @@ impl Packet {
                 rank: reader.read_u8()?,
             },
             0x32 => Packet::RoutingAck,
-            0x33 => Packet::RoutingRetrievePackets,
-            0x34 => Packet::RoutingNoPackets,
 
             0x40 => Packet::MonitorRequest {
                 destination: reader.read_u8()?,
@@ -514,6 +587,7 @@ impl Packet {
                 destination: reader.read_u8()?,
                 id: reader.read_u32()?,
                 run: reader.read_bool()?,
+                timestamp: reader.read_u64()?,
             },
             0xc5 => Packet::SubkernelLoadRunReply {
                 destination: reader.read_u8()?,
@@ -526,14 +600,17 @@ impl Packet {
                 exception_src: reader.read_u8()?,
             },
             0xc9 => Packet::SubkernelExceptionRequest {
+                source: reader.read_u8()?,
                 destination: reader.read_u8()?,
             },
             0xca => {
+                let destination = reader.read_u8()?;
                 let last = reader.read_bool()?;
                 let length = reader.read_u16()?;
-                let mut data: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+                let mut data: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
                 reader.read_exact(&mut data[0..length as usize])?;
                 Packet::SubkernelException {
+                    destination: destination,
                     last: last,
                     length: length,
                     data: data,
@@ -560,6 +637,115 @@ impl Packet {
                 destination: reader.read_u8()?,
             },
 
+            0xd0 => Packet::CoreMgmtGetLogRequest {
+                destination: reader.read_u8()?,
+                clear: reader.read_bool()?,
+            },
+            0xd1 => Packet::CoreMgmtClearLogRequest {
+                destination: reader.read_u8()?,
+            },
+            0xd2 => Packet::CoreMgmtSetLogLevelRequest {
+                destination: reader.read_u8()?,
+                log_level: reader.read_u8()?,
+            },
+            0xd3 => Packet::CoreMgmtSetUartLogLevelRequest {
+                destination: reader.read_u8()?,
+                log_level: reader.read_u8()?,
+            },
+            0xd4 => {
+                let destination = reader.read_u8()?;
+                let length = reader.read_u16()?;
+                let mut key: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut key[0..length as usize])?;
+                Packet::CoreMgmtConfigReadRequest {
+                    destination: destination,
+                    length: length,
+                    key: key,
+                }
+            }
+            0xd5 => Packet::CoreMgmtConfigReadContinue {
+                destination: reader.read_u8()?,
+            },
+            0xd6 => {
+                let destination = reader.read_u8()?;
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut data: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut data[0..length as usize])?;
+                Packet::CoreMgmtConfigWriteRequest {
+                    destination: destination,
+                    last: last,
+                    length: length,
+                    data: data,
+                }
+            }
+            0xd7 => {
+                let destination = reader.read_u8()?;
+                let length = reader.read_u16()?;
+                let mut key: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut key[0..length as usize])?;
+                Packet::CoreMgmtConfigRemoveRequest {
+                    destination: destination,
+                    length: length,
+                    key: key,
+                }
+            }
+            0xd8 => Packet::CoreMgmtConfigEraseRequest {
+                destination: reader.read_u8()?,
+            },
+            0xd9 => Packet::CoreMgmtRebootRequest {
+                destination: reader.read_u8()?,
+            },
+            0xda => Packet::CoreMgmtAllocatorDebugRequest {
+                destination: reader.read_u8()?,
+            },
+            0xdb => Packet::CoreMgmtFlashRequest {
+                destination: reader.read_u8()?,
+                payload_length: reader.read_u32()?,
+            },
+            0xdc => {
+                let destination = reader.read_u8()?;
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut data: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut data[0..length as usize])?;
+                Packet::CoreMgmtFlashAddDataRequest {
+                    destination: destination,
+                    last: last,
+                    length: length,
+                    data: data,
+                }
+            }
+            0xdd => Packet::CoreMgmtDropLinkAck {
+                destination: reader.read_u8()?,
+            },
+            0xde => Packet::CoreMgmtDropLink,
+            0xdf => {
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut data: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut data[0..length as usize])?;
+                Packet::CoreMgmtGetLogReply {
+                    last: last,
+                    length: length,
+                    data: data,
+                }
+            }
+            0xe0 => {
+                let last = reader.read_bool()?;
+                let length = reader.read_u16()?;
+                let mut value: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+                reader.read_exact(&mut value[0..length as usize])?;
+                Packet::CoreMgmtConfigReadReply {
+                    last: last,
+                    length: length,
+                    value: value,
+                }
+            }
+            0xe1 => Packet::CoreMgmtReply {
+                succeeded: reader.read_bool()?,
+            },
+
             ty => return Err(Error::UnknownPacket(ty)),
         })
     }
@@ -602,8 +788,6 @@ impl Packet {
                 writer.write_u8(rank)?;
             }
             Packet::RoutingAck => writer.write_u8(0x32)?,
-            Packet::RoutingRetrievePackets => writer.write_u8(0x33)?,
-            Packet::RoutingNoPackets => writer.write_u8(0x34)?,
 
             Packet::MonitorRequest {
                 destination,
@@ -876,12 +1060,14 @@ impl Packet {
                 destination,
                 id,
                 run,
+                timestamp,
             } => {
                 writer.write_u8(0xc4)?;
                 writer.write_u8(source)?;
                 writer.write_u8(destination)?;
                 writer.write_u32(id)?;
                 writer.write_bool(run)?;
+                writer.write_u64(timestamp)?;
             }
             Packet::SubkernelLoadRunReply { destination, succeeded } => {
                 writer.write_u8(0xc5)?;
@@ -900,12 +1086,19 @@ impl Packet {
                 writer.write_bool(with_exception)?;
                 writer.write_u8(exception_src)?;
             }
-            Packet::SubkernelExceptionRequest { destination } => {
+            Packet::SubkernelExceptionRequest { source, destination } => {
                 writer.write_u8(0xc9)?;
+                writer.write_u8(source)?;
                 writer.write_u8(destination)?;
             }
-            Packet::SubkernelException { last, length, data } => {
+            Packet::SubkernelException {
+                destination,
+                last,
+                length,
+                data,
+            } => {
                 writer.write_u8(0xca)?;
+                writer.write_u8(destination)?;
                 writer.write_bool(last)?;
                 writer.write_u16(length)?;
                 writer.write_all(&data[0..length as usize])?;
@@ -930,6 +1123,115 @@ impl Packet {
                 writer.write_u8(0xcc)?;
                 writer.write_u8(destination)?;
             }
+
+            Packet::CoreMgmtGetLogRequest { destination, clear } => {
+                writer.write_u8(0xd0)?;
+                writer.write_u8(destination)?;
+                writer.write_bool(clear)?;
+            }
+            Packet::CoreMgmtClearLogRequest { destination } => {
+                writer.write_u8(0xd1)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtSetLogLevelRequest { destination, log_level } => {
+                writer.write_u8(0xd2)?;
+                writer.write_u8(destination)?;
+                writer.write_u8(log_level)?;
+            }
+            Packet::CoreMgmtSetUartLogLevelRequest { destination, log_level } => {
+                writer.write_u8(0xd3)?;
+                writer.write_u8(destination)?;
+                writer.write_u8(log_level)?;
+            }
+            Packet::CoreMgmtConfigReadRequest {
+                destination,
+                length,
+                key,
+            } => {
+                writer.write_u8(0xd4)?;
+                writer.write_u8(destination)?;
+                writer.write_u16(length)?;
+                writer.write_all(&key[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigReadContinue { destination } => {
+                writer.write_u8(0xd5)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtConfigWriteRequest {
+                destination,
+                last,
+                length,
+                data,
+            } => {
+                writer.write_u8(0xd6)?;
+                writer.write_u8(destination)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&data[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigRemoveRequest {
+                destination,
+                length,
+                key,
+            } => {
+                writer.write_u8(0xd7)?;
+                writer.write_u8(destination)?;
+                writer.write_u16(length)?;
+                writer.write_all(&key[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigEraseRequest { destination } => {
+                writer.write_u8(0xd8)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtRebootRequest { destination } => {
+                writer.write_u8(0xd9)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtAllocatorDebugRequest { destination } => {
+                writer.write_u8(0xda)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtFlashRequest {
+                destination,
+                payload_length,
+            } => {
+                writer.write_u8(0xdb)?;
+                writer.write_u8(destination)?;
+                writer.write_u32(payload_length)?;
+            }
+            Packet::CoreMgmtFlashAddDataRequest {
+                destination,
+                last,
+                length,
+                data,
+            } => {
+                writer.write_u8(0xdc)?;
+                writer.write_u8(destination)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&data[..length as usize])?;
+            }
+            Packet::CoreMgmtDropLinkAck { destination } => {
+                writer.write_u8(0xdd)?;
+                writer.write_u8(destination)?;
+            }
+            Packet::CoreMgmtDropLink => writer.write_u8(0xde)?,
+            Packet::CoreMgmtGetLogReply { last, length, data } => {
+                writer.write_u8(0xdf)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&data[0..length as usize])?;
+            }
+            Packet::CoreMgmtConfigReadReply { last, length, value } => {
+                writer.write_u8(0xe0)?;
+                writer.write_bool(last)?;
+                writer.write_u16(length)?;
+                writer.write_all(&value[0..length as usize])?;
+            }
+            Packet::CoreMgmtReply { succeeded } => {
+                writer.write_u8(0xe1)?;
+                writer.write_bool(succeeded)?;
+            }
         }
         Ok(())
     }
@@ -947,6 +1249,8 @@ impl Packet {
             Packet::SubkernelLoadRunReply { destination, .. } => Some(*destination),
             Packet::SubkernelMessage { destination, .. } => Some(*destination),
             Packet::SubkernelMessageAck { destination } => Some(*destination),
+            Packet::SubkernelExceptionRequest { destination, .. } => Some(*destination),
+            Packet::SubkernelException { destination, .. } => Some(*destination),
             Packet::DmaPlaybackStatus { destination, .. } => Some(*destination),
             Packet::SubkernelFinished { destination, .. } => Some(*destination),
             _ => None,
@@ -964,7 +1268,9 @@ impl Packet {
             | Packet::SubkernelLoadRunReply { .. }
             | Packet::SubkernelMessageAck { .. }
             | Packet::DmaPlaybackStatus { .. }
-            | Packet::SubkernelFinished { .. } => false,
+            | Packet::SubkernelFinished { .. }
+            | Packet::CoreMgmtDropLinkAck { .. }
+            | Packet::InjectionRequest { .. } => false,
             _ => true,
         }
     }

src/libboard_artiq/src/fiq.rs

@@ -0,0 +1,22 @@
+use libboard_zynq::{println, stdio};
+use libcortex_a9::{interrupt_handler, regs::MPIDR};
+use libregister::RegisterR;
+
+#[cfg(has_si549)]
+use crate::si549;
+
+interrupt_handler!(FIQ, fiq, __irq_stack0_start, __irq_stack1_start, {
+    match MPIDR.read().cpu_id() {
+        0 => {
+            // nFIQ is driven directly and bypass GIC
+            #[cfg(has_si549)]
+            si549::wrpll::interrupt_handler();
+            return;
+        }
+        _ => {}
+    };
+
+    stdio::drop_uart();
+    println!("FIQ");
+    loop {}
+});

src/libboard_artiq/src/io_expander.rs

@@ -1,6 +1,7 @@
 use libboard_zynq::i2c;
 use log::info;
 
+#[cfg(has_virtual_leds)]
 use crate::pl::csr;
 
 // Only the bare minimum registers. Bits/IO connections equivalent between IC types.
@@ -19,11 +20,15 @@ const IODIR_OUT_SFP_LED: u8 = 0x40;
 const IODIR_OUT_SFP0_LED: u8 = 0x40;
 #[cfg(hw_rev = "v1.1")]
 const IODIR_OUT_SFP0_LED: u8 = 0x80;
+#[cfg(has_si549)]
+const IODIR_CLK_SEL: u8 = 0x80; // out
+#[cfg(has_si5324)]
+const IODIR_CLK_SEL: u8 = 0x00; // in
 
 //IO expander port direction
 const IODIR0: [u8; 2] = [
     0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP0_LED,
-    0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP_LED,
+    0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP_LED & !IODIR_CLK_SEL,
 ];
 
 const IODIR1: [u8; 2] = [
@@ -33,6 +38,7 @@ const IODIR1: [u8; 2] = [
 
 pub struct IoExpander {
     address: u8,
+    #[cfg(has_virtual_leds)]
     virtual_led_mapping: &'static [(u8, u8, u8)],
     iodir: [u8; 2],
     out_current: [u8; 2],
@@ -42,17 +48,18 @@ pub struct IoExpander {
 
 impl IoExpander {
     pub fn new(i2c: &mut i2c::I2c, index: u8) -> Result<Self, &'static str> {
-        #[cfg(hw_rev = "v1.0")]
+        #[cfg(all(hw_rev = "v1.0", has_virtual_leds))]
         const VIRTUAL_LED_MAPPING0: [(u8, u8, u8); 2] = [(0, 0, 6), (1, 1, 6)];
-        #[cfg(hw_rev = "v1.1")]
+        #[cfg(all(hw_rev = "v1.1", has_virtual_leds))]
         const VIRTUAL_LED_MAPPING0: [(u8, u8, u8); 2] = [(0, 0, 7), (1, 1, 6)];
-
+        #[cfg(has_virtual_leds)]
         const VIRTUAL_LED_MAPPING1: [(u8, u8, u8); 2] = [(2, 0, 6), (3, 1, 6)];
 
         // Both expanders on SHARED I2C bus
         let mut io_expander = match index {
             0 => IoExpander {
                 address: 0x40,
+                #[cfg(has_virtual_leds)]
                 virtual_led_mapping: &VIRTUAL_LED_MAPPING0,
                 iodir: IODIR0,
                 out_current: [0; 2],
@@ -66,6 +73,7 @@ impl IoExpander {
             },
             1 => IoExpander {
                 address: 0x42,
+                #[cfg(has_virtual_leds)]
                 virtual_led_mapping: &VIRTUAL_LED_MAPPING1,
                 iodir: IODIR1,
                 out_current: [0; 2],

src/libboard_artiq/src/lib.rs

@@ -1,5 +1,7 @@
 #![no_std]
 #![feature(never_type)]
+#![feature(naked_functions)]
+#![feature(asm)]
 
 extern crate core_io;
 extern crate crc;
@@ -19,7 +21,8 @@ pub mod drtioaux;
 #[cfg(has_drtio)]
 pub mod drtioaux_async;
 pub mod drtioaux_proto;
-#[cfg(all(feature = "target_kasli_soc", has_drtio))]
+pub mod fiq;
+#[cfg(feature = "target_kasli_soc")]
 pub mod io_expander;
 pub mod logger;
 #[cfg(has_drtio)]
@@ -35,7 +38,8 @@ pub mod drtio_eem;
 pub mod grabber;
 #[cfg(has_si5324)]
 pub mod si5324;
-
+#[cfg(has_si549)]
+pub mod si549;
 use core::{cmp, str};
 
 pub fn identifier_read(buf: &mut [u8]) -> &str {

src/libboard_artiq/src/si549.rs

@@ -0,0 +1,854 @@
+use embedded_hal::prelude::_embedded_hal_blocking_delay_DelayUs;
+use libboard_zynq::timer::GlobalTimer;
+use log::info;
+
+use crate::pl::csr;
+
+#[cfg(feature = "target_kasli_soc")]
+const ADDRESS: u8 = 0x67;
+
+const ADPLL_MAX: i32 = (950.0 / 0.0001164) as i32;
+
+pub struct DividerConfig {
+    pub hsdiv: u16,
+    pub lsdiv: u8,
+    pub fbdiv: u64,
+}
+
+pub struct FrequencySetting {
+    pub main: DividerConfig,
+    pub helper: DividerConfig,
+}
+
+mod i2c {
+    use super::*;
+
+    #[derive(Clone, Copy)]
+    pub enum DCXO {
+        Main,
+        Helper,
+    }
+
+    fn half_period(timer: &mut GlobalTimer) {
+        timer.delay_us(1)
+    }
+
+    fn sda_i(dcxo: DCXO) -> bool {
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_sda_in_read() == 1 },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_sda_in_read() == 1 },
+        }
+    }
+
+    fn sda_oe(dcxo: DCXO, oe: bool) {
+        let val = if oe { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_sda_oe_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_sda_oe_write(val) },
+        };
+    }
+
+    fn sda_o(dcxo: DCXO, o: bool) {
+        let val = if o { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_sda_out_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_sda_out_write(val) },
+        };
+    }
+
+    fn scl_oe(dcxo: DCXO, oe: bool) {
+        let val = if oe { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_scl_oe_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_scl_oe_write(val) },
+        };
+    }
+
+    fn scl_o(dcxo: DCXO, o: bool) {
+        let val = if o { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_scl_out_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_scl_out_write(val) },
+        };
+    }
+
+    pub fn init(dcxo: DCXO, timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        // Set SCL as output, and high level
+        scl_o(dcxo, true);
+        scl_oe(dcxo, true);
+        // Prepare a zero level on SDA so that sda_oe pulls it down
+        sda_o(dcxo, false);
+        // Release SDA
+        sda_oe(dcxo, false);
+
+        // Check the I2C bus is ready
+        half_period(timer);
+        half_period(timer);
+        if !sda_i(dcxo) {
+            // Try toggling SCL a few times
+            for _bit in 0..8 {
+                scl_o(dcxo, false);
+                half_period(timer);
+                scl_o(dcxo, true);
+                half_period(timer);
+            }
+        }
+
+        if !sda_i(dcxo) {
+            return Err("SDA is stuck low and doesn't get unstuck");
+        }
+        Ok(())
+    }
+
+    pub fn start(dcxo: DCXO, timer: &mut GlobalTimer) {
+        // Set SCL high then SDA low
+        scl_o(dcxo, true);
+        half_period(timer);
+        sda_oe(dcxo, true);
+        half_period(timer);
+    }
+
+    pub fn stop(dcxo: DCXO, timer: &mut GlobalTimer) {
+        // First, make sure SCL is low, so that the target releases the SDA line
+        scl_o(dcxo, false);
+        half_period(timer);
+        // Set SCL high then SDA high
+        sda_oe(dcxo, true);
+        scl_o(dcxo, true);
+        half_period(timer);
+        sda_oe(dcxo, false);
+        half_period(timer);
+    }
+
+    pub fn write(dcxo: DCXO, data: u8, timer: &mut GlobalTimer) -> bool {
+        // MSB first
+        for bit in (0..8).rev() {
+            // Set SCL low and set our bit on SDA
+            scl_o(dcxo, false);
+            sda_oe(dcxo, data & (1 << bit) == 0);
+            half_period(timer);
+            // Set SCL high ; data is shifted on the rising edge of SCL
+            scl_o(dcxo, true);
+            half_period(timer);
+        }
+        // Check ack
+        // Set SCL low, then release SDA so that the I2C target can respond
+        scl_o(dcxo, false);
+        half_period(timer);
+        sda_oe(dcxo, false);
+        // Set SCL high and check for ack
+        scl_o(dcxo, true);
+        half_period(timer);
+        // returns true if acked (I2C target pulled SDA low)
+        !sda_i(dcxo)
+    }
+
+    pub fn read(dcxo: DCXO, ack: bool, timer: &mut GlobalTimer) -> u8 {
+        // Set SCL low first, otherwise setting SDA as input may cause a transition
+        // on SDA with SCL high which will be interpreted as START/STOP condition.
+        scl_o(dcxo, false);
+        half_period(timer); // make sure SCL has settled low
+        sda_oe(dcxo, false);
+
+        let mut data: u8 = 0;
+
+        // MSB first
+        for bit in (0..8).rev() {
+            scl_o(dcxo, false);
+            half_period(timer);
+            // Set SCL high and shift data
+            scl_o(dcxo, true);
+            half_period(timer);
+            if sda_i(dcxo) {
+                data |= 1 << bit
+            }
+        }
+        // Send ack
+        // Set SCL low and pull SDA low when acking
+        scl_o(dcxo, false);
+        if ack {
+            sda_oe(dcxo, true)
+        }
+        half_period(timer);
+        // then set SCL high
+        scl_o(dcxo, true);
+        half_period(timer);
+
+        data
+    }
+}
+
+fn write(dcxo: i2c::DCXO, reg: u8, val: u8, timer: &mut GlobalTimer) -> Result<(), &'static str> {
+    i2c::start(dcxo, timer);
+    if !i2c::write(dcxo, ADDRESS << 1, timer) {
+        return Err("Si549 failed to ack write address");
+    }
+    if !i2c::write(dcxo, reg, timer) {
+        return Err("Si549 failed to ack register");
+    }
+    if !i2c::write(dcxo, val, timer) {
+        return Err("Si549 failed to ack value");
+    }
+    i2c::stop(dcxo, timer);
+    Ok(())
+}
+
+fn read(dcxo: i2c::DCXO, reg: u8, timer: &mut GlobalTimer) -> Result<u8, &'static str> {
+    i2c::start(dcxo, timer);
+    if !i2c::write(dcxo, ADDRESS << 1, timer) {
+        return Err("Si549 failed to ack write address");
+    }
+    if !i2c::write(dcxo, reg, timer) {
+        return Err("Si549 failed to ack register");
+    }
+    i2c::stop(dcxo, timer);
+
+    i2c::start(dcxo, timer);
+    if !i2c::write(dcxo, (ADDRESS << 1) | 1, timer) {
+        return Err("Si549 failed to ack read address");
+    }
+    let val = i2c::read(dcxo, false, timer);
+    i2c::stop(dcxo, timer);
+    Ok(val)
+}
+
+fn setup(dcxo: i2c::DCXO, config: &DividerConfig, timer: &mut GlobalTimer) -> Result<(), &'static str> {
+    i2c::init(dcxo, timer)?;
+
+    write(dcxo, 255, 0x00, timer)?; // PAGE
+    write(dcxo, 69, 0x00, timer)?; // Disable FCAL override.
+    write(dcxo, 17, 0x00, timer)?; // Synchronously disable output
+
+    // The Si549 has no ID register, so we check that it responds correctly
+    // by writing values to a RAM-like register and reading them back.
+    for test_value in 0..255 {
+        write(dcxo, 23, test_value, timer)?;
+        let readback = read(dcxo, 23, timer)?;
+        if readback != test_value {
+            return Err("Si549 detection failed");
+        }
+    }
+
+    write(dcxo, 23, config.hsdiv as u8, timer)?;
+    write(dcxo, 24, (config.hsdiv >> 8) as u8 | (config.lsdiv << 4), timer)?;
+    write(dcxo, 26, config.fbdiv as u8, timer)?;
+    write(dcxo, 27, (config.fbdiv >> 8) as u8, timer)?;
+    write(dcxo, 28, (config.fbdiv >> 16) as u8, timer)?;
+    write(dcxo, 29, (config.fbdiv >> 24) as u8, timer)?;
+    write(dcxo, 30, (config.fbdiv >> 32) as u8, timer)?;
+    write(dcxo, 31, (config.fbdiv >> 40) as u8, timer)?;
+
+    write(dcxo, 7, 0x08, timer)?; // Start FCAL
+    timer.delay_us(30_000); // Internal FCAL VCO calibration
+    write(dcxo, 17, 0x01, timer)?; // Synchronously enable output
+
+    Ok(())
+}
+
+pub fn main_setup(timer: &mut GlobalTimer, settings: &FrequencySetting) -> Result<(), &'static str> {
+    unsafe {
+        csr::wrpll::main_dcxo_bitbang_enable_write(1);
+        csr::wrpll::main_dcxo_i2c_address_write(ADDRESS);
+    }
+
+    setup(i2c::DCXO::Main, &settings.main, timer)?;
+
+    // Si549 maximum settling time for large frequency change.
+    timer.delay_us(40_000);
+
+    unsafe {
+        csr::wrpll::main_dcxo_bitbang_enable_write(0);
+    }
+
+    info!("Main Si549 started");
+    Ok(())
+}
+
+pub fn helper_setup(timer: &mut GlobalTimer, settings: &FrequencySetting) -> Result<(), &'static str> {
+    unsafe {
+        csr::wrpll::helper_reset_write(1);
+        csr::wrpll::helper_dcxo_bitbang_enable_write(1);
+        csr::wrpll::helper_dcxo_i2c_address_write(ADDRESS);
+    }
+
+    setup(i2c::DCXO::Helper, &settings.helper, timer)?;
+
+    // Si549 maximum settling time for large frequency change.
+    timer.delay_us(40_000);
+
+    unsafe {
+        csr::wrpll::helper_reset_write(0);
+        csr::wrpll::helper_dcxo_bitbang_enable_write(0);
+    }
+    info!("Helper Si549 started");
+    Ok(())
+}
+
+fn set_adpll(dcxo: i2c::DCXO, adpll: i32) -> Result<(), &'static str> {
+    if adpll.abs() > ADPLL_MAX {
+        return Err("adpll is too large");
+    }
+
+    match dcxo {
+        i2c::DCXO::Main => unsafe {
+            if csr::wrpll::main_dcxo_bitbang_enable_read() == 1 {
+                return Err("Main si549 bitbang mode is active when using gateware i2c");
+            }
+
+            while csr::wrpll::main_dcxo_adpll_busy_read() == 1 {}
+            if csr::wrpll::main_dcxo_nack_read() == 1 {
+                return Err("Main si549 failed to ack adpll write");
+            }
+
+            csr::wrpll::main_dcxo_i2c_address_write(ADDRESS);
+            csr::wrpll::main_dcxo_adpll_write(adpll as u32);
+
+            csr::wrpll::main_dcxo_adpll_stb_write(1);
+        },
+        i2c::DCXO::Helper => unsafe {
+            if csr::wrpll::helper_dcxo_bitbang_enable_read() == 1 {
+                return Err("Helper si549 bitbang mode is active when using gateware i2c");
+            }
+
+            while csr::wrpll::helper_dcxo_adpll_busy_read() == 1 {}
+            if csr::wrpll::helper_dcxo_nack_read() == 1 {
+                return Err("Helper si549 failed to ack adpll write");
+            }
+
+            csr::wrpll::helper_dcxo_i2c_address_write(ADDRESS);
+            csr::wrpll::helper_dcxo_adpll_write(adpll as u32);
+
+            csr::wrpll::helper_dcxo_adpll_stb_write(1);
+        },
+    };
+
+    Ok(())
+}
+
+#[cfg(has_wrpll)]
+pub mod wrpll {
+
+    use super::*;
+
+    const BEATING_PERIOD: i32 = 0x8000;
+    const BEATING_HALFPERIOD: i32 = 0x4000;
+    const COUNTER_WIDTH: u32 = 24;
+    const DIV_WIDTH: u32 = 2;
+
+    // y[n] = b0*x[n] + b1*x[n-1] + b2*x[n-2] - a1*y[n-1] - a2*y[n-2]
+    struct FilterParameters {
+        pub b0: f64,
+        pub b1: f64,
+        pub b2: f64,
+        pub a1: f64,
+        pub a2: f64,
+    }
+
+    #[cfg(rtio_frequency = "100.0")]
+    const LPF: FilterParameters = FilterParameters {
+        b0: 0.03967479060647884,
+        b1: 0.07934958121295768,
+        b2: 0.03967479060647884,
+        a1: -1.3865593741228928,
+        a2: 0.5452585365488082,
+    };
+
+    #[cfg(rtio_frequency = "125.0")]
+    const LPF: FilterParameters = FilterParameters {
+        b0: 0.07209205036273991,
+        b1: 0.14418410072547982,
+        b2: 0.07209205036273991,
+        a1: -0.6114078511562919,
+        a2: -0.10022394739274834,
+    };
+
+    static mut H_ADPLL1: i32 = 0;
+    static mut H_ADPLL2: i32 = 0;
+    static mut PERIOD_ERR1: i32 = 0;
+    static mut PERIOD_ERR2: i32 = 0;
+
+    static mut M_ADPLL1: i32 = 0;
+    static mut M_ADPLL2: i32 = 0;
+    static mut PHASE_ERR1: i32 = 0;
+    static mut PHASE_ERR2: i32 = 0;
+
+    static mut BASE_ADPLL: i32 = 0;
+
+    #[derive(Clone, Copy)]
+    pub enum ISR {
+        RefTag,
+        MainTag,
+    }
+
+    mod tag_collector {
+        use super::*;
+
+        #[cfg(wrpll_ref_clk = "GT_CDR")]
+        static mut TAG_OFFSET: u32 = 8382;
+        #[cfg(wrpll_ref_clk = "SMA_CLKIN")]
+        static mut TAG_OFFSET: u32 = 0;
+        static mut REF_TAG: u32 = 0;
+        static mut REF_TAG_READY: bool = false;
+        static mut MAIN_TAG: u32 = 0;
+        static mut MAIN_TAG_READY: bool = false;
+
+        pub fn reset() {
+            clear_phase_diff_ready();
+            unsafe {
+                REF_TAG = 0;
+                MAIN_TAG = 0;
+            }
+        }
+
+        pub fn clear_phase_diff_ready() {
+            unsafe {
+                REF_TAG_READY = false;
+                MAIN_TAG_READY = false;
+            }
+        }
+
+        pub fn collect_tags(interrupt: ISR) {
+            match interrupt {
+                ISR::RefTag => unsafe {
+                    REF_TAG = csr::wrpll::ref_tag_read();
+                    REF_TAG_READY = true;
+                },
+                ISR::MainTag => unsafe {
+                    MAIN_TAG = csr::wrpll::main_tag_read();
+                    MAIN_TAG_READY = true;
+                },
+            }
+        }
+
+        pub fn phase_diff_ready() -> bool {
+            unsafe { REF_TAG_READY && MAIN_TAG_READY }
+        }
+
+        #[cfg(feature = "calibrate_wrpll_skew")]
+        pub fn set_tag_offset(offset: u32) {
+            unsafe {
+                TAG_OFFSET = offset;
+            }
+        }
+
+        #[cfg(feature = "calibrate_wrpll_skew")]
+        pub fn get_tag_offset() -> u32 {
+            unsafe { TAG_OFFSET }
+        }
+
+        pub fn get_period_error() -> i32 {
+            // n * BEATING_PERIOD - REF_TAG(n) mod BEATING_PERIOD
+            let mut period_error = unsafe { REF_TAG.overflowing_neg().0.rem_euclid(BEATING_PERIOD as u32) as i32 };
+            // mapping tags from [0, 2π] -> [-π, π]
+            if period_error > BEATING_HALFPERIOD {
+                period_error -= BEATING_PERIOD
+            }
+            period_error
+        }
+
+        pub fn get_phase_error() -> i32 {
+            // MAIN_TAG(n) - REF_TAG(n) - TAG_OFFSET mod BEATING_PERIOD
+            let mut phase_error = unsafe {
+                MAIN_TAG
+                    .overflowing_sub(REF_TAG + TAG_OFFSET)
+                    .0
+                    .rem_euclid(BEATING_PERIOD as u32) as i32
+            };
+
+            // mapping tags from [0, 2π] -> [-π, π]
+            if phase_error > BEATING_HALFPERIOD {
+                phase_error -= BEATING_PERIOD
+            }
+            phase_error
+        }
+    }
+
+    fn set_isr(en: bool) {
+        let val = if en { 1 } else { 0 };
+        unsafe {
+            csr::wrpll::ref_tag_ev_enable_write(val);
+            csr::wrpll::main_tag_ev_enable_write(val);
+        }
+    }
+
+    fn set_base_adpll() -> Result<(), &'static str> {
+        let count2adpll =
+            |error: i32| ((error as f64 * 1e6) / (0.0001164 * (1 << (COUNTER_WIDTH - DIV_WIDTH)) as f64)) as i32;
+
+        let (ref_count, main_count) = get_freq_counts();
+        unsafe {
+            BASE_ADPLL = count2adpll(ref_count as i32 - main_count as i32);
+            set_adpll(i2c::DCXO::Main, BASE_ADPLL)?;
+            set_adpll(i2c::DCXO::Helper, BASE_ADPLL)?;
+        }
+        Ok(())
+    }
+
+    fn get_freq_counts() -> (u32, u32) {
+        unsafe {
+            csr::wrpll::frequency_counter_update_write(1);
+            while csr::wrpll::frequency_counter_busy_read() == 1 {}
+            #[cfg(wrpll_ref_clk = "GT_CDR")]
+            let ref_count = csr::wrpll::frequency_counter_counter_rtio_rx0_read();
+            #[cfg(wrpll_ref_clk = "SMA_CLKIN")]
+            let ref_count = csr::wrpll::frequency_counter_counter_ref_read();
+            let main_count = csr::wrpll::frequency_counter_counter_sys_read();
+
+            (ref_count, main_count)
+        }
+    }
+
+    fn reset_plls(timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        unsafe {
+            H_ADPLL1 = 0;
+            H_ADPLL2 = 0;
+            PERIOD_ERR1 = 0;
+            PERIOD_ERR2 = 0;
+            M_ADPLL1 = 0;
+            M_ADPLL2 = 0;
+            PHASE_ERR1 = 0;
+            PHASE_ERR2 = 0;
+        }
+        set_adpll(i2c::DCXO::Main, 0)?;
+        set_adpll(i2c::DCXO::Helper, 0)?;
+        // wait for adpll to transfer and DCXO to settle
+        timer.delay_us(200);
+        Ok(())
+    }
+
+    fn clear_pending(interrupt: ISR) {
+        match interrupt {
+            ISR::RefTag => unsafe { csr::wrpll::ref_tag_ev_pending_write(1) },
+            ISR::MainTag => unsafe { csr::wrpll::main_tag_ev_pending_write(1) },
+        };
+    }
+
+    fn is_pending(interrupt: ISR) -> bool {
+        match interrupt {
+            ISR::RefTag => unsafe { csr::wrpll::ref_tag_ev_pending_read() == 1 },
+            ISR::MainTag => unsafe { csr::wrpll::main_tag_ev_pending_read() == 1 },
+        }
+    }
+
+    pub fn interrupt_handler() {
+        if is_pending(ISR::RefTag) {
+            tag_collector::collect_tags(ISR::RefTag);
+            clear_pending(ISR::RefTag);
+            helper_pll().expect("failed to run helper DCXO PLL");
+        }
+
+        if is_pending(ISR::MainTag) {
+            tag_collector::collect_tags(ISR::MainTag);
+            clear_pending(ISR::MainTag);
+        }
+
+        if tag_collector::phase_diff_ready() {
+            main_pll().expect("failed to run main DCXO PLL");
+            tag_collector::clear_phase_diff_ready();
+        }
+    }
+
+    fn helper_pll() -> Result<(), &'static str> {
+        let period_err = tag_collector::get_period_error();
+        unsafe {
+            let adpll = ((LPF.b0 * period_err as f64) + (LPF.b1 * PERIOD_ERR1 as f64) + (LPF.b2 * PERIOD_ERR2 as f64)
+                - (LPF.a1 * H_ADPLL1 as f64)
+                - (LPF.a2 * H_ADPLL2 as f64)) as i32;
+            set_adpll(i2c::DCXO::Helper, BASE_ADPLL + adpll)?;
+            H_ADPLL2 = H_ADPLL1;
+            PERIOD_ERR2 = PERIOD_ERR1;
+            H_ADPLL1 = adpll;
+            PERIOD_ERR1 = period_err;
+        };
+        Ok(())
+    }
+
+    fn main_pll() -> Result<(), &'static str> {
+        let phase_err = tag_collector::get_phase_error();
+        unsafe {
+            let adpll = ((LPF.b0 * phase_err as f64) + (LPF.b1 * PHASE_ERR1 as f64) + (LPF.b2 * PHASE_ERR2 as f64)
+                - (LPF.a1 * M_ADPLL1 as f64)
+                - (LPF.a2 * M_ADPLL2 as f64)) as i32;
+            set_adpll(i2c::DCXO::Main, BASE_ADPLL + adpll)?;
+            M_ADPLL2 = M_ADPLL1;
+            PHASE_ERR2 = PHASE_ERR1;
+            M_ADPLL1 = adpll;
+            PHASE_ERR1 = phase_err;
+        };
+        Ok(())
+    }
+
+    #[cfg(wrpll_ref_clk = "GT_CDR")]
+    fn test_skew(timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        // wait for PLL to stabilize
+        timer.delay_us(20_000);
+
+        info!("testing the skew of SYS CLK...");
+        if has_timing_error(timer) {
+            return Err("the skew cannot satisfy setup/hold time constraint of RX synchronizer");
+        }
+        info!("the skew of SYS CLK met the timing constraint");
+        Ok(())
+    }
+
+    #[cfg(wrpll_ref_clk = "GT_CDR")]
+    fn has_timing_error(timer: &mut GlobalTimer) -> bool {
+        unsafe {
+            csr::wrpll_skewtester::error_write(1);
+        }
+        timer.delay_us(5_000);
+        unsafe { csr::wrpll_skewtester::error_read() == 1 }
+    }
+
+    #[cfg(feature = "calibrate_wrpll_skew")]
+    fn find_edge(target: bool, timer: &mut GlobalTimer) -> Result<u32, &'static str> {
+        const STEP: u32 = 8;
+        const STABLE_THRESHOLD: u32 = 10;
+
+        enum FSM {
+            Init,
+            WaitEdge,
+            GotEdge,
+        }
+
+        let mut state: FSM = FSM::Init;
+        let mut offset: u32 = tag_collector::get_tag_offset();
+        let mut median_edge: u32 = 0;
+        let mut stable_counter: u32 = 0;
+
+        for _ in 0..(BEATING_PERIOD as u32 / STEP) as usize {
+            tag_collector::set_tag_offset(offset);
+            offset += STEP;
+            // wait for PLL to stabilize
+            timer.delay_us(20_000);
+
+            let error = has_timing_error(timer);
+            // A median edge deglitcher
+            match state {
+                FSM::Init => {
+                    if error != target {
+                        stable_counter += 1;
+                    } else {
+                        stable_counter = 0;
+                    }
+
+                    if stable_counter >= STABLE_THRESHOLD {
+                        state = FSM::WaitEdge;
+                        stable_counter = 0;
+                    }
+                }
+                FSM::WaitEdge => {
+                    if error == target {
+                        state = FSM::GotEdge;
+                        median_edge = offset;
+                    }
+                }
+                FSM::GotEdge => {
+                    if error != target {
+                        median_edge += STEP;
+                        stable_counter = 0;
+                    } else {
+                        stable_counter += 1;
+                    }
+
+                    if stable_counter >= STABLE_THRESHOLD {
+                        return Ok(median_edge);
+                    }
+                }
+            }
+        }
+        return Err("failed to find timing error edge");
+    }
+
+    #[cfg(feature = "calibrate_wrpll_skew")]
+    fn calibrate_skew(timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        info!("calibrating skew to meet timing constraint...");
+
+        // clear calibrated value
+        tag_collector::set_tag_offset(0);
+        let rising = find_edge(true, timer)? as i32;
+        let falling = find_edge(false, timer)? as i32;
+
+        let width = BEATING_PERIOD - (falling - rising);
+        let result = falling + width / 2;
+        tag_collector::set_tag_offset(result as u32);
+
+        info!(
+            "calibration successful, error zone: {} -> {}, width: {} ({}deg), middle of working region: {}",
+            rising,
+            falling,
+            width,
+            360 * width / BEATING_PERIOD,
+            result,
+        );
+
+        Ok(())
+    }
+
+    pub fn select_recovered_clock(rc: bool, timer: &mut GlobalTimer) {
+        set_isr(false);
+
+        if rc {
+            tag_collector::reset();
+            reset_plls(timer).expect("failed to reset main and helper PLL");
+
+            // get within capture range
+            set_base_adpll().expect("failed to set base adpll");
+
+            // clear gateware pending flag
+            clear_pending(ISR::RefTag);
+            clear_pending(ISR::MainTag);
+
+            // use nFIQ to avoid IRQ being disabled by mutex lock and mess up PLL
+            set_isr(true);
+            info!("WRPLL interrupt enabled");
+
+            #[cfg(feature = "calibrate_wrpll_skew")]
+            calibrate_skew(timer).expect("failed to set the correct skew");
+
+            #[cfg(wrpll_ref_clk = "GT_CDR")]
+            test_skew(timer).expect("skew test failed");
+        }
+    }
+}
+
+#[cfg(has_wrpll_refclk)]
+pub mod wrpll_refclk {
+    use super::*;
+
+    pub struct MmcmSetting {
+        pub clkout0_reg1: u16,  //0x08
+        pub clkout0_reg2: u16,  //0x09
+        pub clkfbout_reg1: u16, //0x14
+        pub clkfbout_reg2: u16, //0x15
+        pub div_reg: u16,       //0x16
+        pub lock_reg1: u16,     //0x18
+        pub lock_reg2: u16,     //0x19
+        pub lock_reg3: u16,     //0x1A
+        pub power_reg: u16,     //0x28
+        pub filt_reg1: u16,     //0x4E
+        pub filt_reg2: u16,     //0x4F
+    }
+
+    fn one_clock_cycle() {
+        unsafe {
+            csr::wrpll_refclk::mmcm_dclk_write(1);
+            csr::wrpll_refclk::mmcm_dclk_write(0);
+        }
+    }
+
+    fn set_addr(address: u8) {
+        unsafe {
+            csr::wrpll_refclk::mmcm_daddr_write(address);
+        }
+    }
+
+    fn set_data(value: u16) {
+        unsafe {
+            csr::wrpll_refclk::mmcm_din_write(value);
+        }
+    }
+
+    fn set_enable(en: bool) {
+        unsafe {
+            let val = if en { 1 } else { 0 };
+            csr::wrpll_refclk::mmcm_den_write(val);
+        }
+    }
+
+    fn set_write_enable(en: bool) {
+        unsafe {
+            let val = if en { 1 } else { 0 };
+            csr::wrpll_refclk::mmcm_dwen_write(val);
+        }
+    }
+
+    fn get_data() -> u16 {
+        unsafe { csr::wrpll_refclk::mmcm_dout_read() }
+    }
+
+    fn drp_ready() -> bool {
+        unsafe { csr::wrpll_refclk::mmcm_dready_read() == 1 }
+    }
+
+    #[allow(dead_code)]
+    fn read(address: u8) -> u16 {
+        set_addr(address);
+        set_enable(true);
+        // Set DADDR on the mmcm and assert DEN for one clock cycle
+        one_clock_cycle();
+
+        set_enable(false);
+        while !drp_ready() {
+            // keep the clock signal until data is ready
+            one_clock_cycle();
+        }
+        get_data()
+    }
+
+    fn write(address: u8, value: u16) {
+        set_addr(address);
+        set_data(value);
+        set_write_enable(true);
+        set_enable(true);
+        // Set DADDR, DI on the mmcm and assert DWE, DEN for one clock cycle
+        one_clock_cycle();
+
+        set_write_enable(false);
+        set_enable(false);
+        while !drp_ready() {
+            // keep the clock signal until write is finished
+            one_clock_cycle();
+        }
+    }
+
+    fn reset(rst: bool) {
+        unsafe {
+            let val = if rst { 1 } else { 0 };
+            csr::wrpll_refclk::mmcm_reset_write(val)
+        }
+    }
+
+    pub fn setup(timer: &mut GlobalTimer, settings: MmcmSetting, mmcm_bypass: bool) -> Result<(), &'static str> {
+        unsafe {
+            csr::wrpll_refclk::refclk_reset_write(1);
+        }
+
+        if mmcm_bypass {
+            info!("Bypassing mmcm");
+            unsafe {
+                csr::wrpll_refclk::mmcm_bypass_write(1);
+            }
+        } else {
+            // Based on "DRP State Machine" from XAPP888
+            // hold reset HIGH during mmcm config
+            reset(true);
+            write(0x08, settings.clkout0_reg1);
+            write(0x09, settings.clkout0_reg2);
+            write(0x14, settings.clkfbout_reg1);
+            write(0x15, settings.clkfbout_reg2);
+            write(0x16, settings.div_reg);
+            write(0x18, settings.lock_reg1);
+            write(0x19, settings.lock_reg2);
+            write(0x1A, settings.lock_reg3);
+            write(0x28, settings.power_reg);
+            write(0x4E, settings.filt_reg1);
+            write(0x4F, settings.filt_reg2);
+            reset(false);
+
+            // wait for the mmcm to lock
+            timer.delay_us(100);
+
+            let locked = unsafe { csr::wrpll_refclk::mmcm_locked_read() == 1 };
+            if !locked {
+                return Err("mmcm failed to generate 125MHz ref clock from SMA CLKIN");
+            }
+        }
+
+        unsafe {
+            csr::wrpll_refclk::refclk_reset_write(0);
+        }
+
+        Ok(())
+    }
+}

src/libbuild_zynq/link.x

@@ -10,7 +10,9 @@ SECTIONS
     __text_start = .;
     .text :
     {
+        __exceptions_start = .;
         KEEP(*(.text.exceptions));
+        __exceptions_end = .;
         *(.text.boot);
         *(.text .text.*);
     } > SDRAM

src/libdwarf/src/eh.rs

@@ -85,10 +85,7 @@ unsafe fn get_ttype_entry(
         encoding | DW_EH_PE_pcrel,
         ttype_base,
     )
-    .map(|v| match v {
+    .map(|v| (v != ttype_base).then(|| v as *const u8))
-        ttype_base => None,
-        ttype_entry => Some(ttype_entry as *const u8),
-    })
 }
 
 pub unsafe fn find_eh_action(

src/libio/Cargo.toml.tpl

@@ -8,7 +8,7 @@ name = "io"
 path = "lib.rs"
 
 [dependencies]
-core_io = { version = "0.1", features = ["collections"] }
+core_io = { git = "https://git.m-labs.hk/M-Labs/rs-core_io.git", rev = "e9d3edf027", features = ["collections"] }
 byteorder = { version = "1.0", default-features = false, optional = true }
 
 libsupport_zynq = { path = "@@ZYNQ_RS@@/libsupport_zynq", default-features = false, features = ["alloc_core"] }

src/libio/cursor.rs

@@ -1,5 +1,6 @@
 #[cfg(feature = "alloc")]
 use alloc::vec::Vec;
+use core::arch::asm;
 
 use core_io::{Error as IoError, Read, Write};
 
@@ -45,7 +46,13 @@ impl<T: AsRef<[u8]>> Read for Cursor<T> {
     fn read(&mut self, buf: &mut [u8]) -> Result<usize, IoError> {
         let data = &self.inner.as_ref()[self.pos..];
         let len = buf.len().min(data.len());
-        buf[..len].copy_from_slice(&data[..len]);
+        // ``copy_from_slice`` generates AXI bursts, use a regular loop instead
+        for i in 0..len {
+            unsafe {
+                asm!("", options(preserves_flags, nostack, readonly));
+            }
+            buf[i] = data[i];
+        }
         self.pos += len;
         Ok(len)
     }
@@ -55,7 +62,12 @@ impl Write for Cursor<&mut [u8]> {
     fn write(&mut self, buf: &[u8]) -> Result<usize, IoError> {
         let data = &mut self.inner[self.pos..];
         let len = buf.len().min(data.len());
-        data[..len].copy_from_slice(&buf[..len]);
+        for i in 0..len {
+            unsafe {
+                asm!("", options(preserves_flags, nostack, readonly));
+            }
+            data[i] = buf[i];
+        }
         self.pos += len;
         Ok(len)
     }
@@ -68,7 +80,6 @@ impl Write for Cursor<&mut [u8]> {
 
 #[cfg(feature = "alloc")]
 impl Write for Cursor<Vec<u8>> {
-    #[inline]
     fn write(&mut self, buf: &[u8]) -> Result<usize, IoError> {
         self.inner.extend_from_slice(buf);
         Ok(buf.len())

src/libio/lib.rs

@@ -1,5 +1,6 @@
 #![no_std]
 #![feature(never_type)]
+#![feature(asm)]
 
 #[cfg(feature = "alloc")]
 extern crate alloc;

src/libksupport/Cargo.toml.tpl

@@ -12,7 +12,7 @@ build_zynq = { path = "../libbuild_zynq" }
 cslice = "0.3"
 log = "0.4"
 nb = "0.1"
-core_io = { version = "0.1", features = ["collections"] }
+core_io = { git = "https://git.m-labs.hk/M-Labs/rs-core_io.git", rev = "e9d3edf027", features = ["collections"] }
 byteorder = { version = "1.3", default-features = false }
 void = { version = "1", default-features = false }
 log_buffer = { version = "1.2" }
@@ -32,3 +32,9 @@ unwind = { path = "../libunwind" }
 libc = { path = "../libc" }
 io = { path = "../libio" }
 libboard_artiq = { path = "../libboard_artiq" }
+
+[dependencies.nalgebra]
+git = "https://git.m-labs.hk/M-Labs/nalgebra.git"
+rev = "ad42410ab0"
+default-features = false
+features = ["libm", "alloc"]

src/libksupport/src/eh_artiq.rs

@@ -14,8 +14,10 @@
 
 use core::mem;
 
-use cslice::CSlice;
+use core_io::Error as ReadError;
+use cslice::{AsCSlice, CSlice};
 use dwarf::eh::{self, EHAction, EHContext};
+use io::{Cursor, ProtoRead};
 use libc::{c_int, c_void, uintptr_t};
 use log::{error, trace};
 use unwind as uw;
@@ -94,29 +96,35 @@ struct ExceptionBuffer {
 }
 
 static mut EXCEPTION_BUFFER: ExceptionBuffer = ExceptionBuffer {
-    uw_exceptions: [uw::_Unwind_Exception {
+    uw_exceptions: [const {
-        exception_class: EXCEPTION_CLASS,
+        uw::_Unwind_Exception {
-        exception_cleanup: cleanup,
+            exception_class: EXCEPTION_CLASS,
-        private: [0; uw::unwinder_private_data_size],
+            exception_cleanup: cleanup,
+            private: [0; uw::unwinder_private_data_size],
+        }
     }; MAX_INFLIGHT_EXCEPTIONS],
     exceptions: [None; MAX_INFLIGHT_EXCEPTIONS + 1],
     exception_stack: [-1; MAX_INFLIGHT_EXCEPTIONS + 1],
     backtrace: [(0, 0); MAX_BACKTRACE_SIZE],
     backtrace_size: 0,
-    stack_pointers: [StackPointerBacktrace {
+    stack_pointers: [const {
-        stack_pointer: 0,
+        StackPointerBacktrace {
-        initial_backtrace_size: 0,
+            stack_pointer: 0,
-        current_backtrace_size: 0,
+            initial_backtrace_size: 0,
+            current_backtrace_size: 0,
+        }
     }; MAX_INFLIGHT_EXCEPTIONS + 1],
     exception_count: 0,
 };
 
 pub unsafe extern "C" fn reset_exception_buffer() {
     trace!("reset exception buffer");
-    EXCEPTION_BUFFER.uw_exceptions = [uw::_Unwind_Exception {
+    EXCEPTION_BUFFER.uw_exceptions = [const {
-        exception_class: EXCEPTION_CLASS,
+        uw::_Unwind_Exception {
-        exception_cleanup: cleanup,
+            exception_class: EXCEPTION_CLASS,
-        private: [0; uw::unwinder_private_data_size],
+            exception_cleanup: cleanup,
+            private: [0; uw::unwinder_private_data_size],
+        }
     }; MAX_INFLIGHT_EXCEPTIONS];
     EXCEPTION_BUFFER.exceptions = [None; MAX_INFLIGHT_EXCEPTIONS + 1];
     EXCEPTION_BUFFER.exception_stack = [-1; MAX_INFLIGHT_EXCEPTIONS + 1];
@@ -220,8 +228,6 @@ pub unsafe fn artiq_personality(
 }
 
 pub unsafe extern "C" fn raise(exception: *const Exception) -> ! {
-    use cslice::AsCSlice;
-
     let count = EXCEPTION_BUFFER.exception_count;
     let stack = &mut EXCEPTION_BUFFER.exception_stack;
     let diff = exception as isize - EXCEPTION_BUFFER.exceptions.as_ptr() as isize;
@@ -295,6 +301,60 @@ pub unsafe extern "C" fn raise(exception: *const Exception) -> ! {
     unreachable!();
 }
 
+fn read_exception_string<'a>(reader: &mut Cursor<&[u8]>) -> Result<CSlice<'a, u8>, ReadError> {
+    let len = reader.read_u32()? as usize;
+    if len == usize::MAX {
+        let data = reader.read_u32()?;
+        Ok(unsafe { CSlice::new(data as *const u8, len) })
+    } else {
+        let pos = reader.position();
+        let slice = unsafe {
+            let ptr = reader.get_ref().as_ptr().offset(pos as isize);
+            CSlice::new(ptr, len)
+        };
+        reader.set_position(pos + len);
+        Ok(slice)
+    }
+}
+
+fn read_exception(raw_exception: &[u8]) -> Result<Exception, ReadError> {
+    let mut reader = Cursor::new(raw_exception);
+
+    let mut byte = reader.read_u8()?;
+    // to sync
+    while byte != 0x5a {
+        byte = reader.read_u8()?;
+    }
+    // skip sync bytes, 0x09 indicates exception
+    while byte != 0x09 {
+        byte = reader.read_u8()?;
+    }
+    let _len = reader.read_u32()?;
+    // ignore the remaining exceptions, stack traces etc. - unwinding from another device would be unwise anyway
+    Ok(Exception {
+        id: reader.read_u32()?,
+        message: read_exception_string(&mut reader)?,
+        param: [
+            reader.read_u64()? as i64,
+            reader.read_u64()? as i64,
+            reader.read_u64()? as i64,
+        ],
+        file: read_exception_string(&mut reader)?,
+        line: reader.read_u32()?,
+        column: reader.read_u32()?,
+        function: read_exception_string(&mut reader)?,
+    })
+}
+
+pub fn raise_raw(raw_exception: &[u8]) -> ! {
+    use crate::artiq_raise;
+    if let Ok(exception) = read_exception(raw_exception) {
+        unsafe { raise(&exception) };
+    } else {
+        artiq_raise!("SubkernelError", "Error passing exception");
+    }
+}
+
 pub unsafe extern "C" fn resume() -> ! {
     trace!("resume");
     assert!(EXCEPTION_BUFFER.exception_count != 0);
@@ -421,20 +481,30 @@ extern "C" fn stop_fn(
     }
 }
 
-// Must be kept in sync with preallocate_runtime_exception_names() in artiq/language/embedding_map.py
+// Must be kept in sync with preallocate_runtime_exception_names() in `artiq.compiler.embedding`
-static EXCEPTION_ID_LOOKUP: [(&str, u32); 12] = [
+static EXCEPTION_ID_LOOKUP: [(&str, u32); 22] = [
-    ("RuntimeError", 0),
+    ("RTIOUnderflow", 0),
-    ("RTIOUnderflow", 1),
+    ("RTIOOverflow", 1),
-    ("RTIOOverflow", 2),
+    ("RTIODestinationUnreachable", 2),
-    ("RTIODestinationUnreachable", 3),
+    ("DMAError", 3),
-    ("DMAError", 4),
+    ("I2CError", 4),
-    ("I2CError", 5),
+    ("CacheError", 5),
-    ("CacheError", 6),
+    ("SPIError", 6),
-    ("SPIError", 7),
+    ("SubkernelError", 7),
-    ("ZeroDivisionError", 8),
+    ("AssertionError", 8),
-    ("IndexError", 9),
+    ("AttributeError", 9),
-    ("UnwrapNoneError", 10),
+    ("IndexError", 10),
-    ("SubkernelError", 11),
+    ("IOError", 11),
+    ("KeyError", 12),
+    ("NotImplementedError", 13),
+    ("OverflowError", 14),
+    ("RuntimeError", 15),
+    ("TimeoutError", 16),
+    ("TypeError", 17),
+    ("ValueError", 18),
+    ("ZeroDivisionError", 19),
+    ("LinAlgError", 20),
+    ("UnwrapNoneError", 21),
 ];
 
 pub fn get_exception_id(name: &str) -> u32 {
@@ -469,3 +539,29 @@ macro_rules! artiq_raise {
     }};
     ($name:expr, $message:expr) => {{ artiq_raise!($name, $message, 0, 0, 0) }};
 }
+
+/// Takes as input exception id from host
+/// Generates a new exception with:
+///   * `id` set to `exn_id`
+///   * `message` set to corresponding exception name from `EXCEPTION_ID_LOOKUP`
+///
+/// The message is matched on host to ensure correct exception is being referred
+/// This test checks the synchronization of exception ids for runtime errors
+#[no_mangle]
+pub extern "C" fn test_exception_id_sync(exn_id: u32) {
+    let message = EXCEPTION_ID_LOOKUP
+        .iter()
+        .find_map(|&(name, id)| if id == exn_id { Some(name) } else { None })
+        .unwrap_or("unallocated internal exception id");
+
+    let exn = Exception {
+        id: exn_id,
+        file: file!().as_c_slice(),
+        line: 0,
+        column: 0,
+        function: "test_exception_id_sync".as_c_slice(),
+        message: message.as_c_slice(),
+        param: [0, 0, 0],
+    };
+    unsafe { raise(&exn) };
+}

src/libksupport/src/kernel/api.rs

@@ -9,7 +9,7 @@ use log::{info, warn};
 use super::subkernel;
 use super::{cache,
             core1::rtio_get_destination_status,
-            dma,
+            dma, linalg,
             rpc::{rpc_recv, rpc_send, rpc_send_async}};
 use crate::{eh_artiq, i2c, rtio};
 
@@ -303,6 +303,7 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api_libm_f64f64f64!(nextafter),
         api_libm_f64f64f64!(pow),
         api_libm_f64f64!(round),
+        api_libm_f64f64!(rint),
         api_libm_f64f64!(sin),
         api_libm_f64f64!(sinh),
         api_libm_f64f64!(sqrt),
@@ -318,6 +319,26 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
             }
             api!(yn = yn)
         },
+
+        // linalg
+        api!(np_linalg_cholesky = linalg::np_linalg_cholesky),
+        api!(np_linalg_qr = linalg::np_linalg_qr),
+        api!(np_linalg_svd = linalg::np_linalg_svd),
+        api!(np_linalg_inv = linalg::np_linalg_inv),
+        api!(np_linalg_pinv = linalg::np_linalg_pinv),
+        api!(np_linalg_matrix_power = linalg::np_linalg_matrix_power),
+        api!(np_linalg_det = linalg::np_linalg_det),
+        api!(sp_linalg_lu = linalg::sp_linalg_lu),
+        api!(sp_linalg_schur = linalg::sp_linalg_schur),
+        api!(sp_linalg_hessenberg = linalg::sp_linalg_hessenberg),
+
+        /*
+         * syscall for unit tests
+         * Used in `artiq.tests.coredevice.test_exceptions.ExceptionTest.test_raise_exceptions_kernel`
+         * This syscall checks that the exception IDs used in the Python `EmbeddingMap` (in `artiq.language.embedding`)
+         * match the `EXCEPTION_ID_LOOKUP` defined in the firmware (`libksupport::src::eh_artiq`)
+         */
+        api!(test_exception_id_sync = eh_artiq::test_exception_id_sync)
     ];
     api.iter()
         .find(|&&(exported, _)| exported.as_bytes() == required)

src/libksupport/src/kernel/dma.rs

@@ -82,7 +82,7 @@ pub extern "C" fn dma_record_stop(duration: i64, enable_ddma: bool) {
 #[inline(always)]
 unsafe fn dma_record_output_prepare(timestamp: i64, target: i32, words: usize) {
     // See gateware/rtio/dma.py.
-    const HEADER_LENGTH: usize = /*length*/1 + /*channel*/3 + /*timestamp*/8 + /*address*/1;
+    const HEADER_LENGTH: usize = /*length*/ 1 + /*channel*/3 + /*timestamp*/8 + /*address*/1;
     let length = HEADER_LENGTH + /*data*/words * 4;
 
     let buffer = &mut RECORDER.as_mut().unwrap().buffer;

src/libksupport/src/kernel/linalg.rs

@@ -0,0 +1,440 @@
+// Uses `nalgebra` crate to invoke `np_linalg` and `sp_linalg` functions
+// When converting between `nalgebra::Matrix` and `NDArray` following considerations are necessary
+//
+// * Both `nalgebra::Matrix` and `NDArray` require their content to be stored in row-major order
+// * `NDArray` data pointer can be directly read and converted to `nalgebra::Matrix` (row and column number must be known)
+// * `nalgebra::Matrix::as_slice` returns the content of matrix in column-major order and initial data needs to be transposed before storing it in `NDArray` data pointer
+
+use alloc::vec::Vec;
+use core::slice;
+
+use nalgebra::DMatrix;
+
+use crate::artiq_raise;
+
+pub struct InputMatrix {
+    pub ndims: usize,
+    pub dims: *const usize,
+    pub data: *mut f64,
+}
+
+impl InputMatrix {
+    fn get_dims(&mut self) -> Vec<usize> {
+        let dims = unsafe { slice::from_raw_parts(self.dims, self.ndims) };
+        dims.to_vec()
+    }
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_cholesky(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix1 = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let result = matrix1.cholesky();
+    match result {
+        Some(res) => {
+            out_slice.copy_from_slice(res.unpack().transpose().as_slice());
+        }
+        None => {
+            artiq_raise!("LinAlgError", "Matrix is not positive definite");
+        }
+    };
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_qr(mat1: *mut InputMatrix, out_q: *mut InputMatrix, out_r: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_q = out_q.as_mut().unwrap();
+    let out_r = out_r.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let outq_dim = (*out_q).get_dims();
+    let outr_dim = (*out_r).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_q_slice = slice::from_raw_parts_mut(out_q.data, outq_dim[0] * outq_dim[1]);
+    let out_r_slice = slice::from_raw_parts_mut(out_r.data, outr_dim[0] * outr_dim[1]);
+
+    // Refer to https://github.com/dimforge/nalgebra/issues/735
+    let matrix1 = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+
+    let res = matrix1.qr();
+    let (q, r) = res.unpack();
+
+    // Uses different algo need to match numpy
+    out_q_slice.copy_from_slice(q.transpose().as_slice());
+    out_r_slice.copy_from_slice(r.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_svd(
+    mat1: *mut InputMatrix,
+    outu: *mut InputMatrix,
+    outs: *mut InputMatrix,
+    outvh: *mut InputMatrix,
+) {
+    let mat1 = mat1.as_mut().unwrap();
+    let outu = outu.as_mut().unwrap();
+    let outs = outs.as_mut().unwrap();
+    let outvh = outvh.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let outu_dim = (*outu).get_dims();
+    let outs_dim = (*outs).get_dims();
+    let outvh_dim = (*outvh).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_u_slice = slice::from_raw_parts_mut(outu.data, outu_dim[0] * outu_dim[1]);
+    let out_s_slice = slice::from_raw_parts_mut(outs.data, outs_dim[0]);
+    let out_vh_slice = slice::from_raw_parts_mut(outvh.data, outvh_dim[0] * outvh_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let result = matrix.svd(true, true);
+    out_u_slice.copy_from_slice(result.u.unwrap().transpose().as_slice());
+    out_s_slice.copy_from_slice(result.singular_values.as_slice());
+    out_vh_slice.copy_from_slice(result.v_t.unwrap().transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_inv(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+    let dim1 = (*mat1).get_dims();
+
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    if !matrix.is_invertible() {
+        artiq_raise!("LinAlgError", "no inverse for Singular Matrix");
+    }
+    let inv = matrix.try_inverse().unwrap();
+    out_slice.copy_from_slice(inv.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_pinv(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+    let dim1 = (*mat1).get_dims();
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let svd = matrix.svd(true, true);
+    let inv = svd.pseudo_inverse(1e-15);
+
+    match inv {
+        Ok(m) => {
+            out_slice.copy_from_slice(m.transpose().as_slice());
+        }
+        Err(_) => {
+            artiq_raise!("LinAlgError", "SVD computation does not converge");
+        }
+    }
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_matrix_power(mat1: *mut InputMatrix, mat2: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let mat2 = mat2.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let power = slice::from_raw_parts_mut(mat2.data, 1);
+    let power = power[0];
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let mut abs_power = power;
+    if abs_power < 0.0 {
+        abs_power = abs_power * -1.0;
+    }
+    let matrix1 = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    if !matrix1.is_square() {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+    let mut result = matrix1.pow(abs_power as u32);
+
+    if power < 0.0 {
+        if !matrix1.is_invertible() {
+            artiq_raise!("LinAlgError", "no inverse for Singular Matrix");
+        }
+        result = result.try_inverse().unwrap();
+    }
+    out_slice.copy_from_slice(result.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_det(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+    let dim1 = (*mat1).get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, 1);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    if !matrix.is_square() {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+    out_slice[0] = matrix.determinant();
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn sp_linalg_lu(mat1: *mut InputMatrix, out_l: *mut InputMatrix, out_u: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_l = out_l.as_mut().unwrap();
+    let out_u = out_u.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let outl_dim = (*out_l).get_dims();
+    let outu_dim = (*out_u).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_l_slice = slice::from_raw_parts_mut(out_l.data, outl_dim[0] * outl_dim[1]);
+    let out_u_slice = slice::from_raw_parts_mut(out_u.data, outu_dim[0] * outu_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let (_, l, u) = matrix.lu().unpack();
+
+    out_l_slice.copy_from_slice(l.transpose().as_slice());
+    out_u_slice.copy_from_slice(u.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn sp_linalg_schur(mat1: *mut InputMatrix, out_t: *mut InputMatrix, out_z: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_t = out_t.as_mut().unwrap();
+    let out_z = out_z.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let out_t_dim = (*out_t).get_dims();
+    let out_z_dim = (*out_z).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_t_slice = slice::from_raw_parts_mut(out_t.data, out_t_dim[0] * out_t_dim[1]);
+    let out_z_slice = slice::from_raw_parts_mut(out_z.data, out_z_dim[0] * out_z_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let (z, t) = matrix.schur().unpack();
+
+    out_t_slice.copy_from_slice(t.transpose().as_slice());
+    out_z_slice.copy_from_slice(z.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn sp_linalg_hessenberg(
+    mat1: *mut InputMatrix,
+    out_h: *mut InputMatrix,
+    out_q: *mut InputMatrix,
+) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_h = out_h.as_mut().unwrap();
+    let out_q = out_q.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let out_h_dim = (*out_h).get_dims();
+    let out_q_dim = (*out_q).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_h_slice = slice::from_raw_parts_mut(out_h.data, out_h_dim[0] * out_h_dim[1]);
+    let out_q_slice = slice::from_raw_parts_mut(out_q.data, out_q_dim[0] * out_q_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let (q, h) = matrix.hessenberg().unpack();
+
+    out_h_slice.copy_from_slice(h.transpose().as_slice());
+    out_q_slice.copy_from_slice(q.transpose().as_slice());
+}

src/libksupport/src/kernel/mod.rs

@@ -13,6 +13,7 @@ mod dma;
 mod rpc;
 pub use dma::DmaRecorder;
 mod cache;
+mod linalg;
 #[cfg(has_drtio)]
 mod subkernel;
 
@@ -23,6 +24,7 @@ pub enum SubkernelStatus {
     Timeout,
     IncorrectState,
     CommLost,
+    Exception(Vec<u8>),
     OtherError,
 }
 
@@ -79,6 +81,7 @@ pub enum Message {
         id: u32,
         destination: u8,
         run: bool,
+        timestamp: u64,
     },
     #[cfg(has_drtio)]
     SubkernelLoadRunReply {
@@ -90,9 +93,7 @@ pub enum Message {
         timeout: i64,
     },
     #[cfg(has_drtio)]
-    SubkernelAwaitFinishReply {
+    SubkernelAwaitFinishReply,
-        status: SubkernelStatus,
-    },
     #[cfg(has_drtio)]
     SubkernelMsgSend {
         id: u32,
@@ -109,9 +110,10 @@ pub enum Message {
     },
     #[cfg(has_drtio)]
     SubkernelMsgRecvReply {
-        status: SubkernelStatus,
         count: u8,
     },
+    #[cfg(has_drtio)]
+    SubkernelError(SubkernelStatus),
 }
 
 static CHANNEL_0TO1: Mutex<Option<sync_channel::Sender<'static, Message>>> = Mutex::new(None);

src/libksupport/src/kernel/subkernel.rs

@@ -3,7 +3,7 @@ use alloc::vec::Vec;
 use cslice::CSlice;
 
 use super::{Message, SubkernelStatus, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0};
-use crate::{artiq_raise, rpc::send_args};
+use crate::{artiq_raise, eh_artiq, rpc::send_args, rtio::now_mu};
 
 pub extern "C" fn load_run(id: u32, destination: u8, run: bool) {
     unsafe {
@@ -14,6 +14,7 @@ pub extern "C" fn load_run(id: u32, destination: u8, run: bool) {
                 id: id,
                 destination: destination,
                 run: run,
+                timestamp: now_mu() as u64,
             });
     }
     match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
@@ -36,21 +37,18 @@ pub extern "C" fn await_finish(id: u32, timeout: i64) {
             });
     }
     match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
-        Message::SubkernelAwaitFinishReply {
+        Message::SubkernelAwaitFinishReply => (),
-            status: SubkernelStatus::NoError,
+        Message::SubkernelError(SubkernelStatus::IncorrectState) => {
-        } => (),
+            artiq_raise!("SubkernelError", "Subkernel not running")
-        Message::SubkernelAwaitFinishReply {
+        }
-            status: SubkernelStatus::IncorrectState,
+        Message::SubkernelError(SubkernelStatus::Timeout) => artiq_raise!("SubkernelError", "Subkernel timed out"),
-        } => artiq_raise!("SubkernelError", "Subkernel not running"),
+        Message::SubkernelError(SubkernelStatus::CommLost) => {
-        Message::SubkernelAwaitFinishReply {
+            artiq_raise!("SubkernelError", "Lost communication with satellite")
-            status: SubkernelStatus::Timeout,
+        }
-        } => artiq_raise!("SubkernelError", "Subkernel timed out"),
+        Message::SubkernelError(SubkernelStatus::OtherError) => {
-        Message::SubkernelAwaitFinishReply {
+            artiq_raise!("SubkernelError", "An error occurred during subkernel operation")
-            status: SubkernelStatus::CommLost,
+        }
-        } => artiq_raise!("SubkernelError", "Lost communication with satellite"),
+        Message::SubkernelError(SubkernelStatus::Exception(raw_exception)) => eh_artiq::raise_raw(&raw_exception),
-        Message::SubkernelAwaitFinishReply {
-            status: SubkernelStatus::OtherError,
-        } => artiq_raise!("SubkernelError", "An error occurred during subkernel operation"),
         _ => panic!("expected SubkernelAwaitFinishReply after SubkernelAwaitFinishRequest"),
     }
 }
@@ -92,30 +90,22 @@ pub extern "C" fn await_message(id: i32, timeout: i64, tags: &CSlice<u8>, min: u
             });
     }
     match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
-        Message::SubkernelMsgRecvReply {
+        Message::SubkernelMsgRecvReply { count } => {
-            status: SubkernelStatus::NoError,
-            count,
-        } => {
             if min > count || count > max {
                 artiq_raise!("SubkernelError", "Received more or less arguments than required")
             }
         }
-        Message::SubkernelMsgRecvReply {
+        Message::SubkernelError(SubkernelStatus::IncorrectState) => {
-            status: SubkernelStatus::IncorrectState,
+            artiq_raise!("SubkernelError", "Subkernel not running")
-            ..
+        }
-        } => artiq_raise!("SubkernelError", "Subkernel not running"),
+        Message::SubkernelError(SubkernelStatus::Timeout) => artiq_raise!("SubkernelError", "Subkernel timed out"),
-        Message::SubkernelMsgRecvReply {
+        Message::SubkernelError(SubkernelStatus::CommLost) => {
-            status: SubkernelStatus::Timeout,
+            artiq_raise!("SubkernelError", "Lost communication with satellite")
-            ..
+        }
-        } => artiq_raise!("SubkernelError", "Subkernel timed out"),
+        Message::SubkernelError(SubkernelStatus::OtherError) => {
-        Message::SubkernelMsgRecvReply {
+            artiq_raise!("SubkernelError", "An error occurred during subkernel operation")
-            status: SubkernelStatus::CommLost,
+        }
-            ..
+        Message::SubkernelError(SubkernelStatus::Exception(raw_exception)) => eh_artiq::raise_raw(&raw_exception),
-        } => artiq_raise!("SubkernelError", "Lost communication with satellite"),
-        Message::SubkernelMsgRecvReply {
-            status: SubkernelStatus::OtherError,
-            ..
-        } => artiq_raise!("SubkernelError", "An error occurred during subkernel operation"),
         _ => panic!("expected SubkernelMsgRecvReply after SubkernelMsgRecvRequest"),
     }
     // RpcRecvRequest should be called after this to receive message data

src/libksupport/src/lib.rs

@@ -1,7 +1,8 @@
 #![no_std]
+#![allow(incomplete_features)]
 #![feature(c_variadic)]
 #![feature(const_btree_new)]
-#![feature(const_in_array_repeat_expressions)]
+#![feature(inline_const)]
 #![feature(naked_functions)]
 #![feature(asm)]
 

src/libunwind/lib.rs

@@ -1,7 +1,7 @@
 #![no_std]
 #![feature(link_cfg)]
 #![feature(nll)]
-#![feature(unwind_attributes)]
+#![feature(c_unwind)]
 #![feature(static_nobundle)]
 #![cfg_attr(not(target_env = "msvc"), feature(libc))]
 

src/libunwind/libunwind.rs

@@ -77,8 +77,7 @@ pub type _Unwind_Exception_Cleanup_Fn =
     all(feature = "llvm-libunwind", any(target_os = "fuchsia", target_os = "linux")),
     link(name = "unwind", kind = "static")
 )]
-extern "C" {
+extern "C-unwind" {
-    #[unwind(allowed)]
     pub fn _Unwind_Resume(exception: *mut _Unwind_Exception) -> !;
     pub fn _Unwind_DeleteException(exception: *mut _Unwind_Exception);
     pub fn _Unwind_GetLanguageSpecificData(ctx: *mut _Unwind_Context) -> *mut c_void;
@@ -226,8 +225,7 @@ if #[cfg(not(all(target_os = "ios", target_arch = "arm")))] {
     #[cfg_attr(all(feature = "llvm-libunwind",
                    any(target_os = "fuchsia", target_os = "linux")),
                link(name = "unwind", kind = "static"))]
-    extern "C" {
+    extern "C-unwind" {
-        #[unwind(allowed)]
         pub fn _Unwind_RaiseException(exception: *mut _Unwind_Exception) -> _Unwind_Reason_Code;
         pub fn _Unwind_Backtrace(trace: _Unwind_Trace_Fn,
                                  trace_argument: *mut c_void)
@@ -238,8 +236,7 @@ if #[cfg(not(all(target_os = "ios", target_arch = "arm")))] {
     #[cfg_attr(all(feature = "llvm-libunwind",
                    any(target_os = "fuchsia", target_os = "linux")),
                link(name = "unwind", kind = "static"))]
-    extern "C" {
+    extern "C-unwind" {
-        #[unwind(allowed)]
         pub fn _Unwind_SjLj_RaiseException(e: *mut _Unwind_Exception) -> _Unwind_Reason_Code;
     }
 

src/runtime/Cargo.toml.tpl

@@ -8,6 +8,7 @@ edition = "2018"
 [features]
 target_zc706 = ["libboard_zynq/target_zc706", "libsupport_zynq/target_zc706", "libconfig/target_zc706", "libboard_artiq/target_zc706"]
 target_kasli_soc = ["libboard_zynq/target_kasli_soc", "libsupport_zynq/target_kasli_soc", "libconfig/target_kasli_soc", "libboard_artiq/target_kasli_soc"]
+target_ebaz4205 = ["libboard_zynq/target_ebaz4205", "libsupport_zynq/target_ebaz4205", "libconfig/target_ebaz4205", "libboard_artiq/target_ebaz4205"]
 default = ["target_zc706"]
 
 [build-dependencies]
@@ -19,7 +20,8 @@ num-derive = "0.3"
 cslice = "0.3"
 log = "0.4"
 embedded-hal = "0.2"
-core_io = { version = "0.1", features = ["collections"] }
+core_io = { git = "https://git.m-labs.hk/M-Labs/rs-core_io.git", rev = "e9d3edf027", features = ["collections"] }
+crc = { version = "1.7", default-features = false }
 byteorder = { version = "1.3", default-features = false }
 void = { version = "1", default-features = false }
 futures = { version = "0.3", default-features = false, features = ["async-await"] }
@@ -44,4 +46,4 @@ libboard_artiq = { path = "../libboard_artiq" }
 
 [dependencies.tar-no-std]
 git = "https://git.m-labs.hk/M-Labs/tar-no-std"
-rev = "2ab6dc5"
+rev = "2ab6dc5"

src/runtime/src/analyzer.rs

@@ -60,7 +60,7 @@ pub mod remote_analyzer {
         routing_table: &drtio_routing::RoutingTable,
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         timer: GlobalTimer,
-    ) -> Result<RemoteBuffer, &'static str> {
+    ) -> Result<RemoteBuffer, drtio::Error> {
         // gets data from satellites and returns consolidated data
         let mut remote_data: Vec<u8> = Vec::new();
         let mut remote_error = false;

src/runtime/src/comms.rs

@@ -10,15 +10,11 @@ use io::Cursor;
 #[cfg(has_drtio)]
 use ksupport::rpc;
 use ksupport::{kernel, resolve_channel_name};
-#[cfg(has_drtio)]
-use libasync::delay;
 use libasync::{smoltcp::{Sockets, TcpStream},
                task};
 use libboard_artiq::drtio_routing;
 #[cfg(feature = "target_kasli_soc")]
 use libboard_zynq::error_led::ErrorLED;
-#[cfg(has_drtio)]
-use libboard_zynq::time::Milliseconds;
 use libboard_zynq::{self as zynq,
                     smoltcp::{self,
                               iface::{EthernetInterfaceBuilder, NeighborCache},
@@ -405,8 +401,9 @@ async fn handle_run_kernel(
                 id,
                 destination: _,
                 run,
+                timestamp,
             } => {
-                let succeeded = match subkernel::load(aux_mutex, routing_table, timer, id, run).await {
+                let succeeded = match subkernel::load(aux_mutex, routing_table, timer, id, run, timestamp).await {
                     Ok(()) => true,
                     Err(e) => {
                         error!("Error loading subkernel: {:?}", e);
@@ -422,33 +419,23 @@ async fn handle_run_kernel(
             #[cfg(has_drtio)]
             kernel::Message::SubkernelAwaitFinishRequest { id, timeout } => {
                 let res = subkernel::await_finish(aux_mutex, routing_table, timer, id, timeout).await;
-                let status = match res {
+                let response = match res {
-                    Ok(ref res) => {
+                    Ok(res) => {
                         if res.status == subkernel::FinishStatus::CommLost {
-                            kernel::SubkernelStatus::CommLost
+                            kernel::Message::SubkernelError(kernel::SubkernelStatus::CommLost)
-                        } else if let Some(exception) = &res.exception {
+                        } else if let Some(exception) = res.exception {
-                            error!("Exception in subkernel");
+                            kernel::Message::SubkernelError(kernel::SubkernelStatus::Exception(exception))
-                            match stream {
-                                None => (),
-                                Some(stream) => {
-                                    write_chunk(stream, exception).await?;
-                                }
-                            }
-                            // will not be called after exception is served
-                            kernel::SubkernelStatus::OtherError
                         } else {
-                            kernel::SubkernelStatus::NoError
+                            kernel::Message::SubkernelAwaitFinishReply
                         }
                     }
-                    Err(SubkernelError::Timeout) => kernel::SubkernelStatus::Timeout,
+                    Err(SubkernelError::Timeout) => kernel::Message::SubkernelError(kernel::SubkernelStatus::Timeout),
-                    Err(SubkernelError::IncorrectState) => kernel::SubkernelStatus::IncorrectState,
+                    Err(SubkernelError::IncorrectState) => {
-                    Err(_) => kernel::SubkernelStatus::OtherError,
+                        kernel::Message::SubkernelError(kernel::SubkernelStatus::IncorrectState)
+                    }
+                    Err(_) => kernel::Message::SubkernelError(kernel::SubkernelStatus::OtherError),
                 };
-                control
+                control.borrow_mut().tx.async_send(response).await;
-                    .borrow_mut()
-                    .tx
-                    .async_send(kernel::Message::SubkernelAwaitFinishReply { status: status })
-                    .await;
             }
             #[cfg(has_drtio)]
             kernel::Message::SubkernelMsgSend { id, destination, data } => {
@@ -469,35 +456,23 @@ async fn handle_run_kernel(
             #[cfg(has_drtio)]
             kernel::Message::SubkernelMsgRecvRequest { id, timeout, tags } => {
                 let message_received = subkernel::message_await(id as u32, timeout, timer).await;
-                let (status, count) = match message_received {
+                let response = match message_received {
-                    Ok(ref message) => (kernel::SubkernelStatus::NoError, message.count),
+                    Ok(ref message) => kernel::Message::SubkernelMsgRecvReply { count: message.count },
-                    Err(SubkernelError::Timeout) => (kernel::SubkernelStatus::Timeout, 0),
+                    Err(SubkernelError::Timeout) => kernel::Message::SubkernelError(kernel::SubkernelStatus::Timeout),
-                    Err(SubkernelError::IncorrectState) => (kernel::SubkernelStatus::IncorrectState, 0),
+                    Err(SubkernelError::IncorrectState) => {
-                    Err(SubkernelError::CommLost) => (kernel::SubkernelStatus::CommLost, 0),
+                        kernel::Message::SubkernelError(kernel::SubkernelStatus::IncorrectState)
+                    }
+                    Err(SubkernelError::CommLost) => kernel::Message::SubkernelError(kernel::SubkernelStatus::CommLost),
                     Err(SubkernelError::SubkernelException) => {
-                        error!("Exception in subkernel");
                         // just retrieve the exception
                         let status = subkernel::await_finish(aux_mutex, routing_table, timer, id as u32, timeout)
                             .await
                             .unwrap();
-                        match stream {
+                        kernel::Message::SubkernelError(kernel::SubkernelStatus::Exception(status.exception.unwrap()))
-                            None => (),
-                            Some(stream) => {
-                                write_chunk(stream, &status.exception.unwrap()).await?;
-                            }
-                        }
-                        (kernel::SubkernelStatus::OtherError, 0)
                     }
-                    Err(_) => (kernel::SubkernelStatus::OtherError, 0),
+                    Err(_) => kernel::Message::SubkernelError(kernel::SubkernelStatus::OtherError),
                 };
-                control
+                control.borrow_mut().tx.async_send(response).await;
-                    .borrow_mut()
-                    .tx
-                    .async_send(kernel::Message::SubkernelMsgRecvReply {
-                        status: status,
-                        count: count,
-                    })
-                    .await;
                 if let Ok(message) = message_received {
                     // receive code almost identical to RPC recv, except we are not reading from a stream
                     let mut reader = Cursor::new(message.data);
@@ -529,7 +504,7 @@ async fn handle_run_kernel(
                             .async_send(kernel::Message::RpcRecvReply(Ok(0)))
                             .await;
                         i += 1;
-                        if i < count {
+                        if i < message.count {
                             current_tags = remaining_tags;
                         } else {
                             break;
@@ -771,14 +746,13 @@ pub fn main(timer: GlobalTimer, cfg: Config) {
     #[cfg(has_drtio_routing)]
     drtio_routing::interconnect_disable_all();
 
-    rtio_mgt::startup(&aux_mutex, &drtio_routing_table, &up_destinations, timer);
+    rtio_mgt::startup(&aux_mutex, &drtio_routing_table, &up_destinations, &cfg, timer);
     ksupport::setup_device_map(&cfg);
 
     analyzer::start(&aux_mutex, &drtio_routing_table, &up_destinations, timer);
     moninj::start(timer, &aux_mutex, &drtio_routing_table);
 
     let control: Rc<RefCell<kernel::Control>> = Rc::new(RefCell::new(kernel::Control::start()));
-    let idle_kernel = Rc::new(cfg.read("idle_kernel").ok());
     if let Ok(buffer) = cfg.read("startup_kernel") {
         info!("Loading startup kernel...");
         let routing_table = drtio_routing_table.borrow();
@@ -805,13 +779,34 @@ pub fn main(timer: GlobalTimer, cfg: Config) {
         }
     }
 
-    mgmt::start(cfg);
+    let cfg = Rc::new(cfg);
+    let restart_idle = Rc::new(Semaphore::new(1, 1));
+    mgmt::start(
+        cfg.clone(),
+        restart_idle.clone(),
+        Some(mgmt::DrtioContext(
+            aux_mutex.clone(),
+            drtio_routing_table.clone(),
+            timer,
+        )),
+    );
 
     task::spawn(async move {
         let connection = Rc::new(Semaphore::new(1, 1));
         let terminate = Rc::new(Semaphore::new(0, 1));
+        let can_restart_idle = Rc::new(Semaphore::new(1, 1));
+        let restart_idle = restart_idle.clone();
         loop {
-            let mut stream = TcpStream::accept(1381, 0x10_000, 0x10_000).await.unwrap();
+            let control = control.clone();
+            let mut maybe_stream = select_biased! {
+                s = (async {
+                        TcpStream::accept(1381, 0x10_000, 0x10_000).await.unwrap()
+                    }).fuse() => Some(s),
+                _ = (async {
+                        restart_idle.async_wait().await;
+                        can_restart_idle.async_wait().await;
+                    }).fuse() => None
+            };
 
             if connection.try_wait().is_none() {
                 // there is an existing connection
@@ -819,47 +814,58 @@ pub fn main(timer: GlobalTimer, cfg: Config) {
                 connection.async_wait().await;
             }
 
+            let maybe_idle_kernel = cfg.read("idle_kernel").ok();
+            if maybe_idle_kernel.is_none() && maybe_stream.is_none() {
+                control.borrow_mut().restart(); // terminate idle kernel if running
+            }
+
             let control = control.clone();
-            let idle_kernel = idle_kernel.clone();
             let connection = connection.clone();
             let terminate = terminate.clone();
+            let can_restart_idle = can_restart_idle.clone();
             let up_destinations = up_destinations.clone();
             let aux_mutex = aux_mutex.clone();
             let routing_table = drtio_routing_table.clone();
 
             // we make sure the value of terminate is 0 before we start
             let _ = terminate.try_wait();
+            let _ = can_restart_idle.try_wait();
             task::spawn(async move {
                 let routing_table = routing_table.borrow();
                 select_biased! {
                     _ = (async {
-                        let _ = handle_connection(&mut stream, control.clone(), &up_destinations, &aux_mutex, &routing_table, timer)
+                        if let Some(stream) = &mut maybe_stream {
-                            .await
+                            let _ = handle_connection(stream, control.clone(), &up_destinations, &aux_mutex, &routing_table, timer)
-                            .map_err(|e| warn!("connection terminated: {}", e));
+                                .await
-                        if let Some(buffer) = &*idle_kernel {
+                                .map_err(|e| warn!("connection terminated: {}", e));
-                            info!("Loading idle kernel");
+                        }
-                            let res = handle_flash_kernel(&buffer, &control, &up_destinations,  &aux_mutex, &routing_table, timer)
+                        can_restart_idle.signal();
-                                .await;
+                        match maybe_idle_kernel {
-                            match res {
+                            Some(buffer) => {
-                                #[cfg(has_drtio)]
+                                loop {
-                                Err(Error::DestinationDown) => {
+                                    info!("loading idle kernel");
-                                    let mut countdown = timer.countdown();
+                                    match handle_flash_kernel(&buffer, &control, &up_destinations, &aux_mutex, &routing_table, timer).await {
-                                    delay(&mut countdown, Milliseconds(500)).await;
+                                        Ok(_) => {
+                                            info!("running idle kernel");
+                                            match handle_run_kernel(None, &control, &up_destinations, &aux_mutex, &routing_table, timer).await {
+                                                Ok(_) => info!("idle kernel finished"),
+                                                Err(_) => warn!("idle kernel running error")
+                                            }
+                                        },
+                                        Err(_) => warn!("idle kernel loading error")
+                                    }
                                 }
-                                Err(_) => warn!("error loading idle kernel"),
+                            },
-                                _ => (),
+                            None => info!("no idle kernel found")
-                            }
-                            info!("Running idle kernel");
-                            let _ = handle_run_kernel(None, &control, &up_destinations, &aux_mutex, &routing_table, timer)
-                                .await.map_err(|_| warn!("error running idle kernel"));
-                            info!("Idle kernel terminated");
                         }
                     }).fuse() => (),
                     _ = terminate.async_wait().fuse() => ()
                 }
                 connection.signal();
-                let _ = stream.flush().await;
+                if let Some(stream) = maybe_stream {
-                let _ = stream.abort().await;
+                    let _ = stream.flush().await;
+                    let _ = stream.abort().await;
+                }
             });
         }
     });
@@ -908,7 +914,8 @@ pub fn soft_panic_main(timer: GlobalTimer, cfg: Config) -> ! {
 
     Sockets::init(32);
 
-    mgmt::start(cfg);
+    let dummy = Rc::new(Semaphore::new(0, 1));
+    mgmt::start(Rc::new(cfg), dummy, None);
 
     // getting eth settings disables the LED as it resets GPIO
     // need to re-enable it here

src/runtime/src/main.rs

@@ -8,20 +8,20 @@
 #[macro_use]
 extern crate alloc;
 
-#[cfg(all(feature = "target_kasli_soc", has_drtio))]
+#[cfg(all(feature = "target_kasli_soc", has_virtual_leds))]
 use core::cell::RefCell;
 
 use ksupport;
 use libasync::task;
 #[cfg(has_drtio_eem)]
 use libboard_artiq::drtio_eem;
-#[cfg(all(feature = "target_kasli_soc", has_drtio))]
+#[cfg(feature = "target_kasli_soc")]
 use libboard_artiq::io_expander;
 use libboard_artiq::{identifier_read, logger, pl};
 use libboard_zynq::{gic, mpcore, timer::GlobalTimer};
 use libconfig::Config;
 use libcortex_a9::l2c::enable_l2_cache;
-use libsupport_zynq::ram;
+use libsupport_zynq::{exception_vectors, ram};
 use log::{info, warn};
 
 mod analyzer;
@@ -38,7 +38,12 @@ mod rtio_mgt;
 #[cfg(has_drtio)]
 mod subkernel;
 
-#[cfg(all(feature = "target_kasli_soc", has_drtio))]
+// linker symbols
+extern "C" {
+    static __exceptions_start: u32;
+}
+
+#[cfg(all(feature = "target_kasli_soc", has_virtual_leds))]
 async fn io_expanders_service(
     i2c_bus: RefCell<&mut libboard_zynq::i2c::I2c>,
     io_expander0: RefCell<io_expander::IoExpander>,
@@ -77,6 +82,9 @@ static mut LOG_BUFFER: [u8; 1 << 17] = [0; 1 << 17];
 
 #[no_mangle]
 pub fn main_core0() {
+    unsafe {
+        exception_vectors::set_vector_table(&__exceptions_start as *const u32 as u32);
+    }
     enable_l2_cache(0x8);
     let mut timer = GlobalTimer::start();
 
@@ -93,7 +101,7 @@ pub fn main_core0() {
     info!("gateware ident: {}", identifier_read(&mut [0; 64]));
 
     ksupport::i2c::init();
-    #[cfg(all(feature = "target_kasli_soc", has_drtio))]
+    #[cfg(feature = "target_kasli_soc")]
     {
         let i2c_bus = unsafe { (ksupport::i2c::I2C_BUS).as_mut().unwrap() };
         let mut io_expander0 = io_expander::IoExpander::new(i2c_bus, 0).unwrap();
@@ -104,6 +112,11 @@ pub fn main_core0() {
         io_expander1
             .init(i2c_bus)
             .expect("I2C I/O expander #1 initialization failed");
+
+        // Drive CLK_SEL to true
+        #[cfg(has_si549)]
+        io_expander0.set(1, 7, true);
+
         // Drive TX_DISABLE to false on SFP0..3
         io_expander0.set(0, 1, false);
         io_expander1.set(0, 1, false);
@@ -111,6 +124,7 @@ pub fn main_core0() {
         io_expander1.set(1, 1, false);
         io_expander0.service(i2c_bus).unwrap();
         io_expander1.service(i2c_bus).unwrap();
+        #[cfg(has_virtual_leds)]
         task::spawn(io_expanders_service(
             RefCell::new(i2c_bus),
             RefCell::new(io_expander0),

src/runtime/src/moninj.rs

@@ -58,10 +58,11 @@ mod remote_moninj {
     use log::error;
 
     use super::*;
-    use crate::rtio_mgt::drtio;
+    use crate::rtio_mgt::{drtio, drtio::Error as DrtioError};
 
     pub async fn read_probe(
         aux_mutex: &Rc<Mutex<bool>>,
+        routing_table: &drtio_routing::RoutingTable,
         timer: GlobalTimer,
         linkno: u8,
         destination: u8,
@@ -71,6 +72,7 @@ mod remote_moninj {
         let reply = drtio::aux_transact(
             aux_mutex,
             linkno,
+            routing_table,
             &drtioaux_async::Packet::MonitorRequest {
                 destination: destination,
                 channel: channel as _,
@@ -82,8 +84,8 @@ mod remote_moninj {
         match reply {
             Ok(drtioaux_async::Packet::MonitorReply { value }) => return value as i64,
             Ok(packet) => error!("received unexpected aux packet: {:?}", packet),
-            Err("link went down") => {
+            Err(DrtioError::LinkDown) => {
-                debug!("link is down");
+                warn!("link is down");
             }
             Err(e) => error!("aux packet error ({})", e),
         }
@@ -92,6 +94,7 @@ mod remote_moninj {
 
     pub async fn inject(
         aux_mutex: &Rc<Mutex<bool>>,
+        _routing_table: &drtio_routing::RoutingTable,
         _timer: GlobalTimer,
         linkno: u8,
         destination: u8,
@@ -99,7 +102,7 @@ mod remote_moninj {
         overrd: i8,
         value: i8,
     ) {
-        let _lock = aux_mutex.lock();
+        let _lock = aux_mutex.async_lock().await;
         drtioaux_async::send(
             linkno,
             &drtioaux_async::Packet::InjectionRequest {
@@ -115,6 +118,7 @@ mod remote_moninj {
 
     pub async fn read_injection_status(
         aux_mutex: &Rc<Mutex<bool>>,
+        routing_table: &drtio_routing::RoutingTable,
         timer: GlobalTimer,
         linkno: u8,
         destination: u8,
@@ -124,6 +128,7 @@ mod remote_moninj {
         let reply = drtio::aux_transact(
             aux_mutex,
             linkno,
+            routing_table,
             &drtioaux_async::Packet::InjectionStatusRequest {
                 destination: destination,
                 channel: channel as _,
@@ -135,8 +140,8 @@ mod remote_moninj {
         match reply {
             Ok(drtioaux_async::Packet::InjectionStatusReply { value }) => return value as i8,
             Ok(packet) => error!("received unexpected aux packet: {:?}", packet),
-            Err("link went down") => {
+            Err(DrtioError::LinkDown) => {
-                debug!("link is down");
+                warn!("link is down");
             }
             Err(e) => error!("aux packet error ({})", e),
         }
@@ -183,7 +188,7 @@ macro_rules! dispatch {
             local_moninj::$func(channel.into(), $($param, )*)
         } else {
             let linkno = hop - 1 as u8;
-            remote_moninj::$func($aux_mutex, $timer, linkno, destination, channel, $($param, )*).await
+            remote_moninj::$func($aux_mutex, $routing_table, $timer, linkno, destination, channel, $($param, )*).await
         }
     }}
 }

src/runtime/src/rtio_clocking.rs

@@ -1,14 +1,20 @@
+#[cfg(not(feature = "target_ebaz4205"))]
 use embedded_hal::blocking::delay::DelayMs;
 #[cfg(has_si5324)]
 use ksupport::i2c;
+#[cfg(not(feature = "target_ebaz4205"))]
 use libboard_artiq::pl;
 #[cfg(has_si5324)]
 use libboard_artiq::si5324;
+#[cfg(has_si549)]
+use libboard_artiq::si549;
 #[cfg(has_si5324)]
 use libboard_zynq::i2c::I2c;
 use libboard_zynq::timer::GlobalTimer;
 use libconfig::Config;
 use log::{info, warn};
+#[cfg(feature = "target_ebaz4205")]
+use {libboard_zynq::slcr, libregister::RegisterRW};
 
 #[derive(Debug, PartialEq, Copy, Clone)]
 #[allow(non_camel_case_types)]
@@ -67,7 +73,7 @@ fn get_rtio_clock_cfg(cfg: &Config) -> RtioClock {
     res
 }
 
-#[cfg(not(has_drtio))]
+#[cfg(not(any(has_drtio, feature = "target_ebaz4205")))]
 fn init_rtio(timer: &mut GlobalTimer) {
     info!("Switching SYS clocks...");
     unsafe {
@@ -260,6 +266,182 @@ fn setup_si5324(i2c: &mut I2c, timer: &mut GlobalTimer, clk: RtioClock) {
     si5324::setup(i2c, &si5324_settings, si5324_ref_input, timer).expect("cannot initialize Si5324");
 }
 
+#[cfg(all(has_si549, has_wrpll))]
+fn wrpll_setup(timer: &mut GlobalTimer, clk: RtioClock, si549_settings: &si549::FrequencySetting) {
+    // register values are directly copied from preconfigured mmcm
+    let (mmcm_setting, mmcm_bypass) = match clk {
+        RtioClock::Ext0_Synth0_10to125 => (
+            si549::wrpll_refclk::MmcmSetting {
+                // CLKFBOUT_MULT = 62.5, DIVCLK_DIVIDE = 1 , CLKOUT0_DIVIDE = 5
+                clkout0_reg1: 0x1083,
+                clkout0_reg2: 0x0080,
+                clkfbout_reg1: 0x179e,
+                clkfbout_reg2: 0x4c00,
+                div_reg: 0x1041,
+                lock_reg1: 0x00fa,
+                lock_reg2: 0x7c01,
+                lock_reg3: 0xffe9,
+                power_reg: 0x9900,
+                filt_reg1: 0x1008,
+                filt_reg2: 0x8800,
+            },
+            false,
+        ),
+        RtioClock::Ext0_Synth0_80to125 => (
+            si549::wrpll_refclk::MmcmSetting {
+                // CLKFBOUT_MULT = 15.625, DIVCLK_DIVIDE = 1 , CLKOUT0_DIVIDE = 10
+                clkout0_reg1: 0x1145,
+                clkout0_reg2: 0x0000,
+                clkfbout_reg1: 0x11c7,
+                clkfbout_reg2: 0x5880,
+                div_reg: 0x1041,
+                lock_reg1: 0x028a,
+                lock_reg2: 0x7c01,
+                lock_reg3: 0xffe9,
+                power_reg: 0x9900,
+                filt_reg1: 0x9908,
+                filt_reg2: 0x8100,
+            },
+            false,
+        ),
+        RtioClock::Ext0_Synth0_100to125 => (
+            si549::wrpll_refclk::MmcmSetting {
+                // CLKFBOUT_MULT = 12.5, DIVCLK_DIVIDE = 1 , CLKOUT0_DIVIDE = 10
+                clkout0_reg1: 0x1145,
+                clkout0_reg2: 0x0000,
+                clkfbout_reg1: 0x1145,
+                clkfbout_reg2: 0x4c00,
+                div_reg: 0x1041,
+                lock_reg1: 0x0339,
+                lock_reg2: 0x7c01,
+                lock_reg3: 0xffe9,
+                power_reg: 0x9900,
+                filt_reg1: 0x9108,
+                filt_reg2: 0x0100,
+            },
+            false,
+        ),
+        RtioClock::Ext0_Synth0_125to125 => (
+            si549::wrpll_refclk::MmcmSetting {
+                // CLKFBOUT_MULT = 10, DIVCLK_DIVIDE = 1 , CLKOUT0_DIVIDE = 10
+                clkout0_reg1: 0x1145,
+                clkout0_reg2: 0x0000,
+                clkfbout_reg1: 0x1145,
+                clkfbout_reg2: 0x0000,
+                div_reg: 0x1041,
+                lock_reg1: 0x03e8,
+                lock_reg2: 0x7001,
+                lock_reg3: 0xf3e9,
+                power_reg: 0x0100,
+                filt_reg1: 0x9908,
+                filt_reg2: 0x1100,
+            },
+            true,
+        ),
+        _ => unreachable!(),
+    };
+
+    si549::helper_setup(timer, &si549_settings).expect("cannot initialize helper Si549");
+    si549::wrpll_refclk::setup(timer, mmcm_setting, mmcm_bypass).expect("cannot initialize ref clk for wrpll");
+    si549::wrpll::select_recovered_clock(true, timer);
+}
+
+#[cfg(has_si549)]
+fn get_si549_setting(clk: RtioClock) -> si549::FrequencySetting {
+    match clk {
+        RtioClock::Ext0_Synth0_10to125 => {
+            info!("using 10MHz reference to make 125MHz RTIO clock with WRPLL");
+        }
+        RtioClock::Ext0_Synth0_80to125 => {
+            info!("using 80MHz reference to make 125MHz RTIO clock with WRPLL");
+        }
+        RtioClock::Ext0_Synth0_100to125 => {
+            info!("using 100MHz reference to make 125MHz RTIO clock with WRPLL");
+        }
+        RtioClock::Ext0_Synth0_125to125 => {
+            info!("using 125MHz reference to make 125MHz RTIO clock with WRPLL");
+        }
+        RtioClock::Int_100 => {
+            info!("using internal 100MHz RTIO clock");
+        }
+        RtioClock::Int_125 => {
+            info!("using internal 125MHz RTIO clock");
+        }
+        _ => {
+            warn!(
+                "rtio_clock setting '{:?}' is unsupported. Falling back to default internal 125MHz RTIO clock.",
+                clk
+            );
+        }
+    };
+
+    match clk {
+        RtioClock::Int_100 => {
+            si549::FrequencySetting {
+                main: si549::DividerConfig {
+                    hsdiv: 0x06C,
+                    lsdiv: 0,
+                    fbdiv: 0x046C5F49797,
+                },
+                helper: si549::DividerConfig {
+                    // 100MHz*32767/32768
+                    hsdiv: 0x06C,
+                    lsdiv: 0,
+                    fbdiv: 0x046C5670BBD,
+                },
+            }
+        }
+        _ => {
+            // Everything else use 125MHz
+            si549::FrequencySetting {
+                main: si549::DividerConfig {
+                    hsdiv: 0x058,
+                    lsdiv: 0,
+                    fbdiv: 0x04815791F25,
+                },
+                helper: si549::DividerConfig {
+                    // 125MHz*32767/32768
+                    hsdiv: 0x058,
+                    lsdiv: 0,
+                    fbdiv: 0x04814E8F442,
+                },
+            }
+        }
+    }
+}
+
+#[cfg(feature = "target_ebaz4205")]
+fn set_fclk0_freq(clk: RtioClock, cfg: &Config) {
+    let io_pll_freq: u32 = 1_000_000_000; // Hardcoded in zynq-rs
+    let mut target_freq = 0;
+    let mut divisor0 = 1u8;
+
+    match clk {
+        RtioClock::Int_100 => {
+            target_freq = 100_000_000;
+            divisor0 = 10;
+        }
+        RtioClock::Int_125 => {
+            target_freq = 125_000_000;
+            divisor0 = 8;
+        }
+        _ => {
+            warn!("Unsupported RTIO Clock: '{:?}'", clk);
+            return;
+        }
+    }
+
+    slcr::RegisterBlock::unlocked(|slcr| {
+        slcr.fpga0_clk_ctrl.modify(|_, w| w.divisor0(divisor0));
+    });
+
+    info!(
+        "Set FCLK0 to {:.2} MHz (target: {} MHz).",
+        io_pll_freq as f64 / divisor0 as f64,
+        target_freq / 1_000_000
+    );
+}
+
 pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
     let clk = get_rtio_clock_cfg(cfg);
     #[cfg(has_si5324)]
@@ -274,9 +456,39 @@ pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
         }
     }
 
+    #[cfg(has_si549)]
+    let si549_settings = get_si549_setting(clk);
+
+    #[cfg(has_si549)]
+    si549::main_setup(timer, &si549_settings).expect("cannot initialize main Si549");
+
     #[cfg(has_drtio)]
     init_drtio(timer);
 
-    #[cfg(not(has_drtio))]
+    #[cfg(not(any(has_drtio, feature = "target_ebaz4205")))]
     init_rtio(timer);
+
+    #[cfg(feature = "target_ebaz4205")]
+    {
+        match clk {
+            RtioClock::Int_100 | RtioClock::Int_125 => {
+                set_fclk0_freq(clk, cfg);
+            }
+            _ => {} // Not set for external clocks
+        }
+    }
+
+    #[cfg(all(has_si549, has_wrpll))]
+    {
+        // SYS CLK switch will reset CSRs that are used by WRPLL
+        match clk {
+            RtioClock::Ext0_Synth0_10to125
+            | RtioClock::Ext0_Synth0_80to125
+            | RtioClock::Ext0_Synth0_100to125
+            | RtioClock::Ext0_Synth0_125to125 => {
+                wrpll_setup(timer, clk, &si549_settings);
+            }
+            _ => {}
+        }
+    }
 }

src/runtime/src/rtio_mgt.rs

@@ -1,19 +1,26 @@
 use alloc::rc::Rc;
 use core::cell::RefCell;
 
-use libboard_artiq::{drtio_routing, pl::csr};
+use libboard_artiq::{drtio_routing, drtio_routing::RoutingTable, pl::csr};
 use libboard_zynq::timer::GlobalTimer;
+use libconfig::Config;
 use libcortex_a9::mutex::Mutex;
+use log::{info, warn};
 
 #[cfg(has_drtio)]
 pub mod drtio {
     use alloc::vec::Vec;
+    use core::fmt;
 
     use embedded_hal::blocking::delay::DelayMs;
+    #[cfg(has_drtio_eem)]
+    use embedded_hal::blocking::delay::DelayUs;
     use ksupport::{resolve_channel_name, ASYNC_ERROR_BUSY, ASYNC_ERROR_COLLISION, ASYNC_ERROR_SEQUENCE_ERROR,
                    SEEN_ASYNC_ERRORS};
     use libasync::{delay, task};
-    use libboard_artiq::{drtioaux::Error,
+    #[cfg(has_drtio_eem)]
+    use libboard_artiq::drtio_eem;
+    use libboard_artiq::{drtioaux::Error as DrtioError,
                          drtioaux_async,
                          drtioaux_async::Packet,
                          drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE}};
@@ -23,9 +30,48 @@ pub mod drtio {
     use super::*;
     use crate::{analyzer::remote_analyzer::RemoteBuffer, rtio_dma::remote_dma, subkernel};
 
+    #[cfg(has_drtio_eem)]
+    const DRTIO_EEM_LINKNOS: core::ops::Range<usize> =
+        (csr::DRTIO.len() - csr::CONFIG_EEM_DRTIO_COUNT as usize)..csr::DRTIO.len();
+
+    #[derive(Debug, PartialEq, Eq, Clone, Copy)]
+    pub enum Error {
+        Timeout,
+        AuxError,
+        LinkDown,
+        UnexpectedReply,
+        DmaAddTraceFail(u8),
+        DmaEraseFail(u8),
+        DmaPlaybackFail(u8),
+        SubkernelAddFail(u8),
+        SubkernelRunFail(u8),
+    }
+
+    impl fmt::Display for Error {
+        fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+            match self {
+                Error::Timeout => write!(f, "timed out"),
+                Error::AuxError => write!(f, "aux packet error"),
+                Error::LinkDown => write!(f, "link down"),
+                Error::UnexpectedReply => write!(f, "unexpected reply"),
+                Error::DmaAddTraceFail(dest) => write!(f, "error adding DMA trace on satellite #{}", dest),
+                Error::DmaEraseFail(dest) => write!(f, "error erasing DMA trace on satellite #{}", dest),
+                Error::DmaPlaybackFail(dest) => write!(f, "error playing back DMA trace on satellite #{}", dest),
+                Error::SubkernelAddFail(dest) => write!(f, "error adding subkernel on satellite #{}", dest),
+                Error::SubkernelRunFail(dest) => write!(f, "error on subkernel run request on satellite #{}", dest),
+            }
+        }
+    }
+
+    impl From<DrtioError> for Error {
+        fn from(_error: DrtioError) -> Self {
+            Error::AuxError
+        }
+    }
+
     pub fn startup(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &Rc<RefCell<drtio_routing::RoutingTable>>,
+        routing_table: &Rc<RefCell<RoutingTable>>,
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         timer: GlobalTimer,
     ) {
@@ -38,156 +84,146 @@ pub mod drtio {
         });
     }
 
-    async fn link_rx_up(linkno: u8) -> bool {
+    async fn link_rx_up(linkno: u8, _timer: &mut GlobalTimer) -> bool {
         let linkno = linkno as usize;
+        #[cfg(has_drtio_eem)]
+        if DRTIO_EEM_LINKNOS.contains(&linkno) {
+            let eem_trx_no = linkno - DRTIO_EEM_LINKNOS.start;
+            unsafe {
+                csr::eem_transceiver::transceiver_sel_write(eem_trx_no as u8);
+                csr::eem_transceiver::comma_align_reset_write(1);
+            }
+            _timer.delay_us(100);
+            return unsafe { csr::eem_transceiver::comma_read() == 1 };
+        }
         unsafe { (csr::DRTIO[linkno].rx_up_read)() == 1 }
     }
 
-    async fn link_has_async_ready(linkno: u8) -> bool {
+    async fn process_async_packets(linkno: u8, routing_table: &RoutingTable, packet: Packet) -> Option<Packet> {
-        let linkno = linkno as usize;
+        match packet {
-        let async_ready;
+            Packet::DmaPlaybackStatus {
-        unsafe {
+                id,
-            async_ready = (csr::DRTIO[linkno].async_messages_ready_read)() == 1;
+                source,
-            (csr::DRTIO[linkno].async_messages_ready_write)(1);
+                destination: 0,
-        }
+                error,
-        async_ready
+                channel,
-    }
+                timestamp,
-
+            } => {
-    async fn process_async_packets(
+                remote_dma::playback_done(id, source, error, channel, timestamp).await;
-        aux_mutex: &Mutex<bool>,
+                None
-        linkno: u8,
+            }
-        routing_table: &drtio_routing::RoutingTable,
+            Packet::SubkernelFinished {
-        timer: GlobalTimer,
+                id,
-    ) {
+                destination: 0,
-        if link_has_async_ready(linkno).await {
+                with_exception,
-            loop {
+                exception_src,
-                let reply = aux_transact(aux_mutex, linkno, &Packet::RoutingRetrievePackets, timer).await;
+            } => {
-                if let Ok(packet) = reply {
+                subkernel::subkernel_finished(id, with_exception, exception_src).await;
-                    match packet {
+                None
-                        Packet::DmaPlaybackStatus {
+            }
-                            id,
+            Packet::SubkernelMessage {
-                            source,
+                id,
-                            destination: 0,
+                source,
-                            error,
+                destination: 0,
-                            channel,
+                status,
-                            timestamp,
+                length,
-                        } => {
+                data,
-                            remote_dma::playback_done(id, source, error, channel, timestamp).await;
+            } => {
-                        }
+                subkernel::message_handle_incoming(id, status, length as usize, &data).await;
-                        Packet::SubkernelFinished {
+                // acknowledge receiving part of the message
-                            id,
+                drtioaux_async::send(linkno, &Packet::SubkernelMessageAck { destination: source })
-                            destination: 0,
+                    .await
-                            with_exception,
+                    .unwrap();
-                            exception_src,
+                None
-                        } => {
+            }
-                            subkernel::subkernel_finished(id, with_exception, exception_src).await;
+            // routable packets
-                        }
+            Packet::DmaAddTraceRequest { destination, .. }
-                        Packet::SubkernelMessage {
+            | Packet::DmaAddTraceReply { destination, .. }
-                            id,
+            | Packet::DmaRemoveTraceRequest { destination, .. }
-                            source,
+            | Packet::DmaRemoveTraceReply { destination, .. }
-                            destination: 0,
+            | Packet::DmaPlaybackRequest { destination, .. }
-                            status,
+            | Packet::DmaPlaybackReply { destination, .. }
-                            length,
+            | Packet::SubkernelLoadRunRequest { destination, .. }
-                            data,
+            | Packet::SubkernelLoadRunReply { destination, .. }
-                        } => {
+            | Packet::SubkernelMessage { destination, .. }
-                            subkernel::message_handle_incoming(id, status, length as usize, &data).await;
+            | Packet::SubkernelMessageAck { destination, .. }
-                            // acknowledge receiving part of the message
+            | Packet::SubkernelException { destination, .. }
-                            let _lock = aux_mutex.async_lock().await;
+            | Packet::SubkernelExceptionRequest { destination, .. }
-                            drtioaux_async::send(linkno, &Packet::SubkernelMessageAck { destination: source })
+            | Packet::DmaPlaybackStatus { destination, .. }
-                                .await
+            | Packet::SubkernelFinished { destination, .. } => {
-                                .unwrap();
+                if destination == 0 {
-                            let mut countdown = timer.countdown();
+                    Some(packet)
-                            // give the satellites some time to process
-                            delay(&mut countdown, Milliseconds(10)).await;
-                        }
-                        // routable packets
-                        Packet::DmaAddTraceRequest { destination, .. }
-                        | Packet::DmaAddTraceReply { destination, .. }
-                        | Packet::DmaRemoveTraceRequest { destination, .. }
-                        | Packet::DmaRemoveTraceReply { destination, .. }
-                        | Packet::DmaPlaybackRequest { destination, .. }
-                        | Packet::DmaPlaybackReply { destination, .. }
-                        | Packet::SubkernelLoadRunRequest { destination, .. }
-                        | Packet::SubkernelLoadRunReply { destination, .. }
-                        | Packet::SubkernelMessage { destination, .. }
-                        | Packet::SubkernelMessageAck { destination, .. }
-                        | Packet::DmaPlaybackStatus { destination, .. }
-                        | Packet::SubkernelFinished { destination, .. } => {
-                            let dest_link = routing_table.0[destination as usize][0] - 1;
-                            if dest_link == linkno {
-                                warn!(
-                                    "[LINK#{}] Re-routed packet would return to the same link, dropping: {:?}",
-                                    linkno, packet
-                                );
-                            } else if destination == 0 {
-                                warn!("[LINK#{}] Received invalid routable packet: {:?}", linkno, packet)
-                            } else {
-                                drtioaux_async::send(dest_link, &packet).await.unwrap();
-                            }
-                        }
-
-                        Packet::RoutingNoPackets => break,
-
-                        other => warn!("[LINK#{}] Received an unroutable packet: {:?}", linkno, other),
-                    }
                 } else {
-                    warn!(
+                    let dest_link = routing_table.0[destination as usize][0] - 1;
-                        "[LINK#{}] Error handling async packets ({})",
+                    if dest_link == linkno {
-                        linkno,
+                        warn!(
-                        reply.unwrap_err()
+                            "[LINK#{}] Re-routed packet would return to the same link, dropping: {:?}",
-                    );
+                            linkno, packet
-                    return;
+                        );
+                    } else {
+                        drtioaux_async::send(dest_link, &packet).await.unwrap();
+                    }
+                    None
                 }
             }
+            other => Some(other),
         }
     }
 
-    async fn recv_aux_timeout(linkno: u8, timeout: u64, timer: GlobalTimer) -> Result<Packet, &'static str> {
+    async fn recv_aux_timeout(linkno: u8, timeout: u64, mut timer: GlobalTimer) -> Result<Packet, Error> {
-        if !link_rx_up(linkno).await {
+        if !link_rx_up(linkno, &mut timer).await {
-            return Err("link went down");
+            return Err(Error::LinkDown);
         }
         match drtioaux_async::recv_timeout(linkno, Some(timeout), timer).await {
             Ok(packet) => return Ok(packet),
-            Err(Error::TimedOut) => return Err("timed out"),
+            Err(DrtioError::TimedOut) => return Err(Error::Timeout),
-            Err(_) => return Err("aux packet error"),
+            Err(_) => return Err(Error::AuxError),
         }
     }
 
     pub async fn aux_transact(
         aux_mutex: &Mutex<bool>,
         linkno: u8,
+        routing_table: &RoutingTable,
         request: &Packet,
-        timer: GlobalTimer,
+        mut timer: GlobalTimer,
-    ) -> Result<Packet, &'static str> {
+    ) -> Result<Packet, Error> {
-        if !link_rx_up(linkno).await {
+        if !link_rx_up(linkno, &mut timer).await {
-            return Err("link went down");
+            return Err(Error::LinkDown);
         }
         let _lock = aux_mutex.async_lock().await;
         drtioaux_async::send(linkno, request).await.unwrap();
-        Ok(recv_aux_timeout(linkno, 200, timer).await?)
+        loop {
+            let packet = recv_aux_timeout(linkno, 200, timer).await?;
+            if let Some(packet) = process_async_packets(linkno, routing_table, packet).await {
+                return Ok(packet);
+            }
+        }
     }
 
     async fn drain_buffer(linkno: u8, draining_time: Milliseconds, timer: GlobalTimer) {
         let max_time = timer.get_time() + draining_time;
-        loop {
+        while timer.get_time() < max_time {
-            if timer.get_time() > max_time {
-                return;
-            } //could this be cut short?
             let _ = drtioaux_async::recv(linkno).await;
         }
     }
 
-    async fn ping_remote(aux_mutex: &Rc<Mutex<bool>>, linkno: u8, timer: GlobalTimer) -> u32 {
+    async fn ping_remote(
+        aux_mutex: &Rc<Mutex<bool>>,
+        linkno: u8,
+        routing_table: &RoutingTable,
+        mut timer: GlobalTimer,
+    ) -> u32 {
         let mut count = 0;
         loop {
-            if !link_rx_up(linkno).await {
+            if !link_rx_up(linkno, &mut timer).await {
                 return 0;
             }
             count += 1;
             if count > 100 {
                 return 0;
             }
-            let reply = aux_transact(aux_mutex, linkno, &Packet::EchoRequest, timer).await;
+            let reply = aux_transact(aux_mutex, linkno, routing_table, &Packet::EchoRequest, timer).await;
             match reply {
                 Ok(Packet::EchoReply) => {
                     // make sure receive buffer is drained
@@ -200,7 +236,7 @@ pub mod drtio {
         }
     }
 
-    async fn sync_tsc(aux_mutex: &Rc<Mutex<bool>>, linkno: u8, timer: GlobalTimer) -> Result<(), &'static str> {
+    async fn sync_tsc(aux_mutex: &Rc<Mutex<bool>>, linkno: u8, timer: GlobalTimer) -> Result<(), Error> {
         let _lock = aux_mutex.async_lock().await;
 
         unsafe {
@@ -211,22 +247,23 @@ pub mod drtio {
         // by the satellite, in response to a TSC set on the RT link.
         let reply = recv_aux_timeout(linkno, 10000, timer).await?;
         if reply == Packet::TSCAck {
-            return Ok(());
+            Ok(())
         } else {
-            return Err("unexpected reply");
+            Err(Error::UnexpectedReply)
         }
     }
 
     async fn load_routing_table(
         aux_mutex: &Rc<Mutex<bool>>,
         linkno: u8,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
         for i in 0..drtio_routing::DEST_COUNT {
             let reply = aux_transact(
                 aux_mutex,
                 linkno,
+                routing_table,
                 &Packet::RoutingSetPath {
                     destination: i as u8,
                     hops: routing_table.0[i],
@@ -235,7 +272,7 @@ pub mod drtio {
             )
             .await?;
             if reply != Packet::RoutingAck {
-                return Err("unexpected reply");
+                return Err(Error::UnexpectedReply);
             }
         }
         Ok(())
@@ -245,13 +282,21 @@ pub mod drtio {
         aux_mutex: &Rc<Mutex<bool>>,
         linkno: u8,
         rank: u8,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
-        let reply = aux_transact(aux_mutex, linkno, &Packet::RoutingSetRank { rank: rank }, timer).await?;
+        let reply = aux_transact(
-        if reply != Packet::RoutingAck {
+            aux_mutex,
-            return Err("unexpected reply");
+            linkno,
+            routing_table,
+            &Packet::RoutingSetRank { rank: rank },
+            timer,
+        )
+        .await?;
+        match reply {
+            Packet::RoutingAck => Ok(()),
+            _ => Err(Error::UnexpectedReply),
         }
-        Ok(())
     }
 
     async fn init_buffer_space(destination: u8, linkno: u8) {
@@ -270,10 +315,14 @@ pub mod drtio {
         }
     }
 
-    async fn process_unsolicited_aux(aux_mutex: &Rc<Mutex<bool>>, linkno: u8) {
+    async fn process_unsolicited_aux(aux_mutex: &Mutex<bool>, linkno: u8, routing_table: &RoutingTable) {
         let _lock = aux_mutex.async_lock().await;
         match drtioaux_async::recv(linkno).await {
-            Ok(Some(packet)) => warn!("[LINK#{}] unsolicited aux packet: {:?}", linkno, packet),
+            Ok(Some(packet)) => {
+                if let Some(packet) = process_async_packets(linkno, routing_table, packet).await {
+                    warn!("[LINK#{}] unsolicited aux packet: {:?}", linkno, packet);
+                }
+            }
             Ok(None) => (),
             Err(_) => warn!("[LINK#{}] aux packet error", linkno),
         }
@@ -301,7 +350,7 @@ pub mod drtio {
     }
 
     async fn destination_set_up(
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         destination: u8,
         up: bool,
@@ -324,7 +373,7 @@ pub mod drtio {
 
     async fn destination_survey(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         up_links: &[bool],
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         timer: GlobalTimer,
@@ -345,6 +394,7 @@ pub mod drtio {
                         let reply = aux_transact(
                             aux_mutex,
                             linkno,
+                            routing_table,
                             &Packet::DestinationStatusRequest {
                                 destination: destination,
                             },
@@ -400,6 +450,7 @@ pub mod drtio {
                         let reply = aux_transact(
                             aux_mutex,
                             linkno,
+                            routing_table,
                             &Packet::DestinationStatusRequest {
                                 destination: destination,
                             },
@@ -427,9 +478,9 @@ pub mod drtio {
 
     pub async fn link_task(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
-        timer: GlobalTimer,
+        mut timer: GlobalTimer,
     ) {
         let mut up_links = [false; csr::DRTIO.len()];
         loop {
@@ -437,19 +488,37 @@ pub mod drtio {
                 let linkno = linkno as u8;
                 if up_links[linkno as usize] {
                     /* link was previously up */
-                    if link_rx_up(linkno).await {
+                    if link_rx_up(linkno, &mut timer).await {
-                        process_async_packets(aux_mutex, linkno, routing_table, timer).await;
+                        process_unsolicited_aux(aux_mutex, linkno, routing_table).await;
-                        process_unsolicited_aux(aux_mutex, linkno).await;
                         process_local_errors(linkno).await;
                     } else {
                         info!("[LINK#{}] link is down", linkno);
                         up_links[linkno as usize] = false;
+
+                        #[cfg(has_drtio_eem)]
+                        if DRTIO_EEM_LINKNOS.contains(&(linkno as usize)) {
+                            unsafe {
+                                csr::eem_transceiver::rx_ready_write(0);
+                            }
+                            while !matches!(drtioaux_async::recv(linkno).await, Ok(None)) {}
+                        }
                     }
                 } else {
                     /* link was previously down */
-                    if link_rx_up(linkno).await {
+                    #[cfg(has_drtio_eem)]
+                    if DRTIO_EEM_LINKNOS.contains(&(linkno as usize)) {
+                        let eem_trx_no = linkno - DRTIO_EEM_LINKNOS.start as u8;
+                        if !unsafe { drtio_eem::align_wordslip(&mut timer, eem_trx_no) } {
+                            continue;
+                        }
+                        unsafe {
+                            csr::eem_transceiver::rx_ready_write(1);
+                        }
+                    }
+
+                    if link_rx_up(linkno, &mut timer).await {
                         info!("[LINK#{}] link RX became up, pinging", linkno);
-                        let ping_count = ping_remote(aux_mutex, linkno, timer).await;
+                        let ping_count = ping_remote(aux_mutex, linkno, routing_table, timer).await;
                         if ping_count > 0 {
                             info!("[LINK#{}] remote replied after {} packets", linkno, ping_count);
                             up_links[linkno as usize] = true;
@@ -459,7 +528,7 @@ pub mod drtio {
                             if let Err(e) = load_routing_table(aux_mutex, linkno, routing_table, timer).await {
                                 error!("[LINK#{}] failed to load routing table ({})", linkno, e);
                             }
-                            if let Err(e) = set_rank(aux_mutex, linkno, 1 as u8, timer).await {
+                            if let Err(e) = set_rank(aux_mutex, linkno, 1 as u8, routing_table, timer).await {
                                 error!("[LINK#{}] failed to set rank ({})", linkno, e);
                             }
                             info!("[LINK#{}] link initialization completed", linkno);
@@ -476,7 +545,7 @@ pub mod drtio {
     }
 
     #[allow(dead_code)]
-    pub fn reset(aux_mutex: Rc<Mutex<bool>>, mut timer: GlobalTimer) {
+    pub fn reset(aux_mutex: Rc<Mutex<bool>>, routing_table: &RoutingTable, mut timer: GlobalTimer) {
         for linkno in 0..csr::DRTIO.len() {
             unsafe {
                 (csr::DRTIO[linkno].reset_write)(1);
@@ -491,8 +560,14 @@ pub mod drtio {
 
         for linkno in 0..csr::DRTIO.len() {
             let linkno = linkno as u8;
-            if task::block_on(link_rx_up(linkno)) {
+            if task::block_on(link_rx_up(linkno, &mut timer)) {
-                let reply = task::block_on(aux_transact(&aux_mutex, linkno, &Packet::ResetRequest, timer));
+                let reply = task::block_on(aux_transact(
+                    &aux_mutex,
+                    linkno,
+                    routing_table,
+                    &Packet::ResetRequest,
+                    timer,
+                ));
                 match reply {
                     Ok(Packet::ResetAck) => (),
                     Ok(_) => error!("[LINK#{}] reset failed, received unexpected aux packet", linkno),
@@ -502,17 +577,18 @@ pub mod drtio {
         }
     }
 
-    async fn partition_data<PacketF, HandlerF>(
+    pub async fn partition_data<PacketF, HandlerF>(
         linkno: u8,
         aux_mutex: &Rc<Mutex<bool>>,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         data: &[u8],
         packet_f: PacketF,
         reply_handler_f: HandlerF,
-    ) -> Result<(), &'static str>
+    ) -> Result<(), Error>
     where
         PacketF: Fn(&[u8; MASTER_PAYLOAD_MAX_SIZE], PayloadStatus, usize) -> Packet,
-        HandlerF: Fn(&Packet) -> Result<(), &'static str>,
+        HandlerF: Fn(&Packet) -> Result<(), Error>,
     {
         let mut i = 0;
         while i < data.len() {
@@ -528,7 +604,7 @@ pub mod drtio {
             i += len;
             let status = PayloadStatus::from_status(first, last);
             let packet = packet_f(&slice, status, len);
-            let reply = aux_transact(aux_mutex, linkno, &packet, timer).await?;
+            let reply = aux_transact(aux_mutex, linkno, routing_table, &packet, timer).await?;
             reply_handler_f(&reply)?;
         }
         Ok(())
@@ -536,16 +612,17 @@ pub mod drtio {
 
     pub async fn ddma_upload_trace(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         id: u32,
         destination: u8,
         trace: &Vec<u8>,
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         partition_data(
             linkno,
             aux_mutex,
+            routing_table,
             timer,
             trace,
             |slice, status, len| Packet::DmaAddTraceRequest {
@@ -566,8 +643,8 @@ pub mod drtio {
                     destination: 0,
                     succeeded: false,
                     ..
-                } => Err("error adding trace on satellite"),
+                } => Err(Error::DmaAddTraceFail(destination)),
-                _ => Err("adding DMA trace failed, unexpected aux packet"),
+                _ => Err(Error::UnexpectedReply),
             },
         )
         .await
@@ -575,15 +652,16 @@ pub mod drtio {
 
     pub async fn ddma_send_erase(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         id: u32,
         destination: u8,
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let reply = aux_transact(
             aux_mutex,
             linkno,
+            routing_table,
             &Packet::DmaRemoveTraceRequest {
                 id: id,
                 source: 0,
@@ -591,33 +669,33 @@ pub mod drtio {
             },
             timer,
         )
-        .await;
+        .await?;
         match reply {
-            Ok(Packet::DmaRemoveTraceReply {
+            Packet::DmaRemoveTraceReply {
                 destination: 0,
                 succeeded: true,
-            }) => Ok(()),
+            } => Ok(()),
-            Ok(Packet::DmaRemoveTraceReply {
+            Packet::DmaRemoveTraceReply {
                 destination: 0,
                 succeeded: false,
-            }) => Err("satellite DMA erase error"),
+            } => Err(Error::DmaEraseFail(destination)),
-            Ok(_) => Err("adding trace failed, unexpected aux packet"),
+            _ => Err(Error::UnexpectedReply),
-            Err(_) => Err("erasing trace failed, aux error"),
         }
     }
 
     pub async fn ddma_send_playback(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         id: u32,
         destination: u8,
         timestamp: u64,
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let reply = aux_transact(
             aux_mutex,
             linkno,
+            routing_table,
             &Packet::DmaPlaybackRequest {
                 id: id,
                 source: 0,
@@ -626,45 +704,44 @@ pub mod drtio {
             },
             timer,
         )
-        .await;
+        .await?;
         match reply {
-            Ok(Packet::DmaPlaybackReply {
+            Packet::DmaPlaybackReply {
                 destination: 0,
                 succeeded: true,
-            }) => Ok(()),
+            } => Ok(()),
-            Ok(Packet::DmaPlaybackReply {
+            Packet::DmaPlaybackReply {
                 destination: 0,
                 succeeded: false,
-            }) => Err("error on DMA playback request"),
+            } => Err(Error::DmaPlaybackFail(destination)),
-            Ok(_) => Err("received unexpected aux packet during DMA playback"),
+            _ => Err(Error::UnexpectedReply),
-            Err(_) => Err("aux error on DMA playback"),
         }
     }
 
     async fn analyzer_get_data(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         destination: u8,
-    ) -> Result<RemoteBuffer, &'static str> {
+    ) -> Result<RemoteBuffer, Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let reply = aux_transact(
             aux_mutex,
             linkno,
+            routing_table,
             &Packet::AnalyzerHeaderRequest {
                 destination: destination,
             },
             timer,
         )
-        .await;
+        .await?;
         let (sent, total, overflow) = match reply {
-            Ok(Packet::AnalyzerHeader {
+            Packet::AnalyzerHeader {
                 sent_bytes,
                 total_byte_count,
                 overflow_occurred,
-            }) => (sent_bytes, total_byte_count, overflow_occurred),
+            } => (sent_bytes, total_byte_count, overflow_occurred),
-            Ok(_) => return Err("received unexpected aux packet during remote analyzer header request"),
+            _ => return Err(Error::UnexpectedReply),
-            Err(e) => return Err(e),
         };
 
         let mut remote_data: Vec<u8> = Vec::new();
@@ -674,19 +751,19 @@ pub mod drtio {
                 let reply = aux_transact(
                     aux_mutex,
                     linkno,
+                    routing_table,
                     &Packet::AnalyzerDataRequest {
                         destination: destination,
                     },
                     timer,
                 )
-                .await;
+                .await?;
                 match reply {
-                    Ok(Packet::AnalyzerData { last, length, data }) => {
+                    Packet::AnalyzerData { last, length, data } => {
                         last_packet = last;
                         remote_data.extend(&data[0..length as usize]);
                     }
-                    Ok(_) => return Err("received unexpected aux packet during remote analyzer data request"),
+                    _ => return Err(Error::UnexpectedReply),
-                    Err(e) => return Err(e),
                 }
             }
         }
@@ -701,10 +778,10 @@ pub mod drtio {
 
     pub async fn analyzer_query(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         timer: GlobalTimer,
-    ) -> Result<Vec<RemoteBuffer>, &'static str> {
+    ) -> Result<Vec<RemoteBuffer>, Error> {
         let mut remote_buffers: Vec<RemoteBuffer> = Vec::new();
         for i in 1..drtio_routing::DEST_COUNT {
             if destination_up(up_destinations, i as u8).await {
@@ -716,16 +793,17 @@ pub mod drtio {
 
     pub async fn subkernel_upload(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         id: u32,
         destination: u8,
         data: &Vec<u8>,
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         partition_data(
             linkno,
             aux_mutex,
+            routing_table,
             timer,
             data,
             |slice, status, len| Packet::SubkernelAddDataRequest {
@@ -737,8 +815,8 @@ pub mod drtio {
             },
             |reply| match reply {
                 Packet::SubkernelAddDataReply { succeeded: true } => Ok(()),
-                Packet::SubkernelAddDataReply { succeeded: false } => Err("error adding subkernel on satellite"),
+                Packet::SubkernelAddDataReply { succeeded: false } => Err(Error::SubkernelAddFail(destination)),
-                _ => Err("adding subkernel failed, unexpected aux packet"),
+                _ => Err(Error::UnexpectedReply),
             },
         )
         .await
@@ -746,21 +824,24 @@ pub mod drtio {
 
     pub async fn subkernel_load(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         id: u32,
         destination: u8,
         run: bool,
-    ) -> Result<(), &'static str> {
+        timestamp: u64,
+    ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let reply = aux_transact(
             aux_mutex,
             linkno,
+            routing_table,
             &Packet::SubkernelLoadRunRequest {
                 id: id,
                 source: 0,
                 destination: destination,
                 run: run,
+                timestamp,
             },
             timer,
         )
@@ -773,53 +854,61 @@ pub mod drtio {
             Packet::SubkernelLoadRunReply {
                 destination: 0,
                 succeeded: false,
-            } => return Err("error on subkernel run request"),
+            } => return Err(Error::SubkernelRunFail(destination)),
-            _ => return Err("received unexpected aux packet during subkernel run"),
+            _ => Err(Error::UnexpectedReply),
         }
     }
 
     pub async fn subkernel_retrieve_exception(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         destination: u8,
-    ) -> Result<Vec<u8>, &'static str> {
+    ) -> Result<Vec<u8>, Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let mut remote_data: Vec<u8> = Vec::new();
         loop {
             let reply = aux_transact(
                 aux_mutex,
                 linkno,
+                routing_table,
                 &Packet::SubkernelExceptionRequest {
+                    source: 0,
                     destination: destination,
                 },
                 timer,
             )
             .await?;
             match reply {
-                Packet::SubkernelException { last, length, data } => {
+                Packet::SubkernelException {
+                    destination: 0,
+                    last,
+                    length,
+                    data,
+                } => {
                     remote_data.extend(&data[0..length as usize]);
                     if last {
                         return Ok(remote_data);
                     }
                 }
-                _ => return Err("received unexpected aux packet during subkernel exception request"),
+                _ => return Err(Error::UnexpectedReply),
             }
         }
     }
 
     pub async fn subkernel_send_message(
         aux_mutex: &Rc<Mutex<bool>>,
-        routing_table: &drtio_routing::RoutingTable,
+        routing_table: &RoutingTable,
         timer: GlobalTimer,
         id: u32,
         destination: u8,
         message: &[u8],
-    ) -> Result<(), &'static str> {
+    ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         partition_data(
             linkno,
             aux_mutex,
+            routing_table,
             timer,
             message,
             |slice, status, len| Packet::SubkernelMessage {
@@ -832,7 +921,7 @@ pub mod drtio {
             },
             |reply| match reply {
                 Packet::SubkernelMessageAck { .. } => Ok(()),
-                _ => Err("sending message to subkernel failed, unexpected aux packet"),
+                _ => Err(Error::UnexpectedReply),
             },
         )
         .await
@@ -845,22 +934,46 @@ pub mod drtio {
 
     pub fn startup(
         _aux_mutex: &Rc<Mutex<bool>>,
-        _routing_table: &Rc<RefCell<drtio_routing::RoutingTable>>,
+        _routing_table: &Rc<RefCell<RoutingTable>>,
         _up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         _timer: GlobalTimer,
     ) {
     }
 
     #[allow(dead_code)]
-    pub fn reset(_aux_mutex: Rc<Mutex<bool>>, mut _timer: GlobalTimer) {}
+    pub fn reset(_aux_mutex: Rc<Mutex<bool>>, _routing_table: &RoutingTable, mut _timer: GlobalTimer) {}
+}
+
+fn toggle_sed_spread(val: u8) {
+    unsafe {
+        csr::rtio_core::sed_spread_enable_write(val);
+    }
+}
+
+fn setup_sed_spread(cfg: &Config) {
+    if let Ok(spread_enable) = cfg.read_str("sed_spread_enable") {
+        match spread_enable.as_ref() {
+            "1" => toggle_sed_spread(1),
+            "0" => toggle_sed_spread(0),
+            _ => {
+                warn!("sed_spread_enable value not supported (only 1, 0 allowed), disabling by default");
+                toggle_sed_spread(0)
+            }
+        };
+    } else {
+        info!("SED spreading disabled by default");
+        toggle_sed_spread(0);
+    }
 }
 
 pub fn startup(
     aux_mutex: &Rc<Mutex<bool>>,
-    routing_table: &Rc<RefCell<drtio_routing::RoutingTable>>,
+    routing_table: &Rc<RefCell<RoutingTable>>,
     up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
+    cfg: &Config,
     timer: GlobalTimer,
 ) {
+    setup_sed_spread(cfg);
     drtio::startup(aux_mutex, routing_table, up_destinations, timer);
     unsafe {
         csr::rtio_core::reset_phy_write(1);
@@ -868,9 +981,9 @@ pub fn startup(
 }
 
 #[allow(dead_code)]
-pub fn reset(aux_mutex: Rc<Mutex<bool>>, timer: GlobalTimer) {
+pub fn reset(aux_mutex: Rc<Mutex<bool>>, routing_table: &RoutingTable, timer: GlobalTimer) {
     unsafe {
         csr::rtio_core::reset_write(1);
     }
-    drtio::reset(aux_mutex, timer)
+    drtio::reset(aux_mutex, routing_table, timer)
 }

src/runtime/src/subkernel.rs

@@ -7,7 +7,7 @@ use libboard_zynq::{time::Milliseconds, timer::GlobalTimer};
 use libcortex_a9::mutex::Mutex;
 use log::{error, warn};
 
-use crate::rtio_mgt::drtio;
+use crate::rtio_mgt::{drtio, drtio::Error as DrtioError};
 
 #[derive(Debug, PartialEq, Clone, Copy)]
 pub enum FinishStatus {
@@ -31,11 +31,11 @@ pub enum Error {
     SubkernelNotFound,
     SubkernelException,
     CommLost,
-    DrtioError(&'static str),
+    DrtioError(DrtioError),
 }
 
-impl From<&'static str> for Error {
+impl From<DrtioError> for Error {
-    fn from(value: &'static str) -> Error {
+    fn from(value: DrtioError) -> Error {
         Error::DrtioError(value)
     }
 }
@@ -100,12 +100,22 @@ pub async fn load(
     timer: GlobalTimer,
     id: u32,
     run: bool,
+    timestamp: u64,
 ) -> Result<(), Error> {
     if let Some(subkernel) = SUBKERNELS.async_lock().await.get_mut(&id) {
         if subkernel.state != SubkernelState::Uploaded {
             return Err(Error::IncorrectState);
         }
-        drtio::subkernel_load(aux_mutex, routing_table, timer, id, subkernel.destination, run).await?;
+        drtio::subkernel_load(
+            aux_mutex,
+            routing_table,
+            timer,
+            id,
+            subkernel.destination,
+            run,
+            timestamp,
+        )
+        .await?;
         if run {
             subkernel.state = SubkernelState::Running;
         }

src/rustfmt.toml

@@ -54,7 +54,7 @@ use_field_init_shorthand = false
 force_explicit_abi = true
 condense_wildcard_suffixes = false
 color = "Auto"
-required_version = "1.4.32"
+required_version = "1.4.37"
 unstable_features = false
 disable_all_formatting = false
 skip_children = false
@@ -65,4 +65,4 @@ report_todo = "Never"
 report_fixme = "Never"
 ignore = []
 emit_mode = "Files"
-make_backup = false
+make_backup = false

src/satman/Cargo.toml.tpl

@@ -7,6 +7,7 @@ build = "build.rs"
 [features]
 target_zc706 = ["libboard_zynq/target_zc706", "libsupport_zynq/target_zc706", "libconfig/target_zc706", "libboard_artiq/target_zc706"]
 target_kasli_soc = ["libboard_zynq/target_kasli_soc", "libsupport_zynq/target_kasli_soc", "libconfig/target_kasli_soc", "libboard_artiq/target_kasli_soc"]
+calibrate_wrpll_skew = ["libboard_artiq/calibrate_wrpll_skew"]
 default = ["target_zc706", ]
 
 [build-dependencies]
@@ -14,7 +15,9 @@ build_zynq = { path = "../libbuild_zynq" }
 
 [dependencies]
 log = { version = "0.4", default-features = false }
-core_io = { version = "0.1", features = ["collections"] }
+byteorder = { version = "1.3", default-features = false }
+core_io = { git = "https://git.m-labs.hk/M-Labs/rs-core_io.git", rev = "e9d3edf027", features = ["collections"] }
+crc = { version = "1.7", default-features = false }
 cslice = "0.3"
 embedded-hal = "0.2"
 

src/satman/src/mgmt.rs

@@ -0,0 +1,149 @@
+use alloc::vec::Vec;
+
+use byteorder::{ByteOrder, NativeEndian};
+use crc::crc32;
+use io::{ProtoRead, ProtoWrite};
+use libboard_artiq::{drtioaux_proto::SAT_PAYLOAD_MAX_SIZE,
+                     logger::{BufferLogger, LogBufferRef}};
+use libconfig::Config;
+use log::{debug, error, info, warn, LevelFilter};
+
+use crate::routing::{SliceMeta, Sliceable};
+
+type Result<T> = core::result::Result<T, ()>;
+
+pub fn byte_to_level_filter(level_byte: u8) -> Result<LevelFilter> {
+    Ok(match level_byte {
+        0 => LevelFilter::Off,
+        1 => LevelFilter::Error,
+        2 => LevelFilter::Warn,
+        3 => LevelFilter::Info,
+        4 => LevelFilter::Debug,
+        5 => LevelFilter::Trace,
+        lv => {
+            error!("unknown log level: {}", lv);
+            return Err(());
+        }
+    })
+}
+
+fn get_logger_buffer() -> LogBufferRef<'static> {
+    let logger = unsafe { BufferLogger::get_logger().as_mut().unwrap() };
+    loop {
+        if let Some(buffer_ref) = logger.buffer() {
+            return buffer_ref;
+        }
+    }
+}
+
+pub fn clear_log() {
+    let mut buffer = get_logger_buffer();
+    buffer.clear();
+}
+
+pub struct Manager<'a> {
+    cfg: &'a mut Config,
+    last_log: Sliceable,
+    config_payload: Vec<u8>,
+    last_value: Sliceable,
+    image_payload: Vec<u8>,
+}
+
+impl<'a> Manager<'_> {
+    pub fn new(cfg: &mut Config) -> Manager {
+        Manager {
+            cfg: cfg,
+            last_log: Sliceable::new(0, Vec::new()),
+            config_payload: Vec::new(),
+            last_value: Sliceable::new(0, Vec::new()),
+            image_payload: Vec::new(),
+        }
+    }
+
+    pub fn log_get_slice(&mut self, data_slice: &mut [u8; SAT_PAYLOAD_MAX_SIZE], consume: bool) -> SliceMeta {
+        // Populate buffer if depleted
+        if self.last_log.at_end() {
+            let mut buffer = get_logger_buffer();
+            self.last_log.extend(buffer.extract().as_bytes());
+            if consume {
+                buffer.clear();
+            }
+        }
+
+        self.last_log.get_slice_satellite(data_slice)
+    }
+
+    pub fn fetch_config_value(&mut self, key: &str) -> Result<()> {
+        self.cfg
+            .read(&key)
+            .map(|value| {
+                debug!("got value");
+                self.last_value = Sliceable::new(0, value)
+            })
+            .map_err(|_| warn!("read error: no such key"))
+    }
+
+    pub fn get_config_value_slice(&mut self, data_slice: &mut [u8; SAT_PAYLOAD_MAX_SIZE]) -> SliceMeta {
+        self.last_value.get_slice_satellite(data_slice)
+    }
+
+    pub fn add_config_data(&mut self, data: &[u8], data_len: usize) {
+        self.config_payload.write_all(&data[..data_len]).unwrap();
+    }
+
+    pub fn clear_config_data(&mut self) {
+        self.config_payload.clear();
+    }
+
+    pub fn write_config(&mut self) -> Result<()> {
+        let mut payload = &self.config_payload[..];
+        let key = payload.read_string().map_err(|_err| error!("error on reading key"))?;
+        debug!("write key: {}", key);
+        let value = payload.read_bytes().unwrap();
+
+        self.cfg
+            .write(&key, value)
+            .map(|()| debug!("write success"))
+            .map_err(|err| error!("failed to write: {:?}", err))
+    }
+
+    pub fn remove_config(&mut self, key: &str) -> Result<()> {
+        debug!("erase key: {}", key);
+        self.cfg
+            .remove(&key)
+            .map(|()| debug!("erase success"))
+            .map_err(|err| warn!("failed to erase: {:?}", err))
+    }
+
+    pub fn allocate_image_buffer(&mut self, image_size: usize) {
+        self.image_payload = Vec::with_capacity(image_size);
+    }
+
+    pub fn add_image_data(&mut self, data: &[u8], data_len: usize) {
+        self.image_payload.extend(&data[..data_len]);
+    }
+
+    pub fn write_image(&self) {
+        let mut image = self.image_payload.clone();
+        let image_ref = &image[..];
+        let bin_len = image.len() - 4;
+
+        let (image_ref, expected_crc) = {
+            let (image_ref, crc_slice) = image_ref.split_at(bin_len);
+            (image_ref, NativeEndian::read_u32(crc_slice))
+        };
+
+        let actual_crc = crc32::checksum_ieee(image_ref);
+
+        if actual_crc == expected_crc {
+            info!("CRC passed. Writing boot image to SD card...");
+            image.truncate(bin_len);
+            self.cfg.write("boot", image).expect("failed to write boot image");
+        } else {
+            panic!(
+                "CRC failed, images have not been written to flash.\n(actual {:08x}, expected {:08x})",
+                actual_crc, expected_crc
+            );
+        }
+    }
+}

src/satman/src/repeater.rs

@@ -87,6 +87,10 @@ impl Repeater {
                 if rep_link_rx_up(self.repno) {
                     if let Ok(Some(drtioaux::Packet::EchoReply)) = drtioaux::recv(self.auxno) {
                         info!("[REP#{}] remote replied after {} packets", self.repno, ping_count);
+                        let max_time = timer.get_time() + Milliseconds(200);
+                        while timer.get_time() < max_time {
+                            let _ = drtioaux::recv(self.auxno);
+                        }
                         self.state = RepeaterState::Up;
                         if let Err(e) = self.sync_tsc(timer) {
                             error!("[REP#{}] failed to sync TSC ({:?})", self.repno, e);
@@ -119,16 +123,11 @@ impl Repeater {
                 }
             }
             RepeaterState::Up => {
-                self.process_unsolicited_aux();
+                self.process_unsolicited_aux(routing_table, rank, destination, router);
                 if !rep_link_rx_up(self.repno) {
                     info!("[REP#{}] link is down", self.repno);
                     self.state = RepeaterState::Down;
                 }
-                if self.async_messages_ready() {
-                    if let Err(e) = self.handle_async(routing_table, rank, destination, router, timer) {
-                        warn!("[REP#{}] Error handling async messages ({:?})", self.repno, e);
-                    }
-                }
             }
             RepeaterState::Failed => {
                 if !rep_link_rx_up(self.repno) {
@@ -139,9 +138,15 @@ impl Repeater {
         }
     }
 
-    fn process_unsolicited_aux(&self) {
+    fn process_unsolicited_aux(
+        &self,
+        routing_table: &drtio_routing::RoutingTable,
+        rank: u8,
+        destination: u8,
+        router: &mut Router,
+    ) {
         match drtioaux::recv(self.auxno) {
-            Ok(Some(packet)) => warn!("[REP#{}] unsolicited aux packet: {:?}", self.repno, packet),
+            Ok(Some(packet)) => router.route(packet, routing_table, rank, destination),
             Ok(None) => (),
             Err(_) => warn!("[REP#{}] aux packet error", self.repno),
         }
@@ -186,34 +191,6 @@ impl Repeater {
         }
     }
 
-    fn async_messages_ready(&self) -> bool {
-        let async_rdy;
-        unsafe {
-            async_rdy = (csr::DRTIOREP[self.repno as usize].async_messages_ready_read)();
-            (csr::DRTIOREP[self.repno as usize].async_messages_ready_write)(0);
-        }
-        async_rdy == 1
-    }
-
-    fn handle_async(
-        &self,
-        routing_table: &drtio_routing::RoutingTable,
-        rank: u8,
-        self_destination: u8,
-        router: &mut Router,
-        timer: &mut GlobalTimer,
-    ) -> Result<(), drtioaux::Error> {
-        loop {
-            drtioaux::send(self.auxno, &drtioaux::Packet::RoutingRetrievePackets).unwrap();
-            let reply = self.recv_aux_timeout(200, timer)?;
-            match reply {
-                drtioaux::Packet::RoutingNoPackets => break,
-                packet => router.route(packet, routing_table, rank, self_destination),
-            }
-        }
-        Ok(())
-    }
-
     fn recv_aux_timeout(&self, timeout: u32, timer: &mut GlobalTimer) -> Result<drtioaux::Packet, drtioaux::Error> {
         let max_time = timer.get_time() + Milliseconds(timeout.into());
         loop {
@@ -231,10 +208,37 @@ impl Repeater {
         }
     }
 
-    pub fn aux_forward(&self, request: &drtioaux::Packet, timer: &mut GlobalTimer) -> Result<(), drtioaux::Error> {
+    pub fn aux_forward(
+        &self,
+        request: &drtioaux::Packet,
+        router: &mut Router,
+        routing_table: &drtio_routing::RoutingTable,
+        rank: u8,
+        self_destination: u8,
+        timer: &mut GlobalTimer,
+    ) -> Result<(), drtioaux::Error> {
         self.aux_send(request)?;
-        let reply = self.recv_aux_timeout(200, timer)?;
+        loop {
-        drtioaux::send(0, &reply).unwrap();
+            let reply = self.recv_aux_timeout(200, timer)?;
+            match reply {
+                // async/locally requested packets to be consumed or routed
+                // these may come while a packet would be forwarded
+                drtioaux::Packet::DmaPlaybackStatus { .. }
+                | drtioaux::Packet::SubkernelFinished { .. }
+                | drtioaux::Packet::SubkernelMessage { .. }
+                | drtioaux::Packet::SubkernelMessageAck { .. }
+                | drtioaux::Packet::SubkernelLoadRunReply { .. }
+                | drtioaux::Packet::SubkernelException { .. }
+                | drtioaux::Packet::DmaAddTraceReply { .. }
+                | drtioaux::Packet::DmaPlaybackReply { .. } => {
+                    router.route(reply, routing_table, rank, self_destination);
+                }
+                _ => {
+                    drtioaux::send(0, &reply).unwrap();
+                    break;
+                }
+            }
+        }
         Ok(())
     }
 

src/satman/src/routing.rs

@@ -57,8 +57,8 @@ impl Sliceable {
         self.data.extend(data);
     }
 
-    get_slice_fn!(get_slice_sat, SAT_PAYLOAD_MAX_SIZE);
     get_slice_fn!(get_slice_master, MASTER_PAYLOAD_MAX_SIZE);
+    get_slice_fn!(get_slice_satellite, SAT_PAYLOAD_MAX_SIZE);
 }
 
 // Packets from downstream (further satellites) are received and routed appropriately.
@@ -75,7 +75,6 @@ pub struct Router {
     local_queue: VecDeque<drtioaux::Packet>,
     #[cfg(has_drtio_routing)]
     downstream_queue: VecDeque<(usize, drtioaux::Packet)>,
-    upstream_notified: bool,
 }
 
 impl Router {
@@ -85,7 +84,6 @@ impl Router {
             local_queue: VecDeque::new(),
             #[cfg(has_drtio_routing)]
             downstream_queue: VecDeque::new(),
-            upstream_notified: false,
         }
     }
 
@@ -161,22 +159,8 @@ impl Router {
         }
     }
 
-    pub fn any_upstream_waiting(&mut self) -> bool {
-        let empty = self.upstream_queue.is_empty();
-        if !empty && !self.upstream_notified {
-            self.upstream_notified = true; // so upstream will not get spammed with notifications
-            true
-        } else {
-            false
-        }
-    }
-
     pub fn get_upstream_packet(&mut self) -> Option<drtioaux::Packet> {
-        let packet = self.upstream_queue.pop_front();
+        self.upstream_queue.pop_front()
-        if packet.is_none() {
-            self.upstream_notified = false;
-        }
-        packet
     }
 
     #[cfg(has_drtio_routing)]

src/satman/src/subkernel.rs

@@ -2,16 +2,16 @@ use alloc::{collections::BTreeMap,
             format,
             string::{String, ToString},
             vec::Vec};
-use core::{option::NoneError, slice, str};
+use core::{slice, str};
 
 use core_io::{Error as IoError, Write};
 use cslice::AsCSlice;
 use dma::{Error as DmaError, Manager as DmaManager};
 use io::{Cursor, ProtoWrite};
-use ksupport::{eh_artiq, kernel, rpc};
+use ksupport::{eh_artiq, kernel, rpc, rtio};
 use libboard_artiq::{drtio_routing::RoutingTable,
                      drtioaux,
-                     drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE, SAT_PAYLOAD_MAX_SIZE},
+                     drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE},
                      pl::csr};
 use libboard_zynq::{time::Milliseconds, timer::GlobalTimer};
 use libcortex_a9::sync_channel::Receiver;
@@ -47,6 +47,9 @@ enum KernelState {
     DmaAwait {
         max_time: Milliseconds,
     },
+    SubkernelRetrievingException {
+        destination: u8,
+    },
 }
 
 #[derive(Debug)]
@@ -62,12 +65,6 @@ pub enum Error {
     DmaError(DmaError),
 }
 
-impl From<NoneError> for Error {
-    fn from(_: NoneError) -> Error {
-        Error::KernelNotFound
-    }
-}
-
 impl From<IoError> for Error {
     fn from(_value: IoError) -> Error {
         Error::SubkernelIoError
@@ -123,10 +120,11 @@ struct MessageManager {
 struct Session {
     id: u32,
     kernel_state: KernelState,
-    last_exception: Option<Sliceable>,
+    last_exception: Option<Sliceable>,   // exceptions raised locally
+    external_exception: Option<Vec<u8>>, // exceptions from sub-subkernels
     messages: MessageManager,
     source: u8, // which destination requested running the kernel
-    subkernels_finished: Vec<u32>,
+    subkernels_finished: Vec<(u32, Option<u8>)>,
 }
 
 impl Session {
@@ -135,6 +133,7 @@ impl Session {
             id: id,
             kernel_state: KernelState::Absent,
             last_exception: None,
+            external_exception: None,
             messages: MessageManager::new(),
             source: 0,
             subkernels_finished: Vec::new(),
@@ -311,7 +310,7 @@ impl<'a> Manager<'_> {
                             complete: false,
                         },
                     );
-                    self.kernels.get_mut(&id)?
+                    self.kernels.get_mut(&id).ok_or_else(|| Error::KernelNotFound)?
                 } else {
                     kernel
                 }
@@ -324,7 +323,7 @@ impl<'a> Manager<'_> {
                         complete: false,
                     },
                 );
-                self.kernels.get_mut(&id)?
+                self.kernels.get_mut(&id).ok_or_else(|| Error::KernelNotFound)?
             }
         };
         kernel.library.extend(&data[0..data_len]);
@@ -344,7 +343,7 @@ impl<'a> Manager<'_> {
         }
     }
 
-    pub fn run(&mut self, source: u8, id: u32) -> Result<(), Error> {
+    pub fn run(&mut self, source: u8, id: u32, timestamp: u64) -> Result<(), Error> {
         if self.session.kernel_state != KernelState::Loaded || self.session.id != id {
             self.load(id)?;
         }
@@ -354,6 +353,7 @@ impl<'a> Manager<'_> {
             csr::cri_con::selected_write(2);
         }
 
+        rtio::at_mu(timestamp as i64);
         self.control.tx.send(kernel::Message::StartRequest);
         Ok(())
     }
@@ -390,15 +390,19 @@ impl<'a> Manager<'_> {
         if self.session.id == id && self.session.kernel_state == KernelState::Loaded {
             return Ok(());
         }
-        if !self.kernels.get(&id)?.complete {
+        if !self.kernels.get(&id).ok_or_else(|| Error::KernelNotFound)?.complete {
             return Err(Error::KernelNotFound);
         }
         self.session = Session::new(id);
         self.control.restart();
 
-        self.control
+        self.control.tx.send(kernel::Message::LoadRequest(
-            .tx
+            self.kernels
-            .send(kernel::Message::LoadRequest(self.kernels.get(&id)?.library.clone()));
+                .get(&id)
+                .ok_or_else(|| Error::KernelNotFound)?
+                .library
+                .clone(),
+        ));
         let reply = self.control.rx.recv();
         match reply {
             kernel::Message::LoadCompleted => Ok(()),
@@ -410,9 +414,9 @@ impl<'a> Manager<'_> {
         }
     }
 
-    pub fn exception_get_slice(&mut self, data_slice: &mut [u8; SAT_PAYLOAD_MAX_SIZE]) -> SliceMeta {
+    pub fn exception_get_slice(&mut self, data_slice: &mut [u8; MASTER_PAYLOAD_MAX_SIZE]) -> SliceMeta {
         match self.session.last_exception.as_mut() {
-            Some(exception) => exception.get_slice_sat(data_slice),
+            Some(exception) => exception.get_slice_master(data_slice),
             None => SliceMeta {
                 destination: 0,
                 len: 0,
@@ -540,7 +544,7 @@ impl<'a> Manager<'_> {
             return;
         }
 
-        match self.process_external_messages(timer) {
+        match self.process_external_messages(router, routing_table, rank, destination, timer) {
             Ok(()) => (),
             Err(Error::AwaitingMessage) => return, // kernel still waiting, do not process kernel messages
             Err(Error::KernelException(exception)) => {
@@ -596,6 +600,41 @@ impl<'a> Manager<'_> {
         }
     }
 
+    fn check_finished_kernels(
+        &mut self,
+        id: u32,
+        router: &mut Router,
+        routing_table: &RoutingTable,
+        rank: u8,
+        self_destination: u8,
+    ) {
+        for (i, (status, exception_source)) in self.session.subkernels_finished.iter().enumerate() {
+            if *status == id {
+                if exception_source.is_none() {
+                    self.control.tx.send(kernel::Message::SubkernelAwaitFinishReply);
+                    self.session.kernel_state = KernelState::Running;
+                    self.session.subkernels_finished.swap_remove(i);
+                } else {
+                    let destination = exception_source.unwrap();
+                    self.session.external_exception = Some(Vec::new());
+                    self.session.kernel_state = KernelState::SubkernelRetrievingException {
+                        destination: destination,
+                    };
+                    router.route(
+                        drtioaux::Packet::SubkernelExceptionRequest {
+                            source: self_destination,
+                            destination: destination,
+                        },
+                        &routing_table,
+                        rank,
+                        self_destination,
+                    );
+                }
+                break;
+            }
+        }
+    }
+
     pub fn subkernel_load_run_reply(&mut self, succeeded: bool) {
         if self.session.kernel_state == KernelState::SubkernelAwaitLoad {
             self.control
@@ -608,16 +647,46 @@ impl<'a> Manager<'_> {
     }
 
     pub fn remote_subkernel_finished(&mut self, id: u32, with_exception: bool, exception_source: u8) {
-        if with_exception {
+        let exception_src = if with_exception { Some(exception_source) } else { None };
-            self.kernel_stop();
+        self.session.subkernels_finished.push((id, exception_src));
-            self.last_finished = Some(SubkernelFinished {
+    }
-                source: self.session.source,
+
-                id: self.session.id,
+    pub fn received_exception(
-                with_exception: true,
+        &mut self,
-                exception_source: exception_source,
+        exception_data: &[u8],
-            })
+        last: bool,
+        router: &mut Router,
+        routing_table: &RoutingTable,
+        rank: u8,
+        self_destination: u8,
+    ) {
+        if let KernelState::SubkernelRetrievingException { destination } = self.session.kernel_state {
+            self.session
+                .external_exception
+                .as_mut()
+                .unwrap()
+                .extend_from_slice(exception_data);
+            if last {
+                self.control
+                    .tx
+                    .send(kernel::Message::SubkernelError(kernel::SubkernelStatus::Exception(
+                        self.session.external_exception.take().unwrap(),
+                    )));
+                self.session.kernel_state = KernelState::Running;
+            } else {
+                /* fetch another slice */
+                router.route(
+                    drtioaux::Packet::SubkernelExceptionRequest {
+                        source: self_destination,
+                        destination: destination,
+                    },
+                    routing_table,
+                    rank,
+                    self_destination,
+                );
+            }
         } else {
-            self.session.subkernels_finished.push(id);
+            warn!("Received unsolicited exception data");
         }
     }
 
@@ -742,6 +811,7 @@ impl<'a> Manager<'_> {
                 id,
                 destination: sk_destination,
                 run,
+                timestamp,
             } => {
                 self.session.kernel_state = KernelState::SubkernelAwaitLoad;
                 router.route(
@@ -750,6 +820,7 @@ impl<'a> Manager<'_> {
                         destination: sk_destination,
                         id: id,
                         run: run,
+                        timestamp,
                     },
                     routing_table,
                     rank,
@@ -780,28 +851,35 @@ impl<'a> Manager<'_> {
         Ok(false)
     }
 
-    fn process_external_messages(&mut self, timer: &GlobalTimer) -> Result<(), Error> {
+    fn process_external_messages(
+        &mut self,
+        router: &mut Router,
+        routing_table: &RoutingTable,
+        rank: u8,
+        self_destination: u8,
+        timer: &GlobalTimer,
+    ) -> Result<(), Error> {
         match &self.session.kernel_state {
             KernelState::MsgAwait { max_time, id, tags } => {
                 if let Some(max_time) = *max_time {
                     if timer.get_time() > max_time {
-                        self.control.tx.send(kernel::Message::SubkernelMsgRecvReply {
+                        self.control
-                            status: kernel::SubkernelStatus::Timeout,
+                            .tx
-                            count: 0,
+                            .send(kernel::Message::SubkernelError(kernel::SubkernelStatus::Timeout));
-                        });
                         self.session.kernel_state = KernelState::Running;
                         return Ok(());
                     }
                 }
                 if let Some(message) = self.session.messages.get_incoming(*id) {
-                    self.control.tx.send(kernel::Message::SubkernelMsgRecvReply {
+                    self.control
-                        status: kernel::SubkernelStatus::NoError,
+                        .tx
-                        count: message.count,
+                        .send(kernel::Message::SubkernelMsgRecvReply { count: message.count });
-                    });
                     let tags = tags.clone();
                     self.session.kernel_state = KernelState::Running;
                     self.pass_message_to_kernel(&message, tags, timer)
                 } else {
+                    let id = *id;
+                    self.check_finished_kernels(id, router, routing_table, rank, self_destination);
                     Err(Error::AwaitingMessage)
                 }
             }
@@ -817,27 +895,18 @@ impl<'a> Manager<'_> {
             KernelState::SubkernelAwaitFinish { max_time, id } => {
                 if let Some(max_time) = *max_time {
                     if timer.get_time() > max_time {
-                        self.control.tx.send(kernel::Message::SubkernelAwaitFinishReply {
+                        self.control
-                            status: kernel::SubkernelStatus::Timeout,
+                            .tx
-                        });
+                            .send(kernel::Message::SubkernelError(kernel::SubkernelStatus::Timeout));
                         self.session.kernel_state = KernelState::Running;
                         return Ok(());
                     }
                 }
-                let mut i = 0;
+                let id = *id;
-                for status in &self.session.subkernels_finished {
+                self.check_finished_kernels(id, router, routing_table, rank, self_destination);
-                    if *status == *id {
-                        self.control.tx.send(kernel::Message::SubkernelAwaitFinishReply {
-                            status: kernel::SubkernelStatus::NoError,
-                        });
-                        self.session.kernel_state = KernelState::Running;
-                        self.session.subkernels_finished.swap_remove(i);
-                        break;
-                    }
-                    i += 1;
-                }
                 Ok(())
             }
+            KernelState::SubkernelRetrievingException { .. } => Err(Error::AwaitingMessage),
             KernelState::DmaAwait { max_time } | KernelState::DmaPendingAwait { max_time, .. } => {
                 if timer.get_time() > *max_time {
                     self.control.tx.send(kernel::Message::DmaAwaitRemoteReply {