drtio: add InjectionRequest to expects_response

Signed-off-by: Jonathan Coates <jonathan.coates@oxionics.com>

.gitignore

@@ -3,3 +3,11 @@ examples/*.elf
 __pycache__
 
 build
+
+src/libboard_artiq/Cargo.toml
+src/libc/Cargo.toml
+src/libdyld/Cargo.toml
+src/libio/Cargo.toml
+src/libksupport/Cargo.toml
+src/runtime/Cargo.toml
+src/satman/Cargo.toml

README.md

@@ -4,68 +4,107 @@ ARTIQ on Zynq
 How to use
 ----------
 
-1. Install ARTIQ-6 or newer.
-2. Select the latest successful build on Hydra: https://nixbld.m-labs.hk/jobset/artiq/zynq
-3. Search for the job named ``<board>-<variant>-sd`` (for example: ``zc706-nist_clock-sd`` or ``zc706-nist_qc2-sd``).
-4. Download the ``boot.bin`` "binary distribution" and place it at the root of a FAT-formatted SD card.
-5. Optionally, create a ``config.txt`` configuration file at the root of the SD card containing ``key=value`` pairs on each line. Use the ``ip``, ``ip6`` and ``mac`` keys to respectively set the IPv4, IPv6 and MAC address of the board. Configuring an IPv6 address is entirely optional. If these keys are not found, the firmware will use default values that may or may not be compatible with your network.
-6. Insert the SD card into the board and set up the board to boot from the SD card. For the ZC706, this is achieved by placing the large DIP switch SW11 in the 00110 position.
-7. Power up the board. After the firmware starts successfully, it should respond to ping at its IP addresses, and boot messages can be observed from its UART at 115200bps.
-8. Create and use an ARTIQ device database as usual, but set ``"target": "cortexa9"`` in the arguments of the core device.
+1. [Install ARTIQ](https://m-labs.hk/artiq/manual/installing.html). Get the corresponding version to the ``artiq-zynq`` version you are targeting.
+2. To obtain firmware binaries, use AFWS or build your own; see [the ARTIQ manual](https://m-labs.hk/artiq/manual/building_developing.html) for detailed instructions or skip to "Development" below. ZC706 variants only can also be downloaded from latest successful build on [Hydra](https://nixbld.m-labs.hk/).
+3. Place ``boot.bin`` file at the root ``/`` of a FAT-formatted SD card.
+4. Optionally, create a ``config.txt`` configuration file containing ``key=value`` pairs on each line and place it at the root of the SD card. See below for valid keys. The ``ip``, ``ip6`` and ``mac`` keys can be used to set networking information. If these keys are not found, the firmware will use default values which may or may not be compatible with your network.
+5. Insert the SD card into the board and set the board to boot from the SD card. For ZC706, this is achieved by placing the large DIP switch SW11 into the 00110 position. On Kasli-SoC, place the BOOT MODE switches to SD.
+6. Power up the board. After successful boot the firmware should respond to ping at its IP addresses. Boot output can be observed from UART at 115200bps 8-N-1.
+7. Create and use an ARTIQ device database as usual.
 
 Configuration
 -------------
 
-Configuring the device is done using the ``config.txt`` text file at the root of the SD card, plus the contents of the ``config`` folder. When searching for a configuration key, the firmware first looks for a file named ``/config/[key].bin`` and, if it exists, returns the contents of that file. If not, it looks into ``/config.txt``, which contains a list of ``key=value`` pairs, one per line. The ``config`` folder allows configuration values that consist in binary data, such as the startup kernel.
+Configuring the device is done using the ``config.txt`` text file at the root of the SD card plus optionally a ``config`` folder. When searching for a configuration key, the firmware first looks for a file named ``/config/[key].bin`` and, if it exists, returns the contents of that file. If not, it looks into ``/config.txt``, which should contain a list of ``key=value`` pairs, one per line. ``config.txt`` should be used for most keys but the ``config`` folder allows for setting configuration values which consist of binary data, such as the startup kernel.
 
-The following configuration keys are available:
+The following configuration keys are available among others:
 
 - ``mac``: Ethernet MAC address.
 - ``ip``: IPv4 address.
 - ``ip6``: IPv6 address.
-- ``startup``: startup kernel in ELF format (as produced by ``artiq_compile``).
-- ``rtioclk``: source of RTIO clock; valid values are ``external`` and ``internal``.
+- ``idle_kernel``: idle kernel in ELF format (as produced by ``artiq_compile``).
+- ``startup_kernel``: startup kernel in ELF format (as produced by ``artiq_compile``).
+- ``rtio_clock``: source of RTIO clock; valid values are ``ext0_bypass`` and ``int_125``.
+
+See [ARTIQ manual](https://m-labs.hk/artiq/manual-beta/core_device.html#configuration-storage) for full list. Configurations can be read/written/removed with ``artiq_coremgmt``. Config erase is not implemented, as it isn't particularly useful.
+
+For convenience, the ``boot`` key can be used with ``artiq_coremgmt`` and a ``boot.bin`` file to replace firmware/gateware in a running system. This key is read-only. When loading ``boot.bin`` onto the SD card directly, place it at the root and not in the ``config`` folder.
 
 Development instructions
 ------------------------
 
-Configure Nix channels:
+ARTIQ on Zynq is packaged using [Nix](https://nixos.org) Flakes. Install Nix 2.8+ and enable flakes by adding ``experimental-features = nix-command flakes`` to ``nix.conf`` (e.g. ``~/.config/nix/nix.conf``).
+
+**Pure build with Nix:**
 
 ```shell
-nix-channel --add https://nixbld.m-labs.hk/channel/custom/artiq/fast-beta/artiq-fast
-nix-channel --update
+nix build .#zc706-nist_clock-jtag  # or zc706-nist_qc2-jtag or zc706-nist_clock-sd or etc
 ```
 
-Note: if you are using Nix channels the first time, you need to be aware of this bug: https://github.com/NixOS/nix/issues/3831
+Run ``nix flake show`` to see all valid build targets. Targets suffixed with ``-jtag`` produce separate firmware and gateware files, intended for use in booting via JTAG server/Ethernet, e.g. ``./remote_run.sh -i`` with a remote JTAG server. Targets suffixed with ``-sd`` will produce ``boot.bin`` file suitable for SD card boot. ``-firmware`` and ``-gateware`` respectively build firmware and gateware only.
 
-Pure build with Nix and execution on a remote JTAG server:
+The Kasli-SoC target requires a system description file as input. See ARTIQ manual for exact instructions or use incremental build.
+
+**Impure incremental build:**
+
+For boards with fixed variants, i.e. ZC706, etc. :
 
 ```shell
-nix-build -A zc706-simple-jtag  # or zc706-nist_qc2-jtag or zc706-nist_clock-jtag
-./remote_run.sh
-```
-
-Impure incremental build and execution on a remote JTAG server:
-
-```shell
-nix-shell
+nix develop
 cd src
-gateware/zc706.py -g ../build/gateware  # build gateware
-make                                    # build firmware
-cd ..
-./remote_run.sh -i
+gateware/<board>.py -g ../build/gateware -V <variant> # gateware
+make GWARGS="-V <variant>" <runtime/satman>    # firmware
+```
+
+For boards with system descriptions, i.e. Kasli-SoC, etc. :
+
+```shell
+nix develop
+cd src
+gateware/<board>.py -g ../build/gateware <description.json> # gateware
+make TARGET=<board> GWARGS="path/to/description.json" <runtime/satman> # firmware
+```
+
+``szl.elf`` can be obtained with:
+
+```shell
+nix build git+https://git.m-labs.hk/m-labs/zynq-rs#<board>-szl
+```
+
+To generate ``boot.bin`` use ``mkbootimage``, e.g.:
+
+```shell
+echo "the_ROM_image:
+    {
+        [bootloader]result/szl.elf
+        gateware/top.bit
+        firmware/armv7-none-eabihf/release/<runtime/satman>
+    }
+    EOF" >> boot.bif
+mkbootimage boot.bif boot.bin
 ```
 
 Notes:
 
-- This is known to work with Nixpkgs 20.03 and the ``nixbld.m-labs.hk`` binary substituter can also be used here (see the ARTIQ manual for the public key and instructions).
 - The impure build process is also compatible with non-Nix systems.
-- If the board is connected to the local machine, use the ``local_run.sh`` script.
+- Firmware type must be either ``runtime`` for DRTIO-less or DRTIO master variants, or ``satman`` for DRTIO satellite.
+- If the board is connected to the local machine by JTAG, use the ``local_run.sh`` script.
+- A known Xilinx hardware bug prevents repeatedly loading the bootloader over JTAG without a POR reset. If booting over JTAG, install a jumper on ``PS_POR_B`` and use the POR reset script [here](https://git.m-labs.hk/M-Labs/zynq-rs/src/branch/master/kasli_soc_por.py).
+
+Pre-Commit Hooks
+----------------
+
+You are strongly recommended to use the provided pre-commit hooks to automatically reformat files and check for non-optimal Rust/C/C++ practices. Run `pre-commit install` to install the hook and `pre-commit` will automatically run `cargo fmt`, `cargo clippy`, and `clang-format` for you.
+
+Several things to note:
+
+- If `cargo fmt`, `cargo clippy`, or `clang-format` returns an error, the pre-commit hook will fail. You should fix all errors before trying to commit again.
+- If `cargo fmt` or `clang-format` reformats some files, the pre-commit hook will also fail. You should review the changes and, if satisfied, try to commit again.
 
 License
 -------
 
-Copyright (C) 2019-2020 M-Labs Limited.
+Copyright (C) 2019-2024 M-Labs Limited.
 
 ARTIQ is free software: you can redistribute it and/or modify
 it under the terms of the GNU Lesser General Public License as published by

cargo-xbuild.nix

@@ -1,23 +0,0 @@
-{ pkgs }:
-pkgs.rustPlatform.buildRustPackage rec {
-  pname = "cargo-xbuild";
-  version = "0.5.21";
-
-  src = pkgs.fetchFromGitHub {
-    owner = "rust-osdev";
-    repo = pname;
-    rev = "v${version}";
-    sha256 = "08mpnj3l6bcm1jg22lw1gcs0lkm4320fwl4p5y1s44w64963kzf7";
-  };
-
-  patches = [ ./cargo-xbuild.patch ];
-
-  cargoSha256 = "1pj4x8y5vfpnn8vhxqqm3vicn29870r3jh0b17q3riq4vz1a2afp";
-
-  meta = with pkgs.stdenv.lib; {
-    description = "Automatically cross-compiles the sysroot crates core, compiler_builtins, and alloc";
-    homepage = "https://github.com/rust-osdev/cargo-xbuild";
-    license = with licenses; [ mit asl20 ];
-    maintainers = with maintainers; [ johntitor xrelkd ];
-  };
-}

cargo-xbuild.patch

@@ -1,13 +0,0 @@
-diff --git a/src/sysroot.rs b/src/sysroot.rs
-index 1f3c8d1..e5615ee 100644
---- a/src/sysroot.rs
-+++ b/src/sysroot.rs
-@@ -163,7 +163,7 @@ version = "0.0.0"
- edition = "2018"
- 
- [dependencies.compiler_builtins]
--version = "0.1.0"
-+git = "https://git.m-labs.hk/M-Labs/compiler-builtins-zynq.git"
- "#;
- 
-     let mut stoml = TOML.to_owned();

default.nix

@@ -1,128 +0,0 @@
-let
-  zynq-rs = (import ./zynq-rs.nix);
-  pkgs = import <nixpkgs> { overlays = [ (import "${zynq-rs}/nix/mozilla-overlay.nix") ]; };
-  rustPlatform = (import "${zynq-rs}/nix/rust-platform.nix" { inherit pkgs; });
-  zc706-szl = (import zynq-rs).zc706-szl;
-  zc706-fsbl = import "${zynq-rs}/nix/fsbl.nix" { inherit pkgs; };
-  mkbootimage = import "${zynq-rs}/nix/mkbootimage.nix" { inherit pkgs; };
-  artiqpkgs = import <artiq-fast/default.nix> { inherit pkgs; };
-  vivado = import <artiq-fast/vivado.nix> { inherit pkgs; };
-  build-zc706 = { variant }: let
-    firmware = rustPlatform.buildRustPackage rec {
-      name = "zc706-${variant}-firmware";
-      version = "0.1.0";
-
-      src = ./src;
-      cargoSha256 = "10hap25cy2qgwr7b86jid73i6fp480iym29r3r97jindfxk0svi0";
-
-      nativeBuildInputs = [
-        pkgs.gnumake
-        (pkgs.python3.withPackages(ps: (with artiqpkgs; [ migen migen-axi misoc artiq ])))
-        pkgs.cargo-xbuild
-        pkgs.llvmPackages_9.llvm
-        pkgs.llvmPackages_9.clang-unwrapped
-      ];
-      buildPhase = ''
-        export XARGO_RUST_SRC="${rustPlatform.rust.rustc.src}/src"
-        export CARGO_HOME=$(mktemp -d cargo-home.XXX)
-        make VARIANT=${variant}
-      '';
-
-      installPhase = ''
-        mkdir -p $out $out/nix-support
-        cp ../build/runtime.bin $out/runtime.bin
-        cp ../build/firmware/armv7-none-eabihf/release/runtime $out/runtime.elf
-        echo file binary-dist $out/runtime.bin >> $out/nix-support/hydra-build-products
-        echo file binary-dist $out/runtime.elf >> $out/nix-support/hydra-build-products
-      '';
-
-      doCheck = false;
-      dontFixup = true;
-    };
-    gateware = pkgs.runCommand "zc706-${variant}-gateware"
-      {
-        nativeBuildInputs = [ 
-          (pkgs.python3.withPackages(ps: (with artiqpkgs; [ migen migen-axi misoc artiq ])))
-          vivado
-        ];
-      }
-      ''
-        python ${./src/gateware}/zc706.py -g build -V ${variant}
-        mkdir -p $out $out/nix-support
-        cp build/top.bit $out
-        echo file binary-dist $out/top.bit >> $out/nix-support/hydra-build-products
-      '';
-
-    # SZL startup
-    jtag = pkgs.runCommand "zc706-${variant}-jtag" {}
-      ''
-        mkdir $out
-        ln -s ${zc706-szl}/szl.elf $out
-        ln -s ${firmware}/runtime.bin $out
-        ln -s ${gateware}/top.bit $out
-      '';
-    sd = pkgs.runCommand "zc706-${variant}-sd"
-      {
-        buildInputs = [ mkbootimage ];
-      }
-      ''
-      # Do not use "long" paths in boot.bif, because embedded developers
-      # can't write software (mkbootimage will segfault).
-      bifdir=`mktemp -d`
-      cd $bifdir
-      ln -s ${zc706-szl}/szl.elf szl.elf
-      ln -s ${firmware}/runtime.elf runtime.elf
-      ln -s ${gateware}/top.bit top.bit
-      cat > boot.bif << EOF
-      the_ROM_image:
-      {
-        [bootloader]szl.elf
-        top.bit
-        runtime.elf
-      }
-      EOF
-      mkdir $out $out/nix-support
-      mkbootimage boot.bif $out/boot.bin
-      echo file binary-dist $out/boot.bin >> $out/nix-support/hydra-build-products
-      '';
-
-    # FSBL startup
-    fsbl-sd = pkgs.runCommand "zc706-${variant}-fsbl-sd"
-      {
-        buildInputs = [ mkbootimage ];
-      }
-      ''
-      bifdir=`mktemp -d`
-      cd $bifdir
-      ln -s ${zc706-fsbl}/fsbl.elf fsbl.elf
-      ln -s ${gateware}/top.bit top.bit
-      ln -s ${firmware}/runtime.elf runtime.elf
-      cat > boot.bif << EOF
-      the_ROM_image:
-      {
-        [bootloader]fsbl.elf
-        top.bit
-        runtime.elf
-      }
-      EOF
-      mkdir $out $out/nix-support
-      mkbootimage boot.bif $out/boot.bin
-      echo file binary-dist $out/boot.bin >> $out/nix-support/hydra-build-products
-      '';
-  in {
-    "zc706-${variant}-firmware" = firmware;
-    "zc706-${variant}-gateware" = gateware;
-    "zc706-${variant}-jtag" = jtag;
-    "zc706-${variant}-sd" = sd;
-    "zc706-${variant}-fsbl-sd" = fsbl-sd;
-  };
-in
-  (
-    (build-zc706 { variant = "simple"; }) //
-    (build-zc706 { variant = "nist_clock"; }) //
-    (build-zc706 { variant = "nist_qc2"; }) //
-    (build-zc706 { variant = "acpki_simple"; }) //
-    (build-zc706 { variant = "acpki_nist_clock"; }) //
-    (build-zc706 { variant = "acpki_nist_qc2"; }) //
-    { inherit zynq-rs; }
-  )

demo.json

@@ -0,0 +1,60 @@
+{
+    "target": "kasli_soc",
+    "variant": "demo",
+    "hw_rev": "v1.0",
+    "base": "standalone",
+    "peripherals": [
+        {
+            "type": "grabber",
+            "ports": [0]
+        },
+        {
+            "type": "dio",
+            "ports": [1],
+            "bank_direction_low": "input",
+            "bank_direction_high": "output"
+        },
+        {
+            "type": "dio",
+            "ports": [2],
+            "bank_direction_low": "output",
+            "bank_direction_high": "output"
+        },
+        {
+            "type": "urukul",
+            "dds": "ad9910",
+            "ports": [3, 4],
+            "clk_sel": 2
+        },
+        {
+            "type": "zotino",
+            "ports": [5]
+        },
+        {
+            "type": "sampler",
+            "ports": [6, 7]
+        },
+        {
+            "type": "mirny",
+            "ports": [8],
+            "clk_sel": 1,
+            "refclk": 125e6
+        },
+        {
+            "type": "fastino",
+            "ports": [9]
+        },
+        {
+            "type": "dio",
+            "ports": [10],
+            "bank_direction_low": "input",
+            "bank_direction_high": "input"
+        },
+        {
+            "type": "dio",
+            "ports": [11],
+            "bank_direction_low": "output",
+            "bank_direction_high": "input"
+        }
+    ]
+}

examples/device_db.py

@@ -8,7 +8,7 @@ device_db = {
         "arguments": {
             "host": "192.168.1.52",
             "ref_period": 1e-9,
-            "ref_multiplier": 1,
+            "ref_multiplier": 8,
             "target": "cortexa9"
         }
     },
@@ -29,30 +29,11 @@ device_db = {
         "class": "PCA9548"
     },
 
-    # led? are common to all variants
     "led0": {
         "type": "local",
         "module": "artiq.coredevice.ttl",
         "class": "TTLOut",
-        "arguments": {"channel": 0},
-    },
-    "led1": {
-        "type": "local",
-        "module": "artiq.coredevice.ttl",
-        "class": "TTLOut",
-        "arguments": {"channel": 1},
-    },
-    "led2": {
-        "type": "local",
-        "module": "artiq.coredevice.ttl",
-        "class": "TTLOut",
-        "arguments": {"channel": 2}
-    },
-    "led3": {
-        "type": "local",
-        "module": "artiq.coredevice.ttl",
-        "class": "TTLOut",
-        "arguments": {"channel": 3}
+        "arguments": {"channel": 41},
     },
 }
 
@@ -62,7 +43,7 @@ for i in range(40):
         "type": "local",
         "module": "artiq.coredevice.ttl",
         "class": "TTLInOut",
-        "arguments": {"channel": 4+i}
+        "arguments": {"channel": i}
     }
 
 device_db["ad9914dds0"] = {
@@ -78,6 +59,14 @@ device_db["ad9914dds1"] = {
     "arguments": {"sysclk": 3e9, "bus_channel": 50, "channel": 1},
 }
 
+for i in range(4):
+    device_db["ttl"+str(i)+"_counter"] = {
+        "type": "local",
+        "module": "artiq.coredevice.edge_counter",
+        "class": "EdgeCounter",
+        "arguments": {"channel": 52+i}
+    }
+
 # for ARTIQ test suite
 device_db.update(
     loop_out="ttl0",

examples/ebaz4205/device_db.py

@@ -0,0 +1,78 @@
+core_addr = "192.168.1.57"
+
+device_db = {
+    "core": {
+        "type": "local",
+        "module": "artiq.coredevice.core",
+        "class": "Core",
+        "arguments": {
+            "host": core_addr,
+            "ref_period": 1e-9,
+            "target": "cortexa9",
+        },
+    },
+    "core_log": {
+        "type": "controller",
+        "host": "::1",
+        "port": 1068,
+        "command": "aqctl_corelog -p {port} --bind {bind} " + core_addr,
+    },
+    "core_moninj": {
+        "type": "controller",
+        "host": "::1",
+        "port_proxy": 1383,
+        "port": 1384,
+        "command": "aqctl_moninj_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
+        + core_addr,
+    },
+    "core_analyzer": {
+        "type": "controller",
+        "host": "::1",
+        "port_proxy": 1385,
+        "port": 1386,
+        "command": "aqctl_coreanalyzer_proxy --port-proxy {port_proxy} --port-control {port} --bind {bind} "
+        + core_addr,
+    },
+    "core_cache": {
+        "type": "local",
+        "module": "artiq.coredevice.cache",
+        "class": "CoreCache",
+    },
+    "core_dma": {"type": "local", "module": "artiq.coredevice.dma", "class": "CoreDMA"},
+    "led0": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 0},
+    },
+    "led1": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 1},
+    },
+}
+
+# TTLs starting at RTIO channel 2, ending at RTIO channel 15
+for i in range(2, 16):
+    device_db["ttl" + str(i)] = {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLInOut",
+        "arguments": {"channel": i},
+    }
+
+device_db.update(
+    spi0={
+        "type": "local",
+        "module": "artiq.coredevice.spi2",
+        "class": "SPIMaster",
+        "arguments": {"channel": 16},
+    },
+    dds0={
+        "type": "local",
+        "module": "artiq.coredevice.ad9834",
+        "class": "AD9834",
+        "arguments": {"spi_device": "spi0"},
+    },
+)

flake.lock

@@ -0,0 +1,248 @@
+{
+  "nodes": {
+    "artiq": {
+      "inputs": {
+        "artiq-comtools": "artiq-comtools",
+        "nixpkgs": "nixpkgs",
+        "rust-overlay": "rust-overlay",
+        "sipyco": "sipyco",
+        "src-migen": "src-migen",
+        "src-misoc": "src-misoc",
+        "src-pythonparser": "src-pythonparser"
+      },
+      "locked": {
+        "lastModified": 1732066716,
+        "narHash": "sha256-krjvt9+RccnAxSEZcFhRpjA2S3CoqE4MSa1JUg421b4=",
+        "ref": "refs/heads/master",
+        "rev": "270a417a28b516d36983779a1adb6d33a3c55a4a",
+        "revCount": 9102,
+        "type": "git",
+        "url": "https://github.com/m-labs/artiq.git"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://github.com/m-labs/artiq.git"
+      }
+    },
+    "artiq-comtools": {
+      "inputs": {
+        "flake-utils": "flake-utils",
+        "nixpkgs": [
+          "artiq",
+          "nixpkgs"
+        ],
+        "sipyco": [
+          "artiq",
+          "sipyco"
+        ]
+      },
+      "locked": {
+        "lastModified": 1720768567,
+        "narHash": "sha256-3VoK7o5MtHtbHLrc6Pv+eQWFtaz5Gd/YWyV5TD3c5Ss=",
+        "owner": "m-labs",
+        "repo": "artiq-comtools",
+        "rev": "f93570d8f2ed5a3cfb3e1c16ab00f2540551e994",
+        "type": "github"
+      },
+      "original": {
+        "owner": "m-labs",
+        "repo": "artiq-comtools",
+        "type": "github"
+      }
+    },
+    "flake-utils": {
+      "inputs": {
+        "systems": "systems"
+      },
+      "locked": {
+        "lastModified": 1710146030,
+        "narHash": "sha256-SZ5L6eA7HJ/nmkzGG7/ISclqe6oZdOZTNoesiInkXPQ=",
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "rev": "b1d9ab70662946ef0850d488da1c9019f3a9752a",
+        "type": "github"
+      },
+      "original": {
+        "owner": "numtide",
+        "repo": "flake-utils",
+        "type": "github"
+      }
+    },
+    "nixpkgs": {
+      "locked": {
+        "lastModified": 1731319897,
+        "narHash": "sha256-PbABj4tnbWFMfBp6OcUK5iGy1QY+/Z96ZcLpooIbuEI=",
+        "owner": "NixOS",
+        "repo": "nixpkgs",
+        "rev": "dc460ec76cbff0e66e269457d7b728432263166c",
+        "type": "github"
+      },
+      "original": {
+        "owner": "NixOS",
+        "ref": "nixos-unstable",
+        "repo": "nixpkgs",
+        "type": "github"
+      }
+    },
+    "root": {
+      "inputs": {
+        "artiq": "artiq",
+        "zynq-rs": "zynq-rs"
+      }
+    },
+    "rust-overlay": {
+      "inputs": {
+        "nixpkgs": [
+          "artiq",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1719454714,
+        "narHash": "sha256-MojqG0lyUINkEk0b3kM2drsU5vyaF8DFZe/FAlZVOGs=",
+        "owner": "oxalica",
+        "repo": "rust-overlay",
+        "rev": "d1c527659cf076ecc4b96a91c702d080b213801e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "oxalica",
+        "ref": "snapshot/2024-08-01",
+        "repo": "rust-overlay",
+        "type": "github"
+      }
+    },
+    "rust-overlay_2": {
+      "inputs": {
+        "nixpkgs": [
+          "zynq-rs",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1719454714,
+        "narHash": "sha256-MojqG0lyUINkEk0b3kM2drsU5vyaF8DFZe/FAlZVOGs=",
+        "owner": "oxalica",
+        "repo": "rust-overlay",
+        "rev": "d1c527659cf076ecc4b96a91c702d080b213801e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "oxalica",
+        "ref": "snapshot/2024-08-01",
+        "repo": "rust-overlay",
+        "type": "github"
+      }
+    },
+    "sipyco": {
+      "inputs": {
+        "nixpkgs": [
+          "artiq",
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1728371104,
+        "narHash": "sha256-PPnAyDedUQ7Og/Cby9x5OT9wMkNGTP8GS53V6N/dk4w=",
+        "owner": "m-labs",
+        "repo": "sipyco",
+        "rev": "094a6cd63ffa980ef63698920170e50dc9ba77fd",
+        "type": "github"
+      },
+      "original": {
+        "owner": "m-labs",
+        "repo": "sipyco",
+        "type": "github"
+      }
+    },
+    "src-migen": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1727677091,
+        "narHash": "sha256-Zg3SQnTwMM/VkOGKogbPyuCC2NhLy8HB2SPEUWWNgCU=",
+        "owner": "m-labs",
+        "repo": "migen",
+        "rev": "c19ae9f8ae162ffe2d310a92bfce53ac2a821bc8",
+        "type": "github"
+      },
+      "original": {
+        "owner": "m-labs",
+        "repo": "migen",
+        "type": "github"
+      }
+    },
+    "src-misoc": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1729234629,
+        "narHash": "sha256-TLsTCXV5AC2xh+bS7EhBVBKqdqIU3eKrnlWcFF9LtAM=",
+        "ref": "refs/heads/master",
+        "rev": "6085a312bca26adeca6584e37d08c8ba2e1d6e38",
+        "revCount": 2460,
+        "submodules": true,
+        "type": "git",
+        "url": "https://github.com/m-labs/misoc.git"
+      },
+      "original": {
+        "submodules": true,
+        "type": "git",
+        "url": "https://github.com/m-labs/misoc.git"
+      }
+    },
+    "src-pythonparser": {
+      "flake": false,
+      "locked": {
+        "lastModified": 1628745371,
+        "narHash": "sha256-p6TgeeaK4NEmbhimEXp31W8hVRo4DgWmcCoqZ+UdN60=",
+        "owner": "m-labs",
+        "repo": "pythonparser",
+        "rev": "5413ee5c9f8760e95c6acd5d6e88dabb831ad201",
+        "type": "github"
+      },
+      "original": {
+        "owner": "m-labs",
+        "repo": "pythonparser",
+        "type": "github"
+      }
+    },
+    "systems": {
+      "locked": {
+        "lastModified": 1681028828,
+        "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
+        "owner": "nix-systems",
+        "repo": "default",
+        "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-systems",
+        "repo": "default",
+        "type": "github"
+      }
+    },
+    "zynq-rs": {
+      "inputs": {
+        "nixpkgs": [
+          "artiq",
+          "nixpkgs"
+        ],
+        "rust-overlay": "rust-overlay_2"
+      },
+      "locked": {
+        "lastModified": 1731749494,
+        "narHash": "sha256-WGigAhvVCGN5YZ1dHPyvoqAh47W1Gtph036O1aKFlLE=",
+        "ref": "refs/heads/master",
+        "rev": "12975de2e110d7948bf47b768559f727d0abc8fc",
+        "revCount": 655,
+        "type": "git",
+        "url": "https://git.m-labs.hk/m-labs/zynq-rs"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.m-labs.hk/m-labs/zynq-rs"
+      }
+    }
+  },
+  "root": "root",
+  "version": 7
+}

flake.nix

@@ -0,0 +1,399 @@
+{
+  description = "ARTIQ port to the Zynq-7000 platform";
+
+  inputs.artiq.url = git+https://github.com/m-labs/artiq.git;
+  inputs.zynq-rs.url = git+https://git.m-labs.hk/m-labs/zynq-rs;
+  inputs.zynq-rs.inputs.nixpkgs.follows = "artiq/nixpkgs";
+
+  outputs = { self, zynq-rs, artiq }:
+  let
+    pkgs = import artiq.inputs.nixpkgs { system = "x86_64-linux"; overlays = [ (import zynq-rs.inputs.rust-overlay) ]; };
+    zynqpkgs = zynq-rs.packages.x86_64-linux;
+    artiqpkgs = artiq.packages.x86_64-linux;
+    llvmPackages_11 = zynq-rs.llvmPackages_11;
+
+    rust = zynq-rs.rust;
+    rustPlatform = zynq-rs.rustPlatform;
+
+    fastnumbers = pkgs.python3Packages.buildPythonPackage rec {
+      pname = "fastnumbers";
+      version = "5.1.0";
+
+      src = pkgs.python3Packages.fetchPypi {
+        inherit pname version;
+        sha256 = "sha256-4JLTP4uVwxcaL7NOV57+DFSwKQ3X+W/6onYkN2AdkKc=";
+      };
+    };
+
+    artiq-netboot = pkgs.python3Packages.buildPythonPackage rec {
+      pname = "artiq-netboot";
+      version = "unstable-2020-10-15";
+
+      src = pkgs.fetchgit {
+        url = "https://git.m-labs.hk/m-labs/artiq-netboot.git";
+        rev = "04f69eb07df73abe4b89fde2c24084f7664f2104";
+        sha256 = "0ql4fr8m8gpb2yql8aqsdqsssxb8zqd6l65kl1f6s9845zy7shs9";
+      };
+    };
+
+    ramda = pkgs.python3Packages.buildPythonPackage {
+      pname = "ramda";
+      version = "unstable-2020-04-11";
+
+      src = pkgs.fetchFromGitHub {
+        owner = "peteut";
+        repo = "ramda.py";
+        rev = "d315a9717ebd639366bf3fe26bad9e3d08ec3c49";
+        sha256 = "sha256-bmSt/IHDnULsZjsC6edELnNH7LoJSVF4L4XhwBAXRkY=";
+      };
+
+      nativeBuildInputs = with pkgs.python3Packages; [ pbr ];
+      propagatedBuildInputs = with pkgs.python3Packages; [ future fastnumbers ];
+
+      checkInputs = with pkgs.python3Packages; [ pytest ];
+      checkPhase = "pytest";
+      doCheck = false;
+
+      preBuild = ''
+        export PBR_VERSION=0.5.5
+      '';
+    };
+
+    migen-axi = pkgs.python3Packages.buildPythonPackage {
+      pname = "migen-axi";
+      version = "unstable-2023-01-06";
+
+      src = pkgs.fetchFromGitHub {
+        owner = "peteut";
+        repo = "migen-axi";
+        rev = "27eaa84a70a3abfe1930c86c36c4de2cd652da35";
+        sha256 = "sha256-3Y9W5ns+1wbVd14iePzgSBzE+LxnGMUDtUw3BccFt80=";
+      };
+
+      format = "pyproject";
+
+      propagatedBuildInputs = with pkgs.python3Packages; [ setuptools click numpy toolz jinja2 ramda artiqpkgs.migen artiqpkgs.misoc ];
+
+      checkInputs = with pkgs.python3Packages; [ pytestCheckHook pytest-timeout ];
+
+      # migen/misoc version checks are broken with pyproject for some reason
+      postPatch = ''
+        sed -i "1,4d" pyproject.toml
+        substituteInPlace pyproject.toml \
+          --replace '"migen@git+https://github.com/m-labs/migen",' ""
+        substituteInPlace pyproject.toml \
+          --replace '"misoc@git+https://github.com/m-labs/misoc.git",' ""
+        # pytest-flake8 is broken with recent flake8. Re-enable after fix.
+        substituteInPlace setup.cfg --replace '--flake8' ""
+      '';
+    };
+    binutils = { platform, target, zlib }: pkgs.stdenv.mkDerivation rec {
+      basename = "binutils";
+      version = "2.30";
+      name = "${basename}-${platform}-${version}";
+      src = pkgs.fetchurl {
+        url = "https://ftp.gnu.org/gnu/binutils/binutils-${version}.tar.bz2";
+        sha256 = "028cklfqaab24glva1ks2aqa1zxa6w6xmc8q34zs1sb7h22dxspg";
+      };
+      configureFlags =
+        [ "--enable-shared" "--enable-deterministic-archives" "--target=${target}"];
+      outputs = [ "out" "info" "man" ];
+      depsBuildBuild = [ pkgs.buildPackages.stdenv.cc ];
+      buildInputs = [ zlib ];
+      enableParallelBuilding = true;
+    };
+    binutils-arm = pkgs.callPackage binutils { platform = "arm"; target = "armv7-unknown-linux-gnueabihf"; };
+
+    # FSBL configuration supplied by Vivado 2020.1 for these boards:
+    fsblTargets = ["zc702" "zc706" "zed"];
+    sat_variants = [
+      # kasli-soc satellite variants
+      "satellite"
+      # zc706 satellite variants
+      "nist_clock_satellite" "nist_qc2_satellite" "acpki_nist_clock_satellite" "acpki_nist_qc2_satellite" 
+      "nist_clock_satellite_100mhz" "nist_qc2_satellite_100mhz" "acpki_nist_clock_satellite_100mhz" "acpki_nist_qc2_satellite_100mhz"
+    ];
+    board-package-set = { target, variant, json ? null }: let
+      szl = zynqpkgs."${target}-szl";
+      fsbl = zynqpkgs."${target}-fsbl";
+      fwtype = if builtins.elem variant sat_variants then "satman" else "runtime";
+
+      firmware = rustPlatform.buildRustPackage rec {
+        name = "firmware";
+        src = ./src;
+        cargoLock = { 
+          lockFile = src/Cargo.lock;
+          outputHashes = {
+              "tar-no-std-0.1.8" = "sha256-xm17108v4smXOqxdLvHl9CxTCJslmeogjm4Y87IXFuM=";
+              "nalgebra-0.32.6" = "sha256-L/YudkVOtfGYoNQKBD7LMk/sMYgRDzPDdpGL5rO7G2I=";
+            };
+        };
+
+        nativeBuildInputs = [
+          pkgs.gnumake
+          (pkgs.python3.withPackages(ps: [ ps.jsonschema artiqpkgs.migen migen-axi artiqpkgs.misoc artiqpkgs.artiq ]))
+          zynqpkgs.cargo-xbuild
+          llvmPackages_11.llvm
+          llvmPackages_11.clang-unwrapped
+        ];
+        buildPhase = ''
+          export XARGO_RUST_SRC="${rust}/lib/rustlib/src/rust/library"
+          export CLANG_EXTRA_INCLUDE_DIR="${llvmPackages_11.clang-unwrapped.lib}/lib/clang/11.1.0/include"
+          export CARGO_HOME=$(mktemp -d cargo-home.XXX)
+          export ZYNQ_RS=${zynq-rs}
+          make TARGET=${target} GWARGS="${if json == null then "-V ${variant}" else json}" ${fwtype}
+        '';
+
+        installPhase = ''
+          mkdir -p $out $out/nix-support
+          cp ../build/${fwtype}.bin $out/${fwtype}.bin
+          cp ../build/firmware/armv7-none-eabihf/release/${fwtype} $out/${fwtype}.elf
+          echo file binary-dist $out/${fwtype}.bin >> $out/nix-support/hydra-build-products
+          echo file binary-dist $out/${fwtype}.elf >> $out/nix-support/hydra-build-products
+        '';
+
+        doCheck = false;
+        dontFixup = true;
+        auditable = false;
+      };
+      gateware = pkgs.runCommand "${target}-${variant}-gateware"
+        {
+          nativeBuildInputs = [ 
+            (pkgs.python3.withPackages(ps: [ ps.jsonschema artiqpkgs.migen migen-axi artiqpkgs.misoc artiqpkgs.artiq ]))
+            artiqpkgs.vivado
+          ];
+        }
+        ''
+          python ${./src/gateware}/${target}.py -g build ${if json == null then "-V ${variant}" else json}
+          mkdir -p $out $out/nix-support
+          cp build/top.bit $out
+          echo file binary-dist $out/top.bit >> $out/nix-support/hydra-build-products
+        '';
+
+      # SZL startup
+      jtag = pkgs.runCommand "${target}-${variant}-jtag" {}
+        ''
+          mkdir $out
+          ln -s ${szl}/szl.elf $out
+          ln -s ${firmware}/${fwtype}.bin $out
+          ln -s ${gateware}/top.bit $out
+        '';
+      sd = pkgs.runCommand "${target}-${variant}-sd"
+        {
+          buildInputs = [ zynqpkgs.mkbootimage ];
+        }
+        ''
+          # Do not use "long" paths in boot.bif, because embedded developers
+          # can't write software (mkbootimage will segfault).
+          bifdir=`mktemp -d`
+          cd $bifdir
+          ln -s ${szl}/szl.elf szl.elf
+          ln -s ${firmware}/${fwtype}.elf ${fwtype}.elf
+          ln -s ${gateware}/top.bit top.bit
+          cat > boot.bif << EOF
+          the_ROM_image:
+          {
+            [bootloader]szl.elf
+            top.bit
+            ${fwtype}.elf
+          }
+          EOF
+          mkdir $out $out/nix-support
+          mkbootimage boot.bif $out/boot.bin
+          echo file binary-dist $out/boot.bin >> $out/nix-support/hydra-build-products
+        '';
+
+      # FSBL startup
+      fsbl-sd = pkgs.runCommand "${target}-${variant}-fsbl-sd"
+        {
+          buildInputs = [ zynqpkgs.mkbootimage ];
+        }
+        ''
+          bifdir=`mktemp -d`
+          cd $bifdir
+          ln -s ${fsbl}/fsbl.elf fsbl.elf
+          ln -s ${gateware}/top.bit top.bit
+          ln -s ${firmware}/${fwtype}.elf ${fwtype}.elf
+          cat > boot.bif << EOF
+          the_ROM_image:
+          {
+            [bootloader]fsbl.elf
+            top.bit
+            ${fwtype}.elf
+          }
+          EOF
+          mkdir $out $out/nix-support
+          mkbootimage boot.bif $out/boot.bin
+          echo file binary-dist $out/boot.bin >> $out/nix-support/hydra-build-products
+        '';
+    in {
+      "${target}-${variant}-firmware" = firmware;
+      "${target}-${variant}-gateware" = gateware;
+      "${target}-${variant}-jtag" = jtag;
+      "${target}-${variant}-sd" = sd;
+    } // (
+      if builtins.elem target fsblTargets
+      then {
+        "${target}-${variant}-fsbl-sd" = fsbl-sd;
+      }
+      else {}
+    );
+
+    gateware-sim = pkgs.stdenv.mkDerivation {
+      name = "gateware-sim";
+      
+      nativeBuildInputs = [ 
+        (pkgs.python3.withPackages(ps: [ artiqpkgs.migen migen-axi artiqpkgs.artiq ]))
+      ];
+
+      phases = [ "buildPhase" ];
+
+      buildPhase =
+        ''
+        python -m unittest discover ${self}/src/gateware -v
+        touch $out
+        '';
+    };
+
+    fmt-check = pkgs.stdenvNoCC.mkDerivation {
+      name = "fmt-check";
+
+      src = ./src;
+
+      nativeBuildInputs = [ rust pkgs.gnumake ];
+
+      phases = [ "unpackPhase" "buildPhase" ];
+
+      buildPhase =
+        ''
+        export ZYNQ_RS=${zynq-rs}
+        make manifests
+        cargo fmt -- --check
+        touch $out
+        '';
+    };
+
+    # for hitl-tests
+    zc706-nist_qc2 = (board-package-set { target = "zc706"; variant = "nist_qc2"; });
+    zc706-hitl-tests = pkgs.stdenv.mkDerivation {
+      name = "zc706-hitl-tests";
+
+      __networked = true;  # compatibility with old patched Nix
+      # breaks hydra, https://github.com/NixOS/hydra/issues/1216
+      #__impure = true;     # Nix 2.8+
+
+      buildInputs = [
+        pkgs.netcat pkgs.openssh pkgs.rsync artiqpkgs.artiq artiq-netboot zynqpkgs.zc706-szl
+      ];
+      phases = [ "buildPhase" ];
+
+      buildPhase =
+        ''
+        export NIX_SSHOPTS="-F /dev/null -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i /opt/hydra_id_ed25519"
+        LOCKCTL=$(mktemp -d)
+        mkfifo $LOCKCTL/lockctl
+
+        cat $LOCKCTL/lockctl | ${pkgs.openssh}/bin/ssh \
+        $NIX_SSHOPTS \
+        rpi-4 \
+        'mkdir -p /tmp/board_lock && flock /tmp/board_lock/zc706-1 -c "echo Ok; cat"' \
+        | (
+          # End remote flock via FIFO
+          atexit_unlock() {
+            echo > $LOCKCTL/lockctl
+          }
+          trap atexit_unlock EXIT
+
+          # Read "Ok" line when remote successfully locked
+          read LOCK_OK
+
+          echo Power cycling board...
+          (echo b; sleep 5; echo B; sleep 5) | nc -N -w6 192.168.1.31 3131
+          echo Power cycle done.
+
+          export USER=hydra
+          export OPENOCD_ZYNQ=${zynq-rs}/openocd
+          export SZL=${zynqpkgs.szl}
+          bash ${self}/remote_run.sh -h rpi-4 -o "$NIX_SSHOPTS" -d ${zc706-nist_qc2.zc706-nist_qc2-jtag}
+
+          echo Waiting for the firmware to boot...
+          sleep 15
+
+          echo Running test kernel...
+          artiq_run --device-db ${self}/examples/device_db.py ${self}/examples/mandelbrot.py
+
+          echo Running ARTIQ unit tests...
+          export ARTIQ_ROOT=${self}/examples
+          export ARTIQ_LOW_LATENCY=1
+          python -m unittest discover artiq.test.coredevice -v
+
+          touch $out
+
+          echo Completed
+
+          (echo b; sleep 5) | nc -N -w6 192.168.1.31 3131
+          echo Board powered off
+        )
+        '';
+    };
+  in rec {
+    packages.x86_64-linux =
+      {
+        inherit fastnumbers artiq-netboot ramda migen-axi binutils-arm;
+      } //
+      (board-package-set { target = "zc706"; variant = "nist_clock"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_master"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_clock_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_master"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "nist_qc2_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_master"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_clock_satellite_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_master"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_master_100mhz"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_satellite"; }) //
+      (board-package-set { target = "zc706"; variant = "acpki_nist_qc2_satellite_100mhz"; }) //
+      (board-package-set { target = "kasli_soc"; variant = "demo"; json = ./demo.json; }) //
+      (board-package-set { target = "kasli_soc"; variant = "master"; json = ./kasli-soc-master.json; }) //
+      (board-package-set { target = "kasli_soc"; variant = "satellite"; json = ./kasli-soc-satellite.json; }) //
+      (board-package-set { target = "ebaz4205"; variant = "base"; });
+
+    hydraJobs = packages.x86_64-linux // { inherit zc706-hitl-tests; inherit gateware-sim; inherit fmt-check; };
+
+    devShell.x86_64-linux = pkgs.mkShell {
+      name = "artiq-zynq-dev-shell";
+      buildInputs = with pkgs; [
+        rust
+        llvmPackages_11.llvm
+        llvmPackages_11.clang-unwrapped
+        gnumake
+        cacert
+        zynqpkgs.cargo-xbuild
+        zynqpkgs.mkbootimage
+        openocd  
+        openssh rsync
+        (python3.withPackages(ps: (with artiqpkgs; [ migen migen-axi misoc artiq artiq-netboot ps.jsonschema ps.pyftdi ])))
+        artiqpkgs.artiq
+        artiqpkgs.vivado
+        binutils-arm
+        pre-commit
+      ];
+      XARGO_RUST_SRC = "${rust}/lib/rustlib/src/rust/library";
+      CLANG_EXTRA_INCLUDE_DIR = "${llvmPackages_11.clang-unwrapped.lib}/lib/clang/11.1.0/include";
+      ZYNQ_RS = "${zynq-rs}";
+      OPENOCD_ZYNQ = "${zynq-rs}/openocd";
+      SZL = "${zynqpkgs.szl}";
+    };
+
+    makeArtiqZynqPackage = board-package-set;
+
+  };
+}

kasli-soc-master.json

@@ -0,0 +1,60 @@
+{
+    "target": "kasli_soc",
+    "variant": "master",
+    "hw_rev": "v1.0",
+    "base": "master",
+    "peripherals": [
+        {
+            "type": "grabber",
+            "ports": [0]
+        },
+        {
+            "type": "dio",
+            "ports": [1],
+            "bank_direction_low": "input",
+            "bank_direction_high": "output"
+        },
+        {
+            "type": "dio",
+            "ports": [2],
+            "bank_direction_low": "output",
+            "bank_direction_high": "output"
+        },
+        {
+            "type": "urukul",
+            "dds": "ad9910",
+            "ports": [3, 4],
+            "clk_sel": 2
+        },
+        {
+            "type": "zotino",
+            "ports": [5]
+        },
+        {
+            "type": "sampler",
+            "ports": [6, 7]
+        },
+        {
+            "type": "mirny",
+            "ports": [8],
+            "clk_sel": 1,
+            "refclk": 125e6
+        },
+        {
+            "type": "fastino",
+            "ports": [9]
+        },
+        {
+            "type": "dio",
+            "ports": [10],
+            "bank_direction_low": "input",
+            "bank_direction_high": "input"
+        },
+        {
+            "type": "dio",
+            "ports": [11],
+            "bank_direction_low": "output",
+            "bank_direction_high": "input"
+        }
+    ]
+}

kasli-soc-satellite.json

@@ -0,0 +1,60 @@
+{
+    "target": "kasli_soc",
+    "variant": "satellite",
+    "hw_rev": "v1.0",
+    "base": "satellite",
+    "peripherals": [
+        {
+            "type": "grabber",
+            "ports": [0]
+        },
+        {
+            "type": "dio",
+            "ports": [1],
+            "bank_direction_low": "input",
+            "bank_direction_high": "output"
+        },
+        {
+            "type": "dio",
+            "ports": [2],
+            "bank_direction_low": "output",
+            "bank_direction_high": "output"
+        },
+        {
+            "type": "urukul",
+            "dds": "ad9910",
+            "ports": [3, 4],
+            "clk_sel": 2
+        },
+        {
+            "type": "zotino",
+            "ports": [5]
+        },
+        {
+            "type": "sampler",
+            "ports": [6, 7]
+        },
+        {
+            "type": "mirny",
+            "ports": [8],
+            "clk_sel": 1,
+            "refclk": 125e6
+        },
+        {
+            "type": "fastino",
+            "ports": [9]
+        },
+        {
+            "type": "dio",
+            "ports": [10],
+            "bank_direction_low": "input",
+            "bank_direction_high": "input"
+        },
+        {
+            "type": "dio",
+            "ports": [11],
+            "bank_direction_low": "output",
+            "bank_direction_high": "input"
+        }
+    ]
+}

local_run.sh

@@ -13,9 +13,10 @@ fi
 
 impure=0
 load_bitstream=1
-board_host="192.168.1.52"
+board_type="kasli_soc"
+fw_type="runtime"
 
-while getopts "ilb:" opt; do
+while getopts "ilb:t:f:" opt; do
     case "$opt" in
     \?) exit 1
         ;;
@@ -25,24 +26,36 @@ while getopts "ilb:" opt; do
         ;;
     b)  board_host=$OPTARG
         ;;
+    t)  board_type=$OPTARG
+        ;;
+    f)  fw_type=$OPTARG
+        ;;
     esac
 done
 
+if [ -z "$board_host" ]; then
+    case $board_type in
+    kasli_soc) board_host="192.168.1.56";;
+    zc706) board_host="192.168.1.52";;
+    *) echo "Unknown board type"; exit 1;;
+    esac
+fi
+
 load_bitstream_cmd=""
 
 build_dir=`pwd`/build
 result_dir=`pwd`/result
 cd $OPENOCD_ZYNQ
-openocd -f zc706.cfg -c "load_image $SZL; resume 0; exit"
+openocd -f $board_type.cfg -c "load_image $SZL/szl-$board_type.elf; resume 0; exit"
 sleep 5
 if [ $impure -eq 1 ]; then
     if [ $load_bitstream -eq 1 ]; then
         load_bitstream_cmd="-g $build_dir/gateware/top.bit"
     fi
-    artiq_netboot $load_bitstream_cmd -f $build_dir/runtime.bin -b $board_host
+    artiq_netboot $load_bitstream_cmd -f $build_dir/$fw_type.bin -b $board_host
 else
     if [ $load_bitstream -eq 1 ]; then
         load_bitstream_cmd="-g $result_dir/top.bit"
     fi
-    artiq_netboot $load_bitstream_cmd -f $result_dir/runtime.bin -b $board_host
+    artiq_netboot $load_bitstream_cmd -f $result_dir/$fw_type.bin -b $board_host
 fi

remote_run.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 
+# Only ZC706 supported for now.
+
 set -e
 
 if [ -z "$OPENOCD_ZYNQ" ]; then
@@ -18,8 +20,9 @@ impure_dir="build"
 sshopts=""
 load_bitstream=1
 board_host="192.168.1.52"
+fw_type="runtime"
 
-while getopts "h:id:o:l" opt; do
+while getopts "h:id:o:lt:" opt; do
     case "$opt" in
     \?) exit 1
         ;;
@@ -36,6 +39,8 @@ while getopts "h:id:o:l" opt; do
         ;;
     b)  board_host=$OPTARG
         ;;
+    t)  fw_type=$OPTARG
+        ;;
     esac
 done
 
@@ -46,17 +51,17 @@ echo "Creating $target_folder..."
 ssh $sshopts $target_host "mkdir -p $target_folder"
 echo "Copying files..."
 rsync -e "ssh $sshopts" -Lc $OPENOCD_ZYNQ/* $target_host:$target_folder
-rsync -e "ssh $sshopts" -Lc $SZL $target_host:$target_folder
+rsync -e "ssh $sshopts" -Lc $SZL/szl-zc706.elf $target_host:$target_folder/szl.elf
 if [ $impure -eq 1 ]; then
     if [ $load_bitstream -eq 1 ]; then
         load_bitstream_cmd="-g build/gateware/top.bit"
     fi
-    firmware="build/runtime.bin"
+    firmware="build/$fw_type.bin"
 else
     if [ $load_bitstream -eq 1 ]; then
         load_bitstream_cmd="-g $pure_dir/top.bit"
     fi
-    firmware="$pure_dir/runtime.bin"    
+    firmware="$pure_dir/$fw_type.bin"    
 fi
 echo "Programming board..."
 ssh $sshopts $target_host "cd $target_folder; openocd -f zc706.cfg -c'load_image szl.elf; resume 0; exit'"

shell.nix

@@ -1,35 +0,0 @@
-let
-  zynq-rs = (import ./zynq-rs.nix);
-  pkgs = import <nixpkgs> { overlays = [ (import "${zynq-rs}/nix/mozilla-overlay.nix") ]; };
-  rustPlatform = (import "${zynq-rs}/nix/rust-platform.nix" { inherit pkgs; });
-  artiq-fast = <artiq-fast>;
-  artiqpkgs = import "${artiq-fast}/default.nix" { inherit pkgs; };
-  vivado = import "${artiq-fast}/vivado.nix" { inherit pkgs; };
-  cargo-xbuild = import ./cargo-xbuild.nix { inherit pkgs; };
-  zc706-szl = (import zynq-rs).zc706-szl;
-in
-  pkgs.stdenv.mkDerivation {
-    name = "artiq-zynq-env";
-    buildInputs = [
-      pkgs.gnumake
-      rustPlatform.rust.rustc
-      rustPlatform.rust.cargo
-      pkgs.llvmPackages_9.llvm
-      pkgs.llvmPackages_9.clang-unwrapped
-      pkgs.cacert
-      cargo-xbuild
-
-      pkgs.openocd
-      pkgs.openssh pkgs.rsync
-
-      (pkgs.python3.withPackages(ps: (with artiqpkgs; [ migen migen-axi misoc artiq ])))
-      vivado
-      artiqpkgs.binutils-arm
-
-      (import "${zynq-rs}/nix/mkbootimage.nix" { inherit pkgs; })
-    ];
-
-    XARGO_RUST_SRC = "${rustPlatform.rust.rustc.src}/src";
-    OPENOCD_ZYNQ = "${zynq-rs}/openocd";
-    SZL = "${zc706-szl}/szl.elf";
-  }

src/.clang-format

@@ -0,0 +1,32 @@
+BasedOnStyle: LLVM
+
+Language: Cpp
+Standard: Cpp11
+
+AccessModifierOffset: -1
+AlignEscapedNewlines: Left
+AlwaysBreakAfterReturnType: None
+AlwaysBreakTemplateDeclarations: Yes
+AllowAllParametersOfDeclarationOnNextLine: false
+AllowShortFunctionsOnASingleLine: Inline
+BinPackParameters: false
+BreakBeforeBinaryOperators: NonAssignment
+BreakBeforeTernaryOperators: true
+BreakConstructorInitializers: AfterColon
+BreakInheritanceList: AfterColon
+ColumnLimit: 120
+ConstructorInitializerAllOnOneLineOrOnePerLine: true
+ContinuationIndentWidth: 4
+DerivePointerAlignment: false
+IndentCaseLabels: true
+IndentPPDirectives: None
+IndentWidth: 4
+MaxEmptyLinesToKeep: 1
+PointerAlignment: Left
+ReflowComments: true
+SortIncludes: false
+SortUsingDeclarations: true
+SpaceAfterTemplateKeyword: false
+SpacesBeforeTrailingComments: 2
+TabWidth: 4
+UseTab: Never

src/.clippy.toml

@@ -0,0 +1 @@
+doc-valid-idents = ["CPython", "NumPy", ".."]

src/.pre-commit-config.yaml

@@ -0,0 +1,32 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+
+default_stages: [commit]
+
+repos:
+  - repo: local
+    hooks:
+      - id: cargo-fmt
+        name: artiq-zynq cargo format
+        entry: nix
+        language: system
+        types: [file, rust]
+        pass_filenames: false
+        description: Runs cargo fmt on the codebase.
+        args: [develop, -c, cargo, fmt, --manifest-path, src/Cargo.toml, --all]
+      - id: cargo-clippy
+        name: artiq-zynq cargo clippy
+        entry: nix
+        language: system
+        types: [file, rust]
+        pass_filenames: false
+        description: Runs cargo clippy on the codebase.
+        args: [develop, -c, cargo, clippy, --manifest-path, src/Cargo.toml, --tests]
+  - repo: https://github.com/pre-commit/mirrors-clang-format
+    rev: v19.1.0
+    hooks:
+      - id: clang-format
+        name: artiq-zynq clang-format
+        description: Runs clang-format on the codebase.
+        files: \.(cpp|h|hpp|c)$
+        args: [-style=file, -fallback-style=none, -assume-filename=src/.clang-format]

src/Cargo.lock

@@ -1,10 +1,27 @@
 # This file is automatically @generated by Cargo.
 # It is not intended for manual editing.
+version = 3
+
+[[package]]
+name = "approx"
+version = "0.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cab112f0a86d568ea0e627cc1d6be74a1e9cd55214684db5561995f6dad897c6"
+dependencies = [
+ "num-traits",
+]
+
+[[package]]
+name = "arrayvec"
+version = "0.7.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "96d30a06541fbafbc7f82ed10c06164cfbd2c401138f6addd8404629c4b16711"
+
 [[package]]
 name = "async-recursion"
-version = "0.3.1"
+version = "0.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e5444eec77a9ec2bfe4524139e09195862e981400c4358d3b760cae634e4c4ee"
+checksum = "d7d78656ba01f1b93024b7c3a0467f1608e4be67d725749fdcd7d2c7678fd7a2"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -13,33 +30,43 @@ dependencies = [
 
 [[package]]
 name = "autocfg"
-version = "1.0.0"
+version = "1.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
+checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
 
 [[package]]
 name = "bit_field"
-version = "0.10.0"
+version = "0.10.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a165d606cf084741d4ac3a28fb6e9b1eb0bd31f6cd999098cfddb0b2ab381dc0"
+checksum = "dcb6dd1c2376d2e096796e234a70e17e94cc2d5d54ff8ce42b28cef1d0d359a4"
 
 [[package]]
 name = "bitflags"
-version = "1.2.1"
+version = "1.3.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
+checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+
+[[package]]
+name = "build_const"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b4ae4235e6dac0694637c763029ecea1a2ec9e4e06ec2729bd21ba4d9c863eb7"
+
+[[package]]
+name = "build_zynq"
+version = "0.0.0"
 
 [[package]]
 name = "byteorder"
-version = "1.3.4"
+version = "1.4.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
+checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
 
 [[package]]
 name = "cc"
-version = "1.0.58"
+version = "1.0.77"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f9a06fb2e53271d7c279ec1efea6ab691c35a2ae67ec0d91d7acec0caf13b518"
+checksum = "e9f73505338f7d905b19d18738976aae232eb46b8efc15554ffc56deb5d9ebe4"
 
 [[package]]
 name = "cfg-if"
@@ -47,20 +74,34 @@ version = "0.1.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
 
+[[package]]
+name = "cfg-if"
+version = "1.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+
 [[package]]
 name = "compiler_builtins"
-version = "0.1.33"
-source = "git+https://git.m-labs.hk/M-Labs/compiler-builtins-zynq.git#62a622ba62c671aaadce7c108854551086df41f8"
-dependencies = [
- "cc",
-]
+version = "0.1.39"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3748f82c7d366a0b4950257d19db685d4958d2fa27c6d164a3f069fec42b748b"
 
 [[package]]
 name = "core_io"
-version = "0.1.20200410"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
+version = "0.1.20210325"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "97f8932064288cc79feb4d343a399d353a6f6f001e586ece47fe518a9e8507df"
 dependencies = [
- "memchr",
+ "rustc_version",
+]
+
+[[package]]
+name = "crc"
+version = "1.8.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d663548de7f5cca343f1e0a48d14dcfb0e9eb4e079ec58883b7251539fa10aeb"
+dependencies = [
+ "build_const",
 ]
 
 [[package]]
@@ -73,8 +114,9 @@ checksum = "0f8cb7306107e4b10e64994de6d3274bd08996a7c1322a27b86482392f96be0a"
 name = "dwarf"
 version = "0.0.0"
 dependencies = [
- "cfg-if",
+ "cfg-if 0.1.10",
  "compiler_builtins",
+ "cslice",
  "libc",
  "unwind",
 ]
@@ -89,9 +131,9 @@ dependencies = [
 
 [[package]]
 name = "embedded-hal"
-version = "0.2.4"
+version = "0.2.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fa998ce59ec9765d15216393af37a58961ddcefb14c753b4816ba2191d865fcb"
+checksum = "35949884794ad573cf46071e41c9b60efb0cb311e3ca01f7af807af1debc66ff"
 dependencies = [
  "nb 0.1.3",
  "void",
@@ -99,9 +141,9 @@ dependencies = [
 
 [[package]]
 name = "fatfs"
-version = "0.3.4"
+version = "0.3.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "93079df23039e52059e1f03b4c29fb0c72da2c792aad91bb2236c9fb81d3592e"
+checksum = "e18f80a87439240dac45d927fd8f8081b6f1e34c03e97271189fa8a8c2e96c8f"
 dependencies = [
  "bitflags",
  "byteorder",
@@ -111,9 +153,9 @@ dependencies = [
 
 [[package]]
 name = "futures"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1e05b85ec287aac0dc34db7d4a569323df697f9c55b99b15d6b4ef8cde49f613"
+checksum = "38390104763dc37a5145a53c29c63c1290b5d316d6086ec32c293f6736051bb0"
 dependencies = [
  "futures-channel",
  "futures-core",
@@ -125,9 +167,9 @@ dependencies = [
 
 [[package]]
 name = "futures-channel"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f366ad74c28cca6ba456d95e6422883cfb4b252a83bed929c83abfdbbf2967d5"
+checksum = "52ba265a92256105f45b719605a571ffe2d1f0fea3807304b522c1d778f79eed"
 dependencies = [
  "futures-core",
  "futures-sink",
@@ -135,23 +177,22 @@ dependencies = [
 
 [[package]]
 name = "futures-core"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "59f5fff90fd5d971f936ad674802482ba441b6f09ba5e15fd8b39145582ca399"
+checksum = "04909a7a7e4633ae6c4a9ab280aeb86da1236243a77b694a49eacd659a4bd3ac"
 
 [[package]]
 name = "futures-io"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "de27142b013a8e869c14957e6d2edeef89e97c289e69d042ee3a49acd8b51789"
+checksum = "00f5fb52a06bdcadeb54e8d3671f8888a39697dcb0b81b23b55174030427f4eb"
 
 [[package]]
 name = "futures-macro"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d0b5a30a4328ab5473878237c447333c093297bded83a4983d10f4deea240d39"
+checksum = "bdfb8ce053d86b91919aad980c220b1fb8401a9394410e1c289ed7e66b61835d"
 dependencies = [
- "proc-macro-hack",
  "proc-macro2",
  "quote",
  "syn",
@@ -159,51 +200,107 @@ dependencies = [
 
 [[package]]
 name = "futures-sink"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3f2032893cb734c7a05d85ce0cc8b8c4075278e93b24b66f9de99d6eb0fa8acc"
+checksum = "39c15cf1a4aa79df40f1bb462fb39676d0ad9e366c2a33b590d7c66f4f81fcf9"
 
 [[package]]
 name = "futures-task"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bdb66b5f09e22019b1ab0830f7785bcea8e7a42148683f99214f73f8ec21a626"
+checksum = "2ffb393ac5d9a6eaa9d3fdf37ae2776656b706e200c8e16b1bdb227f5198e6ea"
 
 [[package]]
 name = "futures-util"
-version = "0.3.5"
+version = "0.3.25"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8764574ff08b701a084482c3c7031349104b07ac897393010494beaa18ce32c6"
+checksum = "197676987abd2f9cadff84926f410af1c183608d36641465df73ae8211dc65d6"
 dependencies = [
  "futures-core",
  "futures-macro",
  "futures-sink",
  "futures-task",
- "pin-project",
+ "pin-project-lite",
  "pin-utils",
- "proc-macro-hack",
- "proc-macro-nested",
+]
+
+[[package]]
+name = "io"
+version = "0.0.0"
+dependencies = [
+ "byteorder",
+ "core_io",
+ "libsupport_zynq",
+]
+
+[[package]]
+name = "ksupport"
+version = "0.1.0"
+dependencies = [
+ "build_zynq",
+ "byteorder",
+ "core_io",
+ "cslice",
+ "dwarf",
+ "dyld",
+ "io",
+ "libasync",
+ "libboard_artiq",
+ "libboard_zynq",
+ "libc",
+ "libconfig",
+ "libcortex_a9",
+ "libm",
+ "libregister",
+ "libsupport_zynq",
+ "log",
+ "log_buffer",
+ "nalgebra",
+ "nb 0.1.3",
+ "unwind",
+ "vcell",
+ "void",
 ]
 
 [[package]]
 name = "libasync"
 version = "0.0.0"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
 dependencies = [
  "embedded-hal",
  "libcortex_a9",
- "nb 0.1.3",
+ "nb 1.0.0",
  "pin-utils",
  "smoltcp",
 ]
 
+[[package]]
+name = "libboard_artiq"
+version = "0.0.0"
+dependencies = [
+ "build_zynq",
+ "core_io",
+ "crc",
+ "embedded-hal",
+ "io",
+ "libasync",
+ "libboard_zynq",
+ "libconfig",
+ "libcortex_a9",
+ "libregister",
+ "libsupport_zynq",
+ "log",
+ "log_buffer",
+ "nb 1.0.0",
+ "void",
+]
+
 [[package]]
 name = "libboard_zynq"
 version = "0.0.0"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
 dependencies = [
  "bit_field",
  "embedded-hal",
+ "libasync",
  "libcortex_a9",
  "libregister",
  "log",
@@ -224,7 +321,6 @@ dependencies = [
 [[package]]
 name = "libconfig"
 version = "0.1.0"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
 dependencies = [
  "core_io",
  "fatfs",
@@ -235,7 +331,6 @@ dependencies = [
 [[package]]
 name = "libcortex_a9"
 version = "0.0.0"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
 dependencies = [
  "bit_field",
  "libregister",
@@ -244,14 +339,13 @@ dependencies = [
 
 [[package]]
 name = "libm"
-version = "0.2.1"
+version = "0.2.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c7d73b3f436185384286bd8098d17ec07c9a7d2388a6599f824d8502b529702a"
+checksum = "348108ab3fba42ec82ff6e9564fc4ca0247bdccdc68dd8af9764bbc79c3c8ffb"
 
 [[package]]
 name = "libregister"
 version = "0.0.0"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
 dependencies = [
  "bit_field",
  "vcell",
@@ -261,8 +355,8 @@ dependencies = [
 [[package]]
 name = "libsupport_zynq"
 version = "0.0.0"
-source = "git+https://git.m-labs.hk/M-Labs/zynq-rs.git#7360984efbd772ae992ef00af09786b0ae8430f0"
 dependencies = [
+ "cc",
  "compiler_builtins",
  "libboard_zynq",
  "libcortex_a9",
@@ -273,17 +367,17 @@ dependencies = [
 
 [[package]]
 name = "linked_list_allocator"
-version = "0.8.5"
+version = "0.8.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "660b26e6156a7d00eefb19052fe1943cf5ab2f353a723a577fad6ba2f99d1f90"
+checksum = "822add9edb1860698b79522510da17bef885171f75aa395cff099d770c609c24"
 
 [[package]]
 name = "log"
-version = "0.4.11"
+version = "0.4.17"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4fabed175da42fed1fa0746b0ea71f412aa9d35e76e95e59b192c64b9dc2bf8b"
+checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
 dependencies = [
- "cfg-if",
+ "cfg-if 1.0.0",
 ]
 
 [[package]]
@@ -299,10 +393,17 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "c75de51135344a4f8ed3cfe2720dc27736f7711989703a0b43aadf3753c55577"
 
 [[package]]
-name = "memchr"
-version = "2.3.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "3728d817d99e5ac407411fa471ff9800a778d88a24685968b36824eaf4bee400"
+name = "nalgebra"
+version = "0.32.6"
+source = "git+https://git.m-labs.hk/M-Labs/nalgebra.git?rev=dd00f9b#dd00f9b46046e0b931d1b470166db02fd29591be"
+dependencies = [
+ "approx",
+ "num-complex",
+ "num-rational",
+ "num-traits",
+ "simba",
+ "typenum",
+]
 
 [[package]]
 name = "nb"
@@ -320,44 +421,66 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "546c37ac5d9e56f55e73b677106873d9d9f5190605e41a856503623648488cae"
 
 [[package]]
-name = "num-derive"
-version = "0.3.1"
+name = "num-complex"
+version = "0.4.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e0396233fb2d5b0ae3f05ff6aba9a09185f7f6e70f87fb01147d545f85364665"
+checksum = "26873667bbbb7c5182d4a37c1add32cdf09f841af72da53318fdb81543c15085"
+dependencies = [
+ "num-traits",
+]
+
+[[package]]
+name = "num-derive"
+version = "0.3.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "876a53fff98e03a936a674b29568b0e605f06b29372c2489ff4de23f1949743d"
 dependencies = [
  "proc-macro2",
  "quote",
  "syn",
 ]
 
+[[package]]
+name = "num-integer"
+version = "0.1.46"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f"
+dependencies = [
+ "num-traits",
+]
+
+[[package]]
+name = "num-rational"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d41702bd167c2df5520b384281bc111a4b5efcf7fbc4c9c222c815b07e0a6a6a"
+dependencies = [
+ "autocfg",
+ "num-integer",
+ "num-traits",
+]
+
 [[package]]
 name = "num-traits"
-version = "0.2.12"
+version = "0.2.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ac267bcc07f48ee5f8935ab0d24f316fb722d7a1292e2913f0cc196b29ffd611"
+checksum = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd"
 dependencies = [
  "autocfg",
+ "libm",
 ]
 
 [[package]]
-name = "pin-project"
-version = "0.4.23"
+name = "paste"
+version = "1.0.15"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ca4433fff2ae79342e497d9f8ee990d174071408f28f726d6d83af93e58e48aa"
-dependencies = [
- "pin-project-internal",
-]
+checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
 
 [[package]]
-name = "pin-project-internal"
-version = "0.4.23"
+name = "pin-project-lite"
+version = "0.2.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "2c0e815c3ee9a031fdf5af21c10aa17c573c9c6a566328d99e3936c34e36461f"
-dependencies = [
- "proc-macro2",
- "quote",
- "syn",
-]
+checksum = "e0a7ae3ac2f1173085d398531c705756c94a4c56843785df85a60c1a0afac116"
 
 [[package]]
 name = "pin-utils"
@@ -365,32 +488,20 @@ version = "0.1.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
 
-[[package]]
-name = "proc-macro-hack"
-version = "0.5.18"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "99c605b9a0adc77b7211c6b1f722dcb613d68d66859a44f3d485a6da332b0598"
-
-[[package]]
-name = "proc-macro-nested"
-version = "0.1.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "eba180dafb9038b050a4c280019bbedf9f2467b61e5d892dcad585bb57aadc5a"
-
 [[package]]
 name = "proc-macro2"
-version = "1.0.19"
+version = "1.0.43"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04f5f085b5d71e2188cb8271e5da0161ad52c3f227a661a3c135fdf28e258b12"
+checksum = "0a2ca2c61bc9f3d74d2886294ab7b9853abd9c1ad903a3ac7815c58989bb7bab"
 dependencies = [
- "unicode-xid",
+ "unicode-ident",
 ]
 
 [[package]]
 name = "quote"
-version = "1.0.7"
+version = "1.0.21"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "aa563d17ecb180e500da1cfd2b028310ac758de548efdd203e18f283af693f37"
+checksum = "bbe448f377a7d6961e30f5955f9b8d106c3f5e449d493ee1b125c1d43c2b5179"
 dependencies = [
  "proc-macro2",
 ]
@@ -406,6 +517,7 @@ name = "runtime"
 version = "0.1.0"
 dependencies = [
  "async-recursion",
+ "build_zynq",
  "byteorder",
  "core_io",
  "cslice",
@@ -413,29 +525,82 @@ dependencies = [
  "dyld",
  "embedded-hal",
  "futures",
+ "io",
+ "ksupport",
  "libasync",
+ "libboard_artiq",
  "libboard_zynq",
  "libc",
  "libconfig",
  "libcortex_a9",
- "libm",
  "libregister",
  "libsupport_zynq",
  "log",
  "log_buffer",
- "nb 0.1.3",
  "num-derive",
  "num-traits",
+ "tar-no-std",
  "unwind",
  "vcell",
  "void",
 ]
 
 [[package]]
-name = "smoltcp"
-version = "0.6.0"
+name = "rustc_version"
+version = "0.1.7"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0fe46639fd2ec79eadf8fe719f237a7a0bd4dac5d957f1ca5bbdbc1c3c39e53a"
+checksum = "c5f5376ea5e30ce23c03eb77cbe4962b988deead10910c372b226388b594c084"
+dependencies = [
+ "semver",
+]
+
+[[package]]
+name = "satman"
+version = "0.0.0"
+dependencies = [
+ "build_zynq",
+ "byteorder",
+ "core_io",
+ "crc",
+ "cslice",
+ "embedded-hal",
+ "io",
+ "ksupport",
+ "libasync",
+ "libboard_artiq",
+ "libboard_zynq",
+ "libc",
+ "libconfig",
+ "libcortex_a9",
+ "libregister",
+ "libsupport_zynq",
+ "log",
+ "unwind",
+]
+
+[[package]]
+name = "semver"
+version = "0.1.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d4f410fedcf71af0345d7607d246e7ad15faaadd49d240ee3b24e5dc21a820ac"
+
+[[package]]
+name = "simba"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "50582927ed6f77e4ac020c057f37a268fc6aebc29225050365aacbb9deeeddc4"
+dependencies = [
+ "approx",
+ "num-complex",
+ "num-traits",
+ "paste",
+]
+
+[[package]]
+name = "smoltcp"
+version = "0.7.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3e4a069bef843d170df47e7c0a8bf8d037f217d9f5b325865acc3e466ffe40d3"
 dependencies = [
  "bitflags",
  "byteorder",
@@ -444,36 +609,52 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "1.0.38"
+version = "1.0.101"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e69abc24912995b3038597a7a593be5053eb0fb44f3cc5beec0deb421790c1f4"
+checksum = "e90cde112c4b9690b8cbe810cba9ddd8bc1d7472e2cae317b69e9438c1cba7d2"
 dependencies = [
  "proc-macro2",
  "quote",
- "unicode-xid",
+ "unicode-ident",
 ]
 
 [[package]]
-name = "unicode-xid"
-version = "0.2.1"
+name = "tar-no-std"
+version = "0.1.8"
+source = "git+https://git.m-labs.hk/M-Labs/tar-no-std?rev=2ab6dc5#2ab6dc58e5249c59c4eb03eaf3a119bcdd678d32"
+dependencies = [
+ "arrayvec",
+ "bitflags",
+ "log",
+]
+
+[[package]]
+name = "typenum"
+version = "1.17.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f7fe0bb3479651439c9112f72b6c505038574c9fbb575ed1bf3b797fa39dd564"
+checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825"
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ceab39d59e4c9499d4e5a8ee0e2735b891bb7308ac83dfb4e80cad195c9f6f3"
 
 [[package]]
 name = "unwind"
 version = "0.0.0"
 dependencies = [
  "cc",
- "cfg-if",
+ "cfg-if 0.1.10",
  "compiler_builtins",
  "libc",
 ]
 
 [[package]]
 name = "vcell"
-version = "0.1.2"
+version = "0.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "876e32dcadfe563a4289e994f7cb391197f362b6315dc45e8ba4aa6f564a4b3c"
+checksum = "77439c1b53d2303b20d9459b1ade71a83c716e3f9c34f3228c00e6f185d6c002"
 
 [[package]]
 name = "void"
@@ -483,9 +664,9 @@ checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d"
 
 [[package]]
 name = "volatile-register"
-version = "0.2.0"
+version = "0.2.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0d67cb4616d99b940db1d6bd28844ff97108b498a6ca850e5b6191a532063286"
+checksum = "9ee8f19f9d74293faf70901bc20ad067dc1ad390d2cbf1e3f75f721ffee908b6"
 dependencies = [
  "vcell",
 ]

src/Cargo.toml

@@ -3,8 +3,11 @@ members = [
   "libc",
   "libdyld",
   "libdwarf",
+  "libio",
   "libunwind",
+  "libksupport",
   "runtime",
+  "satman"
 ]
 
 [profile.release]
@@ -13,7 +16,3 @@ debug = true
 codegen-units = 1
 opt-level = 2
 lto = true
-
-[patch.crates-io]
-core_io = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
-compiler_builtins = { git = "https://git.m-labs.hk/M-Labs/compiler-builtins-zynq.git"}

src/Makefile

@@ -1,16 +1,41 @@
-VARIANT := simple
+TARGET := zc706
+GWARGS := -V nist_clock
 
-all: ../build/firmware/armv7-none-eabihf/release/runtime ../build/runtime.bin
+all: runtime
 
-.PHONY: all
+runtime: ../build/runtime.bin
 
+satman: ../build/satman.bin
 
-../build/pl.rs ../build/rustc-cfg: gateware/*
+.PHONY: all manifests
+
+manifests = libboard_artiq/Cargo.toml libc/Cargo.toml libdyld/Cargo.toml libio/Cargo.toml libksupport/Cargo.toml runtime/Cargo.toml satman/Cargo.toml
+
+$(manifests): %.toml: %.toml.tpl
+	sed s+@@ZYNQ_RS@@+$(ZYNQ_RS)+g $< > $@
+
+manifests: $(manifests)
+
+../build/pl.rs ../build/rustc-cfg ../build/mem.rs: gateware/* 
 	mkdir -p ../build
-	python gateware/zc706.py -r ../build/pl.rs -c ../build/rustc-cfg -V $(VARIANT)
+	python gateware/$(TARGET).py -r ../build/pl.rs -c ../build/rustc-cfg -m ../build/mem.rs $(GWARGS)
 
-../build/firmware/armv7-none-eabihf/release/runtime: ../build/pl.rs ../build/rustc-cfg $(shell find . -path ./szl -prune -o -print)
-	XBUILD_SYSROOT_PATH=`pwd`/../build/sysroot cargo xbuild --release -p runtime --target-dir ../build/firmware
+../build/firmware/armv7-none-eabihf/release/runtime: ../build/pl.rs ../build/rustc-cfg ../build/mem.rs $(manifests) $(shell find . -type f -not -name Cargo.toml -print)
+	cd runtime && \
+	XBUILD_SYSROOT_PATH=`pwd`/../../build/sysroot \
+	cargo xbuild --release \
+		--target-dir ../../build/firmware \
+		--no-default-features --features=target_$(TARGET)
 
 ../build/runtime.bin: ../build/firmware/armv7-none-eabihf/release/runtime
 	llvm-objcopy -O binary ../build/firmware/armv7-none-eabihf/release/runtime ../build/runtime.bin
+
+../build/firmware/armv7-none-eabihf/release/satman: ../build/pl.rs ../build/rustc-cfg ../build/mem.rs $(manifests) $(shell find . -type f -not -name Cargo.toml -print)
+	cd satman && \
+	XBUILD_SYSROOT_PATH=`pwd`/../../build/sysroot \
+	cargo xbuild --release \
+		--target-dir ../../build/firmware \
+		--no-default-features --features=target_$(TARGET)
+
+../build/satman.bin: ../build/firmware/armv7-none-eabihf/release/satman
+	llvm-objcopy -O binary ../build/firmware/armv7-none-eabihf/release/satman ../build/satman.bin

src/gateware/config.py

@@ -0,0 +1,16 @@
+from misoc.integration import cpu_interface
+
+def write_csr_file(soc, filename):
+    with open(filename, "w") as f:
+        f.write(cpu_interface.get_csr_rust(
+            soc.get_csr_regions(), soc.get_csr_groups(), soc.get_constants()))
+
+def write_mem_file(soc, filename):
+    with open(filename, "w") as f:
+        f.write(cpu_interface.get_mem_rust(
+            soc.get_memory_regions(), soc.get_memory_groups(), None))
+
+def write_rustc_cfg_file(soc, filename):
+    with open(filename, "w") as f:
+        f.write(cpu_interface.get_rust_cfg(
+            soc.get_csr_regions(), soc.get_constants()))

src/gateware/ddmtd.py

@@ -0,0 +1,119 @@
+from migen import *
+from migen.genlib.cdc import PulseSynchronizer, MultiReg
+from misoc.interconnect.csr import *
+
+
+class DDMTDSampler(Module):
+    def __init__(self, cd_ref, main_clk_se):
+        self.ref_beating = Signal()
+        self.main_beating = Signal()
+
+        # # #
+
+        ref_clk = Signal()
+        self.specials +=[
+            # ISERDESE2 can only be driven from fabric via IDELAYE2 (see UG471)
+            Instance("IDELAYE2",
+                    p_DELAY_SRC="DATAIN",
+                    p_HIGH_PERFORMANCE_MODE="TRUE",
+                    p_REFCLK_FREQUENCY=208.3,   # REFCLK frequency from IDELAYCTRL
+                    p_IDELAY_VALUE=0,
+
+                    i_DATAIN=cd_ref.clk,
+
+                    o_DATAOUT=ref_clk
+            ),
+            Instance("ISERDESE2",
+                    p_IOBDELAY="IFD",   # use DDLY as input
+                    p_DATA_RATE="SDR",
+                    p_DATA_WIDTH=2,     # min is 2
+                    p_NUM_CE=1,
+
+                    i_DDLY=ref_clk,
+                    i_CE1=1,
+                    i_CLK=ClockSignal("helper"),
+                    i_CLKDIV=ClockSignal("helper"),
+
+                    o_Q1=self.ref_beating
+            ),
+            Instance("ISERDESE2",
+                    p_DATA_RATE="SDR",
+                    p_DATA_WIDTH=2,     # min is 2
+                    p_NUM_CE=1,
+
+                    i_D=main_clk_se,
+                    i_CE1=1,
+                    i_CLK=ClockSignal("helper"),
+                    i_CLKDIV=ClockSignal("helper"),
+
+                    o_Q1=self.main_beating,
+            ),
+        ]
+
+
+class DDMTDDeglitcherMedianEdge(Module):
+    def __init__(self, counter, input_signal, stable_0_period=100, stable_1_period=100):
+        self.tag = Signal(len(counter))
+        self.detect = Signal()
+
+        stable_0_counter = Signal(reset=stable_0_period - 1, max=stable_0_period)
+        stable_1_counter = Signal(reset=stable_1_period - 1, max=stable_1_period)
+
+        # # #
+
+        # Based on CERN's median edge deglitcher FSM
+        # https://white-rabbit.web.cern.ch/documents/Precise_time_and_frequency_transfer_in_a_White_Rabbit_network.pdf (p.72)
+        fsm = ClockDomainsRenamer("helper")(FSM(reset_state="WAIT_STABLE_0"))
+        self.submodules += fsm
+
+        fsm.act("WAIT_STABLE_0",
+            If(stable_0_counter != 0,
+                NextValue(stable_0_counter, stable_0_counter - 1)
+            ).Else(
+                NextValue(stable_0_counter, stable_0_period - 1),
+                NextState("WAIT_EDGE")
+            ),
+            If(input_signal,
+                NextValue(stable_0_counter, stable_0_period - 1)
+            ),
+        )
+        fsm.act("WAIT_EDGE",
+            If(input_signal,
+                NextValue(self.tag, counter),
+                NextState("GOT_EDGE")
+            )
+        )
+        fsm.act("GOT_EDGE",
+            If(stable_1_counter != 0,
+                NextValue(stable_1_counter, stable_1_counter - 1)
+            ).Else(
+                NextValue(stable_1_counter, stable_1_period - 1),
+                self.detect.eq(1),
+                NextState("WAIT_STABLE_0")
+            ),
+            If(~input_signal,
+                NextValue(self.tag, self.tag + 1),
+                NextValue(stable_1_counter, stable_1_period - 1)
+            ),
+        )
+        
+
+class DDMTD(Module):
+    def __init__(self, counter, input_signal):
+
+        # in helper clock domain
+        self.h_tag = Signal(len(counter))
+        self.h_tag_update = Signal()
+
+        # # #
+
+        deglitcher = DDMTDDeglitcherMedianEdge(counter, input_signal)
+        self.submodules += deglitcher
+
+        self.sync.helper += [
+            self.h_tag_update.eq(0),
+            If(deglitcher.detect,
+                self.h_tag_update.eq(1),
+                self.h_tag.eq(deglitcher.tag)
+               )
+        ]

src/gateware/drtio_aux_controller.py

@@ -0,0 +1,85 @@
+"""Auxiliary controller, common to satellite and master"""
+
+from artiq.gateware.drtio.aux_controller import (max_packet, aux_buffer_count,
+    Transmitter, Receiver)
+from migen.fhdl.simplify import FullMemoryWE
+from misoc.interconnect.csr import *
+from migen_axi.interconnect.sram import SRAM
+from migen_axi.interconnect import axi
+
+
+class _DRTIOAuxControllerBase(Module):
+    def __init__(self, link_layer):
+        self.bus = axi.Interface()
+        self.submodules.transmitter = Transmitter(link_layer, len(self.bus.w.data))
+        self.submodules.receiver = Receiver(link_layer, len(self.bus.w.data))
+
+    def get_csrs(self):
+        return self.transmitter.get_csrs() + self.receiver.get_csrs()
+
+
+# TODO: FullMemoryWE should be applied by migen.build
+@FullMemoryWE()
+class DRTIOAuxControllerAxi(_DRTIOAuxControllerBase):
+    def __init__(self, link_layer):
+        _DRTIOAuxControllerBase.__init__(self, link_layer)
+
+        tx_sdram_if = SRAM(self.transmitter.mem, read_only=False)
+        rx_sdram_if = SRAM(self.receiver.mem, read_only=True)
+        aw_decoder = axi.AddressDecoder(self.bus.aw,
+            [(lambda a: a[log2_int(max_packet*aux_buffer_count)] == 0, tx_sdram_if.bus.aw),
+             (lambda a: a[log2_int(max_packet*aux_buffer_count)] == 1, rx_sdram_if.bus.aw)],
+            register=True)
+        ar_decoder = axi.AddressDecoder(self.bus.ar,
+            [(lambda a: a[log2_int(max_packet*aux_buffer_count)] == 0, tx_sdram_if.bus.ar),
+             (lambda a: a[log2_int(max_packet*aux_buffer_count)] == 1, rx_sdram_if.bus.ar)],
+            register=True)
+        # unlike wb, axi address decoder only connects ar/aw lanes,
+        # the rest must also be connected!
+        # not quite unlike an address decoder itself.
+
+        # connect bus.b with tx.b
+        self.comb += [tx_sdram_if.bus.b.ready.eq(self.bus.b.ready),
+                      self.bus.b.id.eq(tx_sdram_if.bus.b.id),
+                      self.bus.b.resp.eq(tx_sdram_if.bus.b.resp),
+                      self.bus.b.valid.eq(tx_sdram_if.bus.b.valid)]
+        # connect bus.w with tx.w
+        # no worries about w.valid and slave sel here, only tx will be written to
+        self.comb += [tx_sdram_if.bus.w.id.eq(self.bus.w.id),
+                      tx_sdram_if.bus.w.data.eq(self.bus.w.data),
+                      tx_sdram_if.bus.w.strb.eq(self.bus.w.strb),
+                      tx_sdram_if.bus.w.last.eq(self.bus.w.last),
+                      tx_sdram_if.bus.w.valid.eq(self.bus.w.valid),
+                      self.bus.w.ready.eq(tx_sdram_if.bus.w.ready)]
+        # connect bus.r with rx.r and tx.r w/o data
+        self.comb += [self.bus.r.id.eq(rx_sdram_if.bus.r.id | tx_sdram_if.bus.r.id),
+                #self.bus.r.data.eq(rx_sdram_if.bus.r.data | tx_sdram_if.bus.r.data),
+                self.bus.r.resp.eq(rx_sdram_if.bus.r.resp | tx_sdram_if.bus.r.resp),
+                self.bus.r.last.eq(rx_sdram_if.bus.r.last | tx_sdram_if.bus.r.last),
+                self.bus.r.valid.eq(rx_sdram_if.bus.r.valid | tx_sdram_if.bus.r.valid),
+                rx_sdram_if.bus.r.ready.eq(self.bus.r.ready),
+                tx_sdram_if.bus.r.ready.eq(self.bus.r.ready)]
+        # connect read data after being masked
+        masked = [Replicate(rx_sdram_if.bus.r.valid,
+                            len(self.bus.r.data)
+                            ) & rx_sdram_if.bus.r.data,
+                 Replicate(tx_sdram_if.bus.r.valid,
+                            len(self.bus.r.data)
+                            ) & tx_sdram_if.bus.r.data]
+        self.comb += self.bus.r.data.eq(reduce(or_, masked))
+
+        self.submodules += tx_sdram_if, rx_sdram_if, aw_decoder, ar_decoder
+
+
+@FullMemoryWE()
+class DRTIOAuxControllerBare(_DRTIOAuxControllerBase):
+    # Barebones version of the AuxController. No SRAM, no decoders.
+    # add memories manually from tx and rx in target code.
+    def get_tx_port(self):
+        return self.transmitter.mem.get_port(write_capable=True)
+
+    def get_rx_port(self):
+        return self.receiver.mem.get_port(write_capable=False)
+
+    def get_mem_size(self):
+        return max_packet*aux_buffer_count

src/gateware/ebaz4205.py

@@ -0,0 +1,307 @@
+#!/usr/bin/env python
+
+import argparse
+
+import analyzer
+import dma
+from artiq.gateware import rtio
+from artiq.gateware.rtio.phy import spi2, ttl_simple
+from artiq.gateware.rtio.xilinx_clocking import fix_serdes_timing_path
+from config import write_csr_file, write_mem_file, write_rustc_cfg_file
+from migen import *
+from migen.build.generic_platform import IOStandard, Misc, Pins, Subsignal
+from migen.build.platforms import ebaz4205
+from migen_axi.integration.soc_core import SoCCore
+from misoc.interconnect.csr import *
+
+_ps = [
+    (
+        "ps",
+        0,
+        Subsignal("clk", Pins("E7"), IOStandard("LVCMOS33"), Misc("SLEW=FAST")),
+        Subsignal("por_b", Pins("C7"), IOStandard("LVCMOS33"), Misc("SLEW=FAST")),
+        Subsignal("srst_b", Pins("B10"), IOStandard("LVCMOS18"), Misc("SLEW=FAST")),
+    )
+]
+
+_ddr = [
+    (
+        "ddr",
+        0,
+        Subsignal(
+            "a",
+            Pins("N2 K2 M3 K3 M4 L1 L4 K4 K1 J4 F5 G4 E4 D4 F4"),
+            IOStandard("SSTL15"),
+        ),
+        Subsignal("ba", Pins("L5 R4 J5"), IOStandard("SSTL15")),
+        Subsignal("cas_n", Pins("P5"), IOStandard("SSTL15")),
+        Subsignal("cke", Pins("N3"), IOStandard("SSTL15")),
+        Subsignal("cs_n", Pins("N1"), IOStandard("SSTL15")),
+        Subsignal("ck_n", Pins("M2"), IOStandard("DIFF_SSTL15"), Misc("SLEW=FAST")),
+        Subsignal("ck_p", Pins("L2"), IOStandard("DIFF_SSTL15"), Misc("SLEW=FAST")),
+        # Pins "T1 Y1" not connected
+        Subsignal("dm", Pins("A1 F1"), IOStandard("SSTL15_T_DCI"), Misc("SLEW=FAST")),
+        Subsignal(
+            "dq",
+            Pins("C3 B3 A2 A4 D3 D1 C1 E1 E2 E3 G3 H3 J3 H2 H1 J1"),
+            # Pins "P1 P3 R3 R1 T4 U4 U2 U3 V1 Y3 W1 Y4 Y2 W3 V2 V3" not connected
+            IOStandard("SSTL15_T_DCI"),
+            Misc("SLEW=FAST"),
+        ),
+        Subsignal(
+            "dqs_n",
+            Pins("B2 F2"),  # Pins "T2 W4" not connected
+            IOStandard("DIFF_SSTL15_T_DCI"),
+            Misc("SLEW=FAST"),
+        ),
+        Subsignal(
+            "dqs_p",
+            Pins("C2 G2"),  # Pins "R2 W5" not connected
+            IOStandard("DIFF_SSTL15_T_DCI"),
+            Misc("SLEW=FAST"),
+        ),
+        Subsignal("vrn", Pins("G5"), IOStandard("SSTL15_T_DCI"), Misc("SLEW=FAST")),
+        Subsignal("vrp", Pins("H5"), IOStandard("SSTL15_T_DCI"), Misc("SLEW=FAST")),
+        Subsignal("drst_n", Pins("B4"), IOStandard("SSTL15"), Misc("SLEW=FAST")),
+        Subsignal("odt", Pins("N5"), IOStandard("SSTL15")),
+        Subsignal("ras_n", Pins("P4"), IOStandard("SSTL15")),
+        Subsignal("we_n", Pins("M5"), IOStandard("SSTL15")),
+    )
+]
+
+# Connector J3
+_i2c = [
+    (
+        "i2c",
+        0,
+        Subsignal("scl", Pins("U12"), IOStandard("LVCMOS33")),
+        Subsignal("sda", Pins("V13"), IOStandard("LVCMOS33")),
+    )
+]
+
+_spi = [
+    (
+        "spi",
+        0,
+        Subsignal("clk", Pins("V20")),
+        Subsignal("mosi", Pins("U20")),
+        Subsignal("cs_n", Pins("P19")),
+        IOStandard("LVCMOS33"),
+    )
+]
+
+
+# Connector DATA1
+def _create_ttl():
+    _ttl = []
+
+    for idx, elem in enumerate([x for x in range(5, 21) if x not in (10, 12)]):
+        _ttl.append(
+            ("ttl", idx, Pins("DATA1:DATA1-{}".format(elem)), IOStandard("LVCMOS33")),
+        )
+    return _ttl
+
+
+class EBAZ4205(SoCCore):
+    def __init__(self, rtio_clk=125e6, acpki=False):
+        self.acpki = acpki
+
+        platform = ebaz4205.Platform()
+        platform.toolchain.bitstream_commands.extend(
+            [
+                "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
+            ]
+        )
+        platform.add_extension(_ps)
+        platform.add_extension(_ddr)
+        platform.add_extension(_i2c)
+        platform.add_extension(_spi)
+        platform.add_extension(_create_ttl())
+
+        gmii = platform.request("gmii")
+        platform.add_period_constraint(gmii.rx_clk, 10)
+        platform.add_period_constraint(gmii.tx_clk, 10)
+        platform.add_platform_command(
+            "set_property CLOCK_DEDICATED_ROUTE FALSE [get_nets gmii_tx_clk_IBUF]"
+        )
+
+        ident = self.__class__.__name__
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident)
+        fix_serdes_timing_path(platform)
+        self.config["RTIO_FREQUENCY"] = str(rtio_clk / 1e6)
+        platform.add_period_constraint(self.ps7.cd_sys.clk, 10)
+
+        self.comb += [
+            self.ps7.enet0.enet.gmii.tx_clk.eq(gmii.tx_clk),
+            self.ps7.enet0.enet.gmii.rx_clk.eq(gmii.rx_clk),
+        ]
+        self.clock_domains.cd_eth_rx = ClockDomain(reset_less=False)
+        self.clock_domains.cd_eth_tx = ClockDomain(reset_less=False)
+        self.comb += [
+            ClockSignal("eth_rx").eq(gmii.rx_clk),
+            ClockSignal("eth_tx").eq(gmii.tx_clk),
+        ]
+        self.sync.eth_tx += [
+            gmii.txd.eq(self.ps7.enet0.enet.gmii.txd),
+            gmii.tx_en.eq(self.ps7.enet0.enet.gmii.tx_en),
+        ]
+        self.sync.eth_rx += [
+            self.ps7.enet0.enet.gmii.rxd.eq(gmii.rxd),
+            self.ps7.enet0.enet.gmii.rx_dv.eq(gmii.rx_dv),
+        ]
+
+        # MDIO
+        mdio = platform.request("mdio")
+        self.comb += mdio.mdc.eq(self.ps7.enet0.enet.mdio.mdc)
+        self.specials += Instance(
+            "IOBUF",
+            i_I=self.ps7.enet0.enet.mdio.o,
+            io_IO=mdio.mdio,
+            o_O=self.ps7.enet0.enet.mdio.i,
+            i_T=~self.ps7.enet0.enet.mdio.t_n,
+        )
+
+        # I2C
+        i2c = self.platform.request("i2c")
+        self.specials += [
+            # SCL
+            Instance(
+                "IOBUF",
+                i_I=self.ps7.i2c0.scl.o,
+                io_IO=i2c.scl,
+                o_O=self.ps7.i2c0.scl.i,
+                i_T=~self.ps7.i2c0.scl.t_n,
+            ),
+            # SDA
+            Instance(
+                "IOBUF",
+                i_I=self.ps7.i2c0.sda.o,
+                io_IO=i2c.sda,
+                o_O=self.ps7.i2c0.sda.i,
+                i_T=~self.ps7.i2c0.sda.t_n,
+            ),
+        ]
+
+        self.rtio_channels = []
+        for i in (0, 1):
+            print("USER LED at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            user_led = self.platform.request("user_led", i)
+            phy = ttl_simple.Output(user_led)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        for i in range(14):
+            print("TTL at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            ttl = self.platform.request("ttl", i)
+            phy = ttl_simple.InOut(ttl)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        print("SPI at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+        spi_phy = spi2.SPIMaster(platform.request("spi"))
+        self.submodules += spi_phy
+        self.rtio_channels.append(rtio.Channel.from_phy(spi_phy, ififo_depth=4))
+        self.config["RTIO_LOG_CHANNEL"] = len(self.rtio_channels)
+        self.rtio_channels.append(rtio.LogChannel())
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+        self.submodules.rtio_core = rtio.Core(self.rtio_tsc, self.rtio_channels)
+        self.csr_devices.append("rtio_core")
+        if self.acpki:
+            import acpki
+
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(
+                self.rtio_tsc,
+                bus=self.ps7.s_axi_acp,
+                user=self.ps7.s_axi_acp_user,
+                evento=self.ps7.event.o,
+            )
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.rtio.cri, self.rtio_dma.cri],
+            [self.rtio_core.cri],
+            enable_routing=True,
+        )
+        self.csr_devices.append("cri_con")
+
+        self.submodules.rtio_moninj = rtio.MonInj(self.rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(
+            self.rtio_tsc, self.rtio_core.cri, self.ps7.s_axi_hp1
+        )
+        self.csr_devices.append("rtio_analyzer")
+
+
+class BASE(EBAZ4205):
+    def __init__(self, rtio_clk, acpki):
+        EBAZ4205.__init__(self, rtio_clk, acpki)
+
+
+VARIANTS = {cls.__name__.lower(): cls for cls in [BASE]}
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="ARTIQ port to the EBAZ4205 control card of Ebit E9+ BTC miner"
+    )
+    parser.add_argument(
+        "-r", default=None, help="build Rust interface into the specified file"
+    )
+    parser.add_argument(
+        "-m", default=None, help="build Rust memory interface into the specified file"
+    )
+    parser.add_argument(
+        "-c",
+        default=None,
+        help="build Rust compiler configuration into the specified file",
+    )
+    parser.add_argument(
+        "-g", default=None, help="build gateware into the specified directory"
+    )
+    parser.add_argument("--rtio-clk", default=125e6, help="RTIO Clock Frequency (Hz)")
+    parser.add_argument(
+        "-V",
+        "--variant",
+        default="base",
+        help="variant: " "[acpki_]base" "(default: %(default)s)",
+    )
+    args = parser.parse_args()
+
+    rtio_clk = int(args.rtio_clk)
+    variant = args.variant.lower()
+    acpki = variant.startswith("acpki_")
+    if acpki:
+        variant = variant[6:]
+
+    try:
+        cls = VARIANTS[variant]
+    except KeyError:
+        raise SystemExit("Invalid variant (-V/--variant)")
+
+    soc = cls(rtio_clk=rtio_clk, acpki=acpki)
+    soc.finalize()
+
+    if args.r is not None:
+        write_csr_file(soc, args.r)
+    if args.m is not None:
+        write_mem_file(soc, args.m)
+    if args.c is not None:
+        write_rustc_cfg_file(soc, args.c)
+    if args.g is not None:
+        soc.build(build_dir=args.g)
+
+
+if __name__ == "__main__":
+    main()

src/gateware/kasli_soc.py

@@ -0,0 +1,672 @@
+#!/usr/bin/env python
+
+import argparse
+from operator import itemgetter
+
+from migen import *
+from migen.build.generic_platform import *
+from migen.genlib.resetsync import AsyncResetSynchronizer
+from migen.genlib.cdc import MultiReg
+from migen_axi.integration.soc_core import SoCCore
+from migen_axi.platforms import kasli_soc
+from misoc.interconnect.csr import *
+from misoc.cores import virtual_leds
+
+from artiq.coredevice import jsondesc
+from artiq.gateware import rtio, eem_7series
+from artiq.gateware.rtio.xilinx_clocking import fix_serdes_timing_path
+from artiq.gateware.rtio.phy import ttl_simple
+from artiq.gateware.drtio.transceiver import gtx_7series, eem_serdes
+from artiq.gateware.drtio.siphaser import SiPhaser7Series
+from artiq.gateware.drtio.rx_synchronizer import XilinxRXSynchronizer
+from artiq.gateware.drtio import *
+from artiq.gateware.wrpll import wrpll
+
+import dma
+import analyzer
+import acpki
+import drtio_aux_controller
+import zynq_clocking
+from config import write_csr_file, write_mem_file, write_rustc_cfg_file
+
+eem_iostandard_dict = {
+     0: "LVDS_25",
+     1: "LVDS_25",
+     2: "LVDS",
+     3: "LVDS",
+     4: "LVDS",
+     5: "LVDS",
+     6: "LVDS",
+     7: "LVDS",
+     8: "LVDS_25",
+     9: "LVDS_25",
+    10: "LVDS",
+    11: "LVDS",
+}
+
+
+def eem_iostandard(eem):
+    return IOStandard(eem_iostandard_dict[eem])
+
+
+class SMAClkinForward(Module):
+    def __init__(self, platform):
+        sma_clkin = platform.request("sma_clkin")
+        sma_clkin_se = Signal()
+        cdr_clk_se = Signal()
+        cdr_clk = platform.request("cdr_clk")
+        self.specials += [
+            Instance("IBUFDS", i_I=sma_clkin.p, i_IB=sma_clkin.n, o_O=sma_clkin_se),
+            Instance("ODDR", i_C=sma_clkin_se, i_CE=1, i_D1=1, i_D2=0, o_Q=cdr_clk_se),
+            Instance("OBUFDS", i_I=cdr_clk_se, o_O=cdr_clk.p, o_OB=cdr_clk.n)
+        ]
+
+
+class GTPBootstrapClock(Module):
+    def __init__(self, platform, freq=125e6):
+        self.clock_domains.cd_bootstrap = ClockDomain(reset_less=True)
+        self.cd_bootstrap.clk.attr.add("keep")
+
+        bootstrap_125 = platform.request("clk125_gtp")
+        bootstrap_se = Signal()
+        clk_out = Signal()
+        platform.add_period_constraint(bootstrap_125.p, 8.0)
+        self.specials += [
+            Instance("IBUFDS_GTE2",
+                i_CEB=0,
+                i_I=bootstrap_125.p, i_IB=bootstrap_125.n, 
+                o_O=bootstrap_se,
+                p_CLKCM_CFG="TRUE",
+                p_CLKRCV_TRST="TRUE",
+                p_CLKSWING_CFG=3),
+            Instance("BUFG", i_I=bootstrap_se, o_O=clk_out)
+        ]
+        if freq == 125e6:
+            self.comb += self.cd_bootstrap.clk.eq(clk_out)
+        elif freq == 100e6:
+            pll_fb = Signal()
+            pll_out = Signal()
+            self.specials += [
+                Instance("PLLE2_BASE",
+                    p_CLKIN1_PERIOD=8.0,
+                    i_CLKIN1=clk_out,
+                    i_CLKFBIN=pll_fb,
+                    o_CLKFBOUT=pll_fb,
+
+                    # VCO @ 1GHz
+                    p_CLKFBOUT_MULT=8, p_DIVCLK_DIVIDE=1,
+
+                    # 100MHz for bootstrap
+                    p_CLKOUT1_DIVIDE=10, p_CLKOUT1_PHASE=0.0, o_CLKOUT1=pll_out,
+                ),
+                Instance("BUFG", i_I=pll_out, o_O=self.cd_bootstrap.clk)
+            ]
+        else:
+            raise ValueError("Bootstrap frequency must be 100 or 125MHz")
+
+
+class GenericStandalone(SoCCore):
+    def __init__(self, description, acpki=False):
+        self.acpki = acpki
+        clk_freq = description["rtio_frequency"]
+        with_wrpll = description["enable_wrpll"]
+
+        platform = kasli_soc.Platform()
+        platform.toolchain.bitstream_commands.extend([
+            "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
+        ])
+        ident = description["variant"]
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
+
+        self.config["HW_REV"] = description["hw_rev"]
+        clk_synth = platform.request("cdr_clk_clean_fabric")
+        clk_synth_se = Signal()
+        clk_synth_se_buf = Signal()
+        platform.add_period_constraint(clk_synth.p, 8.0)
+
+        self.specials += [ 
+            Instance("IBUFGDS",
+                p_DIFF_TERM="TRUE", p_IBUF_LOW_PWR="FALSE",
+                i_I=clk_synth.p, i_IB=clk_synth.n, o_O=clk_synth_se
+            ),
+            Instance("BUFG", i_I=clk_synth_se, o_O=clk_synth_se_buf),
+        ]
+        fix_serdes_timing_path(platform)
+        self.submodules.bootstrap = GTPBootstrapClock(self.platform, clk_freq)
+        self.config["RTIO_FREQUENCY"] = str(clk_freq/1e6)
+        self.config["CLOCK_FREQUENCY"] = int(clk_freq)
+
+        self.submodules.sys_crg = zynq_clocking.SYSCRG(self.platform, self.ps7, clk_synth_se_buf)
+        platform.add_false_path_constraints(
+            self.bootstrap.cd_bootstrap.clk, self.sys_crg.cd_sys.clk)
+        self.csr_devices.append("sys_crg")
+        self.crg = self.ps7 # HACK for eem_7series to find the clock
+        self.crg.cd_sys = self.sys_crg.cd_sys
+
+        if with_wrpll:
+            self.submodules.wrpll_refclk = wrpll.FrequencyMultiplier(platform.request("sma_clkin"))
+            self.submodules.wrpll = wrpll.WRPLL(
+                platform=self.platform,
+                cd_ref=self.wrpll_refclk.cd_ref,
+                main_clk_se=clk_synth_se)
+            self.csr_devices.append("wrpll_refclk")
+            self.csr_devices.append("wrpll")
+            self.comb += self.ps7.core.core0.nfiq.eq(self.wrpll.ev.irq)
+            self.config["HAS_SI549"] = None
+            self.config["WRPLL_REF_CLK"] = "SMA_CLKIN"
+        else:
+            self.submodules += SMAClkinForward(self.platform)
+            self.config["HAS_SI5324"] = None
+            self.config["SI5324_SOFT_RESET"] = None
+
+
+        self.rtio_channels = []
+        has_grabber = any(peripheral["type"] == "grabber" for peripheral in description["peripherals"])
+        if has_grabber:
+            self.grabber_csr_group = []
+        eem_7series.add_peripherals(self, description["peripherals"], iostandard=eem_iostandard)
+        for i in (0, 1):
+            print("USER LED at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            user_led = self.platform.request("user_led", i)
+            phy = ttl_simple.Output(user_led)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+        self.config["RTIO_LOG_CHANNEL"] = len(self.rtio_channels)
+        self.rtio_channels.append(rtio.LogChannel())
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+        self.submodules.rtio_core = rtio.Core(
+            self.rtio_tsc, self.rtio_channels, lane_count=description["sed_lanes"]
+        )
+        self.csr_devices.append("rtio_core")
+
+        if self.acpki:
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+                                                         bus=self.ps7.s_axi_acp,
+                                                         user=self.ps7.s_axi_acp_user,
+                                                         evento=self.ps7.event.o)
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.rtio.cri, self.rtio_dma.cri],
+            [self.rtio_core.cri])
+        self.csr_devices.append("cri_con")
+
+        self.submodules.rtio_moninj = rtio.MonInj(self.rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(self.rtio_tsc, self.rtio_core.cri,
+                                                          self.ps7.s_axi_hp1)
+        self.csr_devices.append("rtio_analyzer")
+
+        if has_grabber:
+            self.config["HAS_GRABBER"] = None
+            self.add_csr_group("grabber", self.grabber_csr_group)
+            for grabber in self.grabber_csr_group:
+                self.platform.add_false_path_constraints(
+                    self.sys_crg.cd_sys.clk, getattr(self, grabber).deserializer.cd_cl.clk)
+
+
+class GenericMaster(SoCCore):
+    def __init__(self, description, acpki=False):
+        clk_freq = description["rtio_frequency"]
+        with_wrpll = description["enable_wrpll"]
+
+        has_drtio_over_eem = any(peripheral["type"] == "shuttler" for peripheral in description["peripherals"])
+        self.acpki = acpki
+
+        platform = kasli_soc.Platform()
+        platform.toolchain.bitstream_commands.extend([
+            "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
+        ])
+        ident = description["variant"]
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
+
+        self.config["HW_REV"] = description["hw_rev"]
+
+        data_pads = [platform.request("sfp", i) for i in range(4)]
+
+        self.submodules.gt_drtio = gtx_7series.GTX(
+            clock_pads=platform.request("clk_gtp"),
+            pads=data_pads,
+            clk_freq=clk_freq)
+        self.csr_devices.append("gt_drtio")
+        self.config["RTIO_FREQUENCY"] = str(clk_freq/1e6)
+        self.config["CLOCK_FREQUENCY"] = int(clk_freq)
+
+        txout_buf = Signal()
+        gtx0 = self.gt_drtio.gtxs[0]
+        self.specials += Instance("BUFG", i_I=gtx0.txoutclk, o_O=txout_buf)
+
+        ext_async_rst = Signal()
+
+        self.submodules.bootstrap = GTPBootstrapClock(self.platform, clk_freq)
+        self.submodules.sys_crg = zynq_clocking.SYSCRG(
+            self.platform,
+            self.ps7,
+            txout_buf,
+            clk_sw=self.gt_drtio.stable_clkin.storage,
+            clk_sw_status=gtx0.tx_init.done,
+            ext_async_rst=ext_async_rst)
+        self.csr_devices.append("sys_crg")
+        self.crg = self.ps7 # HACK for eem_7series to find the clock
+        self.crg.cd_sys = self.sys_crg.cd_sys
+        platform.add_false_path_constraints(
+            self.bootstrap.cd_bootstrap.clk, self.sys_crg.cd_sys.clk)
+        fix_serdes_timing_path(platform)
+
+        self.comb += ext_async_rst.eq(self.sys_crg.clk_sw_fsm.o_clk_sw & ~gtx0.tx_init.done)
+        self.specials += MultiReg(self.sys_crg.clk_sw_fsm.o_clk_sw & self.sys_crg.mmcm_locked, self.gt_drtio.clk_path_ready, odomain="bootstrap")
+
+        if with_wrpll:
+            clk_synth = platform.request("cdr_clk_clean_fabric")
+            clk_synth_se = Signal()
+            platform.add_period_constraint(clk_synth.p, 8.0)
+            self.specials += Instance("IBUFGDS", p_DIFF_TERM="TRUE", p_IBUF_LOW_PWR="FALSE", i_I=clk_synth.p, i_IB=clk_synth.n, o_O=clk_synth_se)
+            self.submodules.wrpll_refclk = wrpll.FrequencyMultiplier(platform.request("sma_clkin"))
+            self.submodules.wrpll = wrpll.WRPLL(
+                platform=self.platform,
+                cd_ref=self.wrpll_refclk.cd_ref,
+                main_clk_se=clk_synth_se)
+            self.csr_devices.append("wrpll_refclk")
+            self.csr_devices.append("wrpll")
+            self.comb += self.ps7.core.core0.nfiq.eq(self.wrpll.ev.irq)
+            self.config["HAS_SI549"] = None
+            self.config["WRPLL_REF_CLK"] = "SMA_CLKIN"
+        else:
+            self.submodules += SMAClkinForward(self.platform)
+            self.config["HAS_SI5324"] = None
+            self.config["SI5324_SOFT_RESET"] = None
+
+        self.rtio_channels = []
+        has_grabber = any(peripheral["type"] == "grabber" for peripheral in description["peripherals"])
+        if has_drtio_over_eem:
+            self.eem_drtio_channels = []
+        if has_grabber:
+            self.grabber_csr_group = []
+        eem_7series.add_peripherals(self, description["peripherals"], iostandard=eem_iostandard)
+        for i in (0, 1):
+            print("USER LED at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            user_led = self.platform.request("user_led", i)
+            phy = ttl_simple.Output(user_led)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+        self.config["RTIO_LOG_CHANNEL"] = len(self.rtio_channels)
+        self.rtio_channels.append(rtio.LogChannel())
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+
+        self.drtio_csr_group = []
+        self.drtioaux_csr_group = []
+        self.drtioaux_memory_group = []
+        self.drtio_cri = []
+        for i in range(len(self.gt_drtio.channels)):
+            core_name = "drtio" + str(i)
+            coreaux_name = "drtioaux" + str(i)
+            memory_name = "drtioaux" + str(i) + "_mem"
+            self.drtio_csr_group.append(core_name)
+            self.drtioaux_csr_group.append(coreaux_name)
+            self.drtioaux_memory_group.append(memory_name)
+
+            cdr = ClockDomainsRenamer({"rtio_rx": "rtio_rx" + str(i)})
+
+            core = cdr(DRTIOMaster(self.rtio_tsc, self.gt_drtio.channels[i]))
+            setattr(self.submodules, core_name, core)
+            self.drtio_cri.append(core.cri)
+            self.csr_devices.append(core_name)
+
+            coreaux = cdr(drtio_aux_controller.DRTIOAuxControllerBare(core.link_layer))
+            setattr(self.submodules, coreaux_name, coreaux)
+            self.csr_devices.append(coreaux_name)
+
+            size = coreaux.get_mem_size()
+            memory_address = self.axi2csr.register_port(coreaux.get_tx_port(), size)
+            self.axi2csr.register_port(coreaux.get_rx_port(), size)
+            self.add_memory_region(memory_name, self.mem_map["csr"] + memory_address, size * 2)
+        self.config["HAS_DRTIO"] = None
+        self.config["HAS_DRTIO_ROUTING"] = None
+
+        if has_drtio_over_eem:
+            self.add_eem_drtio(self.eem_drtio_channels)
+        self.add_drtio_cpuif_groups()
+
+        self.submodules.rtio_core = rtio.Core(
+            self.rtio_tsc, self.rtio_channels, lane_count=description["sed_lanes"]
+        )
+        self.csr_devices.append("rtio_core")
+
+        if self.acpki:
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+                                                         bus=self.ps7.s_axi_acp,
+                                                         user=self.ps7.s_axi_acp_user,
+                                                         evento=self.ps7.event.o)
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.rtio.cri, self.rtio_dma.cri],
+            [self.rtio_core.cri] + self.drtio_cri,
+            enable_routing=True)
+        self.csr_devices.append("cri_con")
+
+        self.submodules.rtio_moninj = rtio.MonInj(self.rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        self.submodules.routing_table = rtio.RoutingTableAccess(self.cri_con)
+        self.csr_devices.append("routing_table")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(self.rtio_tsc, self.rtio_core.cri,
+                                                          self.ps7.s_axi_hp1)
+        self.csr_devices.append("rtio_analyzer")
+
+        if has_grabber:
+            self.config["HAS_GRABBER"] = None
+            self.add_csr_group("grabber", self.grabber_csr_group)
+        
+
+        self.submodules.virtual_leds = virtual_leds.VirtualLeds()
+        self.csr_devices.append("virtual_leds")
+
+        self.comb += [self.virtual_leds.get(i).eq(channel.rx_ready)
+                for i, channel in enumerate(self.gt_drtio.channels)]
+
+    def add_eem_drtio(self, eem_drtio_channels):
+        # Must be called before invoking add_rtio() to construct the CRI
+        # interconnect properly
+        self.submodules.eem_transceiver = eem_serdes.EEMSerdes(self.platform, eem_drtio_channels)
+        self.csr_devices.append("eem_transceiver")
+        self.config["HAS_DRTIO_EEM"] = None
+        self.config["EEM_DRTIO_COUNT"] = len(eem_drtio_channels)
+
+        cdr = ClockDomainsRenamer({"rtio_rx": "sys"})
+        for i in range(len(self.eem_transceiver.channels)):
+            channel = i + len(self.gt_drtio.channels)
+            core_name = "drtio" + str(channel)
+            coreaux_name = "drtioaux" + str(channel)
+            memory_name = "drtioaux" + str(channel) + "_mem"
+            self.drtio_csr_group.append(core_name)
+            self.drtioaux_csr_group.append(coreaux_name)
+            self.drtioaux_memory_group.append(memory_name)
+
+            core = cdr(DRTIOMaster(self.rtio_tsc, self.eem_transceiver.channels[i]))
+            setattr(self.submodules, core_name, core)
+            self.drtio_cri.append(core.cri)
+            self.csr_devices.append(core_name)
+
+            coreaux = cdr(drtio_aux_controller.DRTIOAuxControllerBare(core.link_layer))
+            setattr(self.submodules, coreaux_name, coreaux)
+            self.csr_devices.append(coreaux_name)
+
+            size = coreaux.get_mem_size()
+            memory_address = self.axi2csr.register_port(coreaux.get_tx_port(), size)
+            self.axi2csr.register_port(coreaux.get_rx_port(), size)
+            self.add_memory_region(memory_name, self.mem_map["csr"] + memory_address, size * 2)
+
+    def add_drtio_cpuif_groups(self):
+        self.add_csr_group("drtio", self.drtio_csr_group)
+        self.add_csr_group("drtioaux", self.drtioaux_csr_group)
+        self.add_memory_group("drtioaux_mem", self.drtioaux_memory_group)
+
+
+class GenericSatellite(SoCCore):
+    def __init__(self, description, acpki=False):
+        clk_freq = description["rtio_frequency"]
+        with_wrpll = description["enable_wrpll"]
+
+        self.acpki = acpki
+
+        platform = kasli_soc.Platform()
+        platform.toolchain.bitstream_commands.extend([
+            "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
+        ])
+        ident = description["variant"]
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
+
+        self.config["HW_REV"] = description["hw_rev"]
+
+        data_pads = [platform.request("sfp", i) for i in range(4)]
+        
+        self.submodules.gt_drtio = gtx_7series.GTX(
+            clock_pads=platform.request("clk_gtp"),  
+            pads=data_pads,
+            clk_freq=clk_freq)
+        self.csr_devices.append("gt_drtio")
+
+        txout_buf = Signal()
+        gtx0 = self.gt_drtio.gtxs[0]
+        self.specials += Instance("BUFG", i_I=gtx0.txoutclk, o_O=txout_buf)
+
+        ext_async_rst = Signal()
+
+        self.submodules.bootstrap = GTPBootstrapClock(self.platform, clk_freq)
+        self.submodules.sys_crg = zynq_clocking.SYSCRG(
+            self.platform, 
+            self.ps7,
+            txout_buf,
+            clk_sw=self.gt_drtio.stable_clkin.storage,
+            clk_sw_status=gtx0.tx_init.done,
+            ext_async_rst=ext_async_rst)
+        platform.add_false_path_constraints(
+            self.bootstrap.cd_bootstrap.clk, self.sys_crg.cd_sys.clk)
+        self.csr_devices.append("sys_crg")
+        self.crg = self.ps7 # HACK for eem_7series to find the clock
+        self.crg.cd_sys = self.sys_crg.cd_sys
+
+        fix_serdes_timing_path(platform)
+
+        self.comb += ext_async_rst.eq(self.sys_crg.clk_sw_fsm.o_clk_sw & ~gtx0.tx_init.done)
+        self.specials += MultiReg(self.sys_crg.clk_sw_fsm.o_clk_sw & self.sys_crg.mmcm_locked, self.gt_drtio.clk_path_ready, odomain="bootstrap")
+
+        self.rtio_channels = []
+        has_grabber = any(peripheral["type"] == "grabber" for peripheral in description["peripherals"])
+        if has_grabber:
+            self.grabber_csr_group = []
+        eem_7series.add_peripherals(self, description["peripherals"], iostandard=eem_iostandard)
+        for i in (0, 1):
+            print("USER LED at RTIO channel 0x{:06x}".format(len(self.rtio_channels)))
+            user_led = self.platform.request("user_led", i)
+            phy = ttl_simple.Output(user_led)
+            self.submodules += phy
+            self.rtio_channels.append(rtio.Channel.from_phy(phy))
+        self.config["RTIO_LOG_CHANNEL"] = len(self.rtio_channels)
+        self.rtio_channels.append(rtio.LogChannel())
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+
+        drtioaux_csr_group = []
+        drtioaux_memory_group = []
+        drtiorep_csr_group = []
+        self.drtio_cri = []
+        for i in range(len(self.gt_drtio.channels)):
+            coreaux_name = "drtioaux" + str(i)
+            memory_name = "drtioaux" + str(i) + "_mem"
+            drtioaux_csr_group.append(coreaux_name)
+            drtioaux_memory_group.append(memory_name)
+
+            cdr = ClockDomainsRenamer({"rtio_rx": "rtio_rx" + str(i)})
+
+            if i == 0:
+                self.submodules.rx_synchronizer = cdr(XilinxRXSynchronizer())
+                core = cdr(DRTIOSatellite(
+                    self.rtio_tsc, self.gt_drtio.channels[i],
+                    self.rx_synchronizer))
+                self.submodules.drtiosat = core
+                self.csr_devices.append("drtiosat")
+            else:
+                corerep_name = "drtiorep" + str(i-1)
+                drtiorep_csr_group.append(corerep_name)
+
+                core = cdr(DRTIORepeater(
+                    self.rtio_tsc, self.gt_drtio.channels[i]))
+                setattr(self.submodules, corerep_name, core)
+                self.drtio_cri.append(core.cri)
+                self.csr_devices.append(corerep_name)
+
+            coreaux = cdr(drtio_aux_controller.DRTIOAuxControllerBare(core.link_layer))
+            setattr(self.submodules, coreaux_name, coreaux)
+            self.csr_devices.append(coreaux_name)
+
+            mem_size = coreaux.get_mem_size()
+            tx_port = coreaux.get_tx_port()
+            rx_port = coreaux.get_rx_port()
+            memory_address = self.axi2csr.register_port(tx_port, mem_size)
+            # rcv in upper half of the memory, thus added second
+            self.axi2csr.register_port(rx_port, mem_size)
+            # and registered in PS interface
+            # manually, because software refers to rx/tx by halves of entire memory block, not names
+            self.add_memory_region(memory_name, self.mem_map["csr"] + memory_address, mem_size * 2)
+        self.config["HAS_DRTIO"] = None
+        self.config["HAS_DRTIO_ROUTING"] = None
+        self.add_csr_group("drtioaux", drtioaux_csr_group)
+        self.add_memory_group("drtioaux_mem", drtioaux_memory_group)
+        self.add_csr_group("drtiorep", drtiorep_csr_group)
+
+        if self.acpki:
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+                                                         bus=self.ps7.s_axi_acp,
+                                                         user=self.ps7.s_axi_acp_user,
+                                                         evento=self.ps7.event.o)
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.local_io = SyncRTIO(
+            self.rtio_tsc, self.rtio_channels, lane_count=description["sed_lanes"]
+        )
+        self.comb += [
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
+
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.drtiosat.cri, self.rtio_dma.cri, self.rtio.cri],
+            [self.local_io.cri] + self.drtio_cri,
+            enable_routing=True)
+        self.csr_devices.append("cri_con")
+
+        self.submodules.routing_table = rtio.RoutingTableAccess(self.cri_con)
+        self.csr_devices.append("routing_table")     
+
+        self.submodules.rtio_moninj = rtio.MonInj(self.rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(self.rtio_tsc, self.local_io.cri,
+                                                          self.ps7.s_axi_hp1)
+        self.csr_devices.append("rtio_analyzer")
+
+        rtio_clk_period = 1e9/clk_freq
+        self.config["RTIO_FREQUENCY"] = str(clk_freq/1e6)
+        self.config["CLOCK_FREQUENCY"] = int(clk_freq)
+
+        if with_wrpll:
+            clk_synth = platform.request("cdr_clk_clean_fabric")
+            clk_synth_se = Signal()
+            platform.add_period_constraint(clk_synth.p, 8.0)
+            self.specials += Instance("IBUFGDS", p_DIFF_TERM="TRUE", p_IBUF_LOW_PWR="FALSE", i_I=clk_synth.p, i_IB=clk_synth.n, o_O=clk_synth_se)
+            self.submodules.wrpll = wrpll.WRPLL(
+                platform=self.platform,
+                cd_ref=self.gt_drtio.cd_rtio_rx0,
+                main_clk_se=clk_synth_se)
+            self.submodules.wrpll_skewtester = wrpll.SkewTester(self.rx_synchronizer)
+            self.csr_devices.append("wrpll_skewtester")
+            self.csr_devices.append("wrpll")
+            self.comb += self.ps7.core.core0.nfiq.eq(self.wrpll.ev.irq)
+            self.config["HAS_SI549"] = None
+            self.config["WRPLL_REF_CLK"] = "GT_CDR"
+        else:
+            self.submodules.siphaser = SiPhaser7Series(
+                si5324_clkin=platform.request("cdr_clk"),
+                rx_synchronizer=self.rx_synchronizer,
+                ultrascale=False,
+                rtio_clk_freq=self.gt_drtio.rtio_clk_freq)
+            self.csr_devices.append("siphaser")
+            self.config["HAS_SI5324"] = None
+            self.config["SI5324_SOFT_RESET"] = None
+
+        gtx0 = self.gt_drtio.gtxs[0]
+        platform.add_false_path_constraints(
+            gtx0.txoutclk, gtx0.rxoutclk)
+
+        if has_grabber:
+            self.config["HAS_GRABBER"] = None
+            self.add_csr_group("grabber", self.grabber_csr_group)
+            # no RTIO CRG here
+        
+        self.submodules.virtual_leds = virtual_leds.VirtualLeds()
+        self.csr_devices.append("virtual_leds")
+
+        self.comb += [self.virtual_leds.get(i).eq(channel.rx_ready)
+                for i, channel in enumerate(self.gt_drtio.channels)]
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="ARTIQ device binary builder for generic Kasli-SoC systems")
+    parser.add_argument("-r", default=None,
+        help="build Rust interface into the specified file")
+    parser.add_argument("-c", default=None,
+        help="build Rust compiler configuration into the specified file")
+    parser.add_argument("-m", default=None,
+        help="build Rust memory interface into the specified file")
+    parser.add_argument("-g", default=None,
+        help="build gateware into the specified directory")
+    parser.add_argument("--acpki", default=False, action="store_true",
+        help="enable ACPKI")
+    parser.add_argument("description", metavar="DESCRIPTION",
+                        help="JSON system description file")
+    args = parser.parse_args()
+    description = jsondesc.load(args.description)
+
+    if description["target"] != "kasli_soc":
+        raise ValueError("Description is for a different target")
+
+    if description["drtio_role"] == "standalone":
+        cls = GenericStandalone
+    elif description["drtio_role"] == "master":
+        cls = GenericMaster
+    elif description["drtio_role"] == "satellite":
+        cls = GenericSatellite
+    else:
+        raise ValueError("Invalid DRTIO role")
+
+    soc = cls(description, acpki=args.acpki)
+    soc.finalize()
+
+    if args.r is not None:
+        write_csr_file(soc, args.r)
+    if args.m is not None:
+        write_mem_file(soc, args.m)
+    if args.c is not None:
+        write_rustc_cfg_file(soc, args.c)
+    if args.g is not None:
+        soc.build(build_dir=args.g)
+
+
+if __name__ == "__main__":
+    main()

src/gateware/test_dma.py

@@ -168,7 +168,7 @@ class FullStackTB(Module):
         bus = axi.Interface(ws*8)
         self.memory = AXIMemorySim(bus, sequence)
         self.submodules.dut = dma.DMA(bus)
-        self.submodules.tsc = rtio.TSC("async")
+        self.submodules.tsc = rtio.TSC()
         self.submodules.rtio = rtio.Core(self.tsc, rtio_channels)
         self.comb += self.dut.cri.connect(self.rtio.cri)
 
@@ -229,7 +229,7 @@ class TestDMA(unittest.TestCase):
             do_dma(tb.dut, 0), monitor(),
             (None for _ in range(70)),
             tb.memory.ar(), tb.memory.r()
-        ]}, {"sys": 8, "rsys": 8, "rtio": 8, "rio": 8, "rio_phy": 8})
+        ]}, {"sys": 8, "rsys": 8, "rio": 8, "rio_phy": 8})
 
         correct_changes = [(timestamp + 11, channel)
                            for channel, timestamp, _, _ in test_writes_full_stack]

src/gateware/zc706.py

@@ -10,98 +10,172 @@ from migen.genlib.cdc import MultiReg
 from migen_axi.integration.soc_core import SoCCore
 from migen_axi.platforms import zc706
 from misoc.interconnect.csr import *
-from misoc.integration import cpu_interface
+from misoc.cores import gpio
 
 from artiq.gateware import rtio, nist_clock, nist_qc2
-from artiq.gateware.rtio.phy import ttl_simple, ttl_serdes_7series, dds, spi2
+from artiq.gateware.rtio.phy import ttl_simple, ttl_serdes_7series, dds, spi2, edge_counter
+from artiq.gateware.rtio.xilinx_clocking import fix_serdes_timing_path
+from artiq.gateware.drtio.transceiver import gtx_7series
+from artiq.gateware.drtio.siphaser import SiPhaser7Series
+from artiq.gateware.drtio.rx_synchronizer import XilinxRXSynchronizer
+from artiq.gateware.drtio import *
 
 import dma
 import analyzer
 import acpki
+import drtio_aux_controller
+import zynq_clocking
+from config import write_csr_file, write_mem_file, write_rustc_cfg_file
 
-
-class RTIOCRG(Module, AutoCSR):
-    def __init__(self, platform, rtio_internal_clk):
-        self.clock_sel = CSRStorage()
-        self.pll_reset = CSRStorage(reset=1)
-        self.pll_locked = CSRStatus()
-        self.clock_domains.cd_rtio = ClockDomain()
-        self.clock_domains.cd_rtiox4 = ClockDomain(reset_less=True)
-
-        rtio_external_clk = Signal()
-        user_sma_clock = platform.request("user_sma_clock")
-        platform.add_period_constraint(user_sma_clock.p, 8.0)
-        self.specials += Instance("IBUFDS",
-                                  i_I=user_sma_clock.p, i_IB=user_sma_clock.n,
-                                  o_O=rtio_external_clk)
-
-        pll_locked = Signal()
-        rtio_clk = Signal()
-        rtiox4_clk = Signal()
+class SMAClkinForward(Module):
+    def __init__(self, platform):
+        sma_clkin = platform.request("user_sma_clock")
+        sma_clkin_se = Signal()
+        si5324_clkin_se = Signal()
+        si5324_clkin = platform.request("si5324_clkin")
         self.specials += [
-            Instance("PLLE2_ADV",
-                     p_STARTUP_WAIT="FALSE", o_LOCKED=pll_locked,
-
-                     p_REF_JITTER1=0.01,
-                     p_CLKIN1_PERIOD=8.0, p_CLKIN2_PERIOD=8.0,
-                     i_CLKIN1=rtio_internal_clk, i_CLKIN2=rtio_external_clk,
-                     # Warning: CLKINSEL=0 means CLKIN2 is selected
-                     i_CLKINSEL=~self.clock_sel.storage,
-
-                     # VCO @ 1GHz when using 125MHz input
-                     p_CLKFBOUT_MULT=8, p_DIVCLK_DIVIDE=1,
-                     i_CLKFBIN=self.cd_rtio.clk,
-                     i_RST=self.pll_reset.storage,
-
-                     o_CLKFBOUT=rtio_clk,
-
-                     p_CLKOUT0_DIVIDE=2, p_CLKOUT0_PHASE=0.0,
-                     o_CLKOUT0=rtiox4_clk),
-            Instance("BUFG", i_I=rtio_clk, o_O=self.cd_rtio.clk),
-            Instance("BUFG", i_I=rtiox4_clk, o_O=self.cd_rtiox4.clk),
-            AsyncResetSynchronizer(self.cd_rtio, ~pll_locked),
-            MultiReg(pll_locked, self.pll_locked.status)
+            Instance("IBUFDS", i_I=sma_clkin.p, i_IB=sma_clkin.n, o_O=sma_clkin_se),
+            Instance("ODDR", i_C=sma_clkin_se, i_CE=1, i_D1=1, i_D2=0, o_Q=si5324_clkin_se),
+            Instance("OBUFDS", i_I=si5324_clkin_se, o_O=si5324_clkin.p, o_OB=si5324_clkin.n)
         ]
 
 
+class CLK200BootstrapClock(Module):
+    def __init__(self, platform, freq=125e6):
+        self.clock_domains.cd_bootstrap = ClockDomain(reset_less=True)
+        self.cd_bootstrap.clk.attr.add("keep")
+
+        clk200 = platform.request("clk200")
+        clk200_se = Signal()
+
+        pll_fb = Signal()
+        pll_clkout = Signal()
+        assert freq in [125e6, 100e6]
+        divide = int(1e9/freq)
+        self.specials += [
+            Instance("IBUFDS",
+                i_I=clk200.p, i_IB=clk200.n, o_O=clk200_se),
+            Instance("PLLE2_BASE",
+                p_CLKIN1_PERIOD=5.0,
+                i_CLKIN1=clk200_se,
+                i_CLKFBIN=pll_fb,
+                o_CLKFBOUT=pll_fb,
+
+                # VCO @ 1GHz
+                p_CLKFBOUT_MULT=5, p_DIVCLK_DIVIDE=1,
+
+                # 125MHz/100MHz for bootstrap
+                p_CLKOUT1_DIVIDE=divide, p_CLKOUT1_PHASE=0.0, o_CLKOUT1=pll_clkout,
+            ),
+            Instance("BUFG", i_I=pll_clkout, o_O=self.cd_bootstrap.clk)
+        ]
+
+
+# The NIST backplanes require setting VADJ to 3.3V by reprogramming the power supply.
+# This also changes the I/O standard for some on-board LEDs.
+leds_fmc33 = [
+    ("user_led_33", 0, Pins("Y21"), IOStandard("LVCMOS33")),
+    ("user_led_33", 1, Pins("G2"), IOStandard("LVCMOS15")),
+    ("user_led_33", 2, Pins("W21"), IOStandard("LVCMOS33")),
+    ("user_led_33", 3, Pins("A17"), IOStandard("LVCMOS15")),
+]
+
+# same deal as with LEDs - changed I/O standard.
+si5324_fmc33 = [
+    ("si5324_33", 0,
+        Subsignal("rst_n", Pins("W23"), IOStandard("LVCMOS33")),
+        Subsignal("int", Pins("AJ25"), IOStandard("LVCMOS33"))
+    ),
+]
+
+pmod1_33 = [
+    ("pmod1_33", 0, Pins("AJ21"), IOStandard("LVCMOS33")),
+    ("pmod1_33", 1, Pins("AK21"), IOStandard("LVCMOS33")),
+    ("pmod1_33", 2, Pins("AB21"), IOStandard("LVCMOS33")),
+    ("pmod1_33", 3, Pins("AB16"), IOStandard("LVCMOS33")),
+    # rest removed for use with dummy spi
+]
+
+_ams101_dac = [
+    ("ams101_dac", 0,
+        Subsignal("ldac", Pins("XADC:GPIO0")),
+        Subsignal("clk", Pins("XADC:GPIO1")),
+        Subsignal("mosi", Pins("XADC:GPIO2")),
+        Subsignal("cs_n", Pins("XADC:GPIO3")),
+        IOStandard("LVCMOS15")
+     )
+]
+
+_pmod_spi = [ 
+    ("pmod_spi", 0,
+        # PMOD_1 4-7 pins, same bank as sfp_tx_disable or user_sma_clock
+        Subsignal("miso", Pins("Y20"), IOStandard("LVCMOS25")),
+        Subsignal("clk", Pins("AA20"), IOStandard("LVCMOS25")),
+        Subsignal("mosi", Pins("AC18"), IOStandard("LVCMOS25")),
+        Subsignal("cs_n", Pins("AC19"), IOStandard("LVCMOS25")),
+        IOStandard("LVCMOS25")
+    )
+]
+
+
+def prepare_zc706_platform(platform):
+    platform.toolchain.bitstream_commands.extend([
+        "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
+    ])
+
 class ZC706(SoCCore):
     def __init__(self, acpki=False):
         self.acpki = acpki
-        self.rustc_cfg = dict()
 
         platform = zc706.Platform()
-        platform.toolchain.bitstream_commands.extend([
-            "set_property BITSTREAM.GENERAL.COMPRESS True [current_design]",
-        ])
+        prepare_zc706_platform(platform)
+
         ident = self.__class__.__name__
         if self.acpki:
             ident = "acpki_" + ident
-        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident)
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
 
-        platform.add_platform_command("create_clock -name clk_fpga_0 -period 8 [get_pins \"PS7/FCLKCLK[0]\"]")
-        platform.add_platform_command("set_input_jitter clk_fpga_0 0.24")
+        platform.add_extension(si5324_fmc33)
+        self.comb += platform.request("si5324_33").rst_n.eq(1)
 
-        self.submodules.rtio_crg = RTIOCRG(self.platform, self.ps7.cd_sys.clk)
-        self.csr_devices.append("rtio_crg")
-        self.platform.add_period_constraint(self.rtio_crg.cd_rtio.clk, 8.)
-        self.platform.add_false_path_constraints(
-            self.ps7.cd_sys.clk,
-            self.rtio_crg.cd_rtio.clk)
+        cdr_clk = Signal()
+        cdr_clk_buf = Signal()
+        si5324_out = platform.request("si5324_clkout")
+        platform.add_period_constraint(si5324_out.p, 8.0)
+        self.specials += [
+            Instance("IBUFDS_GTE2",
+                i_CEB=0,
+                i_I=si5324_out.p, i_IB=si5324_out.n,
+                o_O=cdr_clk,
+                p_CLKCM_CFG="TRUE",
+                p_CLKRCV_TRST="TRUE",
+                p_CLKSWING_CFG=3),
+            Instance("BUFG", i_I=cdr_clk, o_O=cdr_clk_buf)
+        ]
+        self.config["HAS_SI5324"] = None
+        self.config["SI5324_AS_SYNTHESIZER"] = None
+        self.config["SI5324_SOFT_RESET"] = None
+        
+        self.submodules.bootstrap = CLK200BootstrapClock(platform)
+        self.submodules.sys_crg = zynq_clocking.SYSCRG(self.platform, self.ps7, cdr_clk_buf)
+        platform.add_false_path_constraints(
+            self.bootstrap.cd_bootstrap.clk, self.sys_crg.cd_sys.clk)
+        self.csr_devices.append("sys_crg")
 
     def add_rtio(self, rtio_channels):
-        self.submodules.rtio_tsc = rtio.TSC("async", glbl_fine_ts_width=3)
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
         self.submodules.rtio_core = rtio.Core(self.rtio_tsc, rtio_channels)
         self.csr_devices.append("rtio_core")
 
         if self.acpki:
-            self.rustc_cfg["ki_impl"] = "acp"
+            self.config["KI_IMPL"] = "acp"
             self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
                                                          bus=self.ps7.s_axi_acp,
                                                          user=self.ps7.s_axi_acp_user,
                                                          evento=self.ps7.event.o)
             self.csr_devices.append("rtio")
         else:
-            self.rustc_cfg["ki_impl"] = "csr"
+            self.config["KI_IMPL"] = "csr"
             self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
             self.csr_devices.append("rtio")
 
@@ -121,52 +195,331 @@ class ZC706(SoCCore):
         self.csr_devices.append("rtio_analyzer")
 
 
-class Simple(ZC706):
-    def __init__(self, **kwargs):
-        ZC706.__init__(self, **kwargs)
+class _MasterBase(SoCCore):
+    def __init__(self, acpki=False, drtio100mhz=False):
+        self.acpki = acpki
 
-        platform = self.platform
+        clk_freq = 100e6 if drtio100mhz else 125e6
 
-        rtio_channels = []
-        for i in range(4):
-            phy = ttl_simple.Output(platform.request("user_led", i))
-            self.submodules += phy
-            rtio_channels.append(rtio.Channel.from_phy(phy))
+        platform = zc706.Platform()
+        prepare_zc706_platform(platform)
+        ident = self.__class__.__name__
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
 
-        self.config["RTIO_LOG_CHANNEL"] = len(rtio_channels)
-        rtio_channels.append(rtio.LogChannel())
+        platform.add_extension(si5324_fmc33)
 
-        self.add_rtio(rtio_channels)
+        self.comb += platform.request("sfp_tx_disable_n").eq(1)
+        data_pads = [
+            platform.request("sfp"),
+            platform.request("user_sma_mgt")
+        ]
+
+        self.submodules += SMAClkinForward(self.platform)
+
+        # 1000BASE_BX10 Ethernet compatible, 125MHz RTIO clock
+        self.submodules.gt_drtio = gtx_7series.GTX(
+            clock_pads=platform.request("si5324_clkout"),
+            pads=data_pads,
+            clk_freq=clk_freq)
+        self.csr_devices.append("gt_drtio")
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+        ext_async_rst = Signal()
+        txout_buf = Signal()
+        gtx0 = self.gt_drtio.gtxs[0]
+        self.specials += Instance("BUFG", i_I=gtx0.txoutclk, o_O=txout_buf)
+        self.submodules.bootstrap = CLK200BootstrapClock(platform, clk_freq)
+        self.submodules.sys_crg = zynq_clocking.SYSCRG(
+            self.platform, 
+            self.ps7, 
+            txout_buf,
+            clk_sw=self.gt_drtio.stable_clkin.storage,
+            clk_sw_status=gtx0.tx_init.done,
+            ext_async_rst=ext_async_rst,
+            freq=clk_freq)
+        platform.add_false_path_constraints(
+            self.bootstrap.cd_bootstrap.clk, self.sys_crg.cd_sys.clk)
+        self.csr_devices.append("sys_crg")
+
+        self.comb += ext_async_rst.eq(self.sys_crg.clk_sw_fsm.o_clk_sw & ~gtx0.tx_init.done)
+        self.specials += MultiReg(self.sys_crg.clk_sw_fsm.o_clk_sw & self.sys_crg.mmcm_locked, self.gt_drtio.clk_path_ready, odomain="bootstrap")
+
+        drtio_csr_group = []
+        drtioaux_csr_group = []
+        drtioaux_memory_group = []
+        self.drtio_cri = []
+        for i in range(len(self.gt_drtio.channels)):
+            core_name = "drtio" + str(i)
+            coreaux_name = "drtioaux" + str(i)
+            memory_name = "drtioaux" + str(i) + "_mem"
+            drtio_csr_group.append(core_name)
+            drtioaux_csr_group.append(coreaux_name)
+            drtioaux_memory_group.append(memory_name)
+
+            cdr = ClockDomainsRenamer({"rtio_rx": "rtio_rx" + str(i)})
+
+            core = cdr(DRTIOMaster(
+                self.rtio_tsc, self.gt_drtio.channels[i]))
+            setattr(self.submodules, core_name, core)
+            self.drtio_cri.append(core.cri)
+            self.csr_devices.append(core_name)
+
+            coreaux = cdr(drtio_aux_controller.DRTIOAuxControllerBare(core.link_layer))
+            setattr(self.submodules, coreaux_name, coreaux)
+            self.csr_devices.append(coreaux_name)
+
+            mem_size = coreaux.get_mem_size()
+            memory_address = self.axi2csr.register_port(coreaux.get_tx_port(), mem_size)
+            self.axi2csr.register_port(coreaux.get_rx_port(), mem_size)
+            self.add_memory_region(memory_name, self.mem_map["csr"] + memory_address, mem_size * 2)
+        self.config["HAS_DRTIO"] = None
+        self.config["HAS_DRTIO_ROUTING"] = None
+        self.add_csr_group("drtio", drtio_csr_group)
+        self.add_csr_group("drtioaux", drtioaux_csr_group)
+        self.add_memory_group("drtioaux_mem", drtioaux_memory_group)
+
+        self.config["RTIO_FREQUENCY"] = str(self.gt_drtio.rtio_clk_freq/1e6)
+
+        self.submodules.si5324_rst_n = gpio.GPIOOut(platform.request("si5324_33").rst_n)
+        self.csr_devices.append("si5324_rst_n")
+        self.config["HAS_SI5324"] = None
+        self.config["SI5324_AS_SYNTHESIZER"] = None
+
+        # Constrain TX & RX timing for the first transceiver channel
+        # (First channel acts as master for phase alignment for all channels' TX)
+        platform.add_false_path_constraints(
+            gtx0.txoutclk, gtx0.rxoutclk)
+        # Constrain RX timing for the each transceiver channel
+        # (Each channel performs single-lane phase alignment for RX)
+        for gtx in self.gt_drtio.gtxs[1:]:
+            platform.add_false_path_constraints(
+                gtx0.txoutclk, gtx.rxoutclk)
+
+        fix_serdes_timing_path(platform)
+
+    def add_rtio(self, rtio_channels):
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+        self.submodules.rtio_core = rtio.Core(self.rtio_tsc, rtio_channels)
+        self.csr_devices.append("rtio_core")
+
+        if self.acpki:
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+                                                         bus=self.ps7.s_axi_acp,
+                                                         user=self.ps7.s_axi_acp_user,
+                                                         evento=self.ps7.event.o)
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.rtio.cri, self.rtio_dma.cri],
+            [self.rtio_core.cri] + self.drtio_cri,
+            enable_routing=True)
+        self.csr_devices.append("cri_con")
+
+        self.submodules.rtio_moninj = rtio.MonInj(rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(self.rtio_tsc, self.rtio_core.cri,
+                                                          self.ps7.s_axi_hp1)
+        self.csr_devices.append("rtio_analyzer")
+
+        self.submodules.routing_table = rtio.RoutingTableAccess(self.cri_con)
+        self.csr_devices.append("routing_table")
 
 
-# The NIST backplanes require setting VADJ to 3.3V by reprogramming the power supply.
-# This also changes the I/O standard for some on-board LEDs.
-leds_fmc33 = [
-    ("user_led_33", 0, Pins("Y21"), IOStandard("LVCMOS33")),
-    ("user_led_33", 1, Pins("G2"), IOStandard("LVCMOS15")),
-    ("user_led_33", 2, Pins("W21"), IOStandard("LVCMOS33")),
-    ("user_led_33", 3, Pins("A17"), IOStandard("LVCMOS15")),
-]
+class _SatelliteBase(SoCCore):
+    def __init__(self, acpki=False, drtio100mhz=False):
+        self.acpki = acpki
+
+        clk_freq = 100e6 if drtio100mhz else 125e6
+
+        platform = zc706.Platform()
+        prepare_zc706_platform(platform)
+        ident = self.__class__.__name__
+        if self.acpki:
+            ident = "acpki_" + ident
+        SoCCore.__init__(self, platform=platform, csr_data_width=32, ident=ident, ps_cd_sys=False)
+
+        platform.add_extension(si5324_fmc33)
+
+        # SFP
+        self.comb += platform.request("sfp_tx_disable_n").eq(0)
+        data_pads = [
+            platform.request("sfp"),
+            platform.request("user_sma_mgt")
+        ]
+
+        self.submodules.rtio_tsc = rtio.TSC(glbl_fine_ts_width=3)
+
+        # 1000BASE_BX10 Ethernet compatible, 125MHz RTIO clock
+        self.submodules.gt_drtio = gtx_7series.GTX(
+            clock_pads=platform.request("si5324_clkout"),
+            pads=data_pads,
+            clk_freq=clk_freq)
+        self.csr_devices.append("gt_drtio")
+
+        ext_async_rst = Signal()
+        txout_buf = Signal()
+        txout_buf.attr.add("keep")
+        gtx0 = self.gt_drtio.gtxs[0]
+        self.specials += Instance(
+            "BUFG", 
+            i_I=gtx0.txoutclk, 
+            o_O=txout_buf)
+        self.submodules.bootstrap = CLK200BootstrapClock(platform, clk_freq)
+        self.submodules.sys_crg = zynq_clocking.SYSCRG(
+            self.platform, 
+            self.ps7, 
+            txout_buf,
+            clk_sw=self.gt_drtio.stable_clkin.storage,
+            clk_sw_status=gtx0.tx_init.done,
+            ext_async_rst=ext_async_rst,
+            freq=clk_freq)
+        platform.add_false_path_constraints(
+            self.bootstrap.cd_bootstrap.clk, self.sys_crg.cd_sys.clk)
+        self.csr_devices.append("sys_crg")
+
+        self.comb += ext_async_rst.eq(self.sys_crg.clk_sw_fsm.o_clk_sw & ~gtx0.tx_init.done)
+        self.specials += MultiReg(self.sys_crg.clk_sw_fsm.o_clk_sw & self.sys_crg.mmcm_locked, self.gt_drtio.clk_path_ready, odomain="bootstrap")
+
+        drtioaux_csr_group = []
+        drtioaux_memory_group = []
+        drtiorep_csr_group = []
+        self.drtio_cri = []
+        for i in range(len(self.gt_drtio.channels)):
+            coreaux_name = "drtioaux" + str(i)
+            memory_name = "drtioaux" + str(i) + "_mem"
+            drtioaux_csr_group.append(coreaux_name)
+            drtioaux_memory_group.append(memory_name)
+
+            cdr = ClockDomainsRenamer({"rtio_rx": "rtio_rx" + str(i)})
+
+            # Satellite
+            if i == 0:
+                self.submodules.rx_synchronizer = cdr(XilinxRXSynchronizer())
+                core = cdr(DRTIOSatellite(
+                    self.rtio_tsc, self.gt_drtio.channels[0], self.rx_synchronizer))
+                self.submodules.drtiosat = core
+                self.csr_devices.append("drtiosat")
+            # Repeaters
+            else:
+                corerep_name = "drtiorep" + str(i-1)
+                drtiorep_csr_group.append(corerep_name)
+                core = cdr(DRTIORepeater(
+                    self.rtio_tsc, self.gt_drtio.channels[i]))
+                setattr(self.submodules, corerep_name, core)
+                self.drtio_cri.append(core.cri)
+                self.csr_devices.append(corerep_name)
+
+            coreaux = cdr(drtio_aux_controller.DRTIOAuxControllerBare(core.link_layer))
+            setattr(self.submodules, coreaux_name, coreaux)
+            self.csr_devices.append(coreaux_name)
+
+            mem_size = coreaux.get_mem_size()
+            tx_port = coreaux.get_tx_port()
+            rx_port = coreaux.get_rx_port()
+            memory_address = self.axi2csr.register_port(tx_port, mem_size)
+            # rcv in upper half of the memory, thus added second
+            self.axi2csr.register_port(rx_port, mem_size)
+            # and registered in PS interface
+            # manually, because software refers to rx/tx by halves of entire memory block, not names
+            self.add_memory_region(memory_name, self.mem_map["csr"] + memory_address, mem_size * 2)
+        self.config["HAS_DRTIO"] = None
+        self.config["HAS_DRTIO_ROUTING"] = None
+        self.add_csr_group("drtioaux", drtioaux_csr_group)
+        self.add_csr_group("drtiorep", drtiorep_csr_group)
+        self.add_memory_group("drtioaux_mem", drtioaux_memory_group)
+
+        self.config["RTIO_FREQUENCY"] = str(self.gt_drtio.rtio_clk_freq/1e6)
+
+        # Si5324 Phaser
+        self.submodules.siphaser = SiPhaser7Series(
+            si5324_clkin=platform.request("si5324_clkin"),
+            rx_synchronizer=self.rx_synchronizer,
+            ultrascale=False,
+            rtio_clk_freq=self.gt_drtio.rtio_clk_freq)
+        platform.add_false_path_constraints(
+            self.sys_crg.cd_sys.clk, self.siphaser.mmcm_freerun_output)
+        self.csr_devices.append("siphaser")
+        self.submodules.si5324_rst_n = gpio.GPIOOut(platform.request("si5324_33").rst_n)
+        self.csr_devices.append("si5324_rst_n")
+        self.config["HAS_SI5324"] = None
+
+        rtio_clk_period = 1e9/self.gt_drtio.rtio_clk_freq
+        # Constrain TX & RX timing for the first transceiver channel
+        # (First channel acts as master for phase alignment for all channels' TX)
+        platform.add_false_path_constraints(
+            gtx0.txoutclk, gtx0.rxoutclk)
+        # Constrain RX timing for the each transceiver channel
+        # (Each channel performs single-lane phase alignment for RX)
+        for gtx in self.gt_drtio.gtxs[1:]:
+            platform.add_false_path_constraints(
+                self.sys_crg.cd_sys.clk, gtx.rxoutclk)
+
+        fix_serdes_timing_path(platform)
+
+    def add_rtio(self, rtio_channels):
+        self.submodules.rtio_moninj = rtio.MonInj(rtio_channels)
+        self.csr_devices.append("rtio_moninj")
+
+        if self.acpki:
+            self.config["KI_IMPL"] = "acp"
+            self.submodules.rtio = acpki.KernelInitiator(self.rtio_tsc,
+                                                         bus=self.ps7.s_axi_acp,
+                                                         user=self.ps7.s_axi_acp_user,
+                                                         evento=self.ps7.event.o)
+            self.csr_devices.append("rtio")
+        else:
+            self.config["KI_IMPL"] = "csr"
+            self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc, now64=True)
+            self.csr_devices.append("rtio")
+
+        self.submodules.rtio_dma = dma.DMA(self.ps7.s_axi_hp0)
+        self.csr_devices.append("rtio_dma")
+
+        self.submodules.local_io = SyncRTIO(self.rtio_tsc, rtio_channels)
+        self.comb += [
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
+        self.submodules.cri_con = rtio.CRIInterconnectShared(
+            [self.drtiosat.cri, self.rtio_dma.cri, self.rtio.cri],
+            [self.local_io.cri] + self.drtio_cri,
+            enable_routing=True)
+        self.csr_devices.append("cri_con")
+
+        self.submodules.rtio_analyzer = analyzer.Analyzer(self.rtio_tsc, self.local_io.cri,
+                                                          self.ps7.s_axi_hp1)
+        self.csr_devices.append("rtio_analyzer")
+
+        self.submodules.routing_table = rtio.RoutingTableAccess(self.cri_con)
+        self.csr_devices.append("routing_table")
 
 
-class NIST_CLOCK(ZC706):
+
+class _NIST_CLOCK_RTIO:
     """
     NIST clock hardware, with old backplane and 11 DDS channels
     """
-    def __init__(self, **kwargs):
-        ZC706.__init__(self, **kwargs)
-
+    def __init__(self):
         platform = self.platform
         platform.add_extension(nist_clock.fmc_adapter_io)
         platform.add_extension(leds_fmc33)
+        platform.add_extension(pmod1_33)
+        platform.add_extension(_ams101_dac)
+        platform.add_extension(_pmod_spi)
 
         rtio_channels = []
 
-        for i in range(4):
-            phy = ttl_simple.Output(platform.request("user_led_33", i))
-            self.submodules += phy
-            rtio_channels.append(rtio.Channel.from_phy(phy))
-
         for i in range(16):
             if i % 4 == 3:
                 phy = ttl_serdes_7series.InOut_8X(platform.request("ttl", i))
@@ -182,16 +535,40 @@ class NIST_CLOCK(ZC706):
             self.submodules += phy
             rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=512))
 
+        # no SMA GPIO, replaced with PMOD1_0
+        phy = ttl_serdes_7series.InOut_8X(platform.request("pmod1_33", 0))
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=512))
+
+        phy = ttl_simple.Output(platform.request("user_led_33", 0))
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        ams101_dac = self.platform.request("ams101_dac", 0)
+        phy = ttl_simple.Output(ams101_dac.ldac)
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy))
+
         phy = ttl_simple.ClockGen(platform.request("la32_p"))
         self.submodules += phy
         rtio_channels.append(rtio.Channel.from_phy(phy))
 
+        phy = spi2.SPIMaster(ams101_dac)
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(
+            phy, ififo_depth=4))
+
         for i in range(3):
             phy = spi2.SPIMaster(self.platform.request("spi", i))
             self.submodules += phy
             rtio_channels.append(rtio.Channel.from_phy(
                 phy, ififo_depth=128))
 
+        # no SDIO on PL side, dummy SPI placeholder instead
+        phy = spi2.SPIMaster(platform.request("pmod_spi"))
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=4))
+
         phy = dds.AD9914(platform.request("dds"), 11, onehot=True)
         self.submodules += phy
         rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=4))
@@ -202,30 +579,43 @@ class NIST_CLOCK(ZC706):
         self.add_rtio(rtio_channels)
 
 
-class NIST_QC2(ZC706):
+class _NIST_QC2_RTIO:
     """
     NIST QC2 hardware, as used in Quantum I and Quantum II, with new backplane
     and 24 DDS channels.  Two backplanes are used.
     """
-    def __init__(self, **kwargs):
-        ZC706.__init__(self, **kwargs)
-
+    def __init__(self):
         platform = self.platform
         platform.add_extension(nist_qc2.fmc_adapter_io)
         platform.add_extension(leds_fmc33)
+        platform.add_extension(_ams101_dac)
+        platform.add_extension(pmod1_33)
 
         rtio_channels = []
-
-        for i in range(4):
-            phy = ttl_simple.Output(platform.request("user_led_33", i))
-            self.submodules += phy
-            rtio_channels.append(rtio.Channel.from_phy(phy))
+        edge_counter_phy = []
 
         # All TTL channels are In+Out capable
         for i in range(40):
             phy = ttl_serdes_7series.InOut_8X(platform.request("ttl", i))
             self.submodules += phy
             rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=512))
+            # first four TTLs will also have edge counters
+            if i < 4:
+                edge_counter_phy.append(phy)
+
+        # no SMA GPIO, replaced with PMOD1_0
+        phy = ttl_serdes_7series.InOut_8X(platform.request("pmod1_33", 0))
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=512))
+
+        phy = ttl_simple.Output(platform.request("user_led_33", 0))
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy))
+
+        ams101_dac = self.platform.request("ams101_dac", 0)
+        phy = ttl_simple.Output(ams101_dac.ldac)
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(phy))
 
         # CLK0, CLK1 are for clock generators, on backplane SMP connectors
         for i in range(2):
@@ -234,6 +624,11 @@ class NIST_QC2(ZC706):
             self.submodules += phy
             rtio_channels.append(rtio.Channel.from_phy(phy))
         
+        phy = spi2.SPIMaster(ams101_dac)
+        self.submodules += phy
+        rtio_channels.append(rtio.Channel.from_phy(
+            phy, ififo_depth=4))
+
         for i in range(4):
             phy = spi2.SPIMaster(self.platform.request("spi", i))
             self.submodules += phy
@@ -246,42 +641,66 @@ class NIST_QC2(ZC706):
             self.submodules += phy
             rtio_channels.append(rtio.Channel.from_phy(phy, ififo_depth=4))
         
+        for phy in edge_counter_phy:
+            counter = edge_counter.SimpleEdgeCounter(phy.input_state)
+            self.submodules += counter
+            rtio_channels.append(rtio.Channel.from_phy(counter))
+
         self.config["RTIO_LOG_CHANNEL"] = len(rtio_channels)
         rtio_channels.append(rtio.LogChannel())
 
         self.add_rtio(rtio_channels)
 
 
-VARIANTS = {cls.__name__.lower(): cls for cls in [Simple, NIST_CLOCK, NIST_QC2]}
+class NIST_CLOCK(ZC706, _NIST_CLOCK_RTIO):
+    def __init__(self, acpki, drtio100mhz):
+        ZC706.__init__(self, acpki)
+        self.submodules += SMAClkinForward(self.platform)
+        _NIST_CLOCK_RTIO.__init__(self)
 
+class NIST_CLOCK_Master(_MasterBase, _NIST_CLOCK_RTIO):
+    def __init__(self, acpki, drtio100mhz):
+        _MasterBase.__init__(self, acpki, drtio100mhz)
+        _NIST_CLOCK_RTIO.__init__(self)
 
-def write_csr_file(soc, filename):
-    with open(filename, "w") as f:
-        f.write(cpu_interface.get_csr_rust(
-            soc.get_csr_regions(), soc.get_csr_groups(), soc.get_constants()))
+class NIST_CLOCK_Satellite(_SatelliteBase, _NIST_CLOCK_RTIO):
+    def __init__(self, acpki, drtio100mhz):
+        _SatelliteBase.__init__(self, acpki, drtio100mhz)
+        _NIST_CLOCK_RTIO.__init__(self)
 
+class NIST_QC2(ZC706, _NIST_QC2_RTIO):
+    def __init__(self, acpki, drtio100mhz):
+        ZC706.__init__(self, acpki)
+        self.submodules += SMAClkinForward(self.platform)
+        _NIST_QC2_RTIO.__init__(self)
 
-def write_rustc_cfg_file(soc, filename):
-    with open(filename, "w") as f:
-        for k, v in sorted(soc.rustc_cfg.items(), key=itemgetter(0)):
-            if v is None:
-                f.write("{}\n".format(k))
-            else:
-                f.write("{}=\"{}\"\n".format(k, v))
+class NIST_QC2_Master(_MasterBase, _NIST_QC2_RTIO):
+    def __init__(self, acpki, drtio100mhz):
+        _MasterBase.__init__(self, acpki, drtio100mhz)
+        _NIST_QC2_RTIO.__init__(self)
 
+class NIST_QC2_Satellite(_SatelliteBase, _NIST_QC2_RTIO):
+    def __init__(self, acpki, drtio100mhz):
+        _SatelliteBase.__init__(self, acpki, drtio100mhz)
+        _NIST_QC2_RTIO.__init__(self)
+
+VARIANTS = {cls.__name__.lower(): cls for cls in [NIST_CLOCK, NIST_CLOCK_Master, NIST_CLOCK_Satellite,
+                                                  NIST_QC2, NIST_QC2_Master, NIST_QC2_Satellite]}
 
 def main():
     parser = argparse.ArgumentParser(
         description="ARTIQ port to the ZC706 Zynq development kit")
     parser.add_argument("-r", default=None,
         help="build Rust interface into the specified file")
+    parser.add_argument("-m", default=None,
+        help="build Rust memory interface into the specified file")
     parser.add_argument("-c", default=None,
         help="build Rust compiler configuration into the specified file")
     parser.add_argument("-g", default=None,
         help="build gateware into the specified directory")
-    parser.add_argument("-V", "--variant", default="simple",
+    parser.add_argument("-V", "--variant", default="nist_clock",
                         help="variant: "
-                             "[acpki_]simple/nist_clock/nist_qc2 "
+                             "[acpki_]nist_clock/nist_qc2[_master/_satellite][_100mhz]"
                              "(default: %(default)s)")
     args = parser.parse_args()
 
@@ -289,21 +708,25 @@ def main():
     acpki = variant.startswith("acpki_")
     if acpki:
         variant = variant[6:]
+    drtio100mhz = variant.endswith("_100mhz")
+    if drtio100mhz:
+        variant = variant[:-7]
     try:
         cls = VARIANTS[variant]
     except KeyError:
         raise SystemExit("Invalid variant (-V/--variant)")
 
-    soc = cls(acpki=acpki)
+    soc = cls(acpki=acpki, drtio100mhz=drtio100mhz)
     soc.finalize()
 
     if args.r is not None:
         write_csr_file(soc, args.r)
+    if args.m is not None:
+        write_mem_file(soc, args.m)
     if args.c is not None:
         write_rustc_cfg_file(soc, args.c)
     if args.g is not None:
         soc.build(build_dir=args.g)
 
-
 if __name__ == "__main__":
     main()

src/gateware/zynq_clocking.py

@@ -0,0 +1,154 @@
+from migen import *
+from migen.genlib.cdc import MultiReg
+from migen.genlib.resetsync import AsyncResetSynchronizer
+from misoc.interconnect.csr import *
+
+
+class ClockSwitchFSM(Module):
+    def __init__(self):
+        self.i_clk_sw = Signal()
+
+        self.o_clk_sw = Signal()
+        self.o_reset = Signal()
+
+        ###
+
+        i_switch = Signal()
+        o_switch = Signal()
+        reset = Signal()
+
+        # at 125MHz bootstrap cd, will get around 0.5ms
+        delay_counter = Signal(16, reset=0xFFFF)
+
+        # register to prevent glitches
+        self.sync.bootstrap += [
+            self.o_clk_sw.eq(o_switch),
+            self.o_reset.eq(reset),
+        ]
+
+        self.o_clk_sw.attr.add("no_retiming")
+        self.o_reset.attr.add("no_retiming")
+        self.i_clk_sw.attr.add("no_retiming")
+        i_switch.attr.add("no_retiming")
+
+        self.specials += MultiReg(self.i_clk_sw, i_switch, "bootstrap")
+
+        fsm = ClockDomainsRenamer("bootstrap")(FSM(reset_state="START"))
+
+        self.submodules += fsm
+
+        fsm.act("START",
+            If(i_switch & ~o_switch,
+                NextState("RESET_START"))
+        )
+        
+        fsm.act("RESET_START",
+            reset.eq(1),
+            If(delay_counter == 0,
+                NextValue(delay_counter, 0xFFFF),
+                NextState("CLOCK_SWITCH")
+            ).Else(
+                NextValue(delay_counter, delay_counter-1),
+            )
+        )
+
+        fsm.act("CLOCK_SWITCH",
+            reset.eq(1),
+            NextValue(o_switch, 1),
+            NextValue(delay_counter, delay_counter-1),
+            If(delay_counter == 0,
+                NextState("END"))
+        )
+        fsm.act("END",
+            NextValue(o_switch, 1),
+            reset.eq(0))
+
+
+class SYSCRG(Module, AutoCSR):
+    def __init__(self, platform, ps7, main_clk, clk_sw=None, clk_sw_status=None, freq=125e6, ext_async_rst=None, ):
+        # assumes bootstrap clock is same freq as main and sys output
+        self.clock_domains.cd_sys = ClockDomain()
+        self.clock_domains.cd_sys4x = ClockDomain(reset_less=True)
+        self.clock_domains.cd_sys5x = ClockDomain(reset_less=True)
+        self.clock_domains.cd_clk200 = ClockDomain()
+
+        self.current_clock = CSRStatus()
+
+        self.cd_sys.clk.attr.add("keep")
+
+        bootstrap_clk = ClockSignal("bootstrap")
+
+        period = 1e9/freq
+
+        self.submodules.clk_sw_fsm = ClockSwitchFSM()
+
+        if clk_sw is None:
+            self.clock_switch = CSRStorage()
+            self.comb += self.clk_sw_fsm.i_clk_sw.eq(self.clock_switch.storage)
+        else:
+            self.comb += self.clk_sw_fsm.i_clk_sw.eq(clk_sw)
+
+        self.mmcm_locked = Signal()
+        mmcm_sys = Signal()
+        mmcm_sys4x = Signal()
+        mmcm_sys5x = Signal()
+        mmcm_clk208 = Signal()
+        mmcm_fb_clk = Signal()
+        self.specials += [
+            Instance("MMCME2_ADV",
+                p_STARTUP_WAIT="FALSE", o_LOCKED=self.mmcm_locked,
+                p_BANDWIDTH="HIGH",
+                p_REF_JITTER1=0.001,
+                p_CLKIN1_PERIOD=period, i_CLKIN1=main_clk,
+                p_CLKIN2_PERIOD=period, i_CLKIN2=bootstrap_clk,
+                i_CLKINSEL=self.clk_sw_fsm.o_clk_sw,
+
+                # VCO @ 1.25GHz
+                p_CLKFBOUT_MULT_F=10, p_DIVCLK_DIVIDE=1,
+                i_CLKFBIN=mmcm_fb_clk,
+                i_RST=self.clk_sw_fsm.o_reset,
+
+                o_CLKFBOUT=mmcm_fb_clk,
+
+                p_CLKOUT0_DIVIDE_F=2.5, p_CLKOUT0_PHASE=0.0, o_CLKOUT0=mmcm_sys4x,
+
+                # 125MHz
+                p_CLKOUT1_DIVIDE=10, p_CLKOUT1_PHASE=0.0, o_CLKOUT1=mmcm_sys,
+
+                # 625MHz
+                p_CLKOUT2_DIVIDE=2, p_CLKOUT2_PHASE=0.0, o_CLKOUT2=mmcm_sys5x,
+
+                # 208MHz
+                p_CLKOUT3_DIVIDE=6, p_CLKOUT3_PHASE=0.0, o_CLKOUT3=mmcm_clk208,
+            ),
+            Instance("BUFG", i_I=mmcm_sys5x, o_O=self.cd_sys5x.clk),
+            Instance("BUFG", i_I=mmcm_sys, o_O=self.cd_sys.clk),
+            Instance("BUFG", i_I=mmcm_sys4x, o_O=self.cd_sys4x.clk),
+            Instance("BUFG", i_I=mmcm_clk208, o_O=self.cd_clk200.clk),
+        ]
+
+        if ext_async_rst is not None:
+            self.specials += [
+                AsyncResetSynchronizer(self.cd_sys, ~self.mmcm_locked | ext_async_rst),
+                AsyncResetSynchronizer(self.cd_clk200, ~self.mmcm_locked | ext_async_rst),
+            ]
+        else:
+            self.specials += [
+                AsyncResetSynchronizer(self.cd_sys, ~self.mmcm_locked),
+                AsyncResetSynchronizer(self.cd_clk200, ~self.mmcm_locked),
+            ]
+
+        reset_counter = Signal(4, reset=15)
+        ic_reset = Signal(reset=1)
+        self.sync.clk200 += \
+            If(reset_counter != 0,
+                reset_counter.eq(reset_counter - 1)
+            ).Else(
+                ic_reset.eq(0)
+            )
+        self.specials += Instance("IDELAYCTRL", i_REFCLK=ClockSignal("clk200"), i_RST=ic_reset)
+
+        if clk_sw_status is None:
+            self.comb += self.current_clock.status.eq(self.clk_sw_fsm.o_clk_sw)
+        else:
+            self.comb += self.current_clock.status.eq(clk_sw_status)

src/libboard_artiq/Cargo.toml.tpl

@@ -0,0 +1,34 @@
+[package]
+name = "libboard_artiq"
+version = "0.0.0"
+authors = ["M-Labs"]
+edition = "2018"
+
+[lib]
+name = "libboard_artiq"
+
+[features]
+target_zc706 = ["libboard_zynq/target_zc706", "libconfig/target_zc706"]
+target_kasli_soc = ["libboard_zynq/target_kasli_soc", "libconfig/target_kasli_soc"]
+target_ebaz4205 = ["libboard_zynq/target_ebaz4205", "libconfig/target_ebaz4205"]
+calibrate_wrpll_skew = []
+
+[build-dependencies]
+build_zynq = { path = "../libbuild_zynq" }
+
+[dependencies]
+log = "0.4"
+log_buffer = { version = "1.2" }
+crc = { version = "1.7", default-features = false }
+core_io = { version = "0.1", features = ["collections"] }
+embedded-hal = "0.2"
+nb = "1.0"
+void = { version = "1", default-features = false }
+
+io = { path = "../libio", features = ["byteorder"] }
+libboard_zynq = { path = "@@ZYNQ_RS@@/libboard_zynq" }
+libsupport_zynq = { path = "@@ZYNQ_RS@@/libsupport_zynq", default-features = false, features = ["alloc_core"] }
+libregister = { path = "@@ZYNQ_RS@@/libregister" }
+libconfig = { path = "@@ZYNQ_RS@@/libconfig", features = ["fat_lfn"] }
+libcortex_a9 = { path = "@@ZYNQ_RS@@/libcortex_a9" }
+libasync = { path = "@@ZYNQ_RS@@/libasync" }

src/libboard_artiq/build.rs

@@ -0,0 +1,5 @@
+extern crate build_zynq;
+
+fn main() {
+    build_zynq::cfg();
+}

src/libboard_artiq/src/drtio_eem.rs

@@ -0,0 +1,245 @@
+use embedded_hal::prelude::_embedded_hal_blocking_delay_DelayUs;
+use libboard_zynq::timer::GlobalTimer;
+use libconfig::Config;
+use libsupport_zynq::alloc::format;
+use log::{debug, error, info};
+
+use crate::pl;
+
+struct SerdesConfig {
+    pub delay: [u8; 4],
+}
+
+impl SerdesConfig {
+    pub fn as_bytes(&self) -> &[u8] {
+        unsafe {
+            core::slice::from_raw_parts(
+                (self as *const SerdesConfig) as *const u8,
+                core::mem::size_of::<SerdesConfig>(),
+            )
+        }
+    }
+}
+
+fn select_lane(lane_no: u8) {
+    unsafe {
+        pl::csr::eem_transceiver::lane_sel_write(lane_no);
+    }
+}
+
+fn apply_delay(tap: u8, timer: &mut GlobalTimer) {
+    unsafe {
+        pl::csr::eem_transceiver::dly_cnt_in_write(tap);
+        pl::csr::eem_transceiver::dly_ld_write(1);
+        timer.delay_us(1);
+        assert!(tap as u8 == pl::csr::eem_transceiver::dly_cnt_out_read());
+    }
+}
+
+fn apply_config(config: &SerdesConfig, timer: &mut GlobalTimer) {
+    for lane_no in 0..4 {
+        select_lane(lane_no as u8);
+        apply_delay(config.delay[lane_no], timer);
+    }
+}
+
+unsafe fn assign_delay(timer: &mut GlobalTimer) -> SerdesConfig {
+    // Select an appropriate delay for lane 0
+    select_lane(0);
+
+    //
+
+    let mut best_dly = None;
+
+    loop {
+        let mut prev = None;
+        for curr_dly in 0..32 {
+            //let read_align = read_align_fn(curr_dly, timer);
+            let curr_low_rate = read_align(curr_dly, timer);
+
+            if let Some(prev_low_rate) = prev {
+                // This is potentially a crossover position
+                if prev_low_rate <= curr_low_rate && curr_low_rate >= 0.5 {
+                    let prev_dev = 0.5 - prev_low_rate;
+                    let curr_dev = curr_low_rate - 0.5;
+                    let selected_idx = if prev_dev < curr_dev { curr_dly - 1 } else { curr_dly };
+
+                    // The setup setup/hold calibration timing (even with
+                    // tolerance) might be invalid in other lanes due to skew.
+                    // 5 taps is very conservative, generally it is 1 or 2
+                    if selected_idx < 5 {
+                        prev = None;
+                        continue;
+                    } else {
+                        best_dly = Some(selected_idx);
+                        break;
+                    }
+                }
+            }
+
+            // Only rising slope from <= 0.5 can result in a rising low rate
+            // crossover at 50%.
+            if curr_low_rate <= 0.5 {
+                prev = Some(curr_low_rate);
+            }
+        }
+
+        if best_dly.is_none() {
+            error!("setup/hold timing calibration failed, retry in 1s...");
+            timer.delay_us(1_000_000);
+        } else {
+            break;
+        }
+    }
+
+    let best_dly = best_dly.unwrap();
+
+    apply_delay(best_dly, timer);
+    let mut delay_list = [best_dly; 4];
+
+    // Assign delay for other lanes
+    for lane_no in 1..=3 {
+        select_lane(lane_no as u8);
+
+        let mut min_deviation = 0.5;
+        let mut min_idx = 0;
+        for dly_delta in -3..=3 {
+            let index = (best_dly as isize + dly_delta) as u8;
+            let low_rate = read_align(index, timer);
+            // abs() from f32 is not available in core library
+            let deviation = if low_rate < 0.5 { 0.5 - low_rate } else { low_rate - 0.5 };
+
+            if deviation < min_deviation {
+                min_deviation = deviation;
+                min_idx = index;
+            }
+        }
+
+        apply_delay(min_idx, timer);
+        delay_list[lane_no] = min_idx;
+    }
+
+    debug!("setup/hold timing calibration: {:?}", delay_list);
+
+    SerdesConfig { delay: delay_list }
+}
+
+fn read_align(dly: u8, timer: &mut GlobalTimer) -> f32 {
+    unsafe {
+        apply_delay(dly, timer);
+        pl::csr::eem_transceiver::counter_reset_write(1);
+
+        pl::csr::eem_transceiver::counter_enable_write(1);
+        timer.delay_us(2000);
+        pl::csr::eem_transceiver::counter_enable_write(0);
+
+        let (high, low) = (
+            pl::csr::eem_transceiver::counter_high_count_read(),
+            pl::csr::eem_transceiver::counter_low_count_read(),
+        );
+        if pl::csr::eem_transceiver::counter_overflow_read() == 1 {
+            panic!("Unexpected phase detector counter overflow");
+        }
+
+        low as f32 / (low + high) as f32
+    }
+}
+
+unsafe fn align_comma(timer: &mut GlobalTimer) {
+    loop {
+        for slip in 1..=10 {
+            // The soft transceiver has 2 8b10b decoders, which receives lane
+            // 0/1 and lane 2/3 respectively. The decoder are time-multiplexed
+            // to decode exactly 1 lane each sysclk cycle.
+            //
+            // The decoder decodes lane 0/2 data on odd sysclk cycles, buffer
+            // on even cycles, and vice versa for lane 1/3. Data/Clock latency
+            // could change timing. The extend bit flips the decoding timing,
+            // so lane 0/2 data are decoded on even cycles, and lane 1/3 data
+            // are decoded on odd cycles.
+            //
+            // This is needed because transmitting/receiving a 8b10b character
+            // takes 2 sysclk cycles. Adjusting bitslip only via ISERDES
+            // limits the range to 1 cycle. The wordslip bit extends the range
+            // to 2 sysclk cycles.
+            pl::csr::eem_transceiver::wordslip_write((slip > 5) as u8);
+
+            // Apply a double bitslip since the ISERDES is 2x oversampled.
+            // Bitslip is used for comma alignment purposes once setup/hold
+            // timing is met.
+            pl::csr::eem_transceiver::bitslip_write(1);
+            pl::csr::eem_transceiver::bitslip_write(1);
+            timer.delay_us(1);
+
+            pl::csr::eem_transceiver::comma_align_reset_write(1);
+            timer.delay_us(100);
+
+            if pl::csr::eem_transceiver::comma_read() == 1 {
+                debug!("comma alignment completed after {} bitslips", slip);
+                return;
+            }
+        }
+
+        error!("comma alignment failed, retrying in 1s...");
+        timer.delay_us(1_000_000);
+    }
+}
+
+pub unsafe fn align_wordslip(timer: &mut GlobalTimer, trx_no: u8) -> bool {
+    pl::csr::eem_transceiver::transceiver_sel_write(trx_no);
+
+    for slip in 0..=1 {
+        pl::csr::eem_transceiver::wordslip_write(slip as u8);
+        timer.delay_us(1);
+        pl::csr::eem_transceiver::comma_align_reset_write(1);
+        timer.delay_us(100);
+
+        if pl::csr::eem_transceiver::comma_read() == 1 {
+            debug!("comma alignment completed with {} wordslip", slip);
+            return true;
+        }
+    }
+
+    false
+}
+
+pub fn init(timer: &mut GlobalTimer, cfg: &Config) {
+    for trx_no in 0..pl::csr::CONFIG_EEM_DRTIO_COUNT {
+        unsafe {
+            pl::csr::eem_transceiver::transceiver_sel_write(trx_no as u8);
+        }
+
+        let key = format!("eem_drtio_delay{}", trx_no);
+
+        let cfg_read = cfg.read(&key);
+        match cfg_read {
+            Ok(record) => {
+                info!("loading calibrated timing values from sd card");
+                unsafe {
+                    apply_config(&*(record.as_ptr() as *const SerdesConfig), timer);
+                }
+            }
+            Err(_) => {
+                info!("calibrating...");
+                let config = unsafe { assign_delay(timer) };
+
+                match cfg.write(&key, config.as_bytes().to_vec()) {
+                    Ok(()) => {
+                        info!("storing calibration timing values into sd card");
+                    }
+                    Err(e) => {
+                        error!(
+                            "calibration successful but calibration timing values cannot be stored into sd card. \
+                             Error:{}",
+                            e
+                        );
+                    }
+                };
+            }
+        }
+
+        unsafe {
+            align_comma(timer);
+        }
+    }
+}

src/libboard_artiq/src/drtio_routing.rs

@@ -0,0 +1,107 @@
+use core::fmt;
+
+use libconfig::Config;
+use log::{info, warn};
+
+#[cfg(has_drtio_routing)]
+use crate::pl::csr;
+
+#[cfg(has_drtio_routing)]
+pub const DEST_COUNT: usize = 256;
+#[cfg(not(has_drtio_routing))]
+pub const DEST_COUNT: usize = 0;
+pub const MAX_HOPS: usize = 32;
+pub const INVALID_HOP: u8 = 0xff;
+
+pub struct RoutingTable(pub [[u8; MAX_HOPS]; DEST_COUNT]);
+
+impl RoutingTable {
+    // default routing table is for star topology with no repeaters
+    pub fn default_master(default_n_links: usize) -> RoutingTable {
+        let mut ret = RoutingTable([[INVALID_HOP; MAX_HOPS]; DEST_COUNT]);
+        let n_entries = default_n_links + 1; // include local RTIO
+        for i in 0..n_entries {
+            ret.0[i][0] = i as u8;
+        }
+        for i in 1..n_entries {
+            ret.0[i][1] = 0x00;
+        }
+        ret
+    }
+
+    // use this by default on satellite, as they receive
+    // the routing table from the master
+    pub fn default_empty() -> RoutingTable {
+        RoutingTable([[INVALID_HOP; MAX_HOPS]; DEST_COUNT])
+    }
+}
+
+impl fmt::Display for RoutingTable {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "RoutingTable {{")?;
+        for i in 0..DEST_COUNT {
+            if self.0[i][0] != INVALID_HOP {
+                write!(f, " {}:", i)?;
+                for j in 0..MAX_HOPS {
+                    if self.0[i][j] == INVALID_HOP {
+                        break;
+                    }
+                    write!(f, " {}", self.0[i][j])?;
+                }
+                write!(f, ";")?;
+            }
+        }
+        write!(f, " }}")?;
+        Ok(())
+    }
+}
+
+pub fn config_routing_table(default_n_links: usize, cfg: &Config) -> RoutingTable {
+    let mut ret = RoutingTable::default_master(default_n_links);
+    if let Ok(data) = cfg.read("routing_table") {
+        if data.len() == DEST_COUNT * MAX_HOPS {
+            for i in 0..DEST_COUNT {
+                for j in 0..MAX_HOPS {
+                    ret.0[i][j] = data[i * MAX_HOPS + j];
+                }
+            }
+        } else {
+            warn!("length of the configured routing table is incorrect, using default");
+        }
+    } else {
+        info!("could not read routing table from configuration, using default");
+    }
+    info!("routing table: {}", ret);
+    ret
+}
+
+#[cfg(has_drtio_routing)]
+pub fn interconnect_enable(routing_table: &RoutingTable, rank: u8, destination: u8) {
+    let hop = routing_table.0[destination as usize][rank as usize];
+    unsafe {
+        csr::routing_table::destination_write(destination);
+        csr::routing_table::hop_write(hop);
+    }
+}
+
+#[cfg(has_drtio_routing)]
+pub fn interconnect_disable(destination: u8) {
+    unsafe {
+        csr::routing_table::destination_write(destination);
+        csr::routing_table::hop_write(INVALID_HOP);
+    }
+}
+
+#[cfg(has_drtio_routing)]
+pub fn interconnect_enable_all(routing_table: &RoutingTable, rank: u8) {
+    for i in 0..DEST_COUNT {
+        interconnect_enable(routing_table, rank, i as u8);
+    }
+}
+
+#[cfg(has_drtio_routing)]
+pub fn interconnect_disable_all() {
+    for i in 0..DEST_COUNT {
+        interconnect_disable(i as u8);
+    }
+}

src/libboard_artiq/src/drtioaux.rs

@@ -0,0 +1,142 @@
+use core::slice;
+
+use core_io::{Error as IoError, ErrorKind as IoErrorKind};
+use crc;
+use io::{proto::{ProtoRead, ProtoWrite},
+         Cursor};
+use libboard_zynq::{time::Milliseconds, timer::GlobalTimer};
+
+pub use crate::drtioaux_proto::Packet;
+use crate::{drtioaux_proto::Error as ProtocolError, mem::mem::DRTIOAUX_MEM, pl::csr::DRTIOAUX};
+
+#[derive(Debug)]
+pub enum Error {
+    GatewareError,
+    CorruptedPacket,
+
+    LinkDown,
+    TimedOut,
+    UnexpectedReply,
+
+    RoutingError,
+
+    Protocol(ProtocolError),
+}
+
+impl From<ProtocolError> for Error {
+    fn from(value: ProtocolError) -> Error {
+        Error::Protocol(value)
+    }
+}
+
+impl From<IoError> for Error {
+    fn from(value: IoError) -> Error {
+        Error::Protocol(ProtocolError::Io(value))
+    }
+}
+
+pub fn reset(linkno: u8) {
+    let linkno = linkno as usize;
+    unsafe {
+        // clear buffer first to limit race window with buffer overflow
+        // error. We assume the CPU is fast enough so that no two packets
+        // will be received between the buffer and the error flag are cleared.
+        (DRTIOAUX[linkno].aux_rx_present_write)(1);
+        (DRTIOAUX[linkno].aux_rx_error_write)(1);
+    }
+}
+
+pub fn has_rx_error(linkno: u8) -> bool {
+    let linkno = linkno as usize;
+    unsafe {
+        let error = (DRTIOAUX[linkno].aux_rx_error_read)() != 0;
+        if error {
+            (DRTIOAUX[linkno].aux_rx_error_write)(1)
+        }
+        error
+    }
+}
+
+fn receive<F, T>(linkno: u8, f: F) -> Result<Option<T>, Error>
+where F: FnOnce(&[u8]) -> Result<T, Error> {
+    let linkidx = linkno as usize;
+    unsafe {
+        if (DRTIOAUX[linkidx].aux_rx_present_read)() == 1 {
+            let read_ptr = (DRTIOAUX[linkidx].aux_read_pointer_read)() as usize;
+            let ptr = (DRTIOAUX_MEM[linkidx].base + DRTIOAUX_MEM[linkidx].size / 2 + read_ptr * 0x400) as *mut u32;
+            let result = f(slice::from_raw_parts(ptr as *mut u8, 0x400 as usize));
+            (DRTIOAUX[linkidx].aux_rx_present_write)(1);
+            Ok(Some(result?))
+        } else {
+            Ok(None)
+        }
+    }
+}
+
+pub fn recv(linkno: u8) -> Result<Option<Packet>, Error> {
+    if has_rx_error(linkno) {
+        return Err(Error::GatewareError);
+    }
+
+    receive(linkno, |buffer| {
+        if buffer.len() < 8 {
+            return Err(IoError::new(IoErrorKind::UnexpectedEof, "Unexpected end").into());
+        }
+
+        let mut reader = Cursor::new(buffer);
+
+        let packet = Packet::read_from(&mut reader)?;
+        let padding = (12 - (reader.position() % 8)) % 8;
+        let checksum_at = reader.position() + padding;
+        let checksum = crc::crc32::checksum_ieee(&reader.get_ref()[0..checksum_at]);
+        reader.set_position(checksum_at);
+        if reader.read_u32()? != checksum {
+            return Err(Error::CorruptedPacket);
+        }
+        Ok(packet)
+    })
+}
+
+pub fn recv_timeout(linkno: u8, timeout_ms: Option<u64>, timer: GlobalTimer) -> Result<Packet, Error> {
+    let timeout_ms = Milliseconds(timeout_ms.unwrap_or(10));
+    let limit = timer.get_time() + timeout_ms;
+    while timer.get_time() < limit {
+        match recv(linkno)? {
+            None => (),
+            Some(packet) => return Ok(packet),
+        }
+    }
+    Err(Error::TimedOut)
+}
+
+fn transmit<F>(linkno: u8, f: F) -> Result<(), Error>
+where F: FnOnce(&mut [u8]) -> Result<usize, Error> {
+    let linkno = linkno as usize;
+    unsafe {
+        while (DRTIOAUX[linkno].aux_tx_read)() != 0 {}
+        let ptr = DRTIOAUX_MEM[linkno].base as *mut u32;
+        let len = f(slice::from_raw_parts_mut(ptr as *mut u8, 0x400 as usize))?;
+        (DRTIOAUX[linkno].aux_tx_length_write)(len as u16);
+        (DRTIOAUX[linkno].aux_tx_write)(1);
+        Ok(())
+    }
+}
+
+pub fn send(linkno: u8, packet: &Packet) -> Result<(), Error> {
+    transmit(linkno, |buffer| {
+        let mut writer = Cursor::new(buffer);
+
+        packet.write_to(&mut writer)?;
+
+        // Pad till offset 4, insert checksum there
+        let padding = (12 - (writer.position() % 8)) % 8;
+        for _ in 0..padding {
+            writer.write_u8(0)?;
+        }
+
+        let checksum = crc::crc32::checksum_ieee(&writer.get_ref()[0..writer.position()]);
+        writer.write_u32(checksum)?;
+
+        Ok(writer.position())
+    })
+}

src/libboard_artiq/src/drtioaux_async.rs

@@ -0,0 +1,130 @@
+use core::slice;
+
+use core_io::{Error as IoError, ErrorKind as IoErrorKind};
+use crc;
+use io::{proto::{ProtoRead, ProtoWrite},
+         Cursor};
+use libasync::{block_async, task};
+use libboard_zynq::{time::Milliseconds, timer::GlobalTimer};
+use nb;
+use void::Void;
+
+pub use crate::drtioaux_proto::Packet;
+use crate::{drtioaux::{has_rx_error, Error},
+            mem::mem::DRTIOAUX_MEM,
+            pl::csr::DRTIOAUX};
+
+pub async fn reset(linkno: u8) {
+    let linkno = linkno as usize;
+    unsafe {
+        // clear buffer first to limit race window with buffer overflow
+        // error. We assume the CPU is fast enough so that no two packets
+        // will be received between the buffer and the error flag are cleared.
+        (DRTIOAUX[linkno].aux_rx_present_write)(1);
+        (DRTIOAUX[linkno].aux_rx_error_write)(1);
+    }
+}
+
+fn tx_ready(linkno: usize) -> nb::Result<(), Void> {
+    unsafe {
+        if (DRTIOAUX[linkno].aux_tx_read)() != 0 {
+            Err(nb::Error::WouldBlock)
+        } else {
+            Ok(())
+        }
+    }
+}
+
+async fn receive<F, T>(linkno: u8, f: F) -> Result<Option<T>, Error>
+where F: FnOnce(&[u8]) -> Result<T, Error> {
+    let linkidx = linkno as usize;
+    unsafe {
+        if (DRTIOAUX[linkidx].aux_rx_present_read)() == 1 {
+            let read_ptr = (DRTIOAUX[linkidx].aux_read_pointer_read)() as usize;
+            let ptr = (DRTIOAUX_MEM[linkidx].base + DRTIOAUX_MEM[linkidx].size / 2 + read_ptr * 0x400) as *mut u32;
+            let result = f(slice::from_raw_parts(ptr as *mut u8, 0x400 as usize));
+            (DRTIOAUX[linkidx].aux_rx_present_write)(1);
+            Ok(Some(result?))
+        } else {
+            Ok(None)
+        }
+    }
+}
+
+pub async fn recv(linkno: u8) -> Result<Option<Packet>, Error> {
+    if has_rx_error(linkno) {
+        return Err(Error::GatewareError);
+    }
+
+    receive(linkno, |buffer| {
+        if buffer.len() < 8 {
+            return Err(IoError::new(IoErrorKind::UnexpectedEof, "Unexpected end").into());
+        }
+
+        let mut reader = Cursor::new(buffer);
+
+        let packet = Packet::read_from(&mut reader)?;
+        let padding = (12 - (reader.position() % 8)) % 8;
+        let checksum_at = reader.position() + padding;
+        let checksum = crc::crc32::checksum_ieee(&reader.get_ref()[0..checksum_at]);
+        reader.set_position(checksum_at);
+        if reader.read_u32()? != checksum {
+            return Err(Error::CorruptedPacket);
+        }
+        Ok(packet)
+    })
+    .await
+}
+
+pub async fn recv_timeout(linkno: u8, timeout_ms: Option<u64>, timer: GlobalTimer) -> Result<Packet, Error> {
+    let timeout_ms = Milliseconds(timeout_ms.unwrap_or(10));
+    let limit = timer.get_time() + timeout_ms;
+    let mut would_block = false;
+    while timer.get_time() < limit {
+        // to ensure one last time recv would run one last time
+        // in case async would return after timeout
+        if would_block {
+            task::r#yield().await;
+        }
+        match recv(linkno).await? {
+            None => {
+                would_block = true;
+            }
+            Some(packet) => return Ok(packet),
+        }
+    }
+    Err(Error::TimedOut)
+}
+
+async fn transmit<F>(linkno: u8, f: F) -> Result<(), Error>
+where F: FnOnce(&mut [u8]) -> Result<usize, Error> {
+    let linkno = linkno as usize;
+    unsafe {
+        let _ = block_async!(tx_ready(linkno)).await;
+        let ptr = DRTIOAUX_MEM[linkno].base as *mut u32;
+        let len = f(slice::from_raw_parts_mut(ptr as *mut u8, 0x400 as usize))?;
+        (DRTIOAUX[linkno].aux_tx_length_write)(len as u16);
+        (DRTIOAUX[linkno].aux_tx_write)(1);
+        Ok(())
+    }
+}
+
+pub async fn send(linkno: u8, packet: &Packet) -> Result<(), Error> {
+    transmit(linkno, |buffer| {
+        let mut writer = Cursor::new(buffer);
+
+        packet.write_to(&mut writer)?;
+
+        // Pad till offset 4, insert checksum there
+        let padding = (12 - (writer.position() % 8)) % 8;
+        for _ in 0..padding {
+            writer.write_u8(0)?;
+        }
+
+        let checksum = crc::crc32::checksum_ieee(&writer.get_ref()[0..writer.position()]);
+        writer.write_u32(checksum)?;
+
+        Ok(writer.position())
+    })
+    .await
+}

src/libboard_artiq/src/fiq.rs

@@ -0,0 +1,22 @@
+use libboard_zynq::{println, stdio};
+use libcortex_a9::{interrupt_handler, regs::MPIDR};
+use libregister::RegisterR;
+
+#[cfg(has_si549)]
+use crate::si549;
+
+interrupt_handler!(FIQ, fiq, __irq_stack0_start, __irq_stack1_start, {
+    match MPIDR.read().cpu_id() {
+        0 => {
+            // nFIQ is driven directly and bypass GIC
+            #[cfg(has_si549)]
+            si549::wrpll::interrupt_handler();
+            return;
+        }
+        _ => {}
+    };
+
+    stdio::drop_uart();
+    println!("FIQ");
+    loop {}
+});

src/libboard_artiq/src/grabber.rs

@@ -0,0 +1,163 @@
+use log::info;
+
+use crate::pl::csr;
+
+#[derive(PartialEq, Clone, Copy)]
+enum State {
+    Reset,
+    ExitReset,
+    Lock,
+    Align,
+    Watch,
+}
+
+#[derive(Clone, Copy)]
+struct Info {
+    state: State,
+    frame_size: (u16, u16),
+}
+
+static mut INFO: [Info; csr::GRABBER_LEN] = [Info {
+    state: State::Reset,
+    frame_size: (0, 0),
+}; csr::GRABBER_LEN];
+
+fn get_pll_reset(g: usize) -> bool {
+    unsafe { (csr::GRABBER[g].pll_reset_read)() != 0 }
+}
+
+fn set_pll_reset(g: usize, reset: bool) {
+    let val = if reset { 1 } else { 0 };
+    unsafe { (csr::GRABBER[g].pll_reset_write)(val) }
+}
+
+fn pll_locked(g: usize) -> bool {
+    unsafe { (csr::GRABBER[g].pll_locked_read)() != 0 }
+}
+
+fn clock_pattern_ok(g: usize) -> bool {
+    unsafe { (csr::GRABBER[g].clk_sampled_read)() == 0b1100011 }
+}
+
+fn clock_pattern_ok_filter(g: usize) -> bool {
+    for _ in 0..128 {
+        if !clock_pattern_ok(g) {
+            return false;
+        }
+    }
+    true
+}
+
+fn phase_shift(g: usize, direction: u8) {
+    unsafe {
+        (csr::GRABBER[g].phase_shift_write)(direction);
+        while (csr::GRABBER[g].phase_shift_done_read)() == 0 {}
+    }
+}
+
+fn clock_align(g: usize) -> bool {
+    while clock_pattern_ok_filter(g) {
+        phase_shift(g, 1);
+    }
+    phase_shift(g, 1);
+
+    let mut count = 0;
+    while !clock_pattern_ok_filter(g) {
+        phase_shift(g, 1);
+        count += 1;
+        if count > 1024 {
+            return false;
+        }
+    }
+
+    let mut window = 1;
+    phase_shift(g, 1);
+    while clock_pattern_ok_filter(g) {
+        phase_shift(g, 1);
+        window += 1;
+    }
+
+    for _ in 0..window / 2 {
+        phase_shift(g, 0);
+    }
+
+    true
+}
+
+fn get_last_pixels(g: usize) -> (u16, u16) {
+    unsafe { ((csr::GRABBER[g].last_x_read)(), (csr::GRABBER[g].last_y_read)()) }
+}
+
+fn get_video_clock(g: usize) -> u32 {
+    let freq_count = unsafe { (csr::GRABBER[g].freq_count_read)() } as u32;
+    2 * freq_count * (csr::CONFIG_CLOCK_FREQUENCY / 1000) / (511 * 1000)
+}
+
+pub fn tick() {
+    for g in 0..csr::GRABBER.len() {
+        let next = match unsafe { INFO[g].state } {
+            State::Reset => {
+                set_pll_reset(g, true);
+                unsafe {
+                    INFO[g].frame_size = (0, 0);
+                }
+                State::ExitReset
+            }
+            State::ExitReset => {
+                if get_pll_reset(g) {
+                    set_pll_reset(g, false);
+                    State::Lock
+                } else {
+                    State::ExitReset
+                }
+            }
+            State::Lock => {
+                if pll_locked(g) {
+                    info!("grabber{} locked: {}MHz", g, get_video_clock(g));
+                    State::Align
+                } else {
+                    State::Lock
+                }
+            }
+            State::Align => {
+                if pll_locked(g) {
+                    if clock_align(g) {
+                        info!("grabber{} alignment success", g);
+                        State::Watch
+                    } else {
+                        info!("grabber{} alignment failure", g);
+                        State::Reset
+                    }
+                } else {
+                    info!("grabber{} lock lost", g);
+                    State::Reset
+                }
+            }
+            State::Watch => {
+                if pll_locked(g) {
+                    if clock_pattern_ok(g) {
+                        let last_xy = get_last_pixels(g);
+                        if last_xy != unsafe { INFO[g].frame_size } {
+                            // x capture is on ~LVAL which is after
+                            // the last increment on DVAL
+                            // y capture is on ~FVAL which coincides with the
+                            // last increment on ~LVAL
+                            info!("grabber{} frame size: {}x{}", g, last_xy.0, last_xy.1 + 1);
+                            unsafe { INFO[g].frame_size = last_xy }
+                        }
+                        State::Watch
+                    } else {
+                        info!("grabber{} alignment lost", g);
+                        State::Reset
+                    }
+                } else {
+                    info!("grabber{} lock lost", g);
+                    State::Reset
+                }
+            }
+        };
+        unsafe {
+            INFO[g].state = next;
+        }
+    }
+}

src/libboard_artiq/src/io_expander.rs

@@ -0,0 +1,183 @@
+use libboard_zynq::i2c;
+use log::info;
+
+#[cfg(has_virtual_leds)]
+use crate::pl::csr;
+
+// Only the bare minimum registers. Bits/IO connections equivalent between IC types.
+struct Registers {
+    // PCA9539 equivalent register names in comments
+    iodira: u8, // Configuration Port 0
+    iodirb: u8, // Configuration Port 1
+    gpioa: u8,  // Output Port 0
+    gpiob: u8,  // Output Port 1
+}
+
+//IO expanders pins
+const IODIR_OUT_SFP_TX_DISABLE: u8 = 0x02;
+const IODIR_OUT_SFP_LED: u8 = 0x40;
+#[cfg(hw_rev = "v1.0")]
+const IODIR_OUT_SFP0_LED: u8 = 0x40;
+#[cfg(hw_rev = "v1.1")]
+const IODIR_OUT_SFP0_LED: u8 = 0x80;
+#[cfg(has_si549)]
+const IODIR_CLK_SEL: u8 = 0x80; // out
+#[cfg(has_si5324)]
+const IODIR_CLK_SEL: u8 = 0x00; // in
+
+//IO expander port direction
+const IODIR0: [u8; 2] = [
+    0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP0_LED,
+    0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP_LED & !IODIR_CLK_SEL,
+];
+
+const IODIR1: [u8; 2] = [
+    0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP_LED,
+    0xFF & !IODIR_OUT_SFP_TX_DISABLE & !IODIR_OUT_SFP_LED,
+];
+
+pub struct IoExpander {
+    address: u8,
+    #[cfg(has_virtual_leds)]
+    virtual_led_mapping: &'static [(u8, u8, u8)],
+    iodir: [u8; 2],
+    out_current: [u8; 2],
+    out_target: [u8; 2],
+    registers: Registers,
+}
+
+impl IoExpander {
+    pub fn new(i2c: &mut i2c::I2c, index: u8) -> Result<Self, &'static str> {
+        #[cfg(all(hw_rev = "v1.0", has_virtual_leds))]
+        const VIRTUAL_LED_MAPPING0: [(u8, u8, u8); 2] = [(0, 0, 6), (1, 1, 6)];
+        #[cfg(all(hw_rev = "v1.1", has_virtual_leds))]
+        const VIRTUAL_LED_MAPPING0: [(u8, u8, u8); 2] = [(0, 0, 7), (1, 1, 6)];
+        #[cfg(has_virtual_leds)]
+        const VIRTUAL_LED_MAPPING1: [(u8, u8, u8); 2] = [(2, 0, 6), (3, 1, 6)];
+
+        // Both expanders on SHARED I2C bus
+        let mut io_expander = match index {
+            0 => IoExpander {
+                address: 0x40,
+                #[cfg(has_virtual_leds)]
+                virtual_led_mapping: &VIRTUAL_LED_MAPPING0,
+                iodir: IODIR0,
+                out_current: [0; 2],
+                out_target: [0; 2],
+                registers: Registers {
+                    iodira: 0x00,
+                    iodirb: 0x01,
+                    gpioa: 0x12,
+                    gpiob: 0x13,
+                },
+            },
+            1 => IoExpander {
+                address: 0x42,
+                #[cfg(has_virtual_leds)]
+                virtual_led_mapping: &VIRTUAL_LED_MAPPING1,
+                iodir: IODIR1,
+                out_current: [0; 2],
+                out_target: [0; 2],
+                registers: Registers {
+                    iodira: 0x00,
+                    iodirb: 0x01,
+                    gpioa: 0x12,
+                    gpiob: 0x13,
+                },
+            },
+            _ => return Err("incorrect I/O expander index"),
+        };
+        if !io_expander.check_ack(i2c)? {
+            info!("MCP23017 io expander {} not found. Checking for PCA9539.", index);
+            io_expander.address += 0xa8; // translate to PCA9539 addresses (see schematic)
+            io_expander.registers = Registers {
+                iodira: 0x06,
+                iodirb: 0x07,
+                gpioa: 0x02,
+                gpiob: 0x03,
+            };
+            if !io_expander.check_ack(i2c)? {
+                return Err("Neither MCP23017 nor PCA9539 io expander found.");
+            };
+        }
+        Ok(io_expander)
+    }
+
+    fn select(&self, i2c: &mut i2c::I2c) -> Result<(), &'static str> {
+        i2c.pca954x_select(0x70, None)?;
+        i2c.pca954x_select(0x71, Some(3))?;
+        Ok(())
+    }
+
+    fn write(&self, i2c: &mut i2c::I2c, addr: u8, value: u8) -> Result<(), &'static str> {
+        i2c.start()?;
+        i2c.write(self.address)?;
+        i2c.write(addr)?;
+        i2c.write(value)?;
+        i2c.stop()?;
+        Ok(())
+    }
+
+    fn check_ack(&self, i2c: &mut i2c::I2c) -> Result<bool, &'static str> {
+        // Check for ack from io expander
+        self.select(i2c)?;
+        i2c.start()?;
+        let ack = i2c.write(self.address)?;
+        i2c.stop()?;
+        Ok(ack)
+    }
+
+    fn update_iodir(&self, i2c: &mut i2c::I2c) -> Result<(), &'static str> {
+        self.write(i2c, self.registers.iodira, self.iodir[0])?;
+        self.write(i2c, self.registers.iodirb, self.iodir[1])?;
+        Ok(())
+    }
+
+    pub fn init(&mut self, i2c: &mut i2c::I2c) -> Result<(), &'static str> {
+        self.select(i2c)?;
+
+        self.update_iodir(i2c)?;
+
+        self.out_current[0] = 0x00;
+        self.write(i2c, self.registers.gpioa, 0x00)?;
+        self.out_current[1] = 0x00;
+        self.write(i2c, self.registers.gpiob, 0x00)?;
+        Ok(())
+    }
+
+    pub fn set_oe(&mut self, i2c: &mut i2c::I2c, port: u8, outputs: u8) -> Result<(), &'static str> {
+        self.iodir[port as usize] &= !outputs;
+        self.update_iodir(i2c)?;
+        Ok(())
+    }
+
+    pub fn set(&mut self, port: u8, bit: u8, high: bool) {
+        if high {
+            self.out_target[port as usize] |= 1 << bit;
+        } else {
+            self.out_target[port as usize] &= !(1 << bit);
+        }
+    }
+
+    pub fn service(&mut self, i2c: &mut i2c::I2c) -> Result<(), &'static str> {
+        #[cfg(has_virtual_leds)]
+        for (led, port, bit) in self.virtual_led_mapping.iter() {
+            let level = unsafe { csr::virtual_leds::status_read() >> led & 1 };
+            self.set(*port, *bit, level != 0);
+        }
+
+        if self.out_target != self.out_current {
+            self.select(i2c)?;
+            if self.out_target[0] != self.out_current[0] {
+                self.write(i2c, self.registers.gpioa, self.out_target[0])?;
+                self.out_current[0] = self.out_target[0];
+            }
+            if self.out_target[1] != self.out_current[1] {
+                self.write(i2c, self.registers.gpiob, self.out_target[1])?;
+                self.out_current[1] = self.out_target[1];
+            }
+        }
+
+        Ok(())
+    }
+}

src/libboard_artiq/src/lib.rs

@@ -0,0 +1,56 @@
+#![no_std]
+#![feature(never_type)]
+#![feature(naked_functions)]
+#![feature(asm)]
+
+extern crate core_io;
+extern crate crc;
+extern crate embedded_hal;
+extern crate io;
+extern crate libasync;
+extern crate libboard_zynq;
+extern crate libconfig;
+extern crate libcortex_a9;
+extern crate libregister;
+extern crate log;
+extern crate log_buffer;
+
+pub mod drtio_routing;
+#[cfg(has_drtio)]
+pub mod drtioaux;
+#[cfg(has_drtio)]
+pub mod drtioaux_async;
+pub mod drtioaux_proto;
+pub mod fiq;
+#[cfg(feature = "target_kasli_soc")]
+pub mod io_expander;
+pub mod logger;
+#[cfg(has_drtio)]
+#[rustfmt::skip]
+#[path = "../../../build/mem.rs"]
+pub mod mem;
+#[rustfmt::skip]
+#[path = "../../../build/pl.rs"]
+pub mod pl;
+#[cfg(has_drtio_eem)]
+pub mod drtio_eem;
+#[cfg(has_grabber)]
+pub mod grabber;
+#[cfg(has_si5324)]
+pub mod si5324;
+#[cfg(has_si549)]
+pub mod si549;
+use core::{cmp, str};
+
+pub fn identifier_read(buf: &mut [u8]) -> &str {
+    unsafe {
+        pl::csr::identifier::address_write(0);
+        let len = pl::csr::identifier::data_read();
+        let len = cmp::min(len, buf.len() as u8);
+        for i in 0..len {
+            pl::csr::identifier::address_write(1 + i);
+            buf[i as usize] = pl::csr::identifier::data_read();
+        }
+        str::from_utf8_unchecked(&buf[..len as usize])
+    }
+}

src/libboard_artiq/src/logger.rs

@@ -1,13 +1,13 @@
-use core::cell::Cell;
-use core::fmt::Write;
-use log::{Log, LevelFilter};
-use log_buffer::LogBuffer;
-use libcortex_a9::mutex::{Mutex, MutexGuard};
+use core::{cell::Cell, fmt::Write};
+
 use libboard_zynq::{println, timer::GlobalTimer};
+use libcortex_a9::mutex::{Mutex, MutexGuard};
+use log::{LevelFilter, Log};
+use log_buffer::LogBuffer;
 
 pub struct LogBufferRef<'a> {
-    buffer:        MutexGuard<'a, LogBuffer<&'static mut [u8]>>,
-    old_log_level: LevelFilter
+    buffer: MutexGuard<'a, LogBuffer<&'static mut [u8]>>,
+    old_log_level: LevelFilter,
 }
 
 impl<'a> LogBufferRef<'a> {
@@ -37,7 +37,7 @@ impl<'a> Drop for LogBufferRef<'a> {
 }
 
 pub struct BufferLogger {
-    buffer:      Mutex<LogBuffer<&'static mut [u8]>>,
+    buffer: Mutex<LogBuffer<&'static mut [u8]>>,
     uart_filter: Cell<LevelFilter>,
     buffer_filter: Cell<LevelFilter>,
 }
@@ -56,8 +56,7 @@ impl BufferLogger {
     pub fn register(self) {
         unsafe {
             LOGGER = Some(self);
-            log::set_logger(LOGGER.as_ref().unwrap())
-                .expect("global logger can only be initialized once");
+            log::set_logger(LOGGER.as_ref().unwrap()).expect("global logger can only be initialized once");
         }
     }
 
@@ -66,9 +65,7 @@ impl BufferLogger {
     }
 
     pub fn buffer<'a>(&'a self) -> Option<LogBufferRef<'a>> {
-        self.buffer
-            .try_lock()
-            .map(LogBufferRef::new)
+        self.buffer.try_lock().map(LogBufferRef::new)
     }
 
     pub fn uart_log_level(&self) -> LevelFilter {
@@ -99,25 +96,36 @@ impl Log for BufferLogger {
 
     fn log(&self, record: &log::Record) {
         if self.enabled(record.metadata()) {
-            let timestamp = unsafe {
-                GlobalTimer::get()
-            }.get_us().0;
-            let seconds   = timestamp / 1_000_000;
-            let micros    = timestamp % 1_000_000;
+            let timestamp = unsafe { GlobalTimer::get() }.get_us().0;
+            let seconds = timestamp / 1_000_000;
+            let micros = timestamp % 1_000_000;
 
             if record.level() <= self.buffer_log_level() {
                 let mut buffer = self.buffer.lock();
-                writeln!(buffer, "[{:6}.{:06}s] {:>5}({}): {}", seconds, micros,
-                         record.level(), record.target(), record.args()).unwrap();
+                writeln!(
+                    buffer,
+                    "[{:6}.{:06}s] {:>5}({}): {}",
+                    seconds,
+                    micros,
+                    record.level(),
+                    record.target(),
+                    record.args()
+                )
+                .unwrap();
             }
 
             if record.level() <= self.uart_log_level() {
-                println!("[{:6}.{:06}s] {:>5}({}): {}", seconds, micros,
-                         record.level(), record.target(), record.args());
+                println!(
+                    "[{:6}.{:06}s] {:>5}({}): {}",
+                    seconds,
+                    micros,
+                    record.level(),
+                    record.target(),
+                    record.args()
+                );
             }
         }
     }
 
-    fn flush(&self) {
-    }
+    fn flush(&self) {}
 }

src/libboard_artiq/src/si5324.rs

@@ -0,0 +1,362 @@
+use core::result;
+
+use embedded_hal::blocking::delay::DelayUs;
+use libboard_zynq::{i2c::I2c, time::Milliseconds, timer::GlobalTimer};
+use log::info;
+
+#[cfg(not(si5324_soft_reset))]
+use crate::pl::csr;
+
+type Result<T> = result::Result<T, &'static str>;
+
+const ADDRESS: u8 = 0x68;
+
+#[cfg(not(si5324_soft_reset))]
+fn hard_reset(timer: &mut GlobalTimer) {
+    unsafe {
+        csr::si5324_rst_n::out_write(0);
+    }
+    timer.delay_us(1_000);
+    unsafe {
+        csr::si5324_rst_n::out_write(1);
+    }
+    timer.delay_us(10_000);
+}
+
+// NOTE: the logical parameters DO NOT MAP to physical values written
+// into registers. They have to be mapped; see the datasheet.
+// DSPLLsim reports the logical parameters in the design summary, not
+// the physical register values.
+pub struct FrequencySettings {
+    pub n1_hs: u8,
+    pub nc1_ls: u32,
+    pub n2_hs: u8,
+    pub n2_ls: u32,
+    pub n31: u32,
+    pub n32: u32,
+    pub bwsel: u8,
+    pub crystal_as_ckin2: bool,
+}
+
+pub enum Input {
+    Ckin1,
+    Ckin2,
+}
+
+fn map_frequency_settings(settings: &FrequencySettings) -> Result<FrequencySettings> {
+    if settings.nc1_ls != 0 && (settings.nc1_ls % 2) == 1 {
+        return Err("NC1_LS must be 0 or even");
+    }
+    if settings.nc1_ls > (1 << 20) {
+        return Err("NC1_LS is too high");
+    }
+    if (settings.n2_ls % 2) == 1 {
+        return Err("N2_LS must be even");
+    }
+    if settings.n2_ls > (1 << 20) {
+        return Err("N2_LS is too high");
+    }
+    if settings.n31 > (1 << 19) {
+        return Err("N31 is too high");
+    }
+    if settings.n32 > (1 << 19) {
+        return Err("N32 is too high");
+    }
+    let r = FrequencySettings {
+        n1_hs: match settings.n1_hs {
+            4 => 0b000,
+            5 => 0b001,
+            6 => 0b010,
+            7 => 0b011,
+            8 => 0b100,
+            9 => 0b101,
+            10 => 0b110,
+            11 => 0b111,
+            _ => return Err("N1_HS has an invalid value"),
+        },
+        nc1_ls: settings.nc1_ls - 1,
+        n2_hs: match settings.n2_hs {
+            4 => 0b000,
+            5 => 0b001,
+            6 => 0b010,
+            7 => 0b011,
+            8 => 0b100,
+            9 => 0b101,
+            10 => 0b110,
+            11 => 0b111,
+            _ => return Err("N2_HS has an invalid value"),
+        },
+        n2_ls: settings.n2_ls - 1,
+        n31: settings.n31 - 1,
+        n32: settings.n32 - 1,
+        bwsel: settings.bwsel,
+        crystal_as_ckin2: settings.crystal_as_ckin2,
+    };
+    Ok(r)
+}
+
+fn write(i2c: &mut I2c, reg: u8, val: u8) -> Result<()> {
+    i2c.start().unwrap();
+    if !i2c.write(ADDRESS << 1).unwrap() {
+        return Err("Si5324 failed to ack write address");
+    }
+    if !i2c.write(reg).unwrap() {
+        return Err("Si5324 failed to ack register");
+    }
+    if !i2c.write(val).unwrap() {
+        return Err("Si5324 failed to ack value");
+    }
+    i2c.stop().unwrap();
+    Ok(())
+}
+
+#[allow(dead_code)]
+fn write_no_ack_value(i2c: &mut I2c, reg: u8, val: u8) -> Result<()> {
+    i2c.start().unwrap();
+    if !i2c.write(ADDRESS << 1).unwrap() {
+        return Err("Si5324 failed to ack write address");
+    }
+    if !i2c.write(reg).unwrap() {
+        return Err("Si5324 failed to ack register");
+    }
+    i2c.write(val).unwrap();
+    i2c.stop().unwrap();
+    Ok(())
+}
+
+fn read(i2c: &mut I2c, reg: u8) -> Result<u8> {
+    i2c.start().unwrap();
+    if !i2c.write(ADDRESS << 1).unwrap() {
+        return Err("Si5324 failed to ack write address");
+    }
+    if !i2c.write(reg).unwrap() {
+        return Err("Si5324 failed to ack register");
+    }
+    i2c.restart().unwrap();
+    if !i2c.write((ADDRESS << 1) | 1).unwrap() {
+        return Err("Si5324 failed to ack read address");
+    }
+    let val = i2c.read(false).unwrap();
+    i2c.stop().unwrap();
+    Ok(val)
+}
+
+fn rmw<F>(i2c: &mut I2c, reg: u8, f: F) -> Result<()>
+where F: Fn(u8) -> u8 {
+    let value = read(i2c, reg)?;
+    write(i2c, reg, f(value))?;
+    Ok(())
+}
+
+fn ident(i2c: &mut I2c) -> Result<u16> {
+    Ok(((read(i2c, 134)? as u16) << 8) | (read(i2c, 135)? as u16))
+}
+
+#[cfg(si5324_soft_reset)]
+fn soft_reset(i2c: &mut I2c, timer: &mut GlobalTimer) -> Result<()> {
+    let val = read(i2c, 136)?;
+    write_no_ack_value(i2c, 136, val | 0x80)?;
+    timer.delay_us(10_000);
+    Ok(())
+}
+
+fn has_xtal(i2c: &mut I2c) -> Result<bool> {
+    Ok((read(i2c, 129)? & 0x01) == 0) // LOSX_INT=0
+}
+
+fn has_ckin(i2c: &mut I2c, input: Input) -> Result<bool> {
+    match input {
+        Input::Ckin1 => Ok((read(i2c, 129)? & 0x02) == 0), // LOS1_INT=0
+        Input::Ckin2 => Ok((read(i2c, 129)? & 0x04) == 0), // LOS2_INT=0
+    }
+}
+
+fn locked(i2c: &mut I2c) -> Result<bool> {
+    Ok((read(i2c, 130)? & 0x01) == 0) // LOL_INT=0
+}
+
+fn monitor_lock(i2c: &mut I2c, timer: &mut GlobalTimer) -> Result<()> {
+    info!("waiting for Si5324 lock...");
+    let timeout = timer.get_time() + Milliseconds(20_000);
+    while !locked(i2c)? {
+        // Yes, lock can be really slow.
+        if timer.get_time() > timeout {
+            return Err("Si5324 lock timeout");
+        }
+    }
+    info!("  ...locked");
+    Ok(())
+}
+
+fn init(i2c: &mut I2c, timer: &mut GlobalTimer) -> Result<()> {
+    #[cfg(not(si5324_soft_reset))]
+    hard_reset(timer);
+
+    #[cfg(feature = "target_kasli_soc")]
+    {
+        i2c.pca954x_select(0x70, None)?;
+        i2c.pca954x_select(0x71, Some(3))?;
+    }
+    #[cfg(feature = "target_zc706")]
+    {
+        i2c.pca954x_select(0x74, Some(4))?;
+    }
+
+    if ident(i2c)? != 0x0182 {
+        return Err("Si5324 does not have expected product number");
+    }
+
+    #[cfg(si5324_soft_reset)]
+    soft_reset(i2c, timer)?;
+    Ok(())
+}
+
+pub fn bypass(i2c: &mut I2c, input: Input, timer: &mut GlobalTimer) -> Result<()> {
+    let cksel_reg = match input {
+        Input::Ckin1 => 0b00,
+        Input::Ckin2 => 0b01,
+    };
+    init(i2c, timer)?;
+    rmw(i2c, 21, |v| v & 0xfe)?; // CKSEL_PIN=0
+    rmw(i2c, 3, |v| (v & 0x3f) | (cksel_reg << 6))?; // CKSEL_REG
+    rmw(i2c, 4, |v| (v & 0x3f) | (0b00 << 6))?; // AUTOSEL_REG=b00
+    rmw(i2c, 6, |v| (v & 0xc0) | 0b111111)?; // SFOUT2_REG=b111 SFOUT1_REG=b111
+    rmw(i2c, 0, |v| (v & 0xfd) | 0x02)?; // BYPASS_REG=1
+    Ok(())
+}
+
+pub fn setup(i2c: &mut I2c, settings: &FrequencySettings, input: Input, timer: &mut GlobalTimer) -> Result<()> {
+    let s = map_frequency_settings(settings)?;
+    let cksel_reg = match input {
+        Input::Ckin1 => 0b00,
+        Input::Ckin2 => 0b01,
+    };
+
+    init(i2c, timer)?;
+    if settings.crystal_as_ckin2 {
+        rmw(i2c, 0, |v| v | 0x40)?; // FREE_RUN=1
+    }
+    rmw(i2c, 2, |v| (v & 0x0f) | (s.bwsel << 4))?;
+    rmw(i2c, 21, |v| v & 0xfe)?; // CKSEL_PIN=0
+    rmw(i2c, 3, |v| (v & 0x2f) | (cksel_reg << 6) | 0x10)?; // CKSEL_REG, SQ_ICAL=1
+    rmw(i2c, 4, |v| (v & 0x3f) | (0b00 << 6))?; // AUTOSEL_REG=b00
+    rmw(i2c, 6, |v| (v & 0xc0) | 0b111111)?; // SFOUT2_REG=b111 SFOUT1_REG=b111
+    write(i2c, 25, (s.n1_hs << 5) as u8)?;
+    write(i2c, 31, (s.nc1_ls >> 16) as u8)?;
+    write(i2c, 32, (s.nc1_ls >> 8) as u8)?;
+    write(i2c, 33, (s.nc1_ls) as u8)?;
+    write(i2c, 34, (s.nc1_ls >> 16) as u8)?; // write to NC2_LS as well
+    write(i2c, 35, (s.nc1_ls >> 8) as u8)?;
+    write(i2c, 36, (s.nc1_ls) as u8)?;
+    write(i2c, 40, (s.n2_hs << 5) as u8 | (s.n2_ls >> 16) as u8)?;
+    write(i2c, 41, (s.n2_ls >> 8) as u8)?;
+    write(i2c, 42, (s.n2_ls) as u8)?;
+    write(i2c, 43, (s.n31 >> 16) as u8)?;
+    write(i2c, 44, (s.n31 >> 8) as u8)?;
+    write(i2c, 45, (s.n31) as u8)?;
+    write(i2c, 46, (s.n32 >> 16) as u8)?;
+    write(i2c, 47, (s.n32 >> 8) as u8)?;
+    write(i2c, 48, (s.n32) as u8)?;
+    rmw(i2c, 137, |v| v | 0x01)?; // FASTLOCK=1
+    rmw(i2c, 136, |v| v | 0x40)?; // ICAL=1
+
+    if !has_xtal(i2c)? {
+        return Err("Si5324 misses XA/XB signal");
+    }
+    if !has_ckin(i2c, input)? {
+        return Err("Si5324 misses clock input signal");
+    }
+
+    monitor_lock(i2c, timer)?;
+    Ok(())
+}
+
+pub fn select_input(i2c: &mut I2c, input: Input, timer: &mut GlobalTimer) -> Result<()> {
+    let cksel_reg = match input {
+        Input::Ckin1 => 0b00,
+        Input::Ckin2 => 0b01,
+    };
+    rmw(i2c, 3, |v| (v & 0x3f) | (cksel_reg << 6))?;
+    if !has_ckin(i2c, input)? {
+        return Err("Si5324 misses clock input signal");
+    }
+    monitor_lock(i2c, timer)?;
+    Ok(())
+}
+
+#[cfg(has_siphaser)]
+pub mod siphaser {
+    use super::*;
+    use crate::pl::csr;
+
+    pub fn select_recovered_clock(i2c: &mut I2c, rc: bool, timer: &mut GlobalTimer) -> Result<()> {
+        let val = read(i2c, 3)?;
+        write(i2c, 3, (val & 0xdf) | (1 << 5))?; // DHOLD=1
+        unsafe {
+            csr::siphaser::switch_clocks_write(if rc { 1 } else { 0 });
+        }
+        let val = read(i2c, 3)?;
+        write(i2c, 3, (val & 0xdf) | (0 << 5))?; // DHOLD=0
+        monitor_lock(i2c, timer)?;
+        Ok(())
+    }
+
+    fn phase_shift(direction: u8, timer: &mut GlobalTimer) {
+        unsafe {
+            csr::siphaser::phase_shift_write(direction);
+            while csr::siphaser::phase_shift_done_read() == 0 {}
+        }
+        // wait for the Si5324 loop to stabilize
+        timer.delay_us(500);
+    }
+
+    fn has_error(timer: &mut GlobalTimer) -> bool {
+        unsafe {
+            csr::siphaser::error_write(1);
+        }
+        timer.delay_us(5_000);
+        unsafe { csr::siphaser::error_read() != 0 }
+    }
+
+    fn find_edge(target: bool, timer: &mut GlobalTimer) -> Result<u32> {
+        let mut nshifts = 0;
+
+        let mut previous = has_error(timer);
+        loop {
+            phase_shift(1, timer);
+            nshifts += 1;
+            let current = has_error(timer);
+            if previous != target && current == target {
+                return Ok(nshifts);
+            }
+            if nshifts > 5000 {
+                return Err("failed to find timing error edge");
+            }
+            previous = current;
+        }
+    }
+
+    pub fn calibrate_skew(timer: &mut GlobalTimer) -> Result<()> {
+        let jitter_margin = 32;
+        let lead = find_edge(false, timer)?;
+        for _ in 0..jitter_margin {
+            phase_shift(1, timer);
+        }
+        let width = find_edge(true, timer)? + jitter_margin;
+        // width is 360 degrees (one full rotation of the phase between s/h limits) minus jitter
+        info!(
+            "calibration successful, lead: {}, width: {} ({}deg)",
+            lead,
+            width,
+            width * 360 / (56 * 8)
+        );
+
+        // Apply reverse phase shift for half the width to get into the
+        // middle of the working region.
+        for _ in 0..width / 2 {
+            phase_shift(0, timer);
+        }
+
+        Ok(())
+    }
+}

src/libboard_artiq/src/si549.rs

@@ -0,0 +1,854 @@
+use embedded_hal::prelude::_embedded_hal_blocking_delay_DelayUs;
+use libboard_zynq::timer::GlobalTimer;
+use log::info;
+
+use crate::pl::csr;
+
+#[cfg(feature = "target_kasli_soc")]
+const ADDRESS: u8 = 0x67;
+
+const ADPLL_MAX: i32 = (950.0 / 0.0001164) as i32;
+
+pub struct DividerConfig {
+    pub hsdiv: u16,
+    pub lsdiv: u8,
+    pub fbdiv: u64,
+}
+
+pub struct FrequencySetting {
+    pub main: DividerConfig,
+    pub helper: DividerConfig,
+}
+
+mod i2c {
+    use super::*;
+
+    #[derive(Clone, Copy)]
+    pub enum DCXO {
+        Main,
+        Helper,
+    }
+
+    fn half_period(timer: &mut GlobalTimer) {
+        timer.delay_us(1)
+    }
+
+    fn sda_i(dcxo: DCXO) -> bool {
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_sda_in_read() == 1 },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_sda_in_read() == 1 },
+        }
+    }
+
+    fn sda_oe(dcxo: DCXO, oe: bool) {
+        let val = if oe { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_sda_oe_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_sda_oe_write(val) },
+        };
+    }
+
+    fn sda_o(dcxo: DCXO, o: bool) {
+        let val = if o { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_sda_out_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_sda_out_write(val) },
+        };
+    }
+
+    fn scl_oe(dcxo: DCXO, oe: bool) {
+        let val = if oe { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_scl_oe_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_scl_oe_write(val) },
+        };
+    }
+
+    fn scl_o(dcxo: DCXO, o: bool) {
+        let val = if o { 1 } else { 0 };
+        match dcxo {
+            DCXO::Main => unsafe { csr::wrpll::main_dcxo_scl_out_write(val) },
+            DCXO::Helper => unsafe { csr::wrpll::helper_dcxo_scl_out_write(val) },
+        };
+    }
+
+    pub fn init(dcxo: DCXO, timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        // Set SCL as output, and high level
+        scl_o(dcxo, true);
+        scl_oe(dcxo, true);
+        // Prepare a zero level on SDA so that sda_oe pulls it down
+        sda_o(dcxo, false);
+        // Release SDA
+        sda_oe(dcxo, false);
+
+        // Check the I2C bus is ready
+        half_period(timer);
+        half_period(timer);
+        if !sda_i(dcxo) {
+            // Try toggling SCL a few times
+            for _bit in 0..8 {
+                scl_o(dcxo, false);
+                half_period(timer);
+                scl_o(dcxo, true);
+                half_period(timer);
+            }
+        }
+
+        if !sda_i(dcxo) {
+            return Err("SDA is stuck low and doesn't get unstuck");
+        }
+        Ok(())
+    }
+
+    pub fn start(dcxo: DCXO, timer: &mut GlobalTimer) {
+        // Set SCL high then SDA low
+        scl_o(dcxo, true);
+        half_period(timer);
+        sda_oe(dcxo, true);
+        half_period(timer);
+    }
+
+    pub fn stop(dcxo: DCXO, timer: &mut GlobalTimer) {
+        // First, make sure SCL is low, so that the target releases the SDA line
+        scl_o(dcxo, false);
+        half_period(timer);
+        // Set SCL high then SDA high
+        sda_oe(dcxo, true);
+        scl_o(dcxo, true);
+        half_period(timer);
+        sda_oe(dcxo, false);
+        half_period(timer);
+    }
+
+    pub fn write(dcxo: DCXO, data: u8, timer: &mut GlobalTimer) -> bool {
+        // MSB first
+        for bit in (0..8).rev() {
+            // Set SCL low and set our bit on SDA
+            scl_o(dcxo, false);
+            sda_oe(dcxo, data & (1 << bit) == 0);
+            half_period(timer);
+            // Set SCL high ; data is shifted on the rising edge of SCL
+            scl_o(dcxo, true);
+            half_period(timer);
+        }
+        // Check ack
+        // Set SCL low, then release SDA so that the I2C target can respond
+        scl_o(dcxo, false);
+        half_period(timer);
+        sda_oe(dcxo, false);
+        // Set SCL high and check for ack
+        scl_o(dcxo, true);
+        half_period(timer);
+        // returns true if acked (I2C target pulled SDA low)
+        !sda_i(dcxo)
+    }
+
+    pub fn read(dcxo: DCXO, ack: bool, timer: &mut GlobalTimer) -> u8 {
+        // Set SCL low first, otherwise setting SDA as input may cause a transition
+        // on SDA with SCL high which will be interpreted as START/STOP condition.
+        scl_o(dcxo, false);
+        half_period(timer); // make sure SCL has settled low
+        sda_oe(dcxo, false);
+
+        let mut data: u8 = 0;
+
+        // MSB first
+        for bit in (0..8).rev() {
+            scl_o(dcxo, false);
+            half_period(timer);
+            // Set SCL high and shift data
+            scl_o(dcxo, true);
+            half_period(timer);
+            if sda_i(dcxo) {
+                data |= 1 << bit
+            }
+        }
+        // Send ack
+        // Set SCL low and pull SDA low when acking
+        scl_o(dcxo, false);
+        if ack {
+            sda_oe(dcxo, true)
+        }
+        half_period(timer);
+        // then set SCL high
+        scl_o(dcxo, true);
+        half_period(timer);
+
+        data
+    }
+}
+
+fn write(dcxo: i2c::DCXO, reg: u8, val: u8, timer: &mut GlobalTimer) -> Result<(), &'static str> {
+    i2c::start(dcxo, timer);
+    if !i2c::write(dcxo, ADDRESS << 1, timer) {
+        return Err("Si549 failed to ack write address");
+    }
+    if !i2c::write(dcxo, reg, timer) {
+        return Err("Si549 failed to ack register");
+    }
+    if !i2c::write(dcxo, val, timer) {
+        return Err("Si549 failed to ack value");
+    }
+    i2c::stop(dcxo, timer);
+    Ok(())
+}
+
+fn read(dcxo: i2c::DCXO, reg: u8, timer: &mut GlobalTimer) -> Result<u8, &'static str> {
+    i2c::start(dcxo, timer);
+    if !i2c::write(dcxo, ADDRESS << 1, timer) {
+        return Err("Si549 failed to ack write address");
+    }
+    if !i2c::write(dcxo, reg, timer) {
+        return Err("Si549 failed to ack register");
+    }
+    i2c::stop(dcxo, timer);
+
+    i2c::start(dcxo, timer);
+    if !i2c::write(dcxo, (ADDRESS << 1) | 1, timer) {
+        return Err("Si549 failed to ack read address");
+    }
+    let val = i2c::read(dcxo, false, timer);
+    i2c::stop(dcxo, timer);
+    Ok(val)
+}
+
+fn setup(dcxo: i2c::DCXO, config: &DividerConfig, timer: &mut GlobalTimer) -> Result<(), &'static str> {
+    i2c::init(dcxo, timer)?;
+
+    write(dcxo, 255, 0x00, timer)?; // PAGE
+    write(dcxo, 69, 0x00, timer)?; // Disable FCAL override.
+    write(dcxo, 17, 0x00, timer)?; // Synchronously disable output
+
+    // The Si549 has no ID register, so we check that it responds correctly
+    // by writing values to a RAM-like register and reading them back.
+    for test_value in 0..255 {
+        write(dcxo, 23, test_value, timer)?;
+        let readback = read(dcxo, 23, timer)?;
+        if readback != test_value {
+            return Err("Si549 detection failed");
+        }
+    }
+
+    write(dcxo, 23, config.hsdiv as u8, timer)?;
+    write(dcxo, 24, (config.hsdiv >> 8) as u8 | (config.lsdiv << 4), timer)?;
+    write(dcxo, 26, config.fbdiv as u8, timer)?;
+    write(dcxo, 27, (config.fbdiv >> 8) as u8, timer)?;
+    write(dcxo, 28, (config.fbdiv >> 16) as u8, timer)?;
+    write(dcxo, 29, (config.fbdiv >> 24) as u8, timer)?;
+    write(dcxo, 30, (config.fbdiv >> 32) as u8, timer)?;
+    write(dcxo, 31, (config.fbdiv >> 40) as u8, timer)?;
+
+    write(dcxo, 7, 0x08, timer)?; // Start FCAL
+    timer.delay_us(30_000); // Internal FCAL VCO calibration
+    write(dcxo, 17, 0x01, timer)?; // Synchronously enable output
+
+    Ok(())
+}
+
+pub fn main_setup(timer: &mut GlobalTimer, settings: &FrequencySetting) -> Result<(), &'static str> {
+    unsafe {
+        csr::wrpll::main_dcxo_bitbang_enable_write(1);
+        csr::wrpll::main_dcxo_i2c_address_write(ADDRESS);
+    }
+
+    setup(i2c::DCXO::Main, &settings.main, timer)?;
+
+    // Si549 maximum settling time for large frequency change.
+    timer.delay_us(40_000);
+
+    unsafe {
+        csr::wrpll::main_dcxo_bitbang_enable_write(0);
+    }
+
+    info!("Main Si549 started");
+    Ok(())
+}
+
+pub fn helper_setup(timer: &mut GlobalTimer, settings: &FrequencySetting) -> Result<(), &'static str> {
+    unsafe {
+        csr::wrpll::helper_reset_write(1);
+        csr::wrpll::helper_dcxo_bitbang_enable_write(1);
+        csr::wrpll::helper_dcxo_i2c_address_write(ADDRESS);
+    }
+
+    setup(i2c::DCXO::Helper, &settings.helper, timer)?;
+
+    // Si549 maximum settling time for large frequency change.
+    timer.delay_us(40_000);
+
+    unsafe {
+        csr::wrpll::helper_reset_write(0);
+        csr::wrpll::helper_dcxo_bitbang_enable_write(0);
+    }
+    info!("Helper Si549 started");
+    Ok(())
+}
+
+fn set_adpll(dcxo: i2c::DCXO, adpll: i32) -> Result<(), &'static str> {
+    if adpll.abs() > ADPLL_MAX {
+        return Err("adpll is too large");
+    }
+
+    match dcxo {
+        i2c::DCXO::Main => unsafe {
+            if csr::wrpll::main_dcxo_bitbang_enable_read() == 1 {
+                return Err("Main si549 bitbang mode is active when using gateware i2c");
+            }
+
+            while csr::wrpll::main_dcxo_adpll_busy_read() == 1 {}
+            if csr::wrpll::main_dcxo_nack_read() == 1 {
+                return Err("Main si549 failed to ack adpll write");
+            }
+
+            csr::wrpll::main_dcxo_i2c_address_write(ADDRESS);
+            csr::wrpll::main_dcxo_adpll_write(adpll as u32);
+
+            csr::wrpll::main_dcxo_adpll_stb_write(1);
+        },
+        i2c::DCXO::Helper => unsafe {
+            if csr::wrpll::helper_dcxo_bitbang_enable_read() == 1 {
+                return Err("Helper si549 bitbang mode is active when using gateware i2c");
+            }
+
+            while csr::wrpll::helper_dcxo_adpll_busy_read() == 1 {}
+            if csr::wrpll::helper_dcxo_nack_read() == 1 {
+                return Err("Helper si549 failed to ack adpll write");
+            }
+
+            csr::wrpll::helper_dcxo_i2c_address_write(ADDRESS);
+            csr::wrpll::helper_dcxo_adpll_write(adpll as u32);
+
+            csr::wrpll::helper_dcxo_adpll_stb_write(1);
+        },
+    };
+
+    Ok(())
+}
+
+#[cfg(has_wrpll)]
+pub mod wrpll {
+
+    use super::*;
+
+    const BEATING_PERIOD: i32 = 0x8000;
+    const BEATING_HALFPERIOD: i32 = 0x4000;
+    const COUNTER_WIDTH: u32 = 24;
+    const DIV_WIDTH: u32 = 2;
+
+    // y[n] = b0*x[n] + b1*x[n-1] + b2*x[n-2] - a1*y[n-1] - a2*y[n-2]
+    struct FilterParameters {
+        pub b0: f64,
+        pub b1: f64,
+        pub b2: f64,
+        pub a1: f64,
+        pub a2: f64,
+    }
+
+    #[cfg(rtio_frequency = "100.0")]
+    const LPF: FilterParameters = FilterParameters {
+        b0: 0.03967479060647884,
+        b1: 0.07934958121295768,
+        b2: 0.03967479060647884,
+        a1: -1.3865593741228928,
+        a2: 0.5452585365488082,
+    };
+
+    #[cfg(rtio_frequency = "125.0")]
+    const LPF: FilterParameters = FilterParameters {
+        b0: 0.07209205036273991,
+        b1: 0.14418410072547982,
+        b2: 0.07209205036273991,
+        a1: -0.6114078511562919,
+        a2: -0.10022394739274834,
+    };
+
+    static mut H_ADPLL1: i32 = 0;
+    static mut H_ADPLL2: i32 = 0;
+    static mut PERIOD_ERR1: i32 = 0;
+    static mut PERIOD_ERR2: i32 = 0;
+
+    static mut M_ADPLL1: i32 = 0;
+    static mut M_ADPLL2: i32 = 0;
+    static mut PHASE_ERR1: i32 = 0;
+    static mut PHASE_ERR2: i32 = 0;
+
+    static mut BASE_ADPLL: i32 = 0;
+
+    #[derive(Clone, Copy)]
+    pub enum ISR {
+        RefTag,
+        MainTag,
+    }
+
+    mod tag_collector {
+        use super::*;
+
+        #[cfg(wrpll_ref_clk = "GT_CDR")]
+        static mut TAG_OFFSET: u32 = 8382;
+        #[cfg(wrpll_ref_clk = "SMA_CLKIN")]
+        static mut TAG_OFFSET: u32 = 0;
+        static mut REF_TAG: u32 = 0;
+        static mut REF_TAG_READY: bool = false;
+        static mut MAIN_TAG: u32 = 0;
+        static mut MAIN_TAG_READY: bool = false;
+
+        pub fn reset() {
+            clear_phase_diff_ready();
+            unsafe {
+                REF_TAG = 0;
+                MAIN_TAG = 0;
+            }
+        }
+
+        pub fn clear_phase_diff_ready() {
+            unsafe {
+                REF_TAG_READY = false;
+                MAIN_TAG_READY = false;
+            }
+        }
+
+        pub fn collect_tags(interrupt: ISR) {
+            match interrupt {
+                ISR::RefTag => unsafe {
+                    REF_TAG = csr::wrpll::ref_tag_read();
+                    REF_TAG_READY = true;
+                },
+                ISR::MainTag => unsafe {
+                    MAIN_TAG = csr::wrpll::main_tag_read();
+                    MAIN_TAG_READY = true;
+                },
+            }
+        }
+
+        pub fn phase_diff_ready() -> bool {
+            unsafe { REF_TAG_READY && MAIN_TAG_READY }
+        }
+
+        #[cfg(feature = "calibrate_wrpll_skew")]
+        pub fn set_tag_offset(offset: u32) {
+            unsafe {
+                TAG_OFFSET = offset;
+            }
+        }
+
+        #[cfg(feature = "calibrate_wrpll_skew")]
+        pub fn get_tag_offset() -> u32 {
+            unsafe { TAG_OFFSET }
+        }
+
+        pub fn get_period_error() -> i32 {
+            // n * BEATING_PERIOD - REF_TAG(n) mod BEATING_PERIOD
+            let mut period_error = unsafe { REF_TAG.overflowing_neg().0.rem_euclid(BEATING_PERIOD as u32) as i32 };
+            // mapping tags from [0, 2π] -> [-π, π]
+            if period_error > BEATING_HALFPERIOD {
+                period_error -= BEATING_PERIOD
+            }
+            period_error
+        }
+
+        pub fn get_phase_error() -> i32 {
+            // MAIN_TAG(n) - REF_TAG(n) - TAG_OFFSET mod BEATING_PERIOD
+            let mut phase_error = unsafe {
+                MAIN_TAG
+                    .overflowing_sub(REF_TAG + TAG_OFFSET)
+                    .0
+                    .rem_euclid(BEATING_PERIOD as u32) as i32
+            };
+
+            // mapping tags from [0, 2π] -> [-π, π]
+            if phase_error > BEATING_HALFPERIOD {
+                phase_error -= BEATING_PERIOD
+            }
+            phase_error
+        }
+    }
+
+    fn set_isr(en: bool) {
+        let val = if en { 1 } else { 0 };
+        unsafe {
+            csr::wrpll::ref_tag_ev_enable_write(val);
+            csr::wrpll::main_tag_ev_enable_write(val);
+        }
+    }
+
+    fn set_base_adpll() -> Result<(), &'static str> {
+        let count2adpll =
+            |error: i32| ((error as f64 * 1e6) / (0.0001164 * (1 << (COUNTER_WIDTH - DIV_WIDTH)) as f64)) as i32;
+
+        let (ref_count, main_count) = get_freq_counts();
+        unsafe {
+            BASE_ADPLL = count2adpll(ref_count as i32 - main_count as i32);
+            set_adpll(i2c::DCXO::Main, BASE_ADPLL)?;
+            set_adpll(i2c::DCXO::Helper, BASE_ADPLL)?;
+        }
+        Ok(())
+    }
+
+    fn get_freq_counts() -> (u32, u32) {
+        unsafe {
+            csr::wrpll::frequency_counter_update_write(1);
+            while csr::wrpll::frequency_counter_busy_read() == 1 {}
+            #[cfg(wrpll_ref_clk = "GT_CDR")]
+            let ref_count = csr::wrpll::frequency_counter_counter_rtio_rx0_read();
+            #[cfg(wrpll_ref_clk = "SMA_CLKIN")]
+            let ref_count = csr::wrpll::frequency_counter_counter_ref_read();
+            let main_count = csr::wrpll::frequency_counter_counter_sys_read();
+
+            (ref_count, main_count)
+        }
+    }
+
+    fn reset_plls(timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        unsafe {
+            H_ADPLL1 = 0;
+            H_ADPLL2 = 0;
+            PERIOD_ERR1 = 0;
+            PERIOD_ERR2 = 0;
+            M_ADPLL1 = 0;
+            M_ADPLL2 = 0;
+            PHASE_ERR1 = 0;
+            PHASE_ERR2 = 0;
+        }
+        set_adpll(i2c::DCXO::Main, 0)?;
+        set_adpll(i2c::DCXO::Helper, 0)?;
+        // wait for adpll to transfer and DCXO to settle
+        timer.delay_us(200);
+        Ok(())
+    }
+
+    fn clear_pending(interrupt: ISR) {
+        match interrupt {
+            ISR::RefTag => unsafe { csr::wrpll::ref_tag_ev_pending_write(1) },
+            ISR::MainTag => unsafe { csr::wrpll::main_tag_ev_pending_write(1) },
+        };
+    }
+
+    fn is_pending(interrupt: ISR) -> bool {
+        match interrupt {
+            ISR::RefTag => unsafe { csr::wrpll::ref_tag_ev_pending_read() == 1 },
+            ISR::MainTag => unsafe { csr::wrpll::main_tag_ev_pending_read() == 1 },
+        }
+    }
+
+    pub fn interrupt_handler() {
+        if is_pending(ISR::RefTag) {
+            tag_collector::collect_tags(ISR::RefTag);
+            clear_pending(ISR::RefTag);
+            helper_pll().expect("failed to run helper DCXO PLL");
+        }
+
+        if is_pending(ISR::MainTag) {
+            tag_collector::collect_tags(ISR::MainTag);
+            clear_pending(ISR::MainTag);
+        }
+
+        if tag_collector::phase_diff_ready() {
+            main_pll().expect("failed to run main DCXO PLL");
+            tag_collector::clear_phase_diff_ready();
+        }
+    }
+
+    fn helper_pll() -> Result<(), &'static str> {
+        let period_err = tag_collector::get_period_error();
+        unsafe {
+            let adpll = ((LPF.b0 * period_err as f64) + (LPF.b1 * PERIOD_ERR1 as f64) + (LPF.b2 * PERIOD_ERR2 as f64)
+                - (LPF.a1 * H_ADPLL1 as f64)
+                - (LPF.a2 * H_ADPLL2 as f64)) as i32;
+            set_adpll(i2c::DCXO::Helper, BASE_ADPLL + adpll)?;
+            H_ADPLL2 = H_ADPLL1;
+            PERIOD_ERR2 = PERIOD_ERR1;
+            H_ADPLL1 = adpll;
+            PERIOD_ERR1 = period_err;
+        };
+        Ok(())
+    }
+
+    fn main_pll() -> Result<(), &'static str> {
+        let phase_err = tag_collector::get_phase_error();
+        unsafe {
+            let adpll = ((LPF.b0 * phase_err as f64) + (LPF.b1 * PHASE_ERR1 as f64) + (LPF.b2 * PHASE_ERR2 as f64)
+                - (LPF.a1 * M_ADPLL1 as f64)
+                - (LPF.a2 * M_ADPLL2 as f64)) as i32;
+            set_adpll(i2c::DCXO::Main, BASE_ADPLL + adpll)?;
+            M_ADPLL2 = M_ADPLL1;
+            PHASE_ERR2 = PHASE_ERR1;
+            M_ADPLL1 = adpll;
+            PHASE_ERR1 = phase_err;
+        };
+        Ok(())
+    }
+
+    #[cfg(wrpll_ref_clk = "GT_CDR")]
+    fn test_skew(timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        // wait for PLL to stabilize
+        timer.delay_us(20_000);
+
+        info!("testing the skew of SYS CLK...");
+        if has_timing_error(timer) {
+            return Err("the skew cannot satisfy setup/hold time constraint of RX synchronizer");
+        }
+        info!("the skew of SYS CLK met the timing constraint");
+        Ok(())
+    }
+
+    #[cfg(wrpll_ref_clk = "GT_CDR")]
+    fn has_timing_error(timer: &mut GlobalTimer) -> bool {
+        unsafe {
+            csr::wrpll_skewtester::error_write(1);
+        }
+        timer.delay_us(5_000);
+        unsafe { csr::wrpll_skewtester::error_read() == 1 }
+    }
+
+    #[cfg(feature = "calibrate_wrpll_skew")]
+    fn find_edge(target: bool, timer: &mut GlobalTimer) -> Result<u32, &'static str> {
+        const STEP: u32 = 8;
+        const STABLE_THRESHOLD: u32 = 10;
+
+        enum FSM {
+            Init,
+            WaitEdge,
+            GotEdge,
+        }
+
+        let mut state: FSM = FSM::Init;
+        let mut offset: u32 = tag_collector::get_tag_offset();
+        let mut median_edge: u32 = 0;
+        let mut stable_counter: u32 = 0;
+
+        for _ in 0..(BEATING_PERIOD as u32 / STEP) as usize {
+            tag_collector::set_tag_offset(offset);
+            offset += STEP;
+            // wait for PLL to stabilize
+            timer.delay_us(20_000);
+
+            let error = has_timing_error(timer);
+            // A median edge deglitcher
+            match state {
+                FSM::Init => {
+                    if error != target {
+                        stable_counter += 1;
+                    } else {
+                        stable_counter = 0;
+                    }
+
+                    if stable_counter >= STABLE_THRESHOLD {
+                        state = FSM::WaitEdge;
+                        stable_counter = 0;
+                    }
+                }
+                FSM::WaitEdge => {
+                    if error == target {
+                        state = FSM::GotEdge;
+                        median_edge = offset;
+                    }
+                }
+                FSM::GotEdge => {
+                    if error != target {
+                        median_edge += STEP;
+                        stable_counter = 0;
+                    } else {
+                        stable_counter += 1;
+                    }
+
+                    if stable_counter >= STABLE_THRESHOLD {
+                        return Ok(median_edge);
+                    }
+                }
+            }
+        }
+        return Err("failed to find timing error edge");
+    }
+
+    #[cfg(feature = "calibrate_wrpll_skew")]
+    fn calibrate_skew(timer: &mut GlobalTimer) -> Result<(), &'static str> {
+        info!("calibrating skew to meet timing constraint...");
+
+        // clear calibrated value
+        tag_collector::set_tag_offset(0);
+        let rising = find_edge(true, timer)? as i32;
+        let falling = find_edge(false, timer)? as i32;
+
+        let width = BEATING_PERIOD - (falling - rising);
+        let result = falling + width / 2;
+        tag_collector::set_tag_offset(result as u32);
+
+        info!(
+            "calibration successful, error zone: {} -> {}, width: {} ({}deg), middle of working region: {}",
+            rising,
+            falling,
+            width,
+            360 * width / BEATING_PERIOD,
+            result,
+        );
+
+        Ok(())
+    }
+
+    pub fn select_recovered_clock(rc: bool, timer: &mut GlobalTimer) {
+        set_isr(false);
+
+        if rc {
+            tag_collector::reset();
+            reset_plls(timer).expect("failed to reset main and helper PLL");
+
+            // get within capture range
+            set_base_adpll().expect("failed to set base adpll");
+
+            // clear gateware pending flag
+            clear_pending(ISR::RefTag);
+            clear_pending(ISR::MainTag);
+
+            // use nFIQ to avoid IRQ being disabled by mutex lock and mess up PLL
+            set_isr(true);
+            info!("WRPLL interrupt enabled");
+
+            #[cfg(feature = "calibrate_wrpll_skew")]
+            calibrate_skew(timer).expect("failed to set the correct skew");
+
+            #[cfg(wrpll_ref_clk = "GT_CDR")]
+            test_skew(timer).expect("skew test failed");
+        }
+    }
+}
+
+#[cfg(has_wrpll_refclk)]
+pub mod wrpll_refclk {
+    use super::*;
+
+    pub struct MmcmSetting {
+        pub clkout0_reg1: u16,  //0x08
+        pub clkout0_reg2: u16,  //0x09
+        pub clkfbout_reg1: u16, //0x14
+        pub clkfbout_reg2: u16, //0x15
+        pub div_reg: u16,       //0x16
+        pub lock_reg1: u16,     //0x18
+        pub lock_reg2: u16,     //0x19
+        pub lock_reg3: u16,     //0x1A
+        pub power_reg: u16,     //0x28
+        pub filt_reg1: u16,     //0x4E
+        pub filt_reg2: u16,     //0x4F
+    }
+
+    fn one_clock_cycle() {
+        unsafe {
+            csr::wrpll_refclk::mmcm_dclk_write(1);
+            csr::wrpll_refclk::mmcm_dclk_write(0);
+        }
+    }
+
+    fn set_addr(address: u8) {
+        unsafe {
+            csr::wrpll_refclk::mmcm_daddr_write(address);
+        }
+    }
+
+    fn set_data(value: u16) {
+        unsafe {
+            csr::wrpll_refclk::mmcm_din_write(value);
+        }
+    }
+
+    fn set_enable(en: bool) {
+        unsafe {
+            let val = if en { 1 } else { 0 };
+            csr::wrpll_refclk::mmcm_den_write(val);
+        }
+    }
+
+    fn set_write_enable(en: bool) {
+        unsafe {
+            let val = if en { 1 } else { 0 };
+            csr::wrpll_refclk::mmcm_dwen_write(val);
+        }
+    }
+
+    fn get_data() -> u16 {
+        unsafe { csr::wrpll_refclk::mmcm_dout_read() }
+    }
+
+    fn drp_ready() -> bool {
+        unsafe { csr::wrpll_refclk::mmcm_dready_read() == 1 }
+    }
+
+    #[allow(dead_code)]
+    fn read(address: u8) -> u16 {
+        set_addr(address);
+        set_enable(true);
+        // Set DADDR on the mmcm and assert DEN for one clock cycle
+        one_clock_cycle();
+
+        set_enable(false);
+        while !drp_ready() {
+            // keep the clock signal until data is ready
+            one_clock_cycle();
+        }
+        get_data()
+    }
+
+    fn write(address: u8, value: u16) {
+        set_addr(address);
+        set_data(value);
+        set_write_enable(true);
+        set_enable(true);
+        // Set DADDR, DI on the mmcm and assert DWE, DEN for one clock cycle
+        one_clock_cycle();
+
+        set_write_enable(false);
+        set_enable(false);
+        while !drp_ready() {
+            // keep the clock signal until write is finished
+            one_clock_cycle();
+        }
+    }
+
+    fn reset(rst: bool) {
+        unsafe {
+            let val = if rst { 1 } else { 0 };
+            csr::wrpll_refclk::mmcm_reset_write(val)
+        }
+    }
+
+    pub fn setup(timer: &mut GlobalTimer, settings: MmcmSetting, mmcm_bypass: bool) -> Result<(), &'static str> {
+        unsafe {
+            csr::wrpll_refclk::refclk_reset_write(1);
+        }
+
+        if mmcm_bypass {
+            info!("Bypassing mmcm");
+            unsafe {
+                csr::wrpll_refclk::mmcm_bypass_write(1);
+            }
+        } else {
+            // Based on "DRP State Machine" from XAPP888
+            // hold reset HIGH during mmcm config
+            reset(true);
+            write(0x08, settings.clkout0_reg1);
+            write(0x09, settings.clkout0_reg2);
+            write(0x14, settings.clkfbout_reg1);
+            write(0x15, settings.clkfbout_reg2);
+            write(0x16, settings.div_reg);
+            write(0x18, settings.lock_reg1);
+            write(0x19, settings.lock_reg2);
+            write(0x1A, settings.lock_reg3);
+            write(0x28, settings.power_reg);
+            write(0x4E, settings.filt_reg1);
+            write(0x4F, settings.filt_reg2);
+            reset(false);
+
+            // wait for the mmcm to lock
+            timer.delay_us(100);
+
+            let locked = unsafe { csr::wrpll_refclk::mmcm_locked_read() == 1 };
+            if !locked {
+                return Err("mmcm failed to generate 125MHz ref clock from SMA CLKIN");
+            }
+        }
+
+        unsafe {
+            csr::wrpll_refclk::refclk_reset_write(0);
+        }
+
+        Ok(())
+    }
+}

src/libbuild_zynq/Cargo.toml

@@ -0,0 +1,8 @@
+[package]
+authors = ["M-Labs"]
+name = "build_zynq"
+version = "0.0.0"
+
+[lib]
+name = "build_zynq"
+path = "lib.rs"

src/libbuild_zynq/lib.rs

@@ -0,0 +1,29 @@
+use std::{env,
+          fs::File,
+          io::{BufRead, BufReader, Write},
+          path::PathBuf};
+
+pub fn add_linker_script() {
+    // Put the linker script somewhere the linker can find it
+    let out = &PathBuf::from(env::var_os("OUT_DIR").unwrap());
+    File::create(out.join("link.x"))
+        .unwrap()
+        .write_all(include_bytes!("link.x"))
+        .unwrap();
+    println!("cargo:rustc-link-search={}", out.display());
+
+    // Only re-run the build script when link.x is changed,
+    // instead of when any part of the source code changes.
+    println!("cargo:rerun-if-changed=link.x");
+}
+
+pub fn cfg() {
+    // Handle rustc-cfg file
+    let cfg_path = "../../build/rustc-cfg";
+    println!("cargo:rerun-if-changed={}", cfg_path);
+
+    let f = BufReader::new(File::open(cfg_path).unwrap());
+    for line in f.lines() {
+        println!("cargo:rustc-cfg={}", line.unwrap());
+    }
+}

src/libbuild_zynq/link.x

@@ -10,7 +10,9 @@ SECTIONS
     __text_start = .;
     .text :
     {
+        __exceptions_start = .;
         KEEP(*(.text.exceptions));
+        __exceptions_end = .;
         *(.text.boot);
         *(.text .text.*);
     } > SDRAM
@@ -69,4 +71,18 @@ SECTIONS
         . += 0x20000;
         __stack0_start = .;
     } > SDRAM
+
+    .irq_stack1 (NOLOAD) : ALIGN(8)
+    {
+        __irq_stack1_end = .;
+        . += 0x100;
+        __irq_stack1_start = .;
+    } > SDRAM
+
+    .irq_stack0 (NOLOAD) : ALIGN(8)
+    {
+        __irq_stack0_end = .;
+        . += 0x100;
+        __irq_stack0_start = .;
+    } > SDRAM
 }

src/libc/Cargo.toml.tpl

@@ -6,7 +6,7 @@ edition = "2018"
 build = "build.rs"
 
 [dependencies]
-libboard_zynq = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
+libboard_zynq = { path = "@@ZYNQ_RS@@/libboard_zynq" }
 
 [build-dependencies]
 cc = { version = "1.0.1" }

src/libc/build.rs

@@ -4,7 +4,8 @@ fn main() {
 }
 
 mod libc {
-    use std::path::Path;
+    use std::{env, path::Path};
+
     pub fn compile() {
         let cfg = &mut cc::Build::new();
         cfg.no_default_flags(true);
@@ -16,6 +17,9 @@ mod libc {
         cfg.flag("-ffreestanding");
         cfg.flag("-fno-PIC");
         cfg.flag("-isystem../include");
+        if let Ok(extra_include) = env::var("CLANG_EXTRA_INCLUDE_DIR") {
+            cfg.flag(&("-isystem".to_owned() + &extra_include));
+        }
         cfg.flag("-fno-stack-protector");
         cfg.flag("--target=armv7-none-eabihf");
         cfg.flag("-O2");
@@ -27,9 +31,7 @@ mod libc {
         cfg.flag("-U_FORTIFY_SOURCE");
         cfg.define("_FORTIFY_SOURCE", Some("0"));
 
-        let sources = vec![
-            "printf.c"
-        ];
+        let sources = vec!["printf.c"];
 
         let root = Path::new("./");
         for src in sources {

src/libdwarf/Cargo.toml

@@ -15,3 +15,4 @@ libc = { path = "../libc" }
 unwind = { path = "../libunwind" }
 compiler_builtins = "0.1.0"
 cfg-if = "0.1.8"
+cslice = "0.3"

src/libdwarf/src/eh.rs

@@ -11,9 +11,12 @@
 #![allow(non_upper_case_globals)]
 #![allow(unused)]
 
-use crate::DwarfReader;
 use core::mem;
 
+use cslice::CSlice;
+
+use crate::DwarfReader;
+
 pub const DW_EH_PE_omit: u8 = 0xFF;
 pub const DW_EH_PE_absptr: u8 = 0x00;
 
@@ -51,10 +54,45 @@ pub enum EHAction {
 
 pub const USING_SJLJ_EXCEPTIONS: bool = cfg!(all(target_os = "ios", target_arch = "arm"));
 
+fn size_of_encoded_value(encoding: u8) -> usize {
+    if encoding == DW_EH_PE_omit {
+        0
+    } else {
+        let encoding = encoding & 0x07;
+        match encoding {
+            DW_EH_PE_absptr => core::mem::size_of::<*const ()>(),
+            DW_EH_PE_udata2 => 2,
+            DW_EH_PE_udata4 => 4,
+            DW_EH_PE_udata8 => 8,
+            _ => unreachable!(),
+        }
+    }
+}
+
+unsafe fn get_ttype_entry(
+    offset: usize,
+    encoding: u8,
+    ttype_base: usize,
+    ttype: *const u8,
+) -> Result<Option<*const u8>, ()> {
+    let i = (offset * size_of_encoded_value(encoding)) as isize;
+    read_encoded_pointer_with_base(
+        &mut DwarfReader::new(ttype.offset(-i)),
+        // the DW_EH_PE_pcrel is a hack.
+        // It seems that the default encoding is absolute, but we have to take reallocation into
+        // account. Unsure if we can fix this in the compiler setting or if this would be affected
+        // by updating the compiler
+        encoding | DW_EH_PE_pcrel,
+        ttype_base,
+    )
+    .map(|v| (v != ttype_base).then(|| v as *const u8))
+}
+
 pub unsafe fn find_eh_action(
     lsda: *const u8,
     context: &EHContext<'_>,
     foreign_exception: bool,
+    id: u32,
 ) -> Result<EHAction, ()> {
     if lsda.is_null() {
         return Ok(EHAction::None);
@@ -72,10 +110,17 @@ pub unsafe fn find_eh_action(
     };
 
     let ttype_encoding = reader.read::<u8>();
-    if ttype_encoding != DW_EH_PE_omit {
-        // Rust doesn't analyze exception types, so we don't care about the type table
-        reader.read_uleb128();
-    }
+    // we do care about the type table
+    let ttype_offset = if ttype_encoding != DW_EH_PE_omit {
+        reader.read_uleb128()
+    } else {
+        0
+    };
+    // for rust functions, it seems that there is no type table, so I just put whatever value here.
+    // we should not return an error, otherwise we would abort unwinding and cannot unwind through
+    // rust functions
+    let ttype_base = get_base(ttype_encoding, context).unwrap_or(1);
+    let ttype_table = reader.ptr.offset(ttype_offset as isize);
 
     let call_site_encoding = reader.read::<u8>();
     let call_site_table_length = reader.read_uleb128();
@@ -94,11 +139,49 @@ pub unsafe fn find_eh_action(
                 break;
             }
             if ip < func_start + cs_start + cs_len {
+                // https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/libsupc%2B%2B/eh_personality.cc#L528
+                let lpad = lpad_base + cs_lpad;
                 if cs_lpad == 0 {
+                    // no cleanups/handler
+                    return Ok(EHAction::None);
+                } else if cs_action == 0 {
+                    return Ok(EHAction::Cleanup(lpad));
+                } else if foreign_exception {
                     return Ok(EHAction::None);
                 } else {
-                    let lpad = lpad_base + cs_lpad;
-                    return Ok(interpret_cs_action(cs_action, lpad, foreign_exception));
+                    let mut saw_cleanup = false;
+                    let mut action_record = action_table.offset(cs_action as isize - 1);
+                    loop {
+                        let mut reader = DwarfReader::new(action_record);
+                        let ar_filter = reader.read_sleb128();
+                        action_record = reader.ptr;
+                        let ar_disp = reader.read_sleb128();
+                        if ar_filter == 0 {
+                            saw_cleanup = true;
+                        } else if ar_filter > 0 {
+                            let catch_type =
+                                get_ttype_entry(ar_filter as usize, ttype_encoding, ttype_base, ttype_table)?;
+                            match catch_type {
+                                Some(clause_ptr) if *(clause_ptr as *const u32) == id => {
+                                    return Ok(EHAction::Catch(lpad));
+                                }
+                                None => return Ok(EHAction::Catch(lpad)),
+                                _ => {}
+                            }
+                        } else if ar_filter < 0 {
+                            // FIXME: how to handle this?
+                            break;
+                        }
+                        if ar_disp == 0 {
+                            break;
+                        }
+                        action_record = action_record.offset((ar_disp as usize) as isize);
+                    }
+                    if saw_cleanup {
+                        return Ok(EHAction::Cleanup(lpad));
+                    } else {
+                        return Ok(EHAction::None);
+                    }
                 }
             }
         }
@@ -106,7 +189,7 @@ pub unsafe fn find_eh_action(
         // So rather than returning EHAction::Terminate, we do this.
         Ok(EHAction::None)
     } else {
-        // SjLj version:
+        // SjLj version: (not yet modified)
         // The "IP" is an index into the call-site table, with two exceptions:
         // -1 means 'no-action', and 0 means 'terminate'.
         match ip as isize {
@@ -146,18 +229,33 @@ fn interpret_cs_action(cs_action: u64, lpad: usize, foreign_exception: bool) ->
 
 #[inline]
 fn round_up(unrounded: usize, align: usize) -> Result<usize, ()> {
-    if align.is_power_of_two() { Ok((unrounded + align - 1) & !(align - 1)) } else { Err(()) }
+    if align.is_power_of_two() {
+        Ok((unrounded + align - 1) & !(align - 1))
+    } else {
+        Err(())
+    }
 }
 
-unsafe fn read_encoded_pointer(
-    reader: &mut DwarfReader,
-    context: &EHContext<'_>,
-    encoding: u8,
-) -> Result<usize, ()> {
+fn get_base(encoding: u8, context: &EHContext<'_>) -> Result<usize, ()> {
+    match encoding & 0x70 {
+        DW_EH_PE_absptr | DW_EH_PE_pcrel | DW_EH_PE_aligned => Ok(0),
+        DW_EH_PE_textrel => Ok((*context.get_text_start)()),
+        DW_EH_PE_datarel => Ok((*context.get_data_start)()),
+        DW_EH_PE_funcrel if context.func_start != 0 => Ok(context.func_start),
+        _ => return Err(()),
+    }
+}
+
+unsafe fn read_encoded_pointer(reader: &mut DwarfReader, context: &EHContext<'_>, encoding: u8) -> Result<usize, ()> {
+    read_encoded_pointer_with_base(reader, encoding, get_base(encoding, context)?)
+}
+
+unsafe fn read_encoded_pointer_with_base(reader: &mut DwarfReader, encoding: u8, base: usize) -> Result<usize, ()> {
     if encoding == DW_EH_PE_omit {
         return Err(());
     }
 
+    let original_ptr = reader.ptr;
     // DW_EH_PE_aligned implies it's an absolute pointer value
     if encoding == DW_EH_PE_aligned {
         reader.ptr = round_up(reader.ptr as usize, mem::size_of::<usize>())? as *const u8;
@@ -177,19 +275,10 @@ unsafe fn read_encoded_pointer(
         _ => return Err(()),
     };
 
-    result += match encoding & 0x70 {
-        DW_EH_PE_absptr => 0,
-        // relative to address of the encoded value, despite the name
-        DW_EH_PE_pcrel => reader.ptr as usize,
-        DW_EH_PE_funcrel => {
-            if context.func_start == 0 {
-                return Err(());
-            }
-            context.func_start
-        }
-        DW_EH_PE_textrel => (*context.get_text_start)(),
-        DW_EH_PE_datarel => (*context.get_data_start)(),
-        _ => return Err(()),
+    result += if (encoding & 0x70) == DW_EH_PE_pcrel {
+        original_ptr as usize
+    } else {
+        base
     };
 
     if encoding & DW_EH_PE_indirect != 0 {

src/libdwarf/src/lib.rs

@@ -26,6 +26,10 @@ impl DwarfReader {
         DwarfReader { ptr }
     }
 
+    pub unsafe fn offset(&mut self, offset: isize) {
+        self.ptr = self.ptr.offset(offset);
+    }
+
     // DWARF streams are packed, so e.g., a u32 would not necessarily be aligned
     // on a 4-byte boundary. This may cause problems on platforms with strict
     // alignment requirements. By wrapping data in a "packed" struct, we are

src/libdyld/Cargo.toml.tpl

@@ -8,4 +8,4 @@ name = "dyld"
 
 [dependencies]
 log = "0.4"
-libcortex_a9 = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
+libcortex_a9 = { path = "@@ZYNQ_RS@@/libcortex_a9" }

src/libdyld/src/elf.rs

@@ -1451,8 +1451,7 @@ pub const R_AARCH64_TLSDESC_CALL: usize = 569;
 pub const R_AARCH64_TLSLE_LDST128_TPREL_LO12: usize = 570;
 pub const R_AARCH64_TLSLE_LDST128_TPREL_LO12_NC: usize = 571;
 pub const R_AARCH64_TLSLD_LDST128_DTPREL_LO12: usize = 572;
-pub const R_AARCH64_TLSLD_LDST128_DTPREL_LO12_NC: usize =
-    573;
+pub const R_AARCH64_TLSLD_LDST128_DTPREL_LO12_NC: usize = 573;
 pub const R_AARCH64_COPY: usize = 1024;
 pub const R_AARCH64_GLOB_DAT: usize = 1025;
 pub const R_AARCH64_JUMP_SLOT: usize = 1026;
@@ -2267,7 +2266,9 @@ pub struct Elf32_Ehdr {
     pub e_shstrndx: Elf32_Half,
 }
 impl Clone for Elf32_Ehdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2288,7 +2289,9 @@ pub struct Elf64_Ehdr {
     pub e_shstrndx: Elf64_Half,
 }
 impl Clone for Elf64_Ehdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2305,7 +2308,9 @@ pub struct Elf32_Shdr {
     pub sh_entsize: Elf32_Word,
 }
 impl Clone for Elf32_Shdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2322,7 +2327,9 @@ pub struct Elf64_Shdr {
     pub sh_entsize: Elf64_Xword,
 }
 impl Clone for Elf64_Shdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2335,7 +2342,9 @@ pub struct Elf32_Sym {
     pub st_shndx: Elf32_Section,
 }
 impl Clone for Elf32_Sym {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2348,7 +2357,9 @@ pub struct Elf64_Sym {
     pub st_size: Elf64_Xword,
 }
 impl Clone for Elf64_Sym {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2357,7 +2368,9 @@ pub struct Elf32_Syminfo {
     pub si_flags: Elf32_Half,
 }
 impl Clone for Elf32_Syminfo {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2366,7 +2379,9 @@ pub struct Elf64_Syminfo {
     pub si_flags: Elf64_Half,
 }
 impl Clone for Elf64_Syminfo {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2375,7 +2390,9 @@ pub struct Elf32_Rel {
     pub r_info: Elf32_Word,
 }
 impl Clone for Elf32_Rel {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2384,7 +2401,9 @@ pub struct Elf64_Rel {
     pub r_info: Elf64_Xword,
 }
 impl Clone for Elf64_Rel {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2394,7 +2413,9 @@ pub struct Elf32_Rela {
     pub r_addend: Elf32_Sword,
 }
 impl Clone for Elf32_Rela {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2404,7 +2425,9 @@ pub struct Elf64_Rela {
     pub r_addend: Elf64_Sxword,
 }
 impl Clone for Elf64_Rela {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2419,7 +2442,9 @@ pub struct Elf32_Phdr {
     pub p_align: Elf32_Word,
 }
 impl Clone for Elf32_Phdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2434,7 +2459,9 @@ pub struct Elf64_Phdr {
     pub p_align: Elf64_Xword,
 }
 impl Clone for Elf64_Phdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Copy)]
@@ -2449,10 +2476,14 @@ pub union Elf32_Dyn__bindgen_ty_1 {
     pub d_ptr: Elf32_Addr,
 }
 impl Clone for Elf32_Dyn__bindgen_ty_1 {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 impl Clone for Elf32_Dyn {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Copy)]
@@ -2467,10 +2498,14 @@ pub union Elf64_Dyn__bindgen_ty_1 {
     pub d_ptr: Elf64_Addr,
 }
 impl Clone for Elf64_Dyn__bindgen_ty_1 {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 impl Clone for Elf64_Dyn {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2484,7 +2519,9 @@ pub struct Elf32_Verdef {
     pub vd_next: Elf32_Word,
 }
 impl Clone for Elf32_Verdef {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2498,7 +2535,9 @@ pub struct Elf64_Verdef {
     pub vd_next: Elf64_Word,
 }
 impl Clone for Elf64_Verdef {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2507,7 +2546,9 @@ pub struct Elf32_Verdaux {
     pub vda_next: Elf32_Word,
 }
 impl Clone for Elf32_Verdaux {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2516,7 +2557,9 @@ pub struct Elf64_Verdaux {
     pub vda_next: Elf64_Word,
 }
 impl Clone for Elf64_Verdaux {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2528,7 +2571,9 @@ pub struct Elf32_Verneed {
     pub vn_next: Elf32_Word,
 }
 impl Clone for Elf32_Verneed {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2540,7 +2585,9 @@ pub struct Elf64_Verneed {
     pub vn_next: Elf64_Word,
 }
 impl Clone for Elf64_Verneed {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2552,7 +2599,9 @@ pub struct Elf32_Vernaux {
     pub vna_next: Elf32_Word,
 }
 impl Clone for Elf32_Vernaux {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2564,7 +2613,9 @@ pub struct Elf64_Vernaux {
     pub vna_next: Elf64_Word,
 }
 impl Clone for Elf64_Vernaux {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Copy)]
@@ -2578,10 +2629,14 @@ pub union Elf32_auxv_t__bindgen_ty_1 {
     pub a_val: u32,
 }
 impl Clone for Elf32_auxv_t__bindgen_ty_1 {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 impl Clone for Elf32_auxv_t {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Copy)]
@@ -2595,10 +2650,14 @@ pub union Elf64_auxv_t__bindgen_ty_1 {
     pub a_val: u64,
 }
 impl Clone for Elf64_auxv_t__bindgen_ty_1 {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 impl Clone for Elf64_auxv_t {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2608,7 +2667,9 @@ pub struct Elf32_Nhdr {
     pub n_type: Elf32_Word,
 }
 impl Clone for Elf32_Nhdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2618,7 +2679,9 @@ pub struct Elf64_Nhdr {
     pub n_type: Elf64_Word,
 }
 impl Clone for Elf64_Nhdr {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2630,7 +2693,9 @@ pub struct Elf32_Move {
     pub m_stride: Elf32_Half,
 }
 impl Clone for Elf32_Move {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2642,7 +2707,9 @@ pub struct Elf64_Move {
     pub m_stride: Elf64_Half,
 }
 impl Clone for Elf64_Move {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Copy)]
@@ -2657,7 +2724,9 @@ pub struct Elf32_gptab__bindgen_ty_1 {
     pub gt_unused: Elf32_Word,
 }
 impl Clone for Elf32_gptab__bindgen_ty_1 {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2666,10 +2735,14 @@ pub struct Elf32_gptab__bindgen_ty_2 {
     pub gt_bytes: Elf32_Word,
 }
 impl Clone for Elf32_gptab__bindgen_ty_2 {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 impl Clone for Elf32_gptab {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2679,7 +2752,9 @@ pub struct Elf32_RegInfo {
     pub ri_gp_value: Elf32_Sword,
 }
 impl Clone for Elf32_RegInfo {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2690,7 +2765,9 @@ pub struct Elf_Options {
     pub info: Elf32_Word,
 }
 impl Clone for Elf_Options {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2699,7 +2776,9 @@ pub struct Elf_Options_Hw {
     pub hwp_flags2: Elf32_Word,
 }
 impl Clone for Elf_Options_Hw {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2711,7 +2790,9 @@ pub struct Elf32_Lib {
     pub l_flags: Elf32_Word,
 }
 impl Clone for Elf32_Lib {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 #[repr(C)]
 #[derive(Debug, Copy)]
@@ -2723,14 +2804,31 @@ pub struct Elf64_Lib {
     pub l_flags: Elf64_Word,
 }
 impl Clone for Elf64_Lib {
-    fn clone(&self) -> Self { *self }
+    fn clone(&self) -> Self {
+        *self
+    }
 }
 pub type Elf32_Conflict = Elf32_Addr;
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct EXIDX_Entry(u32, u32);
 
-pub fn ELF32_R_SYM(info: Elf32_Word) -> Elf32_Word { info >> 8 }
-pub fn ELF32_R_TYPE(info: Elf32_Word) -> u8 { info as u8 }
-pub fn ELF32_R_INFO(sym: Elf32_Word, ty: u8) -> Elf32_Word { sym << 8 | ty as Elf32_Word }
+pub fn ELF32_R_SYM(info: Elf32_Word) -> Elf32_Word {
+    info >> 8
+}
+pub fn ELF32_R_TYPE(info: Elf32_Word) -> u8 {
+    info as u8
+}
+pub fn ELF32_R_INFO(sym: Elf32_Word, ty: u8) -> Elf32_Word {
+    sym << 8 | ty as Elf32_Word
+}
 
-pub fn ELF32_ST_BIND(info: u8) -> u8 { info >> 4 }
-pub fn ELF32_ST_TYPE(info: u8) -> u8 { info & 0xf }
-pub fn ELF32_ST_INFO(bind: u8, ty: u8) -> u8 { (bind << 4) | (ty & 0xf) }
+pub fn ELF32_ST_BIND(info: u8) -> u8 {
+    info >> 4
+}
+pub fn ELF32_ST_TYPE(info: u8) -> u8 {
+    info & 0xf
+}
+pub fn ELF32_ST_INFO(bind: u8, ty: u8) -> u8 {
+    (bind << 4) | (ty & 0xf)
+}

src/libdyld/src/file.rs

@@ -1,8 +1,8 @@
-use core::{mem, ptr, ops::{Deref, Range}};
-use super::{
-    Arch,
-    elf::*,
-};
+use core::{mem,
+           ops::{Deref, Range},
+           ptr};
+
+use super::{elf::*, Arch};
 
 fn read_unaligned<T: Copy>(data: &[u8], offset: usize) -> Option<T> {
     if data.len() < offset + mem::size_of::<T>() {
@@ -31,14 +31,40 @@ impl<'a> File<'a> {
 
     pub fn arch(&self) -> Option<Arch> {
         const IDENT_OPENRISC: [u8; EI_NIDENT] = [
-            ELFMAG0,    ELFMAG1,     ELFMAG2,    ELFMAG3,
-            ELFCLASS32, ELFDATA2MSB, EV_CURRENT, ELFOSABI_NONE,
-            /* ABI version */ 0, /* padding */ 0, 0, 0, 0, 0, 0, 0
+            ELFMAG0,
+            ELFMAG1,
+            ELFMAG2,
+            ELFMAG3,
+            ELFCLASS32,
+            ELFDATA2MSB,
+            EV_CURRENT,
+            ELFOSABI_NONE,
+            /* ABI version */ 0,
+            /* padding */ 0,
+            0,
+            0,
+            0,
+            0,
+            0,
+            0,
         ];
         const IDENT_ARM: [u8; EI_NIDENT] = [
-            ELFMAG0,    ELFMAG1,     ELFMAG2,    ELFMAG3,
-            ELFCLASS32, ELFDATA2LSB, EV_CURRENT, ELFOSABI_NONE,
-            /* ABI version */ 0, /* padding */ 0, 0, 0, 0, 0, 0, 0
+            ELFMAG0,
+            ELFMAG1,
+            ELFMAG2,
+            ELFMAG3,
+            ELFCLASS32,
+            ELFDATA2LSB,
+            EV_CURRENT,
+            ELFOSABI_NONE,
+            /* ABI version */ 0,
+            /* padding */ 0,
+            0,
+            0,
+            0,
+            0,
+            0,
+            0,
         ];
 
         match (self.ehdr.e_ident, self.ehdr.e_machine) {
@@ -48,16 +74,14 @@ impl<'a> File<'a> {
         }
     }
 
-    pub fn program_headers<'b>(&'b self) -> impl Iterator<Item = Option<Elf32_Phdr>> + 'b
-    {
+    pub fn program_headers<'b>(&'b self) -> impl Iterator<Item = Option<Elf32_Phdr>> + 'b {
         (0..self.ehdr.e_phnum).map(move |i| {
             let phdr_off = self.ehdr.e_phoff as usize + mem::size_of::<Elf32_Phdr>() * i as usize;
             self.read_unaligned::<Elf32_Phdr>(phdr_off)
         })
     }
 
-    pub fn section_headers<'b>(&'b self) -> impl Iterator<Item = Option<Elf32_Shdr>> + 'b
-    {
+    pub fn section_headers<'b>(&'b self) -> impl Iterator<Item = Option<Elf32_Shdr>> + 'b {
         (0..self.ehdr.e_shnum).map(move |i| {
             let shdr_off = self.ehdr.e_shoff as usize + mem::size_of::<Elf32_Shdr>() * i as usize;
             self.read_unaligned::<Elf32_Shdr>(shdr_off)

src/libdyld/src/image.rs

@@ -1,13 +1,9 @@
-use core::{
-    ops::{Deref, DerefMut, Range},
-    mem,
-    slice,
-};
-use alloc::alloc::{alloc_zeroed, dealloc, Layout, LayoutErr};
-use super::{
-    elf::*,
-    Error,
-};
+use alloc::alloc::{alloc_zeroed, dealloc, Layout, LayoutError};
+use core::{mem,
+           ops::{Deref, DerefMut, Range},
+           slice};
+
+use super::{elf::*, Error};
 
 pub struct DynamicSection {
     pub strtab: Range<usize>,
@@ -27,24 +23,19 @@ pub struct Image {
 }
 
 impl Image {
-    pub fn new(size: usize, align: usize) -> Result<Self, LayoutErr> {
+    pub fn new(size: usize, align: usize) -> Result<Self, LayoutError> {
         let layout = Layout::from_size_align(size, align)?;
         let data = unsafe {
             let ptr = alloc_zeroed(layout);
             slice::from_raw_parts_mut(ptr, size)
         };
 
-        Ok(Image {
-            layout,
-            data,
-        })
+        Ok(Image { layout, data })
     }
 
     /// assumes that self.data is properly aligned
     pub(crate) fn get_ref<T>(&self, offset: usize) -> Option<&T>
-    where
-        T: Copy,
-    {
+    where T: Copy {
         if self.data.len() < offset + mem::size_of::<T>() {
             None
         } else if (self.data.as_ptr() as usize + offset) & (mem::align_of::<T>() - 1) != 0 {
@@ -66,55 +57,53 @@ impl Image {
         unsafe { slice::from_raw_parts(ptr, len) }
     }
 
-    fn dyn_headers<'a>(&'a self, range: Range<usize>) ->
-        impl Iterator<Item = &'a Elf32_Dyn> + 'a
-    {
+    fn dyn_headers<'a>(&'a self, range: Range<usize>) -> impl Iterator<Item = &'a Elf32_Dyn> + 'a {
         range
             .step_by(mem::size_of::<Elf32_Dyn>())
-            .filter_map(move |offset| {
-                self.get_ref::<Elf32_Dyn>(offset)
-            })
+            .filter_map(move |offset| self.get_ref::<Elf32_Dyn>(offset))
             .take_while(|d| unsafe { d.d_un.d_val } as i32 != DT_NULL)
     }
 
     pub fn dyn_section(&self, range: Range<usize>) -> Result<DynamicSection, Error> {
         let (mut strtab_off, mut strtab_sz) = (0, 0);
-        let (mut rel_off,    mut rel_sz)    = (0, 0);
-        let (mut rela_off,   mut rela_sz)   = (0, 0);
+        let (mut rel_off, mut rel_sz) = (0, 0);
+        let (mut rela_off, mut rela_sz) = (0, 0);
         let (mut pltrel_off, mut pltrel_sz) = (0, 0);
-        let (mut hash_off,   mut hash_sz)   = (0, 0);
+        let (mut hash_off, mut hash_sz) = (0, 0);
         let mut symtab_off = 0;
-        let mut sym_ent    = 0;
-        let mut rel_ent    = 0;
-        let mut rela_ent   = 0;
-        let mut nbucket    = 0;
-        let mut nchain     = 0;
+        let mut sym_ent = 0;
+        let mut rel_ent = 0;
+        let mut rela_ent = 0;
+        let mut nbucket = 0;
+        let mut nchain = 0;
 
         for dyn_header in self.dyn_headers(range) {
             let val = unsafe { dyn_header.d_un.d_val } as usize;
             match dyn_header.d_tag {
-                DT_NULL     => break,
-                DT_STRTAB   => strtab_off = val,
-                DT_STRSZ    => strtab_sz  = val,
-                DT_SYMTAB   => symtab_off = val,
-                DT_SYMENT   => sym_ent    = val,
-                DT_REL      => rel_off    = val,
-                DT_RELSZ    => rel_sz     = val,
-                DT_RELENT   => rel_ent    = val,
-                DT_RELA     => rela_off   = val,
-                DT_RELASZ   => rela_sz    = val,
-                DT_RELAENT  => rela_ent   = val,
-                DT_JMPREL   => pltrel_off = val,
-                DT_PLTRELSZ => pltrel_sz  = val,
-                DT_HASH     => {
-                    nbucket  = *self.get_ref::<Elf32_Word>(val + 0)
+                DT_NULL => break,
+                DT_STRTAB => strtab_off = val,
+                DT_STRSZ => strtab_sz = val,
+                DT_SYMTAB => symtab_off = val,
+                DT_SYMENT => sym_ent = val,
+                DT_REL => rel_off = val,
+                DT_RELSZ => rel_sz = val,
+                DT_RELENT => rel_ent = val,
+                DT_RELA => rela_off = val,
+                DT_RELASZ => rela_sz = val,
+                DT_RELAENT => rela_ent = val,
+                DT_JMPREL => pltrel_off = val,
+                DT_PLTRELSZ => pltrel_sz = val,
+                DT_HASH => {
+                    nbucket = *self
+                        .get_ref::<Elf32_Word>(val + 0)
                         .ok_or("cannot read hash bucket count")? as usize;
-                    nchain   = *self.get_ref::<Elf32_Word>(val + 4)
+                    nchain = *self
+                        .get_ref::<Elf32_Word>(val + 4)
                         .ok_or("cannot read hash chain count")? as usize;
                     hash_off = val + 8;
-                    hash_sz  = (nbucket + nchain) * mem::size_of::<Elf32_Word>();
+                    hash_sz = (nbucket + nchain) * mem::size_of::<Elf32_Word>();
                 }
-                _ => ()
+                _ => (),
             }
         }
 
@@ -123,28 +112,28 @@ impl Image {
         let symtab_sz = nchain * mem::size_of::<Elf32_Sym>();
 
         if strtab_off + strtab_sz > self.data.len() {
-            return Err("invalid strtab offset/size")?
+            return Err("invalid strtab offset/size")?;
         }
         if symtab_off + symtab_sz > self.data.len() {
-            return Err("invalid symtab offset/size")?
+            return Err("invalid symtab offset/size")?;
         }
         if sym_ent != mem::size_of::<Elf32_Sym>() {
-            return Err("incorrect symbol entry size")?
+            return Err("incorrect symbol entry size")?;
         }
         if rel_off + rel_sz > self.data.len() {
-            return Err("invalid rel offset/size")?
+            return Err("invalid rel offset/size")?;
         }
         if rel_ent != 0 && rel_ent != mem::size_of::<Elf32_Rel>() {
-            return Err("incorrect relocation entry size")?
+            return Err("incorrect relocation entry size")?;
         }
         if rela_off + rela_sz > self.data.len() {
-            return Err("invalid rela offset/size")?
+            return Err("invalid rela offset/size")?;
         }
         if rela_ent != 0 && rela_ent != mem::size_of::<Elf32_Rela>() {
-            return Err("incorrect relocation entry size")?
+            return Err("incorrect relocation entry size")?;
         }
         if pltrel_off + pltrel_sz > self.data.len() {
-            return Err("invalid pltrel offset/size")?
+            return Err("invalid pltrel offset/size")?;
         }
 
         Ok(DynamicSection {
@@ -165,7 +154,7 @@ impl Image {
 
     pub fn write(&self, offset: usize, value: Elf32_Word) -> Result<(), Error> {
         if offset + mem::size_of::<Elf32_Addr>() > self.data.len() {
-            return Err("relocation out of image bounds")?
+            return Err("relocation out of image bounds")?;
         }
 
         let ptr = (self.data.as_ptr() as usize + offset) as *mut Elf32_Addr;

src/libdyld/src/lib.rs

@@ -1,13 +1,14 @@
 #![no_std]
 
 extern crate alloc;
-extern crate log;
 extern crate libcortex_a9;
+extern crate log;
 
-use core::{convert, fmt, ops::Range, str};
 use alloc::string::String;
-use log::{debug, trace};
+use core::{convert, fmt, ops::Range, str};
+
 use elf::*;
+use log::{debug, trace};
 
 pub mod elf;
 mod file;
@@ -21,11 +22,10 @@ pub enum Arch {
     OpenRisc,
 }
 
-
 #[derive(Debug)]
 pub enum Error {
     Parsing(&'static str),
-    Lookup(String)
+    Lookup(String),
 }
 
 impl convert::From<&'static str> for Error {
@@ -37,10 +37,8 @@ impl convert::From<&'static str> for Error {
 impl fmt::Display for Error {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match self {
-            &Error::Parsing(desc) =>
-                write!(f, "parse error: {}", desc),
-            &Error::Lookup(ref sym) =>
-                write!(f, "symbol lookup error: {}", sym),
+            &Error::Parsing(desc) => write!(f, "parse error: {}", desc),
+            &Error::Lookup(ref sym) => write!(f, "symbol lookup error: {}", sym),
         }
     }
 }
@@ -103,20 +101,22 @@ impl Library {
         let mut index = self.hash_bucket()[hash as usize % self.hash_bucket().len()] as usize;
 
         loop {
-            if index == STN_UNDEF { return None }
+            if index == STN_UNDEF {
+                return None;
+            }
 
             let sym = &self.symtab()[index];
             let sym_name_off = sym.st_name as usize;
             match self.strtab().get(sym_name_off..sym_name_off + name.len()) {
                 Some(sym_name) if sym_name == name => {
                     if ELF32_ST_BIND(sym.st_info) & STB_GLOBAL == 0 {
-                        return None
+                        return None;
                     }
 
                     match sym.st_shndx {
                         SHN_UNDEF => return None,
                         SHN_ABS => return Some(self.image.ptr() as u32 + sym.st_value),
-                        _ => return Some(self.image.ptr() as u32 + sym.st_value)
+                        _ => return Some(self.image.ptr() as u32 + sym.st_value),
                     }
                 }
                 _ => (),
@@ -127,10 +127,16 @@ impl Library {
     }
 
     pub fn name_starting_at(&self, offset: usize) -> Result<&[u8], Error> {
-        let size = self.strtab().iter().skip(offset).position(|&x| x == 0)
-                              .ok_or("symbol in symbol table not null-terminated")?;
-        Ok(self.strtab().get(offset..offset + size)
-           .ok_or("cannot read symbol name")?)
+        let size = self
+            .strtab()
+            .iter()
+            .skip(offset)
+            .position(|&x| x == 0)
+            .ok_or("symbol in symbol table not null-terminated")?;
+        Ok(self
+            .strtab()
+            .get(offset..offset + size)
+            .ok_or("cannot read symbol name")?)
     }
 
     /// Rebind Rela by `name` to a new `addr`
@@ -138,58 +144,64 @@ impl Library {
         reloc::rebind(self.arch, self, name, addr as Elf32_Word)
     }
 
-    pub fn exidx(&self) -> &[u32] {
+    pub fn exidx(&self) -> &[EXIDX_Entry] {
         self.image.get_ref_slice_unchecked(&self.exidx)
     }
 }
 
-pub fn load(
-    data: &[u8],
-    resolve: &dyn Fn(&[u8]) -> Option<Elf32_Word>
-) -> Result<Library, Error> {
+pub fn load(data: &[u8], resolve: &dyn Fn(&[u8]) -> Option<Elf32_Word>) -> Result<Library, Error> {
     // validate ELF file
-    let file = file::File::new(data)
-        .ok_or("cannot read ELF header")?;
+    let file = file::File::new(data).ok_or("cannot read ELF header")?;
     if file.ehdr.e_type != ET_DYN {
-        return Err("not a shared library")?
+        return Err("not a shared library")?;
     }
-    let arch = file.arch()
-        .ok_or("not for a supported architecture")?;
+    let arch = file.arch().ok_or("not for a supported architecture")?;
 
     // prepare target memory
-    let image_size = file.program_headers()
+    let image_size = file
+        .program_headers()
         .filter_map(|phdr| phdr.map(|phdr| phdr.p_vaddr + phdr.p_memsz))
         .max()
         .unwrap_or(0) as usize;
-    let image_align = file.program_headers()
-        .filter_map(|phdr| phdr.and_then(|phdr| {
-            if phdr.p_type == PT_LOAD {
-                Some(phdr.p_align)
-            } else {
-                None
-            }
-        }))
+    let image_align = file
+        .program_headers()
+        .filter_map(|phdr| {
+            phdr.and_then(|phdr| {
+                if phdr.p_type == PT_LOAD {
+                    Some(phdr.p_align)
+                } else {
+                    None
+                }
+            })
+        })
         .max()
         .unwrap_or(4) as usize;
     // 1 image for all segments
-    let mut image = image::Image::new(image_size, image_align)
-        .map_err(|_| "cannot allocate target image")?;
-    debug!("ELF target: {} bytes, align to {:X}, allocated at {:08X}", image_size, image_align, image.ptr() as usize);
+    let mut image = image::Image::new(image_size, image_align).map_err(|_| "cannot allocate target image")?;
+    debug!(
+        "ELF target: {} bytes, align to {:X}, allocated at {:08X}",
+        image_size,
+        image_align,
+        image.ptr() as usize
+    );
 
     // LOAD
     for phdr in file.program_headers() {
         let phdr = phdr.ok_or("cannot read program header")?;
-        trace!("Program header: {:08X}+{:08X} to {:08X}",
-              phdr.p_offset, phdr.p_filesz,
-              image.ptr() as u32
+        trace!(
+            "Program header: {:08X}+{:08X} to {:08X}",
+            phdr.p_offset,
+            phdr.p_filesz,
+            image.ptr() as u32
         );
         let file_range = phdr.p_offset as usize..(phdr.p_offset + phdr.p_filesz) as usize;
         match phdr.p_type {
             PT_LOAD => {
-                let src = file.get(file_range)
+                let src = file
+                    .get(file_range)
                     .ok_or("program header requests an out of bounds load (in file)")?;
-                let dst = image.get_mut(phdr.p_vaddr as usize..
-                                        (phdr.p_vaddr + phdr.p_filesz) as usize)
+                let dst = image
+                    .get_mut(phdr.p_vaddr as usize..(phdr.p_vaddr + phdr.p_filesz) as usize)
                     .ok_or("program header requests an out of bounds load (in target)")?;
                 dst.copy_from_slice(src);
             }
@@ -203,9 +215,9 @@ pub fn load(
         let shdr = shdr.ok_or("cannot read section header")?;
         match shdr.sh_type as usize {
             SHT_ARM_EXIDX => {
-                let range = shdr.sh_addr as usize..
-                    (shdr.sh_addr + shdr.sh_size) as usize;
-                let _ = image.get(range.clone())
+                let range = shdr.sh_addr as usize..(shdr.sh_addr + shdr.sh_size) as usize;
+                let _ = image
+                    .get(range.clone())
                     .ok_or("section header specifies EXIDX outside of image (in target)")?;
                 exidx = Some(range);
             }
@@ -214,11 +226,14 @@ pub fn load(
     }
 
     // relocate DYNAMIC
-    let dyn_range = file.dyn_header_vaddr()
-        .ok_or("cannot find a dynamic header")?;
+    let dyn_range = file.dyn_header_vaddr().ok_or("cannot find a dynamic header")?;
     let dyn_section = image.dyn_section(dyn_range.clone())?;
-    debug!("Relocating {} rela, {} rel, {} pltrel",
-           dyn_section.rela.len(), dyn_section.rel.len(), dyn_section.pltrel.len());
+    debug!(
+        "Relocating {} rela, {} rel, {} pltrel",
+        dyn_section.rela.len(),
+        dyn_section.rel.len(),
+        dyn_section.pltrel.len()
+    );
     let lib = Library {
         arch,
         image,

src/libdyld/src/reloc.rs

@@ -1,16 +1,10 @@
 use alloc::string::String;
+
+use libcortex_a9::{asm::{dsb, isb},
+                   cache::{bpiall, dcci_slice, iciallu}};
 use log::trace;
-use super::{
-    Arch,
-    elf::*,
-    Error,
-    image::Image,
-    Library,
-};
-use libcortex_a9::{
-    cache::{dcci_slice, iciallu, bpiall},
-    asm::{dsb, isb},
-};
+
+use super::{elf::*, image::Image, Arch, Error, Library};
 
 pub trait Relocatable {
     fn offset(&self) -> usize;
@@ -59,29 +53,25 @@ impl Relocatable for Elf32_Rela {
 enum RelType {
     None,
     Relative,
-    Lookup,
+    LookupAbs,
+    LookupRel,
 }
 
 impl RelType {
     pub fn new(arch: Arch, type_info: u8) -> Option<Self> {
         match type_info {
-            R_OR1K_NONE if arch == Arch::OpenRisc =>
-                Some(RelType::None),
-            R_ARM_NONE if arch == Arch::Arm =>
-                Some(RelType::None),
+            R_OR1K_NONE if arch == Arch::OpenRisc => Some(RelType::None),
+            R_ARM_NONE if arch == Arch::Arm => Some(RelType::None),
 
-            R_OR1K_RELATIVE if arch == Arch::OpenRisc =>
-                Some(RelType::Relative),
-            R_ARM_RELATIVE if arch == Arch::Arm =>
-                Some(RelType::Relative),
+            R_OR1K_RELATIVE if arch == Arch::OpenRisc => Some(RelType::Relative),
+            R_ARM_RELATIVE if arch == Arch::Arm => Some(RelType::Relative),
 
-            R_OR1K_32 | R_OR1K_GLOB_DAT | R_OR1K_JMP_SLOT
-                if arch == Arch::OpenRisc => Some(RelType::Lookup),
-            R_ARM_GLOB_DAT | R_ARM_JUMP_SLOT
-                if arch == Arch::Arm => Some(RelType::Lookup),
+            R_OR1K_32 | R_OR1K_GLOB_DAT | R_OR1K_JMP_SLOT if arch == Arch::OpenRisc => Some(RelType::LookupAbs),
+            R_ARM_GLOB_DAT | R_ARM_JUMP_SLOT | R_ARM_ABS32 if arch == Arch::Arm => Some(RelType::LookupAbs),
 
-            _ =>
-                None
+            R_ARM_PREL31 if arch == Arch::Arm => Some(RelType::LookupRel),
+
+            _ => None,
         }
     }
 }
@@ -93,71 +83,108 @@ fn format_sym_name(sym_name: &[u8]) -> String {
 }
 
 pub fn relocate<R: Relocatable>(
-    arch: Arch, lib: &Library,
-    rel: &R, resolve: &dyn Fn(&[u8]) -> Option<Elf32_Word>
+    arch: Arch,
+    lib: &Library,
+    rel: &R,
+    resolve: &dyn Fn(&[u8]) -> Option<Elf32_Word>,
 ) -> Result<(), Error> {
     let sym;
     if rel.sym_info() == 0 {
         sym = None;
     } else {
-        sym = Some(lib.symtab().get(rel.sym_info() as usize)
-                   .ok_or("symbol out of bounds of symbol table")?)
+        sym = Some(
+            lib.symtab()
+                .get(rel.sym_info() as usize)
+                .ok_or("symbol out of bounds of symbol table")?,
+        )
     }
 
-    let rel_type = RelType::new(arch, rel.type_info())
-        .ok_or("unsupported relocation type")?;
-    let value;
-    match rel_type {
-        RelType::None =>
-            return Ok(()),
+    let rel_type = RelType::new(arch, rel.type_info()).ok_or("unsupported relocation type")?;
+    let value = match rel_type {
+        RelType::None => return Ok(()),
 
         RelType::Relative => {
             let addend = rel.addend(&lib.image);
-            value = lib.image.ptr().wrapping_offset(addend as isize) as Elf32_Word;
+            lib.image.ptr().wrapping_offset(addend as isize) as Elf32_Word
         }
 
-        RelType::Lookup => {
+        RelType::LookupAbs | RelType::LookupRel => {
             let sym = sym.ok_or("relocation requires an associated symbol")?;
             let sym_name = lib.name_starting_at(sym.st_name as usize)?;
 
-            if let Some(addr) = lib.lookup(sym_name) {
+            let sym_addr = if let Some(addr) = lib.lookup(sym_name) {
                 // First, try to resolve against itself.
                 trace!("looked up symbol {} in image", format_sym_name(sym_name));
-                value = lib.image.ptr() as u32 + addr;
+                addr
             } else if let Some(addr) = resolve(sym_name) {
                 // Second, call the user-provided function.
                 trace!("resolved symbol {:?}", format_sym_name(sym_name));
-                value = addr;
+                addr
             } else {
                 // We couldn't find it anywhere.
-                return Err(Error::Lookup(format_sym_name(sym_name)))
+                return Err(Error::Lookup(format_sym_name(sym_name)));
+            };
+
+            match rel_type {
+                RelType::LookupAbs => sym_addr,
+                RelType::LookupRel => {
+                    sym_addr.wrapping_sub(lib.image.ptr().wrapping_offset(rel.offset() as isize) as Elf32_Addr)
+                }
+                _ => unreachable!(),
             }
         }
-    }
+    };
 
-    lib.image.write(rel.offset(), value)
+    match rel.type_info() {
+        R_ARM_PREL31 => {
+            let reloc_word = lib
+                .image
+                .get_ref::<Elf32_Word>(rel.offset())
+                .ok_or("relocation offset cannot be read")?;
+            lib.image
+                .write(rel.offset(), (reloc_word & 0x80000000) | (value & 0x7FFFFFFF))
+        }
+
+        _ => lib.image.write(rel.offset(), value),
+    }
 }
 
-pub fn rebind(
-    arch: Arch, lib: &Library, name: &[u8], value: Elf32_Word
-) -> Result<(), Error> {
-    for rela in lib.pltrel() {
-        let rel_type = RelType::new(arch, rela.type_info())
-            .ok_or("unsupported relocation type")?;
-        match rel_type {
-            RelType::Lookup => {
-                let sym = lib.symtab().get(ELF32_R_SYM(rela.r_info) as usize)
-                    .ok_or("symbol out of bounds of symbol table")?;
-                let sym_name = lib.name_starting_at(sym.st_name as usize)?;
+pub fn rebind(arch: Arch, lib: &Library, name: &[u8], value: Elf32_Word) -> Result<(), Error> {
+    fn rebind_symbol_to_value<R: Relocatable>(
+        arch: Arch,
+        lib: &Library,
+        name: &[u8],
+        value: Elf32_Word,
+        relocs: &[R],
+    ) -> Result<(), Error> {
+        for reloc in relocs {
+            let rel_type = RelType::new(arch, reloc.type_info()).ok_or("unsupported relocation type")?;
+            match rel_type {
+                RelType::LookupAbs => {
+                    let sym = lib
+                        .symtab()
+                        .get(reloc.sym_info() as usize)
+                        .ok_or("symbol out of bounds of symbol table")?;
+                    let sym_name = lib.name_starting_at(sym.st_name as usize)?;
 
-                if sym_name == name {
-                    lib.image.write(rela.offset(), value)?
+                    if sym_name == name {
+                        lib.image.write(reloc.offset(), value)?
+                    }
                 }
+                // No associated symbols for other relocation types.
+                _ => {}
             }
-            // No associated symbols for other relocation types.
-            _ => {}
         }
+
+        Ok(())
     }
+
+    if lib.pltrel().is_empty() {
+        rebind_symbol_to_value(arch, lib, name, value, lib.rela())?;
+    } else {
+        rebind_symbol_to_value(arch, lib, name, value, lib.pltrel())?;
+    }
+
     // FIXME: the cache maintainance operations may be more than enough,
     // may cause performance degradation.
     dcci_slice(lib.image.data);

src/libio/Cargo.toml.tpl

@@ -0,0 +1,17 @@
+[package]
+authors = ["M-Labs"]
+name = "io"
+version = "0.0.0"
+
+[lib]
+name = "io"
+path = "lib.rs"
+
+[dependencies]
+core_io = { version = "0.1", features = ["collections"] }
+byteorder = { version = "1.0", default-features = false, optional = true }
+
+libsupport_zynq = { path = "@@ZYNQ_RS@@/libsupport_zynq", default-features = false, features = ["alloc_core"] }
+
+[features]
+alloc = []

src/libio/cursor.rs

@@ -0,0 +1,85 @@
+#[cfg(feature = "alloc")]
+use alloc::vec::Vec;
+
+use core_io::{Error as IoError, Read, Write};
+
+#[derive(Debug, Clone)]
+pub struct Cursor<T> {
+    inner: T,
+    pos: usize,
+}
+
+impl<T> Cursor<T> {
+    #[inline]
+    pub fn new(inner: T) -> Cursor<T> {
+        Cursor { inner, pos: 0 }
+    }
+
+    #[inline]
+    pub fn into_inner(self) -> T {
+        self.inner
+    }
+
+    #[inline]
+    pub fn get_ref(&self) -> &T {
+        &self.inner
+    }
+
+    #[inline]
+    pub fn get_mut(&mut self) -> &mut T {
+        &mut self.inner
+    }
+
+    #[inline]
+    pub fn position(&self) -> usize {
+        self.pos
+    }
+
+    #[inline]
+    pub fn set_position(&mut self, pos: usize) {
+        self.pos = pos
+    }
+}
+
+impl<T: AsRef<[u8]>> Read for Cursor<T> {
+    fn read(&mut self, buf: &mut [u8]) -> Result<usize, IoError> {
+        let data = &self.inner.as_ref()[self.pos..];
+        let len = buf.len().min(data.len());
+        // ``copy_from_slice`` generates AXI bursts, use a regular loop instead
+        for i in 0..len {
+            buf[i] = data[i];
+        }
+        self.pos += len;
+        Ok(len)
+    }
+}
+
+impl Write for Cursor<&mut [u8]> {
+    fn write(&mut self, buf: &[u8]) -> Result<usize, IoError> {
+        let data = &mut self.inner[self.pos..];
+        let len = buf.len().min(data.len());
+        for i in 0..len {
+            data[i] = buf[i];
+        }
+        self.pos += len;
+        Ok(len)
+    }
+
+    #[inline]
+    fn flush(&mut self) -> Result<(), IoError> {
+        Ok(())
+    }
+}
+
+#[cfg(feature = "alloc")]
+impl Write for Cursor<Vec<u8>> {
+    fn write(&mut self, buf: &[u8]) -> Result<usize, IoError> {
+        self.inner.extend_from_slice(buf);
+        Ok(buf.len())
+    }
+
+    #[inline]
+    fn flush(&mut self) -> Result<(), IoError> {
+        Ok(())
+    }
+}

src/libio/lib.rs

@@ -0,0 +1,19 @@
+#![no_std]
+#![feature(never_type)]
+
+#[cfg(feature = "alloc")]
+extern crate alloc;
+extern crate core_io;
+
+#[cfg(feature = "byteorder")]
+extern crate byteorder;
+
+pub mod cursor;
+#[cfg(feature = "byteorder")]
+pub mod proto;
+
+pub use cursor::Cursor;
+#[cfg(all(feature = "byteorder", feature = "alloc"))]
+pub use proto::ReadStringError;
+#[cfg(feature = "byteorder")]
+pub use proto::{ProtoRead, ProtoWrite};

src/libio/proto.rs

@@ -1,15 +1,15 @@
+#[cfg(feature = "alloc")]
+use alloc::{string::String, vec};
 use core::str::Utf8Error;
-use byteorder::{ByteOrder, NetworkEndian};
-use alloc::vec;
-use alloc::string::String;
 
-use core_io::{Read, Write, Error as IoError};
+use byteorder::{ByteOrder, NativeEndian};
+use core_io::{Error as IoError, Read, Write};
 
 #[allow(dead_code)]
 #[derive(Debug, Clone, PartialEq)]
 pub enum ReadStringError<T> {
     Utf8(Utf8Error),
-    Other(T)
+    Other(T),
 }
 
 pub trait ProtoRead {
@@ -28,21 +28,21 @@ pub trait ProtoRead {
     fn read_u16(&mut self) -> Result<u16, Self::ReadError> {
         let mut bytes = [0; 2];
         self.read_exact(&mut bytes)?;
-        Ok(NetworkEndian::read_u16(&bytes))
+        Ok(NativeEndian::read_u16(&bytes))
     }
 
     #[inline]
     fn read_u32(&mut self) -> Result<u32, Self::ReadError> {
         let mut bytes = [0; 4];
         self.read_exact(&mut bytes)?;
-        Ok(NetworkEndian::read_u32(&bytes))
+        Ok(NativeEndian::read_u32(&bytes))
     }
 
     #[inline]
     fn read_u64(&mut self) -> Result<u64, Self::ReadError> {
         let mut bytes = [0; 8];
         self.read_exact(&mut bytes)?;
-        Ok(NetworkEndian::read_u64(&bytes))
+        Ok(NativeEndian::read_u64(&bytes))
     }
 
     #[inline]
@@ -51,7 +51,8 @@ pub trait ProtoRead {
     }
 
     #[inline]
-    fn read_bytes(&mut self) -> Result<::alloc::vec::Vec<u8>, Self::ReadError> {
+    #[cfg(feature = "alloc")]
+    fn read_bytes(&mut self) -> Result<vec::Vec<u8>, Self::ReadError> {
         let length = self.read_u32()?;
         let mut value = vec![0; length as usize];
         self.read_exact(&mut value)?;
@@ -59,7 +60,8 @@ pub trait ProtoRead {
     }
 
     #[inline]
-    fn read_string(&mut self) -> Result<::alloc::string::String, ReadStringError<Self::ReadError>> {
+    #[cfg(feature = "alloc")]
+    fn read_string(&mut self) -> Result<String, ReadStringError<Self::ReadError>> {
         let bytes = self.read_bytes().map_err(ReadStringError::Other)?;
         String::from_utf8(bytes).map_err(|err| ReadStringError::Utf8(err.utf8_error()))
     }
@@ -85,42 +87,42 @@ pub trait ProtoWrite {
     #[inline]
     fn write_u16(&mut self, value: u16) -> Result<(), Self::WriteError> {
         let mut bytes = [0; 2];
-        NetworkEndian::write_u16(&mut bytes, value);
+        NativeEndian::write_u16(&mut bytes, value);
         self.write_all(&bytes)
     }
 
     #[inline]
     fn write_i16(&mut self, value: i16) -> Result<(), Self::WriteError> {
         let mut bytes = [0; 2];
-        NetworkEndian::write_i16(&mut bytes, value);
+        NativeEndian::write_i16(&mut bytes, value);
         self.write_all(&bytes)
     }
 
     #[inline]
     fn write_u32(&mut self, value: u32) -> Result<(), Self::WriteError> {
         let mut bytes = [0; 4];
-        NetworkEndian::write_u32(&mut bytes, value);
+        NativeEndian::write_u32(&mut bytes, value);
         self.write_all(&bytes)
     }
 
     #[inline]
     fn write_i32(&mut self, value: i32) -> Result<(), Self::WriteError> {
         let mut bytes = [0; 4];
-        NetworkEndian::write_i32(&mut bytes, value);
+        NativeEndian::write_i32(&mut bytes, value);
         self.write_all(&bytes)
     }
 
     #[inline]
     fn write_u64(&mut self, value: u64) -> Result<(), Self::WriteError> {
         let mut bytes = [0; 8];
-        NetworkEndian::write_u64(&mut bytes, value);
+        NativeEndian::write_u64(&mut bytes, value);
         self.write_all(&bytes)
     }
 
     #[inline]
     fn write_i64(&mut self, value: i64) -> Result<(), Self::WriteError> {
         let mut bytes = [0; 8];
-        NetworkEndian::write_i64(&mut bytes, value);
+        NativeEndian::write_i64(&mut bytes, value);
         self.write_all(&bytes)
     }
 
@@ -136,12 +138,15 @@ pub trait ProtoWrite {
     }
 
     #[inline]
+    #[cfg(feature = "alloc")]
     fn write_string(&mut self, value: &str) -> Result<(), Self::WriteError> {
         self.write_bytes(value.as_bytes())
     }
 }
 
-impl<T> ProtoRead for T where T: Read + ?Sized {
+impl<T> ProtoRead for T
+where T: Read + ?Sized
+{
     type ReadError = IoError;
 
     fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Self::ReadError> {
@@ -149,7 +154,9 @@ impl<T> ProtoRead for T where T: Read + ?Sized {
     }
 }
 
-impl<T> ProtoWrite for T where T: Write + ?Sized {
+impl<T> ProtoWrite for T
+where T: Write + ?Sized
+{
     type WriteError = IoError;
 
     fn write_all(&mut self, buf: &[u8]) -> Result<(), Self::WriteError> {

src/libksupport/Cargo.toml.tpl

@@ -0,0 +1,40 @@
+[package]
+name = "ksupport"
+description = "Kernel support for Zynq-based platforms"
+version = "0.1.0"
+authors = ["M-Labs"]
+edition = "2018"
+
+[build-dependencies]
+build_zynq = { path = "../libbuild_zynq" }
+
+[dependencies]
+cslice = "0.3"
+log = "0.4"
+nb = "0.1"
+core_io = { version = "0.1", features = ["collections"] }
+byteorder = { version = "1.3", default-features = false }
+void = { version = "1", default-features = false }
+log_buffer = { version = "1.2" }
+libm = { version = "0.2", features = ["unstable"] }
+vcell = "0.1"
+
+libboard_zynq = { path = "@@ZYNQ_RS@@/libboard_zynq", features = ["ipv6"]}
+libsupport_zynq = { path = "@@ZYNQ_RS@@/libsupport_zynq", default-features = false, features = ["alloc_core"] }
+libcortex_a9 = { path = "@@ZYNQ_RS@@/libcortex_a9" }
+libasync = { path = "@@ZYNQ_RS@@/libasync" }
+libregister = { path = "@@ZYNQ_RS@@/libregister" }
+libconfig = { path = "@@ZYNQ_RS@@/libconfig", features = ["fat_lfn", "ipv6"] }
+
+dyld = { path = "../libdyld" }
+dwarf = { path = "../libdwarf" }
+unwind = { path = "../libunwind" }
+libc = { path = "../libc" }
+io = { path = "../libio" }
+libboard_artiq = { path = "../libboard_artiq" }
+
+[dependencies.nalgebra]
+git = "https://git.m-labs.hk/M-Labs/nalgebra.git"
+rev = "dd00f9b"
+default-features = false
+features = ["libm", "alloc"]

src/libksupport/build.rs

@@ -0,0 +1,5 @@
+extern crate build_zynq;
+
+fn main() {
+    build_zynq::cfg();
+}

src/libksupport/src/eh_artiq.rs

@@ -0,0 +1,561 @@
+// From Current artiq firmware ksupport implementation.
+// Modified to suit the case of artiq-zynq port, for ARM EHABI.
+// Portions of the code in this file are derived from code by:
+//
+// Copyright 2015 The Rust Project Developers. See the COPYRIGHT
+// file at http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+#![allow(non_camel_case_types)]
+
+use core::mem;
+
+use core_io::Error as ReadError;
+use cslice::{AsCSlice, CSlice};
+use dwarf::eh::{self, EHAction, EHContext};
+use io::{Cursor, ProtoRead};
+use libc::{c_int, c_void, uintptr_t};
+use log::{error, trace};
+use unwind as uw;
+
+use crate::kernel::KERNEL_IMAGE;
+
+const EXCEPTION_CLASS: uw::_Unwind_Exception_Class = 0x4d_4c_42_53_41_52_54_51; /* 'MLBSARTQ' */
+
+#[cfg(target_arch = "arm")]
+const UNWIND_DATA_REG: (i32, i32) = (0, 1); // R0, R1
+
+// Note: CSlice within an exception may not be actual cslice, they may be strings that exist only
+// in the host. If the length == usize:MAX, the pointer is actually a string key in the host.
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct Exception<'a> {
+    pub id: u32,
+    pub file: CSlice<'a, u8>,
+    pub line: u32,
+    pub column: u32,
+    pub function: CSlice<'a, u8>,
+    pub message: CSlice<'a, u8>,
+    pub param: [i64; 3],
+}
+
+fn str_err(_: core::str::Utf8Error) -> core::fmt::Error {
+    core::fmt::Error
+}
+
+fn exception_str<'a>(s: &'a CSlice<'a, u8>) -> Result<&'a str, core::str::Utf8Error> {
+    if s.len() == usize::MAX {
+        Ok("<host string>")
+    } else {
+        core::str::from_utf8(s.as_ref())
+    }
+}
+
+impl<'a> core::fmt::Debug for Exception<'a> {
+    fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
+        write!(
+            f,
+            "Exception {} from {} in {}:{}:{}, message: {}",
+            self.id,
+            exception_str(&self.function).map_err(str_err)?,
+            exception_str(&self.file).map_err(str_err)?,
+            self.line,
+            self.column,
+            exception_str(&self.message).map_err(str_err)?
+        )
+    }
+}
+
+const MAX_INFLIGHT_EXCEPTIONS: usize = 10;
+const MAX_BACKTRACE_SIZE: usize = 128;
+
+#[derive(Debug, Default)]
+pub struct StackPointerBacktrace {
+    pub stack_pointer: usize,
+    pub initial_backtrace_size: usize,
+    pub current_backtrace_size: usize,
+}
+
+struct ExceptionBuffer {
+    // we need n _Unwind_Exception, because each will have their own private data
+    uw_exceptions: [uw::_Unwind_Exception; MAX_INFLIGHT_EXCEPTIONS],
+    exceptions: [Option<Exception<'static>>; MAX_INFLIGHT_EXCEPTIONS + 1],
+    exception_stack: [isize; MAX_INFLIGHT_EXCEPTIONS + 1],
+    // nested exceptions will share the backtrace buffer, treated as a tree
+    // backtrace contains a tuple of IP and SP
+    backtrace: [(usize, usize); MAX_BACKTRACE_SIZE],
+    backtrace_size: usize,
+    // stack pointers are stored to reconstruct backtrace for each exception
+    stack_pointers: [StackPointerBacktrace; MAX_INFLIGHT_EXCEPTIONS + 1],
+    // current allocated nested exceptions
+    exception_count: usize,
+}
+
+static mut EXCEPTION_BUFFER: ExceptionBuffer = ExceptionBuffer {
+    uw_exceptions: [uw::_Unwind_Exception {
+        exception_class: EXCEPTION_CLASS,
+        exception_cleanup: cleanup,
+        private: [0; uw::unwinder_private_data_size],
+    }; MAX_INFLIGHT_EXCEPTIONS],
+    exceptions: [None; MAX_INFLIGHT_EXCEPTIONS + 1],
+    exception_stack: [-1; MAX_INFLIGHT_EXCEPTIONS + 1],
+    backtrace: [(0, 0); MAX_BACKTRACE_SIZE],
+    backtrace_size: 0,
+    stack_pointers: [StackPointerBacktrace {
+        stack_pointer: 0,
+        initial_backtrace_size: 0,
+        current_backtrace_size: 0,
+    }; MAX_INFLIGHT_EXCEPTIONS + 1],
+    exception_count: 0,
+};
+
+pub unsafe extern "C" fn reset_exception_buffer() {
+    trace!("reset exception buffer");
+    EXCEPTION_BUFFER.uw_exceptions = [uw::_Unwind_Exception {
+        exception_class: EXCEPTION_CLASS,
+        exception_cleanup: cleanup,
+        private: [0; uw::unwinder_private_data_size],
+    }; MAX_INFLIGHT_EXCEPTIONS];
+    EXCEPTION_BUFFER.exceptions = [None; MAX_INFLIGHT_EXCEPTIONS + 1];
+    EXCEPTION_BUFFER.exception_stack = [-1; MAX_INFLIGHT_EXCEPTIONS + 1];
+    EXCEPTION_BUFFER.backtrace_size = 0;
+    EXCEPTION_BUFFER.exception_count = 0;
+}
+
+type _Unwind_Stop_Fn = extern "C" fn(
+    version: c_int,
+    actions: i32,
+    exception_class: uw::_Unwind_Exception_Class,
+    exception_object: *mut uw::_Unwind_Exception,
+    context: *mut uw::_Unwind_Context,
+    stop_parameter: *mut c_void,
+) -> uw::_Unwind_Reason_Code;
+
+extern "C" {
+    // not defined in EHABI, but LLVM added it and is useful to us
+    fn _Unwind_ForcedUnwind(
+        exception: *mut uw::_Unwind_Exception,
+        stop_fn: _Unwind_Stop_Fn,
+        stop_parameter: *mut c_void,
+    ) -> uw::_Unwind_Reason_Code;
+}
+
+unsafe fn find_eh_action(context: *mut uw::_Unwind_Context, foreign_exception: bool, id: u32) -> Result<EHAction, ()> {
+    let lsda = uw::_Unwind_GetLanguageSpecificData(context) as *const u8;
+    let mut ip_before_instr: c_int = 0;
+    let ip = uw::_Unwind_GetIPInfo(context, &mut ip_before_instr);
+    let eh_context = EHContext {
+        // The return address points 1 byte past the call instruction,
+        // which could be in the next IP range in LSDA range table.
+        ip: if ip_before_instr != 0 { ip } else { ip - 1 },
+        func_start: uw::_Unwind_GetRegionStart(context),
+        get_text_start: &|| uw::_Unwind_GetTextRelBase(context),
+        get_data_start: &|| uw::_Unwind_GetDataRelBase(context),
+    };
+    eh::find_eh_action(lsda, &eh_context, foreign_exception, id)
+}
+
+pub unsafe fn artiq_personality(
+    _state: uw::_Unwind_State,
+    exception_object: *mut uw::_Unwind_Exception,
+    context: *mut uw::_Unwind_Context,
+) -> uw::_Unwind_Reason_Code {
+    // we will only do phase 2 forced unwinding now
+    // The DWARF unwinder assumes that _Unwind_Context holds things like the function
+    // and LSDA pointers, however ARM EHABI places them into the exception object.
+    // To preserve signatures of functions like _Unwind_GetLanguageSpecificData(), which
+    // take only the context pointer, GCC personality routines stash a pointer to
+    // exception_object in the context, using location reserved for ARM's
+    // "scratch register" (r12).
+    uw::_Unwind_SetGR(context, uw::UNWIND_POINTER_REG, exception_object as uw::_Unwind_Ptr);
+    // ...A more principled approach would be to provide the full definition of ARM's
+    // _Unwind_Context in our libunwind bindings and fetch the required data from there
+    // directly, bypassing DWARF compatibility functions.
+
+    let exception_class = (*exception_object).exception_class;
+    let foreign_exception = exception_class != EXCEPTION_CLASS;
+    assert!(!foreign_exception, "we do not expect foreign exceptions");
+    let index = EXCEPTION_BUFFER.exception_stack[EXCEPTION_BUFFER.exception_count - 1];
+    assert!(index != -1);
+    let exception = EXCEPTION_BUFFER.exceptions[index as usize].as_ref().unwrap();
+
+    let id = exception.id;
+    let eh_action = match find_eh_action(context, foreign_exception, id) {
+        Ok(action) => action,
+        Err(_) => return uw::_URC_FAILURE,
+    };
+    match eh_action {
+        EHAction::None => return continue_unwind(exception_object, context),
+        EHAction::Cleanup(lpad) | EHAction::Catch(lpad) => {
+            uw::_Unwind_SetGR(context, UNWIND_DATA_REG.0, exception_object as uintptr_t);
+            uw::_Unwind_SetGR(context, UNWIND_DATA_REG.1, exception as *const _ as uw::_Unwind_Word);
+            uw::_Unwind_SetIP(context, lpad);
+            return uw::_URC_INSTALL_CONTEXT;
+        }
+        EHAction::Terminate => return uw::_URC_FAILURE,
+    }
+
+    // On ARM EHABI the personality routine is responsible for actually
+    // unwinding a single stack frame before returning (ARM EHABI Sec. 6.1).
+    unsafe fn continue_unwind(
+        exception_object: *mut uw::_Unwind_Exception,
+        context: *mut uw::_Unwind_Context,
+    ) -> uw::_Unwind_Reason_Code {
+        let reason = __gnu_unwind_frame(exception_object, context);
+        if reason == uw::_URC_NO_REASON {
+            uw::_URC_CONTINUE_UNWIND
+        } else {
+            reason
+        }
+    }
+    // defined in libgcc
+    extern "C" {
+        fn __gnu_unwind_frame(
+            exception_object: *mut uw::_Unwind_Exception,
+            context: *mut uw::_Unwind_Context,
+        ) -> uw::_Unwind_Reason_Code;
+    }
+}
+
+pub unsafe extern "C" fn raise(exception: *const Exception) -> ! {
+    let count = EXCEPTION_BUFFER.exception_count;
+    let stack = &mut EXCEPTION_BUFFER.exception_stack;
+    let diff = exception as isize - EXCEPTION_BUFFER.exceptions.as_ptr() as isize;
+    if 0 <= diff && diff <= (mem::size_of::<Option<Exception>>() * MAX_INFLIGHT_EXCEPTIONS) as isize {
+        let index = diff / (mem::size_of::<Option<Exception>>() as isize);
+        trace!("reraise at {}", index);
+
+        let mut found = false;
+        for i in 0..=MAX_INFLIGHT_EXCEPTIONS + 1 {
+            if found {
+                if stack[i] == -1 {
+                    stack[i - 1] = index;
+                    assert!(i == count);
+                    break;
+                } else {
+                    stack[i - 1] = stack[i];
+                }
+            } else {
+                if stack[i] == index {
+                    found = true;
+                }
+            }
+        }
+        assert!(found);
+        let _result = _Unwind_ForcedUnwind(
+            &mut EXCEPTION_BUFFER.uw_exceptions[stack[count - 1] as usize],
+            stop_fn,
+            core::ptr::null_mut(),
+        );
+    } else {
+        if count < MAX_INFLIGHT_EXCEPTIONS {
+            trace!("raising exception at level {}", count);
+            let exception = &*exception;
+            for (i, slot) in EXCEPTION_BUFFER.exceptions.iter_mut().enumerate() {
+                // we should always be able to find a slot
+                if slot.is_none() {
+                    *slot = Some(*mem::transmute::<*const Exception, *const Exception<'static>>(
+                        exception,
+                    ));
+                    EXCEPTION_BUFFER.exception_stack[count] = i as isize;
+                    EXCEPTION_BUFFER.uw_exceptions[i].private = [0; uw::unwinder_private_data_size];
+                    EXCEPTION_BUFFER.stack_pointers[i] = StackPointerBacktrace {
+                        stack_pointer: 0,
+                        initial_backtrace_size: EXCEPTION_BUFFER.backtrace_size,
+                        current_backtrace_size: 0,
+                    };
+                    EXCEPTION_BUFFER.exception_count += 1;
+                    let _result =
+                        _Unwind_ForcedUnwind(&mut EXCEPTION_BUFFER.uw_exceptions[i], stop_fn, core::ptr::null_mut());
+                }
+            }
+        } else {
+            error!("too many nested exceptions");
+            // TODO: better reporting?
+            let exception = Exception {
+                id: get_exception_id("RuntimeError"),
+                file: file!().as_c_slice(),
+                line: line!(),
+                column: column!(),
+                // https://github.com/rust-lang/rfcs/pull/1719
+                function: "__artiq_raise".as_c_slice(),
+                message: "too many nested exceptions".as_c_slice(),
+                param: [0, 0, 0],
+            };
+            EXCEPTION_BUFFER.exceptions[MAX_INFLIGHT_EXCEPTIONS] = Some(mem::transmute(exception));
+            EXCEPTION_BUFFER.stack_pointers[MAX_INFLIGHT_EXCEPTIONS] = Default::default();
+            EXCEPTION_BUFFER.exception_count += 1;
+            uncaught_exception()
+        }
+    }
+    unreachable!();
+}
+
+fn read_exception_string<'a>(reader: &mut Cursor<&[u8]>) -> Result<CSlice<'a, u8>, ReadError> {
+    let len = reader.read_u32()? as usize;
+    if len == usize::MAX {
+        let data = reader.read_u32()?;
+        Ok(unsafe { CSlice::new(data as *const u8, len) })
+    } else {
+        let pos = reader.position();
+        let slice = unsafe {
+            let ptr = reader.get_ref().as_ptr().offset(pos as isize);
+            CSlice::new(ptr, len)
+        };
+        reader.set_position(pos + len);
+        Ok(slice)
+    }
+}
+
+fn read_exception(raw_exception: &[u8]) -> Result<Exception, ReadError> {
+    let mut reader = Cursor::new(raw_exception);
+
+    let mut byte = reader.read_u8()?;
+    // to sync
+    while byte != 0x5a {
+        byte = reader.read_u8()?;
+    }
+    // skip sync bytes, 0x09 indicates exception
+    while byte != 0x09 {
+        byte = reader.read_u8()?;
+    }
+    let _len = reader.read_u32()?;
+    // ignore the remaining exceptions, stack traces etc. - unwinding from another device would be unwise anyway
+    Ok(Exception {
+        id: reader.read_u32()?,
+        message: read_exception_string(&mut reader)?,
+        param: [
+            reader.read_u64()? as i64,
+            reader.read_u64()? as i64,
+            reader.read_u64()? as i64,
+        ],
+        file: read_exception_string(&mut reader)?,
+        line: reader.read_u32()?,
+        column: reader.read_u32()?,
+        function: read_exception_string(&mut reader)?,
+    })
+}
+
+pub fn raise_raw(raw_exception: &[u8]) -> ! {
+    use crate::artiq_raise;
+    if let Ok(exception) = read_exception(raw_exception) {
+        unsafe { raise(&exception) };
+    } else {
+        artiq_raise!("SubkernelError", "Error passing exception");
+    }
+}
+
+pub unsafe extern "C" fn resume() -> ! {
+    trace!("resume");
+    assert!(EXCEPTION_BUFFER.exception_count != 0);
+    let i = EXCEPTION_BUFFER.exception_stack[EXCEPTION_BUFFER.exception_count - 1];
+    assert!(i != -1);
+    let _result = _Unwind_ForcedUnwind(
+        &mut EXCEPTION_BUFFER.uw_exceptions[i as usize],
+        stop_fn,
+        core::ptr::null_mut(),
+    );
+    unreachable!()
+}
+
+pub unsafe extern "C" fn end_catch() {
+    let mut count = EXCEPTION_BUFFER.exception_count;
+    assert!(count != 0);
+    // we remove all exceptions with SP <= current exception SP
+    // i.e. the outer exception escapes the finally block
+    let index = EXCEPTION_BUFFER.exception_stack[count - 1] as usize;
+    EXCEPTION_BUFFER.exception_stack[count - 1] = -1;
+    EXCEPTION_BUFFER.exceptions[index] = None;
+    let outer_sp = EXCEPTION_BUFFER.stack_pointers[index].stack_pointer;
+    count -= 1;
+    for i in (0..count).rev() {
+        let index = EXCEPTION_BUFFER.exception_stack[i];
+        assert!(index != -1);
+        let index = index as usize;
+        let sp = EXCEPTION_BUFFER.stack_pointers[index].stack_pointer;
+        if sp >= outer_sp {
+            break;
+        }
+        EXCEPTION_BUFFER.exceptions[index] = None;
+        EXCEPTION_BUFFER.exception_stack[i] = -1;
+        count -= 1;
+    }
+    EXCEPTION_BUFFER.exception_count = count;
+    EXCEPTION_BUFFER.backtrace_size = if count > 0 {
+        let index = EXCEPTION_BUFFER.exception_stack[count - 1];
+        assert!(index != -1);
+        EXCEPTION_BUFFER.stack_pointers[index as usize].current_backtrace_size
+    } else {
+        0
+    };
+}
+
+extern "C" fn cleanup(_unwind_code: uw::_Unwind_Reason_Code, _uw_exception: *mut uw::_Unwind_Exception) {
+    unimplemented!()
+}
+
+fn uncaught_exception() -> ! {
+    unsafe {
+        // dump way to reorder the stack
+        for i in 0..EXCEPTION_BUFFER.exception_count {
+            if EXCEPTION_BUFFER.exception_stack[i] != i as isize {
+                // find the correct index
+                let index = EXCEPTION_BUFFER
+                    .exception_stack
+                    .iter()
+                    .position(|v| *v == i as isize)
+                    .unwrap();
+                let a = EXCEPTION_BUFFER.exception_stack[index];
+                let b = EXCEPTION_BUFFER.exception_stack[i];
+                assert!(a != -1 && b != -1);
+                core::mem::swap(
+                    &mut EXCEPTION_BUFFER.exception_stack[index],
+                    &mut EXCEPTION_BUFFER.exception_stack[i],
+                );
+                core::mem::swap(
+                    &mut EXCEPTION_BUFFER.exceptions[a as usize],
+                    &mut EXCEPTION_BUFFER.exceptions[b as usize],
+                );
+                core::mem::swap(
+                    &mut EXCEPTION_BUFFER.stack_pointers[a as usize],
+                    &mut EXCEPTION_BUFFER.stack_pointers[b as usize],
+                );
+            }
+        }
+    }
+    unsafe {
+        crate::kernel::core1::terminate(
+            EXCEPTION_BUFFER.exceptions[..EXCEPTION_BUFFER.exception_count].as_ref(),
+            EXCEPTION_BUFFER.stack_pointers[..EXCEPTION_BUFFER.exception_count].as_ref(),
+            EXCEPTION_BUFFER.backtrace[..EXCEPTION_BUFFER.backtrace_size].as_mut(),
+        )
+    }
+}
+
+// stop function which would be executed when we unwind each frame
+extern "C" fn stop_fn(
+    _version: c_int,
+    actions: i32,
+    _uw_exception_class: uw::_Unwind_Exception_Class,
+    _uw_exception: *mut uw::_Unwind_Exception,
+    context: *mut uw::_Unwind_Context,
+    _stop_parameter: *mut c_void,
+) -> uw::_Unwind_Reason_Code {
+    unsafe {
+        let load_addr = KERNEL_IMAGE.as_ref().unwrap().get_load_addr();
+        let backtrace_size = EXCEPTION_BUFFER.backtrace_size;
+        // we try to remove unrelated backtrace here to save some buffer size
+        if backtrace_size < MAX_BACKTRACE_SIZE {
+            let ip = uw::_Unwind_GetIP(context);
+            if ip >= load_addr {
+                let ip = ip - load_addr;
+                let sp = uw::_Unwind_GetGR(context, uw::UNWIND_SP_REG);
+                trace!("SP: {:X}, backtrace_size: {}", sp, backtrace_size);
+                EXCEPTION_BUFFER.backtrace[backtrace_size] = (ip, sp);
+                EXCEPTION_BUFFER.backtrace_size += 1;
+                let last_index = EXCEPTION_BUFFER.exception_stack[EXCEPTION_BUFFER.exception_count - 1];
+                assert!(last_index != -1);
+                let sp_info = &mut EXCEPTION_BUFFER.stack_pointers[last_index as usize];
+                sp_info.stack_pointer = sp;
+                sp_info.current_backtrace_size = backtrace_size + 1;
+            }
+        } else {
+            trace!("backtrace size exceeded");
+        }
+
+        if actions as u32 & uw::_US_END_OF_STACK as u32 != 0 {
+            uncaught_exception()
+        } else {
+            uw::_URC_NO_REASON
+        }
+    }
+}
+
+// Must be kept in sync with preallocate_runtime_exception_names() in `artiq.compiler.embedding`
+static EXCEPTION_ID_LOOKUP: [(&str, u32); 22] = [
+    ("RTIOUnderflow", 0),
+    ("RTIOOverflow", 1),
+    ("RTIODestinationUnreachable", 2),
+    ("DMAError", 3),
+    ("I2CError", 4),
+    ("CacheError", 5),
+    ("SPIError", 6),
+    ("SubkernelError", 7),
+    ("AssertionError", 8),
+    ("AttributeError", 9),
+    ("IndexError", 10),
+    ("IOError", 11),
+    ("KeyError", 12),
+    ("NotImplementedError", 13),
+    ("OverflowError", 14),
+    ("RuntimeError", 15),
+    ("TimeoutError", 16),
+    ("TypeError", 17),
+    ("ValueError", 18),
+    ("ZeroDivisionError", 19),
+    ("LinAlgError", 20),
+    ("UnwrapNoneError", 21),
+];
+
+pub fn get_exception_id(name: &str) -> u32 {
+    for (n, id) in EXCEPTION_ID_LOOKUP.iter() {
+        if *n == name {
+            return *id;
+        }
+    }
+    unimplemented!("unallocated internal exception id")
+}
+
+#[macro_export]
+macro_rules! artiq_raise {
+    ($name:expr, $message:expr, $param0:expr, $param1:expr, $param2:expr) => {{
+        use cslice::AsCSlice;
+        let name_id = $crate::eh_artiq::get_exception_id($name);
+        let message_cl = $message.clone();
+        let exn = $crate::eh_artiq::Exception {
+            id: name_id,
+            file: file!().as_c_slice(),
+            line: line!(),
+            column: column!(),
+            // https://github.com/rust-lang/rfcs/pull/1719
+            function: "(Rust function)".as_c_slice(),
+            message: message_cl.as_c_slice(),
+            param: [$param0, $param1, $param2],
+        };
+        #[allow(unused_unsafe)]
+        unsafe {
+            $crate::eh_artiq::raise(&exn)
+        }
+    }};
+    ($name:expr, $message:expr) => {{ artiq_raise!($name, $message, 0, 0, 0) }};
+}
+
+/// Takes as input exception id from host
+/// Generates a new exception with:
+///   * `id` set to `exn_id`
+///   * `message` set to corresponding exception name from `EXCEPTION_ID_LOOKUP`
+///
+/// The message is matched on host to ensure correct exception is being referred
+/// This test checks the synchronization of exception ids for runtime errors
+#[no_mangle]
+pub extern "C" fn test_exception_id_sync(exn_id: u32) {
+    let message = EXCEPTION_ID_LOOKUP
+        .iter()
+        .find_map(|&(name, id)| if id == exn_id { Some(name) } else { None })
+        .unwrap_or("unallocated internal exception id");
+
+    let exn = Exception {
+        id: exn_id,
+        file: file!().as_c_slice(),
+        line: 0,
+        column: 0,
+        function: "test_exception_id_sync".as_c_slice(),
+        message: message.as_c_slice(),
+        param: [0, 0, 0],
+    };
+    unsafe { raise(&exn) };
+}

src/libksupport/src/i2c.rs

@@ -2,9 +2,9 @@ use libboard_zynq;
 
 use crate::artiq_raise;
 
-static mut I2C_BUS: Option<libboard_zynq::i2c::I2c> = None;
+pub static mut I2C_BUS: Option<libboard_zynq::i2c::I2c> = None;
 
-pub extern fn start(busno: i32) {
+pub extern "C" fn start(busno: i32) {
     if busno > 0 {
         artiq_raise!("I2CError", "I2C bus could not be accessed");
     }
@@ -15,7 +15,7 @@ pub extern fn start(busno: i32) {
     }
 }
 
-pub extern fn restart(busno: i32) {
+pub extern "C" fn restart(busno: i32) {
     if busno > 0 {
         artiq_raise!("I2CError", "I2C bus could not be accessed");
     }
@@ -26,7 +26,7 @@ pub extern fn restart(busno: i32) {
     }
 }
 
-pub extern fn stop(busno: i32) {
+pub extern "C" fn stop(busno: i32) {
     if busno > 0 {
         artiq_raise!("I2CError", "I2C bus could not be accessed");
     }
@@ -37,7 +37,7 @@ pub extern fn stop(busno: i32) {
     }
 }
 
-pub extern fn write(busno: i32, data: i32) -> bool {
+pub extern "C" fn write(busno: i32, data: i32) -> bool {
     if busno > 0 {
         artiq_raise!("I2CError", "I2C bus could not be accessed");
     }
@@ -49,7 +49,7 @@ pub extern fn write(busno: i32, data: i32) -> bool {
     }
 }
 
-pub extern fn read(busno: i32, ack: bool) -> i32 {
+pub extern "C" fn read(busno: i32, ack: bool) -> i32 {
     if busno > 0 {
         artiq_raise!("I2CError", "I2C bus could not be accessed");
     }
@@ -61,6 +61,35 @@ pub extern fn read(busno: i32, ack: bool) -> i32 {
     }
 }
 
+pub extern "C" fn switch_select(busno: i32, address: i32, mask: i32) {
+    if busno > 0 {
+        artiq_raise!("I2CError", "I2C bus could not be accessed");
+    }
+    let ch = match mask {
+        //decode from mainline, PCA9548-centric API
+        0x00 => None,
+        0x01 => Some(0),
+        0x02 => Some(1),
+        0x04 => Some(2),
+        0x08 => Some(3),
+        0x10 => Some(4),
+        0x20 => Some(5),
+        0x40 => Some(6),
+        0x80 => Some(7),
+        _ => artiq_raise!("I2CError", "switch select supports only one channel"),
+    };
+    unsafe {
+        if (&mut I2C_BUS)
+            .as_mut()
+            .unwrap()
+            .pca954x_select(address as u8, ch)
+            .is_err()
+        {
+            artiq_raise!("I2CError", "switch select failed");
+        }
+    }
+}
+
 pub fn init() {
     let mut i2c = libboard_zynq::i2c::I2c::i2c0();
     i2c.init().expect("I2C bus initialization failed");

src/libksupport/src/irq.rs

@@ -1,12 +1,9 @@
-use libboard_zynq::{gic, mpcore, println, stdio};
-use libcortex_a9::{
-    asm,
-    regs::{MPIDR, SP},
-    spin_lock_yield, notify_spin_lock
-};
-use libregister::{RegisterR, RegisterW};
 use core::sync::atomic::{AtomicBool, Ordering};
 
+use libboard_zynq::{gic, mpcore, println, stdio};
+use libcortex_a9::{asm, interrupt_handler, notify_spin_lock, regs::MPIDR, spin_lock_yield};
+use libregister::RegisterR;
+
 extern "C" {
     static mut __stack1_start: u32;
     fn main_core1() -> !;
@@ -14,10 +11,7 @@ extern "C" {
 
 static CORE1_RESTART: AtomicBool = AtomicBool::new(false);
 
-#[link_section = ".text.boot"]
-#[no_mangle]
-#[naked]
-pub unsafe extern "C" fn IRQ() {
+interrupt_handler!(IRQ, irq, __irq_stack0_start, __irq_stack1_start, {
     if MPIDR.read().cpu_id() == 1 {
         let mpcore = mpcore::RegisterBlock::mpcore();
         let mut gic = gic::InterruptController::gic(mpcore);
@@ -25,17 +19,23 @@ pub unsafe extern "C" fn IRQ() {
         if id.0 == 0 {
             gic.end_interrupt(id);
             asm::exit_irq();
-            SP.write(&mut __stack1_start as *mut _ as u32);
-            asm::enable_irq();
-            CORE1_RESTART.store(false, Ordering::Relaxed);
-            notify_spin_lock();
-            main_core1();
+            asm!("b core1_restart");
         }
     }
     stdio::drop_uart();
     println!("IRQ");
     loop {}
-}
+});
+
+// This is actually not an interrupt handler, just use the macro for convenience.
+// This function would be called in normal mode (instead of interrupt mode), the outer naked
+// function wrapper is to tell libunwind to stop when it reaches here.
+interrupt_handler!(core1_restart, core1_restart_impl, __stack0_start, __stack1_start, {
+    asm::enable_irq();
+    CORE1_RESTART.store(false, Ordering::Relaxed);
+    notify_spin_lock();
+    main_core1();
+});
 
 pub fn restart_core1() {
     let mut interrupt_controller = gic::InterruptController::gic(mpcore::RegisterBlock::mpcore());

src/libksupport/src/kernel/api.rs

@@ -1,29 +1,27 @@
-use core::ffi::VaList;
-use core::ptr;
-use core::str;
+use alloc::vec;
+use core::{ffi::VaList, ptr, str};
+
 use libc::{c_char, c_int, size_t};
 use libm;
 use log::{info, warn};
 
-use alloc::vec;
-
-use crate::eh_artiq;
-use crate::rtio;
-use crate::i2c;
-use super::rpc::{rpc_send, rpc_send_async, rpc_recv};
-use super::dma;
-use super::cache;
-
+#[cfg(has_drtio)]
+use super::subkernel;
+use super::{cache,
+            core1::rtio_get_destination_status,
+            dma, linalg,
+            rpc::{rpc_recv, rpc_send, rpc_send_async}};
+use crate::{eh_artiq, i2c, rtio};
 
 extern "C" {
     fn vsnprintf_(buffer: *mut c_char, count: size_t, format: *const c_char, va: VaList) -> c_int;
 }
 
-unsafe extern fn core_log(fmt: *const c_char, mut args: ...) {
+unsafe extern "C" fn core_log(fmt: *const c_char, mut args: ...) {
     let size = vsnprintf_(ptr::null_mut(), 0, fmt, args.as_va_list()) as usize;
     let mut buf = vec![0; size + 1];
     vsnprintf_(buf.as_mut_ptr() as *mut i8, size + 1, fmt, args.as_va_list());
-    let buf: &[u8] = &buf.as_slice()[..size-1]; // strip \n and NUL
+    let buf: &[u8] = &buf.as_slice()[..size - 1]; // strip \n and NUL
     match str::from_utf8(buf) {
         Ok(s) => info!("kernel: {}", s),
         Err(e) => {
@@ -33,14 +31,13 @@ unsafe extern fn core_log(fmt: *const c_char, mut args: ...) {
     }
 }
 
-unsafe extern fn rtio_log(fmt: *const c_char, mut args: ...) {
+unsafe extern "C" fn rtio_log(fmt: *const c_char, mut args: ...) {
     let size = vsnprintf_(ptr::null_mut(), 0, fmt, args.as_va_list()) as usize;
     let mut buf = vec![0; size + 1];
     vsnprintf_(buf.as_mut_ptr(), size + 1, fmt, args.as_va_list());
     rtio::write_log(buf.as_slice());
 }
 
-
 macro_rules! api {
     ($i:ident) => ({
         extern { static $i: u8; }
@@ -56,15 +53,25 @@ macro_rules! api {
 }
 
 macro_rules! api_libm_f64f64 {
-    ($i:ident) => ({
-        extern fn $i(x: f64) -> f64 {
+    ($i:ident) => {{
+        extern "C" fn $i(x: f64) -> f64 {
             libm::$i(x)
         }
         api!($i = $i)
-    })
+    }};
+}
+
+macro_rules! api_libm_f64f64f64 {
+    ($i:ident) => {{
+        extern "C" fn $i(x: f64, y: f64) -> f64 {
+            libm::$i(x, y)
+        }
+        api!($i = $i)
+    }};
 }
 
 pub fn resolve(required: &[u8]) -> Option<u32> {
+    #[rustfmt::skip]
     let api = &[
         // timing
         api!(now_mu = rtio::now_mu),
@@ -78,7 +85,7 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
 
         // rtio
         api!(rtio_init = rtio::init),
-        api!(rtio_get_destination_status = rtio::get_destination_status),
+        api!(rtio_get_destination_status = rtio_get_destination_status),
         api!(rtio_get_counter = rtio::get_counter),
         api!(rtio_output = rtio::output),
         api!(rtio_output_wide = rtio::output_wide),
@@ -107,6 +114,17 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(i2c_stop = i2c::stop),
         api!(i2c_write = i2c::write),
         api!(i2c_read = i2c::read),
+        api!(i2c_switch_select = i2c::switch_select),
+
+        // subkernel
+        #[cfg(has_drtio)]
+        api!(subkernel_load_run = subkernel::load_run),
+        #[cfg(has_drtio)]
+        api!(subkernel_await_finish = subkernel::await_finish),
+        #[cfg(has_drtio)]
+        api!(subkernel_send_message = subkernel::send_message),
+        #[cfg(has_drtio)]
+        api!(subkernel_await_message = subkernel::await_message),
 
         // Double-precision floating-point arithmetic helper functions
         // RTABI chapter 4.1.2, Table 2
@@ -114,6 +132,7 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_ddiv),
         api!(__aeabi_dmul),
         api!(__aeabi_dsub),
+
         // Double-precision floating-point comparison helper functions
         // RTABI chapter 4.1.2, Table 3
         api!(__aeabi_dcmpeq),
@@ -123,12 +142,14 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_dcmpge),
         api!(__aeabi_dcmpgt),
         api!(__aeabi_dcmpun),
+
         // Single-precision floating-point arithmetic helper functions
         // RTABI chapter 4.1.2, Table 4
         api!(__aeabi_fadd),
         api!(__aeabi_fdiv),
         api!(__aeabi_fmul),
         api!(__aeabi_fsub),
+
         // Single-precision floating-point comparison helper functions
         // RTABI chapter 4.1.2, Table 5
         api!(__aeabi_fcmpeq),
@@ -138,6 +159,7 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_fcmpge),
         api!(__aeabi_fcmpgt),
         api!(__aeabi_fcmpun),
+
         // Floating-point to integer conversions.
         // RTABI chapter 4.1.2, Table 6
         api!(__aeabi_d2iz),
@@ -148,9 +170,11 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_f2uiz),
         api!(__aeabi_f2lz),
         api!(__aeabi_f2ulz),
+
         // Conversions between floating types.
         // RTABI chapter 4.1.2, Table 7
         api!(__aeabi_f2d),
+
         // Integer to floating-point conversions.
         // RTABI chapter 4.1.2, Table 8
         api!(__aeabi_i2d),
@@ -161,12 +185,14 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_ui2f),
         api!(__aeabi_l2f),
         api!(__aeabi_ul2f),
+
         // Long long helper functions
         // RTABI chapter 4.2, Table 9
         api!(__aeabi_lmul),
         api!(__aeabi_llsl),
         api!(__aeabi_llsr),
         api!(__aeabi_lasr),
+
         // Integer division functions
         // RTABI chapter 4.3.1
         api!(__aeabi_idiv),
@@ -174,6 +200,7 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_idivmod),
         api!(__aeabi_uidiv),
         api!(__aeabi_uldivmod),
+
         // 4.3.4 Memory copying, clearing, and setting
         api!(__aeabi_memcpy8),
         api!(__aeabi_memcpy4),
@@ -189,13 +216,43 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api!(__aeabi_memclr),
 
         // libc
-        api!(memcmp, extern { fn memcmp(a: *const u8, b: *mut u8, size: usize); }),
+        api!(
+            memcpy,
+            extern "C" {
+                fn memcpy(dest: *mut u8, src: *const u8, n: usize) -> *mut u8;
+            }
+        ),
+        api!(
+            memmove,
+            extern "C" {
+                fn memmove(dest: *mut u8, src: *const u8, n: usize) -> *mut u8;
+            }
+        ),
+        api!(
+            memset,
+            extern "C" {
+                fn memset(s: *mut u8, c: i32, n: usize) -> *mut u8;
+            }
+        ),
+        api!(
+            memcmp,
+            extern "C" {
+                fn memcmp(s1: *const u8, s2: *const u8, n: usize) -> i32;
+            }
+        ),
 
         // exceptions
         api!(_Unwind_Resume = unwind::_Unwind_Resume),
+        api!(__nac3_personality = eh_artiq::artiq_personality),
+        api!(__nac3_raise = eh_artiq::raise),
+        api!(__nac3_resume = eh_artiq::resume),
+        api!(__nac3_end_catch = eh_artiq::end_catch),
+
+        // legacy exception symbols
         api!(__artiq_personality = eh_artiq::artiq_personality),
         api!(__artiq_raise = eh_artiq::raise),
-        api!(__artiq_reraise = eh_artiq::reraise),
+        api!(__artiq_resume = eh_artiq::resume),
+        api!(__artiq_end_catch = eh_artiq::end_catch),
 
         // Implementations for LLVM math intrinsics
         api!(__powidf2),
@@ -206,15 +263,11 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api_libm_f64f64!(asin),
         api_libm_f64f64!(asinh),
         api_libm_f64f64!(atan),
-        {
-            extern fn atan2(y: f64, x: f64) -> f64 {
-                libm::atan2(y, x)
-            }
-            api!(atan2 = atan2)
-        },
+        api_libm_f64f64f64!(atan2),
         api_libm_f64f64!(atanh),
         api_libm_f64f64!(cbrt),
         api_libm_f64f64!(ceil),
+        api_libm_f64f64f64!(copysign),
         api_libm_f64f64!(cos),
         api_libm_f64f64!(cosh),
         api_libm_f64f64!(erf),
@@ -226,27 +279,19 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api_libm_f64f64!(fabs),
         api_libm_f64f64!(floor),
         {
-            extern fn fma(x: f64, y: f64, z: f64) -> f64 {
+            extern "C" fn fma(x: f64, y: f64, z: f64) -> f64 {
                 libm::fma(x, y, z)
             }
             api!(fma = fma)
         },
-        {
-            extern fn fmod(x: f64, y: f64) -> f64 {
-                libm::fmod(x, y)
-            }
-            api!(fmod = fmod)
-        },
-        {
-            extern fn hypot(x: f64, y: f64) -> f64 {
-                libm::hypot(x, y)
-            }
-            api!(hypot = hypot)
-        },
+        api_libm_f64f64f64!(fmax),
+        api_libm_f64f64f64!(fmin),
+        api_libm_f64f64f64!(fmod),
+        api_libm_f64f64f64!(hypot),
         api_libm_f64f64!(j0),
         api_libm_f64f64!(j1),
         {
-            extern fn jn(n: i32, x: f64) -> f64 {
+            extern "C" fn jn(n: i32, x: f64) -> f64 {
                 libm::jn(n, x)
             }
             api!(jn = jn)
@@ -255,13 +300,10 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api_libm_f64f64!(log),
         api_libm_f64f64!(log2),
         api_libm_f64f64!(log10),
-        {
-            extern fn pow(x: f64, y: f64) -> f64 {
-                libm::pow(x, y)
-            }
-            api!(pow = pow)
-        },
+        api_libm_f64f64f64!(nextafter),
+        api_libm_f64f64f64!(pow),
         api_libm_f64f64!(round),
+        api_libm_f64f64!(rint),
         api_libm_f64f64!(sin),
         api_libm_f64f64!(sinh),
         api_libm_f64f64!(sqrt),
@@ -272,13 +314,33 @@ pub fn resolve(required: &[u8]) -> Option<u32> {
         api_libm_f64f64!(y0),
         api_libm_f64f64!(y1),
         {
-            extern fn yn(n: i32, x: f64) -> f64 {
+            extern "C" fn yn(n: i32, x: f64) -> f64 {
                 libm::yn(n, x)
             }
             api!(yn = yn)
         },
+
+        // linalg
+        api!(np_linalg_cholesky = linalg::np_linalg_cholesky),
+        api!(np_linalg_qr = linalg::np_linalg_qr),
+        api!(np_linalg_svd = linalg::np_linalg_svd),
+        api!(np_linalg_inv = linalg::np_linalg_inv),
+        api!(np_linalg_pinv = linalg::np_linalg_pinv),
+        api!(np_linalg_matrix_power = linalg::np_linalg_matrix_power),
+        api!(np_linalg_det = linalg::np_linalg_det),
+        api!(sp_linalg_lu = linalg::sp_linalg_lu),
+        api!(sp_linalg_schur = linalg::sp_linalg_schur),
+        api!(sp_linalg_hessenberg = linalg::sp_linalg_hessenberg),
+
+        /*
+         * syscall for unit tests
+         * Used in `artiq.tests.coredevice.test_exceptions.ExceptionTest.test_raise_exceptions_kernel`
+         * This syscall checks that the exception IDs used in the Python `EmbeddingMap` (in `artiq.language.embedding`)
+         * match the `EXCEPTION_ID_LOOKUP` defined in the firmware (`libksupport::src::eh_artiq`)
+         */
+        api!(test_exception_id_sync = eh_artiq::test_exception_id_sync)
     ];
     api.iter()
-       .find(|&&(exported, _)| exported.as_bytes() == required)
-       .map(|&(_, ptr)| ptr as u32)
+        .find(|&&(exported, _)| exported.as_bytes() == required)
+        .map(|&(_, ptr)| ptr as u32)
 }

src/libksupport/src/kernel/cache.rs

@@ -0,0 +1,37 @@
+use alloc::{boxed::Box, string::String};
+use core::mem::{forget, transmute};
+
+use cslice::{AsCSlice, CSlice};
+
+use super::{Message, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0};
+
+pub extern "C" fn get(key: CSlice<u8>) -> &CSlice<'static, i32> {
+    let key = String::from_utf8(key.as_ref().to_vec()).unwrap();
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::CacheGetRequest(key));
+        let msg = KERNEL_CHANNEL_0TO1.as_mut().unwrap().recv();
+        if let Message::CacheGetReply(v) = msg {
+            let leaked = Box::new(v.as_c_slice());
+            let reference = transmute(leaked.as_ref());
+            forget(leaked);
+            forget(v);
+            reference
+        } else {
+            panic!("Expected CacheGetReply for CacheGetRequest");
+        }
+    }
+}
+
+pub extern "C" fn put(key: CSlice<u8>, list: &CSlice<i32>) {
+    let key = String::from_utf8(key.as_ref().to_vec()).unwrap();
+    let value = list.as_ref().to_vec();
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::CachePutRequest(key, value));
+    }
+}

src/libksupport/src/kernel/control.rs

@@ -1,11 +1,11 @@
-use libcortex_a9::sync_channel::{Sender, Receiver};
+use core::mem::{forget, replace};
+
+use libcortex_a9::sync_channel::{Receiver, Sender};
 use libsupport_zynq::boot::Core1;
 
-use super::{CHANNEL_0TO1, CHANNEL_1TO0, CHANNEL_SEM, INIT_LOCK, Message};
+use super::{Message, CHANNEL_0TO1, CHANNEL_1TO0, CHANNEL_SEM, INIT_LOCK};
 use crate::irq::restart_core1;
 
-use core::mem::{forget, replace};
-
 pub struct Control {
     pub tx: Sender<'static, Message>,
     pub rx: Receiver<'static, Message>,
@@ -53,4 +53,3 @@ impl Control {
         forget(replace(&mut self.rx, core0_rx));
     }
 }
-

src/libksupport/src/kernel/core1.rs

@@ -1,54 +1,39 @@
 //! Kernel prologue/epilogue that runs on the 2nd CPU core
 
-use core::{mem, ptr, cell::UnsafeCell};
 use alloc::borrow::ToOwned;
-use log::{debug, info, error};
-use cslice::CSlice;
+use core::{cell::UnsafeCell, mem, ptr};
 
-use libcortex_a9::{
-    enable_fpu,
-    cache::{dcci_slice, iciallu, bpiall},
-    asm::{dsb, isb},
-    sync_channel,
-};
-use libboard_zynq::{mpcore, gic};
+use cslice::CSlice;
+use dyld::{self, elf::EXIDX_Entry, Library};
+use libboard_zynq::{gic, mpcore};
+use libcortex_a9::{asm::{dsb, isb},
+                   cache::{bpiall, dcci_slice, iciallu},
+                   enable_fpu, sync_channel};
 use libsupport_zynq::ram;
-use dyld::{self, Library};
-use crate::{eh_artiq, rtio};
-use super::{
-    api::resolve,
-    rpc::rpc_send_async,
-    INIT_LOCK,
-    CHANNEL_0TO1, CHANNEL_1TO0,
-    CHANNEL_SEM,
-    KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0,
-    KERNEL_IMAGE,
-    Message,
-    dma,
-};
+use log::{debug, error, info};
+
+use super::{api::resolve, dma, rpc::rpc_send_async, Message, CHANNEL_0TO1, CHANNEL_1TO0, CHANNEL_SEM, INIT_LOCK,
+            KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0, KERNEL_IMAGE};
+use crate::{eh_artiq, get_async_errors, rtio};
 
 // linker symbols
 extern "C" {
-    #[no_mangle]
     static __text_start: u32;
-    #[no_mangle]
     static __text_end: u32;
-    #[no_mangle]
-    static __exidx_start: u32;
-    #[no_mangle]
-    static __exidx_end: u32;
+    static __exidx_start: EXIDX_Entry;
+    static __exidx_end: EXIDX_Entry;
 }
 
 unsafe fn attribute_writeback(typeinfo: *const ()) {
     struct Attr {
         offset: usize,
-        tag:    CSlice<'static, u8>,
-        name:   CSlice<'static, u8>
+        tag: CSlice<'static, u8>,
+        name: CSlice<'static, u8>,
     }
 
     struct Type {
         attributes: *const *const Attr,
-        objects:    *const *const ()
+        objects: *const *const (),
     }
 
     let mut tys = typeinfo as *const *const Type;
@@ -67,11 +52,16 @@ unsafe fn attribute_writeback(typeinfo: *const ()) {
                 attributes = attributes.offset(1);
 
                 if (*attribute).tag.len() > 0 {
-                    rpc_send_async(0, &(*attribute).tag, [
-                        &object as *const _ as *const (),
-                        &(*attribute).name as *const _ as *const (),
-                        (object as usize + (*attribute).offset) as *const ()
-                    ].as_ptr());
+                    rpc_send_async(
+                        0,
+                        &(*attribute).tag,
+                        [
+                            &object as *const _ as *const (),
+                            &(*attribute).name as *const _ as *const (),
+                            (object as usize + (*attribute).offset) as *const (),
+                        ]
+                        .as_ptr(),
+                    );
                 }
             }
         }
@@ -86,7 +76,8 @@ pub struct KernelImage {
 
 impl KernelImage {
     pub fn new(library: Library) -> Result<Self, dyld::Error> {
-        let __modinit__ = library.lookup(b"__modinit__")
+        let __modinit__ = library
+            .lookup(b"__modinit__")
             .ok_or(dyld::Error::Lookup("__modinit__".to_owned()))?;
         let typeinfo = library.lookup(b"typeinfo");
 
@@ -94,8 +85,7 @@ impl KernelImage {
         let bss_start = library.lookup(b"__bss_start");
         let end = library.lookup(b"_end");
         if let Some(bss_start) = bss_start {
-            let end = end
-                .ok_or(dyld::Error::Lookup("_end".to_owned()))?;
+            let end = end.ok_or(dyld::Error::Lookup("_end".to_owned()))?;
             unsafe {
                 ptr::write_bytes(bss_start as *mut u8, 0, (end - bss_start) as usize);
             }
@@ -122,7 +112,7 @@ impl KernelImage {
         dsb();
         isb();
 
-        (mem::transmute::<u32, fn()>(self.__modinit__))();
+        (mem::transmute::<u32, extern "C" fn()>(self.__modinit__))();
 
         if let Some(typeinfo) = self.typeinfo {
             attribute_writeback(typeinfo as *const ());
@@ -130,14 +120,12 @@ impl KernelImage {
     }
 
     pub fn get_load_addr(&self) -> usize {
-        unsafe {
-            self.library.get().as_ref().unwrap().image.as_ptr() as usize
-        }
+        unsafe { self.library.get().as_ref().unwrap().image.as_ptr() as usize }
     }
 }
 
 #[no_mangle]
-pub fn main_core1() {
+pub extern "C" fn main_core1() {
     enable_fpu();
     debug!("Core1 started");
 
@@ -168,24 +156,24 @@ pub fn main_core1() {
         let message = core1_rx.recv();
         match message {
             Message::LoadRequest(data) => {
-                let result = dyld::load(&data, &resolve)
-                    .and_then(KernelImage::new);
+                let result = dyld::load(&data, &resolve).and_then(KernelImage::new);
                 match result {
                     Ok(kernel) => {
                         loaded_kernel = Some(kernel);
                         debug!("kernel loaded");
                         core1_tx.send(Message::LoadCompleted);
-                    },
+                    }
                     Err(error) => {
                         error!("failed to load shared library: {}", error);
                         core1_tx.send(Message::LoadFailed);
                     }
                 }
-            },
+            }
             Message::StartRequest => {
                 info!("kernel starting");
                 if let Some(kernel) = loaded_kernel.take() {
                     unsafe {
+                        eh_artiq::reset_exception_buffer();
                         KERNEL_CHANNEL_0TO1 = Some(core1_rx);
                         KERNEL_CHANNEL_1TO0 = Some(core1_tx);
                         KERNEL_IMAGE = &kernel as *const KernelImage;
@@ -196,7 +184,8 @@ pub fn main_core1() {
                     }
                 }
                 info!("kernel finished");
-                core1_tx.send(Message::KernelFinished);
+                let async_errors = unsafe { get_async_errors() };
+                core1_tx.send(Message::KernelFinished(async_errors));
             }
             _ => error!("Core1 received unexpected message: {:?}", message),
         }
@@ -204,44 +193,62 @@ pub fn main_core1() {
 }
 
 /// Called by eh_artiq
-pub fn terminate(exception: &'static eh_artiq::Exception<'static>, backtrace: &'static mut [usize]) -> ! {
-    let load_addr = unsafe {
-        KERNEL_IMAGE.as_ref().unwrap().get_load_addr()
-    };
-    let mut cursor = 0;
-    // The address in the backtrace is relocated, so we have to convert it back to the address in
-    // the original python script, and remove those Rust function backtrace.
-    for i in 0..backtrace.len() {
-        if backtrace[i] >= load_addr {
-            backtrace[cursor] = backtrace[i] - load_addr;
-            cursor += 1;
-        }
-    }
-
+pub fn terminate(
+    exceptions: &'static [Option<eh_artiq::Exception<'static>>],
+    stack_pointers: &'static [eh_artiq::StackPointerBacktrace],
+    backtrace: &'static mut [(usize, usize)],
+) -> ! {
     {
         let core1_tx = unsafe { KERNEL_CHANNEL_1TO0.as_mut().unwrap() };
-        core1_tx.send(Message::KernelException(exception, &backtrace[..cursor]));
+        let errors = unsafe { get_async_errors() };
+        core1_tx.send(Message::KernelException(exceptions, stack_pointers, backtrace, errors));
     }
     loop {}
 }
 
 /// Called by llvm_libunwind
 #[no_mangle]
-extern fn dl_unwind_find_exidx(pc: *const u32, len_ptr: *mut u32) -> *const u32 {
+extern "C" fn dl_unwind_find_exidx(pc: *const u32, len_ptr: *mut u32) -> *const u32 {
     let length;
-    let start: *const u32;
+    let start: *const EXIDX_Entry;
     unsafe {
         if &__text_start as *const u32 <= pc && pc < &__text_end as *const u32 {
-            length = (&__exidx_end as *const u32).offset_from(&__exidx_start) as u32;
+            length = (&__exidx_end as *const EXIDX_Entry).offset_from(&__exidx_start) as u32;
             start = &__exidx_start;
-        } else {
-            let exidx = KERNEL_IMAGE.as_ref()
+        } else if KERNEL_IMAGE != ptr::null() {
+            let exidx = KERNEL_IMAGE
+                .as_ref()
                 .expect("dl_unwind_find_exidx kernel image")
-                .library.get().as_ref().unwrap().exidx();
+                .library
+                .get()
+                .as_ref()
+                .unwrap()
+                .exidx();
             length = exidx.len() as u32;
             start = exidx.as_ptr();
+        } else {
+            length = 0;
+            start = ptr::null();
         }
         *len_ptr = length;
     }
-    start
+    start as *const u32
+}
+
+pub extern "C" fn rtio_get_destination_status(destination: i32) -> bool {
+    #[cfg(has_drtio)]
+    if destination > 0 && destination < 255 {
+        let reply = unsafe {
+            let core1_rx = KERNEL_CHANNEL_0TO1.as_mut().unwrap();
+            let core1_tx = KERNEL_CHANNEL_1TO0.as_mut().unwrap();
+            core1_tx.send(Message::UpDestinationsRequest(destination));
+            core1_rx.recv()
+        };
+        return match reply {
+            Message::UpDestinationsReply(x) => x,
+            _ => panic!("received unexpected reply to UpDestinationsRequest: {:?}", reply),
+        };
+    }
+
+    destination == 0
 }

src/libksupport/src/kernel/dma.rs

@@ -0,0 +1,255 @@
+use alloc::{string::String, vec::Vec};
+use core::mem;
+
+use cslice::CSlice;
+
+use super::{Message, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0, KERNEL_IMAGE};
+use crate::{artiq_raise, pl::csr, rtio};
+
+#[repr(C)]
+pub struct DmaTrace {
+    duration: i64,
+    address: i32,
+    uses_ddma: bool,
+}
+
+#[derive(Clone, Debug)]
+pub struct DmaRecorder {
+    pub name: String,
+    pub buffer: Vec<u8>,
+    pub duration: i64,
+    pub enable_ddma: bool,
+}
+
+static mut RECORDER: Option<DmaRecorder> = None;
+
+pub unsafe fn init_dma_recorder() {
+    // as static would remain after restart, we have to reset it,
+    // without running its destructor.
+    mem::forget(mem::replace(&mut RECORDER, None));
+}
+
+pub extern "C" fn dma_record_start(name: CSlice<u8>) {
+    let name = String::from_utf8(name.as_ref().to_vec()).unwrap();
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::DmaEraseRequest(name.clone()));
+    }
+    unsafe {
+        if RECORDER.is_some() {
+            artiq_raise!("DMAError", "DMA is already recording")
+        }
+
+        let library = KERNEL_IMAGE.as_ref().unwrap();
+        library.rebind(b"rtio_output", dma_record_output as *const ()).unwrap();
+        library
+            .rebind(b"rtio_output_wide", dma_record_output_wide as *const ())
+            .unwrap();
+
+        RECORDER = Some(DmaRecorder {
+            name,
+            buffer: Vec::new(),
+            duration: 0,
+            enable_ddma: false,
+        });
+    }
+}
+
+pub extern "C" fn dma_record_stop(duration: i64, enable_ddma: bool) {
+    unsafe {
+        if RECORDER.is_none() {
+            artiq_raise!("DMAError", "DMA is not recording")
+        }
+
+        let library = KERNEL_IMAGE.as_ref().unwrap();
+        library.rebind(b"rtio_output", rtio::output as *const ()).unwrap();
+        library
+            .rebind(b"rtio_output_wide", rtio::output_wide as *const ())
+            .unwrap();
+
+        let mut recorder = RECORDER.take().unwrap();
+        recorder.duration = duration;
+        recorder.enable_ddma = enable_ddma;
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::DmaPutRequest(recorder));
+    }
+}
+
+#[inline(always)]
+unsafe fn dma_record_output_prepare(timestamp: i64, target: i32, words: usize) {
+    // See gateware/rtio/dma.py.
+    const HEADER_LENGTH: usize = /*length*/1 + /*channel*/3 + /*timestamp*/8 + /*address*/1;
+    let length = HEADER_LENGTH + /*data*/words * 4;
+
+    let buffer = &mut RECORDER.as_mut().unwrap().buffer;
+    buffer.reserve(length);
+    buffer.extend_from_slice(&[
+        (length >> 0) as u8,
+        (target >> 8) as u8,
+        (target >> 16) as u8,
+        (target >> 24) as u8,
+        (timestamp >> 0) as u8,
+        (timestamp >> 8) as u8,
+        (timestamp >> 16) as u8,
+        (timestamp >> 24) as u8,
+        (timestamp >> 32) as u8,
+        (timestamp >> 40) as u8,
+        (timestamp >> 48) as u8,
+        (timestamp >> 56) as u8,
+        (target >> 0) as u8,
+    ]);
+}
+
+pub extern "C" fn dma_record_output(target: i32, word: i32) {
+    unsafe {
+        let timestamp = rtio::now_mu();
+        dma_record_output_prepare(timestamp, target, 1);
+        RECORDER.as_mut().unwrap().buffer.extend_from_slice(&[
+            (word >> 0) as u8,
+            (word >> 8) as u8,
+            (word >> 16) as u8,
+            (word >> 24) as u8,
+        ]);
+    }
+}
+
+pub extern "C" fn dma_record_output_wide(target: i32, words: &CSlice<i32>) {
+    assert!(words.len() <= 16); // enforce the hardware limit
+
+    unsafe {
+        let timestamp = rtio::now_mu();
+        dma_record_output_prepare(timestamp, target, words.len());
+        let buffer = &mut RECORDER.as_mut().unwrap().buffer;
+        for word in words.as_ref().iter() {
+            buffer.extend_from_slice(&[
+                (word >> 0) as u8,
+                (word >> 8) as u8,
+                (word >> 16) as u8,
+                (word >> 24) as u8,
+            ]);
+        }
+    }
+}
+
+pub extern "C" fn dma_erase(name: CSlice<u8>) {
+    let name = String::from_utf8(name.as_ref().to_vec()).unwrap();
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::DmaEraseRequest(name));
+    }
+}
+
+pub extern "C" fn dma_retrieve(name: CSlice<u8>) -> DmaTrace {
+    let name = String::from_utf8(name.as_ref().to_vec()).unwrap();
+    unsafe {
+        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::DmaGetRequest(name));
+    }
+    match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
+        Message::DmaGetReply(None) => (),
+        Message::DmaGetReply(Some((address, duration, uses_ddma))) => {
+            return DmaTrace {
+                address,
+                duration,
+                uses_ddma,
+            };
+        }
+        _ => panic!("Expected DmaGetReply after DmaGetRequest!"),
+    }
+    // we have to defer raising error as we have to drop the message first...
+    artiq_raise!("DMAError", "DMA trace not found");
+}
+
+pub extern "C" fn dma_playback(timestamp: i64, ptr: i32, _uses_ddma: bool) {
+    unsafe {
+        csr::rtio_dma::base_address_write(ptr as u32);
+        csr::rtio_dma::time_offset_write(timestamp as u64);
+
+        let old_cri_master = csr::cri_con::selected_read();
+        csr::cri_con::selected_write(1);
+        csr::rtio_dma::enable_write(1);
+        #[cfg(has_drtio)]
+        if _uses_ddma {
+            KERNEL_CHANNEL_1TO0
+                .as_mut()
+                .unwrap()
+                .send(Message::DmaStartRemoteRequest {
+                    id: ptr,
+                    timestamp: timestamp,
+                });
+        }
+        while csr::rtio_dma::enable_read() != 0 {}
+        csr::cri_con::selected_write(old_cri_master);
+
+        let error = csr::rtio_dma::error_read();
+        if error != 0 {
+            let timestamp = csr::rtio_dma::error_timestamp_read();
+            let channel = csr::rtio_dma::error_channel_read();
+            csr::rtio_dma::error_write(1);
+            if error & 1 != 0 {
+                artiq_raise!(
+                    "RTIOUnderflow",
+                    "RTIO underflow at {1} mu, channel {rtio_channel_info:0}",
+                    channel as i64,
+                    timestamp as i64,
+                    0
+                );
+            }
+            if error & 2 != 0 {
+                artiq_raise!(
+                    "RTIODestinationUnreachable",
+                    "RTIO destination unreachable, output, at {1} mu, channel {rtio_channel_info:0}",
+                    channel as i64,
+                    timestamp as i64,
+                    0
+                );
+            }
+        }
+        #[cfg(has_drtio)]
+        if _uses_ddma {
+            KERNEL_CHANNEL_1TO0
+                .as_mut()
+                .unwrap()
+                .send(Message::DmaAwaitRemoteRequest(ptr));
+            match KERNEL_CHANNEL_0TO1.as_mut().unwrap().recv() {
+                Message::DmaAwaitRemoteReply {
+                    timeout,
+                    error,
+                    channel,
+                    timestamp,
+                } => {
+                    if timeout {
+                        artiq_raise!(
+                            "DMAError",
+                            "Error running DMA on satellite device, timed out waiting for results"
+                        );
+                    }
+                    if error & 1 != 0 {
+                        artiq_raise!(
+                            "RTIOUnderflow",
+                            "RTIO underflow at {1} mu, channel {rtio_channel_info:0}",
+                            channel as i64,
+                            timestamp as i64,
+                            0
+                        );
+                    }
+                    if error & 2 != 0 {
+                        artiq_raise!(
+                            "RTIODestinationUnreachable",
+                            "RTIO destination unreachable, output, at {1} mu, channel {rtio_channel_info:0}",
+                            channel as i64,
+                            timestamp as i64,
+                            0
+                        );
+                    }
+                }
+                _ => panic!("Expected DmaAwaitRemoteReply after DmaAwaitRemoteRequest!"),
+            }
+        }
+    }
+}

src/libksupport/src/kernel/linalg.rs

@@ -0,0 +1,440 @@
+// Uses `nalgebra` crate to invoke `np_linalg` and `sp_linalg` functions
+// When converting between `nalgebra::Matrix` and `NDArray` following considerations are necessary
+//
+// * Both `nalgebra::Matrix` and `NDArray` require their content to be stored in row-major order
+// * `NDArray` data pointer can be directly read and converted to `nalgebra::Matrix` (row and column number must be known)
+// * `nalgebra::Matrix::as_slice` returns the content of matrix in column-major order and initial data needs to be transposed before storing it in `NDArray` data pointer
+
+use alloc::vec::Vec;
+use core::slice;
+
+use nalgebra::DMatrix;
+
+use crate::artiq_raise;
+
+pub struct InputMatrix {
+    pub ndims: usize,
+    pub dims: *const usize,
+    pub data: *mut f64,
+}
+
+impl InputMatrix {
+    fn get_dims(&mut self) -> Vec<usize> {
+        let dims = unsafe { slice::from_raw_parts(self.dims, self.ndims) };
+        dims.to_vec()
+    }
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_cholesky(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix1 = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let result = matrix1.cholesky();
+    match result {
+        Some(res) => {
+            out_slice.copy_from_slice(res.unpack().transpose().as_slice());
+        }
+        None => {
+            artiq_raise!("LinAlgError", "Matrix is not positive definite");
+        }
+    };
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_qr(mat1: *mut InputMatrix, out_q: *mut InputMatrix, out_r: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_q = out_q.as_mut().unwrap();
+    let out_r = out_r.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let outq_dim = (*out_q).get_dims();
+    let outr_dim = (*out_r).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_q_slice = slice::from_raw_parts_mut(out_q.data, outq_dim[0] * outq_dim[1]);
+    let out_r_slice = slice::from_raw_parts_mut(out_r.data, outr_dim[0] * outr_dim[1]);
+
+    // Refer to https://github.com/dimforge/nalgebra/issues/735
+    let matrix1 = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+
+    let res = matrix1.qr();
+    let (q, r) = res.unpack();
+
+    // Uses different algo need to match numpy
+    out_q_slice.copy_from_slice(q.transpose().as_slice());
+    out_r_slice.copy_from_slice(r.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_svd(
+    mat1: *mut InputMatrix,
+    outu: *mut InputMatrix,
+    outs: *mut InputMatrix,
+    outvh: *mut InputMatrix,
+) {
+    let mat1 = mat1.as_mut().unwrap();
+    let outu = outu.as_mut().unwrap();
+    let outs = outs.as_mut().unwrap();
+    let outvh = outvh.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let outu_dim = (*outu).get_dims();
+    let outs_dim = (*outs).get_dims();
+    let outvh_dim = (*outvh).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_u_slice = slice::from_raw_parts_mut(outu.data, outu_dim[0] * outu_dim[1]);
+    let out_s_slice = slice::from_raw_parts_mut(outs.data, outs_dim[0]);
+    let out_vh_slice = slice::from_raw_parts_mut(outvh.data, outvh_dim[0] * outvh_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let result = matrix.svd(true, true);
+    out_u_slice.copy_from_slice(result.u.unwrap().transpose().as_slice());
+    out_s_slice.copy_from_slice(result.singular_values.as_slice());
+    out_vh_slice.copy_from_slice(result.v_t.unwrap().transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_inv(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+    let dim1 = (*mat1).get_dims();
+
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    if !matrix.is_invertible() {
+        artiq_raise!("LinAlgError", "no inverse for Singular Matrix");
+    }
+    let inv = matrix.try_inverse().unwrap();
+    out_slice.copy_from_slice(inv.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_pinv(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+    let dim1 = (*mat1).get_dims();
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let svd = matrix.svd(true, true);
+    let inv = svd.pseudo_inverse(1e-15);
+
+    match inv {
+        Ok(m) => {
+            out_slice.copy_from_slice(m.transpose().as_slice());
+        }
+        Err(_) => {
+            artiq_raise!("LinAlgError", "SVD computation does not converge");
+        }
+    }
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_matrix_power(mat1: *mut InputMatrix, mat2: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let mat2 = mat2.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let power = slice::from_raw_parts_mut(mat2.data, 1);
+    let power = power[0];
+    let outdim = out.get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, outdim[0] * outdim[1]);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let mut abs_power = power;
+    if abs_power < 0.0 {
+        abs_power = abs_power * -1.0;
+    }
+    let matrix1 = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    if !matrix1.is_square() {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+    let mut result = matrix1.pow(abs_power as u32);
+
+    if power < 0.0 {
+        if !matrix1.is_invertible() {
+            artiq_raise!("LinAlgError", "no inverse for Singular Matrix");
+        }
+        result = result.try_inverse().unwrap();
+    }
+    out_slice.copy_from_slice(result.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn np_linalg_det(mat1: *mut InputMatrix, out: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out = out.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+    let dim1 = (*mat1).get_dims();
+    let out_slice = slice::from_raw_parts_mut(out.data, 1);
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    if !matrix.is_square() {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+    out_slice[0] = matrix.determinant();
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn sp_linalg_lu(mat1: *mut InputMatrix, out_l: *mut InputMatrix, out_u: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_l = out_l.as_mut().unwrap();
+    let out_u = out_u.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+    let outl_dim = (*out_l).get_dims();
+    let outu_dim = (*out_u).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_l_slice = slice::from_raw_parts_mut(out_l.data, outl_dim[0] * outl_dim[1]);
+    let out_u_slice = slice::from_raw_parts_mut(out_u.data, outu_dim[0] * outu_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let (_, l, u) = matrix.lu().unpack();
+
+    out_l_slice.copy_from_slice(l.transpose().as_slice());
+    out_u_slice.copy_from_slice(u.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn sp_linalg_schur(mat1: *mut InputMatrix, out_t: *mut InputMatrix, out_z: *mut InputMatrix) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_t = out_t.as_mut().unwrap();
+    let out_z = out_z.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let out_t_dim = (*out_t).get_dims();
+    let out_z_dim = (*out_z).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_t_slice = slice::from_raw_parts_mut(out_t.data, out_t_dim[0] * out_t_dim[1]);
+    let out_z_slice = slice::from_raw_parts_mut(out_z.data, out_z_dim[0] * out_z_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let (z, t) = matrix.schur().unpack();
+
+    out_t_slice.copy_from_slice(t.transpose().as_slice());
+    out_z_slice.copy_from_slice(z.transpose().as_slice());
+}
+
+/// # Safety
+///
+/// `mat1` should point to a valid 2DArray of `f64` floats in row-major order
+#[no_mangle]
+pub unsafe extern "C" fn sp_linalg_hessenberg(
+    mat1: *mut InputMatrix,
+    out_h: *mut InputMatrix,
+    out_q: *mut InputMatrix,
+) {
+    let mat1 = mat1.as_mut().unwrap();
+    let out_h = out_h.as_mut().unwrap();
+    let out_q = out_q.as_mut().unwrap();
+
+    if mat1.ndims != 2 {
+        artiq_raise!(
+            "ValueError",
+            "expected 2D Vector Input, but received {1}D input)",
+            0,
+            mat1.ndims as i64,
+            0
+        );
+    }
+
+    let dim1 = (*mat1).get_dims();
+
+    if dim1[0] != dim1[1] {
+        artiq_raise!(
+            "ValueError",
+            "last 2 dimensions of the array must be square: {1} != {2}",
+            0,
+            dim1[0] as i64,
+            dim1[1] as i64
+        );
+    }
+
+    let out_h_dim = (*out_h).get_dims();
+    let out_q_dim = (*out_q).get_dims();
+
+    let data_slice1 = slice::from_raw_parts_mut(mat1.data, dim1[0] * dim1[1]);
+    let out_h_slice = slice::from_raw_parts_mut(out_h.data, out_h_dim[0] * out_h_dim[1]);
+    let out_q_slice = slice::from_raw_parts_mut(out_q.data, out_q_dim[0] * out_q_dim[1]);
+
+    let matrix = DMatrix::from_row_slice(dim1[0], dim1[1], data_slice1);
+    let (q, h) = matrix.hessenberg().unpack();
+
+    out_h_slice.copy_from_slice(h.transpose().as_slice());
+    out_q_slice.copy_from_slice(q.transpose().as_slice());
+}

src/libksupport/src/kernel/mod.rs

@@ -0,0 +1,128 @@
+use alloc::{string::String, vec::Vec};
+use core::ptr;
+
+use libcortex_a9::{mutex::Mutex, semaphore::Semaphore, sync_channel};
+
+use crate::{eh_artiq, RPCException};
+
+mod control;
+pub use control::Control;
+mod api;
+pub mod core1;
+mod dma;
+mod rpc;
+pub use dma::DmaRecorder;
+mod cache;
+mod linalg;
+#[cfg(has_drtio)]
+mod subkernel;
+
+#[cfg(has_drtio)]
+#[derive(Debug, Clone)]
+pub enum SubkernelStatus {
+    NoError,
+    Timeout,
+    IncorrectState,
+    CommLost,
+    Exception(Vec<u8>),
+    OtherError,
+}
+
+#[derive(Debug, Clone)]
+pub enum Message {
+    LoadRequest(Vec<u8>),
+    LoadCompleted,
+    LoadFailed,
+    StartRequest,
+    KernelFinished(u8),
+    KernelException(
+        &'static [Option<eh_artiq::Exception<'static>>],
+        &'static [eh_artiq::StackPointerBacktrace],
+        &'static [(usize, usize)],
+        u8,
+    ),
+    RpcSend {
+        is_async: bool,
+        data: Vec<u8>,
+    },
+    RpcRecvRequest(*mut ()),
+    RpcRecvReply(Result<usize, RPCException>),
+
+    CacheGetRequest(String),
+    CacheGetReply(Vec<i32>),
+    CachePutRequest(String, Vec<i32>),
+
+    DmaPutRequest(DmaRecorder),
+    DmaEraseRequest(String),
+    DmaGetRequest(String),
+    DmaGetReply(Option<(i32, i64, bool)>),
+    #[cfg(has_drtio)]
+    DmaStartRemoteRequest {
+        id: i32,
+        timestamp: i64,
+    },
+    #[cfg(has_drtio)]
+    DmaAwaitRemoteRequest(i32),
+    #[cfg(has_drtio)]
+    DmaAwaitRemoteReply {
+        timeout: bool,
+        error: u8,
+        channel: u32,
+        timestamp: u64,
+    },
+
+    #[cfg(has_drtio)]
+    UpDestinationsRequest(i32),
+    #[cfg(has_drtio)]
+    UpDestinationsReply(bool),
+
+    #[cfg(has_drtio)]
+    SubkernelLoadRunRequest {
+        id: u32,
+        destination: u8,
+        run: bool,
+        timestamp: u64,
+    },
+    #[cfg(has_drtio)]
+    SubkernelLoadRunReply {
+        succeeded: bool,
+    },
+    #[cfg(has_drtio)]
+    SubkernelAwaitFinishRequest {
+        id: u32,
+        timeout: i64,
+    },
+    #[cfg(has_drtio)]
+    SubkernelAwaitFinishReply,
+    #[cfg(has_drtio)]
+    SubkernelMsgSend {
+        id: u32,
+        destination: Option<u8>,
+        data: Vec<u8>,
+    },
+    #[cfg(has_drtio)]
+    SubkernelMsgSent,
+    #[cfg(has_drtio)]
+    SubkernelMsgRecvRequest {
+        id: i32,
+        timeout: i64,
+        tags: Vec<u8>,
+    },
+    #[cfg(has_drtio)]
+    SubkernelMsgRecvReply {
+        count: u8,
+    },
+    #[cfg(has_drtio)]
+    SubkernelError(SubkernelStatus),
+}
+
+static CHANNEL_0TO1: Mutex<Option<sync_channel::Sender<'static, Message>>> = Mutex::new(None);
+static CHANNEL_1TO0: Mutex<Option<sync_channel::Receiver<'static, Message>>> = Mutex::new(None);
+static CHANNEL_SEM: Semaphore = Semaphore::new(0, 1);
+
+static mut KERNEL_CHANNEL_0TO1: Option<sync_channel::Receiver<'static, Message>> = None;
+static mut KERNEL_CHANNEL_1TO0: Option<sync_channel::Sender<'static, Message>> = None;
+
+pub static mut KERNEL_IMAGE: *const core1::KernelImage = ptr::null();
+
+static INIT_LOCK: Mutex<()> = Mutex::new(());

src/libksupport/src/kernel/rpc.rs

@@ -1,31 +1,28 @@
 //! Kernel-side RPC API
 
 use alloc::vec::Vec;
-use cslice::{CSlice, AsCSlice};
 
-use crate::eh_artiq;
-use crate::rpc::send_args;
-use super::{
-    KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0,
-    Message,
-};
+use cslice::CSlice;
+
+use super::{Message, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0};
+use crate::{eh_artiq, rpc::send_args};
 
 fn rpc_send_common(is_async: bool, service: u32, tag: &CSlice<u8>, data: *const *const ()) {
     let core1_tx = unsafe { KERNEL_CHANNEL_1TO0.as_mut().unwrap() };
     let mut buffer = Vec::<u8>::new();
-    send_args(&mut buffer, service, tag.as_ref(), data).expect("RPC encoding failed");
+    send_args(&mut buffer, service, tag.as_ref(), data, true).expect("RPC encoding failed");
     core1_tx.send(Message::RpcSend { is_async, data: buffer });
 }
 
-pub extern fn rpc_send(service: u32, tag: &CSlice<u8>, data: *const *const ()) {
+pub extern "C" fn rpc_send(service: u32, tag: &CSlice<u8>, data: *const *const ()) {
     rpc_send_common(false, service, tag, data);
 }
 
-pub extern fn rpc_send_async(service: u32, tag: &CSlice<u8>, data: *const *const ()) {
+pub extern "C" fn rpc_send_async(service: u32, tag: &CSlice<u8>, data: *const *const ()) {
     rpc_send_common(true, service, tag, data);
 }
 
-pub extern fn rpc_recv(slot: *mut ()) -> usize {
+pub extern "C" fn rpc_recv(slot: *mut ()) -> usize {
     let reply = unsafe {
         let core1_rx = KERNEL_CHANNEL_0TO1.as_mut().unwrap();
         let core1_tx = KERNEL_CHANNEL_1TO0.as_mut().unwrap();
@@ -36,15 +33,15 @@ pub extern fn rpc_recv(slot: *mut ()) -> usize {
         Message::RpcRecvReply(Ok(alloc_size)) => alloc_size,
         Message::RpcRecvReply(Err(exception)) => unsafe {
             eh_artiq::raise(&eh_artiq::Exception {
-                name:     exception.name.as_bytes().as_c_slice(),
-                file:     exception.file.as_bytes().as_c_slice(),
-                line:     exception.line as u32,
-                column:   exception.column as u32,
-                function: exception.function.as_bytes().as_c_slice(),
-                message:  exception.message.as_bytes().as_c_slice(),
-                param:    exception.param
+                id: exception.id,
+                file: CSlice::new(exception.file as *const u8, usize::MAX),
+                line: exception.line as u32,
+                column: exception.column as u32,
+                function: CSlice::new(exception.function as *const u8, usize::MAX),
+                message: CSlice::new(exception.message as *const u8, usize::MAX),
+                param: exception.param,
             })
         },
-        _ => panic!("received unexpected reply to RpcRecvRequest: {:?}", reply)
+        _ => panic!("received unexpected reply to RpcRecvRequest: {:?}", reply),
     }
 }

src/libksupport/src/kernel/subkernel.rs

@@ -0,0 +1,112 @@
+use alloc::vec::Vec;
+
+use cslice::CSlice;
+
+use super::{Message, SubkernelStatus, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0};
+use crate::{artiq_raise, eh_artiq, rpc::send_args, rtio::now_mu};
+
+pub extern "C" fn load_run(id: u32, destination: u8, run: bool) {
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::SubkernelLoadRunRequest {
+                id: id,
+                destination: destination,
+                run: run,
+                timestamp: now_mu() as u64,
+            });
+    }
+    match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
+        Message::SubkernelLoadRunReply { succeeded: true } => (),
+        Message::SubkernelLoadRunReply { succeeded: false } => {
+            artiq_raise!("SubkernelError", "Error loading or running the subkernel")
+        }
+        _ => panic!("Expected SubkernelLoadRunReply after SubkernelLoadRunRequest!"),
+    }
+}
+
+pub extern "C" fn await_finish(id: u32, timeout: i64) {
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::SubkernelAwaitFinishRequest {
+                id: id,
+                timeout: timeout,
+            });
+    }
+    match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
+        Message::SubkernelAwaitFinishReply => (),
+        Message::SubkernelError(SubkernelStatus::IncorrectState) => {
+            artiq_raise!("SubkernelError", "Subkernel not running")
+        }
+        Message::SubkernelError(SubkernelStatus::Timeout) => artiq_raise!("SubkernelError", "Subkernel timed out"),
+        Message::SubkernelError(SubkernelStatus::CommLost) => {
+            artiq_raise!("SubkernelError", "Lost communication with satellite")
+        }
+        Message::SubkernelError(SubkernelStatus::OtherError) => {
+            artiq_raise!("SubkernelError", "An error occurred during subkernel operation")
+        }
+        Message::SubkernelError(SubkernelStatus::Exception(raw_exception)) => eh_artiq::raise_raw(&raw_exception),
+        _ => panic!("expected SubkernelAwaitFinishReply after SubkernelAwaitFinishRequest"),
+    }
+}
+
+pub extern "C" fn send_message(
+    id: u32,
+    is_return: bool,
+    destination: u8,
+    count: u8,
+    tag: &CSlice<u8>,
+    data: *const *const (),
+) {
+    let mut buffer = Vec::<u8>::new();
+    send_args(&mut buffer, 0, tag.as_ref(), data, false).expect("RPC encoding failed");
+    // overwrite service tag, include how many tags are in the message
+    buffer[3] = count;
+    unsafe {
+        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::SubkernelMsgSend {
+            id: id,
+            destination: if is_return { None } else { Some(destination) },
+            data: buffer[3..].to_vec(),
+        });
+    }
+    match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
+        Message::SubkernelMsgSent => (),
+        _ => panic!("expected SubkernelMsgSent after SubkernelMsgSend"),
+    }
+}
+
+pub extern "C" fn await_message(id: i32, timeout: i64, tags: &CSlice<u8>, min: u8, max: u8) {
+    unsafe {
+        KERNEL_CHANNEL_1TO0
+            .as_mut()
+            .unwrap()
+            .send(Message::SubkernelMsgRecvRequest {
+                id: id,
+                timeout: timeout,
+                tags: tags.as_ref().to_vec(),
+            });
+    }
+    match unsafe { KERNEL_CHANNEL_0TO1.as_mut().unwrap() }.recv() {
+        Message::SubkernelMsgRecvReply { count } => {
+            if min > count || count > max {
+                artiq_raise!("SubkernelError", "Received more or less arguments than required")
+            }
+        }
+        Message::SubkernelError(SubkernelStatus::IncorrectState) => {
+            artiq_raise!("SubkernelError", "Subkernel not running")
+        }
+        Message::SubkernelError(SubkernelStatus::Timeout) => artiq_raise!("SubkernelError", "Subkernel timed out"),
+        Message::SubkernelError(SubkernelStatus::CommLost) => {
+            artiq_raise!("SubkernelError", "Lost communication with satellite")
+        }
+        Message::SubkernelError(SubkernelStatus::OtherError) => {
+            artiq_raise!("SubkernelError", "An error occurred during subkernel operation")
+        }
+        Message::SubkernelError(SubkernelStatus::Exception(raw_exception)) => eh_artiq::raise_raw(&raw_exception),
+        _ => panic!("expected SubkernelMsgRecvReply after SubkernelMsgRecvRequest"),
+    }
+    // RpcRecvRequest should be called after this to receive message data
+}

src/libksupport/src/lib.rs

@@ -0,0 +1,163 @@
+#![no_std]
+#![feature(c_variadic)]
+#![feature(const_btree_new)]
+#![feature(const_in_array_repeat_expressions)]
+#![feature(naked_functions)]
+#![feature(asm)]
+
+#[macro_use]
+extern crate alloc;
+
+use alloc::{collections::BTreeMap, string::String};
+
+use io::{Cursor, ProtoRead};
+use libasync::block_async;
+use libconfig::Config;
+use log::{error, warn};
+#[cfg(has_drtiosat)]
+pub use pl::csr::drtiosat as rtio_core;
+#[cfg(has_rtio_core)]
+pub use pl::csr::rtio_core;
+use void::Void;
+
+pub mod eh_artiq;
+pub mod i2c;
+pub mod irq;
+pub mod kernel;
+pub mod rpc;
+#[cfg(ki_impl = "csr")]
+#[path = "rtio_csr.rs"]
+pub mod rtio;
+#[cfg(ki_impl = "acp")]
+#[path = "rtio_acp.rs"]
+pub mod rtio;
+#[rustfmt::skip]
+#[path = "../../../build/pl.rs"]
+pub mod pl;
+
+
+#[derive(Debug, Clone)]
+pub struct RPCException {
+    pub id: u32,
+    pub message: u32,
+    pub param: [i64; 3],
+    pub file: u32,
+    pub line: i32,
+    pub column: i32,
+    pub function: u32,
+}
+
+pub static mut SEEN_ASYNC_ERRORS: u8 = 0;
+
+pub const ASYNC_ERROR_COLLISION: u8 = 1 << 0;
+pub const ASYNC_ERROR_BUSY: u8 = 1 << 1;
+pub const ASYNC_ERROR_SEQUENCE_ERROR: u8 = 1 << 2;
+
+pub unsafe fn get_async_errors() -> u8 {
+    let errors = SEEN_ASYNC_ERRORS;
+    SEEN_ASYNC_ERRORS = 0;
+    errors
+}
+
+fn wait_for_async_rtio_error() -> nb::Result<(), Void> {
+    unsafe {
+        #[cfg(has_rtio_core)]
+        let errors = rtio_core::async_error_read();
+        #[cfg(has_drtiosat)]
+        let errors = rtio_core::protocol_error_read();
+        if errors != 0 {
+            Ok(())
+        } else {
+            Err(nb::Error::WouldBlock)
+        }
+    }
+}
+
+pub async fn report_async_rtio_errors() {
+    loop {
+        let _ = block_async!(wait_for_async_rtio_error()).await;
+        unsafe {
+            #[cfg(has_rtio_core)]
+            let errors = rtio_core::async_error_read();
+            #[cfg(has_drtiosat)]
+            let errors = rtio_core::protocol_error_read();
+            if errors & ASYNC_ERROR_COLLISION != 0 {
+                let channel = rtio_core::collision_channel_read();
+                error!(
+                    "RTIO collision involving channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                );
+            }
+            if errors & ASYNC_ERROR_BUSY != 0 {
+                let channel = rtio_core::busy_channel_read();
+                error!(
+                    "RTIO busy error involving channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                );
+            }
+            if errors & ASYNC_ERROR_SEQUENCE_ERROR != 0 {
+                let channel = rtio_core::sequence_error_channel_read();
+                error!(
+                    "RTIO sequence error involving channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                );
+            }
+            SEEN_ASYNC_ERRORS = errors;
+            #[cfg(has_rtio_core)]
+            rtio_core::async_error_write(errors);
+            #[cfg(has_drtiosat)]
+            rtio_core::protocol_error_write(errors);
+        }
+    }
+}
+
+static mut RTIO_DEVICE_MAP: BTreeMap<u32, String> = BTreeMap::new();
+
+fn read_device_map(cfg: &Config) -> BTreeMap<u32, String> {
+    let mut device_map: BTreeMap<u32, String> = BTreeMap::new();
+    let _ = cfg
+        .read("device_map")
+        .and_then(|raw_bytes| {
+            let mut bytes_cr = Cursor::new(raw_bytes);
+            let size = bytes_cr.read_u32().unwrap();
+            for _ in 0..size {
+                let channel = bytes_cr.read_u32().unwrap();
+                let device_name = bytes_cr.read_string().unwrap();
+                if let Some(old_entry) = device_map.insert(channel, device_name.clone()) {
+                    warn!(
+                        "conflicting device map entries for RTIO channel {}: '{}' and '{}'",
+                        channel, old_entry, device_name
+                    );
+                }
+            }
+            Ok(())
+        })
+        .or_else(|err| {
+            warn!(
+                "error reading device map ({}), device names will not be available in RTIO error messages",
+                err
+            );
+            Err(err)
+        });
+    device_map
+}
+
+fn _resolve_channel_name(channel: u32, device_map: &BTreeMap<u32, String>) -> String {
+    match device_map.get(&channel) {
+        Some(val) => val.clone(),
+        None => String::from("unknown"),
+    }
+}
+
+pub fn resolve_channel_name(channel: u32) -> String {
+    _resolve_channel_name(channel, unsafe { &RTIO_DEVICE_MAP })
+}
+
+pub fn setup_device_map(cfg: &Config) {
+    unsafe {
+        RTIO_DEVICE_MAP = read_device_map(cfg);
+    }
+}

src/libksupport/src/rpc.rs

@@ -0,0 +1,591 @@
+use core::str;
+
+use byteorder::{ByteOrder, NativeEndian};
+use core_io::{Error, Read, Write};
+use cslice::{CMutSlice, CSlice};
+use io::{ProtoRead, ProtoWrite};
+use log::trace;
+
+use self::tag::{split_tag, Tag, TagIterator};
+
+#[inline]
+pub fn round_up(val: usize, power_of_two: usize) -> usize {
+    assert!(power_of_two.is_power_of_two());
+    let max_rem = power_of_two - 1;
+    (val + max_rem) & (!max_rem)
+}
+
+#[inline]
+pub unsafe fn round_up_mut<T>(ptr: *mut T, power_of_two: usize) -> *mut T {
+    round_up(ptr as usize, power_of_two) as *mut T
+}
+
+#[inline]
+pub unsafe fn round_up_const<T>(ptr: *const T, power_of_two: usize) -> *const T {
+    round_up(ptr as usize, power_of_two) as *const T
+}
+
+#[inline]
+pub unsafe fn align_ptr<T>(ptr: *const ()) -> *const T {
+    round_up_const(ptr, core::mem::align_of::<T>()) as *const T
+}
+
+#[inline]
+pub unsafe fn align_ptr_mut<T>(ptr: *mut ()) -> *mut T {
+    round_up_mut(ptr, core::mem::align_of::<T>()) as *mut T
+}
+
+// versions for reader rather than TcpStream
+// they will be made into sync for satellite subkernels later
+unsafe fn recv_elements<F, R>(
+    reader: &mut R,
+    elt_tag: Tag,
+    length: usize,
+    storage: *mut (),
+    alloc: &mut F,
+) -> Result<(), Error>
+where
+    F: FnMut(usize) -> *mut (),
+    R: Read + ?Sized,
+{
+    match elt_tag {
+        Tag::Bool => {
+            let dest = core::slice::from_raw_parts_mut(storage as *mut u8, length);
+            reader.read_exact(dest)?;
+        }
+        Tag::Int32 => {
+            let ptr = storage as *mut u32;
+            let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 4);
+            reader.read_exact(dest)?;
+            drop(dest);
+            let dest = core::slice::from_raw_parts_mut(ptr, length);
+            NativeEndian::from_slice_u32(dest);
+        }
+        Tag::Int64 | Tag::Float64 => {
+            let ptr = storage as *mut u64;
+            let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 8);
+            reader.read_exact(dest)?;
+            drop(dest);
+            let dest = core::slice::from_raw_parts_mut(ptr, length);
+            NativeEndian::from_slice_u64(dest);
+        }
+        _ => {
+            let mut data = storage;
+            for _ in 0..length {
+                recv_value(reader, elt_tag, &mut data, alloc)?
+            }
+        }
+    }
+    Ok(())
+}
+
+unsafe fn recv_value<F, R>(reader: &mut R, tag: Tag, data: &mut *mut (), alloc: &mut F) -> Result<(), Error>
+where
+    F: FnMut(usize) -> *mut (),
+    R: Read + ?Sized,
+{
+    macro_rules! consume_value {
+        ($ty:ty, | $ptr:ident | $map:expr) => {{
+            let $ptr = align_ptr_mut::<$ty>(*data);
+            *data = $ptr.offset(1) as *mut ();
+            $map
+        }};
+    }
+
+    match tag {
+        Tag::None => Ok(()),
+        Tag::Bool => consume_value!(i8, |ptr| {
+            *ptr = reader.read_u8()? as i8;
+            Ok(())
+        }),
+        Tag::Int32 => consume_value!(i32, |ptr| {
+            *ptr = reader.read_u32()? as i32;
+            Ok(())
+        }),
+        Tag::Int64 | Tag::Float64 => consume_value!(i64, |ptr| {
+            *ptr = reader.read_u64()? as i64;
+            Ok(())
+        }),
+        Tag::String | Tag::Bytes | Tag::ByteArray => {
+            consume_value!(CMutSlice<u8>, |ptr| {
+                let length = reader.read_u32()? as usize;
+                *ptr = CMutSlice::new(alloc(length) as *mut u8, length);
+                reader.read_exact((*ptr).as_mut())?;
+                Ok(())
+            })
+        }
+        Tag::Tuple(it, arity) => {
+            let alignment = tag.alignment();
+            *data = round_up_mut(*data, alignment);
+            let mut it = it.clone();
+            for _ in 0..arity {
+                let tag = it.next().expect("truncated tag");
+                recv_value(reader, tag, data, alloc)?
+            }
+            *data = round_up_mut(*data, alignment);
+            Ok(())
+        }
+        Tag::List(it) => {
+            #[repr(C)]
+            struct List {
+                elements: *mut (),
+                length: usize,
+            }
+            consume_value!(*mut List, |ptr_to_list| {
+                let tag = it.clone().next().expect("truncated tag");
+                let length = reader.read_u32()? as usize;
+
+                let list_size = 4 + 4;
+                let storage_offset = round_up(list_size, tag.alignment());
+                let storage_size = tag.size() * length;
+
+                let allocation = alloc(storage_offset + storage_size) as *mut u8;
+                *ptr_to_list = allocation as *mut List;
+                let storage = allocation.offset(storage_offset as isize) as *mut ();
+
+                (**ptr_to_list).length = length;
+                (**ptr_to_list).elements = storage;
+                recv_elements(reader, tag, length, storage, alloc)
+            })
+        }
+        Tag::Array(it, num_dims) => {
+            consume_value!(*mut (), |buffer| {
+                let mut total_len: usize = 1;
+                for _ in 0..num_dims {
+                    let len = reader.read_u32()? as usize;
+                    total_len *= len;
+                    consume_value!(usize, |ptr| *ptr = len)
+                }
+
+                let elt_tag = it.clone().next().expect("truncated tag");
+                *buffer = alloc(elt_tag.size() * total_len);
+                recv_elements(reader, elt_tag, total_len, *buffer, alloc)
+            })
+        }
+        Tag::Range(it) => {
+            *data = round_up_mut(*data, tag.alignment());
+            let tag = it.clone().next().expect("truncated tag");
+            recv_value(reader, tag, data, alloc)?;
+            recv_value(reader, tag, data, alloc)?;
+            recv_value(reader, tag, data, alloc)?;
+            Ok(())
+        }
+        Tag::Keyword(_) => unreachable!(),
+        Tag::Object => unreachable!(),
+    }
+}
+
+pub fn recv_return<'a, F, R>(
+    reader: &mut R,
+    tag_bytes: &'a [u8],
+    data: *mut (),
+    alloc: &mut F,
+) -> Result<&'a [u8], Error>
+where
+    F: FnMut(usize) -> *mut (),
+    R: Read + ?Sized,
+{
+    let mut it = TagIterator::new(tag_bytes);
+    trace!("recv ...->{}", it);
+
+    let tag = it.next().expect("truncated tag");
+    let mut data = data;
+    unsafe { recv_value(reader, tag, &mut data, alloc)? };
+
+    Ok(it.data)
+}
+
+unsafe fn send_elements<W>(
+    writer: &mut W,
+    elt_tag: Tag,
+    length: usize,
+    data: *const (),
+    write_tags: bool,
+) -> Result<(), Error>
+where
+    W: Write + ?Sized,
+{
+    if write_tags {
+        writer.write_u8(elt_tag.as_u8())?;
+    }
+    match elt_tag {
+        // we cannot use NativeEndian::from_slice_i32 as the data is not mutable,
+        // and that is not needed as the data is already in native endian
+        Tag::Bool => {
+            let slice = core::slice::from_raw_parts(data as *const u8, length);
+            writer.write_all(slice)?;
+        }
+        Tag::Int32 => {
+            let slice = core::slice::from_raw_parts(data as *const u8, length * 4);
+            writer.write_all(slice)?;
+        }
+        Tag::Int64 | Tag::Float64 => {
+            let slice = core::slice::from_raw_parts(data as *const u8, length * 8);
+            writer.write_all(slice)?;
+        }
+        _ => {
+            let mut data = data;
+            for _ in 0..length {
+                send_value(writer, elt_tag, &mut data, write_tags)?;
+            }
+        }
+    }
+    Ok(())
+}
+
+unsafe fn send_value<W>(writer: &mut W, tag: Tag, data: &mut *const (), write_tags: bool) -> Result<(), Error>
+where W: Write + ?Sized {
+    macro_rules! consume_value {
+        ($ty:ty, | $ptr:ident | $map:expr) => {{
+            let $ptr = align_ptr::<$ty>(*data);
+            *data = $ptr.offset(1) as *const ();
+            $map
+        }};
+    }
+
+    if write_tags {
+        writer.write_u8(tag.as_u8())?;
+    }
+    match tag {
+        Tag::None => Ok(()),
+        Tag::Bool => consume_value!(u8, |ptr| writer.write_u8(*ptr)),
+        Tag::Int32 => consume_value!(u32, |ptr| writer.write_u32(*ptr)),
+        Tag::Int64 | Tag::Float64 => consume_value!(u64, |ptr| writer.write_u64(*ptr)),
+        Tag::String => consume_value!(CSlice<u8>, |ptr| {
+            writer.write_string(str::from_utf8((*ptr).as_ref()).unwrap())
+        }),
+        Tag::Bytes | Tag::ByteArray => consume_value!(CSlice<u8>, |ptr| writer.write_bytes((*ptr).as_ref())),
+        Tag::Tuple(it, arity) => {
+            let mut it = it.clone();
+            if write_tags {
+                writer.write_u8(arity)?;
+            }
+            let mut max_alignment = 0;
+            for _ in 0..arity {
+                let tag = it.next().expect("truncated tag");
+                max_alignment = core::cmp::max(max_alignment, tag.alignment());
+                send_value(writer, tag, data, write_tags)?
+            }
+            *data = round_up_const(*data, max_alignment);
+            Ok(())
+        }
+        Tag::List(it) => {
+            #[repr(C)]
+            struct List {
+                elements: *const (),
+                length: u32,
+            }
+            consume_value!(&List, |ptr| {
+                let length = (**ptr).length as usize;
+                writer.write_u32((*ptr).length)?;
+                let tag = it.clone().next().expect("truncated tag");
+                send_elements(writer, tag, length, (**ptr).elements, write_tags)
+            })
+        }
+        Tag::Array(it, num_dims) => {
+            if write_tags {
+                writer.write_u8(num_dims)?;
+            }
+            consume_value!(*const (), |buffer| {
+                let elt_tag = it.clone().next().expect("truncated tag");
+
+                let mut total_len = 1;
+                for _ in 0..num_dims {
+                    consume_value!(u32, |len| {
+                        writer.write_u32(*len)?;
+                        total_len *= *len;
+                    })
+                }
+                let length = total_len as usize;
+                send_elements(writer, elt_tag, length, *buffer, write_tags)
+            })
+        }
+        Tag::Range(it) => {
+            let tag = it.clone().next().expect("truncated tag");
+            send_value(writer, tag, data, write_tags)?;
+            send_value(writer, tag, data, write_tags)?;
+            send_value(writer, tag, data, write_tags)?;
+            Ok(())
+        }
+        Tag::Keyword(it) => {
+            #[repr(C)]
+            struct Keyword<'a> {
+                name: CSlice<'a, u8>,
+            }
+            consume_value!(Keyword, |ptr| {
+                writer.write_string(str::from_utf8((*ptr).name.as_ref()).unwrap())?;
+                let tag = it.clone().next().expect("truncated tag");
+                let mut data = ptr.offset(1) as *const ();
+                send_value(writer, tag, &mut data, write_tags)
+            })
+            // Tag::Keyword never appears in composite types, so we don't have
+            // to accurately advance data.
+        }
+        Tag::Object => {
+            #[repr(C)]
+            struct Object {
+                id: u32,
+            }
+            consume_value!(*const Object, |ptr| writer.write_u32((**ptr).id))
+        }
+    }
+}
+
+pub fn send_args<W>(
+    writer: &mut W,
+    service: u32,
+    tag_bytes: &[u8],
+    data: *const *const (),
+    write_tags: bool,
+) -> Result<(), Error>
+where
+    W: Write + ?Sized,
+{
+    let (arg_tags_bytes, return_tag_bytes) = split_tag(tag_bytes);
+
+    let mut args_it = TagIterator::new(arg_tags_bytes);
+    let return_it = TagIterator::new(return_tag_bytes);
+    trace!("send<{}>({})->{}", service, args_it, return_it);
+
+    writer.write_u32(service)?;
+    for index in 0.. {
+        if let Some(arg_tag) = args_it.next() {
+            let mut data = unsafe { *data.offset(index) };
+            unsafe { send_value(writer, arg_tag, &mut data, write_tags)? };
+        } else {
+            break;
+        }
+    }
+    writer.write_u8(0)?;
+    writer.write_bytes(return_tag_bytes)?;
+
+    Ok(())
+}
+
+pub mod tag {
+    use core::fmt;
+
+    pub fn split_tag(tag_bytes: &[u8]) -> (&[u8], &[u8]) {
+        let tag_separator = tag_bytes
+            .iter()
+            .position(|&b| b == b':')
+            .expect("tag without a return separator");
+        let (arg_tags_bytes, rest) = tag_bytes.split_at(tag_separator);
+        let return_tag_bytes = &rest[1..];
+        (arg_tags_bytes, return_tag_bytes)
+    }
+
+    #[derive(Debug, Clone, Copy)]
+    pub enum Tag<'a> {
+        None,
+        Bool,
+        Int32,
+        Int64,
+        Float64,
+        String,
+        Bytes,
+        ByteArray,
+        Tuple(TagIterator<'a>, u8),
+        List(TagIterator<'a>),
+        Array(TagIterator<'a>, u8),
+        Range(TagIterator<'a>),
+        Keyword(TagIterator<'a>),
+        Object,
+    }
+
+    impl<'a> Tag<'a> {
+        pub fn as_u8(self) -> u8 {
+            match self {
+                Tag::None => b'n',
+                Tag::Bool => b'b',
+                Tag::Int32 => b'i',
+                Tag::Int64 => b'I',
+                Tag::Float64 => b'f',
+                Tag::String => b's',
+                Tag::Bytes => b'B',
+                Tag::ByteArray => b'A',
+                Tag::Tuple(_, _) => b't',
+                Tag::List(_) => b'l',
+                Tag::Array(_, _) => b'a',
+                Tag::Range(_) => b'r',
+                Tag::Keyword(_) => b'k',
+                Tag::Object => b'O',
+            }
+        }
+
+        pub fn alignment(self) -> usize {
+            use cslice::CSlice;
+            match self {
+                Tag::None => 1,
+                Tag::Bool => core::mem::align_of::<u8>(),
+                Tag::Int32 => core::mem::align_of::<i32>(),
+                Tag::Int64 => core::mem::align_of::<i64>(),
+                Tag::Float64 => core::mem::align_of::<f64>(),
+                // struct type: align to largest element
+                Tag::Tuple(it, arity) => {
+                    let it = it.clone();
+                    it.take(arity.into()).map(|t| t.alignment()).max().unwrap()
+                }
+                Tag::Range(it) => {
+                    let it = it.clone();
+                    it.take(3).map(|t| t.alignment()).max().unwrap()
+                }
+                // the ptr/length(s) pair is basically CSlice
+                Tag::Bytes | Tag::String | Tag::ByteArray | Tag::List(_) | Tag::Array(_, _) => {
+                    core::mem::align_of::<CSlice<()>>()
+                }
+                Tag::Keyword(_) => unreachable!("Tag::Keyword should not appear in composite types"),
+                Tag::Object => core::mem::align_of::<u32>(),
+            }
+        }
+
+        pub fn size(self) -> usize {
+            match self {
+                Tag::None => 0,
+                Tag::Bool => 1,
+                Tag::Int32 => 4,
+                Tag::Int64 => 8,
+                Tag::Float64 => 8,
+                Tag::String => 8,
+                Tag::Bytes => 8,
+                Tag::ByteArray => 8,
+                Tag::Tuple(it, arity) => {
+                    let mut size = 0;
+                    let mut max_alignment = 0;
+                    let mut it = it.clone();
+                    for _ in 0..arity {
+                        let tag = it.next().expect("truncated tag");
+                        let alignment = tag.alignment();
+                        max_alignment = core::cmp::max(max_alignment, alignment);
+                        size = super::round_up(size, alignment);
+                        size += tag.size();
+                    }
+                    // Take into account any tail padding (if element(s) with largest
+                    // alignment are not at the end).
+                    size = super::round_up(size, max_alignment);
+                    size
+                }
+                Tag::List(_) => 4,
+                Tag::Array(_, num_dims) => 4 * (1 + num_dims as usize),
+                Tag::Range(it) => {
+                    let tag = it.clone().next().expect("truncated tag");
+                    tag.size() * 3
+                }
+                Tag::Keyword(_) => unreachable!(),
+                Tag::Object => unreachable!(),
+            }
+        }
+    }
+
+    #[derive(Debug, Clone, Copy)]
+    pub struct TagIterator<'a> {
+        pub data: &'a [u8],
+    }
+
+    impl<'a> TagIterator<'a> {
+        pub fn new(data: &'a [u8]) -> TagIterator<'a> {
+            TagIterator { data }
+        }
+
+        fn sub(&mut self, count: u8) -> TagIterator<'a> {
+            let data = self.data;
+            for _ in 0..count {
+                self.next().expect("truncated tag");
+            }
+            TagIterator {
+                data: &data[..(data.len() - self.data.len())],
+            }
+        }
+    }
+
+    impl<'a> core::iter::Iterator for TagIterator<'a> {
+        type Item = Tag<'a>;
+
+        fn next(&mut self) -> Option<Tag<'a>> {
+            if self.data.len() == 0 {
+                return None;
+            }
+
+            let tag_byte = self.data[0];
+            self.data = &self.data[1..];
+            Some(match tag_byte {
+                b'n' => Tag::None,
+                b'b' => Tag::Bool,
+                b'i' => Tag::Int32,
+                b'I' => Tag::Int64,
+                b'f' => Tag::Float64,
+                b's' => Tag::String,
+                b'B' => Tag::Bytes,
+                b'A' => Tag::ByteArray,
+                b't' => {
+                    let count = self.data[0];
+                    self.data = &self.data[1..];
+                    Tag::Tuple(self.sub(count), count)
+                }
+                b'l' => Tag::List(self.sub(1)),
+                b'a' => {
+                    let count = self.data[0];
+                    self.data = &self.data[1..];
+                    Tag::Array(self.sub(1), count)
+                }
+                b'r' => Tag::Range(self.sub(1)),
+                b'k' => Tag::Keyword(self.sub(1)),
+                b'O' => Tag::Object,
+                _ => unreachable!(),
+            })
+        }
+    }
+
+    impl<'a> fmt::Display for TagIterator<'a> {
+        fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+            let mut it = self.clone();
+            let mut first = true;
+            while let Some(tag) = it.next() {
+                if first {
+                    first = false
+                } else {
+                    write!(f, ", ")?
+                }
+
+                match tag {
+                    Tag::None => write!(f, "None")?,
+                    Tag::Bool => write!(f, "Bool")?,
+                    Tag::Int32 => write!(f, "Int32")?,
+                    Tag::Int64 => write!(f, "Int64")?,
+                    Tag::Float64 => write!(f, "Float64")?,
+                    Tag::String => write!(f, "String")?,
+                    Tag::Bytes => write!(f, "Bytes")?,
+                    Tag::ByteArray => write!(f, "ByteArray")?,
+                    Tag::Tuple(it, _) => {
+                        write!(f, "Tuple(")?;
+                        it.fmt(f)?;
+                        write!(f, ")")?;
+                    }
+                    Tag::List(it) => {
+                        write!(f, "List(")?;
+                        it.fmt(f)?;
+                        write!(f, ")")?;
+                    }
+                    Tag::Array(it, num_dims) => {
+                        write!(f, "Array(")?;
+                        it.fmt(f)?;
+                        write!(f, ", {})", num_dims)?;
+                    }
+                    Tag::Range(it) => {
+                        write!(f, "Range(")?;
+                        it.fmt(f)?;
+                        write!(f, ")")?;
+                    }
+                    Tag::Keyword(it) => {
+                        write!(f, "Keyword(")?;
+                        it.fmt(f)?;
+                        write!(f, ")")?;
+                    }
+                    Tag::Object => write!(f, "Object")?,
+                }
+            }
+
+            Ok(())
+        }
+    }
+}

src/libksupport/src/rtio_acp.rs

@@ -1,18 +1,19 @@
-use cslice::CSlice;
-use vcell::VolatileCell;
-use libcortex_a9::asm;
-use crate::artiq_raise;
 use core::sync::atomic::{fence, Ordering};
 
-use crate::pl::csr;
+use cslice::CSlice;
+use libcortex_a9::asm;
+use vcell::VolatileCell;
 
-pub const RTIO_O_STATUS_WAIT:                      i32 = 1;
-pub const RTIO_O_STATUS_UNDERFLOW:                 i32 = 2;
-pub const RTIO_O_STATUS_DESTINATION_UNREACHABLE:   i32 = 4;
-pub const RTIO_I_STATUS_WAIT_EVENT:                i32 = 1;
-pub const RTIO_I_STATUS_OVERFLOW:                  i32 = 2;
-pub const RTIO_I_STATUS_WAIT_STATUS:               i32 = 4; // TODO
-pub const RTIO_I_STATUS_DESTINATION_UNREACHABLE:   i32 = 8;
+use crate::{artiq_raise, pl::csr, resolve_channel_name, rtio_core};
+
+pub const RTIO_O_STATUS_WAIT: i32 = 1;
+pub const RTIO_O_STATUS_UNDERFLOW: i32 = 2;
+pub const RTIO_O_STATUS_DESTINATION_UNREACHABLE: i32 = 4;
+pub const RTIO_I_STATUS_WAIT_EVENT: i32 = 1;
+pub const RTIO_I_STATUS_OVERFLOW: i32 = 2;
+#[allow(unused)]
+pub const RTIO_I_STATUS_WAIT_STATUS: i32 = 4; // TODO
+pub const RTIO_I_STATUS_DESTINATION_UNREACHABLE: i32 = 8;
 
 #[repr(C)]
 pub struct TimestampedData {
@@ -46,23 +47,18 @@ static mut TRANSACTION_BUFFER: Transaction = Transaction {
     reply_timestamp: VolatileCell::new(0),
     padding0: [0; 2],
     padding1: [0; 2],
-    padding2: [0; 2]
+    padding2: [0; 2],
 };
 
-pub extern fn init() {
+pub extern "C" fn init() {
     unsafe {
-        csr::rtio_core::reset_write(1);
+        rtio_core::reset_write(1);
         csr::rtio::engine_addr_base_write(&TRANSACTION_BUFFER as *const Transaction as u32);
         csr::rtio::enable_write(1);
     }
 }
 
-pub extern fn get_destination_status(destination: i32) -> bool {
-    // TODO
-    destination == 0
-}
-
-pub extern fn get_counter() -> i64 {
+pub extern "C" fn get_counter() -> i64 {
     unsafe {
         csr::rtio::counter_update_write(1);
         csr::rtio::counter_read() as i64
@@ -71,15 +67,15 @@ pub extern fn get_counter() -> i64 {
 
 static mut NOW: i64 = 0;
 
-pub extern fn now_mu() -> i64 {
+pub extern "C" fn now_mu() -> i64 {
     unsafe { NOW }
 }
 
-pub extern fn at_mu(t: i64) {
+pub extern "C" fn at_mu(t: i64) {
     unsafe { NOW = t }
 }
 
-pub extern fn delay_mu(dt: i64) {
+pub extern "C" fn delay_mu(dt: i64) {
     unsafe { NOW += dt }
 }
 
@@ -91,18 +87,34 @@ unsafe fn process_exceptional_status(channel: i32, status: i32) {
         while csr::rtio::o_status_read() as i32 & RTIO_O_STATUS_WAIT != 0 {}
     }
     if status & RTIO_O_STATUS_UNDERFLOW != 0 {
-        artiq_raise!("RTIOUnderflow",
-            "RTIO underflow at {0} mu, channel {1}, slack {2} mu",
-            timestamp, channel as i64, timestamp - get_counter());
+        artiq_raise!(
+            "RTIOUnderflow",
+            format!(
+                "RTIO underflow at {{1}} mu, channel 0x{:04x}:{}, slack {{2}} mu",
+                channel,
+                resolve_channel_name(channel as u32)
+            ),
+            channel as i64,
+            timestamp,
+            timestamp - get_counter()
+        );
     }
     if status & RTIO_O_STATUS_DESTINATION_UNREACHABLE != 0 {
-        artiq_raise!("RTIODestinationUnreachable",
-            "RTIO destination unreachable, output, at {0} mu, channel {1}",
-            timestamp, channel as i64, 0);
+        artiq_raise!(
+            "RTIODestinationUnreachable",
+            format!(
+                "RTIO destination unreachable, output, at {{0}} mu, channel 0x{:04x}:{}",
+                channel,
+                resolve_channel_name(channel as u32)
+            ),
+            timestamp,
+            channel as i64,
+            0
+        );
     }
 }
 
-pub extern fn output(target: i32, data: i32) {
+pub extern "C" fn output(target: i32, data: i32) {
     unsafe {
         // Clear status so we can observe response
         TRANSACTION_BUFFER.reply_status.set(0);
@@ -130,7 +142,7 @@ pub extern fn output(target: i32, data: i32) {
     }
 }
 
-pub extern fn output_wide(target: i32, data: CSlice<i32>) {
+pub extern "C" fn output_wide(target: i32, data: CSlice<i32>) {
     unsafe {
         // Clear status so we can observe response
         TRANSACTION_BUFFER.reply_status.set(0);
@@ -147,7 +159,7 @@ pub extern fn output_wide(target: i32, data: CSlice<i32>) {
         loop {
             status = TRANSACTION_BUFFER.reply_status.get();
             if status != 0 {
-                break
+                break;
             }
         }
 
@@ -158,8 +170,8 @@ pub extern fn output_wide(target: i32, data: CSlice<i32>) {
     }
 }
 
-pub extern fn input_timestamp(timeout: i64, channel: i32) -> i64 {
-   unsafe {
+pub extern "C" fn input_timestamp(timeout: i64, channel: i32) -> i64 {
+    unsafe {
         // Clear status so we can observe response
         TRANSACTION_BUFFER.reply_status.set(0);
 
@@ -175,29 +187,45 @@ pub extern fn input_timestamp(timeout: i64, channel: i32) -> i64 {
         loop {
             status = TRANSACTION_BUFFER.reply_status.get();
             if status != 0 {
-                break
+                break;
             }
         }
 
         if status & RTIO_I_STATUS_OVERFLOW != 0 {
-            artiq_raise!("RTIOOverflow",
-                         "RTIO input overflow on channel {0}",
-                         channel as i64, 0, 0);
+            artiq_raise!(
+                "RTIOOverflow",
+                format!(
+                    "RTIO input overflow on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
         }
         if status & RTIO_I_STATUS_WAIT_EVENT != 0 {
-            return -1
+            return -1;
         }
         if status & RTIO_I_STATUS_DESTINATION_UNREACHABLE != 0 {
-            artiq_raise!("RTIODestinationUnreachable",
-                         "RTIO destination unreachable, input, on channel {0}",
-                         channel as i64, 0, 0);
+            artiq_raise!(
+                "RTIODestinationUnreachable",
+                format!(
+                    "RTIO destination unreachable, input, on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
         }
 
         TRANSACTION_BUFFER.reply_timestamp.get()
     }
 }
 
-pub extern fn input_data(channel: i32) -> i32 {
+pub extern "C" fn input_data(channel: i32) -> i32 {
     unsafe {
         TRANSACTION_BUFFER.reply_status.set(0);
 
@@ -213,26 +241,42 @@ pub extern fn input_data(channel: i32) -> i32 {
         loop {
             status = TRANSACTION_BUFFER.reply_status.get();
             if status != 0 {
-                break
+                break;
             }
         }
 
         if status & RTIO_I_STATUS_OVERFLOW != 0 {
-            artiq_raise!("RTIOOverflow",
-                         "RTIO input overflow on channel {0}",
-                         channel as i64, 0, 0);
+            artiq_raise!(
+                "RTIOOverflow",
+                format!(
+                    "RTIO input overflow on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
         }
         if status & RTIO_I_STATUS_DESTINATION_UNREACHABLE != 0 {
-            artiq_raise!("RTIODestinationUnreachable",
-                         "RTIO destination unreachable, input, on channel {0}",
-                         channel as i64, 0, 0);
+            artiq_raise!(
+                "RTIODestinationUnreachable",
+                format!(
+                    "RTIO destination unreachable, input, on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
         }
 
         TRANSACTION_BUFFER.reply_data.get()
     }
 }
 
-pub extern fn input_timestamped_data(timeout: i64, channel: i32) -> TimestampedData {
+pub extern "C" fn input_timestamped_data(timeout: i64, channel: i32) -> TimestampedData {
     unsafe {
         TRANSACTION_BUFFER.reply_status.set(0);
 
@@ -248,19 +292,35 @@ pub extern fn input_timestamped_data(timeout: i64, channel: i32) -> TimestampedD
         loop {
             status = TRANSACTION_BUFFER.reply_status.get();
             if status != 0 {
-                break
+                break;
             }
         }
 
         if status & RTIO_I_STATUS_OVERFLOW != 0 {
-            artiq_raise!("RTIOOverflow",
-                         "RTIO input overflow on channel {0}",
-                         channel as i64, 0, 0);
+            artiq_raise!(
+                "RTIOOverflow",
+                format!(
+                    "RTIO input overflow on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
         }
         if status & RTIO_I_STATUS_DESTINATION_UNREACHABLE != 0 {
-            artiq_raise!("RTIODestinationUnreachable",
-                         "RTIO destination unreachable, input, on channel {0}",
-                         channel as i64, 0, 0);
+            artiq_raise!(
+                "RTIODestinationUnreachable",
+                format!(
+                    "RTIO destination unreachable, input, on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
         }
 
         TimestampedData {

src/libksupport/src/rtio_csr.rs

@@ -0,0 +1,274 @@
+use core::ptr::{read_volatile, write_volatile};
+
+use cslice::CSlice;
+
+use crate::{artiq_raise, pl::csr, resolve_channel_name, rtio_core};
+
+pub const RTIO_O_STATUS_WAIT: u8 = 1;
+pub const RTIO_O_STATUS_UNDERFLOW: u8 = 2;
+pub const RTIO_O_STATUS_DESTINATION_UNREACHABLE: u8 = 4;
+pub const RTIO_I_STATUS_WAIT_EVENT: u8 = 1;
+pub const RTIO_I_STATUS_OVERFLOW: u8 = 2;
+pub const RTIO_I_STATUS_WAIT_STATUS: u8 = 4;
+pub const RTIO_I_STATUS_DESTINATION_UNREACHABLE: u8 = 8;
+
+#[repr(C)]
+pub struct TimestampedData {
+    timestamp: i64,
+    data: i32,
+}
+
+pub extern "C" fn init() {
+    unsafe {
+        rtio_core::reset_write(1);
+    }
+}
+
+pub extern "C" fn get_counter() -> i64 {
+    unsafe {
+        csr::rtio::counter_update_write(1);
+        csr::rtio::counter_read() as i64
+    }
+}
+
+pub extern "C" fn now_mu() -> i64 {
+    unsafe { csr::rtio::now_read() as i64 }
+}
+
+pub extern "C" fn at_mu(t: i64) {
+    unsafe {
+        csr::rtio::now_write(t as u64);
+    }
+}
+
+pub extern "C" fn delay_mu(dt: i64) {
+    unsafe {
+        csr::rtio::now_write(csr::rtio::now_read() + dt as u64);
+    }
+}
+
+// writing the LSB of o_data (offset=0) triggers the RTIO write
+#[inline(always)]
+pub unsafe fn rtio_o_data_write(offset: usize, data: u32) {
+    write_volatile(
+        csr::rtio::O_DATA_ADDR.offset((csr::rtio::O_DATA_SIZE - 1 - offset) as isize),
+        data,
+    );
+}
+
+#[inline(always)]
+pub unsafe fn rtio_i_data_read(offset: usize) -> u32 {
+    read_volatile(csr::rtio::I_DATA_ADDR.offset((csr::rtio::I_DATA_SIZE - 1 - offset) as isize))
+}
+
+#[inline(never)]
+unsafe fn process_exceptional_status(channel: i32, status: u8) {
+    let timestamp = csr::rtio::now_read() as i64;
+    if status & RTIO_O_STATUS_WAIT != 0 {
+        while csr::rtio::o_status_read() & RTIO_O_STATUS_WAIT != 0 {}
+    }
+    if status & RTIO_O_STATUS_UNDERFLOW != 0 {
+        artiq_raise!(
+            "RTIOUnderflow",
+            format!(
+                "RTIO underflow at {{1}} mu, channel 0x{:04x}:{}, slack {{2}} mu",
+                channel,
+                resolve_channel_name(channel as u32)
+            ),
+            channel as i64,
+            timestamp,
+            timestamp - get_counter()
+        );
+    }
+    if status & RTIO_O_STATUS_DESTINATION_UNREACHABLE != 0 {
+        artiq_raise!(
+            "RTIODestinationUnreachable",
+            format!(
+                "RTIO destination unreachable, output, at {{0}} mu, channel 0x{:04x}:{}",
+                channel,
+                resolve_channel_name(channel as u32)
+            ),
+            timestamp,
+            channel as i64,
+            0
+        );
+    }
+}
+
+pub extern "C" fn output(target: i32, data: i32) {
+    unsafe {
+        csr::rtio::target_write(target as u32);
+        // writing target clears o_data
+        rtio_o_data_write(0, data as _);
+        let status = csr::rtio::o_status_read();
+        if status != 0 {
+            process_exceptional_status(target >> 8, status);
+        }
+    }
+}
+
+pub extern "C" fn output_wide(target: i32, data: &CSlice<i32>) {
+    unsafe {
+        csr::rtio::target_write(target as u32);
+        // writing target clears o_data
+        for i in (0..data.len()).rev() {
+            rtio_o_data_write(i, data[i] as _)
+        }
+        let status = csr::rtio::o_status_read();
+        if status != 0 {
+            process_exceptional_status(target >> 8, status);
+        }
+    }
+}
+
+pub extern "C" fn input_timestamp(timeout: i64, channel: i32) -> i64 {
+    unsafe {
+        csr::rtio::target_write((channel as u32) << 8);
+        csr::rtio::i_timeout_write(timeout as u64);
+
+        let mut status = RTIO_I_STATUS_WAIT_STATUS;
+        while status & RTIO_I_STATUS_WAIT_STATUS != 0 {
+            status = csr::rtio::i_status_read();
+        }
+
+        if status & RTIO_I_STATUS_OVERFLOW != 0 {
+            artiq_raise!(
+                "RTIOOverflow",
+                format!(
+                    "RTIO input overflow on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
+        }
+        if status & RTIO_I_STATUS_WAIT_EVENT != 0 {
+            return -1;
+        }
+        if status & RTIO_I_STATUS_DESTINATION_UNREACHABLE != 0 {
+            artiq_raise!(
+                "RTIODestinationUnreachable",
+                format!(
+                    "RTIO destination unreachable, input, on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
+        }
+
+        csr::rtio::i_timestamp_read() as i64
+    }
+}
+
+pub extern "C" fn input_data(channel: i32) -> i32 {
+    unsafe {
+        csr::rtio::target_write((channel as u32) << 8);
+        csr::rtio::i_timeout_write(0xffffffff_ffffffff);
+
+        let mut status = RTIO_I_STATUS_WAIT_STATUS;
+        while status & RTIO_I_STATUS_WAIT_STATUS != 0 {
+            status = csr::rtio::i_status_read();
+        }
+
+        if status & RTIO_I_STATUS_OVERFLOW != 0 {
+            artiq_raise!(
+                "RTIOOverflow",
+                format!(
+                    "RTIO input overflow on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
+        }
+        if status & RTIO_I_STATUS_DESTINATION_UNREACHABLE != 0 {
+            artiq_raise!(
+                "RTIODestinationUnreachable",
+                format!(
+                    "RTIO destination unreachable, input, on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
+        }
+
+        rtio_i_data_read(0) as i32
+    }
+}
+
+pub extern "C" fn input_timestamped_data(timeout: i64, channel: i32) -> TimestampedData {
+    unsafe {
+        csr::rtio::target_write((channel as u32) << 8);
+        csr::rtio::i_timeout_write(timeout as u64);
+
+        let mut status = RTIO_I_STATUS_WAIT_STATUS;
+        while status & RTIO_I_STATUS_WAIT_STATUS != 0 {
+            status = csr::rtio::i_status_read();
+        }
+
+        if status & RTIO_I_STATUS_OVERFLOW != 0 {
+            artiq_raise!(
+                "RTIOOverflow",
+                format!(
+                    "RTIO input overflow on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
+        }
+        if status & RTIO_I_STATUS_WAIT_EVENT != 0 {
+            return TimestampedData { timestamp: -1, data: 0 };
+        }
+        if status & RTIO_I_STATUS_DESTINATION_UNREACHABLE != 0 {
+            artiq_raise!(
+                "RTIODestinationUnreachable",
+                format!(
+                    "RTIO destination unreachable, input, on channel 0x{:04x}:{}",
+                    channel,
+                    resolve_channel_name(channel as u32)
+                ),
+                channel as i64,
+                0,
+                0
+            );
+        }
+
+        TimestampedData {
+            timestamp: csr::rtio::i_timestamp_read() as i64,
+            data: rtio_i_data_read(0) as i32,
+        }
+    }
+}
+
+pub fn write_log(data: &[i8]) {
+    unsafe {
+        csr::rtio::target_write(csr::CONFIG_RTIO_LOG_CHANNEL << 8);
+
+        let mut word: u32 = 0;
+        for i in 0..data.len() {
+            word <<= 8;
+            word |= data[i] as u32;
+            if i % 4 == 3 {
+                rtio_o_data_write(0, word);
+                word = 0;
+            }
+        }
+
+        if word != 0 {
+            rtio_o_data_write(0, word);
+        }
+    }
+}

src/libunwind/backtrace.rs

@@ -1,27 +1,25 @@
-use libc::{c_void, c_int};
+use libc::{c_int, c_void};
+
 use crate::libunwind as uw;
 
 const UW_REG_SP: c_int = 13;
 
 pub fn backtrace<F>(f: F) -> Result<(), uw::_Unwind_Reason_Code>
-    where F: FnMut(usize) -> ()
-{
+where F: FnMut(usize) -> () {
     struct TraceContext<F> {
         step_fn: F,
-        prev_sp: uw::_Unwind_Word
+        prev_sp: uw::_Unwind_Word,
     }
 
-    extern fn trace<F>(context: *mut uw::_Unwind_Context, arg: *mut c_void)
-                      -> uw::_Unwind_Reason_Code
-        where F: FnMut(usize) -> ()
-    {
+    extern "C" fn trace<F>(context: *mut uw::_Unwind_Context, arg: *mut c_void) -> uw::_Unwind_Reason_Code
+    where F: FnMut(usize) -> () {
         unsafe {
             let trace_context = &mut *(arg as *mut TraceContext<F>);
 
             // Detect the root of a libfringe thread
             let cur_sp = uw::_Unwind_GetGR(context, UW_REG_SP);
             if cur_sp == trace_context.prev_sp {
-                return uw::_URC_END_OF_STACK
+                return uw::_URC_END_OF_STACK;
             } else {
                 trace_context.prev_sp = cur_sp;
             }
@@ -35,7 +33,7 @@ pub fn backtrace<F>(f: F) -> Result<(), uw::_Unwind_Reason_Code>
         let mut trace_context = TraceContext { step_fn: f, prev_sp: 0 };
         match uw::_Unwind_Backtrace(trace::<F>, &mut trace_context as *mut _ as *mut c_void) {
             uw::_URC_NO_REASON => Ok(()),
-            err => Err(err)
+            err => Err(err),
         }
     }
 }

src/libunwind/build.rs

@@ -5,7 +5,7 @@ fn main() {
 }
 
 mod llvm_libunwind {
-    use std::path::Path;
+    use std::{env, path::Path};
 
     fn setup_options(cfg: &mut cc::Build) {
         cfg.no_default_flags(true);
@@ -16,9 +16,13 @@ mod llvm_libunwind {
         cfg.flag("-fno-PIC");
         cfg.flag("-Isrc");
         cfg.flag("-isystem../include");
+        if let Ok(extra_include) = env::var("CLANG_EXTRA_INCLUDE_DIR") {
+            cfg.flag(&("-isystem".to_owned() + &extra_include));
+        }
         cfg.flag("-fno-stack-protector");
         cfg.flag("--target=armv7-none-eabihf");
         cfg.flag("-O2");
+        cfg.flag("-flto");
 
         cfg.flag("-std=c99");
         cfg.flag("-fstrict-aliasing");
@@ -77,11 +81,7 @@ mod llvm_libunwind {
         cfg.flag("-fvisibility=hidden");
         cfg.flag_if_supported("-fvisibility-global-new-delete-hidden");
 
-        let unwind_sources = vec![
-            "Unwind-EHABI.cpp",
-            "Unwind-seh.cpp",
-            "libunwind.cpp"
-        ];
+        let unwind_sources = vec!["Unwind-EHABI.cpp", "Unwind-seh.cpp", "libunwind.cpp"];
 
         let root = Path::new("../llvm_libunwind");
         cfg.include(root.join("include"));

src/libunwind/libunwind.rs

@@ -21,8 +21,7 @@ pub use _Unwind_Reason_Code::*;
 pub type _Unwind_Exception_Class = u64;
 pub type _Unwind_Word = uintptr_t;
 pub type _Unwind_Ptr = uintptr_t;
-pub type _Unwind_Trace_Fn =
-    extern "C" fn(ctx: *mut _Unwind_Context, arg: *mut c_void) -> _Unwind_Reason_Code;
+pub type _Unwind_Trace_Fn = extern "C" fn(ctx: *mut _Unwind_Context, arg: *mut c_void) -> _Unwind_Reason_Code;
 
 #[cfg(target_arch = "x86")]
 pub const unwinder_private_data_size: usize = 5;
@@ -279,7 +278,6 @@ if #[cfg(all(windows, target_arch = "x86_64", target_env = "gnu"))] {
 } // cfg_if!
 
 #[no_mangle]
-extern fn abort() {
+extern "C" fn abort() {
     panic!("Abort!");
 }
-

src/llvm_libunwind/include/unwind.h

@@ -107,9 +107,12 @@ struct _Unwind_Control_Block {
 } __attribute__((__aligned__(8)));
 
 typedef _Unwind_Reason_Code (*_Unwind_Stop_Fn)
-      (_Unwind_State state,
-       _Unwind_Exception* exceptionObject,
-       struct _Unwind_Context* context);
+    (int version,
+     _Unwind_Action actions,
+     uint64_t exceptionClass,
+     _Unwind_Exception* exceptionObject,
+     struct _Unwind_Context* context,
+     void* stop_parameter);
 
 typedef _Unwind_Reason_Code (*__personality_routine)
       (_Unwind_State state,

src/llvm_libunwind/src/Unwind-EHABI.cpp

@@ -95,9 +95,11 @@ _Unwind_Reason_Code ProcessDescriptors(
       case Descriptor::LU32:
         descriptor = getNextWord(descriptor, &length);
         descriptor = getNextWord(descriptor, &offset);
+        break;
       case Descriptor::LU16:
         descriptor = getNextNibble(descriptor, &length);
         descriptor = getNextNibble(descriptor, &offset);
+        break;
       default:
         assert(false);
         return _URC_FAILURE;
@@ -183,8 +185,14 @@ static _Unwind_Reason_Code unwindOneFrame(_Unwind_State state,
   if (result != _URC_CONTINUE_UNWIND)
     return result;
 
-  if (__unw_step(reinterpret_cast<unw_cursor_t *>(context)) != UNW_STEP_SUCCESS)
+  switch (__unw_step(reinterpret_cast<unw_cursor_t *>(context))) {
+  case UNW_STEP_SUCCESS:
+    return _URC_CONTINUE_UNWIND;
+  case UNW_STEP_END:
+    return _URC_END_OF_STACK;
+  default:
     return _URC_FAILURE;
+  }
   return _URC_CONTINUE_UNWIND;
 }
 
@@ -677,6 +685,128 @@ static _Unwind_Reason_Code unwind_phase2(unw_context_t *uc, unw_cursor_t *cursor
   return _URC_FATAL_PHASE2_ERROR;
 }
 
+static _Unwind_Reason_Code
+unwind_phase2_forced(unw_context_t *uc, unw_cursor_t *cursor,
+                     _Unwind_Exception *exception_object, _Unwind_Stop_Fn stop,
+                     void *stop_parameter) {
+  // See comment at the start of unwind_phase1 regarding VRS integrity.
+  __unw_init_local(cursor, uc);
+
+  _LIBUNWIND_TRACE_UNWINDING("unwind_phase2_forced(ex_ojb=%p)",
+                             static_cast<void *>(exception_object));
+
+  // Walk each frame until we reach where search phase said to stop.
+  bool end_of_stack = false;
+  // TODO: why can't libunwind handle end of stack properly?
+  // We should fix this kind of hack.
+  unw_word_t forced_phase2_prev_sp = 0x0;
+  while (!end_of_stack) {
+    // Get info about this frame.
+    unw_word_t sp;
+    unw_proc_info_t frameInfo;
+    __unw_get_reg(cursor, UNW_REG_SP, &sp);
+    if (sp == forced_phase2_prev_sp) {
+        break;
+    }
+    forced_phase2_prev_sp = sp;
+    if (__unw_get_proc_info(cursor, &frameInfo) != UNW_ESUCCESS) {
+      _LIBUNWIND_TRACE_UNWINDING(
+          "unwind_phase2_forced(ex_ojb=%p): __unw_get_proc_info "
+          "failed => _URC_FATAL_PHASE2_ERROR",
+          static_cast<void *>(exception_object));
+      return _URC_FATAL_PHASE2_ERROR;
+    }
+
+    // When tracing, print state information.
+    if (_LIBUNWIND_TRACING_UNWINDING) {
+      char functionBuf[512];
+      const char *functionName = functionBuf;
+      unw_word_t offset;
+      if ((__unw_get_proc_name(cursor, functionBuf, sizeof(functionBuf),
+                               &offset) != UNW_ESUCCESS) ||
+          (frameInfo.start_ip + offset > frameInfo.end_ip))
+        functionName = ".anonymous.";
+      _LIBUNWIND_TRACE_UNWINDING(
+          "unwind_phase2_forced(ex_ojb=%p): start_ip=0x%" PRIxPTR ", func=%s, sp=0x%" PRIxPTR ", "
+          "lsda=0x%" PRIxPTR ", personality=0x%" PRIxPTR "",
+          static_cast<void *>(exception_object), frameInfo.start_ip,
+          functionName, sp, frameInfo.lsda,
+          frameInfo.handler);
+    }
+
+    _Unwind_Action action =
+        (_Unwind_Action)(_UA_FORCE_UNWIND | _UA_CLEANUP_PHASE);
+    _Unwind_Reason_Code stopResult =
+        (*stop)(1, action, exception_object->exception_class, exception_object,
+                (_Unwind_Context *)(cursor), stop_parameter);
+
+    _LIBUNWIND_TRACE_UNWINDING(
+        "unwind_phase2_forced(ex_ojb=%p): stop function returned %d",
+        (void *)exception_object, stopResult);
+    if (stopResult != _URC_NO_REASON) {
+      _LIBUNWIND_TRACE_UNWINDING(
+          "unwind_phase2_forced(ex_ojb=%p): stopped by stop function",
+          (void *)exception_object);
+      return _URC_FATAL_PHASE2_ERROR;
+    }
+
+    // If there is a personality routine, tell it we are unwinding.
+    if (frameInfo.handler != 0) {
+      __personality_routine p =
+          (__personality_routine)(long)(frameInfo.handler);
+      struct _Unwind_Context *context = (struct _Unwind_Context *)(cursor);
+      // EHABI #7.2
+      exception_object->pr_cache.fnstart = frameInfo.start_ip;
+      exception_object->pr_cache.ehtp =
+          (_Unwind_EHT_Header *)frameInfo.unwind_info;
+      exception_object->pr_cache.additional = frameInfo.flags;
+      _Unwind_Reason_Code personalityResult =
+          (*p)(_US_FORCE_UNWIND | _US_UNWIND_FRAME_STARTING, exception_object,
+              context);
+      switch (personalityResult) {
+      case _URC_CONTINUE_UNWIND:
+        // Continue unwinding
+        _LIBUNWIND_TRACE_UNWINDING(
+            "unwind_phase2_forced(ex_ojb=%p): _URC_CONTINUE_UNWIND",
+            static_cast<void *>(exception_object));
+        break;
+      case _URC_INSTALL_CONTEXT:
+        _LIBUNWIND_TRACE_UNWINDING(
+            "unwind_phase2_forced(ex_ojb=%p): _URC_INSTALL_CONTEXT",
+            static_cast<void *>(exception_object));
+        {
+          // EHABI #7.4.1 says we need to preserve pc for when _Unwind_Resume
+          // is called back, to find this same frame.
+          unw_word_t pc;
+          __unw_get_reg(cursor, UNW_REG_IP, &pc);
+          exception_object->unwinder_cache.reserved2 = (uint32_t)pc;
+        }
+        // We may get control back if landing pad calls _Unwind_Resume().
+        __unw_resume(cursor);
+        break;
+      case _URC_END_OF_STACK:
+        end_of_stack = true;
+        break;
+      default:
+        // Personality routine returned an unknown result code.
+        _LIBUNWIND_DEBUG_LOG("personality function returned unknown result %d",
+                      personalityResult);
+        return _URC_FATAL_PHASE2_ERROR;
+      }
+    }
+  }
+  _LIBUNWIND_TRACE_UNWINDING("unwind_phase2_forced(ex_ojb=%p): calling stop "
+                             "function with _UA_END_OF_STACK",
+                             (void *)exception_object);
+  _Unwind_Action lastAction =
+      (_Unwind_Action)(_UA_FORCE_UNWIND | _UA_CLEANUP_PHASE | _UA_END_OF_STACK);
+  (*stop)(1, lastAction, exception_object->exception_class, exception_object,
+          (struct _Unwind_Context *)(cursor), stop_parameter);
+  return _URC_FATAL_PHASE2_ERROR;
+}
+
+
+
 /// Called by __cxa_throw.  Only returns if there is a fatal error.
 _LIBUNWIND_EXPORT _Unwind_Reason_Code
 _Unwind_RaiseException(_Unwind_Exception *exception_object) {
@@ -724,15 +854,36 @@ _Unwind_Resume(_Unwind_Exception *exception_object) {
   unw_cursor_t cursor;
   __unw_getcontext(&uc);
 
-  // _Unwind_RaiseException on EHABI will always set the reserved1 field to 0,
-  // which is in the same position as private_1 below.
-  // TODO(ajwong): Who wronte the above? Why is it true?
-  unwind_phase2(&uc, &cursor, exception_object, true);
+  if (exception_object->unwinder_cache.reserved1)
+    unwind_phase2_forced(
+        &uc, &cursor, exception_object,
+        (_Unwind_Stop_Fn)exception_object->unwinder_cache.reserved1,
+        (void *)exception_object->unwinder_cache.reserved3);
+  else
+    unwind_phase2(&uc, &cursor, exception_object, true);
 
   // Clients assume _Unwind_Resume() does not return, so all we can do is abort.
   _LIBUNWIND_ABORT("_Unwind_Resume() can't return");
 }
 
+_LIBUNWIND_EXPORT _Unwind_Reason_Code
+_Unwind_ForcedUnwind(_Unwind_Exception *exception_object, _Unwind_Stop_Fn stop,
+                     void *stop_parameter) {
+  _LIBUNWIND_TRACE_API("_Unwind_ForcedUnwind(ex_obj=%p, stop=%p)",
+                       (void *)exception_object, (void *)(uintptr_t)stop);
+  unw_context_t uc;
+  unw_cursor_t cursor;
+  __unw_getcontext(&uc);
+
+  // Mark that this is a forced unwind, so _Unwind_Resume() can do
+  // the right thing.
+  exception_object->unwinder_cache.reserved1 = (uintptr_t)stop;
+  exception_object->unwinder_cache.reserved3 = (uintptr_t)stop_parameter;
+
+  return unwind_phase2_forced(&uc, &cursor, exception_object, stop,
+                              stop_parameter);
+}
+
 /// Called by personality handler during phase 2 to get LSDA for current frame.
 _LIBUNWIND_EXPORT uintptr_t
 _Unwind_GetLanguageSpecificData(struct _Unwind_Context *context) {
@@ -1002,9 +1153,14 @@ extern "C" _LIBUNWIND_EXPORT _Unwind_Reason_Code
 __gnu_unwind_frame(_Unwind_Exception *exception_object,
                    struct _Unwind_Context *context) {
   unw_cursor_t *cursor = (unw_cursor_t *)context;
-  if (__unw_step(cursor) != UNW_STEP_SUCCESS)
+  switch (__unw_step(cursor)) {
+  case UNW_STEP_SUCCESS:
+    return _URC_OK;
+  case UNW_STEP_END:
+    return _URC_END_OF_STACK;
+  default:
     return _URC_FAILURE;
-  return _URC_OK;
+  }
 }
 
 #endif  // defined(_LIBUNWIND_ARM_EHABI)

src/runtime/Cargo.toml

@@ -1,38 +0,0 @@
-[package]
-name = "runtime"
-description = "ARTIQ runtime on Zynq"
-version = "0.1.0"
-authors = ["M-Labs"]
-edition = "2018"
-
-[features]
-target_zc706 = ["libboard_zynq/target_zc706", "libsupport_zynq/target_zc706"]
-default = ["target_zc706"]
-
-[dependencies]
-num-traits = { version = "0.2", default-features = false }
-num-derive = "0.3"
-cslice = "0.3"
-log = "0.4"
-nb = "0.1"
-embedded-hal = "0.2"
-core_io = { version = "0.1", features = ["collections"] }
-byteorder = { version = "1.3", default-features = false }
-void = { version = "1", default-features = false }
-futures = { version = "0.3", default-features = false, features = ["async-await"] }
-async-recursion = "0.3"
-log_buffer = { version = "1.2" }
-libm = { version = "0.2", features = ["unstable"] }
-vcell = "0.1"
-
-libboard_zynq = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git", features = ["ipv6"]}
-libsupport_zynq = { default-features = false, features = ["alloc_core"], git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
-libcortex_a9 = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
-libasync = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
-libregister = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git" }
-libconfig = { git = "https://git.m-labs.hk/M-Labs/zynq-rs.git", features = ["ipv6"] }
-
-dyld = { path = "../libdyld" }
-dwarf = { path = "../libdwarf" }
-unwind = { path = "../libunwind" }
-libc = { path = "../libc" }

src/runtime/Cargo.toml.tpl

@@ -0,0 +1,49 @@
+[package]
+name = "runtime"
+description = "ARTIQ runtime on Zynq"
+version = "0.1.0"
+authors = ["M-Labs"]
+edition = "2018"
+
+[features]
+target_zc706 = ["libboard_zynq/target_zc706", "libsupport_zynq/target_zc706", "libconfig/target_zc706", "libboard_artiq/target_zc706"]
+target_kasli_soc = ["libboard_zynq/target_kasli_soc", "libsupport_zynq/target_kasli_soc", "libconfig/target_kasli_soc", "libboard_artiq/target_kasli_soc"]
+target_ebaz4205 = ["libboard_zynq/target_ebaz4205", "libsupport_zynq/target_ebaz4205", "libconfig/target_ebaz4205", "libboard_artiq/target_ebaz4205"]
+default = ["target_zc706"]
+
+[build-dependencies]
+build_zynq = { path = "../libbuild_zynq" }
+
+[dependencies]
+num-traits = { version = "0.2", default-features = false }
+num-derive = "0.3"
+cslice = "0.3"
+log = "0.4"
+embedded-hal = "0.2"
+core_io = { version = "0.1", features = ["collections"] }
+crc = { version = "1.7", default-features = false }
+byteorder = { version = "1.3", default-features = false }
+void = { version = "1", default-features = false }
+futures = { version = "0.3", default-features = false, features = ["async-await"] }
+async-recursion = "0.3"
+log_buffer = { version = "1.2" }
+vcell = "0.1"
+
+libboard_zynq = { path = "@@ZYNQ_RS@@/libboard_zynq", features = ["ipv6"]}
+libsupport_zynq = { path = "@@ZYNQ_RS@@/libsupport_zynq", default-features = false, features = ["alloc_core"] }
+libcortex_a9 = { path = "@@ZYNQ_RS@@/libcortex_a9" }
+libasync = { path = "@@ZYNQ_RS@@/libasync" }
+libregister = { path = "@@ZYNQ_RS@@/libregister" }
+libconfig = { path = "@@ZYNQ_RS@@/libconfig", features = ["fat_lfn", "ipv6"] }
+
+dyld = { path = "../libdyld" }
+dwarf = { path = "../libdwarf" }
+unwind = { path = "../libunwind" }
+libc = { path = "../libc" }
+io = { path = "../libio", features = ["alloc"] }
+ksupport = { path = "../libksupport" }
+libboard_artiq = { path = "../libboard_artiq" }
+
+[dependencies.tar-no-std]
+git = "https://git.m-labs.hk/M-Labs/tar-no-std"
+rev = "2ab6dc5"

src/runtime/build.rs

@@ -1,28 +1,6 @@
-use std::env;
-use std::fs::File;
-use std::io::Write;
-use std::io::{BufRead, BufReader};
-use std::path::PathBuf;
+extern crate build_zynq;
 
 fn main() {
-    // Put the linker script somewhere the linker can find it
-    let out = &PathBuf::from(env::var_os("OUT_DIR").unwrap());
-    File::create(out.join("link.x"))
-        .unwrap()
-        .write_all(include_bytes!("link.x"))
-        .unwrap();
-    println!("cargo:rustc-link-search={}", out.display());
-
-    // Only re-run the build script when link.x is changed,
-    // instead of when any part of the source code changes.
-    println!("cargo:rerun-if-changed=link.x");
-
-    // Handle rustc-cfg file
-    let cfg_path = "../../build/rustc-cfg";
-    println!("cargo:rerun-if-changed={}", cfg_path);
-
-    let f = BufReader::new(File::open(cfg_path).unwrap());
-    for line in f.lines() {
-        println!("cargo:rustc-cfg={}", line.unwrap());
-    }
+    build_zynq::add_linker_script();
+    build_zynq::cfg();
 }

src/runtime/src/analyzer.rs

@@ -1,10 +1,15 @@
+use alloc::rc::Rc;
+#[cfg(has_drtio)]
+use alloc::vec::Vec;
+use core::cell::RefCell;
+
 use libasync::{smoltcp::TcpStream, task};
-use libboard_zynq::smoltcp::Error;
-use libcortex_a9::cache;
+use libboard_artiq::drtio_routing;
+use libboard_zynq::{smoltcp::Error, timer::GlobalTimer};
+use libcortex_a9::{cache, mutex::Mutex};
 use log::{debug, info, warn};
 
-use crate::proto_async::*;
-use crate::pl;
+use crate::{pl, proto_async::*};
 
 const BUFFER_SIZE: usize = 512 * 1024;
 
@@ -13,9 +18,7 @@ struct Buffer {
     data: [u8; BUFFER_SIZE],
 }
 
-static mut BUFFER: Buffer = Buffer {
-    data: [0; BUFFER_SIZE]
-};
+static mut BUFFER: Buffer = Buffer { data: [0; BUFFER_SIZE] };
 
 fn arm() {
     debug!("arming RTIO analyzer");
@@ -40,16 +43,61 @@ fn disarm() {
     debug!("RTIO analyzer disarmed");
 }
 
+#[cfg(has_drtio)]
+pub mod remote_analyzer {
+    use super::*;
+    use crate::rtio_mgt::drtio;
+
+    pub struct RemoteBuffer {
+        pub total_byte_count: u64,
+        pub sent_bytes: u32,
+        pub error: bool,
+        pub data: Vec<u8>,
+    }
+
+    pub async fn get_data(
+        aux_mutex: &Rc<Mutex<bool>>,
+        routing_table: &drtio_routing::RoutingTable,
+        up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
+        timer: GlobalTimer,
+    ) -> Result<RemoteBuffer, drtio::Error> {
+        // gets data from satellites and returns consolidated data
+        let mut remote_data: Vec<u8> = Vec::new();
+        let mut remote_error = false;
+        let mut remote_sent_bytes = 0;
+        let mut remote_total_bytes = 0;
+
+        let data_vec = match drtio::analyzer_query(aux_mutex, routing_table, up_destinations, timer).await {
+            Ok(data_vec) => data_vec,
+            Err(e) => return Err(e),
+        };
+        for data in data_vec {
+            remote_total_bytes += data.total_byte_count;
+            remote_sent_bytes += data.sent_bytes;
+            remote_error |= data.error;
+            remote_data.extend(data.data);
+        }
+
+        Ok(RemoteBuffer {
+            total_byte_count: remote_total_bytes,
+            sent_bytes: remote_sent_bytes,
+            error: remote_error,
+            data: remote_data,
+        })
+    }
+}
+
 #[derive(Debug)]
 struct Header {
     sent_bytes: u32,
     total_byte_count: u64,
     error_occurred: bool,
     log_channel: u8,
-    dds_onehot_sel: bool
+    dds_onehot_sel: bool,
 }
 
 async fn write_header(stream: &mut TcpStream, header: &Header) -> Result<(), Error> {
+    stream.send_slice("e".as_bytes()).await?;
     write_i32(stream, header.sent_bytes as i32).await?;
     write_i64(stream, header.total_byte_count as i64).await?;
     write_i8(stream, header.error_occurred as i8).await?;
@@ -58,7 +106,13 @@ async fn write_header(stream: &mut TcpStream, header: &Header) -> Result<(), Err
     Ok(())
 }
 
-async fn handle_connection(stream: &mut TcpStream) -> Result<(), Error> {
+async fn handle_connection(
+    stream: &mut TcpStream,
+    _aux_mutex: &Rc<Mutex<bool>>,
+    _routing_table: &drtio_routing::RoutingTable,
+    _up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
+    _timer: GlobalTimer,
+) -> Result<(), Error> {
     info!("received connection");
 
     let data = unsafe { &BUFFER.data[..] };
@@ -67,6 +121,11 @@ async fn handle_connection(stream: &mut TcpStream) -> Result<(), Error> {
     let total_byte_count = unsafe { pl::csr::rtio_analyzer::dma_byte_count_read() as u64 };
     let pointer = (total_byte_count % BUFFER_SIZE as u64) as usize;
     let wraparound = total_byte_count >= BUFFER_SIZE as u64;
+    let sent_bytes = if wraparound {
+        BUFFER_SIZE as u32
+    } else {
+        total_byte_count as u32
+    };
 
     if overflow_occurred {
         warn!("overflow occured");
@@ -75,12 +134,42 @@ async fn handle_connection(stream: &mut TcpStream) -> Result<(), Error> {
         warn!("bus error occured");
     }
 
+    #[cfg(has_drtio)]
+    let remote = remote_analyzer::get_data(_aux_mutex, _routing_table, _up_destinations, _timer).await;
+    #[cfg(has_drtio)]
+    let (header, remote_data) = match remote {
+        Ok(remote) => (
+            Header {
+                total_byte_count: total_byte_count + remote.total_byte_count,
+                sent_bytes: sent_bytes + remote.sent_bytes,
+                error_occurred: overflow_occurred | bus_error_occurred | remote.error,
+                log_channel: pl::csr::CONFIG_RTIO_LOG_CHANNEL as u8,
+                dds_onehot_sel: true,
+            },
+            remote.data,
+        ),
+        Err(e) => {
+            warn!("Error getting remote analyzer data: {}", e);
+            (
+                Header {
+                    total_byte_count: total_byte_count,
+                    sent_bytes: sent_bytes,
+                    error_occurred: true,
+                    log_channel: pl::csr::CONFIG_RTIO_LOG_CHANNEL as u8,
+                    dds_onehot_sel: true,
+                },
+                Vec::new(),
+            )
+        }
+    };
+
+    #[cfg(not(has_drtio))]
     let header = Header {
         total_byte_count: total_byte_count,
-        sent_bytes: if wraparound { BUFFER_SIZE as u32 } else { total_byte_count as u32 },
+        sent_bytes: sent_bytes,
         error_occurred: overflow_occurred | bus_error_occurred,
         log_channel: pl::csr::CONFIG_RTIO_LOG_CHANNEL as u8,
-        dds_onehot_sel: true  // kept for backward compatibility of analyzer dumps
+        dds_onehot_sel: true, // kept for backward compatibility of analyzer dumps
     };
     debug!("{:?}", header);
 
@@ -91,17 +180,28 @@ async fn handle_connection(stream: &mut TcpStream) -> Result<(), Error> {
     } else {
         stream.send(data[..pointer].iter().copied()).await?;
     }
+    #[cfg(has_drtio)]
+    stream.send(remote_data.iter().copied()).await?;
 
     Ok(())
 }
 
-pub fn start() {
+pub fn start(
+    aux_mutex: &Rc<Mutex<bool>>,
+    routing_table: &Rc<RefCell<drtio_routing::RoutingTable>>,
+    up_destinations: &Rc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
+    timer: GlobalTimer,
+) {
+    let aux_mutex = aux_mutex.clone();
+    let routing_table = routing_table.clone();
+    let up_destinations = up_destinations.clone();
     task::spawn(async move {
         loop {
             arm();
             let mut stream = TcpStream::accept(1382, 2048, 2048).await.unwrap();
             disarm();
-            let _ = handle_connection(&mut stream)
+            let routing_table = routing_table.borrow();
+            let _ = handle_connection(&mut stream, &aux_mutex, &routing_table, &up_destinations, timer)
                 .await
                 .map_err(|e| warn!("connection terminated: {:?}", e));
             let _ = stream.flush().await;

src/runtime/src/eh_artiq.rs

@@ -1,265 +0,0 @@
-// From Current artiq firmware ksupport implementation.
-// Modified to suit the case of artiq-zynq port, for ARM EHABI.
-// Portions of the code in this file are derived from code by:
-//
-// Copyright 2015 The Rust Project Developers. See the COPYRIGHT
-// file at http://rust-lang.org/COPYRIGHT.
-//
-// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
-// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
-// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
-// option. This file may not be copied, modified, or distributed
-// except according to those terms.
-#![allow(non_camel_case_types)]
-
-use core::mem;
-use cslice::CSlice;
-use unwind as uw;
-use libc::{c_int, uintptr_t};
-use log::trace;
-
-use dwarf::eh::{self, EHAction, EHContext};
-
-
-const EXCEPTION_CLASS: uw::_Unwind_Exception_Class = 0x4d_4c_42_53_41_52_54_51; /* 'MLBSARTQ' */
-
-#[cfg(target_arch = "arm")]
-const UNWIND_DATA_REG: (i32, i32) = (0, 1); // R0, R1
-
-#[repr(C)]
-#[derive(Clone, Copy)]
-pub struct Exception<'a> {
-    pub name:     CSlice<'a, u8>,
-    pub file:     CSlice<'a, u8>,
-    pub line:     u32,
-    pub column:   u32,
-    pub function: CSlice<'a, u8>,
-    pub message:  CSlice<'a, u8>,
-    pub param:    [i64; 3]
-}
-
-fn str_err(_: core::str::Utf8Error) -> core::fmt::Error {
-    core::fmt::Error
-}
-
-impl<'a> core::fmt::Debug for Exception<'a> {
-    fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
-        write!(f, "Exception {} from {} in {}:{}:{}, message: {}",
-            core::str::from_utf8(self.name.as_ref()).map_err(str_err)?,
-            core::str::from_utf8(self.function.as_ref()).map_err(str_err)?,
-            core::str::from_utf8(self.file.as_ref()).map_err(str_err)?,
-            self.line, self.column,
-            core::str::from_utf8(self.message.as_ref()).map_err(str_err)?)
-    }
-}
-
-const MAX_BACKTRACE_SIZE: usize = 128;
-
-#[repr(C)]
-struct ExceptionInfo {
-    uw_exception:   uw::_Unwind_Exception,
-    exception:      Option<Exception<'static>>,
-    handled:        bool,
-    backtrace:      [usize; MAX_BACKTRACE_SIZE],
-    backtrace_size: usize
-}
-
-unsafe fn find_eh_action(
-    context: *mut uw::_Unwind_Context,
-    foreign_exception: bool,
-) -> Result<EHAction, ()> {
-    let lsda = uw::_Unwind_GetLanguageSpecificData(context) as *const u8;
-    let mut ip_before_instr: c_int = 0;
-    let ip = uw::_Unwind_GetIPInfo(context, &mut ip_before_instr);
-    let eh_context = EHContext {
-        // The return address points 1 byte past the call instruction,
-        // which could be in the next IP range in LSDA range table.
-        ip: if ip_before_instr != 0 { ip } else { ip - 1 },
-        func_start: uw::_Unwind_GetRegionStart(context),
-        get_text_start: &|| uw::_Unwind_GetTextRelBase(context),
-        get_data_start: &|| uw::_Unwind_GetDataRelBase(context),
-    };
-    eh::find_eh_action(lsda, &eh_context, foreign_exception)
-}
-
-pub unsafe fn artiq_personality(state: uw::_Unwind_State,
-                                         exception_object: *mut uw::_Unwind_Exception,
-                                         context: *mut uw::_Unwind_Context)
-                                         -> uw::_Unwind_Reason_Code {
-    let state = state as c_int;
-    let action = state & uw::_US_ACTION_MASK as c_int;
-    let search_phase = if action == uw::_US_VIRTUAL_UNWIND_FRAME as c_int {
-        // Backtraces on ARM will call the personality routine with
-        // state == _US_VIRTUAL_UNWIND_FRAME | _US_FORCE_UNWIND. In those cases
-        // we want to continue unwinding the stack, otherwise all our backtraces
-        // would end at __rust_try
-        if state & uw::_US_FORCE_UNWIND as c_int != 0 {
-            return continue_unwind(exception_object, context);
-        }
-        true
-    } else if action == uw::_US_UNWIND_FRAME_STARTING as c_int {
-        false
-    } else if action == uw::_US_UNWIND_FRAME_RESUME as c_int {
-        return continue_unwind(exception_object, context);
-    } else {
-        return uw::_URC_FAILURE;
-    };
-
-    // The DWARF unwinder assumes that _Unwind_Context holds things like the function
-    // and LSDA pointers, however ARM EHABI places them into the exception object.
-    // To preserve signatures of functions like _Unwind_GetLanguageSpecificData(), which
-    // take only the context pointer, GCC personality routines stash a pointer to
-    // exception_object in the context, using location reserved for ARM's
-    // "scratch register" (r12).
-    uw::_Unwind_SetGR(context,
-                      uw::UNWIND_POINTER_REG,
-                      exception_object as uw::_Unwind_Ptr);
-    // ...A more principled approach would be to provide the full definition of ARM's
-    // _Unwind_Context in our libunwind bindings and fetch the required data from there
-    // directly, bypassing DWARF compatibility functions.
-
-    let exception_class = (*exception_object).exception_class;
-    let foreign_exception = exception_class != EXCEPTION_CLASS;
-    let eh_action = match find_eh_action(context, foreign_exception) {
-        Ok(action) => action,
-        Err(_) => return uw::_URC_FAILURE,
-    };
-    let exception_info = &mut *(exception_object as *mut ExceptionInfo);
-    let exception = &exception_info.exception.unwrap();
-    if search_phase {
-        match eh_action {
-            EHAction::None => return continue_unwind(exception_object, context),
-            // Actually, cleanup should not return handler found, this is to workaround
-            // the issue of terminating directly when no catch cause is found while
-            // having some cleanup routines defined by finally.
-            // The best way to handle this is to force unwind the stack in the raise
-            // function when end of stack is reached, and call terminate at the end of
-            // the unwind. Unfortunately, there is no forced unwind function defined
-            // for EHABI, and I have no idea how to implement that, so this is a hack.
-            EHAction::Cleanup(_) => return uw::_URC_HANDLER_FOUND,
-            EHAction::Catch(_) => {
-                // EHABI requires the personality routine to update the
-                // SP value in the barrier cache of the exception object.
-                (*exception_object).private[5] =
-                    uw::_Unwind_GetGR(context, uw::UNWIND_SP_REG);
-                return uw::_URC_HANDLER_FOUND;
-            }
-            EHAction::Terminate => return uw::_URC_FAILURE,
-        }
-    } else {
-        match eh_action {
-            EHAction::None => return continue_unwind(exception_object, context),
-            EHAction::Cleanup(lpad) |
-            EHAction::Catch(lpad) => {
-                uw::_Unwind_SetGR(context, UNWIND_DATA_REG.0,
-                                  exception_object as uintptr_t);
-                uw::_Unwind_SetGR(context, UNWIND_DATA_REG.1, exception as *const _ as uw::_Unwind_Word);
-                uw::_Unwind_SetIP(context, lpad);
-                return uw::_URC_INSTALL_CONTEXT;
-            }
-            EHAction::Terminate => return uw::_URC_FAILURE,
-        }
-    }
-
-    // On ARM EHABI the personality routine is responsible for actually
-    // unwinding a single stack frame before returning (ARM EHABI Sec. 6.1).
-    unsafe fn continue_unwind(exception_object: *mut uw::_Unwind_Exception,
-                              context: *mut uw::_Unwind_Context)
-                              -> uw::_Unwind_Reason_Code {
-        if __gnu_unwind_frame(exception_object, context) == uw::_URC_NO_REASON {
-            uw::_URC_CONTINUE_UNWIND
-        } else {
-            uw::_URC_FAILURE
-        }
-    }
-    // defined in libgcc
-    extern "C" {
-        fn __gnu_unwind_frame(exception_object: *mut uw::_Unwind_Exception,
-                              context: *mut uw::_Unwind_Context)
-                              -> uw::_Unwind_Reason_Code;
-    }
-}
-
-extern fn cleanup(_unwind_code: uw::_Unwind_Reason_Code,
-                  uw_exception: *mut uw::_Unwind_Exception) {
-    unsafe {
-        let exception_info = &mut *(uw_exception as *mut ExceptionInfo);
-
-        exception_info.exception = None;
-    }
-}
-
-static mut INFLIGHT: ExceptionInfo = ExceptionInfo {
-    uw_exception: uw::_Unwind_Exception {
-        exception_class:   EXCEPTION_CLASS,
-        exception_cleanup: cleanup,
-        private:           [0; uw::unwinder_private_data_size],
-    },
-    exception:      None,
-    handled:        true,
-    backtrace:      [0; MAX_BACKTRACE_SIZE],
-    backtrace_size: 0
-};
-
-pub unsafe extern fn raise(exception: *const Exception) -> ! {
-    trace!("Trying to raise exception");
-    // FIXME: unsound transmute
-    // This would cause stack memory corruption.
-    INFLIGHT.exception = Some(mem::transmute::<Exception, Exception<'static>>(*exception));
-    INFLIGHT.handled   = false;
-
-    let result = uw::_Unwind_RaiseException(&mut INFLIGHT.uw_exception);
-    assert!(result == uw::_URC_END_OF_STACK);
-
-    INFLIGHT.backtrace_size = 0;
-    // read backtrace
-    let _ = uw::backtrace(|ip| {
-        if INFLIGHT.backtrace_size < MAX_BACKTRACE_SIZE {
-            INFLIGHT.backtrace[INFLIGHT.backtrace_size] = ip;
-            INFLIGHT.backtrace_size += 1;
-        }
-    });
-    crate::kernel::core1::terminate(INFLIGHT.exception.as_ref().unwrap(), INFLIGHT.backtrace[..INFLIGHT.backtrace_size].as_mut());
-}
-
-pub unsafe extern fn reraise() -> ! {
-    use cslice::AsCSlice;
-
-    // Reraise is basically cxa_rethrow, which calls _Unwind_Resume_or_Rethrow,
-    // which for EHABI would always call _Unwind_RaiseException.
-    match INFLIGHT.exception {
-        Some(ref exception) => raise(exception),
-        None => raise(&Exception {
-            name:     "0:artiq.coredevice.exceptions.RuntimeError".as_c_slice(),
-            file:     file!().as_c_slice(),
-            line:     line!(),
-            column:   column!(),
-            // https://github.com/rust-lang/rfcs/pull/1719
-            function: "__artiq_reraise".as_c_slice(),
-            message:  "No active exception to reraise".as_c_slice(),
-            param:    [0, 0, 0]
-        })
-    }
-}
-
-#[macro_export]
-macro_rules! artiq_raise {
-    ($name:expr, $message:expr, $param0:expr, $param1:expr, $param2:expr) => ({
-        use cslice::AsCSlice;
-        let exn = $crate::eh_artiq::Exception {
-            name:     concat!("0:artiq.coredevice.exceptions.", $name).as_c_slice(),
-            file:     file!().as_c_slice(),
-            line:     line!(),
-            column:   column!(),
-            // https://github.com/rust-lang/rfcs/pull/1719
-            function: "(Rust function)".as_c_slice(),
-            message:  $message.as_c_slice(),
-            param:    [$param0, $param1, $param2]
-        };
-        #[allow(unused_unsafe)]
-        unsafe { $crate::eh_artiq::raise(&exn) }
-    });
-    ($name:expr, $message:expr) => ({
-        artiq_raise!($name, $message, 0, 0, 0)
-    });
-}

src/runtime/src/kernel/cache.rs

@@ -1,30 +0,0 @@
-use alloc::string::String;
-use cslice::{CSlice, AsCSlice};
-use core::mem::{transmute, forget};
-use super::{KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0, Message};
-
-pub extern fn get(key: CSlice<u8>) -> CSlice<'static, i32> {
-    let key = String::from_utf8(key.as_ref().to_vec()).unwrap();
-    unsafe {
-        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::CacheGetRequest(key));
-        let msg = KERNEL_CHANNEL_0TO1.as_mut().unwrap().recv();
-        if let Message::CacheGetReply(v) = msg {
-            let slice = transmute(v.as_c_slice());
-            // we intentionally leak the memory here,
-            // which does not matter as core1 would restart
-            forget(v);
-            slice
-        } else {
-            panic!("Expected CacheGetReply for CacheGetRequest");
-        }
-    }
-}
-
-pub extern fn put(key: CSlice<u8>, list: CSlice<i32>) {
-    let key = String::from_utf8(key.as_ref().to_vec()).unwrap();
-    let value = list.as_ref().to_vec();
-    unsafe {
-        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::CachePutRequest(key, value));
-    }
-}
-

src/runtime/src/kernel/dma.rs

@@ -1,212 +0,0 @@
-use crate::{
-    pl::csr,
-    artiq_raise,
-    rtio,
-};
-use alloc::{vec::Vec, string::String, boxed::Box};
-use cslice::CSlice;
-use super::{KERNEL_IMAGE, KERNEL_CHANNEL_0TO1, KERNEL_CHANNEL_1TO0, Message};
-use core::mem;
-
-use libcortex_a9::cache::dcci_slice;
-
-const ALIGNMENT: usize = 16 * 8;
-
-#[repr(C)]
-pub struct DmaTrace {
-    duration: i64,
-    address:  i32,
-}
-
-#[derive(Clone, Debug)]
-pub struct DmaRecorder {
-    pub name: String,
-    pub buffer: Vec<u8>,
-    pub duration: i64,
-}
-
-static mut RECORDER: Option<DmaRecorder> = None;
-
-pub unsafe fn init_dma_recorder() {
-    // as static would remain after restart, we have to reset it,
-    // without running its destructor.
-    mem::forget(mem::replace(&mut RECORDER, None));
-}
-
-pub extern fn dma_record_start(name: CSlice<u8>) {
-    let name = String::from_utf8(name.as_ref().to_vec()).unwrap();
-    unsafe {
-        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::DmaEraseRequest(name.clone()));
-    }
-    unsafe {
-        if RECORDER.is_some() {
-            artiq_raise!("DMAError", "DMA is already recording")
-        }
-
-        let library = KERNEL_IMAGE.as_ref().unwrap();
-        library.rebind(b"rtio_output",
-                       dma_record_output as *const ()).unwrap();
-        library.rebind(b"rtio_output_wide",
-                       dma_record_output_wide as *const ()).unwrap();
-
-        RECORDER = Some(DmaRecorder {
-            name,
-            buffer: Vec::new(),
-            duration: 0,
-        });
-    }
-}
-
-pub extern fn dma_record_stop(duration: i64) {
-    unsafe {
-        if RECORDER.is_none() {
-            artiq_raise!("DMAError", "DMA is not recording")
-        }
-
-        let library = KERNEL_IMAGE.as_ref().unwrap();
-        library.rebind(b"rtio_output",
-                       rtio::output as *const ()).unwrap();
-        library.rebind(b"rtio_output_wide",
-                       rtio::output_wide as *const ()).unwrap();
-
-        let mut recorder = RECORDER.take().unwrap();
-        recorder.duration = duration;
-        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(
-            Message::DmaPutRequest(recorder)
-        );
-    }
-}
-
-#[inline(always)]
-unsafe fn dma_record_output_prepare(timestamp: i64, target: i32,
-                                    words: usize) {
-    // See gateware/rtio/dma.py.
-    const HEADER_LENGTH: usize = /*length*/1 + /*channel*/3 + /*timestamp*/8 + /*address*/1;
-    let length = HEADER_LENGTH + /*data*/words * 4;
-
-    let buffer = &mut RECORDER.as_mut().unwrap().buffer;
-    buffer.reserve(length);
-    buffer.extend_from_slice(&[
-        (length    >>  0) as u8,
-        (target    >>  8) as u8,
-        (target    >>  16) as u8,
-        (target    >>  24) as u8,
-        (timestamp >>  0) as u8,
-        (timestamp >>  8) as u8,
-        (timestamp >> 16) as u8,
-        (timestamp >> 24) as u8,
-        (timestamp >> 32) as u8,
-        (timestamp >> 40) as u8,
-        (timestamp >> 48) as u8,
-        (timestamp >> 56) as u8,
-        (target    >>  0) as u8,
-    ]);
-}
-
-pub extern fn dma_record_output(target: i32, word: i32) {
-    unsafe {
-        let timestamp = rtio::now_mu();
-        dma_record_output_prepare(timestamp, target, 1);
-        RECORDER.as_mut().unwrap().buffer.extend_from_slice(&[
-            (word >>  0) as u8,
-            (word >>  8) as u8,
-            (word >> 16) as u8,
-            (word >> 24) as u8,
-        ]);
-    }
-}
-
-pub extern fn dma_record_output_wide(target: i32, words: CSlice<i32>) {
-    assert!(words.len() <= 16); // enforce the hardware limit
-
-    unsafe {
-        let timestamp = rtio::now_mu();
-        dma_record_output_prepare(timestamp, target, words.len());
-        let buffer = &mut RECORDER.as_mut().unwrap().buffer;
-        for word in words.as_ref().iter() {
-            buffer.extend_from_slice(&[
-                (word >>  0) as u8,
-                (word >>  8) as u8,
-                (word >> 16) as u8,
-                (word >> 24) as u8,
-            ]);
-        }
-    }
-}
-
-pub extern fn dma_erase(name: CSlice<u8>) {
-    let name = String::from_utf8(name.as_ref().to_vec()).unwrap();
-    unsafe {
-        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::DmaEraseRequest(name));
-    }
-}
-
-pub extern fn dma_retrieve(name: CSlice<u8>) -> DmaTrace {
-    let name = String::from_utf8(name.as_ref().to_vec()).unwrap();
-    unsafe {
-        KERNEL_CHANNEL_1TO0.as_mut().unwrap().send(Message::DmaGetRequest(name));
-    }
-    match unsafe {KERNEL_CHANNEL_0TO1.as_mut().unwrap()}.recv() {
-        Message::DmaGetReply(None) => (),
-        Message::DmaGetReply(Some((mut v, duration))) => {
-            v.reserve(ALIGNMENT - 1);
-            let original_length = v.len();
-            let padding = ALIGNMENT - v.as_ptr() as usize % ALIGNMENT;
-            let padding = if padding == ALIGNMENT { 0 } else { padding };
-            for _ in 0..padding {
-                v.push(0);
-            }
-            // trailing zero to indicate end of buffer
-            v.push(0);
-            v.copy_within(0..original_length, padding);
-            dcci_slice(&v);
-            let v = Box::new(v);
-            let address = Box::into_raw(v) as *mut Vec<u8> as i32;
-            return DmaTrace {
-                address,
-                duration,
-            };
-        },
-        _ => panic!("Expected DmaGetReply after DmaGetRequest!"),
-    }
-    // we have to defer raising error as we have to drop the message first...
-    artiq_raise!("DMAError", "DMA trace not found");
-}
-
-pub extern fn dma_playback(timestamp: i64, ptr: i32) {
-    unsafe {
-        let v = Box::from_raw(ptr as *mut Vec<u8>);
-        let padding = ALIGNMENT - v.as_ptr() as usize % ALIGNMENT;
-        let padding = if padding == ALIGNMENT { 0 } else { padding };
-        let ptr = v.as_ptr().add(padding) as i32;
-
-        csr::rtio_dma::base_address_write(ptr as u32);
-        csr::rtio_dma::time_offset_write(timestamp as u64);
-
-        csr::cri_con::selected_write(1);
-        csr::rtio_dma::enable_write(1);
-        while csr::rtio_dma::enable_read() != 0 {}
-        csr::cri_con::selected_write(0);
-
-        // leave the handle as we may try to do playback for another time.
-        mem::forget(v);
-
-        let error = csr::rtio_dma::error_read();
-        if error != 0 {
-            let timestamp = csr::rtio_dma::error_timestamp_read();
-            let channel = csr::rtio_dma::error_channel_read();
-            csr::rtio_dma::error_write(1);
-            if error & 1 != 0 {
-                artiq_raise!("RTIOUnderflow",
-                    "RTIO underflow at {0} mu, channel {1}",
-                    timestamp as i64, channel as i64, 0);
-            }
-            if error & 2 != 0 {
-                artiq_raise!("RTIODestinationUnreachable",
-                    "RTIO destination unreachable, output, at {0} mu, channel {1}",
-                    timestamp as i64, channel as i64, 0);
-            }
-        }
-    }
-}
-

src/runtime/src/kernel/mod.rs

@@ -1,59 +0,0 @@
-use core::ptr;
-use alloc::{vec::Vec, string::String};
-
-use libcortex_a9::{mutex::Mutex, sync_channel, semaphore::Semaphore};
-use crate::eh_artiq;
-
-mod control;
-pub use control::Control;
-pub mod core1;
-mod api;
-mod rpc;
-mod dma;
-pub use dma::DmaRecorder;
-mod cache;
-
-#[derive(Debug, Clone)]
-pub struct RPCException {
-    pub name: String,
-    pub message: String,
-    pub param: [i64; 3],
-    pub file: String,
-    pub line: i32,
-    pub column: i32,
-    pub function: String
-}
-
-#[derive(Debug, Clone)]
-pub enum Message {
-    LoadRequest(Vec<u8>),
-    LoadCompleted,
-    LoadFailed,
-    StartRequest,
-    KernelFinished,
-    KernelException(&'static eh_artiq::Exception<'static>, &'static [usize]),
-    RpcSend { is_async: bool, data: Vec<u8> },
-    RpcRecvRequest(*mut ()),
-    RpcRecvReply(Result<usize, RPCException>),
-
-    CacheGetRequest(String),
-    CacheGetReply(Vec<i32>),
-    CachePutRequest(String, Vec<i32>),
-
-    DmaPutRequest(DmaRecorder),
-    DmaEraseRequest(String),
-    DmaGetRequest(String),
-    DmaGetReply(Option<(Vec<u8>, i64)>),
-}
-
-static CHANNEL_0TO1: Mutex<Option<sync_channel::Sender<'static, Message>>> = Mutex::new(None);
-static CHANNEL_1TO0: Mutex<Option<sync_channel::Receiver<'static, Message>>> = Mutex::new(None);
-static CHANNEL_SEM: Semaphore = Semaphore::new(0, 1);
-
-static mut KERNEL_CHANNEL_0TO1: Option<sync_channel::Receiver<'static, Message>> = None;
-static mut KERNEL_CHANNEL_1TO0: Option<sync_channel::Sender<'static, Message>> = None;
-
-static mut KERNEL_IMAGE: *const core1::KernelImage = ptr::null();
-
-static INIT_LOCK: Mutex<()> = Mutex::new(());
-

src/runtime/src/main.rs

@@ -1,172 +1,94 @@
 #![no_std]
 #![no_main]
-#![recursion_limit="1024"]  // for futures_util::select!
+#![recursion_limit = "1024"] // for futures_util::select!
 #![feature(alloc_error_handler)]
-#![feature(panic_info_message)]
-#![feature(c_variadic)]
 #![feature(const_btree_new)]
-#![feature(ptr_offset_from)]
-#![feature(const_in_array_repeat_expressions)]
-#![feature(naked_functions)]
+#![feature(panic_info_message)]
 
+#[macro_use]
 extern crate alloc;
 
-use core::{cmp, str};
-use log::{info, warn, error};
+#[cfg(all(feature = "target_kasli_soc", has_virtual_leds))]
+use core::cell::RefCell;
 
-use libboard_zynq::{timer::GlobalTimer, mpcore, gic, slcr};
-use libasync::{task, block_async};
-use libsupport_zynq::ram;
-use nb;
-use void::Void;
-use embedded_hal::blocking::delay::DelayMs;
+use ksupport;
+use libasync::task;
+#[cfg(has_drtio_eem)]
+use libboard_artiq::drtio_eem;
+#[cfg(feature = "target_kasli_soc")]
+use libboard_artiq::io_expander;
+use libboard_artiq::{identifier_read, logger, pl};
+use libboard_zynq::{gic, mpcore, timer::GlobalTimer};
 use libconfig::Config;
-use libregister::RegisterW;
+use libcortex_a9::l2c::enable_l2_cache;
+use libsupport_zynq::{exception_vectors, ram};
+use log::{info, warn};
 
-mod proto_core_io;
-mod proto_async;
-mod comms;
-mod rpc;
-#[path = "../../../build/pl.rs"]
-mod pl;
-#[cfg(ki_impl = "csr")]
-#[path = "rtio_csr.rs"]
-mod rtio;
-#[cfg(ki_impl = "acp")]
-#[path = "rtio_acp.rs"]
-mod rtio;
-mod kernel;
-mod moninj;
-mod eh_artiq;
-mod panic;
-mod logger;
-mod mgmt;
 mod analyzer;
-mod irq;
-mod i2c;
+mod comms;
 
-fn init_gateware() {
-    // Set up PS->PL clocks
-    slcr::RegisterBlock::unlocked(|slcr| {
-        // As we are touching the mux, the clock may glitch, so reset the PL.
-        slcr.fpga_rst_ctrl.write(
-            slcr::FpgaRstCtrl::zeroed()
-                .fpga0_out_rst(true)
-                .fpga1_out_rst(true)
-                .fpga2_out_rst(true)
-                .fpga3_out_rst(true)
-        );
-        slcr.fpga0_clk_ctrl.write(
-            slcr::Fpga0ClkCtrl::zeroed()
-                .src_sel(slcr::PllSource::IoPll)
-                .divisor0(8)
-                .divisor1(1)
-        );
-        slcr.fpga_rst_ctrl.write(
-            slcr::FpgaRstCtrl::zeroed()
-        );
-    });
+mod mgmt;
+mod moninj;
+mod panic;
+mod proto_async;
+mod rpc_async;
+mod rtio_clocking;
+mod rtio_dma;
+mod rtio_mgt;
+#[cfg(has_drtio)]
+mod subkernel;
+
+// linker symbols
+extern "C" {
+    static __exceptions_start: u32;
 }
 
-fn identifier_read(buf: &mut [u8]) -> &str {
-    unsafe {
-        pl::csr::identifier::address_write(0);
-        let len = pl::csr::identifier::data_read();
-        let len = cmp::min(len, buf.len() as u8);
-        for i in 0..len {
-            pl::csr::identifier::address_write(1 + i);
-            buf[i as usize] = pl::csr::identifier::data_read();
-        }
-        str::from_utf8_unchecked(&buf[..len as usize])
-    }
-}
-
-fn init_rtio(timer: &mut GlobalTimer, cfg: &Config) {
-    let clock_sel =
-        if let Ok(rtioclk) = cfg.read_str("rtioclk") {
-            match rtioclk.as_ref() {
-                "internal" => {
-                    info!("using internal RTIO clock");
-                    0
-                },
-                "external" => {
-                    info!("using external RTIO clock");
-                    1
-                },
-                other => {
-                    warn!("RTIO clock specification '{}' not recognized", other);
-                    info!("using internal RTIO clock");
-                    0
-                },
-            }
-        } else {
-            info!("using internal RTIO clock (default)");
-            0
-        };
-
+#[cfg(all(feature = "target_kasli_soc", has_virtual_leds))]
+async fn io_expanders_service(
+    i2c_bus: RefCell<&mut libboard_zynq::i2c::I2c>,
+    io_expander0: RefCell<io_expander::IoExpander>,
+    io_expander1: RefCell<io_expander::IoExpander>,
+) {
     loop {
-        unsafe {
-            pl::csr::rtio_crg::pll_reset_write(1);
-            pl::csr::rtio_crg::clock_sel_write(clock_sel);
-            pl::csr::rtio_crg::pll_reset_write(0);
-        }
-        timer.delay_ms(1);
-        let locked = unsafe { pl::csr::rtio_crg::pll_locked_read() != 0 };
-        if locked {
-            info!("RTIO PLL locked");
-            break;
-        } else {
-            warn!("RTIO PLL failed to lock, retrying...");
-            timer.delay_ms(500);
-        }
-    }
-
-    unsafe {
-        pl::csr::rtio_core::reset_phy_write(1);
+        task::r#yield().await;
+        io_expander0
+            .borrow_mut()
+            .service(&mut i2c_bus.borrow_mut())
+            .expect("I2C I/O expander #0 service failed");
+        io_expander1
+            .borrow_mut()
+            .service(&mut i2c_bus.borrow_mut())
+            .expect("I2C I/O expander #1 service failed");
     }
 }
 
-fn wait_for_async_rtio_error() -> nb::Result<(), Void> {
-    unsafe {
-        if pl::csr::rtio_core::async_error_read() != 0 {
-            Ok(())
-        } else {
-            Err(nb::Error::WouldBlock)
+#[cfg(has_grabber)]
+mod grabber {
+    use libasync::delay;
+    use libboard_artiq::grabber;
+    use libboard_zynq::time::Milliseconds;
+
+    use crate::GlobalTimer;
+    pub async fn grabber_thread(timer: GlobalTimer) {
+        let mut countdown = timer.countdown();
+        loop {
+            grabber::tick();
+            delay(&mut countdown, Milliseconds(200)).await;
         }
     }
 }
 
-async fn report_async_rtio_errors() {
-    loop {
-        let _ = block_async!(wait_for_async_rtio_error()).await;
-        unsafe {
-            let errors = pl::csr::rtio_core::async_error_read();
-            if errors & 1 != 0 {
-                error!("RTIO collision involving channel {}",
-                       pl::csr::rtio_core::collision_channel_read());
-            }
-            if errors & 2 != 0 {
-                error!("RTIO busy error involving channel {}",
-                       pl::csr::rtio_core::busy_channel_read());
-            }
-            if errors & 4 != 0 {
-                error!("RTIO sequence error involving channel {}",
-                       pl::csr::rtio_core::sequence_error_channel_read());
-            }
-            pl::csr::rtio_core::async_error_write(errors);
-        }
-    }
-}
-
-static mut LOG_BUFFER: [u8; 1<<17] = [0; 1<<17];
+static mut LOG_BUFFER: [u8; 1 << 17] = [0; 1 << 17];
 
 #[no_mangle]
 pub fn main_core0() {
+    unsafe {
+        exception_vectors::set_vector_table(&__exceptions_start as *const u32 as u32);
+    }
+    enable_l2_cache(0x8);
     let mut timer = GlobalTimer::start();
 
-    let buffer_logger = unsafe {
-        logger::BufferLogger::new(&mut LOG_BUFFER[..])
-    };
+    let buffer_logger = unsafe { logger::BufferLogger::new(&mut LOG_BUFFER[..]) };
     buffer_logger.set_uart_log_level(log::LevelFilter::Info);
     buffer_logger.register();
     log::set_max_level(log::LevelFilter::Info);
@@ -176,10 +98,39 @@ pub fn main_core0() {
     ram::init_alloc_core0();
     gic::InterruptController::gic(mpcore::RegisterBlock::mpcore()).enable_interrupts();
 
-    init_gateware();
-    info!("detected gateware: {}", identifier_read(&mut [0; 64]));
+    info!("gateware ident: {}", identifier_read(&mut [0; 64]));
 
-    i2c::init();
+    ksupport::i2c::init();
+    #[cfg(feature = "target_kasli_soc")]
+    {
+        let i2c_bus = unsafe { (ksupport::i2c::I2C_BUS).as_mut().unwrap() };
+        let mut io_expander0 = io_expander::IoExpander::new(i2c_bus, 0).unwrap();
+        let mut io_expander1 = io_expander::IoExpander::new(i2c_bus, 1).unwrap();
+        io_expander0
+            .init(i2c_bus)
+            .expect("I2C I/O expander #0 initialization failed");
+        io_expander1
+            .init(i2c_bus)
+            .expect("I2C I/O expander #1 initialization failed");
+
+        // Drive CLK_SEL to true
+        #[cfg(has_si549)]
+        io_expander0.set(1, 7, true);
+
+        // Drive TX_DISABLE to false on SFP0..3
+        io_expander0.set(0, 1, false);
+        io_expander1.set(0, 1, false);
+        io_expander0.set(1, 1, false);
+        io_expander1.set(1, 1, false);
+        io_expander0.service(i2c_bus).unwrap();
+        io_expander1.service(i2c_bus).unwrap();
+        #[cfg(has_virtual_leds)]
+        task::spawn(io_expanders_service(
+            RefCell::new(i2c_bus),
+            RefCell::new(io_expander0),
+            RefCell::new(io_expander1),
+        ));
+    }
 
     let cfg = match Config::new() {
         Ok(cfg) => cfg,
@@ -189,8 +140,15 @@ pub fn main_core0() {
         }
     };
 
-    init_rtio(&mut timer, &cfg);
-    task::spawn(report_async_rtio_errors());
+    rtio_clocking::init(&mut timer, &cfg);
 
-    comms::main(timer, &cfg);
+    #[cfg(has_drtio_eem)]
+    drtio_eem::init(&mut timer, &cfg);
+
+    #[cfg(has_grabber)]
+    task::spawn(grabber::grabber_thread(timer));
+
+    task::spawn(ksupport::report_async_rtio_errors());
+
+    comms::main(timer, cfg);
 }

src/runtime/src/moninj.rs

@@ -1,25 +1,23 @@
-use core::fmt;
-use alloc::collections::BTreeMap;
+use alloc::{collections::BTreeMap, rc::Rc};
+use core::{cell::RefCell, fmt};
+
+use futures::{pin_mut, select_biased, FutureExt};
+use libasync::{block_async, nb, smoltcp::TcpStream, task};
+use libboard_artiq::drtio_routing;
+use libboard_zynq::{smoltcp, time::Milliseconds, timer::GlobalTimer};
+use libcortex_a9::mutex::Mutex;
 use log::{debug, info, warn};
-use void::Void;
-
-use libboard_zynq::{smoltcp, timer::GlobalTimer, time::Milliseconds};
-use libasync::{task, smoltcp::TcpStream, block_async, nb};
-
 use num_derive::{FromPrimitive, ToPrimitive};
 use num_traits::{FromPrimitive, ToPrimitive};
-use futures::{pin_mut, select_biased, FutureExt};
+use void::Void;
 
 use crate::proto_async::*;
-use crate::pl::csr;
-
 
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum Error {
     NetworkError(smoltcp::Error),
     UnexpectedPattern,
     UnrecognizedPacket,
-
 }
 
 pub type Result<T> = core::result::Result<T, Error>;
@@ -28,8 +26,8 @@ impl fmt::Display for Error {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         match self {
             &Error::NetworkError(error) => write!(f, "network error: {}", error),
-            &Error::UnexpectedPattern   => write!(f, "unexpected pattern"),
-            &Error::UnrecognizedPacket  => write!(f, "unrecognized packet"),
+            &Error::UnexpectedPattern => write!(f, "unexpected pattern"),
+            &Error::UnrecognizedPacket => write!(f, "unrecognized packet"),
         }
     }
 }
@@ -45,61 +43,191 @@ enum HostMessage {
     MonitorProbe = 0,
     MonitorInjection = 3,
     Inject = 1,
-    GetInjectionStatus = 2
+    GetInjectionStatus = 2,
 }
 
 #[derive(Debug, FromPrimitive, ToPrimitive)]
 enum DeviceMessage {
     MonitorStatus = 0,
-    InjectionStatus = 1
+    InjectionStatus = 1,
 }
 
-fn read_probe(channel: i32, probe: i8) -> i32 {
-    unsafe {
-        csr::rtio_moninj::mon_chan_sel_write(channel as _);
-        csr::rtio_moninj::mon_probe_sel_write(probe as _);
-        csr::rtio_moninj::mon_value_update_write(1);
-        csr::rtio_moninj::mon_value_read() as i32
+#[cfg(has_drtio)]
+mod remote_moninj {
+    use libboard_artiq::drtioaux_async;
+    use log::error;
+
+    use super::*;
+    use crate::rtio_mgt::{drtio, drtio::Error as DrtioError};
+
+    pub async fn read_probe(
+        aux_mutex: &Rc<Mutex<bool>>,
+        routing_table: &drtio_routing::RoutingTable,
+        timer: GlobalTimer,
+        linkno: u8,
+        destination: u8,
+        channel: i32,
+        probe: i8,
+    ) -> i64 {
+        let reply = drtio::aux_transact(
+            aux_mutex,
+            linkno,
+            routing_table,
+            &drtioaux_async::Packet::MonitorRequest {
+                destination: destination,
+                channel: channel as _,
+                probe: probe as _,
+            },
+            timer,
+        )
+        .await;
+        match reply {
+            Ok(drtioaux_async::Packet::MonitorReply { value }) => return value as i64,
+            Ok(packet) => error!("received unexpected aux packet: {:?}", packet),
+            Err(DrtioError::LinkDown) => {
+                warn!("link is down");
+            }
+            Err(e) => error!("aux packet error ({})", e),
+        }
+        0
+    }
+
+    pub async fn inject(
+        aux_mutex: &Rc<Mutex<bool>>,
+        _routing_table: &drtio_routing::RoutingTable,
+        _timer: GlobalTimer,
+        linkno: u8,
+        destination: u8,
+        channel: i32,
+        overrd: i8,
+        value: i8,
+    ) {
+        let _lock = aux_mutex.async_lock().await;
+        drtioaux_async::send(
+            linkno,
+            &drtioaux_async::Packet::InjectionRequest {
+                destination: destination,
+                channel: channel as _,
+                overrd: overrd as _,
+                value: value as _,
+            },
+        )
+        .await
+        .unwrap();
+    }
+
+    pub async fn read_injection_status(
+        aux_mutex: &Rc<Mutex<bool>>,
+        routing_table: &drtio_routing::RoutingTable,
+        timer: GlobalTimer,
+        linkno: u8,
+        destination: u8,
+        channel: i32,
+        overrd: i8,
+    ) -> i8 {
+        let reply = drtio::aux_transact(
+            aux_mutex,
+            linkno,
+            routing_table,
+            &drtioaux_async::Packet::InjectionStatusRequest {
+                destination: destination,
+                channel: channel as _,
+                overrd: overrd as _,
+            },
+            timer,
+        )
+        .await;
+        match reply {
+            Ok(drtioaux_async::Packet::InjectionStatusReply { value }) => return value as i8,
+            Ok(packet) => error!("received unexpected aux packet: {:?}", packet),
+            Err(DrtioError::LinkDown) => {
+                warn!("link is down");
+            }
+            Err(e) => error!("aux packet error ({})", e),
+        }
+        0
     }
 }
 
-fn inject(channel: i32, overrd: i8, value: i8) {
-    unsafe {
-        csr::rtio_moninj::inj_chan_sel_write(channel as _);
-        csr::rtio_moninj::inj_override_sel_write(overrd as _);
-        csr::rtio_moninj::inj_value_write(value as _);
+mod local_moninj {
+    use libboard_artiq::pl::csr;
+
+    pub fn read_probe(channel: i32, probe: i8) -> i64 {
+        unsafe {
+            csr::rtio_moninj::mon_chan_sel_write(channel as _);
+            csr::rtio_moninj::mon_probe_sel_write(probe as _);
+            csr::rtio_moninj::mon_value_update_write(1);
+            csr::rtio_moninj::mon_value_read() as i64
+        }
+    }
+
+    pub fn inject(channel: i32, overrd: i8, value: i8) {
+        unsafe {
+            csr::rtio_moninj::inj_chan_sel_write(channel as _);
+            csr::rtio_moninj::inj_override_sel_write(overrd as _);
+            csr::rtio_moninj::inj_value_write(value as _);
+        }
+    }
+
+    pub fn read_injection_status(channel: i32, overrd: i8) -> i8 {
+        unsafe {
+            csr::rtio_moninj::inj_chan_sel_write(channel as _);
+            csr::rtio_moninj::inj_override_sel_write(overrd as _);
+            csr::rtio_moninj::inj_value_read() as i8
+        }
     }
 }
 
-fn read_injection_status(channel: i32, overrd: i8) -> i8 {
-    unsafe {
-        csr::rtio_moninj::inj_chan_sel_write(channel as _);
-        csr::rtio_moninj::inj_override_sel_write(overrd as _);
-        csr::rtio_moninj::inj_value_read() as i8
-    }
+#[cfg(has_drtio)]
+macro_rules! dispatch {
+    ($timer:ident, $aux_mutex:ident, $routing_table:ident, $channel:expr, $func:ident $(, $param:expr)*) => {{
+        let destination = ($channel >> 16) as u8;
+        let channel = $channel;
+        let hop = $routing_table.0[destination as usize][0];
+        if hop == 0 {
+            local_moninj::$func(channel.into(), $($param, )*)
+        } else {
+            let linkno = hop - 1 as u8;
+            remote_moninj::$func($aux_mutex, $routing_table, $timer, linkno, destination, channel, $($param, )*).await
+        }
+    }}
 }
 
-async fn handle_connection(stream: &TcpStream, timer: GlobalTimer) -> Result<()> {
+#[cfg(not(has_drtio))]
+macro_rules! dispatch {
+    ($timer:ident, $aux_mutex:ident, $routing_table:ident, $channel:expr, $func:ident $(, $param:expr)*) => {{
+        let channel = $channel as u16;
+        local_moninj::$func(channel.into(), $($param, )*)
+    }}
+}
+
+async fn handle_connection(
+    stream: &TcpStream,
+    timer: GlobalTimer,
+    _aux_mutex: &Rc<Mutex<bool>>,
+    _routing_table: &drtio_routing::RoutingTable,
+) -> Result<()> {
     if !expect(&stream, b"ARTIQ moninj\n").await? {
         return Err(Error::UnexpectedPattern);
     }
 
-    let mut probe_watch_list: BTreeMap<(i32, i8), Option<i32>> = BTreeMap::new();
+    let mut probe_watch_list: BTreeMap<(i32, i8), Option<i64>> = BTreeMap::new();
     let mut inject_watch_list: BTreeMap<(i32, i8), Option<i8>> = BTreeMap::new();
-    let mut next_check = Milliseconds(0);
+    let mut next_check = timer.get_time();
+    let timeout = |next_check: Milliseconds| -> nb::Result<(), Void> {
+        if timer.get_time() < next_check {
+            Err(nb::Error::WouldBlock)
+        } else {
+            Ok(())
+        }
+    };
     loop {
         // TODO: we don't need fuse() here.
         // remove after https://github.com/rust-lang/futures-rs/issues/1989 lands
         let read_message_f = read_i8(&stream).fuse();
         let next_check_c = next_check.clone();
-        let timeout = || -> nb::Result<(), Void> {
-            if timer.get_time() < next_check_c {
-                Err(nb::Error::WouldBlock)
-            } else {
-                Ok(())
-            }
-        };
-        let timeout_f = block_async!(timeout()).fuse();
+
+        let timeout_f = block_async!(timeout(next_check_c)).fuse();
         pin_mut!(read_message_f, timeout_f);
         select_biased! {
             message = read_message_f => {
@@ -134,13 +262,13 @@ async fn handle_connection(stream: &TcpStream, timer: GlobalTimer) -> Result<()>
                         let channel = read_i32(&stream).await?;
                         let overrd = read_i8(&stream).await?;
                         let value = read_i8(&stream).await?;
-                        inject(channel, overrd, value);
+                        dispatch!(timer, _aux_mutex, _routing_table, channel, inject, overrd, value);
                         debug!("INJECT channel {}, overrd {}, value {}", channel, overrd, value);
                     },
                     HostMessage::GetInjectionStatus => {
                         let channel = read_i32(&stream).await?;
                         let overrd = read_i8(&stream).await?;
-                        let value = read_injection_status(channel, overrd);
+                        let value = dispatch!(timer, _aux_mutex, _routing_table, channel, read_injection_status, overrd);
                         write_i8(&stream, DeviceMessage::InjectionStatus.to_i8().unwrap()).await?;
                         write_i32(&stream, channel).await?;
                         write_i8(&stream, overrd).await?;
@@ -150,17 +278,17 @@ async fn handle_connection(stream: &TcpStream, timer: GlobalTimer) -> Result<()>
             },
             _ = timeout_f => {
                 for (&(channel, probe), previous) in probe_watch_list.iter_mut() {
-                    let current = read_probe(channel, probe);
+                    let current = dispatch!(timer, _aux_mutex, _routing_table, channel, read_probe, probe);
                     if previous.is_none() || previous.unwrap() != current {
                         write_i8(&stream, DeviceMessage::MonitorStatus.to_i8().unwrap()).await?;
                         write_i32(&stream, channel).await?;
                         write_i8(&stream, probe).await?;
-                        write_i32(&stream, current).await?;
+                        write_i64(&stream, current).await?;
                         *previous = Some(current);
                     }
                 }
                 for (&(channel, overrd), previous) in inject_watch_list.iter_mut() {
-                    let current = read_injection_status(channel, overrd);
+                    let current = dispatch!(timer, _aux_mutex, _routing_table, channel, read_injection_status, overrd);
                     if previous.is_none() || previous.unwrap() != current {
                         write_i8(&stream, DeviceMessage::InjectionStatus.to_i8().unwrap()).await?;
                         write_i32(&stream, channel).await?;
@@ -169,21 +297,30 @@ async fn handle_connection(stream: &TcpStream, timer: GlobalTimer) -> Result<()>
                         *previous = Some(current);
                     }
                 }
-                next_check = next_check + Milliseconds(200);
+                next_check = timer.get_time() + Milliseconds(200);
             }
         }
     }
 }
 
-pub fn start(timer: GlobalTimer) {
+pub fn start(
+    timer: GlobalTimer,
+    aux_mutex: &Rc<Mutex<bool>>,
+    routing_table: &Rc<RefCell<drtio_routing::RoutingTable>>,
+) {
+    let aux_mutex = aux_mutex.clone();
+    let routing_table = routing_table.clone();
     task::spawn(async move {
         loop {
+            let aux_mutex = aux_mutex.clone();
+            let routing_table = routing_table.clone();
             let stream = TcpStream::accept(1383, 2048, 2048).await.unwrap();
             task::spawn(async move {
                 info!("received connection");
-                let result = handle_connection(&stream, timer).await;
+                let routing_table = routing_table.borrow();
+                let result = handle_connection(&stream, timer, &aux_mutex, &routing_table).await;
                 match result {
-                    Err(Error::NetworkError(smoltcp::Error::Illegal)) => info!("peer closed connection"),
+                    Err(Error::NetworkError(smoltcp::Error::Finished)) => info!("peer closed connection"),
                     Err(error) => warn!("connection terminated: {}", error),
                     _ => (),
                 }

src/runtime/src/panic.rs

@@ -1,22 +1,22 @@
-use libboard_zynq::{print, println};
-use libregister::RegisterR;
+#[cfg(feature = "target_kasli_soc")]
+use libboard_zynq::error_led::ErrorLED;
+use libboard_zynq::{print, println, timer::GlobalTimer};
+use libconfig::Config;
 use libcortex_a9::regs::MPIDR;
+use libregister::RegisterR;
+use log::error;
 use unwind::backtrace;
 
+use crate::comms::soft_panic_main;
+
 static mut PANICKED: [bool; 2] = [false; 2];
+static mut SOFT_PANICKED: bool = false;
 
 #[panic_handler]
 fn panic(info: &core::panic::PanicInfo) -> ! {
     let id = MPIDR.read().cpu_id() as usize;
-    print!("Core {} ", id);
-    unsafe {
-        if PANICKED[id] {
-            println!("nested panic!");
-            loop {}
-        }
-        PANICKED[id] = true;
-    }
-    print!("panic at ");
+    let soft_panicked = unsafe { SOFT_PANICKED };
+    print!("Core {} panic at ", id);
     if let Some(location) = info.location() {
         print!("{}:{}:{}", location.file(), location.line(), location.column());
     } else {
@@ -27,6 +27,20 @@ fn panic(info: &core::panic::PanicInfo) -> ! {
     } else {
         println!("");
     }
+    unsafe {
+        // soft panics only allowed for core 0
+        if PANICKED[id] && (SOFT_PANICKED || id == 1) {
+            println!("nested panic!");
+            loop {}
+        }
+        SOFT_PANICKED = true;
+        PANICKED[id] = true;
+    }
+    #[cfg(feature = "target_kasli_soc")]
+    {
+        let mut err_led = ErrorLED::error_led();
+        err_led.toggle(true);
+    }
     println!("Backtrace: ");
     let _ = backtrace(|ip| {
         // Backtrace gives us the return address, i.e. the address after the delay slot,
@@ -34,6 +48,26 @@ fn panic(info: &core::panic::PanicInfo) -> ! {
         print!("{:#08x} ", ip - 2 * 4);
     });
     println!("\nEnd backtrace");
-
+    if !soft_panicked && id == 0 {
+        soft_panic(info);
+    }
     loop {}
 }
+
+fn soft_panic(info: &core::panic::PanicInfo) -> ! {
+    // write panic info to log, so coremgmt can also read it
+    if let Some(location) = info.location() {
+        error!("panic at {}:{}:{}", location.file(), location.line(), location.column());
+    } else {
+        error!("panic at unknown location");
+    }
+    if let Some(message) = info.message() {
+        error!("panic message: {}", message);
+    }
+    let timer = GlobalTimer::start();
+    let cfg = match Config::new() {
+        Ok(cfg) => cfg,
+        Err(_) => Config::new_dummy(),
+    };
+    soft_panic_main(timer, cfg);
+}

src/runtime/src/proto_async.rs

@@ -1,8 +1,7 @@
-use core::cmp::min;
-use core::cell::RefCell;
+use core::{cell::RefCell, cmp::min};
 
-use libboard_zynq::smoltcp;
 use libasync::smoltcp::TcpStream;
+use libboard_zynq::smoltcp;
 
 type Result<T> = core::result::Result<T, smoltcp::Error>;
 
@@ -14,25 +13,27 @@ enum RecvState<T> {
 pub async fn expect(stream: &TcpStream, pattern: &[u8]) -> Result<bool> {
     let mut state = RecvState::NeedsMore(0, true);
     loop {
-        state = stream.recv(|buf| {
-            let mut consumed = 0;
-            if let RecvState::NeedsMore(mut cur_index, _) = state {
-                for b in buf.iter() {
-                    consumed += 1;
-                    if *b == pattern[cur_index] {
-                        if cur_index + 1 == pattern.len() {
-                            return (consumed, RecvState::Completed(true));
+        state = stream
+            .recv(|buf| {
+                let mut consumed = 0;
+                if let RecvState::NeedsMore(mut cur_index, _) = state {
+                    for b in buf.iter() {
+                        consumed += 1;
+                        if *b == pattern[cur_index] {
+                            if cur_index + 1 == pattern.len() {
+                                return (consumed, RecvState::Completed(true));
+                            }
+                        } else {
+                            return (consumed, RecvState::Completed(false));
                         }
-                    } else {
-                        return (consumed, RecvState::Completed(false));
+                        cur_index += 1;
                     }
-                    cur_index += 1;
+                    (consumed, RecvState::NeedsMore(cur_index, true))
+                } else {
+                    unreachable!();
                 }
-                (consumed, RecvState::NeedsMore(cur_index, true))
-            } else {
-                unreachable!();
-            }
-        }).await?;
+            })
+            .await?;
         if let RecvState::Completed(result) = state {
             return Ok(result);
         }
@@ -40,27 +41,23 @@ pub async fn expect(stream: &TcpStream, pattern: &[u8]) -> Result<bool> {
 }
 
 pub async fn read_bool(stream: &TcpStream) -> Result<bool> {
-    Ok(stream.recv(|buf| {
-        (1, buf[0] != 0)
-    }).await?)
+    Ok(stream.recv(|buf| (1, buf[0] != 0)).await?)
 }
 
 pub async fn read_i8(stream: &TcpStream) -> Result<i8> {
-    Ok(stream.recv(|buf| {
-        (1, buf[0] as i8)
-    }).await?)
+    Ok(stream.recv(|buf| (1, buf[0] as i8)).await?)
 }
 
 pub async fn read_i32(stream: &TcpStream) -> Result<i32> {
     let mut buffer: [u8; 4] = [0; 4];
     read_chunk(stream, &mut buffer).await?;
-    Ok(i32::from_be_bytes(buffer))
+    Ok(i32::from_le_bytes(buffer))
 }
 
 pub async fn read_i64(stream: &TcpStream) -> Result<i64> {
     let mut buffer: [u8; 8] = [0; 8];
     read_chunk(stream, &mut buffer).await?;
-    Ok(i64::from_be_bytes(buffer))
+    Ok(i64::from_le_bytes(buffer))
 }
 
 pub async fn read_chunk(stream: &TcpStream, destination: &mut [u8]) -> Result<()> {
@@ -68,12 +65,14 @@ pub async fn read_chunk(stream: &TcpStream, destination: &mut [u8]) -> Result<()
     let destination = RefCell::new(destination);
     let mut done = 0;
     while done < total {
-        let count = stream.recv(|buf| {
-            let mut destination = destination.borrow_mut();
-            let count = min(total - done, buf.len());
-            destination[done..done + count].copy_from_slice(&buf[..count]);
-            (count, count)
-        }).await?;
+        let count = stream
+            .recv(|buf| {
+                let mut destination = destination.borrow_mut();
+                let count = min(total - done, buf.len());
+                destination[done..done + count].copy_from_slice(&buf[..count]);
+                (count, count)
+            })
+            .await?;
         done += count;
     }
     Ok(())
@@ -90,12 +89,12 @@ pub async fn write_bool(stream: &TcpStream, value: bool) -> Result<()> {
 }
 
 pub async fn write_i32(stream: &TcpStream, value: i32) -> Result<()> {
-    stream.send_slice(&value.to_be_bytes()).await?;
+    stream.send_slice(&value.to_le_bytes()).await?;
     Ok(())
 }
 
 pub async fn write_i64(stream: &TcpStream, value: i64) -> Result<()> {
-    stream.send_slice(&value.to_be_bytes()).await?;
+    stream.send_slice(&value.to_le_bytes()).await?;
     Ok(())
 }
 

src/runtime/src/rpc.rs

@@ -1,579 +0,0 @@
-use core::str;
-use core::future::Future;
-use cslice::{CSlice, CMutSlice};
-use log::trace;
-use byteorder::{NetworkEndian, ByteOrder};
-
-use core_io::{Write, Error};
-use libboard_zynq::smoltcp;
-use libasync::smoltcp::TcpStream;
-use alloc::boxed::Box;
-use async_recursion::async_recursion;
-
-use crate::proto_core_io::ProtoWrite;
-use crate::proto_async;
-use self::tag::{Tag, TagIterator, split_tag};
-
-unsafe fn align_ptr<T>(ptr: *const ()) -> *const T {
-    let alignment = core::mem::align_of::<T>() as isize;
-    let fix = (alignment - (ptr as isize) % alignment) % alignment;
-    ((ptr as isize) + fix) as *const T
-}
-
-unsafe fn align_ptr_mut<T>(ptr: *mut ()) -> *mut T {
-    let alignment = core::mem::align_of::<T>() as isize;
-    let fix = (alignment - (ptr as isize) % alignment) % alignment;
-    ((ptr as isize) + fix) as *mut T
-}
-
-#[async_recursion(?Send)]
-async unsafe fn recv_value<F>(stream: &TcpStream, tag: Tag<'async_recursion>, data: &mut *mut (),
-                              alloc: &(impl Fn(usize) -> F + 'async_recursion))
-                             -> Result<(), smoltcp::Error>
-                          where F: Future<Output=*mut ()>
-{
-    macro_rules! consume_value {
-        ($ty:ty, |$ptr:ident| $map:expr) => ({
-            let $ptr = align_ptr_mut::<$ty>(*data);
-            *data = $ptr.offset(1) as *mut ();
-            $map
-        })
-    }
-
-    match tag {
-        Tag::None => Ok(()),
-        Tag::Bool =>
-            consume_value!(i8, |ptr| {
-                *ptr = proto_async::read_i8(stream).await?;
-                Ok(())
-            }),
-        Tag::Int32 =>
-            consume_value!(i32, |ptr| {
-                *ptr = proto_async::read_i32(stream).await?;
-                Ok(())
-            }),
-        Tag::Int64 | Tag::Float64 =>
-            consume_value!(i64, |ptr| {
-                *ptr = proto_async::read_i64(stream).await?;
-                Ok(())
-            }),
-        Tag::String | Tag::Bytes | Tag::ByteArray => {
-            consume_value!(CMutSlice<u8>, |ptr| {
-                let length = proto_async::read_i32(stream).await? as usize;
-                *ptr = CMutSlice::new(alloc(length).await as *mut u8, length);
-                proto_async::read_chunk(stream, (*ptr).as_mut()).await?;
-                Ok(())
-            })
-        }
-        Tag::Tuple(it, arity) => {
-            let mut it = it.clone();
-            for _ in 0..arity {
-                let tag = it.next().expect("truncated tag");
-                recv_value(stream, tag, data, alloc).await?;
-            }
-            Ok(())
-        }
-        Tag::List(it) => {
-            #[repr(C)]
-            struct List { elements: *mut (), length: u32 };
-            consume_value!(List, |ptr| {
-                let length = proto_async::read_i32(stream).await? as usize;
-                (*ptr).length  = length as u32;
-                let tag = it.clone().next().expect("truncated tag");
-                let mut data = alloc(tag.size() * length as usize).await;
-
-                (*ptr).elements = data;
-                match tag {
-                    Tag::Bool => {
-                        let ptr = align_ptr_mut::<u8>(data);
-                        let dest = core::slice::from_raw_parts_mut(ptr, length);
-                        proto_async::read_chunk(stream, dest).await?;
-                    },
-                    Tag::Int32 => {
-                        let ptr = align_ptr_mut::<u32>(data);
-                        // reading as raw bytes and do endianness conversion later
-                        let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 4);
-                        proto_async::read_chunk(stream, dest).await?;
-                        drop(dest);
-                        let dest = core::slice::from_raw_parts_mut(ptr, length);
-                        NetworkEndian::from_slice_u32(dest);
-                    },
-                    Tag::Int64 | Tag::Float64 => {
-                        let ptr = align_ptr_mut::<u64>(data);
-                        let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 8);
-                        proto_async::read_chunk(stream, dest).await?;
-                        drop(dest);
-                        let dest = core::slice::from_raw_parts_mut(ptr, length);
-                        NetworkEndian::from_slice_u64(dest);
-                    },
-                    _ => {
-                        for _ in 0..(*ptr).length as usize {
-                            recv_value(stream, tag, &mut data, alloc).await?
-                        }
-                    }
-                }
-                Ok(())
-            })
-        }
-        Tag::Array(it, num_dims) => {
-            consume_value!(*mut (), |buffer| {
-                let mut total_len: u32 = 1;
-                for _ in 0..num_dims {
-                    let len = proto_async::read_i32(stream).await? as u32;
-                    total_len *= len;
-                    consume_value!(u32, |ptr| *ptr = len )
-                }
-
-                let elt_tag = it.clone().next().expect("truncated tag");
-                *buffer = alloc(elt_tag.size() * total_len as usize).await;
-
-                let length = total_len as usize;
-                let mut data = *buffer;
-                match elt_tag {
-                    Tag::Bool => {
-                        let ptr = align_ptr_mut::<u8>(data);
-                        let dest = core::slice::from_raw_parts_mut(ptr, length);
-                        proto_async::read_chunk(stream, dest).await?;
-                    },
-                    Tag::Int32 => {
-                        let ptr = align_ptr_mut::<u32>(data);
-                        let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 4);
-                        proto_async::read_chunk(stream, dest).await?;
-                        drop(dest);
-                        let dest = core::slice::from_raw_parts_mut(ptr, length);
-                        NetworkEndian::from_slice_u32(dest);
-                    },
-                    Tag::Int64 | Tag::Float64 => {
-                        let ptr = align_ptr_mut::<u64>(data);
-                        let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 8);
-                        proto_async::read_chunk(stream, dest).await?;
-                        drop(dest);
-                        let dest = core::slice::from_raw_parts_mut(ptr, length);
-                        NetworkEndian::from_slice_u64(dest);
-                    },
-                    _ => {
-                        for _ in 0..length {
-                            recv_value(stream, elt_tag, &mut data, alloc).await?
-                        }
-                    }
-                }
-                Ok(())
-            })
-        }
-        Tag::Range(it) => {
-            let tag = it.clone().next().expect("truncated tag");
-            recv_value(stream, tag, data, alloc).await?;
-            recv_value(stream, tag, data, alloc).await?;
-            recv_value(stream, tag, data, alloc).await?;
-            Ok(())
-        }
-        Tag::Keyword(_) => unreachable!(),
-        Tag::Object => unreachable!()
-    }
-}
-
-pub async fn recv_return<F>(stream: &TcpStream, tag_bytes: &[u8], data: *mut (),
-                            alloc: &impl Fn(usize) -> F)
-                           -> Result<(), smoltcp::Error>
-                        where F: Future<Output=*mut ()>
-{
-    let mut it = TagIterator::new(tag_bytes);
-    trace!("recv ...->{}", it);
-
-    let tag = it.next().expect("truncated tag");
-    let mut data = data;
-    unsafe { recv_value(stream, tag, &mut data, alloc).await? };
-
-    Ok(())
-}
-
-unsafe fn send_value<W>(writer: &mut W, tag: Tag, data: &mut *const ())
-                       -> Result<(), Error>
-    where W: Write + ?Sized
-{
-    macro_rules! consume_value {
-        ($ty:ty, |$ptr:ident| $map:expr) => ({
-            let $ptr = align_ptr::<$ty>(*data);
-            *data = $ptr.offset(1) as *const ();
-            $map
-        })
-    }
-
-    writer.write_u8(tag.as_u8())?;
-    match tag {
-        Tag::None => Ok(()),
-        Tag::Bool =>
-            consume_value!(u8, |ptr|
-                writer.write_u8(*ptr)),
-        Tag::Int32 =>
-            consume_value!(u32, |ptr|
-                writer.write_u32(*ptr)),
-        Tag::Int64 | Tag::Float64 =>
-            consume_value!(u64, |ptr|
-                writer.write_u64(*ptr)),
-        Tag::String =>
-            consume_value!(CSlice<u8>, |ptr|
-                writer.write_string(str::from_utf8((*ptr).as_ref()).unwrap())),
-        Tag::Bytes | Tag::ByteArray =>
-            consume_value!(CSlice<u8>, |ptr|
-                writer.write_bytes((*ptr).as_ref())),
-        Tag::Tuple(it, arity) => {
-            let mut it = it.clone();
-            writer.write_u8(arity)?;
-            for _ in 0..arity {
-                let tag = it.next().expect("truncated tag");
-                send_value(writer, tag, data)?
-            }
-            Ok(())
-        }
-        Tag::List(it) => {
-            #[repr(C)]
-            struct List { elements: *const (), length: u32 };
-            consume_value!(List, |ptr| {
-                let length = (*ptr).length as isize;
-                writer.write_u32((*ptr).length)?;
-                let tag = it.clone().next().expect("truncated tag");
-                let mut data = (*ptr).elements;
-                writer.write_u8(tag.as_u8())?;
-                match tag {
-                    Tag::Bool => {
-                        // we can pretend this is u8...
-                        let ptr1 = align_ptr::<u8>(data);
-                        let slice = core::slice::from_raw_parts(ptr1, length as usize);
-                        writer.write_all(slice)?;
-                    },
-                    Tag::Int32 => {
-                        let ptr1 = align_ptr::<i32>(data);
-                        let slice = core::slice::from_raw_parts(ptr1, length as usize);
-                        let mut v: alloc::vec::Vec<i32> = slice.to_vec();
-                        NetworkEndian::from_slice_i32(&mut v);
-                        let slice2 = core::slice::from_raw_parts(
-                            v.as_ptr() as usize as *const u8,
-                            length as usize * 4
-                        );
-                        writer.write_all(slice2)?;
-                    },
-                    Tag::Int64 | Tag::Float64 => {
-                        let ptr1 = align_ptr::<i64>(data);
-                        let slice = core::slice::from_raw_parts(ptr1, length as usize);
-                        let mut v: alloc::vec::Vec<i64> = slice.to_vec();
-                        NetworkEndian::from_slice_i64(&mut v);
-                        let slice2 = core::slice::from_raw_parts(
-                            v.as_ptr() as usize as *const u8,
-                            length as usize * 8
-                        );
-                        writer.write_all(slice2)?;
-                    },
-                    // non-primitive types, not sure if this would happen but we can handle it...
-                    _ => {
-                        for _ in 0..length {
-                            send_value(writer, tag, &mut data)?;
-                        }
-                    }
-                };
-                Ok(())
-            })
-        }
-        Tag::Array(it, num_dims) => {
-            writer.write_u8(num_dims)?;
-            consume_value!(*const(), |buffer| {
-                let elt_tag = it.clone().next().expect("truncated tag");
-
-                let mut total_len = 1;
-                for _ in 0..num_dims {
-                    consume_value!(u32, |len| {
-                        writer.write_u32(*len)?;
-                        total_len *= *len;
-                    })
-                }
-                let mut data = *buffer;
-                let length = total_len as isize;
-                writer.write_u8(elt_tag.as_u8())?;
-                match elt_tag {
-                    Tag::Bool => {
-                        let ptr1 = align_ptr::<u8>(data);
-                        let slice = core::slice::from_raw_parts(ptr1, length as usize);
-                        writer.write_all(slice)?;
-                    },
-                    Tag::Int32 => {
-                        let ptr1 = align_ptr::<i32>(data);
-                        let slice = core::slice::from_raw_parts(ptr1, length as usize);
-                        let mut v: alloc::vec::Vec<i32> = slice.to_vec();
-                        NetworkEndian::from_slice_i32(&mut v);
-                        let slice2 = core::slice::from_raw_parts(
-                            v.as_ptr() as usize as *const u8,
-                            length as usize * 4
-                        );
-                        writer.write_all(slice2)?;
-                    },
-                    Tag::Int64 | Tag::Float64 => {
-                        let ptr1 = align_ptr::<i64>(data);
-                        let slice = core::slice::from_raw_parts(ptr1, length as usize);
-                        let mut v: alloc::vec::Vec<i64> = slice.to_vec();
-                        NetworkEndian::from_slice_i64(&mut v);
-                        let slice2 = core::slice::from_raw_parts(
-                            v.as_ptr() as usize as *const u8,
-                            length as usize * 8
-                        );
-                        writer.write_all(slice2)?;
-                    },
-                    // non-primitive types, not sure if this would happen but we can handle it...
-                    _ => {
-                        for _ in 0..length {
-                            send_value(writer, elt_tag, &mut data)?;
-                        }
-                    }
-                };
-                Ok(())
-            })
-        }
-        Tag::Range(it) => {
-            let tag = it.clone().next().expect("truncated tag");
-            send_value(writer, tag, data)?;
-            send_value(writer, tag, data)?;
-            send_value(writer, tag, data)?;
-            Ok(())
-        }
-        Tag::Keyword(it) => {
-            #[repr(C)]
-            struct Keyword<'a> { name: CSlice<'a, u8> };
-            consume_value!(Keyword, |ptr| {
-                writer.write_string(str::from_utf8((*ptr).name.as_ref()).unwrap())?;
-                let tag = it.clone().next().expect("truncated tag");
-                let mut data = ptr.offset(1) as *const ();
-                send_value(writer, tag, &mut data)
-            })
-            // Tag::Keyword never appears in composite types, so we don't have
-            // to accurately advance data.
-        }
-        Tag::Object => {
-            #[repr(C)]
-            struct Object { id: u32 };
-            consume_value!(*const Object, |ptr|
-                writer.write_u32((**ptr).id))
-        }
-    }
-}
-
-pub fn send_args<W>(writer: &mut W, service: u32, tag_bytes: &[u8], data: *const *const ())
-                   -> Result<(), Error>
-    where W: Write + ?Sized
-{
-    let (arg_tags_bytes, return_tag_bytes) = split_tag(tag_bytes);
-
-    let mut args_it = TagIterator::new(arg_tags_bytes);
-    let return_it = TagIterator::new(return_tag_bytes);
-    trace!("send<{}>({})->{}", service, args_it, return_it);
-
-    writer.write_u32(service)?;
-    for index in 0.. {
-        if let Some(arg_tag) = args_it.next() {
-            let mut data = unsafe { *data.offset(index) };
-            unsafe { send_value(writer, arg_tag, &mut data)? };
-        } else {
-            break
-        }
-    }
-    writer.write_u8(0)?;
-    writer.write_bytes(return_tag_bytes)?;
-
-    Ok(())
-}
-
-mod tag {
-    use core::fmt;
-
-    pub fn split_tag(tag_bytes: &[u8]) -> (&[u8], &[u8]) {
-        let tag_separator =
-            tag_bytes.iter()
-                     .position(|&b| b == b':')
-                     .expect("tag without a return separator");
-        let (arg_tags_bytes, rest) = tag_bytes.split_at(tag_separator);
-        let return_tag_bytes = &rest[1..];
-        (arg_tags_bytes, return_tag_bytes)
-    }
-
-    #[derive(Debug, Clone, Copy)]
-    pub enum Tag<'a> {
-        None,
-        Bool,
-        Int32,
-        Int64,
-        Float64,
-        String,
-        Bytes,
-        ByteArray,
-        Tuple(TagIterator<'a>, u8),
-        List(TagIterator<'a>),
-        Array(TagIterator<'a>, u8),
-        Range(TagIterator<'a>),
-        Keyword(TagIterator<'a>),
-        Object
-    }
-
-    impl<'a> Tag<'a> {
-        pub fn as_u8(self) -> u8 {
-            match self {
-                Tag::None => b'n',
-                Tag::Bool => b'b',
-                Tag::Int32 => b'i',
-                Tag::Int64 => b'I',
-                Tag::Float64 => b'f',
-                Tag::String => b's',
-                Tag::Bytes => b'B',
-                Tag::ByteArray => b'A',
-                Tag::Tuple(_, _) => b't',
-                Tag::List(_) => b'l',
-                Tag::Array(_, _) => b'a',
-                Tag::Range(_) => b'r',
-                Tag::Keyword(_) => b'k',
-                Tag::Object => b'O',
-            }
-        }
-
-        pub fn size(self) -> usize {
-            match self {
-                Tag::None => 0,
-                Tag::Bool => 1,
-                Tag::Int32 => 4,
-                Tag::Int64 => 8,
-                Tag::Float64 => 8,
-                Tag::String => 8,
-                Tag::Bytes => 8,
-                Tag::ByteArray => 8,
-                Tag::Tuple(it, arity) => {
-                    let mut size = 0;
-                    let mut it = it.clone();
-                    for _ in 0..arity {
-                        let tag = it.next().expect("truncated tag");
-                        size += tag.size();
-                    }
-                    size
-                }
-                Tag::List(_) => 8,
-                Tag::Array(_, num_dims) => 4 * (1 + num_dims as usize),
-                Tag::Range(it) => {
-                    let tag = it.clone().next().expect("truncated tag");
-                    tag.size() * 3
-                }
-                Tag::Keyword(_) => unreachable!(),
-                Tag::Object => unreachable!(),
-            }
-        }
-    }
-
-    #[derive(Debug, Clone, Copy)]
-    pub struct TagIterator<'a> {
-        data: &'a [u8]
-    }
-
-    impl<'a> TagIterator<'a> {
-        pub fn new(data: &'a [u8]) -> TagIterator<'a> {
-            TagIterator { data: data }
-        }
-
-        pub fn next(&mut self) -> Option<Tag<'a>> {
-            if self.data.len() == 0 {
-                return None
-            }
-
-            let tag_byte = self.data[0];
-            self.data = &self.data[1..];
-            Some(match tag_byte {
-                b'n' => Tag::None,
-                b'b' => Tag::Bool,
-                b'i' => Tag::Int32,
-                b'I' => Tag::Int64,
-                b'f' => Tag::Float64,
-                b's' => Tag::String,
-                b'B' => Tag::Bytes,
-                b'A' => Tag::ByteArray,
-                b't' => {
-                    let count = self.data[0];
-                    self.data = &self.data[1..];
-                    Tag::Tuple(self.sub(count), count)
-                }
-                b'l' => Tag::List(self.sub(1)),
-                b'a' => {
-                    let count = self.data[0];
-                    self.data = &self.data[1..];
-                    Tag::Array(self.sub(1), count)
-                }
-                b'r' => Tag::Range(self.sub(1)),
-                b'k' => Tag::Keyword(self.sub(1)),
-                b'O' => Tag::Object,
-                _    => unreachable!()
-            })
-        }
-
-        fn sub(&mut self, count: u8) -> TagIterator<'a> {
-            let data = self.data;
-            for _ in 0..count {
-                self.next().expect("truncated tag");
-            }
-            TagIterator { data: &data[..(data.len() - self.data.len())] }
-        }
-    }
-
-    impl<'a> fmt::Display for TagIterator<'a> {
-        fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-            let mut it = self.clone();
-            let mut first = true;
-            while let Some(tag) = it.next() {
-                if first {
-                    first = false
-                } else {
-                    write!(f, ", ")?
-                }
-
-                match tag {
-                    Tag::None =>
-                        write!(f, "None")?,
-                    Tag::Bool =>
-                        write!(f, "Bool")?,
-                    Tag::Int32 =>
-                        write!(f, "Int32")?,
-                    Tag::Int64 =>
-                        write!(f, "Int64")?,
-                    Tag::Float64 =>
-                        write!(f, "Float64")?,
-                    Tag::String =>
-                        write!(f, "String")?,
-                    Tag::Bytes =>
-                        write!(f, "Bytes")?,
-                    Tag::ByteArray =>
-                        write!(f, "ByteArray")?,
-                    Tag::Tuple(it, _) => {
-                        write!(f, "Tuple(")?;
-                        it.fmt(f)?;
-                        write!(f, ")")?;
-                    }
-                    Tag::List(it) => {
-                        write!(f, "List(")?;
-                        it.fmt(f)?;
-                        write!(f, ")")?;
-                    }
-                    Tag::Array(it, num_dims) => {
-                        write!(f, "Array(")?;
-                        it.fmt(f)?;
-                        write!(f, ", {})", num_dims)?;
-                    }
-                    Tag::Range(it) => {
-                        write!(f, "Range(")?;
-                        it.fmt(f)?;
-                        write!(f, ")")?;
-                    }
-                    Tag::Keyword(it) => {
-                        write!(f, "Keyword(")?;
-                        it.fmt(f)?;
-                        write!(f, ")")?;
-                    }
-                    Tag::Object =>
-                        write!(f, "Object")?,
-                }
-            }
-
-            Ok(())
-        }
-    }
-}

src/runtime/src/rpc_async.rs

@@ -0,0 +1,197 @@
+use alloc::boxed::Box;
+use core::future::Future;
+
+use async_recursion::async_recursion;
+use byteorder::{ByteOrder, NativeEndian};
+use cslice::CMutSlice;
+use ksupport::rpc::{tag::{Tag, TagIterator},
+                    *};
+use libasync::smoltcp::TcpStream;
+use libboard_zynq::smoltcp;
+use log::trace;
+
+use crate::proto_async;
+
+/// Reads (deserializes) `length` array or list elements of type `tag` from `stream`,
+/// writing them into the buffer given by `storage`.
+///
+/// `alloc` is used for nested allocations (if elements themselves contain
+/// lists/arrays), see [recv_value].
+#[async_recursion(?Send)]
+async unsafe fn recv_elements<F>(
+    stream: &TcpStream,
+    elt_tag: Tag<'async_recursion>,
+    length: usize,
+    storage: *mut (),
+    alloc: &(impl Fn(usize) -> F + 'async_recursion),
+) -> Result<(), smoltcp::Error>
+where
+    F: Future<Output = *mut ()>,
+{
+    // List of simple types are special-cased in the protocol for performance.
+    match elt_tag {
+        Tag::Bool => {
+            let dest = core::slice::from_raw_parts_mut(storage as *mut u8, length);
+            proto_async::read_chunk(stream, dest).await?;
+        }
+        Tag::Int32 => {
+            let ptr = storage as *mut u32;
+            let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 4);
+            proto_async::read_chunk(stream, dest).await?;
+            drop(dest);
+            let dest = core::slice::from_raw_parts_mut(ptr, length);
+            NativeEndian::from_slice_u32(dest);
+        }
+        Tag::Int64 | Tag::Float64 => {
+            let ptr = storage as *mut u64;
+            let dest = core::slice::from_raw_parts_mut(ptr as *mut u8, length * 8);
+            proto_async::read_chunk(stream, dest).await?;
+            drop(dest);
+            let dest = core::slice::from_raw_parts_mut(ptr, length);
+            NativeEndian::from_slice_u64(dest);
+        }
+        _ => {
+            let mut data = storage;
+            for _ in 0..length {
+                recv_value(stream, elt_tag, &mut data, alloc).await?
+            }
+        }
+    }
+    Ok(())
+}
+
+/// Reads (deserializes) a value of type `tag` from `stream`, writing the results to
+/// the kernel-side buffer `data` (the passed pointer to which is incremented to point
+/// past the just-received data). For nested allocations (lists/arrays), `alloc` is
+/// invoked any number of times with the size of the required allocation as a parameter
+/// (which is assumed to be correctly aligned for all payload types).
+#[async_recursion(?Send)]
+async unsafe fn recv_value<F>(
+    stream: &TcpStream,
+    tag: Tag<'async_recursion>,
+    data: &mut *mut (),
+    alloc: &(impl Fn(usize) -> F + 'async_recursion),
+) -> Result<(), smoltcp::Error>
+where
+    F: Future<Output = *mut ()>,
+{
+    macro_rules! consume_value {
+        ($ty:ty, | $ptr:ident | $map:expr) => {{
+            let $ptr = align_ptr_mut::<$ty>(*data);
+            *data = $ptr.offset(1) as *mut ();
+            $map
+        }};
+    }
+
+    match tag {
+        Tag::None => Ok(()),
+        Tag::Bool => consume_value!(i8, |ptr| {
+            *ptr = proto_async::read_i8(stream).await?;
+            Ok(())
+        }),
+        Tag::Int32 => consume_value!(i32, |ptr| {
+            *ptr = proto_async::read_i32(stream).await?;
+            Ok(())
+        }),
+        Tag::Int64 | Tag::Float64 => consume_value!(i64, |ptr| {
+            *ptr = proto_async::read_i64(stream).await?;
+            Ok(())
+        }),
+        Tag::String | Tag::Bytes | Tag::ByteArray => {
+            consume_value!(CMutSlice<u8>, |ptr| {
+                let length = proto_async::read_i32(stream).await? as usize;
+                *ptr = CMutSlice::new(alloc(length).await as *mut u8, length);
+                proto_async::read_chunk(stream, (*ptr).as_mut()).await?;
+                Ok(())
+            })
+        }
+        Tag::Tuple(it, arity) => {
+            let alignment = tag.alignment();
+            *data = round_up_mut(*data, alignment);
+            let mut it = it.clone();
+            for _ in 0..arity {
+                let tag = it.next().expect("truncated tag");
+                recv_value(stream, tag, data, alloc).await?
+            }
+            // Take into account any tail padding (if element(s) with largest alignment
+            // are not at the end).
+            *data = round_up_mut(*data, alignment);
+            Ok(())
+        }
+        Tag::List(it) => {
+            #[repr(C)]
+            struct List {
+                elements: *mut (),
+                length: usize,
+            }
+            consume_value!(*mut List, |ptr_to_list| {
+                let tag = it.clone().next().expect("truncated tag");
+                let length = proto_async::read_i32(stream).await? as usize;
+
+                // To avoid multiple kernel CPU roundtrips, use a single allocation for
+                // both the pointer/length List (slice) and the backing storage for the
+                // elements. We can assume that alloc() is aligned suitably, so just
+                // need to take into account any extra padding required.
+                // (Note: At the time of writing, there will never actually be any types
+                // with alignment larger than 8 bytes, so storage_offset == 0 always.)
+                let list_size = 4 + 4;
+                let storage_offset = round_up(list_size, tag.alignment());
+                let storage_size = tag.size() * length;
+
+                let allocation = alloc(storage_offset + storage_size).await as *mut u8;
+                *ptr_to_list = allocation as *mut List;
+                let storage = allocation.offset(storage_offset as isize) as *mut ();
+
+                (**ptr_to_list).length = length;
+                (**ptr_to_list).elements = storage;
+                recv_elements(stream, tag, length, storage, alloc).await
+            })
+        }
+        Tag::Array(it, num_dims) => {
+            consume_value!(*mut (), |buffer| {
+                // Deserialize length along each dimension and compute total number of
+                // elements.
+                let mut total_len: usize = 1;
+                for _ in 0..num_dims {
+                    let len = proto_async::read_i32(stream).await? as usize;
+                    total_len *= len;
+                    consume_value!(usize, |ptr| *ptr = len)
+                }
+
+                // Allocate backing storage for elements; deserialize them.
+                let elt_tag = it.clone().next().expect("truncated tag");
+                *buffer = alloc(elt_tag.size() * total_len).await;
+                recv_elements(stream, elt_tag, total_len, *buffer, alloc).await
+            })
+        }
+        Tag::Range(it) => {
+            *data = round_up_mut(*data, tag.alignment());
+            let tag = it.clone().next().expect("truncated tag");
+            recv_value(stream, tag, data, alloc).await?;
+            recv_value(stream, tag, data, alloc).await?;
+            recv_value(stream, tag, data, alloc).await?;
+            Ok(())
+        }
+        Tag::Keyword(_) => unreachable!(),
+        Tag::Object => unreachable!(),
+    }
+}
+
+pub async fn recv_return<F>(
+    stream: &TcpStream,
+    tag_bytes: &[u8],
+    data: *mut (),
+    alloc: &impl Fn(usize) -> F,
+) -> Result<(), smoltcp::Error>
+where
+    F: Future<Output = *mut ()>,
+{
+    let mut it = TagIterator::new(tag_bytes);
+    trace!("recv ...->{}", it);
+
+    let tag = it.next().expect("truncated tag");
+    let mut data = data;
+    unsafe { recv_value(stream, tag, &mut data, alloc).await? };
+
+    Ok(())
+}