doc: Fix example flake

Since DRTIO setups now have sequences from multiple cores, it can have more than one StoppedMessage.

Signed-off-by: Egor Savkin <es@m-labs.hk>

.github/pull_request_template.md

@@ -51,7 +51,7 @@ Closes #XXX
 
 ### Documentation Changes
 
-- [ ] Check, test, and update the documentation in [doc/](../doc/). Build documentation (`cd doc/manual/; make html`) to ensure no errors.
+- [ ] Check, test, and update the documentation in [doc/](../doc/). Build documentation (`nix build .#artiq-manual-html; nix build .#artiq-manual-pdf`) to ensure no errors.
 
 ### Git Logistics
 

.gitignore

@@ -11,6 +11,7 @@ __pycache__/
 .ipynb_checkpoints
 /doc/manual/_build
 /build
+/result
 /dist
 /*.egg-info
 /.coverage
@@ -23,7 +24,8 @@ __pycache__/
 /artiq/test/results
 /artiq/examples/*/results
 /artiq/examples/*/last_rid.pyon
-/artiq/examples/*/dataset_db.pyon
+/artiq/examples/*/dataset_db.mdb
+/artiq/examples/*/dataset_db.mdb-lock
 
 # when testing ad-hoc experiments at the root:
 /repository/

CONTRIBUTING.rst

@@ -8,27 +8,27 @@ Reporting Issues/Bugs
 Thanks for `reporting issues to ARTIQ
 <https://github.com/m-labs/artiq/issues/new>`_! You can also discuss issues and
 ask questions on IRC (the #m-labs channel on OFTC), the `Mattermost chat
-<https://chat.m-labs.hk>`_, or on the `forum <https://forum.m-labs.hk>`_.
+<https://chat.m-labs.hk>`_, or in the `forum <https://forum.m-labs.hk>`_.
 
 The best bug reports are those which contain sufficient information. With
 accurate and comprehensive context, an issue can be resolved quickly and
 efficiently. Please consider adding the following data to your issue
 report if possible:
 
-* A clear and unique summary that fits into one line. Also check that
+* A clear and unique summary that fits into one line. Check that this
-  this issue has not yet been reported. If it has, add additional information there.
+  issue has not yet been reported; if it has, add additional information there.
-* Precise steps to reproduce (list of actions that leads to the issue)
+* Precise steps to reproduce (a list of actions that leads to the issue)
 * Expected behavior (what should happen)
 * Actual behavior (what happens instead)
-* Logging message, trace backs, screen shots where relevant
+* Logging message, tracebacks, screenshots, where applicable
 * Components involved (omit irrelevant parts):
 
-  * Operating System
+  * Operating system used
-  * ARTIQ version (with recent versions of ARTIQ, run ``artiq_client --version``)
+  * ARTIQ version (run any command in the form of ``artiq_client --version``)
-  * Version of the gateware and runtime loaded in the core device (in the output of ``artiq_coremgmt -D .... log``)
+  * Gateware and firmware loaded to the core device (in the output of
+    ``artiq_coremgmt [-D ....] log``)
   * Hardware involved
 
-
 For in-depth information on bug reporting, see:
 
 http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
@@ -38,10 +38,10 @@ https://developer.mozilla.org/en-US/docs/Mozilla/QA/Bug_writing_guidelines
 Contributing Code
 =================
 
-ARTIQ welcomes contributions. Write bite-sized patches that can stand alone,
+ARTIQ welcomes contributions. Write bite-size patches that can stand alone,
-clean them up, write proper commit messages, add docstrings and unittests. Then
+clean them up, write proper commit messages, add docstrings and unit tests;
 ``git rebase`` them onto the current master or merge the current master. Verify
-that the testsuite passes. Then submit a pull request. Expect your contribution
+that the test suite passes. Then submit a pull request. Expect your contribution
 to be held up to coding standards (e.g. use ``flake8`` to check yourself).
 
 Checklist for Code Contributions
@@ -51,7 +51,7 @@ Checklist for Code Contributions
 - Use correct spelling and grammar. Use your code editor to help you with
   syntax, spelling, and style
 - Style: PEP-8 (``flake8``)
-- Add, check docstrings and comments
+- Add or update docstrings and comments
 - Split your contribution into logically separate changes (``git rebase
   --interactive``). Merge (squash, fixup) commits that just fix previous commits
   or amend them. Remove unintended changes. Clean up your commits.
@@ -63,12 +63,37 @@ Checklist for Code Contributions
 - Review each of your commits for the above items (``git show``)
 - Update ``RELEASE_NOTES.md`` if there are noteworthy changes, especially if
   there are changes to existing APIs
-- Check, test, and update the documentation in `doc/`
+- Check, test, and update the documentation in ``doc/``
-- Check, test, and update the unittests
+- Check, test, and update the unit tests
 - Close and/or update issues
 
+
+Contributing Documentation
+==========================
+
+ARTIQ welcomes documentation contributions. The ARTIQ manual is hosted online in HTML
+form `here <https://m-labs.hk/artiq/manual/>`__ and in PDF form
+`here <https://m-labs.hk/artiq/manual.pdf>`__. It is generated from source files
+in ``doc/manual``, written in a variant of the
+`reStructured Text <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_
+markup language processed by `Sphinx <https://www.sphinx-doc.org/en/master/>`_, with
+some of the additional reference material processed from inline documentation
+in the ARTIQ source itself.
+
+Write bite-size patches that can stand alone, clean them up, write proper commit 
+messages. Check that your edits render properly and compile without errors: ::
+
+  $ nix build .#artiq-manual-pdf
+  $ nix build .#artiq-manual-html
+
+Elaborations, improvements, clarifications and corrections to any of the material
+are happily accepted, but special attention is drawn to the manual
+`FAQ <https://m-labs.hk/artiq/manual/faq.html>`_, where tips and solutions
+are especially easy to add. See also the FAQ's own
+`section on the subject <https://m-labs.hk/artiq/manual/faq.html#build-documentation>`_.
+
 Copyright and Sign-Off
-----------------------
+======================
 
 Authors retain copyright of their contributions to ARTIQ, but whenever possible
 should use the GNU LGPL version 3 license for them to be merged.
@@ -108,7 +133,7 @@ can certify the below:
             maintained indefinitely and may be redistributed consistent with
             this project or the open source license(s) involved.
 
-then you just add a line saying
+then add a line saying
 
         Signed-off-by: Random J Developer <random@developer.example.org>
 

README.rst

@@ -29,7 +29,7 @@ Website: https://m-labs.hk/artiq
 License
 =======
 
-Copyright (C) 2014-2024 M-Labs Limited.
+Copyright (C) 2014-2025 M-Labs Limited.
 
 ARTIQ is free software: you can redistribute it and/or modify
 it under the terms of the GNU Lesser General Public License as published by

RELEASE_NOTES.rst

@@ -3,8 +3,8 @@
 Release notes
 =============
 
-ARTIQ-8 (Unreleased)
+ARTIQ-8
---------------------
+-------
 
 Highlights:
 

artiq/applets/plot_xy.py

@@ -24,11 +24,11 @@ class XYPlot(pyqtgraph.PlotWidget):
             y = value[self.args.y]
         except KeyError:
             return
-        x = value.get(self.args.x, (False, None))
+        x = value.get(self.args.x)
         if x is None:
             x = np.arange(len(y))
-        error = value.get(self.args.error, (False, None))
+        error = value.get(self.args.error)
-        fit = value.get(self.args.fit, (False, None))
+        fit = value.get(self.args.fit)
 
         if not len(y) or len(y) != len(x):
             self.mismatch['X values'] = True

artiq/applets/simple.py

@@ -24,21 +24,21 @@ class _AppletRequestInterface:
     def set_dataset(self, key, value, unit=None, scale=None, precision=None, persist=None):
         """
         Set a dataset.
-        See documentation of ``artiq.language.environment.set_dataset``.
+        See documentation of :meth:`~artiq.language.environment.HasEnvironment.set_dataset`.
         """
         raise NotImplementedError
 
     def mutate_dataset(self, key, index, value):
         """
         Mutate a dataset.
-        See documentation of ``artiq.language.environment.mutate_dataset``.
+        See documentation of :meth:`~artiq.language.environment.HasEnvironment.mutate_dataset`.
         """
         raise NotImplementedError
 
     def append_to_dataset(self, key, value):
         """
         Append to a dataset.
-        See documentation of ``artiq.language.environment.append_to_dataset``.
+        See documentation of :meth:`~artiq.language.environment.HasEnvironment.append_to_dataset`.
         """
         raise NotImplementedError
 
@@ -49,8 +49,9 @@ class _AppletRequestInterface:
 
         :param expurl: Experiment URL identifying the experiment in the dashboard. Example: 'repo:ArgumentsDemo'.
         :param key: Name of the argument in the experiment.
-        :param value: Object representing the new temporary value of the argument. For ``Scannable`` arguments, this parameter
+        :param value: Object representing the new temporary value of the argument. For :class:`~artiq.language.scan.Scannable` arguments, 
-            should be a ``ScanObject``. The type of the ``ScanObject`` will be set as the selected type when this function is called. 
+            this parameter should be a :class:`~artiq.language.scan.ScanObject`. The type of the :class:`~artiq.language.scan.ScanObject` 
+            will be set as the selected type when this function is called. 
         """
         raise NotImplementedError
 

artiq/browser/files.py

@@ -83,7 +83,7 @@ class ZoomIconView(QtWidgets.QListView):
             w = self.iconSize().width()*self.zoom_step**(
                 ev.angleDelta().y()/120.)
             if a <= w <= b:
-                self.setIconSize(QtCore.QSize(w, w*self.aspect))
+                self.setIconSize(QtCore.QSize(int(w), int(w*self.aspect)))
         else:
             QtWidgets.QListView.wheelEvent(self, ev)
 

artiq/compiler/embedding.py

@@ -88,7 +88,11 @@ class EmbeddingMap:
                 self.subkernel_message_map[msg_type.name] = msg_id
                 self.object_reverse_map[obj_id] = msg_id
 
-        self.preallocate_runtime_exception_names(["RuntimeError",
+        # Keep this list of exceptions in sync with `EXCEPTION_ID_LOOKUP` in `artiq::firmware::ksupport::eh_artiq`
+        # The exceptions declared here must be defined in `artiq.coredevice.exceptions`
+        # Verify synchronization by running the test cases in `artiq.test.coredevice.test_exceptions`
+        self.preallocate_runtime_exception_names([
+            "0:RuntimeError",
             "RTIOUnderflow",
             "RTIOOverflow",
             "RTIODestinationUnreachable",
@@ -99,7 +103,8 @@ class EmbeddingMap:
             "0:ZeroDivisionError",
             "0:IndexError",
             "UnwrapNoneError",
-                                                  "SubkernelError"])
+            "SubkernelError",
+        ])
 
     def preallocate_runtime_exception_names(self, names):
         for i, name in enumerate(names):
@@ -747,9 +752,9 @@ class StitchingInferencer(Inferencer):
                 if elt.__class__ == float:
                     state |= IS_FLOAT
                 elif elt.__class__ == int:
-                    if -2**31 < elt < 2**31-1:
+                    if -2**31 <= elt <= 2**31-1:
                         state |= IS_INT32
-                    elif -2**63 < elt < 2**63-1:
+                    elif -2**63 <= elt <= 2**63-1:
                         state |= IS_INT64
                     else:
                         state = -1

artiq/compiler/ir.py

@@ -1047,6 +1047,42 @@ class Builtin(Instruction):
     def opcode(self):
         return "builtin({})".format(self.op)
 
+class BuiltinInvoke(Terminator):
+    """
+    A builtin operation which can raise exceptions.
+
+    :ivar op: (string) operation name
+    """
+
+    """
+    :param op: (string) operation name
+    :param normal: (:class:`BasicBlock`) normal target
+    :param exn: (:class:`BasicBlock`) exceptional target
+    """
+    def __init__(self, op, operands, typ, normal, exn, name=None):
+        assert isinstance(op, str)
+        for operand in operands: assert isinstance(operand, Value)
+        assert isinstance(normal, BasicBlock)
+        assert isinstance(exn, BasicBlock)
+        if name is None:
+            name = "BLTINV.{}".format(op)
+        super().__init__(operands + [normal, exn], typ, name)
+        self.op = op
+
+    def copy(self, mapper):
+        self_copy = super().copy(mapper)
+        self_copy.op = self.op
+        return self_copy
+
+    def normal_target(self):
+        return self.operands[-2]
+
+    def exception_target(self):
+        return self.operands[-1]
+
+    def opcode(self):
+        return "builtinInvokable({})".format(self.op)
+
 class Closure(Instruction):
     """
     A closure creation operation.

artiq/compiler/math_fns.py

@@ -61,22 +61,6 @@ unary_fp_runtime_calls = [
     ("cbrt", "cbrt"),
 ]
 
-#: float -> float numpy.* math functions lowered to runtime calls.
-unary_fp_runtime_calls = [
-    ("tan", "tan"),
-    ("arcsin", "asin"),
-    ("arccos", "acos"),
-    ("arctan", "atan"),
-    ("sinh", "sinh"),
-    ("cosh", "cosh"),
-    ("tanh", "tanh"),
-    ("arcsinh", "asinh"),
-    ("arccosh", "acosh"),
-    ("arctanh", "atanh"),
-    ("expm1", "expm1"),
-    ("cbrt", "cbrt"),
-]
-
 scipy_special_unary_runtime_calls = [
     ("erf", "erf"),
     ("erfc", "erfc"),

artiq/compiler/transforms/artiq_ir_generator.py

@@ -65,10 +65,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
     :ivar catch_clauses: (list of (:class:`ir.BasicBlock`, :class:`types.Type` or None))
         a list of catch clauses that should be appended to inner try block
         landingpad
-    :ivar final_branch: (function (target: :class:`ir.BasicBlock`, block: :class:`ir.BasicBlock)
-                         or None)
-        the function that appends to ``block`` a jump through the ``finally`` statement
-        to ``target``
 
     There is, additionally, some global state that is used to translate
     the results of analyses on AST level to IR level:
@@ -114,7 +110,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
         self.return_target = None
         self.unwind_target = None
         self.catch_clauses = []
-        self.final_branch = None
         self.function_map = dict()
         self.variable_map = dict()
         self.method_map = defaultdict(lambda: [])
@@ -635,11 +630,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
         self.append(ir.Branch(self.continue_target))
 
     def raise_exn(self, exn=None, loc=None):
-        if self.final_branch is not None:
-            raise_proxy = self.add_block("try.raise")
-            self.final_branch(raise_proxy, self.current_block)
-            self.current_block = raise_proxy
-
         if exn is not None:
             # if we need to raise the exception in a final body, we have to
             # lazy-evaluate the exception object to make sure that we generate
@@ -713,7 +703,7 @@ class ARTIQIRGenerator(algorithm.Visitor):
                 return_action.append(ir.Return(value))
                 final_branch(return_action, return_proxy)
         else:
-            landingpad.has_cleanup = False
+            landingpad.has_cleanup = self.unwind_target is not None
 
         # we should propagate the clauses to nested try catch blocks
         # so nested try catch will jump to our clause if the inner one does not
@@ -777,7 +767,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
                 self.continue_target = old_continue
             self.return_target = old_return
 
-        if any(node.finalbody):
             # create new unwind target for cleanup
             final_dispatcher = self.add_block("try.final.dispatch")
             final_landingpad = ir.LandingPad(cleanup)
@@ -786,7 +775,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
             # make sure that exception clauses are unwinded to the finally block
             old_unwind, self.unwind_target = self.unwind_target, final_dispatcher
 
-        if any(node.finalbody):
             # if we have a while:try/finally continue must execute finally
             # before continuing the while
             redirect = final_branch
@@ -1114,13 +1102,11 @@ class ARTIQIRGenerator(algorithm.Visitor):
             entry = self.add_block("entry")
             old_block, self.current_block = self.current_block, entry
 
-            old_final_branch, self.final_branch = self.final_branch, None
             old_unwind, self.unwind_target = self.unwind_target, None
             self.raise_exn(lambda: exn_gen(*args[1:]), loc=loc)
         finally:
             self.current_function = old_func
             self.current_block = old_block
-            self.final_branch = old_final_branch
             self.unwind_target = old_unwind
 
         # cond:    bool Value, condition
@@ -1539,7 +1525,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
             entry = self.add_block("entry")
             old_block, self.current_block = self.current_block, entry
 
-            old_final_branch, self.final_branch = self.final_branch, None
             old_unwind, self.unwind_target = self.unwind_target, None
 
             shape = self.append(ir.GetAttr(arg, "shape"))
@@ -1565,7 +1550,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
             self.current_loc = old_loc
             self.current_function = old_func
             self.current_block = old_block
-            self.final_branch = old_final_branch
             self.unwind_target = old_unwind
 
     def _get_array_unaryop(self, name, make_op, result_type, arg_type):
@@ -1666,7 +1650,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
             entry = self.add_block("entry")
             old_block, self.current_block = self.current_block, entry
 
-            old_final_branch, self.final_branch = self.final_branch, None
             old_unwind, self.unwind_target = self.unwind_target, None
 
             body_gen(result, lhs, rhs)
@@ -1677,7 +1660,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
             self.current_loc = old_loc
             self.current_function = old_func
             self.current_block = old_block
-            self.final_branch = old_final_branch
             self.unwind_target = old_unwind
 
     def _make_array_elementwise_binop(self, name, result_type, lhs_type,
@@ -2544,10 +2526,22 @@ class ARTIQIRGenerator(algorithm.Visitor):
                 fn = types.get_method_function(fn)
             sid = ir.Constant(fn.sid, builtins.TInt32())
             if not builtins.is_none(fn.ret):
+                if self.unwind_target is None:
                     ret = self.append(ir.Builtin("subkernel_retrieve_return", [sid, timeout], fn.ret))
+                else:
+                    after_invoke = self.add_block("invoke")
+                    ret = self.append(ir.BuiltinInvoke("subkernel_retrieve_return", [sid, timeout], 
+                                                       fn.ret, after_invoke, self.unwind_target))
+                    self.current_block = after_invoke
             else:
                 ret = ir.Constant(None, builtins.TNone())
+            if self.unwind_target is None:
                 self.append(ir.Builtin("subkernel_await_finish", [sid, timeout], builtins.TNone()))
+            else:
+                after_invoke = self.add_block("invoke")
+                self.append(ir.BuiltinInvoke("subkernel_await_finish", [sid, timeout], 
+                                                builtins.TNone(), after_invoke, self.unwind_target))
+                self.current_block = after_invoke
             return ret
         elif types.is_builtin(typ, "subkernel_preload"):
             if len(node.args) == 1 and len(node.keywords) == 0:
@@ -2594,7 +2588,14 @@ class ARTIQIRGenerator(algorithm.Visitor):
                     {"name": name, "recv": vartype, "send": msg.value_type},
                     node.loc)
                 self.engine.process(diag)
-            return self.append(ir.Builtin("subkernel_recv", [msg_id, timeout], vartype))
+            if self.unwind_target is None:
+                ret = self.append(ir.Builtin("subkernel_recv", [msg_id, timeout], vartype))
+            else:
+                after_invoke = self.add_block("invoke")
+                ret = self.append(ir.BuiltinInvoke("subkernel_recv", [msg_id, timeout], 
+                                                   vartype, after_invoke, self.unwind_target))
+                self.current_block = after_invoke
+            return ret
         elif types.is_exn_constructor(typ):
             return self.alloc_exn(node.type, *[self.visit(arg_node) for arg_node in node.args])
         elif types.is_constructor(typ):
@@ -2801,7 +2802,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
 
             entry = self.add_block("entry")
             old_block, self.current_block = self.current_block, entry
-            old_final_branch, self.final_branch = self.final_branch, None
             old_unwind, self.unwind_target = self.unwind_target, None
 
             exn = self.alloc_exn(builtins.TException("AssertionError"),
@@ -2814,7 +2814,6 @@ class ARTIQIRGenerator(algorithm.Visitor):
         finally:
             self.current_function = old_func
             self.current_block = old_block
-            self.final_branch = old_final_branch
             self.unwind_target = old_unwind
 
         self.raise_assert_func = func
@@ -2946,7 +2945,7 @@ class ARTIQIRGenerator(algorithm.Visitor):
                 format_string += ")"
             elif builtins.is_exception(value.type):
                 # message may not be an actual string...
-                # so we cannot really print it
+                # so we cannot really print itInvoke
                 name    = self.append(ir.GetAttr(value, "#__name__"))
                 param1  = self.append(ir.GetAttr(value, "#__param0__"))
                 param2  = self.append(ir.GetAttr(value, "#__param1__"))

artiq/compiler/transforms/int_monomorphizer.py

@@ -14,9 +14,9 @@ class IntMonomorphizer(algorithm.Visitor):
     def visit_NumT(self, node):
         if builtins.is_int(node.type):
             if types.is_var(node.type["width"]):
-                if -2**31 < node.n < 2**31-1:
+                if -2**31 <= node.n <= 2**31-1:
                     width = 32
-                elif -2**63 < node.n < 2**63-1:
+                elif -2**63 <= node.n <= 2**63-1:
                     width = 64
                 else:
                     diag = diagnostic.Diagnostic("error",

artiq/compiler/transforms/llvm_ir_generator.py

@@ -1437,6 +1437,44 @@ class LLVMIRGenerator:
         else:
             assert False
 
+    def process_BuiltinInvoke(self, insn):
+        llnormalblock = self.map(insn.normal_target())
+        llunwindblock = self.map(insn.exception_target())
+        if insn.op == "subkernel_retrieve_return":
+            llsid = self.map(insn.operands[0])
+            lltimeout = self.map(insn.operands[1])
+            lltagptr = self._build_subkernel_tags([insn.type])
+            llheadu = self.llbuilder.append_basic_block(name="subkernel.await.unwind")
+            self.llbuilder.invoke(self.llbuiltin("subkernel_await_message"), 
+                                  [llsid, lltimeout, lltagptr, ll.Constant(lli8, 1), ll.Constant(lli8, 1)],
+                                  llheadu, llunwindblock,
+                                  name="subkernel.await.message")
+            self.llbuilder.position_at_end(llheadu)
+            llstackptr = self.llbuilder.call(self.llbuiltin("llvm.stacksave"), [],
+                                             name="subkernel.arg.stack")
+            return self._build_rpc_recv(insn.type, llstackptr, llnormalblock, llunwindblock)
+        elif insn.op == "subkernel_await_finish":
+            llsid = self.map(insn.operands[0])
+            lltimeout = self.map(insn.operands[1])
+            return self.llbuilder.invoke(self.llbuiltin("subkernel_await_finish"), [llsid, lltimeout],
+                                         llnormalblock, llunwindblock,
+                                         name="subkernel.await.finish")
+        elif insn.op == "subkernel_recv":
+            llmsgid = self.map(insn.operands[0])
+            lltimeout = self.map(insn.operands[1])
+            lltagptr = self._build_subkernel_tags([insn.type])
+            llheadu = self.llbuilder.append_basic_block(name="subkernel.await.unwind")
+            self.llbuilder.invoke(self.llbuiltin("subkernel_await_message"), 
+                                  [llmsgid, lltimeout, lltagptr, ll.Constant(lli8, 1), ll.Constant(lli8, 1)],
+                                  llheadu, llunwindblock,
+                                  name="subkernel.await.message")
+            self.llbuilder.position_at_end(llheadu)
+            llstackptr = self.llbuilder.call(self.llbuiltin("llvm.stacksave"), [],
+                                             name="subkernel.arg.stack")
+            return self._build_rpc_recv(insn.type, llstackptr, llnormalblock, llunwindblock)
+        else:
+            assert False
+
     def process_SubkernelAwaitArgs(self, insn):
         llmin = self.map(insn.operands[0])
         llmax = self.map(insn.operands[1])

artiq/coredevice/ad53xx.py

@@ -1,8 +1,8 @@
-""""RTIO driver for the Analog Devices AD53[67][0123] family of multi-channel
+"""RTIO driver for the Analog Devices AD53[67][0123] family of multi-channel
 Digital to Analog Converters.
 
 Output event replacement is not supported and issuing commands at the same
-time is an error.
+time results in a collision error.
 """
 
 # Designed from the data sheets and somewhat after the linux kernel
@@ -131,10 +131,10 @@ class AD53xx:
       optimized for speed; datasheet says t22: 25ns min SCLK edge to SDO
       valid, and suggests the SPI speed for reads should be <=20 MHz)
     :param vref: DAC reference voltage (default: 5.)
-    :param offset_dacs: Initial register value for the two offset DACs, device
+    :param offset_dacs: Initial register value for the two offset DACs 
-      dependent and must be set correctly for correct voltage to mu
+      (default: 8192). Device dependent and must be set correctly for 
-      conversions. Knowledge of his state is not transferred between
+      correct voltage-to-mu conversions. Knowledge of this state is 
-      experiments. (default: 8192)
+      not transferred between experiments.
     :param core_device: Core device name (default: "core")
     """
     kernel_invariants = {"bus", "ldac", "clr", "chip_select", "div_write",
@@ -202,7 +202,7 @@ class AD53xx:
         :param op: Operation to perform, one of :const:`AD53XX_READ_X1A`,
           :const:`AD53XX_READ_X1B`, :const:`AD53XX_READ_OFFSET`,
           :const:`AD53XX_READ_GAIN` etc. (default: :const:`AD53XX_READ_X1A`).
-        :return: The 16 bit register value
+        :return: The 16-bit register value
         """
         self.bus.write(ad53xx_cmd_read_ch(channel, op) << 8)
         self.bus.set_config_mu(SPI_AD53XX_CONFIG | spi.SPI_INPUT, 24,
@@ -309,7 +309,7 @@ class AD53xx:
 
         This method does not advance the timeline; write events are scheduled
         in the past. The DACs will synchronously start changing their output
-        levels `now`.
+        levels ``now``.
 
         If no LDAC device was defined, the LDAC pulse is skipped.
 
@@ -364,8 +364,8 @@ class AD53xx:
         high) can be calibrated in this fashion.
 
         :param channel: The number of the calibrated channel
-        :params vzs: Measured voltage with the DAC set to zero-scale (0x0000)
+        :param vzs: Measured voltage with the DAC set to zero-scale (0x0000)
-        :params vfs: Measured voltage with the DAC set to full-scale (0xffff)
+        :param vfs: Measured voltage with the DAC set to full-scale (0xffff)
         """
         offset_err = voltage_to_mu(vzs, self.offset_dacs, self.vref)
         gain_err = voltage_to_mu(vfs, self.offset_dacs, self.vref) - (

artiq/coredevice/ad9910.py

@@ -114,27 +114,27 @@ class AD9910:
         (as configured through CFG_MASK_NU), 4-7 for individual channels.
     :param cpld_device: Name of the Urukul CPLD this device is on.
     :param sw_device: Name of the RF switch device. The RF switch is a
-        TTLOut channel available as the :attr:`sw` attribute of this instance.
+        TTLOut channel available as the ``sw`` attribute of this instance.
     :param pll_n: DDS PLL multiplier. The DDS sample clock is
-        f_ref/clk_div*pll_n where f_ref is the reference frequency and
+        ``f_ref / clk_div * pll_n`` where ``f_ref`` is the reference frequency and
-        clk_div is the reference clock divider (both set in the parent
+        ``clk_div`` is the reference clock divider (both set in the parent
         Urukul CPLD instance).
     :param pll_en: PLL enable bit, set to 0 to bypass PLL (default: 1).
         Note that when bypassing the PLL the red front panel LED may remain on.
     :param pll_cp: DDS PLL charge pump setting.
     :param pll_vco: DDS PLL VCO range selection.
-    :param sync_delay_seed: SYNC_IN delay tuning starting value.
+    :param sync_delay_seed: ``SYNC_IN`` delay tuning starting value.
-        To stabilize the SYNC_IN delay tuning, run :meth:`tune_sync_delay` once
+        To stabilize the ``SYNC_IN`` delay tuning, run :meth:`tune_sync_delay` once
         and set this to the delay tap number returned (default: -1 to signal no
         synchronization and no tuning during :meth:`init`).
-        Can be a string of the form "eeprom_device:byte_offset" to read the
+        Can be a string of the form ``eeprom_device:byte_offset`` to read the
-        value from a I2C EEPROM; in which case, `io_update_delay` must be set
+        value from a I2C EEPROM, in which case ``io_update_delay`` must be set
         to the same string value.
-    :param io_update_delay: IO_UPDATE pulse alignment delay.
+    :param io_update_delay: ``IO_UPDATE`` pulse alignment delay.
-        To align IO_UPDATE to SYNC_CLK, run :meth:`tune_io_update_delay` and
+        To align ``IO_UPDATE`` to ``SYNC_CLK``, run :meth:`tune_io_update_delay` and
         set this to the delay tap number returned.
-        Can be a string of the form "eeprom_device:byte_offset" to read the
+        Can be a string of the form ``eeprom_device:byte_offset`` to read the
-        value from a I2C EEPROM; in which case, `sync_delay_seed` must be set
+        value from a I2C EEPROM, in which case ``sync_delay_seed`` must be set
         to the same string value.
     """
 
@@ -188,16 +188,14 @@ class AD9910:
 
     @kernel
     def set_phase_mode(self, phase_mode: TInt32):
-        r"""Set the default phase mode.
+        r"""Set the default phase mode for future calls to :meth:`set` and
-
-        for future calls to :meth:`set` and
         :meth:`set_mu`. Supported phase modes are:
 
         * :const:`PHASE_MODE_CONTINUOUS`: the phase accumulator is unchanged
           when changing frequency or phase. The DDS phase is the sum of the
           phase accumulator and the phase offset. The only discontinuous
           changes in the DDS output phase come from changes to the phase
-          offset. This mode is also knows as "relative phase mode".
+          offset. This mode is also known as "relative phase mode".
           :math:`\phi(t) = q(t^\prime) + p + (t - t^\prime) f`
 
         * :const:`PHASE_MODE_ABSOLUTE`: the phase accumulator is reset when
@@ -233,7 +231,7 @@ class AD9910:
 
     @kernel
     def write16(self, addr: TInt32, data: TInt32):
-        """Write to 16 bit register.
+        """Write to 16-bit register.
 
         :param addr: Register address
         :param data: Data to be written
@@ -244,7 +242,7 @@ class AD9910:
 
     @kernel
     def write32(self, addr: TInt32, data: TInt32):
-        """Write to 32 bit register.
+        """Write to 32-bit register.
 
         :param addr: Register address
         :param data: Data to be written
@@ -258,7 +256,7 @@ class AD9910:
 
     @kernel
     def read16(self, addr: TInt32) -> TInt32:
-        """Read from 16 bit register.
+        """Read from 16-bit register.
 
         :param addr: Register address
         """
@@ -273,7 +271,7 @@ class AD9910:
 
     @kernel
     def read32(self, addr: TInt32) -> TInt32:
-        """Read from 32 bit register.
+        """Read from 32-bit register.
 
         :param addr: Register address
         """
@@ -288,10 +286,10 @@ class AD9910:
 
     @kernel
     def read64(self, addr: TInt32) -> TInt64:
-        """Read from 64 bit register.
+        """Read from 64-bit register.
 
         :param addr: Register address
-        :return: 64 bit integer register value
+        :return: 64-bit integer register value
         """
         self.bus.set_config_mu(
             urukul.SPI_CONFIG, 8,
@@ -311,10 +309,10 @@ class AD9910:
 
     @kernel
     def write64(self, addr: TInt32, data_high: TInt32, data_low: TInt32):
-        """Write to 64 bit register.
+        """Write to 64-bit register.
 
         :param addr: Register address
-        :param data_high: High (MSB) 32 bits of the data
+        :param data_high: High (MSB) 32 data bits 
         :param data_low: Low (LSB) 32 data bits
         """
         self.bus.set_config_mu(urukul.SPI_CONFIG, 8,
@@ -332,8 +330,9 @@ class AD9910:
         """Write data to RAM.
 
         The profile to write to and the step, start, and end address
-        need to be configured before and separately using
+        need to be configured in advance and separately using
-        :meth:`set_profile_ram` and the parent CPLD `set_profile`.
+        :meth:`set_profile_ram` and the parent CPLD
+        :meth:`~artiq.coredevice.urukul.CPLD.set_profile`.
 
         :param data: Data to be written to RAM.
         """
@@ -354,7 +353,8 @@ class AD9910:
 
         The profile to read from and the step, start, and end address
         need to be configured before and separately using
-        :meth:`set_profile_ram` and the parent CPLD `set_profile`.
+        :meth:`set_profile_ram` and the parent CPLD 
+        :meth:`~artiq.coredevice.urukul.CPLD.set_profile`.
 
         :param data: List to be filled with data read from RAM.
         """
@@ -390,9 +390,9 @@ class AD9910:
                  manual_osk_external: TInt32 = 0,
                  osk_enable: TInt32 = 0,
                  select_auto_osk: TInt32 = 0):
-        """Set CFR1. See the AD9910 datasheet for parameter meanings.
+        """Set CFR1. See the AD9910 datasheet for parameter meanings and sizes.
 
-        This method does not pulse IO_UPDATE.
+        This method does not pulse ``IO_UPDATE.``
 
         :param power_down: Power down bits.
         :param phase_autoclear: Autoclear phase accumulator.
@@ -429,9 +429,9 @@ class AD9910:
                  effective_ftw: TInt32 = 1,
                  sync_validation_disable: TInt32 = 0, 
                  matched_latency_enable: TInt32 = 0):
-        """Set CFR2. See the AD9910 datasheet for parameter meanings.
+        """Set CFR2. See the AD9910 datasheet for parameter meanings and sizes.
 
-        This method does not pulse IO_UPDATE.
+        This method does not pulse ``IO_UPDATE``.
 
         :param asf_profile_enable: Enable amplitude scale from single tone profiles.
         :param drg_enable: Digital ramp enable.
@@ -456,14 +456,14 @@ class AD9910:
         """Initialize and configure the DDS.
 
         Sets up SPI mode, confirms chip presence, powers down unused blocks,
-        configures the PLL, waits for PLL lock. Uses the
+        configures the PLL, waits for PLL lock. Uses the ``IO_UPDATE`` 
-        IO_UPDATE signal multiple times.
+        signal multiple times.
 
         :param blind: Do not read back DDS identity and do not wait for lock.
         """
         self.sync_data.init()
         if self.sync_data.sync_delay_seed >= 0 and not self.cpld.sync_div:
-            raise ValueError("parent cpld does not drive SYNC")
+            raise ValueError("parent CPLD does not drive SYNC")
         if self.sync_data.sync_delay_seed >= 0:
             if self.sysclk_per_mu != self.sysclk * self.core.ref_period:
                 raise ValueError("incorrect clock ratio for synchronization")
@@ -514,7 +514,7 @@ class AD9910:
     def power_down(self, bits: TInt32 = 0b1111):
         """Power down DDS.
 
-        :param bits: Power down bits, see datasheet
+        :param bits: Power-down bits, see datasheet
         """
         self.set_cfr1(power_down=bits)
         self.cpld.io_update.pulse(1 * us)
@@ -534,23 +534,31 @@ class AD9910:
         After the SPI transfer, the shared IO update pin is pulsed to
         activate the data.
 
-        .. seealso: :meth:`set_phase_mode` for a definition of the different
+        .. seealso:: :meth:`AD9910.set_phase_mode` for a definition of the different
             phase modes.
 
-        :param ftw: Frequency tuning word: 32 bit.
+        .. warning::
-        :param pow_: Phase tuning word: 16 bit unsigned.
+            Deterministic phase control depends on correct alignment of operations
-        :param asf: Amplitude scale factor: 14 bit unsigned.
+            to a 4ns grid (``SYNC_CLK``). This function uses :meth:`~artiq.language.core.now_mu()`
+            to ensure such alignment automatically. When replayed over DMA, however, the ensuing
+            event sequence *must* be started at the same offset relative to ``SYNC_CLK``, or
+            unstable ``SYNC_CLK`` cycle assignment (i.e. inconsistent delays of exactly 4ns) will
+            result.
+
+        :param ftw: Frequency tuning word: 32-bit.
+        :param pow_: Phase tuning word: 16-bit unsigned.
+        :param asf: Amplitude scale factor: 14-bit unsigned.
         :param phase_mode: If specified, overrides the default phase mode set
             by :meth:`set_phase_mode` for this call.
         :param ref_time_mu: Fiducial time used to compute absolute or tracking
-            phase updates. In machine units as obtained by `now_mu()`.
+            phase updates. In machine units as obtained by :meth:`~artiq.language.core.now_mu()`.
         :param profile: Single tone profile number to set (0-7, default: 7).
-            Ineffective if `ram_destination` is specified.
+            Ineffective if ``ram_destination`` is specified.
         :param ram_destination: RAM destination (:const:`RAM_DEST_FTW`,
             :const:`RAM_DEST_POW`, :const:`RAM_DEST_ASF`,
             :const:`RAM_DEST_POWASF`). If specified, write free DDS parameters
             to the ASF/FTW/POW registers instead of to the single tone profile
-            register (default behaviour, see `profile`).
+            register (default behaviour, see ``profile``).
         :return: Resulting phase offset word after application of phase
             tracking offset. When using :const:`PHASE_MODE_CONTINUOUS` in
             subsequent calls, use this value as the "current" phase.
@@ -598,10 +606,10 @@ class AD9910:
         """Get the frequency tuning word, phase offset word,
         and amplitude scale factor.
 
-        .. seealso:: :meth:`get`
+        See also :meth:`AD9910.get`.
 
         :param profile: Profile number to get (0-7, default: 7)
-        :return: A tuple ``(ftw, pow, asf)``
+        :return: A tuple (FTW, POW, ASF)
         """
 
         # Read data
@@ -617,12 +625,12 @@ class AD9910:
                         profile: TInt32 = _DEFAULT_PROFILE_RAM,
                         nodwell_high: TInt32 = 0, zero_crossing: TInt32 = 0,
                         mode: TInt32 = 1):
-        """Set the RAM profile settings.
+        """Set the RAM profile settings. See also AD9910 datasheet.
 
-        :param start: Profile start address in RAM.
+        :param start: Profile start address in RAM (10-bit).
-        :param end: Profile end address in RAM (last address).
+        :param end: Profile end address in RAM, inclusive (10-bit).
-        :param step: Profile time step in units of t_DDS, typically 4 ns
+        :param step: Profile time step, counted in DDS sample clock
-            (default: 1).
+            cycles, typically 4 ns (16-bit, default: 1)
         :param profile: Profile index (0 to 7) (default: 0).
         :param nodwell_high: No-dwell high bit (default: 0,
             see AD9910 documentation).
@@ -850,7 +858,7 @@ class AD9910:
             ram_destination: TInt32 = -1) -> TFloat:
         """Set DDS data in SI units.
 
-        .. seealso:: :meth:`set_mu`
+        See also :meth:`AD9910.set_mu`.
 
         :param frequency: Frequency in Hz
         :param phase: Phase tuning word in turns
@@ -871,10 +879,10 @@ class AD9910:
             ) -> TTuple([TFloat, TFloat, TFloat]):
         """Get the frequency, phase, and amplitude.
 
-        .. seealso:: :meth:`get_mu`
+        See also :meth:`AD9910.get_mu`.
 
         :param profile: Profile number to get (0-7, default: 7)
-        :return: A tuple ``(frequency, phase, amplitude)``
+        :return: A tuple (frequency, phase, amplitude)
         """
 
         # Get values
@@ -887,11 +895,10 @@ class AD9910:
     def set_att_mu(self, att: TInt32):
         """Set digital step attenuator in machine units.
 
-        This method will write the attenuator settings of all four channels.
+        This method will write the attenuator settings of all four channels. See also
+        :meth:`CPLD.get_channel_att <artiq.coredevice.urukul.CPLD.set_att_mu>`.
 
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.set_att_mu`
+        :param att: Attenuation setting, 8-bit digital.
-
-        :param att: Attenuation setting, 8 bit digital.
         """
         self.cpld.set_att_mu(self.chip_select - 4, att)
 
@@ -899,9 +906,8 @@ class AD9910:
     def set_att(self, att: TFloat):
         """Set digital step attenuator in SI units.
 
-        This method will write the attenuator settings of all four channels.
+        This method will write the attenuator settings of all four channels. See also 
-
+        :meth:`CPLD.get_channel_att <artiq.coredevice.urukul.CPLD.set_att>`.
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.set_att`
 
         :param att: Attenuation in dB.
         """
@@ -909,19 +915,17 @@ class AD9910:
 
     @kernel
     def get_att_mu(self) -> TInt32:
-        """Get digital step attenuator value in machine units.
+        """Get digital step attenuator value in machine units. See also
+        :meth:`CPLD.get_channel_att <artiq.coredevice.urukul.CPLD.get_channel_att_mu>`.
 
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.get_channel_att_mu`
+        :return: Attenuation setting, 8-bit digital.
-
-        :return: Attenuation setting, 8 bit digital.
         """
         return self.cpld.get_channel_att_mu(self.chip_select - 4)
 
     @kernel
     def get_att(self) -> TFloat:
-        """Get digital step attenuator value in SI units.
+        """Get digital step attenuator value in SI units. See also 
-
+        :meth:`CPLD.get_channel_att <artiq.coredevice.urukul.CPLD.get_channel_att>`.
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.get_channel_att`
 
         :return: Attenuation in dB.
         """
@@ -943,16 +947,16 @@ class AD9910:
                  window: TInt32, 
                  en_sync_gen: TInt32 = 0):
         """Set the relevant parameters in the multi device synchronization
-        register. See the AD9910 datasheet for details. The SYNC clock
+        register. See the AD9910 datasheet for details. The ``SYNC`` clock
-        generator preset value is set to zero, and the SYNC_OUT generator is
+        generator preset value is set to zero, and the ``SYNC_OUT`` generator is
         disabled by default.
 
-        :param in_delay: SYNC_IN delay tap (0-31) in steps of ~75ps
+        :param in_delay: ``SYNC_IN`` delay tap (0-31) in steps of ~75ps
-        :param window: Symmetric SYNC_IN validation window (0-15) in
+        :param window: Symmetric ``SYNC_IN`` validation window (0-15) in
             steps of ~75ps for both hold and setup margin.
         :param en_sync_gen: Whether to enable the DDS-internal sync generator
-            (SYNC_OUT, cf. sync_sel == 1). Should be left off for the normal
+            (``SYNC_OUT``, cf. ``sync_sel == 1``). Should be left off for the normal
-            use case, where the SYNC clock is supplied by the core device.
+            use case, where the ``SYNC`` clock is supplied by the core device.
         """
         self.write32(_AD9910_REG_SYNC,
                      (window << 28) |  # SYNC S/H validation delay
@@ -965,9 +969,9 @@ class AD9910:
 
     @kernel
     def clear_smp_err(self):
-        """Clear the SMP_ERR flag and enables SMP_ERR validity monitoring.
+        """Clear the ``SMP_ERR`` flag and enables ``SMP_ERR`` validity monitoring.
 
-        Violations of the SYNC_IN sample and hold margins will result in
+        Violations of the ``SYNC_IN`` sample and hold margins will result in
         SMP_ERR being asserted. This then also activates the red LED on
         the respective Urukul channel.
 
@@ -982,9 +986,9 @@ class AD9910:
     @kernel
     def tune_sync_delay(self,
                         search_seed: TInt32 = 15) -> TTuple([TInt32, TInt32]):
-        """Find a stable SYNC_IN delay.
+        """Find a stable ``SYNC_IN`` delay.
 
-        This method first locates a valid SYNC_IN delay at zero validation
+        This method first locates a valid ``SYNC_IN`` delay at zero validation
         window size (setup/hold margin) by scanning around `search_seed`. It
         then looks for similar valid delays at successively larger validation
         window sizes until none can be found. It then decreases the validation
@@ -993,7 +997,7 @@ class AD9910:
 
         This method and :meth:`tune_io_update_delay` can be run in any order.
 
-        :param search_seed: Start value for valid SYNC_IN delay search.
+        :param search_seed: Start value for valid ``SYNC_IN`` delay search.
             Defaults to 15 (half range).
         :return: Tuple of optimal delay and window size.
         """
@@ -1040,16 +1044,16 @@ class AD9910:
     def measure_io_update_alignment(self, delay_start: TInt64,
                                     delay_stop: TInt64) -> TInt32:
         """Use the digital ramp generator to locate the alignment between
-        IO_UPDATE and SYNC_CLK.
+        ``IO_UPDATE`` and ``SYNC_CLK``.
 
         The ramp generator is set up to a linear frequency ramp
-        (dFTW/t_SYNC_CLK=1) and started at a coarse RTIO time stamp plus
+        ``(dFTW/t_SYNC_CLK=1)`` and started at a coarse RTIO time stamp plus
-        `delay_start` and stopped at a coarse RTIO time stamp plus
+        ``delay_start`` and stopped at a coarse RTIO time stamp plus
-        `delay_stop`.
+        ``delay_stop``.
 
-        :param delay_start: Start IO_UPDATE delay in machine units.
+        :param delay_start: Start ``IO_UPDATE`` delay in machine units.
-        :param delay_stop: Stop IO_UPDATE delay in machine units.
+        :param delay_stop: Stop ``IO_UPDATE`` delay in machine units.
-        :return: Odd/even SYNC_CLK cycle indicator.
+        :return: Odd/even ``SYNC_CLK`` cycle indicator.
         """
         # set up DRG
         self.set_cfr1(drg_load_lrr=1, drg_autoclear=1)
@@ -1081,19 +1085,19 @@ class AD9910:
 
     @kernel
     def tune_io_update_delay(self) -> TInt32:
-        """Find a stable IO_UPDATE delay alignment.
+        """Find a stable ``IO_UPDATE`` delay alignment.
 
-        Scan through increasing IO_UPDATE delays until a delay is found that
+        Scan through increasing ``IO_UPDATE`` delays until a delay is found that
-        lets IO_UPDATE be registered in the next SYNC_CLK cycle. Return a
+        lets ``IO_UPDATE`` be registered in the next ``SYNC_CLK`` cycle. Return a
-        IO_UPDATE delay that is as far away from that SYNC_CLK edge
+        ``IO_UPDATE`` delay that is as far away from that ``SYNC_CLK`` edge
         as possible.
 
-        This method assumes that the IO_UPDATE TTLOut device has one machine
+        This method assumes that the ``IO_UPDATE`` TTLOut device has one machine
         unit resolution (SERDES).
 
         This method and :meth:`tune_sync_delay` can be run in any order.
 
-        :return: Stable IO_UPDATE delay to be passed to the constructor
+        :return: Stable ``IO_UPDATE`` delay to be passed to the constructor
             :class:`AD9910` via the device database.
         """
         period = self.sysclk_per_mu * 4  # SYNC_CLK period

artiq/coredevice/ad9912.py

@@ -11,7 +11,7 @@ from artiq.coredevice import urukul
 
 class AD9912:
     """
-    AD9912 DDS channel on Urukul
+    AD9912 DDS channel on Urukul.
 
     This class supports a single DDS channel and exposes the DDS,
     the digital step attenuator, and the RF switch.
@@ -22,9 +22,9 @@ class AD9912:
     :param sw_device: Name of the RF switch device. The RF switch is a
         TTLOut channel available as the :attr:`sw` attribute of this instance.
     :param pll_n: DDS PLL multiplier. The DDS sample clock is
-        f_ref/clk_div*pll_n where f_ref is the reference frequency and clk_div
+        ``f_ref / clk_div * pll_n`` where ``f_ref`` is the reference frequency and 
-        is the reference clock divider (both set in the parent Urukul CPLD
+        ``clk_div`` is the reference clock divider (both set in the parent 
-        instance).
+        Urukul CPLD instance).
     :param pll_en: PLL enable bit, set to 0 to bypass PLL (default: 1).
         Note that when bypassing the PLL the red front panel LED may remain on.
     """
@@ -44,7 +44,11 @@ class AD9912:
         self.pll_en = pll_en
         self.pll_n = pll_n
         if pll_en:
-            sysclk = self.cpld.refclk / [1, 1, 2, 4][self.cpld.clk_div] * pll_n
+            refclk = self.cpld.refclk
+            if refclk < 11e6:
+                # use SYSCLK PLL Doubler
+                refclk = refclk * 2
+            sysclk = refclk / [1, 1, 2, 4][self.cpld.clk_div] * pll_n
         else:
             sysclk = self.cpld.refclk
         assert sysclk <= 1e9
@@ -97,7 +101,7 @@ class AD9912:
 
         Sets up SPI mode, confirms chip presence, powers down unused blocks,
         and configures the PLL. Does not wait for PLL lock. Uses the
-        IO_UPDATE signal multiple times.
+        ``IO_UPDATE`` signal multiple times.
         """
         # SPI mode
         self.write(AD9912_SER_CONF, 0x99, length=1)
@@ -115,6 +119,10 @@ class AD9912:
             self.write(AD9912_N_DIV, self.pll_n // 2 - 2, length=1)
             self.cpld.io_update.pulse(2 * us)
             # I_cp = 375 µA, VCO high range
+            if self.cpld.refclk < 11e6:
+                # enable SYSCLK PLL Doubler
+                self.write(AD9912_PLLCFG, 0b00001101, length=1)
+            else:
                 self.write(AD9912_PLLCFG, 0b00000101, length=1)
             self.cpld.io_update.pulse(2 * us)
         delay(1 * ms)
@@ -125,9 +133,9 @@ class AD9912:
 
         This method will write the attenuator settings of all four channels.
 
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.set_att_mu`
+        See also :meth:`~artiq.coredevice.urukul.CPLD.set_att_mu`.
 
-        :param att: Attenuation setting, 8 bit digital.
+        :param att: Attenuation setting, 8-bit digital.
         """
         self.cpld.set_att_mu(self.chip_select - 4, att)
 
@@ -137,7 +145,7 @@ class AD9912:
 
         This method will write the attenuator settings of all four channels.
 
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.set_att`
+        See also :meth:`~artiq.coredevice.urukul.CPLD.set_att`.
 
         :param att: Attenuation in dB. Higher values mean more attenuation.
         """
@@ -147,9 +155,9 @@ class AD9912:
     def get_att_mu(self) -> TInt32:
         """Get digital step attenuator value in machine units.
 
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.get_channel_att_mu`
+        See also :meth:`~artiq.coredevice.urukul.CPLD.get_channel_att_mu`.
 
-        :return: Attenuation setting, 8 bit digital.
+        :return: Attenuation setting, 8-bit digital.
         """
         return self.cpld.get_channel_att_mu(self.chip_select - 4)
 
@@ -157,7 +165,7 @@ class AD9912:
     def get_att(self) -> TFloat:
         """Get digital step attenuator value in SI units.
 
-        .. seealso:: :meth:`artiq.coredevice.urukul.CPLD.get_channel_att`
+        See also :meth:`~artiq.coredevice.urukul.CPLD.get_channel_att`.
 
         :return: Attenuation in dB.
         """
@@ -170,8 +178,8 @@ class AD9912:
         After the SPI transfer, the shared IO update pin is pulsed to
         activate the data.
 
-        :param ftw: Frequency tuning word: 48 bit unsigned.
+        :param ftw: Frequency tuning word: 48-bit unsigned.
-        :param pow_: Phase tuning word: 16 bit unsigned.
+        :param pow_: Phase tuning word: 16-bit unsigned.
         """
         # streaming transfer of FTW and POW
         self.bus.set_config_mu(urukul.SPI_CONFIG, 16,
@@ -189,9 +197,9 @@ class AD9912:
     def get_mu(self) -> TTuple([TInt64, TInt32]):
         """Get the frequency tuning word and phase offset word.
 
-        .. seealso:: :meth:`get`
+        See also :meth:`AD9912.get`.
 
-        :return: A tuple ``(ftw, pow)``.
+        :return: A tuple (FTW, POW).
         """
 
         # Read data
@@ -239,7 +247,7 @@ class AD9912:
     def set(self, frequency: TFloat, phase: TFloat = 0.0):
         """Set profile 0 data in SI units.
 
-        .. seealso:: :meth:`set_mu`
+        See also :meth:`AD9912.set_mu`.
 
         :param frequency: Frequency in Hz
         :param phase: Phase tuning word in turns
@@ -251,9 +259,9 @@ class AD9912:
     def get(self) -> TTuple([TFloat, TFloat]):
         """Get the frequency and phase.
 
-        .. seealso:: :meth:`get_mu`
+        See also :meth:`AD9912.get_mu`.
 
-        :return: A tuple ``(frequency, phase)``.
+        :return: A tuple (frequency, phase).
         """
 
         # Get values

artiq/coredevice/ad9914.py

@@ -49,7 +49,7 @@ class AD9914:
     The time cursor is not modified by any function in this class.
 
     Output event replacement is not supported and issuing commands at the same
-    time is an error.
+    time results in collision errors.
 
     :param sysclk: DDS system frequency. The DDS system clock must be a
         phase-locked multiple of the RTIO clock.
@@ -134,7 +134,7 @@ class AD9914:
         timing margin.
 
         :param sync_delay: integer from 0 to 0x3f that sets the value of
-            SYNC_OUT (bits 3-5) and SYNC_IN (bits 0-2) delay ADJ bits.
+            ``SYNC_OUT`` (bits 3-5) and ``SYNC_IN`` (bits 0-2) delay ADJ bits.
         """
         delay_mu(-self.init_sync_duration_mu)
         self.write(AD9914_GPIO,      (1 << self.channel) << 1)

artiq/coredevice/adf5356.py

@@ -112,7 +112,7 @@ class ADF5356:
 
         This method will write the attenuator settings of the channel.
 
-        .. seealso:: :meth:`artiq.coredevice.mirny.Mirny.set_att`
+        See also :meth:`Mirny.set_att<artiq.coredevice.mirny.Mirny.set_att>`.
 
         :param att: Attenuation in dB.
         """
@@ -122,7 +122,7 @@ class ADF5356:
     def set_att_mu(self, att):
         """Set digital step attenuator in machine units.
 
-        :param att: Attenuation setting, 8 bit digital.
+        :param att: Attenuation setting, 8-bit digital.
         """
         self.cpld.set_att_mu(self.channel, att)
 
@@ -531,14 +531,14 @@ class ADF5356:
     @portable
     def _compute_pfd_frequency(self, r, d, t) -> TInt64:
         """
-        Calculate the PFD frequency from the given reference path parameters
+        Calculate the PFD frequency from the given reference path parameters.
         """
         return int64(self.sysclk * ((1 + d) / (r * (1 + t))))
 
     @portable
     def _compute_reference_counter(self) -> TInt32:
         """
-        Determine the reference counter R that maximizes the PFD frequency
+        Determine the reference counter R that maximizes the PFD frequency.
         """
         d = ADF5356_REG4_R_DOUBLER_GET(self.regs[4])
         t = ADF5356_REG4_R_DIVIDER_GET(self.regs[4])
@@ -565,14 +565,15 @@ def calculate_pll(f_vco: TInt64, f_pfd: TInt64):
     """
     Calculate fractional-N PLL parameters such that
 
-    ``f_vco`` = ``f_pfd`` * (``n`` + (``frac1`` + ``frac2``/``mod2``) / ``mod1``)
+    ``f_vco = f_pfd * (n + (frac1 + frac2/mod2) / mod1)``
 
     where
+        
     ``mod1 = 2**24`` and ``mod2 <= 2**28``
 
     :param f_vco: target VCO frequency
     :param f_pfd: PFD frequency
-    :return: ``(n, frac1, (frac2_msb, frac2_lsb), (mod2_msb, mod2_lsb))``
+    :return: (``n``, ``frac1``, ``(frac2_msb, frac2_lsb)``, ``(mod2_msb, mod2_lsb)``)
     """
     f_pfd = int64(f_pfd)
     f_vco = int64(f_vco)

artiq/coredevice/almazny.py

@@ -1,4 +1,4 @@
-from artiq.language.core import kernel, portable
+from artiq.language.core import kernel, portable, delay
 from artiq.language.units import us
 
 from numpy import int32
@@ -17,12 +17,12 @@ ALMAZNY_LEGACY_SPIT_WR = 32
 
 class AlmaznyLegacy:
     """
-    Almazny (High frequency mezzanine board for Mirny)
+    Almazny (High-frequency mezzanine board for Mirny)
 
     This applies to Almazny hardware v1.1 and earlier.
-    Use :class:`artiq.coredevice.almazny.AlmaznyChannel` for Almazny v1.2 and later.
+    Use :class:`~artiq.coredevice.almazny.AlmaznyChannel` for Almazny v1.2 and later.
 
-    :param host_mirny: Mirny device Almazny is connected to
+    :param host_mirny: :class:`~artiq.coredevice.mirny.Mirny` device Almazny is connected to
     """
 
     def __init__(self, dmgr, host_mirny):
@@ -121,12 +121,13 @@ class AlmaznyLegacy:
 
 class AlmaznyChannel:
     """
-    One Almazny channel
+    Driver for one Almazny channel.
+
     Almazny is a mezzanine for the Quad PLL RF source Mirny that exposes and
     controls the frequency-doubled outputs.
     This driver requires Almazny hardware revision v1.2 or later
     and Mirny CPLD gateware v0.3 or later.
-    Use :class:`artiq.coredevice.almazny.AlmaznyLegacy` for Almazny hardware v1.1 and earlier.
+    Use :class:`~artiq.coredevice.almazny.AlmaznyLegacy` for Almazny hardware v1.1 and earlier.
 
     :param host_mirny: Mirny CPLD device name
     :param channel: channel index (0-3)

artiq/coredevice/cache.py

@@ -21,9 +21,9 @@ class CoreCache:
         """Extract a value from the core device cache.
         After a value is extracted, it cannot be replaced with another value using
         :meth:`put` until all kernel functions finish executing; attempting
-        to replace it will result in a :class:`artiq.coredevice.exceptions.CacheError`.
+        to replace it will result in a :class:`~artiq.coredevice.exceptions.CacheError`.
 
-        If the cache does not contain any value associated with ``key``, an empty list
+        If the cache does not contain any value associated with `key`, an empty list
         is returned.
 
         The value is not copied, so mutating it will change what's stored in the cache.

artiq/coredevice/comm_analyzer.py

@@ -728,15 +728,7 @@ def decoded_dump_to_target(manager, devices, dump, uniform_interval):
         logger.warning("unable to determine DDS sysclk")
         dds_sysclk = 3e9  # guess
 
-    if isinstance(dump.messages[-1], StoppedMessage):
+    messages = sorted(dump.messages, key=get_message_time)
-        m = dump.messages[-1]
-        end_time = get_message_time(m)
-        manager.set_end_time(end_time)
-        messages = dump.messages[:-1]
-    else:
-        logger.warning("StoppedMessage missing")
-        messages = dump.messages
-    messages = sorted(messages, key=get_message_time)
 
     channel_handlers = create_channel_handlers(
         manager, devices, ref_period,
@@ -752,6 +744,8 @@ def decoded_dump_to_target(manager, devices, dump, uniform_interval):
         interval = manager.get_channel("interval", 64, ty=WaveformType.ANALOG)
     slack = manager.get_channel("rtio_slack", 64, ty=WaveformType.ANALOG)
 
+    stopped_messages = []
+
     manager.set_time(0)
     start_time = 0
     for m in messages:
@@ -762,7 +756,10 @@ def decoded_dump_to_target(manager, devices, dump, uniform_interval):
         manager.set_start_time(start_time)
     t0 = start_time
     for i, message in enumerate(messages):
-        if message.channel in channel_handlers:
+        if isinstance(message, StoppedMessage):
+            stopped_messages.append(message)
+            logger.debug(f"StoppedMessage at {get_message_time(message)}")
+        elif message.channel in channel_handlers:
             t = get_message_time(message)
             if t >= 0:
                 if uniform_interval:
@@ -776,3 +773,9 @@ def decoded_dump_to_target(manager, devices, dump, uniform_interval):
             if isinstance(message, OutputMessage):
                 slack.set_value_double(
                     (message.timestamp - message.rtio_counter)*ref_period)
+
+    if not stopped_messages:
+        logger.warning("StoppedMessage missing")
+    else:
+        end_time = get_message_time(stopped_messages[-1])
+        manager.set_end_time(end_time)

artiq/coredevice/comm_kernel.py

@@ -3,6 +3,7 @@ import logging
 import traceback
 import numpy
 import socket
+import builtins
 from enum import Enum
 from fractions import Fraction
 from collections import namedtuple
@@ -465,12 +466,12 @@ class CommKernel:
             self._write_bool(value)
         elif tag == "i":
             check(isinstance(value, (int, numpy.int32)) and
-                  (-2**31 <= value < 2**31),
+                  (-2**31 <= value <= 2**31-1),
                   lambda: "32-bit int")
             self._write_int32(value)
         elif tag == "I":
             check(isinstance(value, (int, numpy.int32, numpy.int64)) and
-                  (-2**63 <= value < 2**63),
+                  (-2**63 <= value <= 2**63-1),
                   lambda: "64-bit int")
             self._write_int64(value)
         elif tag == "f":
@@ -479,8 +480,8 @@ class CommKernel:
             self._write_float64(value)
         elif tag == "F":
             check(isinstance(value, Fraction) and
-                  (-2**63 <= value.numerator < 2**63) and
+                  (-2**63 <= value.numerator <= 2**63-1) and
-                  (-2**63 <= value.denominator < 2**63),
+                  (-2**63 <= value.denominator <= 2**63-1),
                   lambda: "64-bit Fraction")
             self._write_int64(value.numerator)
             self._write_int64(value.denominator)
@@ -616,9 +617,10 @@ class CommKernel:
                 self._write_int32(embedding_map.store_str(function))
             else:
                 exn_type = type(exn)
-                if exn_type in (ZeroDivisionError, ValueError, IndexError, RuntimeError) or \
+                if exn_type in builtins.__dict__.values():
-                        hasattr(exn, "artiq_builtin"):
+                    name = "0:{}".format(exn_type.__qualname__)
-                    name = "0:{}".format(exn_type.__name__)
+                elif hasattr(exn, "artiq_builtin"):
+                    name = "0:{}.{}".format(exn_type.__module__, exn_type.__qualname__)
                 else:
                     exn_id = embedding_map.store_object(exn_type)
                     name = "{}:{}.{}".format(exn_id,

artiq/coredevice/core.py

@@ -53,6 +53,9 @@ def rtio_get_destination_status(linkno: TInt32) -> TBool:
 def rtio_get_counter() -> TInt64:
     raise NotImplementedError("syscall not simulated")
 
+@syscall
+def test_exception_id_sync(id: TInt32) -> TNone:
+    raise NotImplementedError("syscall not simulated")
 
 def get_target_cls(target):
     if target == "rv32g":
@@ -73,8 +76,8 @@ class Core:
         On platforms that use clock multiplication and SERDES-based PHYs,
         this is the period after multiplication. For example, with a RTIO core
         clocked at 125MHz and a SERDES multiplication factor of 8, the
-        reference period is 1ns.
+        reference period is ``1 ns``.
-        The time machine unit is equal to this period.
+        The machine time unit (``mu``) is equal to this period.
     :param ref_multiplier: ratio between the RTIO fine timestamp frequency
         and the RTIO coarse timestamp frequency (e.g. SERDES multiplication
         factor).
@@ -116,6 +119,8 @@ class Core:
             self.trigger_analyzer_proxy()
 
     def close(self):
+        """Disconnect core device and close sockets. 
+        """
         self.comm.close()
 
     def compile(self, function, args, kwargs, set_result=None,
@@ -241,8 +246,8 @@ class Core:
         Similarly, modified values are not written back, and explicit RPC should be used
         to modify host objects.
         Carefully review the source code of drivers calls used in precompiled kernels, as
-        they may rely on host object attributes being transfered between kernel calls.
+        they may rely on host object attributes being transferred between kernel calls.
-        Examples include code used to control DDS phase, and Urukul RF switch control
+        Examples include code used to control DDS phase and Urukul RF switch control
         via the CPLD register.
 
         The return value of the callable is the return value of the kernel, if any.
@@ -273,7 +278,7 @@ class Core:
     @portable
     def seconds_to_mu(self, seconds):
         """Convert seconds to the corresponding number of machine units
-        (RTIO cycles).
+        (fine RTIO cycles).
 
         :param seconds: time (in seconds) to convert.
         """
@@ -281,7 +286,7 @@ class Core:
 
     @portable
     def mu_to_seconds(self, mu):
-        """Convert machine units (RTIO cycles) to seconds.
+        """Convert machine units (fine RTIO cycles) to seconds.
 
         :param mu: cycle count to convert.
         """
@@ -296,7 +301,7 @@ class Core:
         for the actual value of the hardware register at the instant when
         execution resumes in the caller.
 
-        For a more detailed description of these concepts, see :doc:`/rtio`.
+        For a more detailed description of these concepts, see :doc:`rtio`.
         """
         return rtio_get_counter()
 
@@ -315,7 +320,7 @@ class Core:
     def get_rtio_destination_status(self, destination):
         """Returns whether the specified RTIO destination is up.
         This is particularly useful in startup kernels to delay
-        startup until certain DRTIO destinations are up."""
+        startup until certain DRTIO destinations are available."""
         return rtio_get_destination_status(destination)
 
     @kernel
@@ -343,7 +348,7 @@ class Core:
 
         Returns only after the dump has been retrieved from the device.
 
-        Raises IOError if no analyzer proxy has been configured, or if the
+        Raises :exc:`IOError` if no analyzer proxy has been configured, or if the
         analyzer proxy fails. In the latter case, more details would be
         available in the proxy log.
         """

artiq/coredevice/dma.py

@@ -76,11 +76,11 @@ class CoreDMA:
 
     @kernel
     def record(self, name, enable_ddma=False):
-        """Returns a context manager that will record a DMA trace called ``name``.
+        """Returns a context manager that will record a DMA trace called `name`.
         Any previously recorded trace with the same name is overwritten.
         The trace will persist across kernel switches.
 
-        In DRTIO context, distributed DMA can be toggled with ``enable_ddma``.
+        In DRTIO context, distributed DMA can be toggled with `enable_ddma`.
         Enabling it allows running DMA on satellites, rather than sending all
         events from the master.
 
@@ -116,7 +116,7 @@ class CoreDMA:
     def playback_handle(self, handle):
         """Replays a handle obtained with :meth:`get_handle`. Using this function
         is much faster than :meth:`playback` for replaying a set of traces repeatedly,
-        but incurs the overhead of managing the handles onto the programmer."""
+        but offloads the overhead of managing the handles onto the programmer."""
         (epoch, advance_mu, ptr, uses_ddma) = handle
         if self.epoch != epoch:
             raise DMAError("Invalid handle")

artiq/coredevice/edge_counter.py

@@ -1,9 +1,9 @@
 """Driver for RTIO-enabled TTL edge counter.
 
-Like for the TTL input PHYs, sensitivity can be configured over RTIO
+As for the TTL input PHYs, sensitivity can be configured over RTIO
-(``gate_rising()``, etc.). In contrast to the former, however, the count is
+(:meth:`gate_rising<EdgeCounter.gate_rising>`, etc.). In contrast to the former, however, the count is
 accumulated in gateware, and only a single input event is generated at the end
-of each gate period::
+of each gate period: ::
 
     with parallel:
         doppler_cool()
@@ -17,12 +17,12 @@ of each gate period::
     print("Readout counts:", self.pmt_counter.fetch_count())
 
 For applications where the timestamps of the individual input events are not
-required, this has two advantages over ``TTLInOut.count()`` beyond raw
+required, this has two advantages over :meth:`TTLInOut.count<artiq.coredevice.ttl.TTLInOut.count>` 
-throughput. First, it is easy to count events during multiple separate periods
+beyond raw throughput. First, it is easy to count events during multiple separate 
-without blocking to read back counts in between, as illustrated in the above
+periods without blocking to read back counts in between, as illustrated in the 
-example. Secondly, as each count total only takes up a single input event, it
+above example. Secondly, as each count total only takes up a single input event, 
-is much easier to acquire counts on several channels in parallel without
+it is much easier to acquire counts on several channels in parallel without
-risking input FIFO overflows::
+risking input RTIO overflows: ::
 
     # Using the TTLInOut driver, pmt_1 input events are only processed
     # after pmt_0 is done counting. To avoid RTIOOverflows, a round-robin
@@ -35,8 +35,6 @@ risking input FIFO overflows::
     counts_0 = self.pmt_0.count(now_mu()) # blocks
     counts_1 = self.pmt_1.count(now_mu())
 
-    #
-
     # Using gateware counters, only a single input event each is
     # generated, greatly reducing the load on the input FIFOs:
 
@@ -47,7 +45,7 @@ risking input FIFO overflows::
     counts_0 = self.pmt_0_counter.fetch_count() # blocks
     counts_1 = self.pmt_1_counter.fetch_count()
 
-See :mod:`artiq.gateware.rtio.phy.edge_counter` and
+See the sources of :mod:`artiq.gateware.rtio.phy.edge_counter` and
 :meth:`artiq.gateware.eem.DIO.add_std` for the gateware components.
 """
 
@@ -176,13 +174,13 @@ class EdgeCounter:
         """Emit an RTIO event at the current timeline position to set the
         gateware configuration.
 
-        For most use cases, the `gate_*` wrappers will be more convenient.
+        For most use cases, the ``gate_*`` wrappers will be more convenient.
 
         :param count_rising: Whether to count rising signal edges.
         :param count_falling: Whether to count falling signal edges.
-        :param send_count_event: If `True`, an input event with the current
+        :param send_count_event: If ``True``, an input event with the current
             counter value is generated on the next clock cycle (once).
-        :param reset_to_zero: If `True`, the counter value is reset to zero on
+        :param reset_to_zero: If ``True``, the counter value is reset to zero on
             the next clock cycle (once).
         """
         config = int32(0)

artiq/coredevice/exceptions.py

@@ -6,12 +6,26 @@ import os
 from artiq import __artiq_dir__ as artiq_dir
 from artiq.coredevice.runtime import source_loader
 
+"""
+This file provides class definition for all the exceptions declared in `EmbeddingMap` in `artiq.compiler.embedding`
+
+For Python builtin exceptions, use the `builtins` module
+For ARTIQ specific exceptions, inherit from `Exception` class
+"""
 
-ZeroDivisionError = builtins.ZeroDivisionError
-ValueError = builtins.ValueError
-IndexError = builtins.IndexError
-RuntimeError = builtins.RuntimeError
 AssertionError = builtins.AssertionError
+AttributeError = builtins.AttributeError
+IndexError = builtins.IndexError
+IOError = builtins.IOError
+KeyError = builtins.KeyError
+NotImplementedError = builtins.NotImplementedError
+OverflowError = builtins.OverflowError
+RuntimeError = builtins.RuntimeError
+TimeoutError = builtins.TimeoutError
+TypeError = builtins.TypeError
+ValueError = builtins.ValueError
+ZeroDivisionError = builtins.ZeroDivisionError
+OSError = builtins.OSError
 
 
 class CoreException:
@@ -137,7 +151,7 @@ class RTIOOverflow(Exception):
 
 
 class RTIODestinationUnreachable(Exception):
-    """Raised with a RTIO operation could not be completed due to a DRTIO link
+    """Raised when a RTIO operation could not be completed due to a DRTIO link
     being down.
     """
     artiq_builtin = True
@@ -157,13 +171,17 @@ class SubkernelError(Exception):
 
 class ClockFailure(Exception):
     """Raised when RTIO PLL has lost lock."""
-
+    artiq_builtin = True
 
 class I2CError(Exception):
     """Raised when a I2C transaction fails."""
-    pass
+    artiq_builtin = True
 
 
 class SPIError(Exception):
     """Raised when a SPI transaction fails."""
-    pass
+    artiq_builtin = True
+
+class UnwrapNoneError(Exception):
+    """Raised when unwrapping a none Option."""
+    artiq_builtin = True

artiq/coredevice/fastino.py

@@ -1,4 +1,4 @@
-"""RTIO driver for the Fastino 32channel, 16 bit, 2.5 MS/s per channel,
+"""RTIO driver for the Fastino 32-channel, 16-bit, 2.5 MS/s per channel
 streaming DAC.
 """
 from numpy import int32, int64
@@ -17,22 +17,22 @@ class Fastino:
     to the DAC RTIO addresses, if a channel is not "held" by setting its bit
     using :meth:`set_hold`, the next frame will contain the update. For the
     DACs held, the update is triggered explicitly by setting the corresponding
-    bit using :meth:`set_update`. Update is self-clearing. This enables atomic
+    bit using :meth:`update`. Update is self-clearing. This enables atomic
     DAC updates synchronized to a frame edge.
 
-    The `log2_width=0` RTIO layout uses one DAC channel per RTIO address and a
+    The ``log2_width=0`` RTIO layout uses one DAC channel per RTIO address and a
-    dense RTIO address space. The RTIO words are narrow (32 bit) and
+    dense RTIO address space. The RTIO words are narrow (32-bit) and
     few-channel updates are efficient. There is the least amount of DAC state
     tracking in kernels, at the cost of more DMA and RTIO data.
     The setting here and in the RTIO PHY (gateware) must match.
 
-    Other `log2_width` (up to `log2_width=5`) settings pack multiple
+    Other ``log2_width`` (up to ``log2_width=5``) settings pack multiple
     (in powers of two) DAC channels into one group and into one RTIO write.
-    The RTIO data width increases accordingly. The `log2_width`
+    The RTIO data width increases accordingly. The ``log2_width``
     LSBs of the RTIO address for a DAC channel write must be zero and the
-    address space is sparse. For `log2_width=5` the RTIO data is 512 bit wide.
+    address space is sparse. For ``log2_width=5`` the RTIO data is 512-bit wide.
 
-    If `log2_width` is zero, the :meth:`set_dac`/:meth:`set_dac_mu` interface
+    If ``log2_width`` is zero, the :meth:`set_dac`/:meth:`set_dac_mu` interface
     must be used. If non-zero, the :meth:`set_group`/:meth:`set_group_mu`
     interface must be used.
 
@@ -63,14 +63,15 @@ class Fastino:
             * disables RESET, DAC_CLR, enables AFE_PWR
             * clears error counters, enables error counting
             * turns LEDs off
-            * clears `hold` and `continuous` on all channels
+            * clears ``hold`` and ``continuous`` on all channels
             * clear and resets interpolators to unit rate change on all
               channels
 
         It does not change set channel voltages and does not reset the PLLs or clock
         domains.
 
-        Note: On Fastino gateware before v0.2 this may lead to 0 voltage being emitted
+        .. warning::
+            On Fastino gateware before v0.2 this may lead to 0 voltage being emitted
             transiently.
         """
         self.set_cfg(reset=0, afe_power_down=0, dac_clr=0, clr_err=1)
@@ -115,7 +116,7 @@ class Fastino:
         """Write DAC data in machine units.
 
         :param dac: DAC channel to write to (0-31).
-        :param data: DAC word to write, 16 bit unsigned integer, in machine
+        :param data: DAC word to write, 16-bit unsigned integer, in machine
             units.
         """
         self.write(dac, data)
@@ -124,9 +125,9 @@ class Fastino:
     def set_group_mu(self, dac: TInt32, data: TList(TInt32)):
         """Write a group of DAC channels in machine units.
 
-        :param dac: First channel in DAC channel group (0-31). The `log2_width`
+        :param dac: First channel in DAC channel group (0-31). The ``log2_width``
             LSBs must be zero.
-        :param data: List of DAC data pairs (2x16 bit unsigned) to write,
+        :param data: List of DAC data pairs (2x16-bit unsigned) to write,
             in machine units. Data exceeding group size is ignored.
             If the list length is less than group size, the remaining
             DAC channels within the group are cleared to 0 (machine units).
@@ -137,10 +138,10 @@ class Fastino:
 
     @portable
     def voltage_to_mu(self, voltage):
-        """Convert SI Volts to DAC machine units.
+        """Convert SI volts to DAC machine units.
 
-        :param voltage: Voltage in SI Volts.
+        :param voltage: Voltage in SI volts.
-        :return: DAC data word in machine units, 16 bit integer.
+        :return: DAC data word in machine units, 16-bit integer.
         """
         data = int32(round((0x8000/10.)*voltage)) + int32(0x8000)
         if data < 0 or data > 0xffff:
@@ -149,9 +150,9 @@ class Fastino:
 
     @portable
     def voltage_group_to_mu(self, voltage, data):
-        """Convert SI Volts to packed DAC channel group machine units.
+        """Convert SI volts to packed DAC channel group machine units.
 
-        :param voltage: List of SI Volt voltages.
+        :param voltage: List of SI volt voltages.
         :param data: List of DAC channel data pairs to write to.
             Half the length of `voltage`.
         """
@@ -185,7 +186,7 @@ class Fastino:
     def update(self, update):
         """Schedule channels for update.
 
-        :param update: Bit mask of channels to update (32 bit).
+        :param update: Bit mask of channels to update (32-bit).
         """
         self.write(0x20, update)
 
@@ -193,7 +194,7 @@ class Fastino:
     def set_hold(self, hold):
         """Set channels to manual update.
 
-        :param hold: Bit mask of channels to hold (32 bit).
+        :param hold: Bit mask of channels to hold (32-bit).
         """
         self.write(0x21, hold)
 
@@ -214,9 +215,9 @@ class Fastino:
 
     @kernel
     def set_leds(self, leds):
-        """Set the green user-defined LEDs
+        """Set the green user-defined LEDs.
 
-        :param leds: LED status, 8 bit integer each bit corresponding to one
+        :param leds: LED status, 8-bit integer each bit corresponding to one
             green LED.
         """
         self.write(0x23, leds)
@@ -245,16 +246,16 @@ class Fastino:
     def stage_cic(self, rate) -> TInt32:
         """Compute and stage interpolator configuration.
 
-        This method approximates the desired interpolation rate using a 10 bit
+        This method approximates the desired interpolation rate using a 10-bit
-        floating point representation (6 bit mantissa, 4 bit exponent) and
+        floating point representation (6-bit mantissa, 4-bit exponent) and
         then determines an optimal interpolation gain compensation exponent
         to avoid clipping. Gains for rates that are powers of two are accurately
         compensated. Other rates lead to overall less than unity gain (but more
         than 0.5 gain).
 
-        The overall gain including gain compensation is
+        The overall gain including gain compensation is ``actual_rate ** order / 
-        `actual_rate**order/2**ceil(log2(actual_rate**order))`
+        2 ** ceil(log2(actual_rate ** order))``
-        where `order = 3`.
+        where ``order = 3``.
 
         Returns the actual interpolation rate.
         """
@@ -293,7 +294,7 @@ class Fastino:
         their output is supposed to be constant.
 
         This method resets and settles the affected interpolators. There will be
-        no output updates for the next `order = 3` input samples.
+        no output updates for the next ``order = 3`` input samples.
         Affected channels will only accept one input sample per input sample
         period. This method synchronizes the input sample period to the current
         frame on the affected channels.

artiq/coredevice/grabber.py

@@ -102,7 +102,7 @@ class Grabber:
         this call or the next.
         
         If the timeout is reached before data is available, the exception
-        GrabberTimeoutException is raised.
+        :exc:`GrabberTimeoutException` is raised.
 
         :param timeout_mu: Timestamp at which a timeout will occur. Set to -1
                            (default) to disable timeout. 

artiq/coredevice/i2c.py

@@ -43,7 +43,7 @@ def i2c_poll(busno, busaddr):
     """Poll I2C device at address.
 
     :param busno: I2C bus number
-    :param busaddr: 8 bit I2C device address (LSB=0)
+    :param busaddr: 8-bit I2C device address (LSB=0)
     :returns: True if the poll was ACKed
     """
     i2c_start(busno)
@@ -57,7 +57,7 @@ def i2c_write_byte(busno, busaddr, data, ack=True):
     """Write one byte to a device.
 
     :param busno: I2C bus number
-    :param busaddr: 8 bit I2C device address (LSB=0)
+    :param busaddr: 8-bit I2C device address (LSB=0)
     :param data: Data byte to be written
     :param nack: Allow NACK
     """
@@ -76,7 +76,7 @@ def i2c_read_byte(busno, busaddr):
     """Read one byte from a device.
 
     :param busno: I2C bus number
-    :param busaddr: 8 bit I2C device address (LSB=0)
+    :param busaddr: 8-bit I2C device address (LSB=0)
     :returns: Byte read
     """
     i2c_start(busno)
@@ -95,10 +95,10 @@ def i2c_write_many(busno, busaddr, addr, data, ack_last=True):
     """Transfer multiple bytes to a device.
 
     :param busno: I2c bus number
-    :param busaddr: 8 bit I2C device address (LSB=0)
+    :param busaddr: 8-bit I2C device address (LSB=0)
-    :param addr: 8 bit data address
+    :param addr: 8-bit data address
     :param data: Data bytes to be written
-    :param ack_last: Expect I2C ACK of the last byte written. If `False`,
+    :param ack_last: Expect I2C ACK of the last byte written. If ``False``,
         the last byte may be NACKed (e.g. EEPROM full page writes).
     """
     n = len(data)
@@ -121,8 +121,8 @@ def i2c_read_many(busno, busaddr, addr, data):
     """Transfer multiple bytes from a device.
 
     :param busno: I2c bus number
-    :param busaddr: 8 bit I2C device address (LSB=0)
+    :param busaddr: 8-bit I2C device address (LSB=0)
-    :param addr: 8 bit data address
+    :param addr: 8-bit data address
     :param data: List of integers to be filled with the data read.
         One entry ber byte.
     """
@@ -147,7 +147,7 @@ class I2CSwitch:
 
     PCA954X (or other) type detection is done by the CPU during I2C init.
 
-    I2C transactions not real-time, and are performed by the CPU without
+    I2C transactions are not real-time, and are performed by the CPU without
     involving RTIO.
 
     On the KC705, this chip is used for selecting the I2C buses on the two FMC
@@ -176,7 +176,7 @@ class I2CSwitch:
 class TCA6424A:
     """Driver for the TCA6424A I2C I/O expander.
 
-    I2C transactions not real-time, and are performed by the CPU without
+    I2C transactions are not real-time, and are performed by the CPU without
     involving RTIO.
 
     On the NIST QC2 hardware, this chip is used for switching the directions
@@ -212,7 +212,7 @@ class TCA6424A:
 class PCF8574A:
     """Driver for the PCF8574 I2C remote 8-bit I/O expander.
 
-    I2C transactions not real-time, and are performed by the CPU without
+    I2C transactions are not real-time, and are performed by the CPU without
     involving RTIO.
     """
     def __init__(self, dmgr, busno=0, address=0x7c, core_device="core"):

artiq/coredevice/mirny.py

@@ -1,4 +1,4 @@
-"""RTIO driver for Mirny (4 channel GHz PLLs)
+"""RTIO driver for Mirny (4-channel GHz PLLs)
 """
 
 from artiq.language.core import kernel, delay, portable
@@ -82,7 +82,7 @@ class Mirny:
 
     @kernel
     def read_reg(self, addr):
-        """Read a register"""
+        """Read a register."""
         self.bus.set_config_mu(
             SPI_CONFIG | spi.SPI_INPUT | spi.SPI_END, 24, SPIT_RD, SPI_CS
         )
@@ -91,7 +91,7 @@ class Mirny:
 
     @kernel
     def write_reg(self, addr, data):
-        """Write a register"""
+        """Write a register."""
         self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END, 24, SPIT_WR, SPI_CS)
         self.bus.write((addr << 25) | WE | ((data & 0xFFFF) << 8))
 
@@ -101,9 +101,9 @@ class Mirny:
         Initialize and detect Mirny.
 
         Select the clock source based the board's hardware revision.
-        Raise ValueError if the board's hardware revision is not supported.
+        Raise :exc:`ValueError` if the board's hardware revision is not supported.
 
-        :param blind: Verify presence and protocol compatibility. Raise ValueError on failure.
+        :param blind: Verify presence and protocol compatibility. Raise :exc:`ValueError` on failure.
         """
         reg0 = self.read_reg(0)
         self.hw_rev = reg0 & 0x3
@@ -138,7 +138,7 @@ class Mirny:
     def set_att_mu(self, channel, att):
         """Set digital step attenuator in machine units.
 
-        :param att: Attenuation setting, 8 bit digital.
+        :param att: Attenuation setting, 8-bit digital.
         """
         self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END, 16, SPIT_WR, SPI_CS)
         self.bus.write(((channel | 8) << 25) | (att << 16))
@@ -149,7 +149,7 @@ class Mirny:
 
         This method will write the attenuator settings of the selected channel.
 
-        .. seealso:: :meth:`set_att_mu`
+        See also :meth:`Mirny.set_att_mu`.
 
         :param channel: Attenuator channel (0-3).
         :param att: Attenuation setting in dB. Higher value is more
@@ -160,7 +160,7 @@ class Mirny:
 
     @kernel
     def write_ext(self, addr, length, data, ext_div=SPIT_WR):
-        """Perform SPI write to a prefixed address"""
+        """Perform SPI write to a prefixed address."""
         self.bus.set_config_mu(SPI_CONFIG, 8, SPIT_WR, SPI_CS)
         self.bus.write(addr << 25)
         self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END, length, ext_div, SPI_CS)

artiq/coredevice/novogorny.py

@@ -16,31 +16,31 @@ SPI_CS_SR = 2
 
 @portable
 def adc_ctrl(channel=1, softspan=0b111, valid=1):
-    """Build a LTC2335-16 control word"""
+    """Build a LTC2335-16 control word."""
     return (valid << 7) | (channel << 3) | softspan
 
 
 @portable
 def adc_softspan(data):
-    """Return the softspan configuration index from a result packet"""
+    """Return the softspan configuration index from a result packet."""
     return data & 0x7
 
 
 @portable
 def adc_channel(data):
-    """Return the channel index from a result packet"""
+    """Return the channel index from a result packet."""
     return (data >> 3) & 0x7
 
 
 @portable
 def adc_data(data):
-    """Return the ADC value from a result packet"""
+    """Return the ADC value from a result packet."""
     return (data >> 8) & 0xffff
 
 
 @portable
 def adc_value(data, v_ref=5.):
-    """Convert a ADC result packet to SI units (Volt)"""
+    """Convert a ADC result packet to SI units (volts)."""
     softspan = adc_softspan(data)
     data = adc_data(data)
     g = 625
@@ -107,7 +107,7 @@ class Novogorny:
     def configure(self, data):
         """Set up the ADC sequencer.
 
-        :param data: List of 8 bit control words to write into the sequencer
+        :param data: List of 8-bit control words to write into the sequencer
             table.
         """
         if len(data) > 1:
@@ -137,12 +137,10 @@ class Novogorny:
 
     @kernel
     def sample(self, next_ctrl=0):
-        """Acquire a sample
+        """Acquire a sample. See also :meth:`Novogorny.sample_mu`. 
-
-        .. seealso:: :meth:`sample_mu`
 
         :param next_ctrl: ADC control word for the next sample
-        :return: The ADC result packet (Volt)
+        :return: The ADC result packet (volts)
         """
         return adc_value(self.sample_mu(), self.v_ref)
 
@@ -151,7 +149,7 @@ class Novogorny:
         """Acquire a burst of samples.
 
         If the burst is too long and the sample rate too high, there will be
-        RTIO input overflows.
+        :exc:RTIOOverflow exceptions.
 
         High sample rates lead to gain errors since the impedance between the
         instrumentation amplifier and the ADC is high.

artiq/coredevice/phaser.py

@@ -85,7 +85,7 @@ SERVO_T_CYCLE = (32+12+192+24+4)*ns  # Must match gateware ADC parameters
 class Phaser:
     """Phaser 4-channel, 16-bit, 1 GS/s DAC coredevice driver.
 
-    Phaser contains a 4 channel, 1 GS/s DAC chip with integrated upconversion,
+    Phaser contains a 4-channel, 1 GS/s DAC chip with integrated upconversion,
     quadrature modulation compensation and interpolation features.
 
     The coredevice RTIO PHY and the Phaser gateware come in different modes
@@ -109,9 +109,9 @@ class Phaser:
     **Base mode**
 
     The coredevice produces 2 IQ (in-phase and quadrature) data streams with 25
-    MS/s and 14 bit per quadrature. Each data stream supports 5 independent
+    MS/s and 14 bits per quadrature. Each data stream supports 5 independent
-    numerically controlled IQ oscillators (NCOs, DDSs with 32 bit frequency, 16
+    numerically controlled IQ oscillators (NCOs, DDSs with 32-bit frequency, 
-    bit phase, 15 bit amplitude, and phase accumulator clear functionality)
+    16-bit phase, 15-bit amplitude, and phase accumulator clear functionality)
     added together. See :class:`PhaserChannel` and :class:`PhaserOscillator`.
 
     Together with a data clock, framing marker, a checksum and metadata for
@@ -119,30 +119,28 @@ class Phaser:
     FastLink via a single EEM connector from coredevice to Phaser.
 
     On Phaser in the FPGA the data streams are buffered and interpolated
-    from 25 MS/s to 500 MS/s 16 bit followed by a 500 MS/s digital upconverter
+    from 25 MS/s to 500 MS/s 16-bit followed by a 500 MS/s digital upconverter
     with adjustable frequency and phase. The interpolation passband is 20 MHz
     wide, passband ripple is less than 1e-3 amplitude, stopband attenuation
     is better than 75 dB at offsets > 15 MHz and better than 90 dB at offsets
     > 30 MHz.
 
-    The four 16 bit 500 MS/s DAC data streams are sent via a 32 bit parallel
+    The four 16-bit 500 MS/s DAC data streams are sent via a 32-bit parallel
     LVDS bus operating at 1 Gb/s per pin pair and processed in the DAC (Texas
     Instruments DAC34H84). On the DAC 2x interpolation, sinx/x compensation,
     quadrature modulator compensation, fine and coarse mixing as well as group
     delay capabilities are available. If desired, these features my be
-    configured via the `dac` dictionary.
+    configured via the ``dac`` dictionary.
 
     The latency/group delay from the RTIO events setting
     :class:`PhaserOscillator` or :class:`PhaserChannel` DUC parameters all the
     way to the DAC outputs is deterministic. This enables deterministic
     absolute phase with respect to other RTIO input and output events
-    (see `get_next_frame_mu()`).
+    (see :meth:`get_next_frame_mu()`).
 
     **Miqro mode**
 
-    See :class:`Miqro`
+    See :class:`Miqro`. Here the DAC operates in 4x interpolation.
-
-    Here the DAC operates in 4x interpolation.
 
     **Analog flow**
 
@@ -171,7 +169,7 @@ class Phaser:
     and Q datastreams from the DUC by the IIR output. The IIR state is updated at
     the 3.788 MHz ADC sampling rate.
 
-    Each channel IIR features 4 profiles, each consisting of the [b0, b1, a1] filter
+    Each channel IIR features 4 profiles, each consisting of the ``[b0, b1, a1]`` filter
     coefficients as well as an output offset. The coefficients and offset can be
     set for each profile individually and the profiles each have their own ``y0``,
     ``y1`` output registers (the ``x0``, ``x1`` inputs are shared). To avoid
@@ -185,25 +183,25 @@ class Phaser:
     still ingests samples and updates its input ``x0`` and ``x1`` registers, but
     does not update the ``y0``, ``y1`` output registers.
 
-    After power-up the servo is disabled, in profile 0, with coefficients [0, 0, 0]
+    After power-up the servo is disabled, in profile 0, with coefficients ``[0, 0, 0]``
     and hold is enabled. If older gateware without ther servo is loaded onto the
     Phaser FPGA, the device simply behaves as if the servo is disabled and none of
     the servo functions have any effect.
 
     .. note:: Various register settings of the DAC and the quadrature
-        upconverters are available to be modified through the `dac`, `trf0`,
+        upconverters are available to be modified through the ``dac``, ``trf0``,
-        `trf1` dictionaries. These can be set through the device database
+        ``trf1`` dictionaries. These can be set through the device database
-        (`device_db.py`). The settings are frozen during instantiation of the
+        (``device_db.py``). The settings are frozen during instantiation of the
-        class and applied during `init()`. See the :class:`DAC34H84` and
+        class and applied during ``init()``. See the :class:`dac34H84` and
-        :class:`TRF372017` source for details.
+        :class:`trf372017` source for details.
 
     .. note:: To establish deterministic latency between RTIO time base and DAC
-        output, the DAC FIFO read pointer value (`fifo_offset`) must be
+        output, the DAC FIFO read pointer value (``fifo_offset``) must be
-        fixed. If `tune_fifo_offset=True` (the default) a value with maximum
+        fixed. If `tune_fifo_offset` = ``True`` (the default) a value with maximum
         margin is determined automatically by `dac_tune_fifo_offset` each time
-        `init()` is called. This value should be used for the `fifo_offset` key
+        :meth:`init` is called. This value should be used for the ``fifo_offset`` key
-        of the `dac` settings of Phaser in `device_db.py` and automatic
+        of the ``dac`` settings of Phaser in ``device_db.py`` and automatic
-        tuning should be disabled by `tune_fifo_offset=False`.
+        tuning should be disabled by `tune_fifo_offset` = ``False```.
 
     :param channel: Base RTIO channel number
     :param core_device: Core device name (default: "core")
@@ -219,9 +217,9 @@ class Phaser:
     :param trf1: Channel 1 TRF372017 quadrature upconverter settings as a
         dictionary.
 
-    Attributes:
+    **Attributes:**
 
-    * :attr:`channel`: List of two :class:`PhaserChannel`
+    * :attr:`channel`: List of two instances of :class:`PhaserChannel`
         To access oscillators, digital upconverters, PLL/VCO analog
         quadrature upconverters and attenuators.
     """
@@ -463,8 +461,8 @@ class Phaser:
     def write8(self, addr, data):
         """Write data to FPGA register.
 
-        :param addr: Address to write to (7 bit)
+        :param addr: Address to write to (7-bit)
-        :param data: Data to write (8 bit)
+        :param data: Data to write (8-bit)
         """
         rtio_output((self.channel_base << 8) | (addr & 0x7f) | 0x80, data)
         delay_mu(int64(self.t_frame))
@@ -473,8 +471,8 @@ class Phaser:
     def read8(self, addr) -> TInt32:
         """Read from FPGA register.
 
-        :param addr: Address to read from (7 bit)
+        :param addr: Address to read from (7-bit)
-        :return: Data read (8 bit)
+        :return: Data read (8-bit)
         """
         rtio_output((self.channel_base << 8) | (addr & 0x7f), 0)
         response = rtio_input_data(self.channel_base)
@@ -482,13 +480,13 @@ class Phaser:
 
     @kernel
     def write16(self, addr, data: TInt32):
-        """Write 16 bit to a sequence of FPGA registers."""
+        """Write 16 bits to a sequence of FPGA registers."""
         self.write8(addr, data >> 8)
         self.write8(addr + 1, data)
 
     @kernel
     def write32(self, addr, data: TInt32):
-        """Write 32 bit to a sequence of FPGA registers."""
+        """Write 32 bits to a sequence of FPGA registers."""
         for offset in range(4):
             byte = data >> 24
             self.write8(addr + offset, byte)
@@ -496,7 +494,7 @@ class Phaser:
 
     @kernel
     def read32(self, addr) -> TInt32:
-        """Read 32 bit from a sequence of FPGA registers."""
+        """Read 32 bits from a sequence of FPGA registers."""
         data = 0
         for offset in range(4):
             data <<= 8
@@ -508,15 +506,15 @@ class Phaser:
     def set_leds(self, leds):
         """Set the front panel LEDs.
 
-        :param leds: LED settings (6 bit)
+        :param leds: LED settings (6-bit)
         """
         self.write8(PHASER_ADDR_LED, leds)
 
     @kernel
     def set_fan_mu(self, pwm):
-        """Set the fan duty cycle.
+        """Set the fan duty cycle in machine units.
 
-        :param pwm: Duty cycle in machine units (8 bit)
+        :param pwm: Duty cycle in machine units (8-bit)
         """
         self.write8(PHASER_ADDR_FAN, pwm)
 
@@ -581,10 +579,10 @@ class Phaser:
 
     @kernel
     def measure_frame_timestamp(self):
-        """Measure the timestamp of an arbitrary frame and store it in `self.frame_tstamp`.
+        """Measure the timestamp of an arbitrary frame and store it in ``self.frame_tstamp``.
 
         To be used as reference for aligning updates to the FastLink frames.
-        See `get_next_frame_mu()`.
+        See :meth:`get_next_frame_mu()`.
         """
         rtio_output(self.channel_base << 8, 0)  # read any register
         self.frame_tstamp = rtio_input_timestamp(now_mu() + 4 * self.t_frame, self.channel_base)
@@ -592,10 +590,10 @@ class Phaser:
 
     @kernel
     def get_next_frame_mu(self):
-        """Return the timestamp of the frame strictly after `now_mu()`.
+        """Return the timestamp of the frame strictly after :meth:`~artiq.language.core.now_mu()`.
 
         Register updates (DUC, DAC, TRF, etc.) scheduled at this timestamp and multiples
-        of `self.t_frame` later will have deterministic latency to output.
+        of ``self.t_frame`` later will have deterministic latency to output.
         """
         n = int64((now_mu() - self.frame_tstamp) / self.t_frame)
         return self.frame_tstamp + (n + 1) * self.t_frame
@@ -658,7 +656,7 @@ class Phaser:
 
     @kernel
     def dac_write(self, addr, data):
-        """Write 16 bit to a DAC register.
+        """Write 16 bits to a DAC register.
 
         :param addr: Register address
         :param data: Register data to write
@@ -708,16 +706,16 @@ class Phaser:
     def dac_sync(self):
         """Trigger DAC synchronisation for both output channels.
 
-        The DAC sif_sync is de-asserts, then asserted. The synchronisation is
+        The DAC ``sif_sync`` is de-asserted, then asserted. The synchronisation is
         triggered on assertion.
 
         By default, the fine-mixer (NCO) and QMC are synchronised. This
         includes applying the latest register settings.
 
-        The synchronisation sources may be configured through the `syncsel_x`
+        The synchronisation sources may be configured through the ``syncsel_x``
-        fields in the `dac` configuration dictionary (see `__init__()`).
+        fields in the ``dac`` configuration dictionary (see :class:`Phaser`).
 
-        .. note:: Synchronising the NCO clears the phase-accumulator
+        .. note:: Synchronising the NCO clears the phase-accumulator.
         """
         config1f = self.dac_read(0x1f)
         delay(.4*ms)
@@ -726,11 +724,11 @@ class Phaser:
 
     @kernel
     def set_dac_cmix(self, fs_8_step):
-        """Set the DAC coarse mixer frequency for both channels
+        """Set the DAC coarse mixer frequency for both channels.
 
         Use of the coarse mixer requires the DAC mixer to be enabled. The mixer
-        can be configured via the `dac` configuration dictionary (see
+        can be configured via the ``dac`` configuration dictionary (see
-        `__init__()`).
+        :class:`Phaser`).
 
         The selected coarse mixer frequency becomes active without explicit
         synchronisation.
@@ -763,8 +761,8 @@ class Phaser:
     def dac_iotest(self, pattern) -> TInt32:
         """Performs a DAC IO test according to the datasheet.
 
-        :param pattern: List of four int32 containing the pattern
+        :param pattern: List of four int32s containing the pattern
-        :return: Bit error mask (16 bits)
+        :return: Bit error mask (16-bit)
         """
         if len(pattern) != 4:
             raise ValueError("pattern length out of bounds")
@@ -803,9 +801,9 @@ class Phaser:
 
     @kernel
     def dac_tune_fifo_offset(self):
-        """Scan through `fifo_offset` and configure midpoint setting.
+        """Scan through ``fifo_offset`` and configure midpoint setting.
 
-        :return: Optimal `fifo_offset` setting with maximum margin to write
+        :return: Optimal ``fifo_offset`` setting with maximum margin to write
             pointer.
         """
         # expect two or three error free offsets:
@@ -865,7 +863,7 @@ class PhaserChannel:
 
     Attributes:
 
-    * :attr:`oscillator`: List of five :class:`PhaserOscillator`.
+    * :attr:`oscillator`: List of five instances of :class:`PhaserOscillator`.
     * :attr:`miqro`: A :class:`Miqro`.
 
     .. note:: The amplitude sum of the oscillators must be less than one to
@@ -879,7 +877,7 @@ class PhaserChannel:
         changes in oscillator parameters, the overshoot can lead to clipping
         or overflow after the interpolation. Either band-limit any changes
         in the oscillator parameters or back off the amplitude sufficiently.
-        Miqro is not affected by this. But both the oscillators and Miqro can
+        Miqro is not affected by this, but both the oscillators and Miqro can
         be affected by intrinsic overshoot of the interpolator on the DAC.
     """
     kernel_invariants = {"index", "phaser", "trf_mmap"}
@@ -899,7 +897,7 @@ class PhaserChannel:
         The data is split accross multiple registers and thus the data
         is only valid if constant.
 
-        :return: DAC data as 32 bit IQ. I/DACA/DACC in the 16 LSB,
+        :return: DAC data as 32-bit IQ. I/DACA/DACC in the 16 LSB,
             Q/DACB/DACD in the 16 MSB
         """
         return self.phaser.read32(PHASER_ADDR_DAC0_DATA + (self.index << 4))
@@ -908,7 +906,7 @@ class PhaserChannel:
     def set_dac_test(self, data: TInt32):
         """Set the DAC test data.
 
-        :param data: 32 bit IQ test data, I/DACA/DACC in the 16 LSB,
+        :param data: 32-bit IQ test data, I/DACA/DACC in the 16 LSB,
             Q/DACB/DACD in the 16 MSB
         """
         self.phaser.write32(PHASER_ADDR_DAC0_TEST + (self.index << 4), data)
@@ -930,7 +928,7 @@ class PhaserChannel:
     def set_duc_frequency_mu(self, ftw):
         """Set the DUC frequency.
 
-        :param ftw: DUC frequency tuning word (32 bit)
+        :param ftw: DUC frequency tuning word (32-bit)
         """
         self.phaser.write32(PHASER_ADDR_DUC0_F + (self.index << 4), ftw)
 
@@ -948,7 +946,7 @@ class PhaserChannel:
     def set_duc_phase_mu(self, pow):
         """Set the DUC phase offset.
 
-        :param pow: DUC phase offset word (16 bit)
+        :param pow: DUC phase offset word (16-bit)
         """
         addr = PHASER_ADDR_DUC0_P + (self.index << 4)
         self.phaser.write8(addr, pow >> 8)
@@ -970,10 +968,10 @@ class PhaserChannel:
         This method stages the new NCO frequency, but does not apply it.
 
         Use of the DAC-NCO requires the DAC mixer and NCO to be enabled. These
-        can be configured via the `dac` configuration dictionary (see
+        can be configured via the ``dac`` configuration dictionary (see 
-        `__init__()`).
+        :class:`Phaser`).
 
-        :param ftw: NCO frequency tuning word (32 bit)
+        :param ftw: NCO frequency tuning word (32-bit)
         """
         self.phaser.dac_write(0x15 + (self.index << 1), ftw >> 16)
         self.phaser.dac_write(0x14 + (self.index << 1), ftw)
@@ -985,8 +983,8 @@ class PhaserChannel:
         This method stages the new NCO frequency, but does not apply it.
 
         Use of the DAC-NCO requires the DAC mixer and NCO to be enabled. These
-        can be configured via the `dac` configuration dictionary (see
+        can be configured via the ``dac`` configuration dictionary (see
-        `__init__()`).
+        :class:`Phaser`).
 
         :param frequency: NCO frequency in Hz (passband from -400 MHz
             to 400 MHz, wrapping around at +- 500 MHz)
@@ -1001,14 +999,13 @@ class PhaserChannel:
         By default, the new NCO phase applies on completion of the SPI
         transfer. This also causes a staged NCO frequency to be applied.
         Different triggers for applying NCO settings may be configured through
-        the `syncsel_mixerxx` fields in the `dac` configuration dictionary (see
+        the ``syncsel_mixerxx`` fields in the ``dac`` configuration dictionary (see
-        `__init__()`).
+        :class:`Phaser`).
 
         Use of the DAC-NCO requires the DAC mixer and NCO to be enabled. These
-        can be configured via the `dac` configuration dictionary (see
+        can be configured via the ``dac`` configuration dictionary.
-        `__init__()`).
 
-        :param pow: NCO phase offset word (16 bit)
+        :param pow: NCO phase offset word (16-bit)
         """
         self.phaser.dac_write(0x12 + self.index, pow)
 
@@ -1019,12 +1016,11 @@ class PhaserChannel:
         By default, the new NCO phase applies on completion of the SPI
         transfer. This also causes a staged NCO frequency to be applied.
         Different triggers for applying NCO settings may be configured through
-        the `syncsel_mixerxx` fields in the `dac` configuration dictionary (see
+        the ``syncsel_mixerxx`` fields in the ``dac`` configuration dictionary (see
-        `__init__()`).
+        :class:`Phaser`).
 
         Use of the DAC-NCO requires the DAC mixer and NCO to be enabled. These
-        can be configured via the `dac` configuration dictionary (see
+        can be configured via the ``dac`` configuration dictionary.
-        `__init__()`).
 
         :param phase: NCO phase in turns
         """
@@ -1035,7 +1031,7 @@ class PhaserChannel:
     def set_att_mu(self, data):
         """Set channel attenuation.
 
-        :param data: Attenuator data in machine units (8 bit)
+        :param data: Attenuator data in machine units (8-bit)
         """
         div = 34  # 30 ns min period
         t_xfer = self.phaser.core.seconds_to_mu((8 + 1)*div*4*ns)
@@ -1082,7 +1078,7 @@ class PhaserChannel:
     def trf_write(self, data, readback=False):
         """Write 32 bits to quadrature upconverter register.
 
-        :param data: Register data (32 bit) containing encoded address
+        :param data: Register data (32-bit) containing encoded address
         :param readback: Whether to return the read back MISO data
         """
         div = 34  # 50 ns min period
@@ -1114,7 +1110,7 @@ class PhaserChannel:
 
         :param addr: Register address to read (0 to 7)
         :param cnt_mux_sel: Report VCO counter min or max frequency
-        :return: Register data (32 bit)
+        :return: Register data (32-bit)
         """
         self.trf_write(0x80000008 | (addr << 28) | (cnt_mux_sel << 27))
         # single clk pulse with ~LE to start readback
@@ -1189,13 +1185,13 @@ class PhaserChannel:
             * :math:`b_0` and :math:`b_1` are the feedforward gains for the two
               delays
 
-        .. seealso:: :meth:`set_iir`
+        See also :meth:`PhaserChannel.set_iir`.
 
         :param profile: Profile to set (0 to 3)
-        :param b0: b0 filter coefficient (16 bit signed)
+        :param b0: b0 filter coefficient (16-bit signed)
-        :param b1: b1 filter coefficient (16 bit signed)
+        :param b1: b1 filter coefficient (16-bit signed)
-        :param a1: a1 filter coefficient (16 bit signed)
+        :param a1: a1 filter coefficient (16-bit signed)
-        :param offset: Output offset (16 bit signed)
+        :param offset: Output offset (16-bit signed)
         """
         if (profile < 0) or (profile > 3):
             raise ValueError("invalid profile index")
@@ -1240,7 +1236,7 @@ class PhaserChannel:
             integrator gain limit is infinite. Same sign as ``ki``.
         :param x_offset: IIR input offset. Used as the negative
             setpoint when stabilizing to a desired input setpoint. Will
-            be converted to an equivalent output offset and added to y_offset.
+            be converted to an equivalent output offset and added to ``y_offset``.
         :param y_offset: IIR output offset.
         """
         NORM = 1 << SERVO_COEFF_SHIFT
@@ -1296,7 +1292,7 @@ class PhaserOscillator:
     def set_frequency_mu(self, ftw):
         """Set Phaser MultiDDS frequency tuning word.
 
-        :param ftw: Frequency tuning word (32 bit)
+        :param ftw: Frequency tuning word (32-bit)
         """
         rtio_output(self.base_addr, ftw)
 
@@ -1314,8 +1310,8 @@ class PhaserOscillator:
     def set_amplitude_phase_mu(self, asf=0x7fff, pow=0, clr=0):
         """Set Phaser MultiDDS amplitude, phase offset and accumulator clear.
 
-        :param asf: Amplitude (15 bit)
+        :param asf: Amplitude (15-bit)
-        :param pow: Phase offset word (16 bit)
+        :param pow: Phase offset word (16-bit)
         :param clr: Clear the phase accumulator (persistent)
         """
         data = (asf & 0x7fff) | ((clr & 1) << 15) | (pow << 16)
@@ -1346,38 +1342,42 @@ class Miqro:
 
     **Oscillators**
 
-    * There are n_osc = 16 oscillators with oscillator IDs 0..n_osc-1.
+    * There are ``n_osc = 16`` oscillators with oscillator IDs ``0``... ``n_osc-1``.
     * Each oscillator outputs one tone at any given time
 
-        * I/Q (quadrature, a.k.a. complex) 2x16 bit signed data
+        * I/Q (quadrature, a.k.a. complex) 2x16-bit signed data
           at tau = 4 ns sample intervals, 250 MS/s, Nyquist 125 MHz, bandwidth 200 MHz
           (from f = -100..+100 MHz, taking into account the interpolation anti-aliasing
           filters in subsequent interpolators),
-        * 32 bit frequency (f) resolution (~ 1/16 Hz),
+        * 32-bit frequency (f) resolution (~ 1/16 Hz),
-        * 16 bit unsigned amplitude (a) resolution
+        * 16-bit unsigned amplitude (a) resolution
-        * 16 bit phase offset (p) resolution
+        * 16-bit phase offset (p) resolution
 
-    * The output phase p' of each oscillator at time t (boot/reset/initialization of the
+    * The output phase ``p'`` of each oscillator at time ``t`` (boot/reset/initialization of the
-      device at t=0) is then p' = f*t + p (mod 1 turn) where f and p are the (currently
+      device at ``t=0``) is then ``p' = f*t + p (mod 1 turn)`` where ``f`` and ``p`` are the (currently
       active) profile frequency and phase offset.
-    * Note: The terms  "phase coherent" and "phase tracking" are defined to refer to this
+    
-      choice of oscillator output phase p'. Note that the phase offset p is not relative to
+    .. note :: 
+        The terms  "phase coherent" and "phase tracking" are defined to refer to this
+        choice of oscillator output phase ``p'``. Note that the phase offset ``p`` is not relative to
         (on top of previous phase/profiles/oscillator history).
-      It is "absolute" in the sense that frequency f and phase offset p fully determine
+        It is "absolute" in the sense that frequency ``f`` and phase offset ``p`` fully determine
-      oscillator output phase p' at time t. This is unlike typical DDS behavior.
+        oscillator output phase ``p'`` at time ``t``. This is unlike typical DDS behavior.
+    
     * Frequency, phase, and amplitude of each oscillator are configurable by selecting one of
-      n_profile = 32 profiles 0..n_profile-1. This selection is fast and can be done for
+      ``n_profiles = 32`` profiles ``0``... ``n_profile-1``. This selection is fast and can be 
-      each pulse. The phase coherence defined above is guaranteed for each
+      done for each pulse. The phase coherence defined above is guaranteed for each
       profile individually.
     * Note: one profile per oscillator (usually profile index 0) should be reserved
       for the NOP (no operation, identity) profile, usually with zero amplitude.
     * Data for each profile for each oscillator can be configured
       individually. Storing profile data should be considered "expensive".
-    * Note: The annotation that some operation is "expensive" does not mean it is
+    
-      impossible, just that it may take a significant amount of time and
+    .. note:: 
-      resources to execute such that it may be impractical when used often or
+        To refer to an operation as "expensive" does not mean it is impossible, 
-      during fast pulse sequences. They are intended for use in calibration and
+        merely that it may take a significant amount of time and resources to 
-      initialization.
+        execute, such that it may be impractical when used often or during fast 
+        pulse sequences. They are intended for use in calibration and initialization.  
 
     **Summation**
 
@@ -1394,18 +1394,18 @@ class Miqro:
       the RF output.
     * Selected profiles become active simultaneously (on the same output sample) when
       triggering the shaper with the first shaper output sample.
-    * The shaper reads (replays) window samples from a memory of size n_window = 1 << 10.
+    * The shaper reads (replays) window samples from a memory of size ``n_window = 1 << 10``.
     * The window memory can be segmented by choosing different start indices
       to support different windows.
     * Each window memory segment starts with a header determining segment
       length and interpolation parameters.
     * The window samples are interpolated by a factor (rate change) between 1 and
-      r = 1 << 12.
+      ``r = 1 << 12``.
     * The interpolation order is constant, linear, quadratic, or cubic. This
       corresponds to interpolation modes from rectangular window (1st order CIC)
       or zero order hold) to Parzen window (4th order CIC or cubic spline).
     * This results in support for single shot pulse lengths (envelope support) between
-      tau and a bit more than r * n_window * tau = (1 << 12 + 10) tau ~ 17 ms.
+      tau and a bit more than ``r * n_window * tau = (1 << 12 + 10) tau ~ 17 ms``.
     * Windows can be configured to be head-less and/or tail-less, meaning, they
       do not feed zero-amplitude samples into the shaper before and after
       each window respectively. This is used to implement pulses with arbitrary
@@ -1413,18 +1413,18 @@ class Miqro:
 
     **Overall properties**
 
-    * The DAC may upconvert the signal by applying a frequency offset f1 with
+    * The DAC may upconvert the signal by applying a frequency offset ``f1`` with
-      phase p1.
+      phase ``p1``.
     * In the Upconverter Phaser variant, the analog quadrature upconverter
-      applies another frequency of f2 and phase p2.
+      applies another frequency of ``f2`` and phase ``p2``.
     * The resulting phase of the signal from one oscillator at the SMA output is
-      (f + f1 + f2)*t + p + s(t - t0) + p1 + p2 (mod 1 turn)
+      ``(f + f1 + f2)*t + p + s(t - t0) + p1 + p2 (mod 1 turn)``
-      where s(t - t0) is the phase of the interpolated
+      where ``s(t - t0)`` is the phase of the interpolated
-      shaper output, and t0 is the trigger time (fiducial of the shaper).
+      shaper output, and ``t0`` is the trigger time (fiducial of the shaper).
       Unsurprisingly the frequency is the derivative of the phase.
     * Group delays between pulse parameter updates are matched across oscillators,
       shapers, and channels.
-    * The minimum time to change profiles and phase offsets is ~128 ns (estimate, TBC).
+    * The minimum time to change profiles and phase offsets is ``~128 ns`` (estimate, TBC).
       This is the minimum pulse interval.
       The sustained pulse rate of the RTIO PHY/Fastlink is one pulse per Fastlink frame
       (may be increased, TBC).
@@ -1455,9 +1455,9 @@ class Miqro:
 
         :param oscillator: Oscillator index (0 to 15)
         :param profile: Profile index (0 to 31)
-        :param ftw: Frequency tuning word (32 bit signed integer on a 250 MHz clock)
+        :param ftw: Frequency tuning word (32-bit signed integer on a 250 MHz clock)
-        :param asf: Amplitude scale factor (16 bit unsigned integer)
+        :param asf: Amplitude scale factor (16-bit unsigned integer)
-        :param pow_: Phase offset word (16 bit integer)
+        :param pow_: Phase offset word (16-bit integer)
         """
         if oscillator >= 16:
             raise ValueError("invalid oscillator index")
@@ -1481,7 +1481,7 @@ class Miqro:
         :param amplitude: Amplitude in units of full scale (0. to 1.)
         :param phase: Phase in turns. See :class:`Miqro` for a definition of
             phase in this context.
-        :return: The quantized 32 bit frequency tuning word
+        :return: The quantized 32-bit frequency tuning word
         """
         ftw = int32(round(frequency*((1 << 30)/(62.5*MHz))))
         asf = int32(round(amplitude*0xffff))
@@ -1493,7 +1493,7 @@ class Miqro:
 
     @kernel
     def set_window_mu(self, start, iq, rate=1, shift=0, order=3, head=1, tail=1):
-        """Store a window segment (machine units)
+        """Store a window segment (machine units).
 
         :param start: Window start address (0 to 0x3ff)
         :param iq: List of IQ window samples. Each window sample is an integer
@@ -1540,7 +1540,7 @@ class Miqro:
 
     @kernel
     def set_window(self, start, iq, period=4*ns, order=3, head=1, tail=1):
-        """Store a window segment
+        """Store a window segment.
 
         :param start: Window start address (0 to 0x3ff)
         :param iq: List of IQ window samples. Each window sample is a pair of
@@ -1577,7 +1577,7 @@ class Miqro:
 
     @kernel
     def encode(self, window, profiles, data):
-        """Encode window and profile selection
+        """Encode window and profile selection.
 
         :param window: Window start address (0 to 0x3ff)
         :param profiles: List of profile indices for the oscillators. Maximum

artiq/coredevice/rtio.py

@@ -25,7 +25,7 @@ def rtio_input_data(channel: TInt32) -> TInt32:
 @syscall(flags={"nowrite"})
 def rtio_input_timestamped_data(timeout_mu: TInt64,
                                 channel: TInt32) -> TTuple([TInt64, TInt32]):
-    """Wait for an input event up to timeout_mu on the given channel, and
+    """Wait for an input event up to ``timeout_mu`` on the given channel, and
     return a tuple of timestamp and attached data, or (-1, 0) if the timeout is
     reached."""
     raise NotImplementedError("syscall not simulated")

artiq/coredevice/sampler.py

@@ -16,13 +16,13 @@ SPI_CS_PGIA = 1  # separate SPI bus, CS used as RCLK
 
 @portable
 def adc_mu_to_volt(data, gain=0, corrected_fs=True):
-    """Convert ADC data in machine units to Volts.
+    """Convert ADC data in machine units to volts.
 
-    :param data: 16 bit signed ADC word
+    :param data: 16-bit signed ADC word
     :param gain: PGIA gain setting (0: 1, ..., 3: 1000)
     :param corrected_fs: use corrected ADC FS reference.
-        Should be True for Samplers' revisions after v2.1. False for v2.1 and earlier.
+        Should be ``True`` for Sampler revisions after v2.1. ``False`` for v2.1 and earlier.
-    :return: Voltage in Volts
+    :return: Voltage in volts
     """
     if gain == 0:
         volt_per_lsb = 20.48 / (1 << 16) if corrected_fs else 20. / (1 << 16)
@@ -40,7 +40,7 @@ def adc_mu_to_volt(data, gain=0, corrected_fs=True):
 class Sampler:
     """Sampler ADC.
 
-    Controls the LTC2320-16 8 channel 16 bit ADC with SPI interface and
+    Controls the LTC2320-16 8-channel 16-bit ADC with SPI interface and
     the switchable gain instrumentation amplifiers.
 
     :param spi_adc_device: ADC SPI bus device name
@@ -119,12 +119,12 @@ class Sampler:
         Perform a conversion and transfer the samples.
 
         This assumes that the input FIFO of the ADC SPI RTIO channel is deep
-        enough to buffer the samples (half the length of `data` deep).
+        enough to buffer the samples (half the length of ``data`` deep).
         If it is not, there will be RTIO input overflows.
 
         :param data: List of data samples to fill. Must have even length.
             Samples are always read from the last channel (channel 7) down.
-            The `data` list will always be filled with the last item
+            The ``data`` list will always be filled with the last item
             holding to the sample from channel 7.
         """
         self.cnv.pulse(30*ns)  # t_CNVH
@@ -142,7 +142,7 @@ class Sampler:
     def sample(self, data):
         """Acquire a set of samples.
 
-        .. seealso:: :meth:`sample_mu`
+        See also :meth:`Sampler.sample_mu`.
 
         :param data: List of floating point data samples to fill.
         """

artiq/coredevice/shuttler.py

@@ -16,8 +16,8 @@ def shuttler_volt_to_mu(volt):
 class Config:
     """Shuttler configuration registers interface.
 
-    The configuration registers control waveform phase auto-clear, and pre-DAC
+    The configuration registers control waveform phase auto-clear, pre-DAC
-    gain & offset values for calibration with ADC on the Shuttler AFE card.
+    gain and offset values for calibration with ADC on the Shuttler AFE card.
 
     To find the calibrated DAC code, the Shuttler Core first multiplies the
     output data with pre-DAC gain, then adds the offset.
@@ -84,8 +84,7 @@ class Config:
     def set_offset(self, channel, offset):
         """Set the 16-bits pre-DAC offset register of a Shuttler Core channel.
 
-        .. seealso::
+        See also :meth:`shuttler_volt_to_mu`.
-            :meth:`shuttler_volt_to_mu`
 
         :param channel: Shuttler Core channel to be configured.
         :param offset: Shuttler Core channel offset.
@@ -114,13 +113,13 @@ class DCBias:
     .. math::
         w(t) = a(t) + b(t) * cos(c(t))
     
-    And `t` corresponds to time in seconds.
+    and `t` corresponds to time in seconds.
     This class controls the cubic spline `a(t)`, in which
 
     .. math::
         a(t) = p_0 + p_1t + \\frac{p_2t^2}{2} + \\frac{p_3t^3}{6}
     
-    And `a(t)` is in Volt.
+    and `a(t)` is measured in volts. 
 
     :param channel: RTIO channel number of this DC-bias spline interface.
     :param core_device: Core device name.
@@ -137,7 +136,7 @@ class DCBias:
         """Set the DC-bias spline waveform.
 
         Given `a(t)` as defined in :class:`DCBias`, the coefficients should be
-        configured by the following formulae.
+        configured by the following formulae:
 
         .. math::
             T &= 8*10^{-9}
@@ -154,8 +153,10 @@ class DCBias:
         and 48 bits in width respectively. See :meth:`shuttler_volt_to_mu` for
         machine unit conversion.
 
-        Note: The waveform is not updated to the Shuttler Core until
+        .. note:: 
-        triggered. See :class:`Trigger` for the update triggering mechanism.
+            The waveform is not updated to the Shuttler Core until
+            triggered. See :class:`Trigger` for the update triggering 
+            mechanism.
 
         :param a0: The :math:`a_0` coefficient in machine unit.
         :param a1: The :math:`a_1` coefficient in machine unit.
@@ -189,7 +190,7 @@ class DDS:
     .. math::
         w(t) = a(t) + b(t) * cos(c(t))
     
-    And `t` corresponds to time in seconds.
+    and `t` corresponds to time in seconds.
     This class controls the cubic spline `b(t)` and quadratic spline `c(t)`,
     in which
 
@@ -198,7 +199,7 @@ class DDS:
 
         c(t) &= r_0 + r_1t + \\frac{r_2t^2}{2}
     
-    And `b(t)` is in Volt, `c(t)` is in number of turns. Note that `b(t)`
+    `b(t)` is in volts, `c(t)` is in number of turns. Note that `b(t)`
     contributes to a constant gain of :math:`g=1.64676`.
 
     :param channel: RTIO channel number of this DC-bias spline interface.
@@ -244,13 +245,13 @@ class DDS:
         Note: The waveform is not updated to the Shuttler Core until
         triggered. See :class:`Trigger` for the update triggering mechanism.
 
-        :param b0: The :math:`b_0` coefficient in machine unit.
+        :param b0: The :math:`b_0` coefficient in machine units.
-        :param b1: The :math:`b_1` coefficient in machine unit.
+        :param b1: The :math:`b_1` coefficient in machine units.
-        :param b2: The :math:`b_2` coefficient in machine unit.
+        :param b2: The :math:`b_2` coefficient in machine units.
-        :param b3: The :math:`b_3` coefficient in machine unit.
+        :param b3: The :math:`b_3` coefficient in machine units.
-        :param c0: The :math:`c_0` coefficient in machine unit.
+        :param c0: The :math:`c_0` coefficient in machine units.
-        :param c1: The :math:`c_1` coefficient in machine unit.
+        :param c1: The :math:`c_1` coefficient in machine units.
-        :param c2: The :math:`c_2` coefficient in machine unit.
+        :param c2: The :math:`c_2` coefficient in machine units.
         """
         coef_words = [
             b0,
@@ -292,8 +293,8 @@ class Trigger:
         """Triggers coefficient update of (a) Shuttler Core channel(s).
 
         Each bit corresponds to a Shuttler waveform generator core. Setting
-        `trig_out` bits commits the pending coefficient update (from
+        ``trig_out`` bits commits the pending coefficient update (from
-        `set_waveform` in :class:`DCBias` and :class:`DDS`) to the Shuttler Core
+        ``set_waveform`` in :class:`DCBias` and :class:`DDS`) to the Shuttler Core
         synchronously.
 
         :param trig_out: Coefficient update trigger bits. The MSB corresponds
@@ -336,8 +337,8 @@ _AD4115_REG_SETUPCON0 = 0x20
 class Relay:
     """Shuttler AFE relay switches.
 
-    It controls the AFE relay switches and the LEDs. Switch on the relay to
+    This class controls the AFE relay switches and the LEDs. Switch the relay on to
-    enable AFE output; And off to disable the output. The LEDs indicates the
+    enable AFE output; off to disable the output. The LEDs indicate the
     relay status.
 
     .. note::
@@ -357,7 +358,7 @@ class Relay:
     def init(self):
         """Initialize SPI device.
 
-        Configures the SPI bus to 16-bits, write-only, simultaneous relay
+        Configures the SPI bus to 16 bits, write-only, simultaneous relay
         switches and LED control.
         """
         self.bus.set_config_mu(
@@ -365,10 +366,10 @@ class Relay:
 
     @kernel
     def enable(self, en: TInt32):
-        """Enable/Disable relay switches of corresponding channels.
+        """Enable/disable relay switches of corresponding channels.
 
         Each bit corresponds to the relay switch of a channel. Asserting a bit
-        turns on the corresponding relay switch; Deasserting the same bit
+        turns on the corresponding relay switch; deasserting the same bit
         turns off the switch instead.
 
         :param en: Switch enable bits. The MSB corresponds to Channel 15, LSB
@@ -403,12 +404,12 @@ class ADC:
     def reset(self):
         """AD4115 reset procedure.
 
-        This performs a write operation of 96 serial clock cycles with DIN
+        Performs a write operation of 96 serial clock cycles with DIN
-        held at high. It resets the entire device, including the register
+        held at high. This resets the entire device, including the register
         contents.
 
         .. note::
-            The datasheet only requires 64 cycles, but reasserting `CS_n` right
+            The datasheet only requires 64 cycles, but reasserting ``CS_n`` right
             after the transfer appears to interrupt the start-up sequence.
         """
         self.bus.set_config_mu(ADC_SPI_CONFIG, 32, SPIT_ADC_WR, CS_ADC)
@@ -420,7 +421,7 @@ class ADC:
 
     @kernel
     def read8(self, addr: TInt32) -> TInt32:
-        """Read from 8 bit register.
+        """Read from 8-bit register.
 
         :param addr: Register address.
         :return: Read-back register content.
@@ -433,7 +434,7 @@ class ADC:
 
     @kernel
     def read16(self, addr: TInt32) -> TInt32:
-        """Read from 16 bit register.
+        """Read from 16-bit register.
 
         :param addr: Register address.
         :return: Read-back register content.
@@ -446,7 +447,7 @@ class ADC:
 
     @kernel
     def read24(self, addr: TInt32) -> TInt32:
-        """Read from 24 bit register.
+        """Read from 24-bit register.
 
         :param addr: Register address.
         :return: Read-back register content.
@@ -459,7 +460,7 @@ class ADC:
 
     @kernel
     def write8(self, addr: TInt32, data: TInt32):
-        """Write to 8 bit register.
+        """Write to 8-bit register.
 
         :param addr: Register address.
         :param data: Data to be written.
@@ -470,7 +471,7 @@ class ADC:
 
     @kernel
     def write16(self, addr: TInt32, data: TInt32):
-        """Write to 16 bit register.
+        """Write to 16-bit register.
 
         :param addr: Register address.
         :param data: Data to be written.
@@ -481,7 +482,7 @@ class ADC:
 
     @kernel
     def write24(self, addr: TInt32, data: TInt32):
-        """Write to 24 bit register.
+        """Write to 24-bit register.
 
         :param addr: Register address.
         :param data: Data to be written.
@@ -494,11 +495,11 @@ class ADC:
     def read_ch(self, channel: TInt32) -> TFloat:
         """Sample a Shuttler channel on the AFE.
 
-        It performs a single conversion using profile 0 and setup 0, on the
+        Performs a single conversion using profile 0 and setup 0 on the
-        selected channel. The sample is then recovered and converted to volt.
+        selected channel. The sample is then recovered and converted to volts.
 
         :param channel: Shuttler channel to be sampled.
-        :return: Voltage sample in volt.
+        :return: Voltage sample in volts.
         """
         # Always configure Profile 0 for single conversion
         self.write16(_AD4115_REG_CH0, 0x8000 | ((channel * 2 + 1) << 4))
@@ -519,7 +520,7 @@ class ADC:
 
     @kernel
     def standby(self):
-        """Place the ADC in standby mode and disables power down the clock.
+        """Place the ADC in standby mode and disable power down the clock.
 
         The ADC can be returned to single conversion mode by calling
         :meth:`single_conversion`.
@@ -536,13 +537,7 @@ class ADC:
         .. note::
             The AD4115 datasheet suggests placing the ADC in standby mode
             before power-down. This is to prevent accidental entry into the
-            power-down mode.
+            power-down mode. See also :meth:`standby` and :meth:`power_up`.
-
-        .. seealso::
-            :meth:`standby`
-
-            :meth:`power_up`
-
         """
         self.write16(_AD4115_REG_ADCMODE, 0x8030)
 
@@ -552,8 +547,7 @@ class ADC:
         
         The ADC should be in power-down mode before calling this method.
 
-        .. seealso::
+        See also :meth:`power_down`.
-            :meth:`power_down`
         """
         self.reset()
         # Although the datasheet claims 500 us reset wait time, only waiting
@@ -564,22 +558,18 @@ class ADC:
     def calibrate(self, volts, trigger, config, samples=[-5.0, 0.0, 5.0]):
         """Calibrate the Shuttler waveform generator using the ADC on the AFE.
 
-        It finds the average slope rate and average offset by samples, and
+        Finds the average slope rate and average offset by samples, and
-        compensate by writing the pre-DAC gain and offset registers in the
+        compensates by writing the pre-DAC gain and offset registers in the
         configuration registers.
 
         .. note::
-            If the pre-calibration slope rate < 1, the calibration procedure
+            If the pre-calibration slope rate is less than 1, the calibration 
-            will introduce a pre-DAC gain compensation. However, this may
+            procedure will introduce a pre-DAC gain compensation. However, this 
-            saturate the pre-DAC voltage code. (See :class:`Config` notes).
+            may saturate the pre-DAC voltage code (see :class:`Config` notes).
             Shuttler cannot cover the entire +/- 10 V range in this case.
+            See also :meth:`Config.set_gain` and :meth:`Config.set_offset`.
 
-        .. seealso::
+        :param volts: A list of all 16 cubic DC-bias splines.
-            :meth:`Config.set_gain`
-
-            :meth:`Config.set_offset`
-
-        :param volts: A list of all 16 cubic DC-bias spline.
             (See :class:`DCBias`)
         :param trigger: The Shuttler spline coefficient update trigger.
         :param config: The Shuttler Core configuration registers.

artiq/coredevice/spi2.py

@@ -4,7 +4,7 @@ Driver for generic SPI on RTIO.
 This ARTIQ coredevice driver corresponds to the "new" MiSoC SPI core (v2).
 
 Output event replacement is not supported and issuing commands at the same
-time is an error.
+time results in collision errors.
 """
 
 from artiq.language.core import syscall, kernel, portable, delay_mu
@@ -51,7 +51,7 @@ class SPIMaster:
       event (``SPI_INPUT`` set), then :meth:`read` the ``data``.
     * If ``SPI_END`` was not set, repeat the transfer sequence.
 
-    A **transaction** consists of one or more **transfers**. The chip select
+    A *transaction* consists of one or more *transfers*. The chip select
     pattern is asserted for the entire length of the transaction. All but the
     last transfer are submitted with ``SPI_END`` cleared in the configuration
     register.
@@ -138,10 +138,10 @@ class SPIMaster:
         * :const:`SPI_LSB_FIRST`: LSB is the first bit on the wire (reset=0)
         * :const:`SPI_HALF_DUPLEX`: 3-wire SPI, in/out on ``mosi`` (reset=0)
 
-        :param flags: A bit map of `SPI_*` flags.
+        :param flags: A bit map of :const:`SPI_*` flags.
         :param length: Number of bits to write during the next transfer.
             (reset=1)
-        :param freq: Desired SPI clock frequency. (reset=f_rtio/2)
+        :param freq: Desired SPI clock frequency. (reset= ``f_rtio/2``)
         :param cs: Bit pattern of chip selects to assert.
             Or number of the chip select to assert if ``cs`` is decoded
             downstream. (reset=0)
@@ -152,16 +152,15 @@ class SPIMaster:
     def set_config_mu(self, flags, length, div, cs):
         """Set the ``config`` register (in SPI bus machine units).
 
-        .. seealso:: :meth:`set_config`
+        See also :meth:`set_config`.
 
         :param flags: A bit map of `SPI_*` flags.
         :param length: Number of bits to write during the next transfer.
             (reset=1)
         :param div: Counter load value to divide the RTIO
-          clock by to generate the SPI clock. (minimum=2, reset=2)
+          clock by to generate the SPI clock; ``f_rtio_clk/f_spi == div``. 
-          ``f_rtio_clk/f_spi == div``. If ``div`` is odd,
+          If ``div`` is odd, the setup phase of the SPI clock is one 
-          the setup phase of the SPI clock is one coarse RTIO clock cycle
+          coarse RTIO clock cycle longer than the hold phase. (minimum=2, reset=2)
-          longer than the hold phase.
         :param cs: Bit pattern of chip selects to assert.
             Or number of the chip select to assert if ``cs`` is decoded
             downstream. (reset=0)
@@ -188,7 +187,7 @@ class SPIMaster:
         experiments and are known.
 
         This method is portable and can also be called from e.g.
-        :meth:`__init__`.
+        ``__init__``.
 
         .. warning:: If this method is called while recording a DMA
            sequence, the playback of the sequence will not update the
@@ -208,7 +207,7 @@ class SPIMaster:
         * The ``data`` register and the shift register are 32 bits wide.
         * Data writes take one ``ref_period`` cycle.
         * A transaction consisting of a single transfer (``SPI_END``) takes
-          :attr:`xfer_duration_mu` ``=(n + 1)*div`` cycles RTIO time where
+          :attr:`xfer_duration_mu` `` = (n + 1) * div`` cycles RTIO time, where
           ``n`` is the number of bits and ``div`` is the SPI clock divider.
         * Transfers in a multi-transfer transaction take up to one SPI clock
           cycle less time depending on multiple parameters. Advanced users may

artiq/coredevice/suservo.py

@@ -24,7 +24,7 @@ def y_mu_to_full_scale(y):
 
 @portable
 def adc_mu_to_volts(x, gain, corrected_fs=True):
-    """Convert servo ADC data from machine units to Volt."""
+    """Convert servo ADC data from machine units to volts."""
     val = (x >> 1) & 0xffff
     mask = 1 << 15
     val = -(val & mask) + (val & ~mask)
@@ -155,7 +155,7 @@ class SUServo:
         This method advances the timeline by one servo memory access.
         It does not support RTIO event replacement.
 
-        :param enable (int): Enable servo operation. Enabling starts servo
+        :param int enable: Enable servo operation. Enabling starts servo
             iterations beginning with the ADC sampling stage. The first DDS
             update will happen about two servo cycles (~2.3 µs) after enabling
             the servo. The delay is deterministic.
@@ -198,7 +198,7 @@ class SUServo:
         consistent and valid data, stop the servo before using this method.
 
         :param adc: ADC channel number (0-7)
-        :return: 17 bit signed X0
+        :return: 17-bit signed X0
         """
         # State memory entries are 25 bits. Due to the pre-adder dynamic
         # range, X0/X1/OFFSET are only 24 bits. Finally, the RTIO interface
@@ -288,12 +288,12 @@ class Channel:
     def set_dds_mu(self, profile, ftw, offs, pow_=0):
         """Set profile DDS coefficients in machine units.
 
-        .. seealso:: :meth:`set_amplitude`
+        See also :meth:`Channel.set_dds`.
 
         :param profile: Profile number (0-31)
-        :param ftw: Frequency tuning word (32 bit unsigned)
+        :param ftw: Frequency tuning word (32-bit unsigned)
-        :param offs: IIR offset (17 bit signed)
+        :param offs: IIR offset (17-bit signed)
-        :param pow_: Phase offset word (16 bit)
+        :param pow_: Phase offset word (16-bit)
         """
         base = (self.servo_channel << 8) | (profile << 3)
         self.servo.write(base + 0, ftw >> 16)
@@ -327,7 +327,7 @@ class Channel:
         See :meth:`set_dds_mu` for setting the complete DDS profile.
 
         :param profile: Profile number (0-31)
-        :param offs: IIR offset (17 bit signed)
+        :param offs: IIR offset (17-bit signed)
         """
         base = (self.servo_channel << 8) | (profile << 3)
         self.servo.write(base + 4, offs)
@@ -375,15 +375,15 @@ class Channel:
             * :math:`b_0` and :math:`b_1` are the feedforward gains for the two
               delays
 
-        .. seealso:: :meth:`set_iir`
+        See also :meth:`Channel.set_iir`.
 
         :param profile: Profile number (0-31)
         :param adc: ADC channel to take IIR input from (0-7)
-        :param a1: 18 bit signed A1 coefficient (Y1 coefficient,
+        :param a1: 18-bit signed A1 coefficient (Y1 coefficient,
             feedback, integrator gain)
-        :param b0: 18 bit signed B0 coefficient (recent,
+        :param b0: 18-bit signed B0 coefficient (recent,
             X0 coefficient, feed forward, proportional gain)
-        :param b1: 18 bit signed B1 coefficient (old,
+        :param b1: 18-bit signed B1 coefficient (old,
             X1 coefficient, feed forward, proportional gain)
         :param dly: IIR update suppression time. In units of IIR cycles
             (~1.2 µs, 0-255).
@@ -489,7 +489,7 @@ class Channel:
     def get_y_mu(self, profile):
         """Get a profile's IIR state (filter output, Y0) in machine units.
 
-        The IIR state is also know as the "integrator", or the DDS amplitude
+        The IIR state is also known as the "integrator", or the DDS amplitude
         scale factor. It is 17 bits wide and unsigned.
 
         This method does not advance the timeline but consumes all slack.
@@ -499,7 +499,7 @@ class Channel:
         consistent and valid data, stop the servo before using this method.
 
         :param profile: Profile number (0-31)
-        :return: 17 bit unsigned Y0
+        :return: 17-bit unsigned Y0
         """
         return self.servo.read(STATE_SEL | (self.servo_channel << 5) | profile)
 
@@ -507,7 +507,7 @@ class Channel:
     def get_y(self, profile):
         """Get a profile's IIR state (filter output, Y0).
 
-        The IIR state is also know as the "integrator", or the DDS amplitude
+        The IIR state is also known as the "integrator", or the DDS amplitude
         scale factor. It is 17 bits wide and unsigned.
 
         This method does not advance the timeline but consumes all slack.
@@ -525,7 +525,7 @@ class Channel:
     def set_y_mu(self, profile, y):
         """Set a profile's IIR state (filter output, Y0) in machine units.
 
-        The IIR state is also know as the "integrator", or the DDS amplitude
+        The IIR state is also known as the "integrator", or the DDS amplitude
         scale factor. It is 17 bits wide and unsigned.
 
         This method must not be used when the servo could be writing to the
@@ -535,7 +535,7 @@ class Channel:
         This method advances the timeline by one servo memory access.
 
         :param profile: Profile number (0-31)
-        :param y: 17 bit unsigned Y0
+        :param y: 17-bit unsigned Y0
         """
         # State memory is 25 bits wide and signed.
         # Reads interact with the 18 MSBs (coefficient memory width)
@@ -545,7 +545,7 @@ class Channel:
     def set_y(self, profile, y):
         """Set a profile's IIR state (filter output, Y0).
 
-        The IIR state is also know as the "integrator", or the DDS amplitude
+        The IIR state is also known as the "integrator", or the DDS amplitude
         scale factor. It is 17 bits wide and unsigned.
 
         This method must not be used when the servo could be writing to the

artiq/coredevice/ttl.py

@@ -27,7 +27,7 @@ class TTLOut:
 
     This should be used with output-only channels.
 
-    :param channel: channel number
+    :param channel: Channel number
     """
     kernel_invariants = {"core", "channel", "target_o"}
 
@@ -109,7 +109,7 @@ class TTLInOut:
     API is active (e.g. the gate is open, or the input events have not been
     fully read out), another API must not be used simultaneously.
 
-    :param channel: channel number
+    :param channel: Channel number
     """
     kernel_invariants = {"core", "channel", "gate_latency_mu",
         "target_o", "target_oe", "target_sens", "target_sample"}
@@ -145,7 +145,7 @@ class TTLInOut:
         """Set the direction to output at the current position of the time
         cursor.
 
-        There must be a delay of at least one RTIO clock cycle before any
+        A delay of at least one RTIO clock cycle is necessary before any
         other command can be issued.
 
         This method only configures the direction at the FPGA. When using
@@ -158,7 +158,7 @@ class TTLInOut:
         """Set the direction to input at the current position of the time
         cursor.
 
-        There must be a delay of at least one RTIO clock cycle before any
+        A delay of at least one RTIO clock cycle is necessary before any
         other command can be issued.
 
         This method only configures the direction at the FPGA. When using
@@ -326,17 +326,18 @@ class TTLInOut:
         :return: The number of events before the timeout elapsed (0 if none
             observed).
 
-        Examples:
+        **Examples:**
+
             To count events on channel ``ttl_input``, up to the current timeline
-            position::
+            position: ::
 
                 ttl_input.count(now_mu())
 
             If other events are scheduled between the end of the input gate
-            period and when the number of events is counted, using ``now_mu()``
+            period and when the number of events is counted, using 
-            as timeout consumes an unnecessary amount of timeline slack. In
+            :meth:`~artiq.language.core.now_mu()` as timeout consumes an 
-            such cases, it can be beneficial to pass a more precise timestamp,
+            unnecessary amount of timeline slack. In such cases, it can be 
-            for example::
+            beneficial to pass a more precise timestamp, for example: ::
 
                 gate_end_mu = ttl_input.gate_rising(100 * us)
 
@@ -350,7 +351,7 @@ class TTLInOut:
                 num_rising_edges = ttl_input.count(gate_end_mu)
 
             The ``gate_*()`` family of methods return the cursor at the end
-            of the window, allowing this to be expressed in a compact fashion::
+            of the window, allowing this to be expressed in a compact fashion: ::
 
                 ttl_input.count(ttl_input.gate_rising(100 * us))
         """
@@ -441,7 +442,7 @@ class TTLInOut:
         was being watched.
 
         The time cursor is not modified by this function. This function
-        always makes the slack negative.
+        always results in negative slack.
         """
         rtio_output(self.target_sens, 0)
         success = True

artiq/coredevice/urukul.py

@@ -130,7 +130,7 @@ class CPLD:
     :param spi_device: SPI bus device name
     :param io_update_device: IO update RTIO TTLOut channel name
     :param dds_reset_device: DDS reset RTIO TTLOut channel name
-    :param sync_device: AD9910 SYNC_IN RTIO TTLClockGen channel name
+    :param sync_device: AD9910 ``SYNC_IN`` RTIO TTLClockGen channel name
     :param refclk: Reference clock (SMA, MMCX or on-board 100 MHz oscillator)
         frequency in Hz
     :param clk_sel: Reference clock selection. For hardware revision >= 1.3
@@ -143,9 +143,9 @@ class CPLD:
         1: divide-by-1; 2: divide-by-2; 3: divide-by-4.
         On Urukul boards with CPLD gateware before v1.3.1 only the default
         (0, i.e. variant dependent divider) is valid.
-    :param sync_sel: SYNC (multi-chip synchronisation) signal source selection.
+    :param sync_sel: ``SYNC`` (multi-chip synchronisation) signal source selection.
-        0 corresponds to SYNC_IN being supplied by the FPGA via the EEM
+        0 corresponds to ``SYNC_IN`` being supplied by the FPGA via the EEM
-        connector. 1 corresponds to SYNC_OUT from DDS0 being distributed to the
+        connector. 1 corresponds to ``SYNC_OUT`` from DDS0 being distributed to the
         other chips.
     :param rf_sw: Initial CPLD RF switch register setting (default: 0x0).
         Knowledge of this state is not transferred between experiments.
@@ -153,8 +153,8 @@ class CPLD:
         0x00000000). See also :meth:`get_att_mu` which retrieves the hardware
         state without side effects. Knowledge of this state is not transferred
         between experiments.
-    :param sync_div: SYNC_IN generator divider. The ratio between the coarse
+    :param sync_div: ``SYNC_IN`` generator divider. The ratio between the coarse
-        RTIO frequency and the SYNC_IN generator frequency (default: 2 if
+        RTIO frequency and the ``SYNC_IN`` generator frequency (default: 2 if
         `sync_device` was specified).
     :param core_device: Core device name
 
@@ -204,7 +204,7 @@ class CPLD:
 
         See :func:`urukul_cfg` for possible flags.
 
-        :param cfg: 24 bit data to be written. Will be stored at
+        :param cfg: 24-bit data to be written. Will be stored at
             :attr:`cfg_reg`.
         """
         self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END, 24,
@@ -237,7 +237,7 @@ class CPLD:
 
         Resets the DDS I/O interface and verifies correct CPLD gateware
         version.
-        Does not pulse the DDS MASTER_RESET as that confuses the AD9910.
+        Does not pulse the DDS ``MASTER_RESET`` as that confuses the AD9910.
 
         :param blind: Do not attempt to verify presence and compatibility.
         """
@@ -283,7 +283,7 @@ class CPLD:
     def cfg_switches(self, state: TInt32):
         """Configure all four RF switches through the configuration register.
 
-        :param state: RF switch state as a 4 bit integer.
+        :param state: RF switch state as a 4-bit integer.
         """
         self.cfg_write((self.cfg_reg & ~0xf) | state)
 
@@ -327,10 +327,9 @@ class CPLD:
     @kernel
     def set_all_att_mu(self, att_reg: TInt32):
         """Set all four digital step attenuators (in machine units). 
+        See also :meth:`set_att_mu`.
 
-        .. seealso:: :meth:`set_att_mu`
+        :param att_reg: Attenuator setting string (32-bit)
-
-        :param att_reg: Attenuator setting string (32 bit)
         """
         self.bus.set_config_mu(SPI_CONFIG | spi.SPI_END, 32,
                                SPIT_ATT_WR, CS_ATT)
@@ -342,8 +341,7 @@ class CPLD:
         """Set digital step attenuator in SI units.
 
         This method will write the attenuator settings of all four channels.
-
+        See also :meth:`set_att_mu`.
-        .. seealso:: :meth:`set_att_mu`
 
         :param channel: Attenuator channel (0-3).
         :param att: Attenuation setting in dB. Higher value is more
@@ -359,9 +357,9 @@ class CPLD:
         The result is stored and will be used in future calls of
         :meth:`set_att_mu` and :meth:`set_att`.
 
-        .. seealso:: :meth:`get_channel_att_mu`
+        See also :meth:`get_channel_att_mu`.
 
-        :return: 32 bit attenuator settings
+        :return: 32-bit attenuator settings
         """
         self.bus.set_config_mu(SPI_CONFIG | spi.SPI_INPUT, 32,
                                SPIT_ATT_RD, CS_ATT)
@@ -380,7 +378,7 @@ class CPLD:
         The result is stored and will be used in future calls of
         :meth:`set_att_mu` and :meth:`set_att`.
 
-        .. seealso:: :meth:`get_att_mu`
+        See also :meth:`get_att_mu`.
 
         :param channel: Attenuator channel (0-3).
         :return: 8-bit digital attenuation setting:
@@ -392,7 +390,7 @@ class CPLD:
     def get_channel_att(self, channel: TInt32) -> TFloat:
         """Get digital step attenuator value for a channel in SI units.
 
-        .. seealso:: :meth:`get_channel_att_mu`
+        See also :meth:`get_channel_att_mu`.
 
         :param channel: Attenuator channel (0-3).
         :return: Attenuation setting in dB. Higher value is more
@@ -403,14 +401,14 @@ class CPLD:
 
     @kernel
     def set_sync_div(self, div: TInt32):
-        """Set the SYNC_IN AD9910 pulse generator frequency
+        """Set the ``SYNC_IN`` AD9910 pulse generator frequency
         and align it to the current RTIO timestamp.
 
-        The SYNC_IN signal is derived from the coarse RTIO clock
+        The ``SYNC_IN`` signal is derived from the coarse RTIO clock
         and the divider must be a power of two.
         Configure ``sync_sel == 0``.
 
-        :param div: SYNC_IN frequency divider. Must be a power of two.
+        :param div: ``SYNC_IN`` frequency divider. Must be a power of two.
             Minimum division ratio is 2. Maximum division ratio is 16.
         """
         ftw_max = 1 << 4

artiq/coredevice/zotino.py

@@ -1,7 +1,7 @@
 """RTIO driver for the Zotino 32-channel, 16-bit 1MSPS DAC.
 
 Output event replacement is not supported and issuing commands at the same
-time is an error.
+time results in a collision error.
 """
 
 from artiq.language.core import kernel

artiq/dashboard/moninj.py

@@ -348,7 +348,7 @@ class _DACWidget(QtWidgets.QFrame):
         grid.setHorizontalSpacing(0)
         grid.setVerticalSpacing(0)
         self.setLayout(grid)
-        label = QtWidgets.QLabel("{} ch{}".format(title, channel))
+        label = QtWidgets.QLabel("{}_ch{}".format(title, channel))
         label.setAlignment(QtCore.Qt.AlignCenter)
         grid.addWidget(label, 1, 1)
 
@@ -373,7 +373,7 @@ class _DACWidget(QtWidgets.QFrame):
         return (self.title, self.channel)
 
     def to_model_path(self):
-        return "dac/{} ch{}".format(self.title, self.channel)
+        return "dac/{}_ch{}".format(self.title, self.channel)
 
 
 _WidgetDesc = namedtuple("_WidgetDesc", "uid comment cls arguments")
@@ -391,8 +391,6 @@ def setup_from_ddb(ddb):
                 comment = v.get("comment")
                 if v["type"] == "local":
                     if v["module"] == "artiq.coredevice.ttl":
-                        if "ttl_urukul" in k:
-                            continue
                         channel = v["arguments"]["channel"]
                         force_out = v["class"] == "TTLOut"
                         widget = _WidgetDesc(k, comment, _TTLWidget, (channel, force_out, k))
@@ -414,7 +412,7 @@ def setup_from_ddb(ddb):
                         bus_channel = ddb[spi_dev]["arguments"]["channel"]
                         pll = v["arguments"]["pll_n"]
                         refclk = ddb[dds_cpld]["arguments"]["refclk"]
-                        clk_div = v["arguments"].get("clk_div", 0)
+                        clk_div = ddb[dds_cpld]["arguments"].get("clk_div", 0)
                         widget = _WidgetDesc(k, comment, _DDSWidget,
                                              (k, bus_channel, channel, v["class"],
                                               refclk, dds_cpld, pll, clk_div))
@@ -861,9 +859,10 @@ class _MonInjDock(QDockWidgetCloseDetect):
 
     def delete_widget(self, index, checked):
         widget = self.flow.itemAt(index).widget()
-        widget.hide()
         self.manager.dm.setup_monitoring(False, widget)
         self.flow.layout.takeAt(index)
+        widget.setParent(self.manager.main_window)
+        widget.hide()
 
     def add_channels(self):
         channels = self.channel_dialog.channels
@@ -871,9 +870,9 @@ class _MonInjDock(QDockWidgetCloseDetect):
 
     def layout_widgets(self, widgets):
         for widget in sorted(widgets, key=lambda w: w.sort_key()):
-            widget.show()
             self.manager.dm.setup_monitoring(True, widget)
             self.flow.addWidget(widget)
+            widget.show()
 
     def restore_widgets(self):
         if self.widget_uids is not None:
@@ -948,7 +947,7 @@ class MonInj:
         del self.docks[name]
         self.update_closable()
         dock.delete_all_widgets()
-        dock.hide()  # dock may be parent, only delete on exit
+        dock.deleteLater()
 
     def update_closable(self):
         flags = (QtWidgets.QDockWidget.DockWidgetMovable |

artiq/examples/kasli_suservo/repository/suservo.py

@@ -36,7 +36,7 @@ class SUServo(EnvExperiment):
             self.suservo0.set_pgia_mu(i, 0)
             delay(10*us)
         # DDS attenuator
-        self.suservo0.cpld0.set_att(0, 10.)
+        self.suservo0.cplds[0].set_att(0, 10.)
         delay(1*us)
         # Servo is done and disabled
         assert self.suservo0.get_status() & 0xff == 2

artiq/firmware/bootloader/main.rs

@@ -500,7 +500,7 @@ pub extern fn main() -> i32 {
     println!(r"|_|  |_|_|____/ \___/ \____|");
     println!("");
     println!("MiSoC Bootloader");
-    println!("Copyright (c) 2017-2024 M-Labs Limited");
+    println!("Copyright (c) 2017-2025 M-Labs Limited");
     println!("");
 
     #[cfg(has_ethmac)]

artiq/firmware/ksupport/api.rs

@@ -105,6 +105,7 @@ static mut API: &'static [(&'static str, *const ())] = &[
     api!(nextafter),
     api!(pow),
     api!(round),
+    api!(rint),
     api!(sin),
     api!(sinh),
     api!(sqrt),
@@ -172,4 +173,12 @@ static mut API: &'static [(&'static str, *const ())] = &[
     api!(spi_set_config = ::nrt_bus::spi::set_config),
     api!(spi_write = ::nrt_bus::spi::write),
     api!(spi_read = ::nrt_bus::spi::read),
+
+    /*
+     * syscall for unit tests
+     * Used in `artiq.tests.coredevice.test_exceptions.ExceptionTest.test_raise_exceptions_kernel`
+     * This syscall checks that the exception IDs used in the Python `EmbeddingMap` (in `artiq.compiler.embedding`)
+     * match the `EXCEPTION_ID_LOOKUP` defined in the firmware (`artiq::firmware::ksupport::eh_artiq`)
+     */
+    api!(test_exception_id_sync = ::eh_artiq::test_exception_id_sync)
 ];

artiq/firmware/ksupport/eh_artiq.rs

@@ -328,6 +328,7 @@ extern fn stop_fn(_version: c_int,
     }
 }
 
+// Must be kept in sync with `artiq.compiler.embedding`
 static EXCEPTION_ID_LOOKUP: [(&str, u32); 12] = [
     ("RuntimeError", 0),
     ("RTIOUnderflow", 1),
@@ -340,7 +341,7 @@ static EXCEPTION_ID_LOOKUP: [(&str, u32); 12] = [
     ("ZeroDivisionError", 8),
     ("IndexError", 9),
     ("UnwrapNoneError", 10),
-    ("SubkernelError", 11)
+    ("SubkernelError", 11),
 ];
 
 pub fn get_exception_id(name: &str) -> u32 {
@@ -352,3 +353,29 @@ pub fn get_exception_id(name: &str) -> u32 {
     unimplemented!("unallocated internal exception id")
 }
 
+/// Takes as input exception id from host
+/// Generates a new exception with:
+///   * `id` set to `exn_id`
+///   * `message` set to corresponding exception name from `EXCEPTION_ID_LOOKUP`
+///
+/// The message is matched on host to ensure correct exception is being referred 
+/// This test checks the synchronization of exception ids for runtime errors
+#[no_mangle]
+pub extern "C-unwind" fn test_exception_id_sync(exn_id: u32) {
+    let message = EXCEPTION_ID_LOOKUP
+        .iter()
+        .find_map(|&(name, id)| if id == exn_id { Some(name) } else { None })
+        .unwrap_or("unallocated internal exception id");
+    
+    let exn = Exception {
+        id:       exn_id,
+        file:     file!().as_c_slice(),
+        line:     0,
+        column:   0,
+        function: "test_exception_id_sync".as_c_slice(),
+        message:  message.as_c_slice(),
+        param:    [0, 0, 0]
+    };
+    unsafe { raise(&exn) };
+}
+

artiq/firmware/ksupport/lib.rs

@@ -37,6 +37,14 @@ fn recv<R, F: FnOnce(&Message) -> R>(f: F) -> R {
     result
 }
 
+fn try_recv<F: FnOnce(&Message)>(f: F) {
+    let msg_ptr = mailbox::receive();
+    if msg_ptr != 0 {
+        f(unsafe { &*(msg_ptr as *const Message) });
+        mailbox::acknowledge();
+    }
+}
+
 macro_rules! recv {
     ($p:pat => $e:expr) => {
         recv(move |request| {
@@ -473,7 +481,15 @@ extern "C-unwind" fn dma_playback(timestamp: i64, ptr: i32, _uses_ddma: bool) {
 
 
 extern "C-unwind" fn subkernel_load_run(id: u32, destination: u8, run: bool) {
-    send(&SubkernelLoadRunRequest { id: id, destination: destination, run: run });
+    let timestamp = unsafe {
+        ((csr::rtio::now_hi_read() as u64) << 32) | (csr::rtio::now_lo_read() as u64)
+    };
+    send(&SubkernelLoadRunRequest { 
+        id: id, 
+        destination: destination, 
+        run: run, 
+        timestamp: timestamp,
+    });
     recv!(&SubkernelLoadRunReply { succeeded } => {
         if !succeeded {
             raise!("SubkernelError",
@@ -484,9 +500,10 @@ extern "C-unwind" fn subkernel_load_run(id: u32, destination: u8, run: bool) {
 
 extern "C-unwind" fn subkernel_await_finish(id: u32, timeout: i64) {
     send(&SubkernelAwaitFinishRequest { id: id, timeout: timeout });
-    recv!(SubkernelAwaitFinishReply { status } => {
+    recv(move |request| {
+        if let SubkernelAwaitFinishReply = request { }
+        else if let SubkernelError(status) = request {
             match status {
-            SubkernelStatus::NoError => (),
                 SubkernelStatus::IncorrectState => raise!("SubkernelError",
                     "Subkernel not running"),
                 SubkernelStatus::Timeout => raise!("SubkernelError",
@@ -494,7 +511,12 @@ extern "C-unwind" fn subkernel_await_finish(id: u32, timeout: i64) {
                 SubkernelStatus::CommLost => raise!("SubkernelError",
                     "Lost communication with satellite"),
                 SubkernelStatus::OtherError => raise!("SubkernelError",
-                "An error occurred during subkernel operation")
+                    "An error occurred during subkernel operation"),
+                SubkernelStatus::Exception(e) => unsafe { crate::eh_artiq::raise(e) },
+            }
+        } else {
+            send(&Log(format_args!("unexpected reply: {:?}\n", request)));
+            loop {}
         }
     })
 }
@@ -512,15 +534,15 @@ extern fn subkernel_send_message(id: u32, is_return: bool, destination: u8,
 
 extern "C-unwind" fn subkernel_await_message(id: i32, timeout: i64, tags: &CSlice<u8>, min: u8, max: u8) -> u8 {
     send(&SubkernelMsgRecvRequest { id: id, timeout: timeout, tags: tags.as_ref() });
-    recv!(SubkernelMsgRecvReply { status, count } => {
+    recv(move |request| {
-        match status {
+        if let SubkernelMsgRecvReply { count } = request {
-            SubkernelStatus::NoError => {
             if count < &min || count > &max {
                 raise!("SubkernelError",
                     "Received less or more arguments than expected");
             }
             *count
-            }
+        } else if let SubkernelError(status) = request {
+            match status {
                 SubkernelStatus::IncorrectState => raise!("SubkernelError",
                     "Subkernel not running"),
                 SubkernelStatus::Timeout => raise!("SubkernelError",
@@ -528,7 +550,12 @@ extern "C-unwind" fn subkernel_await_message(id: i32, timeout: i64, tags: &CSlic
                 SubkernelStatus::CommLost => raise!("SubkernelError",
                     "Lost communication with satellite"),
                 SubkernelStatus::OtherError => raise!("SubkernelError",
-                "An error occurred during subkernel operation")
+                    "An error occurred during subkernel operation"),
+                SubkernelStatus::Exception(e) => unsafe { crate::eh_artiq::raise(e) },
+            }
+        } else {
+            send(&Log(format_args!("unexpected reply: {:?}\n", request)));
+            loop {}
         }
     })
     // RpcRecvRequest should be called `count` times after this to receive message data
@@ -590,6 +617,15 @@ pub unsafe fn main() {
             },
             Ok(library) => {
                 send(&LoadReply(Ok(())));
+                // Master kernel would just acknowledge kernel load
+                // Satellites may send UpdateNow
+                try_recv(move |msg| match msg {
+                    UpdateNow(timestamp) => unsafe {
+                        csr::rtio::now_hi_write((*timestamp >> 32) as u32);
+                        csr::rtio::now_lo_write(*timestamp as u32);
+                    }
+                    _ => unreachable!()
+                });
                 library
             }
         }

artiq/firmware/libproto_artiq/drtioaux_proto.rs

@@ -120,11 +120,11 @@ pub enum Packet {
 
     SubkernelAddDataRequest { destination: u8, id: u32, status: PayloadStatus, length: u16, data: [u8; MASTER_PAYLOAD_MAX_SIZE] },
     SubkernelAddDataReply { succeeded: bool },
-    SubkernelLoadRunRequest { source: u8, destination: u8, id: u32, run: bool },
+    SubkernelLoadRunRequest { source: u8, destination: u8, id: u32, run: bool, timestamp: u64 },
     SubkernelLoadRunReply { destination: u8, succeeded: bool },
     SubkernelFinished { destination: u8, id: u32, with_exception: bool, exception_src: u8 },
-    SubkernelExceptionRequest { destination: u8 },
+    SubkernelExceptionRequest { source: u8, destination: u8 },
-    SubkernelException { last: bool, length: u16, data: [u8; SAT_PAYLOAD_MAX_SIZE] },
+    SubkernelException { destination: u8, last: bool, length: u16, data: [u8; MASTER_PAYLOAD_MAX_SIZE] },
     SubkernelMessage { source: u8, destination: u8, id: u32, status: PayloadStatus, length: u16, data: [u8; MASTER_PAYLOAD_MAX_SIZE] },
     SubkernelMessageAck { destination: u8 },
 }
@@ -354,7 +354,8 @@ impl Packet {
                 source: reader.read_u8()?,
                 destination: reader.read_u8()?,
                 id: reader.read_u32()?,
-                run: reader.read_bool()?
+                run: reader.read_bool()?,
+                timestamp: reader.read_u64()?
             },
             0xc5 => Packet::SubkernelLoadRunReply {
                 destination: reader.read_u8()?,
@@ -367,14 +368,17 @@ impl Packet {
                 exception_src: reader.read_u8()?
             },
             0xc9 => Packet::SubkernelExceptionRequest {
+                source: reader.read_u8()?,
                 destination: reader.read_u8()?
             },
             0xca => {
+                let destination = reader.read_u8()?;
                 let last = reader.read_bool()?;
                 let length = reader.read_u16()?;
-                let mut data: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+                let mut data: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
                 reader.read_exact(&mut data[0..length as usize])?;
                 Packet::SubkernelException {
+                    destination: destination,
                     last: last,
                     length: length,
                     data: data
@@ -644,12 +648,13 @@ impl Packet {
                 writer.write_u8(0xc1)?;
                 writer.write_bool(succeeded)?;
             },
-            Packet::SubkernelLoadRunRequest { source, destination, id, run } => {
+            Packet::SubkernelLoadRunRequest { source, destination, id, run, timestamp } => {
                 writer.write_u8(0xc4)?;
                 writer.write_u8(source)?;
                 writer.write_u8(destination)?;
                 writer.write_u32(id)?;
                 writer.write_bool(run)?;
+                writer.write_u64(timestamp)?;
             },
             Packet::SubkernelLoadRunReply { destination, succeeded } => {
                 writer.write_u8(0xc5)?;
@@ -663,12 +668,14 @@ impl Packet {
                 writer.write_bool(with_exception)?;
                 writer.write_u8(exception_src)?;
             },
-            Packet::SubkernelExceptionRequest { destination } => {
+            Packet::SubkernelExceptionRequest { source, destination } => {
                 writer.write_u8(0xc9)?;
+                writer.write_u8(source)?;
                 writer.write_u8(destination)?;
             },
-            Packet::SubkernelException { last, length, data } => {
+            Packet::SubkernelException { destination, last, length, data } => {
                 writer.write_u8(0xca)?;
+                writer.write_u8(destination)?;
                 writer.write_bool(last)?;
                 writer.write_u16(length)?;
                 writer.write_all(&data[0..length as usize])?;
@@ -703,6 +710,8 @@ impl Packet {
             Packet::SubkernelLoadRunReply     { destination, .. } => Some(*destination),
             Packet::SubkernelMessage          { destination, .. } => Some(*destination),
             Packet::SubkernelMessageAck       { destination, .. } => Some(*destination),
+            Packet::SubkernelExceptionRequest { destination, .. } => Some(*destination),
+            Packet::SubkernelException        { destination, .. } => Some(*destination),
             Packet::DmaPlaybackStatus         { destination, .. } => Some(*destination),
             Packet::SubkernelFinished         { destination, .. } => Some(*destination),
             _ => None
@@ -717,7 +726,7 @@ impl Packet {
             Packet::DmaAddTraceReply { .. } | Packet::DmaRemoveTraceReply { .. } |
                 Packet::DmaPlaybackReply { .. } | Packet::SubkernelLoadRunReply { .. } |
                 Packet::SubkernelMessageAck { .. } | Packet::DmaPlaybackStatus { .. } |
-                Packet::SubkernelFinished { .. } => false,
+                Packet::SubkernelFinished { .. } | Packet::InjectionRequest { .. } => false,
             _ => true
         }
     }

artiq/firmware/libproto_artiq/kernel_proto.rs

@@ -11,12 +11,12 @@ pub const KERNELCPU_LAST_ADDRESS:    usize = 0x4fffffff;
 pub const KSUPPORT_HEADER_SIZE: usize = 0x74;
 
 #[derive(Debug)]
-pub enum SubkernelStatus {
+pub enum SubkernelStatus<'a> {
-    NoError,
     Timeout,
     IncorrectState,
     CommLost,
-    OtherError
+    Exception(eh::eh_artiq::Exception<'a>),
+    OtherError,
 }
 
 #[derive(Debug)]
@@ -103,13 +103,16 @@ pub enum Message<'a> {
     SpiReadReply { succeeded: bool, data: u32 },
     SpiBasicReply { succeeded: bool },
 
-    SubkernelLoadRunRequest { id: u32, destination: u8, run: bool },
+    SubkernelLoadRunRequest { id: u32, destination: u8, run: bool, timestamp: u64 },
     SubkernelLoadRunReply { succeeded: bool },
     SubkernelAwaitFinishRequest { id: u32, timeout: i64 },
-    SubkernelAwaitFinishReply { status: SubkernelStatus },
+    SubkernelAwaitFinishReply,
     SubkernelMsgSend { id: u32, destination: Option<u8>, count: u8, tag: &'a [u8], data: *const *const () },
     SubkernelMsgRecvRequest { id: i32, timeout: i64, tags: &'a [u8] },
-    SubkernelMsgRecvReply { status: SubkernelStatus, count: u8 },
+    SubkernelMsgRecvReply { count: u8 },
+    SubkernelError(SubkernelStatus<'a>),
+
+    UpdateNow(u64),
 
     Log(fmt::Arguments<'a>),
     LogSlice(&'a str)

artiq/firmware/runtime/kernel.rs

@@ -95,7 +95,9 @@ pub mod subkernel {
     use board_artiq::drtio_routing::RoutingTable;
     use board_misoc::clock;
     use proto_artiq::{drtioaux_proto::{PayloadStatus, MASTER_PAYLOAD_MAX_SIZE}, rpc_proto as rpc};
-    use io::Cursor;
+    use io::{Cursor, ProtoRead};
+    use eh::eh_artiq::Exception;
+    use cslice::CSlice;
     use rtio_mgt::drtio;
     use sched::{Io, Mutex, Error as SchedError};
 
@@ -192,14 +194,15 @@ pub mod subkernel {
     }
 
     pub fn load(io: &Io, aux_mutex: &Mutex, ddma_mutex: &Mutex, subkernel_mutex: &Mutex, routing_table: &RoutingTable,
-            id: u32, run: bool) -> Result<(), Error> {
+            id: u32, run: bool, timestamp: u64) -> Result<(), Error> {
         let _lock = subkernel_mutex.lock(io)?;
         let subkernel = unsafe { SUBKERNELS.get_mut(&id).unwrap() };
         if subkernel.state != SubkernelState::Uploaded {
             error!("for id: {} expected Uploaded, got: {:?}", id, subkernel.state);
             return Err(Error::IncorrectState);
         }
-        drtio::subkernel_load(io, aux_mutex, ddma_mutex, subkernel_mutex, routing_table, id, subkernel.destination, run)?;
+        drtio::subkernel_load(io, aux_mutex, ddma_mutex, subkernel_mutex, 
+            routing_table, id, subkernel.destination, run, timestamp)?;
         if run {
             subkernel.state = SubkernelState::Running;
         }
@@ -256,6 +259,49 @@ pub mod subkernel {
         }
     }
 
+    fn read_exception_string<'a>(reader: &mut Cursor<&[u8]>) -> Result<CSlice<'a, u8>, Error> {
+        let len = reader.read_u32()? as usize;
+        if len == usize::MAX {
+            let data = reader.read_u32()?;
+            Ok(unsafe { CSlice::new(data as *const u8, len) })
+        } else {
+            let pos = reader.position();
+            let slice = unsafe {
+                let ptr = reader.get_ref().as_ptr().offset(pos as isize);
+                CSlice::new(ptr, len)
+            };
+            reader.set_position(pos + len);
+            Ok(slice)
+        }
+    }
+
+    pub fn read_exception(buffer: &[u8]) -> Result<Exception, Error>
+    {
+        let mut reader = Cursor::new(buffer);
+
+        let mut byte = reader.read_u8()?;
+        // to sync
+        while byte != 0x5a {
+            byte = reader.read_u8()?;
+        }
+        // skip sync bytes, 0x09 indicates exception
+        while byte != 0x09 {
+            byte = reader.read_u8()?;
+        }
+        let _len = reader.read_u32()?;
+        // ignore the remaining exceptions, stack traces etc. - unwinding from another device would be unwise anyway
+        Ok(Exception {
+            id:       reader.read_u32()?,
+            message:  read_exception_string(&mut reader)?,
+            param:    [reader.read_u64()? as i64, reader.read_u64()? as i64, reader.read_u64()? as i64],
+            file:     read_exception_string(&mut reader)?,
+            line:     reader.read_u32()?,
+            column:   reader.read_u32()?,
+            function: read_exception_string(&mut reader)?
+        })
+    }
+
+
     pub fn retrieve_finish_status(io: &Io, aux_mutex: &Mutex, ddma_mutex: &Mutex, subkernel_mutex: &Mutex,
         routing_table: &RoutingTable, id: u32) -> Result<SubkernelFinished, Error> {
         let _lock = subkernel_mutex.lock(io)?;

artiq/firmware/runtime/main.rs

@@ -230,7 +230,7 @@ fn startup() {
         let subkernel_mutex = subkernel_mutex.clone();
         let drtio_routing_table = drtio_routing_table.clone();
         let up_destinations = up_destinations.clone();
-        io.spawn(8192, move |io| { analyzer::thread(io, &aux_mutex, &ddma_mutex, &subkernel_mutex, &drtio_routing_table, &up_destinations) });
+        io.spawn(16384, move |io| { analyzer::thread(io, &aux_mutex, &ddma_mutex, &subkernel_mutex, &drtio_routing_table, &up_destinations) });
     }
 
     #[cfg(has_grabber)]

artiq/firmware/runtime/rtio_mgt.rs

@@ -128,6 +128,8 @@ pub mod drtio {
                 drtioaux::Packet::SubkernelLoadRunReply     { destination, .. } |
                 drtioaux::Packet::SubkernelMessage          { destination, .. } |
                 drtioaux::Packet::SubkernelMessageAck       { destination, .. } |
+                drtioaux::Packet::SubkernelExceptionRequest { destination, .. } |
+                drtioaux::Packet::SubkernelException        { destination, .. } |
                 drtioaux::Packet::DmaPlaybackStatus         { destination, .. } |
                 drtioaux::Packet::SubkernelFinished         { destination, .. } => {
                 if *destination == 0 {
@@ -592,11 +594,14 @@ pub mod drtio {
     }
 
     pub fn subkernel_load(io: &Io, aux_mutex: &Mutex, ddma_mutex: &Mutex, subkernel_mutex: &Mutex, 
-            routing_table: &drtio_routing::RoutingTable, id: u32, destination: u8, run: bool
+            routing_table: &drtio_routing::RoutingTable, id: u32, destination: u8, run: bool, timestamp: u64
         ) -> Result<(), Error> {
         let linkno = routing_table.0[destination as usize][0] - 1;
         let reply = aux_transact(io, aux_mutex, ddma_mutex, subkernel_mutex, routing_table, linkno, 
-            &drtioaux::Packet::SubkernelLoadRunRequest{ id: id, source: 0, destination: destination, run: run })?;
+            &drtioaux::Packet::SubkernelLoadRunRequest{ 
+                id: id, source: 0, destination: destination,
+                run: run, timestamp: timestamp
+            })?;
         match reply {
             drtioaux::Packet::SubkernelLoadRunReply { destination: 0, succeeded: true } => Ok(()),
             drtioaux::Packet::SubkernelLoadRunReply { destination: 0, succeeded: false } =>
@@ -612,9 +617,9 @@ pub mod drtio {
         let mut remote_data: Vec<u8> = Vec::new();
         loop {
             let reply = aux_transact(io, aux_mutex, ddma_mutex, subkernel_mutex, routing_table, linkno, 
-                &drtioaux::Packet::SubkernelExceptionRequest { destination: destination })?;
+                &drtioaux::Packet::SubkernelExceptionRequest { source: 0, destination: destination })?;
             match reply {
-                drtioaux::Packet::SubkernelException { last, length, data } => { 
+                drtioaux::Packet::SubkernelException { destination: 0, last, length, data } => { 
                     remote_data.extend(&data[0..length as usize]);
                     if last {
                         return Ok(remote_data);
@@ -710,11 +715,30 @@ fn read_device_map() -> DeviceMap {
     device_map
 }
 
+fn toggle_sed_spread(val: u8) {
+    unsafe { csr::rtio_core::sed_spread_enable_write(val); }
+}
+
+fn setup_sed_spread() {
+    config::read_str("sed_spread_enable", |r| {
+        match r {
+            Ok("1") => { info!("SED spreading enabled"); toggle_sed_spread(1); },
+            Ok("0") => { info!("SED spreading disabled"); toggle_sed_spread(0); },
+            Ok(_) => { 
+                warn!("sed_spread_enable value not supported (only 1, 0 allowed), disabling by default");
+                toggle_sed_spread(0);
+            },
+            Err(_) => { info!("SED spreading disabled by default"); toggle_sed_spread(0) },
+        }
+    });
+}
+
 pub fn startup(io: &Io, aux_mutex: &Mutex,
         routing_table: &Urc<RefCell<drtio_routing::RoutingTable>>,
         up_destinations: &Urc<RefCell<[bool; drtio_routing::DEST_COUNT]>>,
         ddma_mutex: &Mutex, subkernel_mutex: &Mutex) {
     set_device_map(read_device_map());
+    setup_sed_spread();
     drtio::startup(io, aux_mutex, routing_table, up_destinations, ddma_mutex, subkernel_mutex);
     unsafe {
         csr::rtio_core::reset_phy_write(1);

artiq/firmware/runtime/sched.rs

@@ -402,16 +402,18 @@ impl<'a> TcpListener<'a> {
             socket.may_send() || socket.may_recv()
         })?;
 
-        let accepted = self.handle.get();
+        let accepted = TcpStream {
+            io: self.io,
+            handle: self.handle.get(),
+        };
+        accepted.with_lower(|s| s.set_nagle_enabled(false));
+
         self.handle.set(Self::new_lower(self.io, self.buffer_size.get()));
         match self.listen(self.endpoint.get()) {
             Ok(()) => (),
             _ => unreachable!()
         }
-        Ok(TcpStream {
+        Ok(accepted)
-            io:     self.io,
-            handle: accepted
-        })
     }
 
     pub fn close(&self) {

artiq/firmware/runtime/session.rs

@@ -126,19 +126,6 @@ macro_rules! unexpected {
      ($($arg:tt)*) => (return Err(Error::Unexpected(format!($($arg)*))));
 }
 
-#[cfg(has_drtio)]
-macro_rules! propagate_subkernel_exception {
-    ( $exception:ident, $stream:ident ) => {
-        error!("Exception in subkernel");
-        match $stream {
-            None => return Ok(true),
-            Some(ref mut $stream) => { 
-                $stream.write_all($exception)?;
-            }
-        }
-    }
-}
-
 // Persistent state
 #[derive(Debug)]
 struct Congress {
@@ -674,9 +661,9 @@ fn process_kern_message(io: &Io, aux_mutex: &Mutex,
                 }
             }
             #[cfg(has_drtio)]
-            &kern::SubkernelLoadRunRequest { id, destination: _, run } => {
+            &kern::SubkernelLoadRunRequest { id, destination: _, run, timestamp } => {
                 let succeeded = match subkernel::load(
-                    io, aux_mutex, ddma_mutex, subkernel_mutex, routing_table, id, run) {
+                    io, aux_mutex, ddma_mutex, subkernel_mutex, routing_table, id, run, timestamp) {
                         Ok(()) => true,
                         Err(e) => { error!("Error loading subkernel: {}", e); false }
                     };
@@ -686,23 +673,26 @@ fn process_kern_message(io: &Io, aux_mutex: &Mutex,
             &kern::SubkernelAwaitFinishRequest{ id, timeout } => {
                 let res = subkernel::await_finish(io, aux_mutex, ddma_mutex, subkernel_mutex, routing_table,
                     id, timeout);
-                let status = match res {
+                let response = match res {
                     Ok(ref res) => {
                             if res.comm_lost {
-                                kern::SubkernelStatus::CommLost
+                                kern::SubkernelError(kern::SubkernelStatus::CommLost)
-                            } else if let Some(exception) = &res.exception {
+                            } else if let Some(raw_exception) = &res.exception {
-                                propagate_subkernel_exception!(exception, stream);
+                                let exception = subkernel::read_exception(raw_exception);
-                                // will not be called after exception is served
+                                if let Ok(exception) = exception {
-                                kern::SubkernelStatus::OtherError
+                                    kern::SubkernelError(kern::SubkernelStatus::Exception(exception))
                                 } else {
-                                kern::SubkernelStatus::NoError
+                                    kern::SubkernelError(kern::SubkernelStatus::OtherError)
+                                }
+                            } else {
+                                kern::SubkernelAwaitFinishReply
                             }
                         },
-                    Err(SubkernelError::Timeout) => kern::SubkernelStatus::Timeout,
+                    Err(SubkernelError::Timeout) => kern::SubkernelError(kern::SubkernelStatus::Timeout),
-                    Err(SubkernelError::IncorrectState) => kern::SubkernelStatus::IncorrectState,
+                    Err(SubkernelError::IncorrectState) => kern::SubkernelError(kern::SubkernelStatus::IncorrectState),
-                    Err(_) => kern::SubkernelStatus::OtherError
+                    Err(_) => kern::SubkernelError(kern::SubkernelStatus::OtherError)
                 };
-                kern_send(io, &kern::SubkernelAwaitFinishReply { status: status })
+                kern_send(io, &response)
             }
             #[cfg(has_drtio)]
             &kern::SubkernelMsgSend { id, destination, count, tag, data } => {
@@ -712,25 +702,34 @@ fn process_kern_message(io: &Io, aux_mutex: &Mutex,
             #[cfg(has_drtio)]
             &kern::SubkernelMsgRecvRequest { id, timeout, tags } => {
                 let message_received = subkernel::message_await(io, subkernel_mutex, id as u32, timeout);
-                let (status, count) = match message_received {
+                if let Err(SubkernelError::SubkernelFinished) = message_received {
-                    Ok(ref message) => (kern::SubkernelStatus::NoError, message.count),
-                    Err(SubkernelError::Timeout) => (kern::SubkernelStatus::Timeout, 0),
-                    Err(SubkernelError::IncorrectState) => (kern::SubkernelStatus::IncorrectState, 0),
-                    Err(SubkernelError::SubkernelFinished) => {
                     let res = subkernel::retrieve_finish_status(io, aux_mutex, ddma_mutex, subkernel_mutex,
                         routing_table, id as u32)?;
                     if res.comm_lost {
-                            (kern::SubkernelStatus::CommLost, 0)
+                        kern_send(io, 
-                        } else if let Some(exception) = &res.exception {
+                            &kern::SubkernelError(kern::SubkernelStatus::CommLost))?;
-                            propagate_subkernel_exception!(exception, stream);
+                    } else if let Some(raw_exception) = &res.exception {
-                            (kern::SubkernelStatus::OtherError, 0)
+                        let exception = subkernel::read_exception(raw_exception);
+                        if let Ok(exception) = exception {
+                            kern_send(io,
+                                &kern::SubkernelError(kern::SubkernelStatus::Exception(exception)))?;
                         } else {
-                            (kern::SubkernelStatus::OtherError, 0)
+                            kern_send(io,
+                                &kern::SubkernelError(kern::SubkernelStatus::OtherError))?;
                         }
+                    } else {
+                        kern_send(io,
+                            &kern::SubkernelError(kern::SubkernelStatus::OtherError))?;
                     }
-                    Err(_) => (kern::SubkernelStatus::OtherError, 0)
+                } else {
+                    let message = match message_received {
+                        Ok(ref message) => kern::SubkernelMsgRecvReply { count: message.count },
+                        Err(SubkernelError::Timeout) => kern::SubkernelError(kern::SubkernelStatus::Timeout),
+                        Err(SubkernelError::IncorrectState) => kern::SubkernelError(kern::SubkernelStatus::IncorrectState),
+                        Err(SubkernelError::SubkernelFinished) => unreachable!(), // taken care of above
+                        Err(_) => kern::SubkernelError(kern::SubkernelStatus::OtherError)
                     };
-                kern_send(io, &kern::SubkernelMsgRecvReply { status: status, count: count})?;
+                    kern_send(io, &message)?;
                     if let Ok(message) = message_received {
                         // receive code almost identical to RPC recv, except we are not reading from a stream
                         let mut reader = Cursor::new(message.data);
@@ -773,11 +772,10 @@ fn process_kern_message(io: &Io, aux_mutex: &Mutex,
                                 Err(_) => unexpected!("expected valid subkernel message data")
                             };
                         }
-                    Ok(())
-                } else {
-                    // if timed out, no data has been received, exception should be raised by kernel
-                    Ok(())
                     }
+                    // if timed out, no data has been received, exception should be raised by kernel
+                }
+                Ok(())
             },
 
             request => unexpected!("unexpected request {:?} from kernel CPU", request)

artiq/firmware/satman/kernel.rs

@@ -1,6 +1,6 @@
 use core::mem;
 use alloc::{string::String, format, vec::Vec, collections::btree_map::BTreeMap};
-use cslice::AsCSlice;
+use cslice::{CSlice, AsCSlice};
 
 use board_artiq::{drtioaux, drtio_routing::RoutingTable, mailbox, spi};
 use board_misoc::{csr, clock, i2c};
@@ -10,14 +10,13 @@ use proto_artiq::{
     session_proto::Reply::KernelException as HostKernelException, 
     rpc_proto as rpc};
 use eh::eh_artiq;
-use io::Cursor;
+use io::{Cursor, ProtoRead};
 use kernel::eh_artiq::StackPointerBacktrace;
 
 use ::{cricon_select, RtioMaster};
 use cache::Cache;
 use dma::{Manager as DmaManager, Error as DmaError};
 use routing::{Router, Sliceable, SliceMeta};
-use SAT_PAYLOAD_MAX_SIZE;
 use MASTER_PAYLOAD_MAX_SIZE;
 
 mod kernel_cpu {
@@ -69,6 +68,7 @@ enum KernelState {
     SubkernelAwaitFinish { max_time: i64, id: u32 },
     DmaUploading { max_time: u64 },
     DmaAwait { max_time: u64 },
+    SubkernelRetrievingException { destination: u8 },
 }
 
 #[derive(Debug)]
@@ -134,10 +134,13 @@ struct MessageManager {
 struct Session {
     kernel_state: KernelState,
     log_buffer: String,
-    last_exception: Option<Sliceable>,
+    last_exception: Option<Sliceable>,  // exceptions raised locally
-    source: u8, // which destination requested running the kernel
+    external_exception: Vec<u8>,  // exceptions from sub-subkernels
+    // which destination requested running the kernel
+    source: u8,
     messages: MessageManager,
-    subkernels_finished: Vec<u32> // ids of subkernels finished
+    // ids of subkernels finished (with exception)
+    subkernels_finished: Vec<(u32, Option<u8>)>,
 }
 
 #[derive(Debug)]
@@ -277,6 +280,7 @@ impl Session {
             kernel_state: KernelState::Absent,
             log_buffer: String::new(),
             last_exception: None,
+            external_exception: Vec::new(),
             source: 0,
             messages: MessageManager::new(),
             subkernels_finished: Vec::new()
@@ -359,7 +363,7 @@ impl Manager {
         unsafe { self.cache.unborrow() }
     }
 
-    pub fn run(&mut self, source: u8, id: u32) -> Result<(), Error> {
+    pub fn run(&mut self, source: u8, id: u32, timestamp: u64) -> Result<(), Error> {
         info!("starting subkernel #{}", id);
         if self.session.kernel_state != KernelState::Loaded
             || self.current_id != id {
@@ -369,7 +373,7 @@ impl Manager {
         self.session.kernel_state = KernelState::Running;
         cricon_select(RtioMaster::Kernel);
     
-        kern_acknowledge()
+        kern_send(&kern::UpdateNow(timestamp))
     }
 
     pub fn message_handle_incoming(&mut self, status: PayloadStatus, length: usize, id: u32, slice: &[u8; MASTER_PAYLOAD_MAX_SIZE]) {
@@ -428,9 +432,9 @@ impl Manager {
         }
     }
 
-    pub fn exception_get_slice(&mut self, data_slice: &mut [u8; SAT_PAYLOAD_MAX_SIZE]) -> SliceMeta {
+    pub fn exception_get_slice(&mut self, data_slice: &mut [u8; MASTER_PAYLOAD_MAX_SIZE]) -> SliceMeta {
         match self.session.last_exception.as_mut() {
-            Some(exception) => exception.get_slice_sat(data_slice),
+            Some(exception) => exception.get_slice_master(data_slice),
             None => SliceMeta { destination: 0, len: 0, status: PayloadStatus::FirstAndLast }
         }
     }
@@ -517,7 +521,7 @@ impl Manager {
             return;
         }
 
-        match self.process_external_messages() {
+        match self.process_external_messages(router, routing_table, rank, destination) {
             Ok(()) => (),
             Err(Error::AwaitingMessage) => return, // kernel still waiting, do not process kernel messages
             Err(Error::KernelException(exception)) => {
@@ -549,20 +553,42 @@ impl Manager {
         }
     }
 
-    fn process_external_messages(&mut self) -> Result<(), Error> {
+    fn check_finished_kernels(&mut self, id: u32, router: &mut Router, routing_table: &RoutingTable, rank: u8, self_destination: u8) {
+        for (i, (status, exception_source)) in self.session.subkernels_finished.iter().enumerate() {
+            if *status == id {
+                if exception_source.is_none() {
+                    kern_send(&kern::SubkernelAwaitFinishReply).unwrap();
+                    self.session.kernel_state = KernelState::Running;
+                    self.session.subkernels_finished.swap_remove(i);
+                } else {
+                    let destination = exception_source.unwrap();
+                    self.session.external_exception = Vec::new();
+                    self.session.kernel_state = KernelState::SubkernelRetrievingException { destination: destination };
+                    router.route(drtioaux::Packet::SubkernelExceptionRequest {
+                        source: self_destination, destination: destination
+                    }, &routing_table, rank, self_destination);
+                }
+                break;
+            }
+        }
+    }
+
+    fn process_external_messages(&mut self, router: &mut Router, routing_table: &RoutingTable, rank: u8, self_destination: u8) -> Result<(), Error> {
         match &self.session.kernel_state {
             KernelState::MsgAwait { id, max_time, tags } => {
                 if *max_time > 0 && clock::get_ms() > *max_time as u64 {
-                    kern_send(&kern::SubkernelMsgRecvReply { status: kern::SubkernelStatus::Timeout, count: 0 })?;
+                    kern_send(&kern::SubkernelError(kern::SubkernelStatus::Timeout))?;
                     self.session.kernel_state = KernelState::Running;
                     return Ok(())
                 }
                 if let Some(message) = self.session.messages.get_incoming(*id) {
-                    kern_send(&kern::SubkernelMsgRecvReply { status: kern::SubkernelStatus::NoError, count: message.count })?;
+                    kern_send(&kern::SubkernelMsgRecvReply { count: message.count })?;
                     let tags = tags.clone();
                     self.session.kernel_state = KernelState::Running;
                     pass_message_to_kernel(&message, &tags)
                 } else {
+                    let id = *id;
+                    self.check_finished_kernels(id, router, routing_table, rank, self_destination);
                     Err(Error::AwaitingMessage)
                 }
             },
@@ -576,19 +602,11 @@ impl Manager {
             },
             KernelState::SubkernelAwaitFinish { max_time, id } => {
                 if *max_time > 0 && clock::get_ms() > *max_time as u64 {
-                    kern_send(&kern::SubkernelAwaitFinishReply { status: kern::SubkernelStatus::Timeout })?;
+                    kern_send(&kern::SubkernelError(kern::SubkernelStatus::Timeout))?;
                     self.session.kernel_state = KernelState::Running;
                 } else {
-                    let mut i = 0;
+                    let id = *id;
-                    for status in &self.session.subkernels_finished {
+                    self.check_finished_kernels(id, router, routing_table, rank, self_destination);
-                        if *status == *id {
-                            kern_send(&kern::SubkernelAwaitFinishReply { status: kern::SubkernelStatus::NoError })?;
-                            self.session.kernel_state = KernelState::Running;
-                            self.session.subkernels_finished.swap_remove(i);
-                            break;
-                        }
-                        i += 1;
-                    }
                 }
                 Ok(())
             }
@@ -606,6 +624,9 @@ impl Manager {
                 }
                 Ok(())
             }
+            KernelState::SubkernelRetrievingException { destination: _ } => {
+                Err(Error::AwaitingMessage)
+            }
             _ => Ok(())
         }
     }
@@ -628,16 +649,30 @@ impl Manager {
     }
 
     pub fn remote_subkernel_finished(&mut self, id: u32, with_exception: bool, exception_source: u8) {
-        if with_exception {
+        let exception_src = if with_exception { Some(exception_source) } else { None };
-            unsafe { kernel_cpu::stop() }
+        self.session.subkernels_finished.push((id, exception_src));
-            self.session.kernel_state = KernelState::Absent;
+    }
-            unsafe { self.cache.unborrow() }
+
-            self.last_finished = Some(SubkernelFinished {
+    pub fn received_exception(&mut self, exception_data: &[u8], last: bool, router: &mut Router, routing_table: &RoutingTable,
-                source: self.session.source, id: self.current_id,
+        rank: u8, self_destination: u8) {
-                with_exception: true, exception_source: exception_source
+        if let KernelState::SubkernelRetrievingException { destination } = self.session.kernel_state {
-            })
+            self.session.external_exception.extend_from_slice(exception_data);
+            if last {
+                if let Ok(exception) = read_exception(&self.session.external_exception) {
+                    kern_send(&kern::SubkernelError(kern::SubkernelStatus::Exception(exception))).unwrap();
                 } else {
-            self.session.subkernels_finished.push(id);
+                    kern_send(
+                        &kern::SubkernelError(kern::SubkernelStatus::OtherError)).unwrap();
+                }
+                self.session.kernel_state = KernelState::Running;
+            } else {
+                /* fetch another slice */
+                router.route(drtioaux::Packet::SubkernelExceptionRequest {
+                    source: self_destination, destination: destination
+                }, routing_table, rank, self_destination);
+            }
+        } else {
+            warn!("Received unsolicited exception data");
         }
     }
 
@@ -655,6 +690,7 @@ impl Manager {
                     (_, KernelState::DmaAwait { .. }) |
                     (_, KernelState::MsgSending) |
                     (_, KernelState::SubkernelAwaitLoad) | 
+                    (_, KernelState::SubkernelRetrievingException { .. }) |
                     (_, KernelState::SubkernelAwaitFinish { .. }) => {
                     // We're standing by; ignore the message.
                     return Ok(None)
@@ -789,21 +825,21 @@ impl Manager {
                     // ID equal to -1 indicates wildcard for receiving arguments
                     let id = if id == -1 { self.current_id } else { id as u32 };
                     self.session.kernel_state = KernelState::MsgAwait { 
-                        id: id, max_time: max_time, tags: tags.to_vec() };
+                        id, max_time, tags: tags.to_vec() };
                     Ok(())
                 },
 
-                &kern::SubkernelLoadRunRequest { id, destination: sk_destination, run } => {
+                &kern::SubkernelLoadRunRequest { id, destination: sk_destination, run, timestamp } => {
                     self.session.kernel_state = KernelState::SubkernelAwaitLoad;
                     router.route(drtioaux::Packet::SubkernelLoadRunRequest { 
-                        source: destination, destination: sk_destination, id: id, run: run 
+                        source: destination, destination: sk_destination, id, run, timestamp
                     }, routing_table, rank, destination);
                     Ok(())
                 }
 
-                &kern::SubkernelAwaitFinishRequest{ id, timeout } => {
+                &kern::SubkernelAwaitFinishRequest { id, timeout } => {
                     let max_time = if timeout > 0 { clock::get_ms() as i64 + timeout } else { timeout };
-                    self.session.kernel_state = KernelState::SubkernelAwaitFinish { max_time: max_time, id: id };
+                    self.session.kernel_state = KernelState::SubkernelAwaitFinish { max_time, id };
                     Ok(())
                 }
 
@@ -822,6 +858,48 @@ impl Drop for Manager {
     }
 }
 
+fn read_exception_string<'a>(reader: &mut Cursor<&[u8]>) -> Result<CSlice<'a, u8>, Error> {
+    let len = reader.read_u32()? as usize;
+    if len == usize::MAX {
+        let data = reader.read_u32()?;
+        Ok(unsafe { CSlice::new(data as *const u8, len) })
+    } else {
+        let pos = reader.position();
+        let slice = unsafe {
+            let ptr = reader.get_ref().as_ptr().offset(pos as isize);
+            CSlice::new(ptr, len)
+        };
+        reader.set_position(pos + len);
+        Ok(slice)
+    }
+}
+
+fn read_exception(buffer: &[u8]) -> Result<eh_artiq::Exception, Error>
+{
+    let mut reader = Cursor::new(buffer);
+
+    let mut byte = reader.read_u8()?;
+    // to sync
+    while byte != 0x5a {
+        byte = reader.read_u8()?;
+    }
+    // skip sync bytes, 0x09 indicates exception
+    while byte != 0x09 {
+        byte = reader.read_u8()?;
+    }
+    let _len = reader.read_u32()?;
+    // ignore the remaining exceptions, stack traces etc. - unwinding from another device would be unwise anyway
+    Ok(eh_artiq::Exception {
+        id:       reader.read_u32()?,
+        message:  read_exception_string(&mut reader)?,
+        param:    [reader.read_u64()? as i64, reader.read_u64()? as i64, reader.read_u64()? as i64],
+        file:     read_exception_string(&mut reader)?,
+        line:     reader.read_u32()?,
+        column:   reader.read_u32()?,
+        function: read_exception_string(&mut reader)?
+    })
+}
+
 fn kern_recv<R, F>(f: F) -> Result<R, Error>
         where F: FnOnce(&kern::Message) -> Result<R, Error> {
     if mailbox::receive() == 0 {

artiq/firmware/satman/main.rs

@@ -14,7 +14,7 @@ extern crate io;
 extern crate eh;
 
 use core::convert::TryFrom;
-use board_misoc::{csr, ident, clock, uart_logger, i2c, pmp};
+use board_misoc::{csr, ident, clock, config, uart_logger, i2c, pmp};
 #[cfg(has_si5324)]
 use board_artiq::si5324;
 #[cfg(has_si549)]
@@ -70,6 +70,10 @@ fn drtiosat_tsc_loaded() -> bool {
     }
 }
 
+fn toggle_sed_spread(val: u8) {
+    unsafe { csr::drtiosat::sed_spread_enable_write(val); }
+}
+
 #[derive(Clone, Copy)]
 pub enum RtioMaster {
     Drtio,
@@ -100,13 +104,13 @@ pub fn cricon_read() -> RtioMaster {
 
 #[cfg(has_drtio_routing)]
 macro_rules! forward {
-    ($routing_table:expr, $destination:expr, $rank:expr, $repeaters:expr, $packet:expr) => {{
+    ($router:expr, $routing_table:expr, $destination:expr, $rank:expr, $self_destination:expr, $repeaters:expr, $packet:expr) => {{
         let hop = $routing_table.0[$destination as usize][$rank as usize];
         if hop != 0 {
             let repno = (hop - 1) as usize;
             if repno < $repeaters.len() {
                 if $packet.expects_response() {
-                    return $repeaters[repno].aux_forward($packet);
+                    return $repeaters[repno].aux_forward($packet, $router, $routing_table, $rank, $self_destination);
                 } else {
                     let res = $repeaters[repno].aux_send($packet);
                     // allow the satellite to parse the packet before next
@@ -122,7 +126,7 @@ macro_rules! forward {
 
 #[cfg(not(has_drtio_routing))]
 macro_rules! forward {
-    ($routing_table:expr, $destination:expr, $rank:expr, $repeaters:expr, $packet:expr) => {}
+    ($router:expr, $routing_table:expr, $destination:expr, $rank:expr, $self_destination:expr, $repeaters:expr, $packet:expr) => {}
 }
 
 fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmgr: &mut KernelManager,
@@ -197,7 +201,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
                         let repno = hop - 1;
                         match _repeaters[repno].aux_forward(&drtioaux::Packet::DestinationStatusRequest {
                             destination: destination
-                        }) {
+                        }, router, _routing_table, *rank, *self_destination) {
                             Ok(()) => (),
                             Err(drtioaux::Error::LinkDown) => drtioaux::send(0, &drtioaux::Packet::DestinationDownReply)?,
                             Err(e) => {
@@ -251,7 +255,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
         }
 
         drtioaux::Packet::MonitorRequest { destination: _destination, channel, probe } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let value;
             #[cfg(has_rtio_moninj)]
             unsafe {
@@ -268,7 +272,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             drtioaux::send(0, &reply)
         },
         drtioaux::Packet::InjectionRequest { destination: _destination, channel, overrd, value } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             #[cfg(has_rtio_moninj)]
             unsafe {
                 csr::rtio_moninj::inj_chan_sel_write(channel as _);
@@ -278,7 +282,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             Ok(())
         },
         drtioaux::Packet::InjectionStatusRequest { destination: _destination, channel, overrd } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let value;
             #[cfg(has_rtio_moninj)]
             unsafe {
@@ -294,22 +298,22 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
         },
 
         drtioaux::Packet::I2cStartRequest { destination: _destination, busno } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = i2c::start(busno).is_ok();
             drtioaux::send(0, &drtioaux::Packet::I2cBasicReply { succeeded: succeeded })
         }
         drtioaux::Packet::I2cRestartRequest { destination: _destination, busno } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = i2c::restart(busno).is_ok();
             drtioaux::send(0, &drtioaux::Packet::I2cBasicReply { succeeded: succeeded })
         }
         drtioaux::Packet::I2cStopRequest { destination: _destination, busno } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = i2c::stop(busno).is_ok();
             drtioaux::send(0, &drtioaux::Packet::I2cBasicReply { succeeded: succeeded })
         }
         drtioaux::Packet::I2cWriteRequest { destination: _destination, busno, data } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             match i2c::write(busno, data) {
                 Ok(ack) => drtioaux::send(0,
                     &drtioaux::Packet::I2cWriteReply { succeeded: true, ack: ack }),
@@ -318,7 +322,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             }
         }
         drtioaux::Packet::I2cReadRequest { destination: _destination, busno, ack } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             match i2c::read(busno, ack) {
                 Ok(data) => drtioaux::send(0,
                     &drtioaux::Packet::I2cReadReply { succeeded: true, data: data }),
@@ -327,25 +331,25 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             }
         }
         drtioaux::Packet::I2cSwitchSelectRequest { destination: _destination, busno, address, mask } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = i2c::switch_select(busno, address, mask).is_ok();
             drtioaux::send(0, &drtioaux::Packet::I2cBasicReply { succeeded: succeeded })
         }
 
         drtioaux::Packet::SpiSetConfigRequest { destination: _destination, busno, flags, length, div, cs } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = spi::set_config(busno, flags, length, div, cs).is_ok();
             drtioaux::send(0,
                 &drtioaux::Packet::SpiBasicReply { succeeded: succeeded })
         },
         drtioaux::Packet::SpiWriteRequest { destination: _destination, busno, data } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = spi::write(busno, data).is_ok();
             drtioaux::send(0,
                 &drtioaux::Packet::SpiBasicReply { succeeded: succeeded })
         }
         drtioaux::Packet::SpiReadRequest { destination: _destination, busno } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             match spi::read(busno) {
                 Ok(data) => drtioaux::send(0,
                     &drtioaux::Packet::SpiReadReply { succeeded: true, data: data }),
@@ -355,7 +359,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
         }
 
         drtioaux::Packet::AnalyzerHeaderRequest { destination: _destination } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let header = analyzer.get_header();
             drtioaux::send(0, &drtioaux::Packet::AnalyzerHeader {
                 total_byte_count: header.total_byte_count,
@@ -365,7 +369,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
         }
 
         drtioaux::Packet::AnalyzerDataRequest { destination: _destination } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let mut data_slice: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
             let meta = analyzer.get_data(&mut data_slice);
             drtioaux::send(0, &drtioaux::Packet::AnalyzerData {
@@ -376,7 +380,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
         }
 
         drtioaux::Packet::DmaAddTraceRequest { source, destination, id, status, length, trace } => {
-            forward!(_routing_table, destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, destination, *rank, *self_destination, _repeaters, &packet);
             *self_destination = destination;
             let succeeded = dmamgr.add(source, id, status, &trace, length as usize).is_ok();
             router.send(drtioaux::Packet::DmaAddTraceReply { 
@@ -384,19 +388,19 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             }, _routing_table, *rank, *self_destination)
         }
         drtioaux::Packet::DmaAddTraceReply { source, destination: _destination, id, succeeded } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             dmamgr.ack_upload(kernelmgr, source, id, succeeded, router, *rank, *self_destination, _routing_table);
             Ok(())
         }
         drtioaux::Packet::DmaRemoveTraceRequest { source, destination: _destination, id } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let succeeded = dmamgr.erase(source, id).is_ok();
             router.send(drtioaux::Packet::DmaRemoveTraceReply { 
                 destination: source, succeeded: succeeded 
             }, _routing_table, *rank, *self_destination)
         }
         drtioaux::Packet::DmaPlaybackRequest { source, destination: _destination, id, timestamp } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             // no DMA with a running kernel
             let succeeded = !kernelmgr.is_running() && dmamgr.playback(source, id, timestamp).is_ok();
             router.send(drtioaux::Packet::DmaPlaybackReply { 
@@ -404,27 +408,27 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             }, _routing_table, *rank, *self_destination)
         }
         drtioaux::Packet::DmaPlaybackReply { destination: _destination, succeeded } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             if !succeeded {
                 kernelmgr.ddma_nack();
             }
             Ok(())
         }
         drtioaux::Packet::DmaPlaybackStatus { source: _, destination: _destination, id, error, channel, timestamp } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             dmamgr.remote_finished(kernelmgr, id, error, channel, timestamp);
             Ok(())
         }
 
         drtioaux::Packet::SubkernelAddDataRequest { destination, id, status, length, data } => {
-            forward!(_routing_table, destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, destination, *rank, *self_destination, _repeaters, &packet);
             *self_destination = destination;
             let succeeded = kernelmgr.add(id, status, &data, length as usize).is_ok();
             drtioaux::send(0,
                 &drtioaux::Packet::SubkernelAddDataReply { succeeded: succeeded })
         }
-        drtioaux::Packet::SubkernelLoadRunRequest { source, destination: _destination, id, run } => {
+        drtioaux::Packet::SubkernelLoadRunRequest { source, destination: _destination, id, run, timestamp } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             let mut succeeded = kernelmgr.load(id).is_ok();
             // allow preloading a kernel with delayed run
             if run {
@@ -432,7 +436,7 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
                     // cannot run kernel while DDMA is running
                     succeeded = false;
                 } else {
-                    succeeded |= kernelmgr.run(source, id).is_ok();
+                    succeeded |= kernelmgr.run(source, id, timestamp).is_ok();
                 }
             }
             router.send(drtioaux::Packet::SubkernelLoadRunReply { 
@@ -441,35 +445,41 @@ fn process_aux_packet(dmamgr: &mut DmaManager, analyzer: &mut Analyzer, kernelmg
             _routing_table, *rank, *self_destination)
         }
         drtioaux::Packet::SubkernelLoadRunReply { destination: _destination, succeeded } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             // received if local subkernel started another, remote subkernel
             kernelmgr.subkernel_load_run_reply(succeeded, *self_destination);
             Ok(())
         }
         drtioaux::Packet::SubkernelFinished { destination: _destination, id, with_exception, exception_src } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             kernelmgr.remote_subkernel_finished(id, with_exception, exception_src);
             Ok(())
         }
-        drtioaux::Packet::SubkernelExceptionRequest { destination: _destination } => {
+        drtioaux::Packet::SubkernelExceptionRequest { source, destination: _destination } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
-            let mut data_slice: [u8; SAT_PAYLOAD_MAX_SIZE] = [0; SAT_PAYLOAD_MAX_SIZE];
+            let mut data_slice: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
             let meta = kernelmgr.exception_get_slice(&mut data_slice);
-            drtioaux::send(0, &drtioaux::Packet::SubkernelException {
+            router.send(drtioaux::Packet::SubkernelException {
+                destination: source,
                 last: meta.status.is_last(),
                 length: meta.len,
                 data: data_slice,
-            })
+            }, _routing_table, *rank, *self_destination)
+        }
+        drtioaux::Packet::SubkernelException { destination: _destination, last, length, data } => {
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
+            kernelmgr.received_exception(&data[..length as usize], last, router, _routing_table, *rank, *self_destination);
+            Ok(())
         }
         drtioaux::Packet::SubkernelMessage { source, destination: _destination, id, status, length, data } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             kernelmgr.message_handle_incoming(status, length as usize, id, &data);
             router.send(drtioaux::Packet::SubkernelMessageAck {
                     destination: source
                 }, _routing_table, *rank, *self_destination)
         }
         drtioaux::Packet::SubkernelMessageAck { destination: _destination } => {
-            forward!(_routing_table, _destination, *rank, _repeaters, &packet);
+            forward!(router, _routing_table, _destination, *rank, *self_destination, _repeaters, &packet);
             if kernelmgr.message_ack_slice() {
                 let mut data_slice: [u8; MASTER_PAYLOAD_MAX_SIZE] = [0; MASTER_PAYLOAD_MAX_SIZE];
                 if let Some(meta) = kernelmgr.message_get_slice(&mut data_slice) {
@@ -742,7 +752,7 @@ pub extern fn main() -> i32 {
         io_expander.service().unwrap();
     }
 
-    #[cfg(not(soc_platform = "efc"))]
+    #[cfg(not(has_drtio_eem))]
     unsafe {
         csr::gt_drtio::txenable_write(0xffffffffu32 as _);
     }
@@ -754,6 +764,18 @@ pub extern fn main() -> i32 {
 
     init_rtio_crg();
 
+    config::read_str("sed_spread_enable", |r| {
+        match r {
+            Ok("1") => { info!("SED spreading enabled"); toggle_sed_spread(1); },
+            Ok("0") => { info!("SED spreading disabled"); toggle_sed_spread(0); },
+            Ok(_) => { 
+                warn!("sed_spread_enable value not supported (only 1, 0 allowed), disabling by default");
+                toggle_sed_spread(0);
+            },
+            Err(_) => { info!("SED spreading disabled by default"); toggle_sed_spread(0) },
+        }
+    });
+
     #[cfg(has_drtio_eem)]
     drtio_eem::init();
 

artiq/firmware/satman/repeater.rs

@@ -75,6 +75,11 @@ impl Repeater {
                 if rep_link_rx_up(self.repno) {
                     if let Ok(Some(drtioaux::Packet::EchoReply)) = drtioaux::recv(self.auxno) {
                         info!("[REP#{}] remote replied after {} packets", self.repno, ping_count);
+                        // clear the aux buffer
+                        let max_time = clock::get_ms() + 200;
+                        while clock::get_ms() < max_time {
+                            let _ = drtioaux::recv(self.auxno);
+                        }
                         self.state = RepeaterState::Up;
                         if let Err(e) = self.sync_tsc() {
                             error!("[REP#{}] failed to sync TSC ({})", self.repno, e);
@@ -181,10 +186,31 @@ impl Repeater {
         }
     }
 
-    pub fn aux_forward(&self, request: &drtioaux::Packet) -> Result<(), drtioaux::Error<!>> {
+    pub fn aux_forward(&self, request: &drtioaux::Packet, router: &mut Router, 
+        routing_table: &drtio_routing::RoutingTable, rank: u8,
+        self_destination: u8) -> Result<(), drtioaux::Error<!>> {
         self.aux_send(request)?;
+        loop {
             let reply = self.recv_aux_timeout(200)?;
+            match reply {
+                // async/locally requested packets to be consumed or routed
+                // these may come while a packet would be forwarded
+                drtioaux::Packet::DmaPlaybackStatus { .. } |
+                drtioaux::Packet::SubkernelFinished { .. } |
+                drtioaux::Packet::SubkernelMessage  { .. } | 
+                drtioaux::Packet::SubkernelMessageAck { .. } | 
+                drtioaux::Packet::SubkernelLoadRunReply { .. } |
+                drtioaux::Packet::SubkernelException { .. } |
+                drtioaux::Packet::DmaAddTraceReply { .. } |
+                drtioaux::Packet::DmaPlaybackReply { .. } => {
+                    router.route(reply, routing_table, rank, self_destination);
+                }
+                _ => {
                     drtioaux::send(0, &reply).unwrap();
+                    break;
+                }
+            }
+        }
         Ok(())
     }
 

artiq/firmware/satman/routing.rs

@@ -4,7 +4,6 @@ use board_artiq::{drtioaux, drtio_routing};
 use board_misoc::csr;
 use core::cmp::min;
 use proto_artiq::drtioaux_proto::PayloadStatus;
-use SAT_PAYLOAD_MAX_SIZE;
 use MASTER_PAYLOAD_MAX_SIZE;
 
 /* represents data that has to be sent with the aux protocol */
@@ -57,7 +56,6 @@ impl Sliceable {
         self.data.extend(data);
     }
 
-    get_slice_fn!(get_slice_sat, SAT_PAYLOAD_MAX_SIZE);
     get_slice_fn!(get_slice_master, MASTER_PAYLOAD_MAX_SIZE);
 }
 

artiq/frontend/afws_client.py

@@ -158,11 +158,11 @@ class Client:
         if reply[0] != "OK":
             return reply[0], None
         length = int(reply[1])
-        json_str = self.fsocket.read(length).decode("ascii")
+        json_bytes = self.fsocket.read(length)
-        return "OK", json_str
+        return "OK", json_bytes
 
 
-def main():
+def get_argparser():
     parser = argparse.ArgumentParser()
     parser.add_argument("--server", default="afws.m-labs.hk", help="server to connect to (default: %(default)s)")
     parser.add_argument("--port", default=80, type=int, help="port to connect to (default: %(default)d)")
@@ -183,8 +183,11 @@ def main():
     act_get_json.add_argument("variant", nargs="?", default=None, help="variant to get (can be omitted if user is authorised to build only one)")
     act_get_json.add_argument("-o", "--out", default=None, help="output JSON file")
     act_get_json.add_argument("-f", "--force", action="store_true", help="overwrite file if it already exists")
-    args = parser.parse_args()
+    return parser
 
+
+def main():
+    args = get_argparser().parse_args()
     client = Client(args.server, args.port, args.cert)
     try:
         if args.action == "build":
@@ -256,7 +259,7 @@ def main():
                 variant = args.variant
             else:
                 variant = client.get_single_variant(error_msg="User can get JSON of more than 1 variant - need to specify")
-            result, json_str = client.get_json(variant)
+            result, json_bytes = client.get_json(variant)
             if result != "OK":
                 if result == "UNAUTHORIZED":
                     print(f"You are not authorized to get JSON of variant {variant}. Your firmware subscription may have expired. Contact helpdesk\x40m-labs.hk.")
@@ -265,10 +268,10 @@ def main():
                 if not args.force and os.path.exists(args.out):
                     print(f"File {args.out} already exists. You can use -f to overwrite the existing file.")
                     sys.exit(1)
-                with open(args.out, "w") as f:
+                with open(args.out, "wb") as f:
-                    f.write(json_str)
+                    f.write(json_bytes)
             else:
-                print(json_str)
+                sys.stdout.buffer.write(json_bytes)
         else:
             raise ValueError
     finally:

artiq/frontend/artiq_client.py

@@ -1,10 +1,4 @@
 #!/usr/bin/env python3
-"""
-Client to send commands to :mod:`artiq_master` and display results locally.
-
-The client can perform actions such as accessing/setting datasets,
-scanning devices, scheduling experiments, and looking for experiments/devices.
-"""
 
 import argparse
 import logging
@@ -40,10 +34,10 @@ def get_argparser():
     parser = argparse.ArgumentParser(description="ARTIQ CLI client")
     parser.add_argument(
         "-s", "--server", default="::1",
-        help="hostname or IP of the master to connect to")
+        help="hostname or IP of the master to connect to (default: %(default)s)")
     parser.add_argument(
         "--port", default=None, type=int,
-        help="TCP port to use to connect to the master")
+        help="TCP port to use to connect to the master (default: %(default)s)")
     parser.add_argument("--version", action="version",
                         version="ARTIQ v{}".format(artiq_version),
                         help="print the ARTIQ version number")
@@ -59,7 +53,8 @@ def get_argparser():
                             help="priority (higher value means sooner "
                                  "scheduling, default: %(default)s)")
     parser_add.add_argument("-t", "--timed", default=None, type=str,
-                            help="set a due date for the experiment")
+                            help="set a due date for the experiment "
+                            "(default: %(default)s)")
     parser_add.add_argument("-f", "--flush", default=False,
                             action="store_true",
                             help="flush the pipeline before preparing "
@@ -80,7 +75,7 @@ def get_argparser():
     parser_add.add_argument("file", metavar="FILE",
                             help="file containing the experiment to run")
     parser_add.add_argument("arguments", metavar="ARGUMENTS", nargs="*",
-                            help="run arguments")
+                            help="run arguments, use format KEY=VALUE")
 
     parser_delete = subparsers.add_parser("delete",
                                           help="delete an experiment "

artiq/frontend/artiq_dashboard.py

@@ -32,31 +32,31 @@ def get_argparser():
                         help="print the ARTIQ version number")
     parser.add_argument(
         "-s", "--server", default="::1",
-        help="hostname or IP of the master to connect to")
+        help="hostname or IP of the master to connect to (default: %(default)s)")
     parser.add_argument(
         "--port-notify", default=3250, type=int,
-        help="TCP port to connect to for notifications")
+        help="TCP port to connect to for notifications (default: %(default)s)")
     parser.add_argument(
         "--port-control", default=3251, type=int,
-        help="TCP port to connect to for control")
+        help="TCP port to connect to for control (default: %(default)s)")
     parser.add_argument(
         "--port-broadcast", default=1067, type=int,
-        help="TCP port to connect to for broadcasts")
+        help="TCP port to connect to for broadcasts (default: %(default)s)")
     parser.add_argument(
         "--db-file", default=None,
-        help="database file for local GUI settings")
+        help="database file for local GUI settings (default: %(default)s)")
     parser.add_argument(
         "-p", "--load-plugin", dest="plugin_modules", action="append",
         help="Python module to load on startup")
     parser.add_argument(
         "--analyzer-proxy-timeout", default=5, type=float,
-        help="connection timeout to core analyzer proxy")
+        help="connection timeout to core analyzer proxy (default: %(default)s)")
     parser.add_argument(
         "--analyzer-proxy-timer", default=5, type=float,
-        help="retry timer to core analyzer proxy")
+        help="retry timer to core analyzer proxy (default: %(default)s)")
     parser.add_argument(
         "--analyzer-proxy-timer-backoff", default=1.1, type=float,
-        help="retry timer backoff multiplier to core analyzer proxy")
+        help="retry timer backoff multiplier to core analyzer proxy, (default: %(default)s)")
     common_args.verbosity_args(parser)
     return parser
 

artiq/frontend/artiq_ddb_template.py

@@ -833,7 +833,7 @@ def process(output, primary_description, satellites):
         processor(peripheral)
 
 
-def main():
+def get_argparser(): 
     parser = argparse.ArgumentParser(
         description="ARTIQ device database template builder")
     parser.add_argument("--version", action="version",
@@ -847,8 +847,11 @@ def main():
                         default=[], metavar=("DESTINATION", "DESCRIPTION"), type=str,
                         help="add DRTIO satellite at the given destination number with "
                              "devices from the given JSON description")
+    return parser
 
-    args = parser.parse_args()
+
+def main():
+    args = get_argparser().parse_args()
 
     primary_description = jsondesc.load(args.primary_description)
 

artiq/frontend/artiq_flash.py

@@ -30,7 +30,8 @@ Valid actions:
     * firmware: write firmware to flash
     * load: load main gateware bitstream into device (volatile but fast)
     * erase: erase flash memory
-    * start: trigger the target to (re)load its gateware bitstream from flash
+    * start: trigger the target to (re)load its gateware bitstream from flash.
+      If your core device is reachable by network, prefer 'artiq_coremgmt reboot'. 
 
 Prerequisites:
 

artiq/frontend/artiq_master.py

@@ -40,21 +40,21 @@ def get_argparser():
 
     group = parser.add_argument_group("databases")
     group.add_argument("--device-db", default="device_db.py",
-                       help="device database file (default: '%(default)s')")
+                       help="device database file (default: %(default)s)")
     group.add_argument("--dataset-db", default="dataset_db.mdb",
-                       help="dataset file (default: '%(default)s')")
+                       help="dataset file (default: %(default)s)")
 
     group = parser.add_argument_group("repository")
     group.add_argument(
         "-g", "--git", default=False, action="store_true",
-        help="use the Git repository backend")
+        help="use the Git repository backend (default: %(default)s)")
     group.add_argument(
         "-r", "--repository", default="repository",
-        help="path to the repository (default: '%(default)s')")
+        help="path to the repository (default: %(default)s)")
     group.add_argument(
         "--experiment-subdir", default="",
         help=("path to the experiment folder from the repository root "
-              "(default: '%(default)s')"))
+              "(default: %(default)s)"))
     log_args(parser)
 
     parser.add_argument("--name",

artiq/frontend/artiq_run.py

@@ -161,7 +161,7 @@ def get_argparser(with_file=True):
         parser.add_argument("file", metavar="FILE",
                             help="file containing the experiment to run")
     parser.add_argument("arguments", metavar="ARGUMENTS", nargs="*",
-                        help="run arguments")
+                        help="run arguments, use format KEY=VALUE")
 
     return parser
 

artiq/frontend/artiq_session.py

@@ -11,7 +11,7 @@ def get_argparser():
         description="ARTIQ session manager. "
                     "Automatically runs the master, dashboard and "
                     "local controller manager on the current machine. "
-                    "The latter requires the artiq-comtools package to "
+                    "The latter requires the ``artiq-comtools`` package to "
                     "be installed.")
     parser.add_argument("--version", action="version",
                         version="ARTIQ v{}".format(artiq_version),

artiq/gateware/drtio/core.py

@@ -1,6 +1,7 @@
 from types import SimpleNamespace
 
 from migen import *
+from migen.genlib.cdc import MultiReg
 from migen.genlib.resetsync import AsyncResetSynchronizer
 from misoc.interconnect.csr import *
 
@@ -51,6 +52,7 @@ class SyncRTIO(Module):
     def __init__(self, tsc, channels, lane_count=8, fifo_depth=128):
         self.cri = cri.Interface()
         self.async_errors = Record(async_errors_layout)
+        self.sed_spread_enable = Signal()
 
         chan_fine_ts_width = max(max(rtlink.get_fine_ts_width(channel.interface.o)
                                      for channel in channels),
@@ -61,10 +63,11 @@ class SyncRTIO(Module):
         self.submodules.outputs = ClockDomainsRenamer("rio")(
             SED(channels, tsc.glbl_fine_ts_width,
                 lane_count=lane_count, fifo_depth=fifo_depth,
-                enable_spread=True, fifo_high_watermark=0.75,
+                fifo_high_watermark=0.75, report_buffer_space=True,
-                report_buffer_space=True, interface=self.cri))
+                interface=self.cri))
         self.comb += self.outputs.coarse_timestamp.eq(tsc.coarse_ts)
         self.sync += self.outputs.minimum_coarse_timestamp.eq(tsc.coarse_ts + 16)
+        self.specials += MultiReg(self.sed_spread_enable, self.outputs.enable_spread, "rio")
 
         self.submodules.inputs = ClockDomainsRenamer("rio")(
             InputCollector(tsc, channels, interface=self.cri))
@@ -78,6 +81,7 @@ class DRTIOSatellite(Module):
         self.reset = CSRStorage(reset=1)
         self.reset_phy = CSRStorage(reset=1)
         self.tsc_loaded = CSR()
+        self.sed_spread_enable = CSRStorage()
         # master interface in the sys domain
         self.cri = cri.Interface()
         self.async_errors = Record(async_errors_layout)
@@ -143,7 +147,7 @@ class DRTIOSatellite(Module):
             self.rt_packet, tsc, self.async_errors)
 
     def get_csrs(self):
-        return ([self.reset, self.reset_phy, self.tsc_loaded] +
+        return ([self.reset, self.sed_spread_enable, self.reset_phy, self.tsc_loaded] +
                 self.link_layer.get_csrs() + self.link_stats.get_csrs() +
                 self.rt_errors.get_csrs())
 

artiq/gateware/rtio/core.py

@@ -3,7 +3,7 @@ from operator import and_
 
 from migen import *
 from migen.genlib.resetsync import AsyncResetSynchronizer
-from migen.genlib.cdc import BlindTransfer
+from migen.genlib.cdc import MultiReg, BlindTransfer
 from misoc.interconnect.csr import *
 
 from artiq.gateware.rtio import cri
@@ -18,6 +18,7 @@ class Core(Module, AutoCSR):
         self.cri = cri.Interface()
         self.reset = CSR()
         self.reset_phy = CSR()
+        self.sed_spread_enable = CSRStorage()
         self.async_error = CSR(3)
         self.collision_channel = CSRStatus(16)
         self.busy_channel = CSRStatus(16)
@@ -67,6 +68,7 @@ class Core(Module, AutoCSR):
         self.submodules += outputs
         self.comb += outputs.coarse_timestamp.eq(tsc.coarse_ts)
         self.sync += outputs.minimum_coarse_timestamp.eq(tsc.coarse_ts + 12)
+        self.specials += MultiReg(self.sed_spread_enable.storage, outputs.enable_spread, "rio")
 
         inputs = ClockDomainsRenamer("rio")(InputCollector(tsc, channels,
             quash_channels=quash_channels,

artiq/gateware/rtio/sed/core.py

@@ -12,7 +12,7 @@ __all__ = ["SED"]
 
 class SED(Module):
     def __init__(self, channels, glbl_fine_ts_width,
-                 lane_count=8, fifo_depth=128, fifo_high_watermark=1.0, enable_spread=True,
+                 lane_count=8, fifo_depth=128, fifo_high_watermark=1.0,
                  quash_channels=[], report_buffer_space=False, interface=None):
         seqn_width = layouts.seqn_width(lane_count, fifo_depth)
 
@@ -23,7 +23,6 @@ class SED(Module):
             layouts.fifo_payload(channels),
             [channel.interface.o.delay for channel in channels],
             glbl_fine_ts_width,
-            enable_spread=enable_spread,
             quash_channels=quash_channels,
             interface=interface)
         self.submodules.fifos = FIFOs(lane_count, fifo_depth, fifo_high_watermark,
@@ -47,6 +46,10 @@ class SED(Module):
                 self.cri.o_buffer_space.eq(self.fifos.buffer_space)
             ]
 
+    @property
+    def enable_spread(self):
+        return self.lane_dist.enable_spread
+
     @property
     def cri(self):
         return self.lane_dist.cri

artiq/gateware/rtio/sed/lane_distributor.py

@@ -10,7 +10,7 @@ __all__ = ["LaneDistributor"]
 class LaneDistributor(Module):
     def __init__(self, lane_count, seqn_width, layout_payload,
                  compensation, glbl_fine_ts_width,
-                 enable_spread=True, quash_channels=[], interface=None):
+                 quash_channels=[], interface=None):
         if lane_count & (lane_count - 1):
             raise NotImplementedError("lane count must be a power of 2")
 
@@ -28,6 +28,8 @@ class LaneDistributor(Module):
         self.output = [Record(layouts.fifo_ingress(seqn_width, layout_payload))
                        for _ in range(lane_count)]
 
+        self.enable_spread = Signal()
+
         # # #
 
         o_status_wait = Signal()
@@ -173,9 +175,8 @@ class LaneDistributor(Module):
         ]
 
         # current lane has reached high watermark, spread events by switching to the next.
-        if enable_spread:
         self.sync += [
-                If(current_lane_high_watermark | ~current_lane_writable,
+            If(self.enable_spread & (current_lane_high_watermark | ~current_lane_writable),
                 force_laneB.eq(1)
             ),
             If(do_write,

artiq/gateware/targets/efc.py

@@ -217,7 +217,10 @@ class Satellite(BaseSoC, AMPSoC):
 
         # satellite (master-controlled) RTIO
         self.submodules.local_io = SyncRTIO(self.rtio_tsc, rtio_channels, lane_count=sed_lanes)
-        self.comb += self.drtiosat.async_errors.eq(self.local_io.async_errors)
+        self.comb += [ 
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
 
         # subkernel RTIO
         self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc)

artiq/gateware/targets/kasli.py

@@ -120,6 +120,7 @@ class StandaloneBase(MiniSoC, AMPSoC):
             self.config["HAS_SI549"] = None
             self.config["WRPLL_REF_CLK"] = "SMA_CLKIN"
         else:
+            if self.platform.hw_rev == "v2.0":
                 self.submodules += SMAClkinForward(self.platform)
             self.config["HAS_SI5324"] = None
             self.config["SI5324_SOFT_RESET"] = None
@@ -596,7 +597,10 @@ class SatelliteBase(BaseSoC, AMPSoC):
 
         # satellite (master-controlled) RTIO
         self.submodules.local_io = SyncRTIO(self.rtio_tsc, rtio_channels, lane_count=sed_lanes)
-        self.comb += self.drtiosat.async_errors.eq(self.local_io.async_errors)
+        self.comb += [ 
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
 
         # subkernel RTIO
         self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc)
@@ -842,6 +846,8 @@ def main():
     has_shuttler = any(peripheral["type"] == "shuttler" for peripheral in description["peripherals"])
     if has_shuttler and (description["drtio_role"] == "standalone"):
         raise ValueError("Shuttler requires DRTIO, please switch role to master")
+    if description["enable_wrpll"] and description["hw_rev"] in ["v1.0", "v1.1"]:
+        raise ValueError("Kasli {} does not support WRPLL".format(description["hw_rev"])) 
 
     soc = cls(description, gateware_identifier_str=args.gateware_identifier_str, **soc_kasli_argdict(args))
     args.variant = description["variant"]

artiq/gateware/targets/kc705.py

@@ -468,7 +468,10 @@ class _SatelliteBase(BaseSoC, AMPSoC):
 
         # DRTIO
         self.submodules.local_io = SyncRTIO(self.rtio_tsc, rtio_channels)
-        self.comb += self.drtiosat.async_errors.eq(self.local_io.async_errors)
+        self.comb += [ 
+            self.drtiosat.async_errors.eq(self.local_io.async_errors),
+            self.local_io.sed_spread_enable.eq(self.drtiosat.sed_spread_enable.storage)
+        ]
 
         # subkernel RTIO
         self.submodules.rtio = rtio.KernelInitiator(self.rtio_tsc)

artiq/gateware/test/rtio/test_sed_lane_distributor.py

@@ -14,6 +14,7 @@ def simulate(input_events, compensation=None, wait=True):
     if compensation is None:
         compensation = [0]*256
     dut = lane_distributor.LaneDistributor(LANE_COUNT, 8, layout, compensation, 3)
+    dut.comb += dut.enable_spread.eq(1)
 
     output = []
     access_results = []

artiq/gateware/test/rtio/test_sed_top.py

@@ -27,6 +27,7 @@ class DUT(Module):
             self.sed.coarse_timestamp.eq(self.sed.coarse_timestamp + 1),
             self.sed.minimum_coarse_timestamp.eq(self.sed.coarse_timestamp + 16)
         ]
+        self.comb += self.sed.enable_spread.eq(0)
 
 
 def simulate(input_events, **kwargs):
@@ -110,6 +111,6 @@ class TestSED(unittest.TestCase):
             input_events += [(now, 1), (now, 0)]
 
         ttl_changes, access_results = simulate(input_events,
-            lane_count=2, fifo_depth=2, enable_spread=False)
+            lane_count=2, fifo_depth=2)
         self.assertEqual([r[0] for r in access_results], ["ok"]*len(input_events))
         self.assertEqual(ttl_changes, list(range(40, 40+40*20, 10)))

artiq/gui/applets.py

@@ -399,7 +399,7 @@ class _CompleterDelegate(QtWidgets.QStyledItemDelegate):
             # case, but causes unnecessary flickering and trashing of the user
             # selection when datasets are modified due to Qt's naive handler.
             # Doing this is of course convoluted due to Qt's arrogance
-            # about private fields and not letting users knows what
+            # about private fields and not letting users know what
             # slots are connected to signals, but thanks to the complicated
             # model system there is a short dirty hack in this particular case.
             nodatachanged_model = QtCore.QIdentityProxyModel()

artiq/language/core.py

@@ -28,7 +28,7 @@ def kernel(arg=None, flags={}):
     This decorator marks an object's method for execution on the core
     device.
 
-    When a decorated method is called from the Python interpreter, the :attr:`core`
+    When a decorated method is called from the Python interpreter, the ``core``
     attribute of the object is retrieved and used as core device driver. The
     core device driver will typically compile, transfer and run the method
     (kernel) on the device.
@@ -41,7 +41,7 @@ def kernel(arg=None, flags={}):
         - if the method is a regular Python method (not a kernel), it generates
           a remote procedure call (RPC) for execution on the host.
 
-    The decorator takes an optional parameter that defaults to :attr`core` and
+    The decorator takes an optional parameter that defaults to ``core`` and
     specifies the name of the attribute to use as core device driver.
 
     This decorator must be present in the global namespace of all modules using
@@ -134,7 +134,7 @@ def portable(arg=None, flags={}):
 def rpc(arg=None, flags={}):
     """
     This decorator marks a function for execution on the host interpreter.
-    This is also the default behavior of ARTIQ; however, this decorator allows
+    This is also the default behavior of ARTIQ; however, this decorator allows for
     specifying additional flags.
     """
     if arg is None:
@@ -256,7 +256,7 @@ _time_manager = _DummyTimeManager()
 def set_time_manager(time_manager):
     """Set the time manager used for simulating kernels by running them
     directly inside the Python interpreter. The time manager responds to the
-    entering and leaving of interleave/parallel/sequential blocks, delays, etc. and
+    entering and leaving of parallel/sequential blocks, delays, etc. and
     provides a time-stamped logging facility for events.
     """
     global _time_manager
@@ -280,7 +280,7 @@ class _Parallel:
 
     The execution time of a parallel block is the execution time of its longest
     statement. A parallel block may contain sequential blocks, which themselves
-    may contain interleave blocks, etc.
+    may contain parallel blocks, etc.
     """
     def __enter__(self):
         _time_manager.enter_parallel()
@@ -340,5 +340,5 @@ def watchdog(timeout):
 
 
 class TerminationRequested(Exception):
-    """Raised by ``pause`` when the user has requested termination."""
+    """Raised by :meth:`pause` when the user has requested termination."""
     pass

artiq/language/environment.py

@@ -28,7 +28,7 @@ class DefaultMissing(Exception):
 
 
 class CancelledArgsError(Exception):
-    """Raised by the ``interactive`` context manager when an interactive
+    """Raised by the :meth:`~artiq.language.environment.HasEnvironment.interactive` context manager when an interactive
     arguments request is cancelled."""
     pass
 
@@ -117,7 +117,7 @@ class NumberValue(_SimpleArgProcessor):
     ``int`` will also result in an error unless these conditions are met.
 
     When ``scale`` is not specified, and the unit is a common one (i.e.
-    defined in ``artiq.language.units``), then the scale is obtained from
+    defined in :class:`~artiq.language.units`), then the scale is obtained from
     the unit using a simple string match. For example, milliseconds (``"ms"``)
     units set the scale to 0.001. No unit (default) corresponds to a scale of
     1.0.
@@ -321,7 +321,8 @@ class HasEnvironment:
 
         :param key: Name of the argument.
         :param processor: A description of how to process the argument, such
-            as instances of ``BooleanValue`` and ``NumberValue``.
+            as instances of :mod:`~artiq.language.environment.BooleanValue` and 
+            :mod:`~artiq.language.environment.NumberValue`.
         :param group: An optional string that defines what group the argument
             belongs to, for user interface purposes.
         :param tooltip: An optional string to describe the argument in more
@@ -347,7 +348,8 @@ class HasEnvironment:
         """Request arguments from the user interactively.
 
         This context manager returns a namespace object on which the method
-        `setattr_argument` should be called, with the usual semantics.
+        :meth:`~artiq.language.environment.HasEnvironment.setattr_argument` should be called, 
+        with the usual semantics.
 
         When the context manager terminates, the experiment is blocked
         and the user is presented with the requested argument widgets.
@@ -355,7 +357,7 @@ class HasEnvironment:
         the namespace contains the values of the arguments.
 
         If the interactive arguments request is cancelled, raises
-        ``CancelledArgsError``."""
+        :exc:`~artiq.language.environment.CancelledArgsError`."""
         interactive_arglist = []
         namespace = SimpleNamespace()
         def setattr_argument(key, processor=None, group=None, tooltip=None):
@@ -478,7 +480,7 @@ class HasEnvironment:
 
         This function is used to get additional information for displaying the dataset.
 
-        See ``set_dataset`` for documentation of metadata items.
+        See :meth:`set_dataset` for documentation of metadata items.
         """
         try:
             return self.__dataset_mgr.get_metadata(key)

artiq/language/scan.py

@@ -1,20 +1,6 @@
 """
 Implementation and management of scan objects.
 
-A scan object (e.g. :class:`artiq.language.scan.RangeScan`) represents a
-one-dimensional sweep of a numerical range. Multi-dimensional scans are
-constructed by combining several scan objects, for example using
-:class:`artiq.language.scan.MultiScanManager`.
-
-Iterate on a scan object to scan it, e.g. ::
-
-    for variable in self.scan:
-        do_something(variable)
-
-Iterating multiple times on the same scan object is possible, with the scan
-yielding the same values each time. Iterating concurrently on the
-same scan object (e.g. via nested loops) is also supported, and the
-iterators are independent from each other.
 """
 
 import random
@@ -32,6 +18,21 @@ __all__ = ["ScanObject",
 
 
 class ScanObject:
+    """
+    Represents a one-dimensional sweep of a numerical range. Multi-dimensional scans are
+    constructed by combining several scan objects, for example using
+    :class:`MultiScanManager`.
+
+    Iterate on a scan object to scan it, e.g. ::
+
+        for variable in self.scan:
+            do_something(variable)
+
+    Iterating multiple times on the same scan object is possible, with the scan
+    yielding the same values each time. Iterating concurrently on the
+    same scan object (e.g. via nested loops) is also supported, and the
+    iterators are independent from each other.
+    """
     def __iter__(self):
         raise NotImplementedError
 
@@ -163,7 +164,7 @@ class Scannable:
     takes a scan object.
 
     When ``scale`` is not specified, and the unit is a common one (i.e.
-    defined in ``artiq.language.units``), then the scale is obtained from
+    defined in :class:`artiq.language.units`), then the scale is obtained from
     the unit using a simple string match. For example, milliseconds (``"ms"``)
     units set the scale to 0.001. No unit (default) corresponds to a scale of
     1.0.

artiq/master/experiments.py

@@ -206,10 +206,9 @@ class GitBackend:
         a git hash
         """
         commit, _ = self.git.resolve_refish(rev)
-
+        commit_id = str(commit.id)
-        logger.debug('Resolved git ref "%s" into "%s"', rev, commit.hex)
+        logger.debug('Resolved git ref "%s" into "%s"', rev, commit_id)
-
+        return commit_id
-        return commit.hex
 
     def request_rev(self, rev):
         rev = self._get_pinned_rev(rev)

artiq/master/scheduler.py

@@ -477,16 +477,16 @@ class Scheduler:
         return self.notifier.raw_view
 
     def check_pause(self, rid):
-        """Returns ``True`` if there is a condition that could make ``pause``
+        """Returns ``True`` if there is a condition that could make :meth:`pause`
         not return immediately (termination requested or higher priority run).
 
         The typical purpose of this function is to check from a kernel
         whether returning control to the host and pausing would have an effect,
         in order to avoid the cost of switching kernels in the common case
-        where ``pause`` does nothing.
+        where :meth:`pause` does nothing.
 
         This function does not have side effects, and does not have to be
-        followed by a call to ``pause``.
+        followed by a call to :meth:`pause`.
         """
         for pipeline in self._pipelines.values():
             if rid in pipeline.pool.runs:

artiq/master/worker_impl.py

@@ -362,7 +362,13 @@ def main():
                     write_results()
                     raise
                 finally:
+                    try:
                         device_mgr.notify_run_end()
+                    except:
+                        # Also write results immediately if the `notify_run_end`
+                        # callbacks produce an exception
+                        write_results()
+                        raise
                 put_completed()
             elif action == "analyze":
                 try:

artiq/test/coredevice/test_embedding.py

@@ -573,3 +573,29 @@ class NumpyQuotingTest(ExperimentCase):
     def test_issue_1871(self):
         """Ensure numpy.array() does not break NumPy math functions"""
         self.create(_NumpyQuoting).run()
+
+
+class _IntBoundary(EnvExperiment):
+    def build(self):
+        self.setattr_device("core")
+        self.int32_min = numpy.iinfo(numpy.int32).min
+        self.int32_max = numpy.iinfo(numpy.int32).max
+        self.int64_min = numpy.iinfo(numpy.int64).min
+        self.int64_max = numpy.iinfo(numpy.int64).max
+
+    @kernel
+    def test_int32_bounds(self, min_val: TInt32, max_val: TInt32):
+        return min_val == self.int32_min and max_val == self.int32_max
+
+    @kernel
+    def test_int64_bounds(self, min_val: TInt64, max_val: TInt64):
+        return min_val == self.int64_min and max_val == self.int64_max
+
+    @kernel
+    def run(self):
+        self.test_int32_bounds(self.int32_min, self.int32_max)
+        self.test_int64_bounds(self.int64_min, self.int64_max)
+
+class IntBoundaryTest(ExperimentCase):
+    def test_int_boundary(self):
+        self.create(_IntBoundary).run()

artiq/test/coredevice/test_exceptions.py

@@ -0,0 +1,59 @@
+import unittest
+import artiq.coredevice.exceptions as exceptions
+
+from artiq.experiment import *
+from artiq.test.hardware_testbench import ExperimentCase
+from artiq.compiler.embedding import EmbeddingMap
+from artiq.coredevice.core import test_exception_id_sync
+
+"""
+Test sync in exceptions raised between host and kernel
+Check `artiq.compiler.embedding` and `artiq::firmware::ksupport::eh_artiq`
+
+Considers the following two cases:
+    1) Exception raised on kernel and passed to host
+    2) Exception raised in a host function called from kernel
+Ensures same exception is raised on both kernel and host in either case
+"""
+
+exception_names = EmbeddingMap().str_reverse_map
+
+
+class _TestExceptionSync(EnvExperiment):
+    def build(self):
+        self.setattr_device("core")
+    
+    @rpc
+    def _raise_exception_host(self, id):
+        exn = exception_names[id].split('.')[-1].split(':')[-1]
+        exn = getattr(exceptions, exn)
+        raise exn
+
+    @kernel
+    def raise_exception_host(self, id):
+        self._raise_exception_host(id)
+
+    @kernel
+    def raise_exception_kernel(self, id):
+        test_exception_id_sync(id)
+
+    
+class ExceptionTest(ExperimentCase):
+    def test_raise_exceptions_kernel(self):
+        exp = self.create(_TestExceptionSync)
+        
+        for id, name in list(exception_names.items())[::-1]:
+            name = name.split('.')[-1].split(':')[-1]
+            with self.assertRaises(getattr(exceptions, name)) as ctx:
+                exp.raise_exception_kernel(id)
+            self.assertEqual(str(ctx.exception).strip("'"), name)
+            
+            
+    def test_raise_exceptions_host(self):
+        exp = self.create(_TestExceptionSync)
+
+        for id, name in exception_names.items():
+            name = name.split('.')[-1].split(':')[-1]
+            with self.assertRaises(getattr(exceptions, name)) as ctx:
+                exp.raise_exception_host(id)
+

artiq/test/lit/exceptions/finally_catch_try.py

@@ -0,0 +1,19 @@
+# RUN: %python -m artiq.compiler.testbench.jit %s >%t
+# RUN: OutputCheck %s --file-to-check=%t
+# REQUIRES: exceptions
+
+def doit():
+    try:
+        try:
+            raise RuntimeError("Error")
+        except ValueError:
+            print("ValueError")
+    except RuntimeError:
+        print("Caught")
+    finally:
+        print("Cleanup")
+
+doit()
+
+# CHECK-L: Caught
+# CHECK-NEXT-L: Cleanup

artiq/test/lit/exceptions/finally_try.py

@@ -0,0 +1,20 @@
+# RUN: %python -m artiq.compiler.testbench.jit %s >%t
+# RUN: OutputCheck %s --file-to-check=%t
+# REQUIRES: exceptions
+
+def doit():
+    try:
+        try:
+            raise RuntimeError("Error")
+        except ValueError:
+            print("ValueError")
+    finally:
+        print("Cleanup")
+
+try:
+    doit()
+except RuntimeError:
+    print("Caught")
+
+# CHECK-L: Cleanup
+# CHECK-NEXT-L: Caught

artiq/test/test_scheduler.py

@@ -141,132 +141,21 @@ class SchedulerCase(unittest.TestCase):
         high_priority = 3
         middle_priority = 2
         low_priority = 1
+
+        # "late" is far in the future, beyond any reasonable test timeout.
         late = time() + 100000
         early = time() + 1
 
-        expect = [
+        middle_priority_done = asyncio.Event()
-            {
+        def notify(mod):
-                "path": [],
+            # Watch for the "middle_priority" experiment's completion
-                "action": "setitem",
+            if mod == {
-                "value": {
-                    "repo_msg": None,
-                    "priority": low_priority,
-                    "pipeline": "main",
-                    "due_date": None,
-                    "status": "pending",
-                    "expid": expid_bg,
-                    "flush": False
-                },
-                "key": 0
-            },
-            {
-                "path": [],
-                "action": "setitem",
-                "value": {
-                    "repo_msg": None,
-                    "priority": high_priority,
-                    "pipeline": "main",
-                    "due_date": late,
-                    "status": "pending",
-                    "expid": expid_empty,
-                    "flush": False
-                },
-                "key": 1
-            },
-            {
-                "path": [],
-                "action": "setitem",
-                "value": {
-                    "repo_msg": None,
-                    "priority": middle_priority,
-                    "pipeline": "main",
-                    "due_date": early,
-                    "status": "pending",
-                    "expid": expid_empty,
-                    "flush": False
-                },
-                "key": 2
-            },
-            {
-                "path": [0],
-                "action": "setitem",
-                "value": "preparing",
-                "key": "status"
-            },
-            {
-                "path": [0],
-                "action": "setitem",
-                "value": "prepare_done",
-                "key": "status"
-            },
-            {
-                "path": [0],
-                "action": "setitem",
-                "value": "running",
-                "key": "status"
-            },
-            {
-                "path": [2],
-                "action": "setitem",
-                "value": "preparing",
-                "key": "status"
-            },
-            {
-                "path": [2],
-                "action": "setitem",
-                "value": "prepare_done",
-                "key": "status"
-            },
-            {
-                "path": [0],
-                "action": "setitem",
-                "value": "paused",
-                "key": "status"
-            },
-            {
-                "path": [2],
-                "action": "setitem",
-                "value": "running",
-                "key": "status"
-            },
-            {
                 "path": [2],
                 "action": "setitem",
                 "value": "run_done",
                 "key": "status"
-            },
+            }:
-            {
+                middle_priority_done.set()
-                "path": [0],
-                "action": "setitem",
-                "value": "running",
-                "key": "status"
-            },
-            {
-                "path": [2],
-                "action": "setitem",
-                "value": "analyzing",
-                "key": "status"
-            },
-            {
-                "path": [2],
-                "action": "setitem",
-                "value": "deleting",
-                "key": "status"
-            },
-            {
-                "path": [],
-                "action": "delitem",
-                "key": 2
-            },
-        ]
-        done = asyncio.Event()
-        expect_idx = 0
-        def notify(mod):
-            nonlocal expect_idx
-            self.assertEqual(mod, expect[expect_idx])
-            expect_idx += 1
-            if expect_idx >= len(expect):
-                done.set()
         scheduler.notifier.publish = notify
 
         scheduler.start(loop=loop)
@@ -275,7 +164,9 @@ class SchedulerCase(unittest.TestCase):
         scheduler.submit("main", expid_empty, high_priority, late)
         scheduler.submit("main", expid_empty, middle_priority, early)
 
-        loop.run_until_complete(done.wait())
+        # Check that the "middle_priority" experiment finishes. This would hang for a
+        # long time (until "late") if the due times were not being respected.
+        loop.run_until_complete(middle_priority_done.wait())
         scheduler.notifier.publish = None
         loop.run_until_complete(scheduler.stop())
 

doc/manual/building_developing.rst

@@ -0,0 +1,267 @@
+Building and developing ARTIQ
+=============================
+
+.. warning::
+    This section is only for software or FPGA developers who want to modify ARTIQ. The steps described here are not required if you simply want to run experiments with ARTIQ. If you purchased a system from M-Labs or QUARTIQ, we usually provide board binaries for you; you can use the AFWS client to get updated versions if necessary, as described in :ref:`obtaining-binaries`. It is not necessary to build them yourself.
+
+The easiest way to obtain an ARTIQ development environment is via the `Nix package manager <https://nixos.org/nix/>`_ on Linux. The Nix system is used on the `M-Labs Hydra server <https://nixbld.m-labs.hk/>`_ to build ARTIQ and its dependencies continuously; it ensures that all build instructions are up-to-date and allows binary packages to be used on developers' machines, in particular for large tools such as the Rust compiler.
+
+ARTIQ itself does not depend on Nix, and it is also possible to obtain everything from source (look into the ``flake.nix`` file to see what resources are used, and run the commands manually, adapting to your system) - but Nix makes the process a lot easier.
+
+Installing Vivado
+-----------------
+
+It is necessary to independently install AMD's `Vivado <https://www.xilinx.com/support/download.html>`_, which requires a login for download and can't be automatically obtained by package managers. The "appropriate" Vivado version to use for building gateware and firmware can vary. Some versions contain bugs that lead to hidden or visible failures, others work fine. Refer to the ``flake.nix`` file from the ARTIQ repository in order to determine which version is used at M-Labs.
+
+.. tip::
+    Text-search ``flake.nix`` for a mention of ``/opt/Xilinx/Vivado``. Given e.g. the line ::
+
+        profile = "set -e; source /opt/Xilinx/Vivado/2022.2/settings64.sh"
+
+    the intended Vivado version is 2022.2.
+
+Download and run the official installer. If using NixOS, note that this will require a FHS chroot environment; the ARTIQ flake provides such an environment, which you can enter with the command ``vivado-env`` from the development environment (i.e. after ``nix develop``). Other tips:
+
+- Be aware that Vivado is notoriously not a lightweight piece of software and you will likely need **at least 70GB+** of free space to install it.
+- If you do not want to write to ``/opt``, you can install into a folder of your home directory.
+- During the Vivado installation, uncheck ``Install cable drivers`` (they are not required, as we use better open source alternatives).
+- If the Vivado GUI installer crashes, you may be able to work around the problem by running it in unattended mode with a command such as ``./xsetup -a XilinxEULA,3rdPartyEULA,WebTalkTerms -b Install -e 'Vitis Unified Software Platform' -l /opt/Xilinx/``.
+- Vivado installation issues are not uncommon. Searching for similar problems on `the M-Labs forum <https://forum.m-labs.hk/>`_ or `Vivado's own support forums <https://support.xilinx.com/s/topic/0TO2E000000YKXwWAO/installation-and-licensing>`_ might be helpful when looking for solutions.
+
+.. _system-description:
+
+System description file
+-----------------------
+
+ARTIQ gateware and firmware binaries are dependent on the system configuration. In other words, a specific set of ARTIQ binaries is bound to the exact arrangement of real-time hardware it was generated for: the core device itself, its role in a DRTIO context (master, satellite, or standalone), the (real-time) peripherals in use, the physical EEM ports they will be connected to, and various other basic specifications. This information is normally provided to the software in the form of a JSON file called the system description or system configuration file.
+
+.. warning::
+
+    System configuration files are only used with Kasli and Kasli-SoC boards. KC705 and ZC706 ARTIQ configurations, due to their relative rarity and specialization, are handled on a case-by-case basis and selected through a variant name such as ``nist_clock``, with no system description file necessary. See below in :ref:`building` for where to find the list of supported variants. Writing new KC705 or ZC706 variants is not a trivial task, and not particularly recommended, unless you are an FPGA developer and know what you're doing.
+
+If you already have your system configuration file on hand, you can edit it to reflect any changes in configuration. If you purchased your original system from M-Labs, or recently purchased new hardware to add to it, you can obtain your up-to-date system configuration file through AFWS at any time using the command ``$ afws_client get_json`` (see :ref:`AFWS client<afws-client>`). If you are starting from scratch, a close reading of ``coredevice_generic.schema.json`` in ``artiq/coredevice`` will be helpful.
+
+System descriptions do not need to be very complex. At its most basic, a system description looks something like: ::
+
+    {
+        "target": "kasli",
+        "variant": "example",
+        "hw_rev": "v2.0",
+        "base": "master",
+        "peripherals": [
+            {
+                "type": "grabber",
+                "ports": [0]
+            }
+        ]
+    }
+
+Only these five fields are required, and the ``peripherals`` list can in principle be empty. A limited number of more extensive examples can currently be found in `the ARTIQ-Zynq repository <https://git.m-labs.hk/M-Labs/artiq-zynq/src/branch/master>`_, as well as in the main repository under ``artiq/examples/kasli_shuttler``. Once your system description file is complete, you can use ``artiq_ddb_template`` (see also :ref:`Utilities <ddb-template-tool>`) to test it and to generate a template for the corresponding :ref:`device database <device-db>`.
+
+DRTIO descriptions
+^^^^^^^^^^^^^^^^^^
+
+Note that in DRTIO systems it is necessary to create one description file *per core device*. Satellites and their connected peripherals must be described separately. Satellites also need to be reflashed separately, albeit only if their personal system descriptions have changed. (The layout of satellites relative to the master is configurable on the fly and will be established much later, in the routing table; see :ref:`drtio-routing`. It is not necessary to rebuild or reflash if only changing the DRTIO routing table).
+
+In contrast, only one device database should be generated even for a DRTIO system. Use a command of the form: ::
+
+    $ artiq_ddb_template -s 1 <satellite1>.json -s 2 <satellite2>.json <master>.json
+
+The numbers designate the respective satellite's destination number, which must correspond to the destination numbers used when generating the routing table later.
+
+Common system description changes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To add or remove peripherals from the system, add or remove their entries from the ``peripherals`` field. When replacing hardware with upgraded versions, update the corresponding ``hw_rev`` (hardware revision) field. Other fields to consider include:
+
+    - ``enable_wrpll`` (a simple boolean, see :ref:`core-device-clocking`)
+    - ``sed_lanes`` (increasing the number of SED lanes can reduce sequence errors, but correspondingly consumes more FPGA resources, see :ref:`sequence-errors`)
+    - various defaults (e.g. ``core_addr`` defines a default IP address, which can be freely reconfigured later).
+
+Nix development environment
+---------------------------
+
+* Install `Nix <http://nixos.org/nix/>`_ if you haven't already. Prefer a single-user installation for simplicity.
+* Configure Nix to support building ARTIQ:
+
+    - Enable flakes, for example by adding ``experimental-features = nix-command flakes`` to ``nix.conf``. See also the `NixOS Wiki on flakes <https://nixos.wiki/wiki/flakes>`_.
+    - Add ``/opt`` (or your Vivado location) as an Nix sandbox, for example by adding ``extra-sandbox-paths = /opt`` to ``nix.conf``.
+    - Make sure that you have accepted and marked as permanent the additional settings described in :ref:`installing-details`. You can check on this manually by ensuring the file ``trusted-settings.json`` in ``~/.local/share/nix/`` exists and contains the following: ::
+
+        {
+            "extra-sandbox-paths":{
+                "/opt":true
+            },
+            "extra-substituters":{
+                "https://nixbld.m-labs.hk":true
+            },
+            "extra-trusted-public-keys":{
+                "nixbld.m-labs.hk-1:5aSRVA5b320xbNvu30tqxVPXpld73bhtOeH6uAjRyHc=":true
+            }
+        }
+
+    - If using NixOS, make the equivalent changes to your ``configuration.nix`` instead.
+
+* Clone `the ARTIQ Git repository <https://github.com/m-labs/artiq>`_, or `the ARTIQ-Zynq repository <https://git.m-labs.hk/M-Labs/artiq-zynq>`__ for Zynq devices (Kasli-SoC or ZC706). By default, you are working with the ``master`` branch, which represents the beta version and is not stable (see :doc:`releases`). Checkout the most recent release (``git checkout release-[number]``) for a stable version.
+* If your Vivado installation is not in its default location ``/opt``, open ``flake.nix`` and edit it accordingly (note that the edits must be made in the main ARTIQ flake, even if you are working with Zynq, see also tip below).
+* Run ``nix develop`` at the root of the repository, where ``flake.nix`` is.
+
+.. note::
+    You can also target legacy versions of ARTIQ; use Git to checkout older release branches. Note however that older releases of ARTIQ required different processes for developing and building, which you are broadly more likely to figure out by (also) consulting the corresponding older versions of the manual.
+
+Once you have run ``nix develop`` you are in the ARTIQ development environment. All ARTIQ commands and utilities -- :mod:`~artiq.frontend.artiq_run`, :mod:`~artiq.frontend.artiq_master`, etc. -- should be available, as well as all the packages necessary to build or run ARTIQ itself. You can exit the environment at any time using Control+D or the ``exit`` command and re-enter it by re-running ``nix develop`` again in the same location.
+
+.. tip::
+    If you are developing for Zynq, you will have noted that the ARTIQ-Zynq repository consists largely of firmware. The firmware for Zynq (NAR3) is more modern than that used for current mainline ARTIQ, and is intended to eventually replace it; for now it constitutes most of the difference between the two ARTIQ variants. The gateware for Zynq, on the other hand, is largely imported from mainline ARTIQ.
+
+    If you intend to modify the source housed in the original ARTIQ repository, but build and test the results on a Zynq device, clone both repositories and set your ``PYTHONPATH`` after entering the ARTIQ-Zynq development shell: ::
+
+        $ export PYTHONPATH=/absolute/path/to/your/artiq:$PYTHONPATH
+
+    Note that this only applies for incremental builds. If you want to use ``nix build``, or make changes to the dependencies, look into changing the inputs of the ``flake.nix`` instead. You can do this by replacing the URL of the GitHub ARTIQ repository with ``path:/absolute/path/to/your/artiq``; remember that Nix pins dependencies, so to incorporate new changes you will need to exit the development shell, update the environment with ``nix flake update``, and re-run ``nix develop``.
+
+Building only standard binaries
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you are working with original ARTIQ, and you only want to build a set of standard binaries (i.e. without changing the source code), you can also enter the development shell without cloning the repository, using ``nix develop`` as follows: ::
+
+    $ nix develop git+https://github.com/m-labs/artiq.git\?ref=release-[number]#boards
+
+Leave off ``\?ref=release-[number]`` to prefer the current beta version instead of a numbered release.
+
+.. note::
+    Adding ``#boards`` makes use of the ARTIQ flake's provided ``artiq-boards-shell``, a lighter environment optimized for building firmware and flashing boards, which can also be accessed by running ``nix develop .#boards`` if you have already cloned the repository. Developers should be aware that in this shell the current copy of the ARTIQ sources is not added to your ``PYTHONPATH``. Run ``nix flake show`` and read ``flake.nix`` carefully to understand the different available shells.
+
+The parallel command does exist for ARTIQ-Zynq: ::
+
+    $ nix develop git+https://git.m-labs.hk/m-labs/artiq-zynq\?ref=release-[number]
+
+but if you are building ARTIQ-Zynq without intention to change the source, it is not actually necessary to enter the development environment at all; Nix is capable of accessing the official flake directly to set up the build, eliminating the requirement for any particular environment.
+
+This is equally possible for original ARTIQ, but not as useful, as the development environment (specifically the ``#boards`` shell) is still the easiest way to access the necessary tools for flashing the board. On the other hand, with Zynq, it is normally recommended to boot from SD card, which requires no further special tools. As long as you have a functioning Nix installation with flakes enabled, you can progress directly to the building instructions below.
+
+.. _building:
+
+Building ARTIQ
+--------------
+
+For general troubleshooting and debugging, especially with a 'fresh' board, see also :ref:`connecting-uart`.
+
+Kasli or KC705 (ARTIQ original)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For Kasli, if you have your system description file on-hand, you can at this point build both firmware and gateware with a command of the form: ::
+
+    $ python -m artiq.gateware.targets.kasli <description>.json
+
+With KC705, use: ::
+
+    $ python -m artiq.gateware.targets.kc705 -V <variant>
+
+This will create a directory ``artiq_kasli`` or ``artiq_kc705`` containing the binaries in a subdirectory named after your description file or variant. Flash the board as described in :ref:`writing-flash`, adding the option ``--srcbuild``, e.g., assuming your board is already connected by JTAG USB: ::
+
+    $ artiq_flash --srcbuild [-t kc705] -d artiq_<board>/<variant>
+
+.. note::
+    To see supported KC705 variants, run: ::
+
+        $ python -m artiq.gateware.targets.kc705 --help
+
+    Look for the option ``-V VARIANT, --variant VARIANT``.
+
+Kasli-SoC or ZC706 (ARTIQ on Zynq)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The building process for Zynq devices is a little more complex. The easiest method is to leverage ``nix build`` and the ``makeArtiqZynqPackage`` utility provided by the official flake. The ensuing command is rather long, because it uses a multi-clause expression in the Nix language to describe the desired result; it can be executed piece-by-piece using the `Nix REPL <https://nix.dev/manual/nix/2.18/command-ref/new-cli/nix3-repl.html>`_, but ``nix build`` provides a lot of useful conveniences.
+
+For Kasli-SoC, run: ::
+
+    $ nix build --print-build-logs --impure --expr 'let fl = builtins.getFlake "git+https://git.m-labs.hk/m-labs/artiq-zynq?ref=release-[number]"; in (fl.makeArtiqZynqPackage {target="kasli_soc"; variant="<variant>"; json=<path/to/description.json>;}).kasli_soc-<variant>-sd'
+
+Replace ``<variant>`` with ``master``, ``satellite``, or ``standalone``, depending on your targeted DRTIO role. Remove ``?ref=release-[number]`` to use the current beta version rather than a numbered release. If you have cloned the repository and prefer to use your local copy of the flake, replace the corresponding clause with ``builtins.getFlake "/absolute/path/to/your/artiq-zynq"``.
+
+For ZC706, you can use a command of the same form: ::
+
+    $ nix build --print-build-logs --impure --expr 'let fl = builtins.getFlake "git+https://git.m-labs.hk/m-labs/artiq-zynq?ref=release-[number]"; in (fl.makeArtiqZynqPackage {target="zc706"; variant="<variant>";}).zc706-<variant>-sd'
+
+or you can use the more direct version: ::
+
+    $ nix build --print-build-logs git+https://git.m-labs.hk/m-labs/artiq-zynq\?ref=release-[number]#zc706-<variant>-sd
+
+(which is possible for ZC706 because there is no need to be able to specify a system description file in the arguments.)
+
+.. note::
+    To see supported ZC706 variants, you can run the following at the root of the repository: ::
+
+        $ src/gateware/zc706.py --help
+
+    Look for the option ``-V VARIANT, --variant VARIANT``. If you have not cloned the repository or are not in the development environment, try: ::
+
+        $ nix flake show git+https://git.m-labs.hk/m-labs/artiq-zynq\?ref=release-[number] | grep "package 'zc706.*sd"
+
+    to see the list of suitable build targets directly.
+
+Any of these commands should produce a directory ``result`` which contains a file ``boot.bin``. As described in :ref:`writing-flash`, if your core device is currently accessible over the network, it can be flashed with :mod:`~artiq.frontend.artiq_coremgmt`. If it is not connected to the network:
+
+1. Power off the board, extract the SD card and load ``boot.bin`` onto it manually.
+2. Insert the SD card back into the board.
+3. Ensure that the DIP switches (labeled BOOT MODE) are set correctly, to SD.
+4. Power the board back on.
+
+Optionally, the SD card may also be loaded at the same time with an additional file ``config.txt``, which can contain preset configuration values in the format ``key=value``, one per line. The keys are those used with :mod:`~artiq.frontend.artiq_coremgmt`. This allows e.g. presetting an IP address and any other configuration information.
+
+After a successful boot, the "FPGA DONE" light should be illuminated and the board should respond to ping when plugged into Ethernet.
+
+.. _zynq-jtag-boot :
+
+Booting over JTAG/Ethernet
+""""""""""""""""""""""""""
+
+It is also possible to boot Zynq devices over USB and Ethernet. Flip the DIP switches to JTAG. The scripts ``remote_run.sh`` and ``local_run.sh`` in the ARTIQ-Zynq repository, intended for use with a remote JTAG server or a local connection to the core device respectively, are used at M-Labs to accomplish this. Both make use of the netboot tool ``artiq_netboot``, see also its source `here <https://git.m-labs.hk/M-Labs/artiq-netboot>`__, which is included in the ARTIQ-Zynq development environment. Adapt the relevant script to your system or read it closely to understand the options and the commands being run; note for example that ``remote_run.sh`` as written only supports ZC706.
+
+You will need to generate the gateware, firmware and bootloader first, either through ``nix build`` or incrementally as below. After an incremental build add the option ``-i`` when running either of the scripts. If using ``nix build``, note that target names of the form ``<board>-<variant>-jtag`` (run ``nix flake show`` to see all targets) will output the three necessary files without combining them into ``boot.bin``.
+
+.. warning::
+
+    A known Xilinx hardware bug on Zynq prevents repeatedly loading the SZL bootloader over JTAG (i.e. repeated calls of the ``*_run.sh`` scripts) without a POR reset. On Kasli-SoC, you can physically set a jumper on the ``PS_POR_B`` pins of your board and use the M-Labs `POR reset script <https://git.m-labs.hk/M-Labs/zynq-rs/src/branch/master/kasli_soc_por.py>`_.
+
+Zynq incremental build
+^^^^^^^^^^^^^^^^^^^^^^
+
+The ``boot.bin`` file used in a Zynq SD card boot is in practice the combination of several files, normally ``top.bit`` (the gateware), ``runtime`` or ``satman`` (the firmware) and ``szl.elf`` (an open-source bootloader for Zynq `written by M-Labs <https://git.m-labs.hk/M-Labs/zynq-rs/src/branch/master/szl>`_, used in ARTIQ in place of Xilinx's FSBL). In some circumstances, especially if you are developing ARTIQ, you may prefer to construct these components separately. Be sure that you have cloned the repository and entered the development environment as described above.
+
+To compile the gateware and firmware, enter the ``src`` directory and run two commands as follows:
+
+For Kasli-SoC:
+    ::
+
+    $ gateware/kasli_soc.py -g ../build/gateware <description.json>
+    $ make TARGET=kasli_soc GWARGS="path/to/description.json" <fw-type>
+
+For ZC706:
+    ::
+
+    $ gateware/zc706.py -g ../build/gateware -V <variant>
+    $ make TARGET=zc706 GWARGS="-V <variant>" <fw-type>
+
+where ``fw-type`` is ``runtime`` for standalone or DRTIO master builds and ``satman`` for DRTIO satellites. Both the gateware and the firmware will generate into the ``../build`` destination directory. At this stage you can :ref:`boot from JTAG <zynq-jtag-boot>`; either of the ``*_run.sh`` scripts will expect the gateware and firmware files at their default locations, and the ``szl.elf`` bootloader is retrieved automatically.
+
+If you prefer to boot from SD card, you will need to construct your own ``boot.bin``. Build ``szl.elf`` from source by running a command of the form: ::
+
+    $ nix build git+https://git.m-labs.hk/m-labs/zynq-rs#<board>-szl
+
+For easiest access run this command in the ``build`` directory. The ``szl.elf`` file will be in the subdirectory ``result``. To combine all three files into the boot image, create a file called ``boot.bif`` in ``build`` with the following contents: ::
+
+    the_ROM_image:
+        {
+            [bootloader]result/szl.elf
+            gateware/top.bit
+            firmware/armv7-none-eabihf/release/<fw-type>
+        }
+        EOF
+
+Save this file. Now use ``mkbootimage`` to create ``boot.bin``. ::
+
+$   mkbootimage boot.bif boot.bin
+
+Boot from SD card as above.

doc/manual/compiler.rst

@@ -1,89 +1,124 @@
 Compiler
 ========
 
-The ARTIQ compiler transforms the Python code of the kernels into machine code executable on the core device. It is invoked automatically when calling a function that uses the ``@kernel`` decorator.
+The ARTIQ compiler transforms the Python code of the kernels into machine code executable on the core device. For limited purposes (normally, obtaining executable binaries of idle and startup kernels), it can be accessed through :mod:`~artiq.frontend.artiq_compile`. Otherwise it is invoked automatically whenever a function with an applicable decorator is called.
 
-Supported Python features
+ARTIQ kernel code accepts *nearly,* but not quite, a strict subset of Python 3. The necessities of real-time operation impose a harsher set of limitations; as a result, many Python features are necessarily omitted, and there are some specific discrepancies (see also :ref:`compiler-pitfalls`).
--------------------------
 
-A number of Python features can be used inside a kernel for compilation and execution on the core device. They include ``for`` and ``while`` loops, conditionals (``if``, ``else``, ``elif``), functions, exceptions, and statically typed variables of the following types:
+In general, ARTIQ Python supports only statically typed variables; it implements no heap allocation or garbage collection systems, essentially disallowing any heap-based data structures (although lists and arrays remain available in a stack-based form); and it cannot use runtime dispatch, meaning that, for example, all elements of an array must be of the same type. Nonetheless, technical details aside, a basic knowledge of Python is entirely sufficient to write ARTIQ experiments.
 
-* Booleans
+.. note::
-* 32-bit signed integers (default size)
+    The ARTIQ compiler is now in its second iteration. The third generation, known as NAC3, is `currently in development <https://git.m-labs.hk/M-Labs/nac3>`_, and available for pre-alpha experimental use. NAC3 represents a major overhaul of ARTIQ compilation, and will feature much faster compilation speeds, a greatly improved type system, and more predictable and transparent operation. It is compatible with ARTIQ firmware starting at ARTIQ-7. Instructions for installation and basic usage differences can also be found `on the M-Labs Forum <https://forum.m-labs.hk/d/392-nac3-new-artiq-compiler-3-prealpha-release>`_. While NAC3 is a work in progress and many important features remain unimplemented, installation and feedback is welcomed.
-* 64-bit signed integers (use ``numpy.int64`` to convert)
-* Double-precision floating point numbers
-* Lists of any supported types
-* String constants
-* User-defined classes, with attributes of any supported types (attributes that are not used anywhere in the kernel are ignored)
 
-For a demonstration of some of these features, see the ``mandelbrot.py`` example.
+ARTIQ Python code
+-----------------
 
-When several instances of a user-defined class are referenced from the same kernel, every attribute must have the same type in every instance of the class.
+A variety of short experiments can be found in the subfolders of ``artiq/examples``, especially under ``kc705_nist_clock/repository`` and ``no_hardware/repository``. Reading through these will give you a general idea of what ARTIQ Python is capable of and how to use it.
 
-Remote procedure calls
+Functions and decorators
-----------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Kernel code can call host functions without any additional ceremony. However, such functions are assumed to return `None`, and if a value other than `None` is returned, an exception is raised. To call a host function returning a value other than `None` its return type must be annotated using the standard Python syntax, e.g.: ::
+The ARTIQ compiler recognizes several specialized decorators, which determine the way the decorated function will be compiled and handled.
+
+``@kernel`` (see :meth:`~artiq.language.core.kernel`) designates kernel functions, which will be compiled for and executed on the core device; the basic setup and background for kernels is detailed on the :doc:`getting_started_core` page. ``@subkernel`` (:meth:`~artiq.language.core.subkernel`) designates subkernel functions, which are largely similar to kernels except that they are executed on satellite devices in a DRTIO setting, with some associated limitations; they are described in more detail on the :doc:`using_drtio_subkernels` page.
+
+``@rpc`` (:meth:`~artiq.language.core.rpc`) designates functions to be executed on the host machine, which are compiled and run in regular Python, outside of the core device's real-time limitations. Notably, functions without decorators are assumed to be host-bound by default, and treated identically to an explicitly marked ``@rpc``. As a result, the explicit decorator is only really necessary when specifying additional flags (for example, ``flags={"async"}``, see below).
+
+``@portable`` (:meth:`~artiq.language.core.portable`) designates functions to be executed *on the same device they are called.* In other words, when called from a kernel, a portable is executed as a kernel; when called from a subkernel, it is executed as a kernel, on the same satellite device as the calling subkernel; when called from a host function, it is executed on the host machine.
+
+``@host_only`` (:meth:`~artiq.language.core.host_only`) functions are executed fully on the host, similarly to ``@rpc``, but calling them from a kernel as an RPC will be refused by the compiler. It can be used to mark functions which should only ever be called by the host.
+
+.. warning::
+    ARTIQ goes to some lengths to cache code used in experiments correctly, so that experiments run according to the state of the code when they were started, even if the source is changed during the run time. Python itself annoyingly fails to implement this (see also `issue #416 <https://github.com/m-labs/artiq/issues/416>`_), necessitating a workaround on ARTIQ's part. One particular downstream limitation is that the ARTIQ compiler is unable to recognize decorators with path prefixes, i.e.: ::
+
+         import artiq.experiment as aq
+
+         [...]
+
+            @aq.kernel
+            def run(self):
+                pass
+
+    will fail to compile. As long as ``from artiq.experiment import *`` is used as in the examples, this is never an issue. If prefixes are strongly preferred, a possible workaround is to import decorators separately, as e.g. ``from artiq.language.core import kernel``.
+
+.. _compiler-types:
+
+ARTIQ types
+^^^^^^^^^^^
+
+Python/NumPy types correspond to ARTIQ types as follows:
+
++------------------------+-------------------------+
+| Python                 | ARTIQ                   |
++========================+=========================+
+| NoneType               | TNone                   |
++------------------------+-------------------------+
+| bool                   | TBool                   |
++------------------------+-------------------------+
+| int                    | TInt32 or TInt64        |
++------------------------+-------------------------+
+| float                  | TFloat                  |
++------------------------+-------------------------+
+| str                    | TStr                    |
++------------------------+-------------------------+
+| bytes                  | TBytes                  |
++------------------------+-------------------------+
+| bytearray              | TByteArray              |
++------------------------+-------------------------+
+| list of T              | TList(T)                |
++------------------------+-------------------------+
+| NumPy array            | TArray(T, num_dims)     |
++------------------------+-------------------------+
+| tuple of (T1, T2, ...) | TTuple([T1, T2, ...])   |
++------------------------+-------------------------+
+| range                  | TRange32, TRange64      |
++------------------------+-------------------------+
+| numpy.int32            | TInt32                  |
++------------------------+-------------------------+
+| numpy.int64            | TInt64                  |
++------------------------+-------------------------+
+| numpy.float64          | TFloat                  |
++------------------------+-------------------------+
+
+Integers are 32-bit by default but may be converted to 64-bit with ``numpy.int64``.
+
+The ARTIQ compiler can be thought of as overriding all built-in Python types, and types in kernel code cannot always be assumed to behave as they would in host Python. In particular, normally heap-allocated types such as arrays, lists, and strings are very limited in what they support. Strings must be constant and lists and arrays must be of constant size. Methods like ``append``, ``push``, and ``pop`` are unavailable as a matter of principle, and will not compile. Certain types, notably dictionaries, have no ARTIQ implementation and cannot be used in kernels at all.
+
+.. tip::
+    Instead of pushing or appending, preallocate for the maximum number of elements you expect with a list comprehension, i.e. ``x = [0 for _ in range(1024)]``, and then keep a variable ``n`` noting the last filled element of the array. Afterwards, ``x[0:n]`` will give you a list with that number of elements.
+
+Multidimensional arrays are allowed (using NumPy syntax). Element-wise operations (e.g. ``+``, ``/``), matrix multiplication (``@``) and multidimensional indexing are supported; slices and views (currently) are not.
+
+Tuples may contain a mixture of different types. They cannot be iterated over or dynamically indexed, although they may be indexed by constants and multiple assignment is supported.
+
+User-defined classes are supported, provided their attributes are of other supported types (attributes that are not used in the kernel are ignored and thus unrestricted). When several instances of a user-defined class are referenced from the same kernel, every attribute must have the same type in every instance of the class.
+
+.. _basic-artiq-python:
+
+Basic ARTIQ Python
+^^^^^^^^^^^^^^^^^^
+
+Basic Python features can broadly be used inside kernels without further compunctions. This includes loops (``for`` / ``while`` / ``break`` / ``continue``), conditionals (``if`` / ``else`` / ``elif``), functions, exceptions, ``try`` / ``except`` / ``else`` blocks,  and statically typed variables of any supported types.
+
+Kernel code can call host functions without any additional ceremony. However, such functions are assumed to return ``None``, and if a value other than ``None`` is returned, an exception is raised. To call a host function returning a value other than ``None`` its return type must be annotated, using the standard Python syntax, e.g.: ::
 
     def return_four() -> TInt32:
         return 4
 
-The Python types correspond to ARTIQ type annotations as follows:
+.. tip::
+    Multiple variables of different types can be sent in one RPC call by returning a tuple, e.g. ::
 
-+---------------+-------------------------+
+        def return_many() -> TTuple([TInt32, TFloat, TStr]):
-| Python        | ARTIQ                   |
+            return (4, 12.34, "hello",)
-+===============+=========================+
-| NoneType      | TNone                   |
-+---------------+-------------------------+
-| bool          | TBool                   |
-+---------------+-------------------------+
-| int           | TInt32 or TInt64        |
-+---------------+-------------------------+
-| float         | TFloat                  |
-+---------------+-------------------------+
-| str           | TStr                    |
-+---------------+-------------------------+
-| list of T     | TList(T)                |
-+---------------+-------------------------+
-| NumPy array   | TArray(T, num_dims)     |
-+---------------+-------------------------+
-| range         | TRange32, TRange64      |
-+---------------+-------------------------+
-| numpy.int32   | TInt32                  |
-+---------------+-------------------------+
-| numpy.int64   | TInt64                  |
-+---------------+-------------------------+
-| numpy.float64 | TFloat                  |
-+---------------+-------------------------+
 
-Pitfalls
+    Which can be retrieved from a kernel with ``(a, b, c) = return_many()``
---------
 
-The ARTIQ compiler accepts *nearly* a strict subset of Python 3. However, by necessity there
+Kernels can freely modify attributes of objects shared with the host. However, by necessity, these modifications are actually applied to local copies of the objects, as the latency of immediate writeback would be unsupportable in a real-time environment. Instead, modifications are written back *when the kernel completes;* notably, this means RPCs called by a kernel itself will only have access to the unmodified host version of the object, as the kernel hasn't finished execution yet. In some cases, accessing data on the host is better handled by calling RPCs specifically to make the desired modifications.
-is a number of differences that can lead to bugs.
 
-Arbitrary-length integers are not supported at all on the core device; all integers are
+.. warning::
-either 32-bit or 64-bit. This especially affects calculations that result in a 32-bit signed
-overflow; if the compiler detects a constant that doesn't fit into 32 bits, the entire expression
-will be upgraded to 64-bit arithmetics, however if all constants are small, 32-bit arithmetics
-will be used even if the result will overflow. Overflows are not detected.
 
-The result of calling the builtin ``round`` function is different when used with
+    Kernels *cannot and should not* return lists, arrays, or strings they have created, or any objects containing them; in the absence of a heap, the way these values are allocated means they cannot outlive the kernels they are created in. Trying to do so will normally be discovered by lifetime tracking and result in compilation errors, but in certain cases lifetime tracking will fail to detect a problem and experiments will encounter memory corruption at runtime. For example: ::
-the builtin ``float`` type and the ``numpy.float64`` type on the host interpreter; ``round(1.0)``
-returns an integer value 1, whereas ``round(numpy.float64(1.0))`` returns a floating point value
-``numpy.float64(1.0)``. Since both ``float`` and ``numpy.float64`` are mapped to
-the builtin ``float`` type on the core device, this can lead to problems in functions marked
-``@portable``; the workaround is to explicitly cast the argument of ``round`` to ``float``:
-``round(float(numpy.float64(1.0)))`` returns an integer on the core device as well as on the host
-interpreter.
 
-Empty lists do not have valid list element types, so they cannot be used in the kernel.
-
-In kernels, lifetime of allocated values (e.g. lists or numpy arrays) might not be correctly
-tracked across function calls (see `#1497 <https://github.com/m-labs/artiq/issues/1497>`_,
-`#1677 <https://github.com/m-labs/artiq/issues/1677>`_) like in this example ::
-
-    @kernel
         def func(a):
             return a
 
@@ -96,14 +131,72 @@ tracked across function calls (see `#1497 <https://github.com/m-labs/artiq/issue
                 # results in memory corruption
                 return func([1, 2, 3])
 
-This results in memory corruption at runtime.
+    will compile, **but corrupts at runtime.** On the other hand, lists, arrays, or strings can and should be used as inputs for RPCs, and this is the preferred method of returning data to the host. In this way the data is sent before the kernel completes and there are no allocation issues.
+
+Available built-in functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ARTIQ makes various useful built-in and mathematical functions from Python, NumPy, and SciPy available in kernel code. They are not guaranteed to be perfectly equivalent to their host namesakes (for example, ``numpy.rint()`` normally rounds-to-even, but in kernel code rounds toward zero) but their behavior should be basically predictable.
+
+
+.. list-table::
+    :header-rows: 1
+
+    +   * Reference
+        * Functions
+    +   * `Python built-ins <https://docs.python.org/3/library/functions.html>`_
+        *   - ``len()``, ``round()``, ``abs()``, ``min()``, ``max()``
+            - ``print()`` (with caveats; see below)
+            - all basic type conversions (``int()``, ``float()`` etc.)
+    +   * `NumPy mathematic utilities <https://numpy.org/doc/stable/reference/routines.math.html>`_
+        *   - ``sqrt()``, ``cbrt()``
+            - ``fabs()``, ``fmax()``, ``fmin()``
+            - ``floor()``, ``ceil()``, ``trunc()``, ``rint()``
+    +   * `NumPy exponents and logarithms <https://numpy.org/doc/stable/reference/routines.math.html#exponents-and-logarithms>`_
+        *   - ``exp()``, ``exp2()``, ``expm1()``
+            - ``log()``, ``log2()``, ``log10()``
+    +   * `NumPy trigonometric and hyperbolic functions <https://numpy.org/doc/stable/reference/routines.math.html#trigonometric-functions>`_
+        *   - ``sin()``, ``cos()``, ``tan()``,
+            - ``arcsin()``, ``arccos()``, ``arctan()``
+            - ``sinh()``, ``cosh()``, ``tanh()``
+            - ``arcsinh()``, ``arccosh()``, ``arctanh()``
+            - ``hypot()``, ``arctan2()``
+    +   * `NumPy floating point routines <https://numpy.org/doc/stable/reference/routines.math.html#floating-point-routines>`_
+        *   - ``copysign()``, ``nextafter()``
+    +   * `SciPy special functions <https://docs.scipy.org/doc/scipy/reference/special.html>`_
+        *   - ``erf()``, ``erfc()``
+            - ``gamma()``, ``gammaln()``
+            - ``j0()``, ``j1()``, ``y0()``, ``y1()``
+
+Basic NumPy array handling (``np.array()``, ``numpy.transpose()``, ``numpy.full()``, ``@``, element-wise operation, etc.) is also available. NumPy functions are implicitly broadcast when applied to arrays.
+
+Print and logging functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ARTIQ offers two native built-in logging functions: ``rtio_log()``, which prints to the :ref:`RTIO log <rtio-analyzer>`, as retrieved by :mod:`~artiq.frontend.artiq_coreanalyzer`, and ``core_log()``, which prints directly to the core log, regardless of context or network connection status. Both exist for debugging purposes, especially in contexts where a ``print()`` RPC is not suitable, such as in idle/startup kernels or when debugging delicate RTIO slack issues which may be significantly affected by the overhead of ``print()``.
+
+``print()`` itself is in practice an RPC to the regular host Python ``print()``, i.e. with output either in the terminal of :mod:`~artiq.frontend.artiq_run` or in the client logs when using :mod:`~artiq.frontend.artiq_dashboard` or :mod:`~artiq.frontend.artiq_compile`. This means on one hand that it should not be used in idle, startup, or subkernels, and on the other hand that it suffers of some of the timing limitations of any other RPC, especially if the RPC queue is full. Accordingly, it is important to be aware that the timing of ``print()`` outputs can't reliably be used to debug timing in kernels, and especially not the timing of other RPCs.
+
+.. _compiler-pitfalls:
+
+Pitfalls
+--------
+
+Empty lists do not have valid list element types, so they cannot be used in the kernel.
+
+Arbitrary-length integers are not supported at all on the core device; all integers are either 32-bit or 64-bit. This especially affects calculations that result in a 32-bit signed overflow. If the compiler detects a constant that can't fit into 32 bits, the entire expression will be upgraded to 64-bit arithmetic, but if all constants are small, 32-bit arithmetic is used even if the result will overflow. Overflows are not detected.
+
+The result of calling the builtin ``round`` function is different when used with the builtin ``float`` type and the ``numpy.float64`` type on the host interpreter; ``round(1.0)`` returns an integer value 1, whereas ``round(numpy.float64(1.0))`` returns a floating point value ``numpy.float64(1.0)``. Since both ``float`` and ``numpy.float64`` are mapped to the builtin ``float`` type on the core device, this can lead to problems in functions marked ``@portable``; the workaround is to explicitly cast the argument of ``round`` to ``float``: ``round(float(numpy.float64(1.0)))`` returns an integer on the core device as well as on the host interpreter.
+
+Flags and optimizations
+-----------------------
+
+The ARTIQ compiler runs many optimizations, most of which perform well on code that has pristine Python semantics. It also contains more powerful, and more invasive, optimizations that require opt-in to activate.
 
 Asynchronous RPCs
------------------
+^^^^^^^^^^^^^^^^^
 
-If an RPC returns no value, it can be invoked in a way that does not block until the RPC finishes
+If an RPC returns no value, it can be invoked in a way that does not block until the RPC finishes execution, but only until it is queued. (Submitting asynchronous RPCs too rapidly, as well as submitting asynchronous RPCs with arguments that are too large, can still block until completion.)
-execution, but only until it is queued. (Submitting asynchronous RPCs too rapidly, as well as
-submitting asynchronous RPCs with arguments that are too large, can still block until completion.)
 
 To define an asynchronous RPC, use the ``@rpc`` annotation with a flag: ::
 
@@ -111,17 +204,12 @@ To define an asynchronous RPC, use the ``@rpc`` annotation with a flag: ::
     def record_result(x):
         self.results.append(x)
 
-Additional optimizations
-------------------------
-
-The ARTIQ compiler runs many optimizations, most of which perform well on code that has pristine Python semantics. It also contains more powerful, and more invasive, optimizations that require opt-in to activate.
-
 Fast-math flags
-+++++++++++++++
+^^^^^^^^^^^^^^^
 
-The compiler does not normally perform algebraically equivalent transformations on floating-point expressions, because this can dramatically change the result. However, it can be instructed to do so if all of the following is true:
+The compiler does not normally perform algebraically equivalent transformations on floating-point expressions, because this can dramatically change the result. However, it can be instructed to do so if all of the following are true:
 
-* Arguments and results will not be not-a-number or infinity values;
+* Arguments and results will not be Not-a-Number or infinite;
 * The sign of a zero value is insignificant;
 * Any algebraically equivalent transformations, such as reassociation or replacing division with multiplication by reciprocal, are legal to perform.
 
@@ -134,7 +222,7 @@ If this is the case for a given kernel, a ``fast-math`` flag can be specified to
 This flag particularly benefits loops with I/O delays performed in fractional seconds rather than machine units, as well as updates to DDS phase and frequency.
 
 Kernel invariants
-+++++++++++++++++
+^^^^^^^^^^^^^^^^^
 
 The compiler attempts to remove or hoist out of loops any redundant memory load operations, as well as propagate known constants into function bodies, which can enable further optimization. However, it must make conservative assumptions about code that it is unable to observe, because such code can change the value of the attribute, making the optimization invalid.
 
@@ -171,7 +259,7 @@ In the synthetic example above, the compiler will be able to detect that the res
                 delay(self.worker.interval / 5.0)
                 self.worker.work()
 
-In the synthetic example above, the compiler will be able to detect that the result of evaluating ``self.interval / 5.0`` never changes, even though it neither knows the value of ``self.worker.interval`` beforehand nor can it see through the ``self.worker.work()`` function call, and hoist the expensive floating-point division out of the loop, transforming the code for ``loop`` into an equivalent of the following: ::
+In the synthetic example above, the compiler will be able to detect that the result of evaluating ``self.interval / 5.0`` never changes, even though it neither knows the value of ``self.worker.interval`` beforehand nor can see through the ``self.worker.work()`` function call, and thus can hoist the expensive floating-point division out of the loop, transforming the code for ``loop`` into an equivalent of the following: ::
 
         @kernel
         def loop(self):

doc/manual/conf.py

@@ -29,26 +29,17 @@ builtins.__in_sphinx__ = True
 # we cannot use autodoc_mock_imports (does not help with argparse)
 mock_modules = ["artiq.gui.waitingspinnerwidget",
                 "artiq.gui.flowlayout",
-                "artiq.gui.state",
-                "artiq.gui.log",
                 "artiq.gui.models",
                 "artiq.compiler.module",
                 "artiq.compiler.embedding",
                 "artiq.dashboard.waveform",
-                "artiq.dashboard.interactive_args",
+                "artiq.coredevice.jsondesc",
-                "qasync", "pyqtgraph", "matplotlib", "lmdb",
+                "qasync", "lmdb", "dateutil.parser", "prettytable", "PyQt5",
-                "numpy", "dateutil", "dateutil.parser", "prettytable", "PyQt5",
+                "h5py", "llvmlite", "pythonparser", "tqdm", "jsonschema"]
-                "h5py", "serial", "scipy", "scipy.interpolate",
-                "llvmlite", "Levenshtein", "pythonparser",
-                "sipyco", "sipyco.pc_rpc", "sipyco.sync_struct",
-                "sipyco.asyncio_tools", "sipyco.logging_tools",
-                "sipyco.broadcast", "sipyco.packed_exceptions",
-                "sipyco.keepalive", "sipyco.pipe_ipc"]
 
 for module in mock_modules:
     sys.modules[module] = Mock()
 
-
 # https://stackoverflow.com/questions/29992444/sphinx-autodoc-skips-classes-inherited-from-mock
 class MockApplets:
     class AppletsDock:
@@ -79,6 +70,7 @@ extensions = [
     'sphinx.ext.napoleon',
     'sphinxarg.ext',
     'sphinxcontrib.wavedrom',  # see also below for config
+    "sphinxcontrib.jquery",
 ]
 
 mathjax_path = "https://m-labs.hk/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML.js"
@@ -97,7 +89,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'ARTIQ'
-copyright = '2014-2024, M-Labs Limited'
+copyright = '2014-2025, M-Labs Limited'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -146,6 +138,29 @@ pygments_style = 'sphinx'
 # If true, keep warnings as "system message" paragraphs in the built documents.
 #keep_warnings = False
 
+# If true, Sphinx will warn about *all* references where the target cannot be found.
+nitpicky = True
+
+# (type, target) regex tuples to ignore when generating warnings in 'nitpicky' mode
+# i.e. objects that are not documented in this manual and do not need to be
+nitpick_ignore_regex = [
+    (r'py:.*', r'numpy..*'),
+    (r'py:.*', r'sipyco..*'),
+    ('py:const', r'.*'), # no constants are documented anyway
+    ('py.attr', r'.*'), # nor attributes
+    (r'py:.*', r'artiq.gateware.*'),
+    ('py:mod', r'artiq.test.*'),
+    ('py:mod', r'artiq.applets.*'),
+    # we can't use artiq.master.* because we shouldn't ignore the scheduler
+    ('py:class', r'artiq.master.experiments.*'),
+    ('py:class', r'artiq.master.databases.*'),
+    ('py:.*', r'artiq.master.worker.*'),
+    ('py:class', 'dac34H84'),
+    ('py:class', 'trf372017'),
+    ('py:class', r'list(.*)'),
+    ('py.class', 'NoneType'),
+    ('py:meth', 'pause')
+]
 
 # -- Options for HTML output ----------------------------------------------
 

doc/manual/configuring.rst

@@ -0,0 +1,129 @@
+Networking and configuration
+============================
+
+.. _core-device-networking:
+
+Setting up core device networking
+---------------------------------
+
+For Kasli, insert a SFP/RJ45 transceiver (normally included with purchases from M-Labs and QUARTIQ) into the SFP0 port and connect it to an Ethernet port in your network. If the port is 10Mbps or 100Mbps and not 1000Mbps, make sure that the SFP/RJ45 transceiver supports the lower rate. Many SFP/RJ45 transceivers only support the 1000Mbps rate. If you do not have a SFP/RJ45 transceiver that supports 10Mbps and 100Mbps rates, you may instead use a gigabit Ethernet switch in the middle to perform rate conversion.
+
+You can also insert other types of SFP transceivers into Kasli if you wish to use it directly in e.g. an optical fiber Ethernet network. Kasli-SoC already directly features RJ45 10/100/1000 Ethernet.
+
+IP address and ping
+^^^^^^^^^^^^^^^^^^^
+
+If you purchased a Kasli or Kasli-SoC device from M-Labs, it will arrive with an IP address already set, normally the address requested in the web shop at time of purchase. If you did not specify an address at purchase, the default IP M-Labs uses is ``192.168.1.75``. If you did not obtain your hardware from M-Labs, or if you have just reflashed your core device, see :ref:`networking-tips` below.
+
+Once you know the IP, check that you can ping your device: ::
+
+  $ ping <IP_address>
+
+If ping fails, check that the Ethernet LED is ON; on Kasli, it is the LED next to the SFP0 connector. As a next step, try connecting to the serial port to read the UART log. See :ref:`connecting-UART`.
+
+Core management tool
+^^^^^^^^^^^^^^^^^^^^
+
+The tool used to configure the core device is the command-line utility :mod:`~artiq.frontend.artiq_coremgmt`. In order for it to connect to your core device, it is necessary to supply it somehow with the correct IP address for your core device. This can be done directly through use of the ``-D`` option, for example in: ::
+
+    $ artiq_coremgmt -D <IP_address> log
+
+.. note::
+  This command reads and displays the core log. If you have recently rebooted or reflashed your core device, you should see the startup logs in your terminal.
+
+Normally, however, the core device IP is supplied through the *device database* for your system, which comes in the form of a Python script called ``device_db.py`` (see also :ref:`device-db`). If you purchased a system from M-Labs, the ``device_db.py`` for your system will have been provided for you, either on the USB stick, inside ``~/artiq`` on your NUC, or sent by email.
+
+Make sure the field ``core_addr`` at the top of the file is set to your core device's correct IP address, and always execute :mod:`~artiq.frontend.artiq_coremgmt` from the same directory the device database is placed in.
+
+Once you can reach your core device, the IP can be changed at any time by running: ::
+
+  $ artiq_coremgmt [-D old_IP] config write -s ip <new_IP>
+
+and then rebooting the device: ::
+
+  $ artiq_coremgmt [-D old_IP] reboot
+
+Make sure to correspondingly edit your ``device_db.py`` after rebooting.
+
+.. _networking-tips:
+
+Tips and troubleshooting
+^^^^^^^^^^^^^^^^^^^^^^^^
+For Kasli-SoC:
+    If the ``ip`` config is not set, Kasli-SoC firmware defaults to using the IP address ``192.168.1.56``.
+
+For ZC706:
+    If the ``ip`` config is not set, ZC706 firmware defaults to using the IP address ``192.168.1.52``.
+
+For Kasli or KC705:
+    If the ``ip`` config field is not set or set to ``use_dhcp``, the device will attempt to obtain an IP address and default gateway using DHCP. The chosen IP address will be in log output, which can be accessed via the :ref:`UART log <connecting-UART>`.
+
+    If a static IP address is preferred, it can be flashed directly (OpenOCD must be installed and configured, as in :doc:`flashing`), along with, as necessary, default gateway, IPv6, and/or MAC address: ::
+
+        $ artiq_mkfs flash_storage.img [-s mac xx:xx:xx:xx:xx:xx] [-s ip xx.xx.xx.xx/xx] [-s ipv4_default_route xx.xx.xx.xx] [-s ip6 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/xx] [-s ipv6_default_route xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]
+        $ artiq_flash -t [board] -V [variant] -f flash_storage.img storage start
+
+On Kasli or Kasli-SoC devices, specifying the MAC address is unnecessary, as they can obtain it from their EEPROM. If you only want to access the core device from the same subnet, default gateway and IPv4 prefix length may also be ommitted. On any board, once a device can be reached by :mod:`~artiq.frontend.artiq_coremgmt`, these values can be set and edited at any time, following the procedure for IP above.
+
+Regarding IPv6, note that the device also has a link-local address that corresponds to its EUI-64, which can be used simultaneously to the (potentially unrelated) IPv6 address defined by using the ``ip6`` configuration key.
+
+If problems persist, see the :ref:`network troubleshooting <faq-networking>` section of the FAQ.
+
+.. _core-device-config:
+
+Configuring the core device
+---------------------------
+
+.. note::
+  The following steps are optional, and you only need to execute them if they are necessary for your specific system. To learn more about how ARTIQ works and how to use it first, you might skip to the first tutorial page, :doc:`rtio`. For all configuration options, the core device generally must be restarted for changes to take effect.
+
+Flash idle and/or startup kernel
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The *idle kernel* is the kernel (that is, a piece of code running on the core device; see :doc:`rtio` for further explanation) which the core device runs in between experiments and whenever not connected to the host. It is saved directly to the core device's flash storage in compiled form. Potential uses include cleanup of the environment between experiments, state maintenance for certain hardware, or anything else that should run continuously whenever the system is not otherwise occupied.
+
+To flash an idle kernel, first write an idle experiment. Note that since the idle kernel runs regardless of whether the core device is connected to the host, remote procedure calls or RPCs (functions called by a kernel to run on the host) are forbidden and the ``run()`` method must be a kernel marked with ``@kernel``. Once written, you can compile and flash your idle experiment: ::
+
+  $ artiq_compile idle.py
+  $ artiq_coremgmt config write -f idle_kernel idle.elf
+
+The *startup kernel* is a kernel executed once and only once immediately whenever the core device powers on. Uses include initializing DDSes and setting TTL directions. For DRTIO systems, the startup kernel should wait until the desired destinations, including local RTIO, are up, using ``self.core.get_rtio_destination_status`` (see :meth:`~artiq.coredevice.core.Core.get_rtio_destination_status`).
+
+To flash a startup kernel, proceed as with the idle kernel, but using the ``startup_kernel`` key in the :mod:`~artiq.frontend.artiq_coremgmt` command.
+
+.. note::
+  Subkernels (see :doc:`using_drtio_subkernels`) are allowed in idle (and startup) experiments without any additional ceremony. :mod:`~artiq.frontend.artiq_compile` will produce a ``.tar`` rather than a ``.elf``; simply substitute ``idle.tar`` for ``idle.elf`` in the ``artiq_coremgmt config write`` command.
+
+Select the RTIO clock source
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The core device may use any of an external clock signal, its internal clock with external frequency reference, or its internal clock with internal crystal reference. Clock source and timing are set at power-up. To find out what clock signal you are using, check the startup logs with ``artiq_coremgmt log``.
+
+The default is to use an internal 125MHz clock. To select a source, use a command of the form: ::
+
+  $ artiq_coremgmt config write -s rtio_clock int_125  # internal 125MHz clock (default)
+  $ artiq_coremgmt config write -s rtio_clock ext0_synth0_10to125  # external 10MHz reference used to synthesize internal 125MHz
+
+See :ref:`core-device-clocking` for availability of specific options.
+
+.. _config-rtiomap:
+
+Set up resolving RTIO channels to their names
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This feature allows you to print the channels' respective names alongside with their numbers in RTIO error messages. To enable it, run the :mod:`~artiq.frontend.artiq_rtiomap` tool and write its result into the device config at the ``device_map`` key: ::
+
+  $ artiq_rtiomap dev_map.bin
+  $ artiq_coremgmt config write -f device_map dev_map.bin
+
+More information on the ``artiq_rtiomap`` utility can be found on the :ref:`Utilities <rtiomap-tool>` page.
+
+Enable event spreading
+^^^^^^^^^^^^^^^^^^^^^^
+
+This feature changes the logic used for queueing RTIO output events in the core device for a more efficient use of FPGA resources, at the cost of introducing nondeterminism and potential unpredictability in certain timing errors (specifically gateware :ref:`sequence errors<sequence-errors>`). It can be enabled with the config key ``sed_spread_enable``. See :ref:`sed-event-spreading`.
+
+Load the DRTIO routing table
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you are using DRTIO and the default routing table (for a star topology) is not suitable to your needs, you will first need to prepare and load a different routing table. See :ref:`Using DRTIO <drtio-routing>`.

doc/manual/core_device.rst

@@ -1,82 +1,152 @@
 Core device
 ===========
 
-The core device is a FPGA-based hardware component that contains a softcore CPU tightly coupled with the so-called RTIO core that provides precision timing. The CPU executes Python code that is statically compiled by the ARTIQ compiler, and communicates with the core device peripherals (TTL, DDS, etc.) over the RTIO core. This architecture provides high timing resolution, low latency, low jitter, high level programming capabilities, and good integration with the rest of the Python experiment code.
+The core device is a FPGA-based hardware component that contains a softcore or hardcore CPU tightly coupled with the so-called RTIO core, which runs in gateware and provides precision timing. The CPU executes Python code that is statically compiled by the ARTIQ compiler and communicates with peripherals (TTL, DDS, etc.) through the RTIO core, as described in :doc:`rtio`. This architecture provides high timing resolution, low latency, low jitter, high-level programming capabilities, and good integration with the rest of the Python experiment code.
 
-While it is possible to use all the other parts of ARTIQ (controllers, master, GUI, dataset management, etc.) without a core device, many experiments require it.
+While it is possible to use the other parts of ARTIQ (controllers, master, GUI, dataset management, etc.) without a core device, most use cases will require it.
 
+.. _configuration-storage:
 
-.. _core-device-flash-storage:
+Configuration storage
+---------------------
 
-Flash storage
+The core device reserves some storage space (either flash or directly on SD card, depending on target board) to store configuration data. The configuration data is organized as a list of key-value records, accessible either through :mod:`~artiq.frontend.artiq_mkfs` and :mod:`~artiq.frontend.artiq_flash` or, preferably in most cases, the ``config`` option of the :mod:`~artiq.frontend.artiq_coremgmt` core management tool (see below). Information can be stored to keys of any name, but the specific keys currently used and referenced by ARTIQ are summarized below:
-*************
 
-The core device contains some flash space that can be used to store configuration data.
+``idle_kernel``
+  Stores (compiled ``.tar`` or ``.elf`` binary of) idle kernel. See :ref:`core-device-config`.
+``startup_kernel``
+  Stores (compiled ``.tar`` or ``.elf`` binary of) startup kernel. See :ref:`core-device-config`.
+``ip``
+  Sets IP address of core device. For this and other networking options see also :ref:`core-device-networking`.
+``mac``
+  Sets MAC address of core device. Unnecessary on Kasli or Kasli-SoC, which can obtain it from EEPROM.
+``ipv4_default_route``
+  Sets IPv4 default route.
+``ip6``
+  Sets IPv6 address of core device. Can be set irrespective of and used simultaneously as IPv4 address above.
+``ipv6_default_route``
+  Sets IPv6 default route.
+``sed_spread_enable``
+  If set to ``1``, will activate :ref:`sed-event-spreading` in this core device. Needs to be set separately for satellite devices in a DRTIO setting.
+``log_level``
+  Sets core device log level. Possible levels are ``TRACE``, ``DEBUG``, ``INFO``, ``WARN``, ``ERROR``, and ``OFF``. Note that enabling higher log levels will produce some core device slowdown.
+``uart_log_level``
+  Sets UART log level, with same options. Printing a large number of messages to UART log will produce a significant slowdown.
+``rtio_clock``
+  Sets the RTIO clock; see :ref:`core-device-clocking`.
+``routing_table``
+  Sets the routing table in DRTIO systems; see :ref:`drtio-routing`. If not set, a star topology is assumed.
+``device_map``
+  If set, allows the core log to connect RTIO channels to device names and use device names as well as channel numbers in log output. A correctly formatted table can be automatically generated with :mod:`~artiq.frontend.artiq_rtiomap`, see :ref:`Utilities<rtiomap-tool>`.
+``net_trace``
+  If set to ``1``, will activate net trace (print all packets sent and received to UART and core log). This will considerably slow down all network response from the core. Not applicable for ARTIQ-Zynq (Kasli-SoC, ZC706).
+``panic_reset``
+  If set to ``1``, core device will restart automatically.  Not applicable for ARTIQ-Zynq.
+``no_flash_boot``
+  If set to ``1``, will disable flash boot. Network boot is attempted if possible. Not applicable for ARTIQ-Zynq.
+``boot``
+  Allows full firmware/gateware (``boot.bin``) to be written with :mod:`~artiq.frontend.artiq_coremgmt`, on ARTIQ-Zynq systems only.
 
-This storage area is used to store the core device MAC address, IP address and even the idle kernel.
+Common configuration commands
+-----------------------------
 
-The flash storage area is one sector (typically 64 kB) large and is organized as a list of key-value records.
+To write, then read, the value ``test_value`` in the key ``my_key``::
 
-This flash storage space can be accessed by using ``artiq_coremgmt`` (see: :ref:`core-device-management-tool`).
+    $ artiq_coremgmt config write -s my_key test_value
+    $ artiq_coremgmt config read my_key
+    b'test_value'
 
-.. _board-ports:
+You do not need to remove a record in order to change its value. Just overwrite it::
+
+    $ artiq_coremgmt config write -s my_key some_value
+    $ artiq_coremgmt config write -s my_key some_other_value
+    $ artiq_coremgmt config read my_key
+    b'some_other_value'
+
+You can write several records at once::
+
+    $ artiq_coremgmt config write -s key1 value1 -f key2 filename -s key3 value3
+
+You can also write entire files in a record using the ``-f`` option. This is useful for instance to write the startup and idle kernels into the flash storage::
+
+    $ artiq_coremgmt config write -f idle_kernel idle.elf
+    $ artiq_coremgmt config read idle_kernel | head -c9
+    b'\x7fELF
+
+The same option is used to write ``boot.bin`` in ARTIQ-Zynq. Note that the ``boot`` key is write-only.
+
+See also the full reference of :mod:`~artiq.frontend.artiq_coremgmt` in :ref:`Utilities <core-device-management-tool>`.
+
+.. _core-device-clocking:
+
+Clocking
+--------
+
+The core device generates the RTIO clock using a PLL locked either to an internal crystal or to an external frequency reference. If choosing the latter, external reference must be provided (via front panel SMA input on Kasli boards). Valid configuration options include:
+
+  * ``int_100`` - internal crystal reference is used to synthesize a 100MHz RTIO clock,
+  * ``int_125`` - internal crystal reference is used to synthesize a 125MHz RTIO clock (default option),
+  * ``int_150`` - internal crystal reference is used to synthesize a 150MHz RTIO clock.
+  * ``ext0_synth0_10to125`` - external 10MHz reference clock used to synthesize a 125MHz RTIO clock,
+  * ``ext0_synth0_80to125`` - external 80MHz reference clock used to synthesize a 125MHz RTIO clock,
+  * ``ext0_synth0_100to125`` - external 100MHz reference clock used to synthesize a 125MHz RTIO clock,
+  * ``ext0_synth0_125to125`` - external 125MHz reference clock used to synthesize a 125MHz RTIO clock.
+
+The selected option can be observed in the core device boot logs and accessed using ``artiq_coremgmt config`` with key ``rtio_clock``.
+
+As of ARTIQ 8, it is now possible for Kasli and Kasli-SoC configurations to enable WRPLL -- a clock recovery method using `DDMTD <http://white-rabbit.web.cern.ch/documents/DDMTD_for_Sub-ns_Synchronization.pdf>`_ and Si549 oscillators -- both to lock the main RTIO clock and (in DRTIO configurations) to lock satellites to master. This is set by the ``enable_wrpll`` option in the :ref:`JSON description file <system-description>`. Because WRPLL requires slightly different gateware and firmware, it is necessary to re-flash devices to enable or disable it in extant systems. If you would like to obtain the firmware for a different WRPLL setting through AFWS, write to the helpdesk@ email.
+
+If phase noise performance is the priority, it is recommended to use ``ext0_synth0_125to125`` over other ``ext0`` options, as this bypasses the (noisy) MMCM.
+
+If not using WRPLL, PLL can also be bypassed entirely with the options
+
+    * ``ext0_bypass`` (input clock used directly)
+    * ``ext0_bypass_125`` (explicit alias)
+    * ``ext0_bypass_100`` (explicit alias)
+
+Bypassing the PLL ensures the skews between input clock, downstream clock outputs, and RTIO clock are deterministic across reboots of the system. This is useful when phase determinism is required in situations where the reference clock fans out to other devices before reaching the master.
+
+Board details
+-------------
 
 FPGA board ports
-****************
+^^^^^^^^^^^^^^^^
 
 All boards have a serial interface running at 115200bps 8-N-1 that can be used for debugging.
 
-Kasli
+Kasli and Kasli-SoC
------
+^^^^^^^^^^^^^^^^^^^
 
-`Kasli <https://github.com/m-labs/sinara/wiki/Kasli>`_ is a versatile core device designed for ARTIQ as part of the `Sinara <https://github.com/m-labs/sinara/wiki>`_ family of boards. All variants support interfacing to various EEM daughterboards (TTL, DDS, ADC, DAC...) connected directly to it.
+`Kasli <https://github.com/sinara-hw/Kasli/wiki>`_ and `Kasli-SoC <https://github.com/sinara-hw/Kasli-SOC/wiki>`_ are versatile core devices designed for ARTIQ as part of the open-source `Sinara <https://github.com/sinara-hw/meta/wiki>`_ family of boards. All support interfacing to various EEM daughterboards (TTL, DDS, ADC, DAC...) through twelve onboard EEM ports. Kasli is based on a Xilinx Artix-7 FPGA, and Kasli-SoC, which runs on a separate `Zynq port <https://git.m-labs.hk/M-Labs/artiq-zynq>`_ of the ARTIQ firmware, is based on a Zynq-7000 SoC, notably including an ARM CPU allowing for much heavier software computations at high speeds. They are architecturally very different but supply similar feature sets. Kasli itself exists in two versions, of which the improved Kasli v2.0 is now in more common use, but the original v1.0 remains supported by ARTIQ.
 
-Standalone variants
+Kasli can be connected to the network using a 1000Base-X SFP module, installed into the SFP0 cage. Kasli-SoC features a built-in Ethernet port to use instead. If configured as a DRTIO satellite, both boards instead reserve SFP0 for the upstream DRTIO connection; remaining SFP cages are available for downstream connections. Equally, if used as a DRTIO master, all free SFP cages are available for downstream connections (i.e. all but SFP0 on Kasli, all four on Kasli-SoC).
-+++++++++++++++++++
 
-Kasli is connected to the network using a 1000Base-X SFP module. `No-name <https://www.fs.com>`_ BiDi (1000Base-BX) modules have been used successfully. The SFP module for the network should be installed into the SFP0 cage.
+The DRTIO line rate depends upon the RTIO clock frequency running, e.g., at 125MHz the line rate is 2.5Gbps, at 150MHz 3.0Gbps, etc. See below for information on RTIO clocks.
-The other SFP cages are not used.
 
-The RTIO clock frequency is 125MHz or 150MHz, which is generated by the Si5324.
+KC705 and ZC706
+^^^^^^^^^^^^^^^
 
-DRTIO master variants
+Two high-end evaluation kits are also supported as alternative ARTIQ core device target boards, respectively the Kintex7 `KC705 <https://www.xilinx.com/products/boards-and-kits/ek-k7-kc705-g.html>`_ and Zynq-SoC `ZC706 <https://www.xilinx.com/products/boards-and-kits/ek-z7-zc706-g.html>`_, both from Xilinx. ZC706, like Kasli-SoC, runs on the ARTIQ-Zynq port. Both are supported in several set variants, namely NIST CLOCK and QC2 (FMC), either available in master, satellite, or standalone variants. See also :doc:`building_developing` for more on system variants.
-+++++++++++++++++++++
 
-Kasli can be used as a DRTIO master that provides local RTIO channels and can additionally control one DRTIO satellite.
+Common KC705 problems
-
+"""""""""""""""""""""
-The RTIO clock frequency is 125MHz or 150MHz, which is generated by the Si5324. The DRTIO line rate is 2.5Gbps or 3Gbps.
-
-As with the standalone configuration, the SFP module for the Ethernet network should be installed into the SFP0 cage. The DRTIO connections are on SFP1 and SFP2, and optionally on the SATA connector.
-
-DRTIO satellite/repeater variants
-+++++++++++++++++++++++++++++++++
-
-Kasli can be used as a DRTIO satellite with a 125MHz or 150MHz RTIO clock and a 2.5Gbps or 3Gbps DRTIO line rate.
-
-The DRTIO upstream connection is on SFP0 or optionally on the SATA connector, and the remaining SFPs are downstream ports.
-
-KC705
------
-
-An alternative target board for the ARTIQ core device is the KC705 development board from Xilinx. It supports the NIST CLOCK and QC2 hardware (FMC).
-
-Common problems
-+++++++++++++++
 
 * The SW13 switches on the board need to be set to 00001.
 * When connected, the CLOCK adapter breaks the JTAG chain due to TDI not being connected to TDO on the FMC mezzanine.
 * On some boards, the JTAG USB connector is not correctly soldered.
 
 VADJ
-++++
+""""
 
 With the NIST CLOCK and QC2 adapters, for safe operation of the DDS buses (to prevent damage to the IO banks of the FPGA), the FMC VADJ rail of the KC705 should be changed to 3.3V. Plug the Texas Instruments USB-TO-GPIO PMBus adapter into the PMBus connector in the corner of the KC705 and use the Fusion Digital Power Designer software to configure (requires Windows). Write to chip number U55 (address 52), channel 4, which is the VADJ rail, to make it 3.3V instead of 2.5V.  Power cycle the KC705 board to check that the startup voltage on the VADJ rail is now 3.3V.
 
+Variant details
+---------------
 
 NIST CLOCK
-++++++++++
+^^^^^^^^^^
 
-With the CLOCK hardware, the TTL lines are mapped as follows:
+With the KC705 CLOCK hardware, the TTL lines are mapped as follows:
 
 +--------------------+-----------------------+--------------+
 | RTIO channel       | TTL line              | Capability   |
@@ -116,11 +186,21 @@ The board has RTIO SPI buses mapped as follows:
 
 The DDS bus is on channel 27.
 
+The ZC706 variant is identical except for the following differences:
+
+    - The SMA GPIO on channel 18 is replaced by an Input+Output capable PMOD1_0 line.
+    - Since there is no SDIO on the programmable logic side, channel 26 is instead occupied by an additional SPI:
+
++--------------+------------------+--------------+--------------+--------------+
+| RTIO channel | CS_N             | MOSI         | MISO         | CLK          |
++==============+==================+==============+==============+==============+
+| 26           | PMOD_SPI_CS_N    | PMOD_SPI_MOSI| PMOD_SPI_MISO| PMOD_SPI_CLK |
++--------------+------------------+--------------+--------------+--------------+
 
 NIST QC2
-++++++++
+^^^^^^^^
 
-With the QC2 hardware, the TTL lines are mapped as follows:
+With the KC705 QC2 hardware, the TTL lines are mapped as follows:
 
 +--------------------+-----------------------+--------------+
 | RTIO channel       | TTL line              | Capability   |
@@ -154,32 +234,11 @@ The board has RTIO SPI buses mapped as follows:
 
 There are two DDS buses on channels 50 (LPC, DDS0-DDS11) and 51 (HPC, DDS12-DDS23).
 
-
 The QC2 hardware uses TCA6424A I2C I/O expanders to define the directions of its TTL buffers. There is one such expander per FMC card, and they are selected using the PCA9548 on the KC705.
 
-To avoid I/O contention, the startup kernel should first program the TCA6424A expanders and then call ``output()`` on all ``TTLInOut`` channels that should be configured as outputs.
+To avoid I/O contention, the startup kernel should first program the TCA6424A expanders and then call ``output()`` on all ``TTLInOut`` channels that should be configured as outputs. See :mod:`artiq.coredevice.i2c` for more details.
 
-See :mod:`artiq.coredevice.i2c` for more details.
+The ZC706 is identical except for the following differences:
 
-Clocking
+    - The SMA GPIO is once again replaced with PMOD1_0.
-++++++++
+    - The first four TTLs also have edge counters, on channels 52, 53, 54, and 55.
-
-The KC705 in standalone variants supports an internal 125 MHz RTIO clock (based on its crystal oscillator, or external reference for PLL for DRTIO variants) and an external clock, that can be selected using the ``rtio_clock`` configuration entry. Valid values are:
-
-    * ``int_125`` - internal crystal oscillator, 125 MHz output (default),
-    * ``ext0_bypass`` - external clock.
-
-KC705 in DRTIO variants and Kasli generates the RTIO clock using a PLL locked either to an internal crystal or to an external frequency reference. Valid values are:
-
-    * ``int_125`` - internal crystal oscillator using PLL, 125 MHz output (default),
-    * ``int_100`` - internal crystal oscillator using PLL, 100 MHz output,
-    * ``int_150`` - internal crystal oscillator using PLL, 150 MHz output,
-    * ``ext0_synth0_10to125`` - external 10 MHz reference using PLL, 125 MHz output,
-    * ``ext0_synth0_80to125`` - external 80 MHz reference using PLL, 125 MHz output,
-    * ``ext0_synth0_100to125`` - external 100 MHz reference using PLL, 125 MHz output,
-    * ``ext0_synth0_125to125`` - external 125 MHz reference using PLL, 125 MHz output,
-    * ``ext0_bypass``, ``ext0_bypass_125``, ``ext0_bypass_100`` - external clock - with explicit aliases available.
-
-The selected option can be observed in the core device boot logs.
-
-Options ``rtio_clock=int_XXX`` and ``rtio_clock=ext0_synth0_XXXXX`` generate the RTIO clock using a PLL locked either to an internal crystal or to an external frequency reference (depending on exact option). ``rtio_clock=ext0_bypass`` bypasses that PLL and the user must supply the RTIO clock (typically 125 MHz) at the Kasli front panel SMA input. Bypassing the PLL ensures the skews between input clock, Kasli downstream clock outputs, and RTIO clock are deterministic accross reboots of the system. This is useful when phase determinism is required in situtations where the reference clock fans out to other devices before reaching Kasli.

doc/manual/core_drivers_reference.rst

@@ -1,32 +1,31 @@
-Core drivers reference
+Core real-time drivers
 ======================
 
 These drivers are for the core device and the peripherals closely integrated into it, which do not use the controller mechanism.
 
-
 System drivers
 --------------
 
 :mod:`artiq.coredevice.core` module
-+++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.core
     :members:
 
 :mod:`artiq.coredevice.exceptions` module
-+++++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.exceptions
     :members:
 
 :mod:`artiq.coredevice.dma` module
-++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.dma
     :members:
 
 :mod:`artiq.coredevice.cache` module
-++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.cache
     :members:
@@ -36,25 +35,25 @@ Digital I/O drivers
 -------------------
 
 :mod:`artiq.coredevice.ttl` module
-++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.ttl
     :members:
 
 :mod:`artiq.coredevice.edge_counter` module
-++++++++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.edge_counter
     :members:
 
 :mod:`artiq.coredevice.spi2` module
-+++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.spi2
     :members:
 
 :mod:`artiq.coredevice.i2c` module
-++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.i2c
     :members:
@@ -63,49 +62,49 @@ RF generation drivers
 ---------------------
 
 :mod:`artiq.coredevice.urukul` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.urukul
     :members:
 
 :mod:`artiq.coredevice.ad9910` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.ad9910
     :members:
 
 :mod:`artiq.coredevice.ad9912` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.ad9912
     :members:
 
 :mod:`artiq.coredevice.ad9914` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.ad9914
     :members:
 
 :mod:`artiq.coredevice.mirny` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.mirny
     :members:
 
 :mod:`artiq.coredevice.almazny` module
-++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.almazny
     :members:
 
 :mod:`artiq.coredevice.adf5356` module
-+++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.adf5356
     :members:
 
 :mod:`artiq.coredevice.phaser` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.phaser
     :members:
@@ -114,37 +113,37 @@ DAC/ADC drivers
 ---------------
 
 :mod:`artiq.coredevice.ad53xx` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.ad53xx
     :members:
 
 :mod:`artiq.coredevice.zotino` module
-+++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.zotino
     :members:
 
 :mod:`artiq.coredevice.sampler` module
-++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.sampler
     :members:
 
 :mod:`artiq.coredevice.novogorny` module
-++++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.novogorny
     :members:
 
 :mod:`artiq.coredevice.fastino` module
-++++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.fastino
     :members:
 
 :mod:`artiq.coredevice.shuttler` module
-++++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.shuttler
     :members:
@@ -154,14 +153,14 @@ Miscellaneous
 -------------
 
 :mod:`artiq.coredevice.suservo` module
-++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.suservo
     :members:
 
 
 :mod:`artiq.coredevice.grabber` module
-++++++++++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. automodule:: artiq.coredevice.grabber
     :members:

doc/manual/core_language_reference.rst

@@ -1,28 +1,34 @@
-Core language reference
+Core language and environment
-=======================
+=============================
 
-The most commonly used features from the ARTIQ language modules and from the core device modules are bundled together in :mod:`artiq.experiment` and can be imported with ``from artiq.experiment import *``.
+The most commonly used features from the ARTIQ language modules and from the core device modules are bundled together in ``artiq.experiment`` and can be imported with ``from artiq.experiment import *``.
 
 :mod:`artiq.language.core` module
 ---------------------------------
 
 .. automodule:: artiq.language.core
+    :member-order: bysource
     :members:
 
 :mod:`artiq.language.environment` module
 ----------------------------------------
 
 .. automodule:: artiq.language.environment
+    :member-order: bysource
     :members:
 
 :mod:`artiq.language.scan` module
 ----------------------------------------
 
 .. automodule:: artiq.language.scan
+    :member-order: bysource
     :members:
 
 :mod:`artiq.language.units` module
 ----------------------------------
 
-This module contains floating point constants that correspond to common physical units (ns, MHz, ...).
+.. displays nothing, but makes references work
-They are provided for convenience (e.g write ``MHz`` instead of ``1000000.0``) and code clarity purposes.
+.. automodule:: artiq.language.units
+    :members:
+
+    This module contains floating point constants that correspond to common physical units (ns, MHz, ...). They are provided for convenience (e.g write ``MHz`` instead of ``1000000.0``) and code clarity purposes.

doc/manual/default_network_ports.rst

@@ -10,9 +10,9 @@ Default network ports
 +---------------------------------+--------------+
 | Core device (analyzer)          | 1382         |
 +---------------------------------+--------------+
-| Moninj (core device or proxy)   | 1383         |
+| MonInj (core device or proxy)   | 1383         |
 +---------------------------------+--------------+
-| Moninj (proxy control)          | 1384         |
+| MonInj (proxy control)          | 1384         |
 +---------------------------------+--------------+
 | Core analyzer proxy (proxy)     | 1385         |
 +---------------------------------+--------------+

doc/manual/developing.rst

@@ -1,22 +0,0 @@
-.. _developing-artiq:
-
-Developing ARTIQ
-^^^^^^^^^^^^^^^^
-
-.. warning::
-    This section is only for software or FPGA developers who want to modify ARTIQ. The steps described here are not required if you simply want to run experiments with ARTIQ. If you purchased a system from M-Labs or QUARTIQ, we normally provide board binaries for you.
-
-The easiest way to obtain an ARTIQ development environment is via the Nix package manager on Linux. The Nix system is used on the `M-Labs Hydra server <https://nixbld.m-labs.hk/>`_ to build ARTIQ and its dependencies continuously; it ensures that all build instructions are up-to-date and allows binary packages to be used on developers' machines, in particular for large tools such as the Rust compiler.
-ARTIQ itself does not depend on Nix, and it is also possible to compile everything from source (look into the ``flake.nix`` file and/or nixpkgs, and run the commands manually) - but Nix makes the process a lot easier.
-
-* Download Vivado from Xilinx and install it (by running the official installer in a FHS chroot environment if using NixOS; the ARTIQ flake provides such an environment, which can be entered with the command `vivado-env`). If you do not want to write to ``/opt``, you can install it in a folder of your home directory. The "appropriate" Vivado version to use for building the bitstream can vary. Some versions contain bugs that lead to hidden or visible failures, others work fine. Refer to `Hydra <https://nixbld.m-labs.hk/>`_ and/or the ``flake.nix`` file from the ARTIQ repository in order to determine which version is used at M-Labs. If the Vivado GUI installer crashes, you may be able to work around the problem by running it in unattended mode with a command such as ``./xsetup -a XilinxEULA,3rdPartyEULA,WebTalkTerms -b Install -e 'Vitis Unified Software Platform' -l /opt/Xilinx/``.
-* During the Vivado installation, uncheck ``Install cable drivers`` (they are not required as we use better and open source alternatives).
-* Install the `Nix package manager <http://nixos.org/nix/>`_, version 2.4 or later. Prefer a single-user installation for simplicity.
-* If you did not install Vivado in its default location ``/opt``, clone the ARTIQ Git repository and edit ``flake.nix`` accordingly.
-* Enable flakes in Nix by e.g. adding ``experimental-features = nix-command flakes`` to ``nix.conf`` (for example ``~/.config/nix/nix.conf``).
-* Clone the ARTIQ Git repository and run ``nix develop`` at the root (where ``flake.nix`` is).
-* Make the current source code of ARTIQ available to the Python interpreter by running ``export PYTHONPATH=`pwd`:$PYTHONPATH``.
-* You can then build the firmware and gateware with a command such as ``$ python -m artiq.gateware.targets.kasli <description>.json``, using a JSON system description file. 
-* Flash the binaries into the FPGA board with a command such as ``$ artiq_flash --srcbuild -d artiq_kasli/<your_variant>``. You need to configure OpenOCD as explained :ref:`in the user section <configuring-openocd>`. OpenOCD is already part of the flake's development environment.
-* Check that the board boots and examine the UART messages by running a serial terminal program, e.g. ``$ flterm /dev/ttyUSB1`` (``flterm`` is part of MiSoC and installed in the flake's development environment). Leave the terminal running while you are flashing the board, so that you see the startup messages when the board boots immediately after flashing. You can also restart the board (without reflashing it) with ``$ artiq_flash start``.
-* The communication parameters are 115200 8-N-1. Ensure that your user has access to the serial device (e.g. by adding the user account to the ``dialout`` group).

doc/manual/developing_a_ndsp.rst

@@ -1,39 +1,41 @@
 Developing a Network Device Support Package (NDSP)
 ==================================================
 
-Most ARTIQ devices are interfaced through "controllers" that expose RPC interfaces to the network (based on SiPyCo). The master never does direct I/O to the devices, but issues RPCs to the controllers when needed. As opposed to running everything on the master, this architecture has those main advantages:
+Besides the kind of specialized real-time hardware most of ARTIQ is concerned with the control and management of, ARTIQ also easily handles more conventional 'slow' devices. This is done through *controllers*, based on `SiPyCo <https://github.com/m-labs/sipyco>`_ (manual hosted `here <https://m-labs.hk/artiq/sipyco-manual/>`_), which expose remote procedure call (RPC) interfaces to the network. This allows experiments to issue RPCs to the controllers as necessary, without needing to do direct I/O to the devices. Some advantages of this architecture include:
 
-* Each driver can be run on a different machine, which alleviates cabling issues and OS compatibility problems.
+* Controllers/drivers can be run on different machines, alleviating cabling issues and OS compatibility problems.
 * Reduces the impact of driver crashes.
 * Reduces the impact of driver memory leaks.
 
-This mechanism is for "slow" devices that are directly controlled by a PC, typically over a non-realtime channel such as USB.
+Certain devices (such as the PDQ2) may still perform real-time operations by having certain controls physically connected to the core device (for example, the trigger and frame selection signals on the PDQ2). For handling such cases, parts of the support infrastructure may be kernels executed on the core device.
 
-Certain devices (such as the PDQ2) may still perform real-time operations by having certain controls physically connected to the core device (for example, the trigger and frame selection signals on the PDQ2). For handling such cases, parts of the NDSPs may be kernels executed on the core device.
+.. seealso::
+    Some NDSPs for particular devices have already been written and made available to the public. A (non-exhaustive) list can be found on the page :doc:`list_of_ndsps`.
 
-A network device support package (NDSP) is composed of several parts:
+Components of a NDSP
+--------------------
 
-1. The `driver`, which contains the Python API functions to be called over the network, and performs the I/O to the device. The top-level module of the driver is called ``artiq.devices.XXX.driver``.
+Full support for a specific device, called a network device support package or NDSP, requires several parts:
-2. The `controller`, which instantiates, initializes and terminates the driver, and sets up the RPC server. The controller is a front-end command-line tool to the user and is called ``artiq.frontend.aqctl_XXX``. A ``setup.py`` entry must also be created to install it.
+
-3. An optional `client`, which connects to the controller and exposes the functions of the driver as a command-line interface. Clients are front-end tools (called ``artiq.frontend.aqcli_XXX``) that have ``setup.py`` entries. In most cases, a custom client is not needed and the generic ``sipyco_rpctool`` utility can be used instead. Custom clients are only required when large amounts of data must be transferred over the network API, that would be unwieldy to pass as ``sipyco_rpctool`` command-line parameters.
+1. The `driver`, which contains the Python API functions to be called over the network and performs the I/O to the device. The top-level module of the driver should be called ``artiq.devices.XXX.driver``.
+2. The `controller`, which instantiates, initializes and terminates the driver, and sets up the RPC server. The controller is a front-end command-line tool to the user and should be called ``artiq.frontend.aqctl_XXX``. A ``setup.py`` entry must also be created to install it.
+3. An optional `client`, which connects to the controller and exposes the functions of the driver as a command-line interface. Clients are front-end tools (called ``artiq.frontend.aqcli_XXX``) that have ``setup.py`` entries. In most cases, a custom client is not needed and the generic ``sipyco_rpctool`` utility can be used instead. Custom clients are only required when large amounts of data, which would be unwieldy to pass as ``sipyco_rpctool`` command-line parameters, must be transferred over the network API.
 4. An optional `mediator`, which is code executed on the client that supplements the network API. A mediator may contain kernels that control real-time signals such as TTL lines connected to the device. Simple devices use the network API directly and do not have a mediator. Mediator modules are called ``artiq.devices.XXX.mediator`` and their public classes are exported at the ``artiq.devices.XXX`` level (via ``__init__.py``) for direct import and use by the experiments.
 
 The driver and controller
 -------------------------
 
-A controller is a piece of software that receives commands from a client over the network (or the ``localhost`` interface), drives a device, and returns information about the device to the client. The mechanism used is remote procedure calls (RPCs) using ``sipyco.pc_rpc``, which makes the network layers transparent for the driver's user.
+As an example, we will develop a controller for a "device" that is very easy to work with: the console from which the controller is run. The operation that the driver will implement (and offer as an RPC) is writing a message to that console.
 
-The controller we will develop is for a "device" that is very easy to work with: the console from which the controller is run. The operation that the driver will implement is writing a message to that console.
+To use RPCs, the functions that a driver provides must be the methods of a single object. We will thus define a class that provides our message-printing method: ::
-
-For using RPC, the functions that a driver provides must be the methods of a single object. We will thus define a class that provides our message-printing method: ::
 
     class Hello:
         def message(self, msg):
             print("message: " + msg)
 
-For a more complex driver, you would put this class definition into a separate Python module called ``driver``.
+For a more complex driver, we would place this class definition into a separate Python module called ``driver``. In this example, for simplicity, we can include it in the controller module.
 
-To turn it into a server, we use ``sipyco.pc_rpc``. Import the function we will use: ::
+For the controller itself, we will turn this method into a server using ``sipyco.pc_rpc``. Import the function we will use: ::
 
     from sipyco.pc_rpc import simple_server_loop
 
@@ -45,13 +47,14 @@ and add a ``main`` function that is run when the program is executed: ::
     if __name__ == "__main__":
         main()
 
-:tip: Defining the ``main`` function instead of putting its code directly in the ``if __name__ == "__main__"`` body enables the controller to be used as well as a setuptools entry point.
+.. tip::
+     Defining the ``main`` function instead of putting its code directly in the ``if __name__ == "__main__"`` body enables the controller to be used as a setuptools entry point as well.
 
-The parameters ``::1`` and ``3249`` are respectively the address to bind the server to (IPv6 localhost) and the TCP port to use. Then add a line: ::
+The parameters ``::1`` and ``3249`` are respectively the address to bind the server to (in this case, we use IPv6 localhost) and the TCP port. Add a line: ::
 
     #!/usr/bin/env python3
 
-at the beginning of the file, save it to ``aqctl_hello.py`` and set its execution permissions: ::
+at the beginning of the file, save it as ``aqctl_hello.py``, and set its execution permissions: ::
 
     $ chmod 755 aqctl_hello.py
 
@@ -59,16 +62,18 @@ Run it as: ::
 
     $ ./aqctl_hello.py
 
-and verify that you can connect to the TCP port: ::
+In a different console, verify that you can connect to the TCP port: ::
 
     $ telnet ::1 3249
     Trying ::1...
     Connected to ::1.
     Escape character is '^]'.
 
-:tip: Use the key combination Ctrl-AltGr-9 to get the ``telnet>`` prompt, and enter ``close`` to quit Telnet. Quit the controller with Ctrl-C.
+.. tip ::
 
-Also verify that a target (service) named "hello" (as passed in the first argument to ``simple_server_loop``) exists using the ``sipyco_rpctool`` program from the ARTIQ front-end tools: ::
+    To exit telnet, use the escape character combination (Ctrl + ]) to access the ``telnet>`` prompt, and then enter ``quit`` or ``close`` to close the connection.
+
+Also verify that a target (i.e. available service for RPC) named "hello" exists: ::
 
     $ sipyco_rpctool ::1 3249 list-targets
     Target(s):   hello
@@ -76,7 +81,7 @@ Also verify that a target (service) named "hello" (as passed in the first argume
 The client
 ----------
 
-Clients are small command-line utilities that expose certain functionalities of the drivers. The ``sipyco_rpctool`` utility contains a generic client that can be used in most cases, and developing a custom client is not required. Try these commands ::
+Clients are small command-line utilities that expose certain functionalities of the drivers. The ``sipyco_rpctool`` utility contains a generic client that can be used in most cases, and developing a custom client is not required. You have already used it above in the form of ``list-targets``. Try these commands: ::
 
     $ sipyco_rpctool ::1 3249 list-methods
     $ sipyco_rpctool ::1 3249 call message test
@@ -98,21 +103,54 @@ In case you are developing a NDSP that is complex enough to need a custom client
     if __name__ == "__main__":
         main()
 
-Run it as before, while the controller is running. You should see the message appearing on the controller's terminal: ::
+Run it as before, making sure the controller is running first. You should see the message appear in the controller's terminal: ::
 
     $ ./aqctl_hello.py
     message: Hello World!
 
-When using the driver in an experiment, the ``Client`` instance can be returned by the environment mechanism (via the ``get_device`` and ``attr_device`` methods of :class:`artiq.language.environment.HasEnvironment`) and used normally as a device.
+We see that the client has made a request to the server, which has, through the driver, performed the requisite I/O with the "device" (its console), resulting in the operation we wanted. Success!
 
-:warning: RPC servers operate on copies of objects provided by the client, and modifications to mutable types are not written back. For example, if the client passes a list as a parameter of an RPC method, and that method ``append()s`` an element to the list, the element is not appended to the client's list.
+.. warning::
+    Note that RPC servers operate on copies of objects provided by the client, and modifications to mutable types are not written back. For example, if the client passes a list as a parameter of an RPC method, and that method ``append()s`` an element to the list, the element is not appended to the client's list.
+
+To access this driver in an experiment, we can retrieve the ``Client`` instance with the ``get_device`` and ``set_device`` methods of :class:`artiq.language.environment.HasEnvironment`, and then use it like any other device (provided the controller is running and accessible at the time).
+
+.. _ndsp-integration:
+
+Integration with ARTIQ experiments
+----------------------------------
+
+Generally we will want to add the device to our :ref:`device database <device-db>` so that we can add it to an experiment with ``self.setattr_device`` and so the controller can be started and stopped automatically by a controller manager (the :mod:`~artiq_comtools.artiq_ctlmgr` utility from ``artiq-comtools``). To do so, add an entry to your device database in this format: ::
+
+	device_db.update({
+            "hello": {
+        	"type": "controller",
+        	"host": "::1",
+        	"port": 3249,
+        	"command": "python /abs/path/to/aqctl_hello.py -p {port}"
+    	    },
+	})
+
+Now it can be added using ``self.setattr_device("hello")`` in the ``build()`` phase of the experiment, and its methods accessed via: ::
+
+	self.hello.message("Hello world!")
+
+.. note::
+    In order to be correctly started and stopped by a controller manager, your controller must additionally implement a ``ping()`` method, which should simply return true, e.g. ::
+
+        def ping(self):
+            return True
+
+
+Remote execution support
+------------------------
+
+If you wish to support remote execution in your controller, you may do so by simply replacing ``simple_server_loop`` with :class:`sipyco.remote_exec.simple_rexec_server_loop`.
 
 Command-line arguments
 ----------------------
 
-Use the Python ``argparse`` module to make the bind address(es) and port configurable on the controller, and the server address, port and message configurable on the client.
+Use the Python ``argparse`` module to make the bind address(es) and port configurable on the controller, and the server address, port and message configurable on the client. We suggest naming the controller parameters ``--bind`` (which adds a bind address in addition to a default binding to localhost), ``--no-bind-localhost`` (which disables the default binding to localhost), and ``--port``, so that those parameters stay consistent across controllers. Use ``-s/--server`` and ``--port`` on the client. The :meth:`sipyco.common_args.simple_network_args` library function adds such arguments for the controller, and the :meth:`sipyco.common_args.bind_address_from_args` function processes them.
-
-We suggest naming the controller parameters ``--bind`` (which adds a bind address in addition to a default binding to localhost), ``--no-bind-localhost`` (which disables the default binding to localhost), and ``--port``, so that those parameters stay consistent across controllers. Use ``-s/--server`` and ``--port`` on the client. The ``sipyco.common_args.simple_network_args`` library function adds such arguments for the controller, and the ``sipyco.common_args.bind_address_from_args`` function processes them.
 
 The controller's code would contain something similar to this: ::
 
@@ -132,7 +170,7 @@ We suggest that you define a function ``get_argparser`` that returns the argumen
 Logging
 -------
 
-For the debug, information and warning messages, use the ``logging`` Python module and print the log on the standard error output (the default setting). The logging level is by default "WARNING", meaning that only warning messages and more critical messages will get printed (and no debug nor information messages). By calling ``sipyco.common_args.verbosity_args`` with the parser as argument, you add support for the ``--verbose`` (``-v``) and ``--quiet`` (``-q``) arguments in the parser. Each occurrence of ``-v`` (resp. ``-q``) in the arguments will increase (resp. decrease) the log level of the logging module. For instance, if only one ``-v`` is present in the arguments, then more messages (info, warning and above) will get printed. If only one ``-q`` is present in the arguments, then only errors and critical messages will get printed. If ``-qq`` is present in the arguments, then only critical messages will get printed, but no debug/info/warning/error.
+For debug, information and warning messages, use the ``logging`` Python module and print the log on the standard error output (the default setting). As in other areas, there are five logging levels, from most to least critical, ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, and ``DEBUG``. By default, the logging level starts at ``WARNING``, meaning it will print messages of level WARNING and above (and no debug nor information messages). By calling ``sipyco.common_args.verbosity_args`` with the parser as argument, you add support for the ``--verbose`` (``-v``) and ``--quiet`` (``-q``) arguments in your controller. Each occurrence of ``-v`` (resp. ``-q``) in the arguments will increase (resp. decrease) the log level of the logging module. For instance, if only one ``-v`` is present, then more messages (INFO and above) will be printed. If only one ``-q`` is present in the arguments, then ERROR and above will be printed. If ``-qq`` is present in the arguments, then only CRITICAL will be printed.
 
 The program below exemplifies how to use logging: ::
 
@@ -168,46 +206,26 @@ The program below exemplifies how to use logging: ::
     if __name__ == "__main__":
         main()
 
+Additional guidelines
+---------------------
 
-Integration with ARTIQ experiments
+Command line and options
-----------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^
 
-To integrate the controller into an experiment, simply add an entry into the ``device_db.py`` following the example shown below for the ``aqctl_hello.py`` from above: ::
+* Controllers should be able to operate in "simulation" mode, specified with ``--simulation``, where they behave properly even if the associated hardware is not connected. For example, they can print the data to the console instead of sending it to the device, or dump it into a file.
+* The device identification (e.g. serial number, or entry in ``/dev``) to attach to must be passed as a command-line parameter to the controller. We suggest using ``-d`` and ``--device`` as parameter names.
+* Keep command line parameters consistent across clients/controllers. When adding new command line options, look for a client/controller that does a similar thing and follow its use of ``argparse``. If the original client/controller could use ``argparse`` in a better way, improve it.
 
-	device_db.update({
+Style
-    	"hello": {
+^^^^^
-        	"type": "controller",
-        	"host": "::1",
-        	"port": 3249,
-        	"command": "python /abs/path/to/aqctl_hello.py -p {port}" 
-    	},
-	})
-	
-From the experiment this can now be added using ``self.setattr_device("hello")`` in the ``build()`` phase of the experiment, and methods accessed via: ::
-	
-	self.hello.message("Hello world!")
-
-To automatically initiate controllers the ``artiq_ctlmgr`` utility from ``artiq-comtools`` can be used. By default, this initiates all controllers hosted on the local address of master connection.
-
-Remote execution support
-------------------------
-
-If you wish to support remote execution in your controller, you may do so by simply replacing ``simple_server_loop`` with :class:`sipyco.remote_exec.simple_rexec_server_loop`.
-
-General guidelines
-------------------
 
 * Do not use ``__del__`` to implement the cleanup code of your driver. Instead, define a ``close`` method, and call it using a ``try...finally...`` block in the controller.
 * Format your source code according to PEP8. We suggest using ``flake8`` to check for compliance.
 * Use new-style formatting (``str.format``) except for logging where it is not well supported, and double quotes for strings.
-* The device identification (e.g. serial number, or entry in ``/dev``) to attach to must be passed as a command-line parameter to the controller. We suggest using ``-d`` and ``--device`` as parameter name.
-* Controllers must be able to operate in "simulation" mode, where they behave properly even if the associated hardware is not connected. For example, they can print the data to the console instead of sending it to the device, or dump it into a file.
-* The simulation mode is entered whenever the ``--simulation`` option is specified.
-* Keep command line parameters consistent across clients/controllers. When adding new command line options, look for a client/controller that does a similar thing and follow its use of ``argparse``. If the original client/controller could use ``argparse`` in a better way, improve it.
 * Use docstrings for all public methods of the driver (note that those will be retrieved by ``sipyco_rpctool``).
-* Choose a free default TCP port and add it to the default port list in this manual.
+* Choose a free default TCP port and add it to the :doc:`default port list<default_network_ports>` in this manual.
 
 Hosting your code
 -----------------
 
-We suggest that you create a Git repository for your code, and publish it on https://git.m-labs.hk/, GitLab, GitHub, or a similar website of your choosing. Then send us a message or pull request for your NDSP to be added to the list in this manual.
+We suggest that you create a Git repository for your code, and publish it on https://git.m-labs.hk/, GitLab, GitHub, or a similar website of your choosing. Then send us a message or pull request for your NDSP to be added to :doc:`the list in this manual <list_of_ndsps>`.

doc/manual/drtio.rst

@@ -1,17 +1,11 @@
-Distributed Real Time Input/Output (DRTIO)
+DRTIO system
-==========================================
+============
 
-DRTIO is a time and data transfer system that allows ARTIQ RTIO channels to be distributed among several satellite devices synchronized and controlled by a central core device.
+DRTIO is the time and data transfer system that allows ARTIQ RTIO channels to be distributed among several satellite devices, synchronized and controlled by a central core device. The main source of DRTIO traffic is the remote control of RTIO output and input channels. The protocol is optimized to maximize throughput and minimize latency, and handles flow control and error conditions (underflows, overflows, etc.)
 
-The link is a high speed duplex serial line operating at 1Gbps or more, over copper or optical fiber.
+The DRTIO protocol also supports auxiliary traffic, which is low-priority and non-realtime, e.g., to override and monitor TTL I/Os. Auxiliary traffic never interrupts or delays the main traffic, so it cannot cause unexpected latencies or exceptions (e.g. RTIO underflows).
 
-The main source of DRTIO traffic is the remote control of RTIO output and input channels. The protocol is optimized to maximize throughput and minimize latency, and handles flow control and error conditions (underflows, overflows, etc.)
+The lower layers of DRTIO are similar to `White Rabbit <https://white-rabbit.web.cern.ch/>`_ , with the following main differences:
-
-The DRTIO protocol also supports auxiliary, low-priority and non-realtime traffic. The auxiliary channel supports overriding and monitoring TTL I/Os. Auxiliary traffic never interrupts or delays the main traffic, so that it cannot cause unexpected poor performance (e.g. RTIO underflows).
-
-Time transfer and clock syntonization is typically done over the serial link alone. The DRTIO code is organized as much as possible to support porting to different types of transceivers (Xilinx MGTs, Altera MGTs, soft transceivers running off regular FPGA IOs, etc.) and different synchronization mechanisms.
-
-The lower layers of DRTIO are similar to White Rabbit, with the following main differences:
 
 * lower latency
 * deterministic latency
@@ -20,80 +14,31 @@ The lower layers of DRTIO are similar to White Rabbit, with the following main d
 * no Ethernet compatibility
 * only star or tree topologies are supported
 
-From ARTIQ kernels, DRTIO channels are used in the same way as local RTIO channels.
+Time transfer and clock syntonization is typically done over the serial link alone. The DRTIO code is written as much as possible to support porting to different types of transceivers (Xilinx MGTs, Altera MGTs, soft transceivers running off regular FPGA IOs, etc.) and different synchronization mechanisms.
-
-.. _using-drtio:
-
-Using DRTIO
------------
 
 Terminology
-+++++++++++
+-----------
 
-In a system of interconnected DRTIO devices, each RTIO core (driving RTIO PHYs; for example a RTIO core would connect to a large bank of TTL signals) is assigned a number and is called a *destination*. One DRTIO device normally contains one RTIO core.
+In a system of interconnected DRTIO devices, each RTIO core (controlling a certain set of associated RTIO channels) is assigned a number and called a *destination*. One DRTIO device normally contains one RTIO core.
 
 On one DRTIO device, the immediate path that a RTIO request must take is called a *hop*: the request can be sent to the local RTIO core, or to another device downstream. Each possible hop is assigned a number. Hop 0 is normally the local RTIO core, and hops 1 and above correspond to the respective downstream ports of the device.
 
 DRTIO devices are arranged in a tree topology, with the core device at the root. For each device, its distance from the root (in number of devices that are crossed) is called its *rank*. The root has rank 0, the devices immediately connected to it have rank 1, and so on.
 
 The routing table
-+++++++++++++++++
+-----------------
 
 The routing table defines, for each destination, the list of hops ("route") that must be taken from the root in order to reach it.
 
-It is stored in a binary format that can be manipulated with the :ref:`artiq_route utility <routing-table-tool>`. The binary file is then programmed into the flash storage of the core device under the ``routing_table`` key. It is automatically distributed to downstream devices when the connections are established. Modifying the routing table requires rebooting the core device for the new table to be taken into account.
+It is stored in a binary format that can be generated and manipulated with the utility :mod:`~artiq.frontend.artiq_route`, see :ref:`drtio-routing`. The binary file is programmed into the flash storage of the core device under the ``routing_table`` key. It is automatically distributed to downstream devices when the connections are established. Modifying the routing table requires rebooting the core device for the new table to be taken into account.
-
-All routes must end with the local RTIO core of the last device (0).
-
-The local RTIO core of the core device is a destination like any other, and it needs to be explicitly part of the routing table for kernels to be able to access it.
-
-If no routing table is programmed, the core device takes a default routing table for a star topology (i.e. with no devices of rank 2 or above), with destination 0 being the core device's local RTIO core and destinations 1 and above corresponding to devices on the respective downstream ports.
-
-Here is an example of creating and programming a routing table for a chain of 3 devices: ::
-
-    # create an empty routing table
-    $ artiq_route rt.bin init
-
-    # set destination 0 to the local RTIO core
-    $ artiq_route rt.bin set 0 0
-
-    # for destination 1, first use hop 1 (the first downstream port)
-    # then use the local RTIO core of that second device.
-    $ artiq_route rt.bin set 1 1 0
-
-    # for destination 2, use hop 1 and reach the second device as
-    # before, then use hop 1 on that device to reach the third
-    # device, and finally use the local RTIO core (hop 0) of the
-    # third device.
-    $ artiq_route rt.bin set 2 1 1 0
-
-    $ artiq_route rt.bin show
-      0:   0
-      1:   1   0
-      2:   1   1   0
-
-    $ artiq_coremgmt config write -f routing_table rt.bin
-
-Addressing distributed RTIO cores from kernels
-++++++++++++++++++++++++++++++++++++++++++++++
-
-Remote RTIO channels are accessed in the same way as local ones. Bits 16-24 of the RTIO channel number define the destination. Bits 0-15 of the RTIO channel number select the channel within the destination.
-
-Link establishment
-++++++++++++++++++
-
-After devices have booted, it takes several seconds for all links in a DRTIO system to become established (especially with the long locking times of low-bandwidth PLLs that are used for jitter reduction purposes). Kernels should not attempt to access destinations until all required links are up (when this happens, the ``RTIODestinationUnreachable`` exception is raised). ARTIQ provides the method :meth:`~artiq.coredevice.core.Core.get_rtio_destination_status` that determines whether a destination can be reached. We recommend calling it in a loop in your startup kernel for each important destination, to delay startup until they all can be reached.
-
-Latency
-+++++++
-
-Each hop increases the RTIO latency of a destination by a significant amount; that latency is however constant and can be compensated for in kernels. To limit latency in a system, fully utilize the downstream ports of devices to reduce the depth of the tree, instead of creating chains.
 
 Internal details
 ----------------
 
+Bits 16-24 of the RTIO channel number (assigned to a respective device in the initial :ref:`system description JSON <system-description>`, and specified again for use of the ARTIQ front-end in the device database) define the destination. Bits 0-15 of the RTIO channel number select the channel within the destination.
+
 Real-time and auxiliary packets
-+++++++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 DRTIO is a packet-based protocol that uses two types of packets:
 
@@ -101,9 +46,9 @@ DRTIO is a packet-based protocol that uses two types of packets:
 * auxiliary packets, which are lower-bandwidth and are used for ancillary tasks such as housekeeping and monitoring/injection. Auxiliary packets are low-priority and their transmission has no impact on the timing of real-time packets (however, transmission of real-time packets slows down the transmission of auxiliary packets). In the ARTIQ DRTIO implementation, the contents of the auxiliary packets are read and written directly by the firmware, with the gateware simply handling the transmission of the raw data.
 
 Link layer
-++++++++++
+^^^^^^^^^^
 
-The lower layer of the DRTIO protocol stack is the link layer, which is responsible for delimiting real-time and auxiliary packets, and assisting with the establishment of a fixed-latency high speed serial transceiver link.
+The lower layer of the DRTIO protocol stack is the link layer, which is responsible for delimiting real-time and auxiliary packets and assisting with the establishment of a fixed-latency high speed serial transceiver link.
 
 DRTIO uses the IBM (Widmer and Franaszek) 8b/10b encoding. D characters (the encoded 8b symbols) always transmit real-time packet data, whereas K characters are used for idling and transmitting auxiliary packet data.
 
@@ -113,7 +58,8 @@ A real-time packet is defined by a series of D characters containing the packet'
 
 K characters, which are transmitted whenever there is no real-time data to transmit and to delimit real-time packets, are chosen using a 3-bit K selection word. If this K character is the first character in the set of N characters processed by the transceiver in the logic clock cycle, the mapping between the K selection word and the 8b/10b K space contains commas. If the K character is any of the subsequent characters processed by the transceiver, a different mapping is used that does not contain any commas. This scheme allows the receiver to align its logic clock with that of the transmitter, simply by shifting its logic clock so that commas are received into the first character position.
 
-.. note:: Due to the shoddy design of transceiver hardware, this simple process of clock and comma alignment is difficult to perform in practice. The paper "High-speed, fixed-latency serial links with Xilinx FPGAs" (by Xue LIU, Qing-xu DENG, Bo-ning HOU and Ze-ke WANG) discusses techniques that can be used. The ARTIQ implementation simply keeps resetting the receiver until the comma is aligned, since relatively long lock times are acceptable.
+.. note::
+    Due to the shoddy design of transceiver hardware, this simple process of clock and comma alignment is difficult to perform in practice. The paper "High-speed, fixed-latency serial links with Xilinx FPGAs" (by Xue LIU, Qing-xu DENG, Bo-ning HOU and Ze-ke WANG) discusses techniques that can be used. The ARTIQ implementation simply keeps resetting the receiver until the comma is aligned, since relatively long lock times are acceptable.
 
 The series of K selection words is then used to form auxiliary packets and the idle pattern. When there is no auxiliary packet to transfer or to delimitate auxiliary packets, the K selection word ``100`` is used. To transfer data from an auxiliary packet, the K selection word ``0ab`` is used, with ``ab`` containing two bits of data from the packet. An auxiliary packet is delimited by at least one ``100`` K selection word.
 
@@ -122,23 +68,23 @@ Both real-time traffic and K selection words are scrambled in order to make the
 Due to the use of K characters both as delimiters for real-time packets and as information carrier for auxiliary packets, auxiliary traffic is guaranteed a minimum bandwidth simply by having a maximum size limit on real-time packets.
 
 Clocking
-++++++++
+^^^^^^^^
 
 At the DRTIO satellite device, the recovered and aligned transceiver clock is used for clocking RTIO channels, after appropriate jitter filtering using devices such as the Si5324. The same clock is also used for clocking the DRTIO transmitter (loop timing), which simplifies clock domain transfers and allows for precise round-trip-time measurements to be done.
 
 RTIO clock synchronization
-++++++++++++++++++++++++++
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 As part of the DRTIO link initialization, a real-time packet is sent by the core device to each satellite device to make them load their respective timestamp counters with the timestamp values from their respective packets.
 
 RTIO outputs
-++++++++++++
+^^^^^^^^^^^^
 
 Controlling a remote RTIO output involves placing the RTIO event into the buffer of the destination. The core device maintains a cache of the buffer space available in each destination. If, according to the cache, there is space available, then a packet containing the event information (timestamp, address, channel, data) is sent immediately and the cached value is decremented by one. If, according to the cache, no space is available, then the core device sends a request for the space available in the destination and updates the cache. The process repeats until at least one remote buffer entry is available for the event, at which point a packet containing the event information is sent as before.
 
 Detecting underflow conditions is the responsibility of the core device; should an underflow occur then no DRTIO packet is transmitted. Sequence errors are handled similarly.
 
 RTIO inputs
-+++++++++++
+^^^^^^^^^^^
 
 The core device sends a request to the satellite for reading data from one of its channels. The request contains a timeout, which is the RTIO timestamp to wait for until an input event appears. The satellite then replies with either an input event (containing timestamp and data), a timeout, or an overflow error.

doc/manual/environment.rst

@@ -1,53 +1,102 @@
-The environment
+Environment
-===============
+===========
 
-Experiments interact with an environment that consists of devices, arguments and datasets. Access to the environment is handled by the class :class:`artiq.language.environment.EnvExperiment` that experiments should derive from.
+ARTIQ experiments exist in an environment, which consists of devices, arguments, and datasets. Access to the environment is handled through the :class:`~artiq.language.environment.HasEnvironment` manager provided by the :class:`~artiq.language.environment.EnvExperiment` class that experiments should derive from.
 
 .. _device-db:
 
 The device database
 -------------------
 
-The device database contains information about the devices available in a ARTIQ installation, what drivers to use, what controllers to use and on what machine, and where the devices are connected.
+Information about available devices is provided to ARTIQ through a file called the device database, typically called ``device_db.py``, which many of the ARTIQ front-end tools require access to in order to run. The device database specifies:
 
-The master (or :mod:`~artiq.frontend.artiq_run`) instantiates the device drivers (and the RPC clients in the case of controllers) for the experiments based on the contents of the device database.
+    * what devices are available to an ARTIQ installation
+    * what drivers to use
+    * what controllers to use
+    * how and where to contact each device, i.e.
 
-The device database is stored in the memory of the master and is generated by a Python script typically called ``device_db.py``. That script must define a global variable ``device_db`` with the contents of the database. The device database is a Python dictionary whose keys are the device names, and values can have several types.
+      - at which RTIO channel(s) each local device can be reached
+      - at which network address each controller can be reached
+
+as well as, if present, how and where to contact the core device itself (e.g., its IP address, often by a field named ``core_addr``).
+
+This is stored in a Python dictionary whose keys are the device names, which the file must define as a global variable with the name ``device_db``. Examples for various system configurations can be found inside the subfolders of ``artiq/examples``. A typical device database entry looks like this: ::
+
+    "led": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 19}
+    },
+
+Note that the key (the name of the device) is ``led`` and the value is itself a Python dictionary. Names will later be used to gain access to a device through methods such as ``self.setattr_device("led")``. While in this case ``led`` can be replaced with another name, provided it is used consistently, some names (in particular, ``core``) are used internally by ARTIQ and will cause problems if changed. It is often more convenient to use aliases for renaming purposes, see below.
+
+.. note::
+    The device database is generated and stored in the memory of the master when the master is first started. Changes to the ``device_db.py`` file will not immediately affect a running master. In order to update the device database, right-click in the Explorer window and select 'Scan device database', or run the command ``artiq_client scan-devices``.
+
+.. warning::
+    It is important to understand that the device database does not *set* your system configuration, only *describe* it. If you change the devices available to your system, it is usually necessary to edit the device database, but editing the database will not change what devices are available to your system.
+
+    Remote (normally, non-realtime) devices must have accessible, suitable controllers and drivers; see :doc:`developing_a_ndsp` for more information, including how to add entries for new remote devices to your device database. Local devices (normally, real-time, e.g. your Sinara hardware) must be connected to your system, and more importantly, your gateware and firmware must have been compiled to account for them, and to expect them at those ports.
+
+    While controllers can be added and removed to your device database on an *ad hoc* basis, in order to make new real-time hardware accessible, it is generally also necessary to recompile and reflash your gateware and firmware. (If you purchase your hardware from M-Labs, you will be provided with new binaries and necessary assistance.) See :doc:`building_developing`.
+
+    Adding or removing new real-time hardware is a difference in *system configuration,* which must be specified at compilation time of gateware and firmware. For Kasli and Kasli-SoC, this is managed in the form of a JSON usually called the :ref:`system description file<system-description>`. The device database generally provides that information to ARTIQ which can change from instance to instance ARTIQ is run, e.g., device names and aliases, network addresses, clock frequencies, and so on. The system configuration defines that information which is *not* permitted to change, e.g., what device is associated with which EEM port or RTIO channels. Insofar as data is duplicated between the two, the device database is obliged to agree with the system description, not the other way around.
+
+If you obtain your hardware from M-Labs, you will always be provided with a ``device_db.py`` to match your system configuration, which you can edit as necessary to add controllers, aliases, and so on. In the relatively unlikely case that you are writing a device database from scratch, the :mod:`~artiq.frontend.artiq_ddb_template` utility can be used to generate a template device database directly from the JSON system description used to compile your gateware and firmware. This is the easiest way to ensure that details such as the allocation of RTIO channel numbers will be represented in the device database correctly. See also the corresponding entry in :ref:`Utilities <ddb-template-tool>`.
 
 Local devices
-+++++++++++++
+^^^^^^^^^^^^^
 
-Local device entries are dictionaries that contain a ``type`` field set to ``local``. They correspond to device drivers that are created locally on the master (as opposed to going through the controller mechanism). The fields ``module`` and ``class`` determine the location of the Python class that the driver consists of. The ``arguments`` field is another (possibly empty) dictionary that contains arguments to pass to the device driver constructor.
+Local device entries are dictionaries which contain a ``type`` field set to ``local``. They correspond to device drivers that are created locally on the master as opposed to using the controller mechanism; this is normally the real-time hardware of the system, including the core, which is itself considered a local device. The ``led`` example above is a local device entry.
+
+The fields ``module`` and ``class`` determine the location of the Python class of the driver. The ``arguments`` field is another (possibly empty) dictionary that contains arguments to pass to the device driver constructor. ``arguments`` is often used to specify the RTIO channel number of a peripheral, which must match the channel number in gateware.
+
+On Kasli and Kasli-SoC, the allocation of RTIO channels to EEM ports is done automatically when the gateware is compiled, and while conceptually simple (channels are assigned one after the other, from zero upwards, for each device entry in the system description file) it is not entirely straightforward (different devices require different numbers of RTIO channels). Again, the easiest way to handle this when writing a new device database is automatically, using :mod:`~artiq.frontend.artiq_ddb_template`.
+
+.. _environment-ctlrs:
 
 Controllers
-+++++++++++
+^^^^^^^^^^^
 
-Controller entries are dictionaries whose ``type`` field is set to ``controller``. When an experiment requests such a device, a RPC client (see ``sipyco.pc_rpc``) is created and connected to the appropriate controller. Controller entries are also used by controller managers to determine what controllers to run.
+Controller entries are dictionaries which contain a ``type`` field set to ``controller``. When an experiment requests such a device, a RPC client (see ``sipyco.pc_rpc``) is created and connected to the appropriate controller. Controller entries are also used by controller managers to determine what controllers to run. For an example, see :ref:`the NDSP development page <ndsp-integration>`.
 
-The ``best_effort`` field is a boolean that determines whether to use ``sipyco.pc_rpc.Client`` or ``sipyco.pc_rpc.BestEffortClient``. The ``host`` and ``port`` fields configure the TCP connection. The ``target`` field contains the name of the RPC target to use (you may use ``sipyco_rpctool`` on a controller to list its targets). Controller managers run the ``command`` field in a shell to launch the controller, after replacing ``{port}`` and ``{bind}`` by respectively the TCP port the controller should listen to (matches the ``port`` field) and an appropriate bind address for the controller's listening socket.
+The ``host`` and ``port`` fields configure the TCP connection. The ``target`` field contains the name of the RPC target to use (you may use ``sipyco_rpctool`` on a controller to list its targets). Controller managers run the ``command`` field in a shell to launch the controller, after replacing ``{port}`` and ``{bind}`` by respectively the TCP port the controller should listen to (matching the ``port`` field) and an appropriate bind address for the controller's listening socket.
+
+An optional ``best_effort`` boolean field determines whether to use ``sipyco.pc_rpc.Client`` or ``sipyco.pc_rpc.BestEffortClient``. ``BestEffortClient`` is very similar to ``Client``, but suppresses network errors and automatically retries connections in the background. If no ``best_effort`` field is present, ``Client`` is used by default.
 
 Aliases
-+++++++
+^^^^^^^
 
 If an entry is a string, that string is used as a key for another lookup in the device database.
 
 Arguments
 ---------
 
-Arguments are values that parameterize the behavior of an experiment and are set before the experiment is executed.
+Arguments are values that parameterize the behavior of an experiment. ARTIQ supports both interactive arguments, requested and supplied at some point while an experiment is running, and submission-time arguments, requested in the build phase and set before the experiment is executed. For more on arguments in practice, see the tutorial section :ref:`mgmt-arguments`. For supported argument types, see the reference for :mod:`artiq.language.environment`; for specific methods, see the reference for :class:`~artiq.language.environment.HasEnvironment`.
-
-Requesting the values of arguments can only be done in the build phase of an experiment. The value requests are also used to define the GUI widgets shown in the explorer when the experiment is selected.
 
+.. _environment-datasets:
 
 Datasets
 --------
 
-Datasets are values (possibly arrays) that are read and written by experiments and live in a key-value store.
+Datasets are values that are read and written by experiments kept in a key-value store. They exist to facilitate the exchange and preservation of information between experiments, from experiments to the management system, and from experiments to long-term storage. Datasets may be either scalars (``bool``, ``int``, ``float``, or NumPy scalar) or NumPy arrays. For basic use of datasets, see the :ref:`data interfaces tutorial <mgmt-datasets>`.
 
-A dataset may be broadcasted, that is, distributed to all clients connected to the master. For example, the ARTIQ GUI may plot it while the experiment is in progress to give rapid feedback to the user. Broadcasted datasets live in a global key-value store; experiments should use distinctive real-time result names in order to avoid conflicts. Broadcasted datasets may be used to communicate values across experiments; for example, a periodic calibration experiment may update a dataset read by payload experiments. Broadcasted datasets are replaced when a new dataset with the same key (name) is produced.
+A dataset may be broadcast (``broadcast=True``), that is, distributed to all clients connected to the master. This is useful e.g. for the ARTIQ dashboard to plot results while an experiment is in progress and give rapid feedback to the user. Broadcasted datasets live in a global key-value store owned by the master. Care should be taken that experiments use distinctive real-time result names in order to avoid conflicts. Broadcasted datasets may be used to communicate values across experiments; for instance, a periodic calibration experiment might update a dataset read by payload experiments.
 
-Datasets can attach metadata for numerical formatting with the ``unit``, ``scale`` and ``precision`` parameters. In experiment code, values are assumed to be in the SI unit. In code, setting a dataset with a value of `1000` and the unit `kV` represents the quantity `1 kV`. It is recommended to use the globals defined by `artiq.language.units` and write `1*kV` instead of `1000` for the value. In dashboards and clients these globals are not available. There, setting a dataset with a value of `1` and the unit `kV` simply represents the quantity `1 kV`. ``precision`` refers to the max number of decimal places to display. This parameter does not affect the underlying value, and is only used for display purposes.
+Broadcasted datasets are replaced when a new dataset with the same key (name) is produced. By default, they are erased when the master halts. Broadcasted datasets may be made persistent (``persistent=True``, which also implies ``broadcast=True``), in which case the master stores them in a LMDB database typically called ``dataset_db.mdb``, where they are saved across master restarts.
 
-Broadcasted datasets may be persistent: the master stores them in a file typically called ``dataset_db.pyon`` so they are saved across master restarts.
+By default, datasets are archived in the ``results`` HDF5 output for that run, although this can be opted against (``archive=False``). They can be viewed and analyzed with the ARTIQ browser, or with an HDF5 viewer of your choice.
+
+Datasets and units
+^^^^^^^^^^^^^^^^^^
+
+Datasets accept metadata for numerical formatting with the ``unit``, ``scale`` and ``precision`` parameters of ``set_dataset``.
+
+.. note::
+    In experiment code, values are assumed to be in the SI base unit. Setting a dataset with a value of ``1000`` and the unit ``kV`` represents the quantity ``1 kV``. It is recommended to use the globals defined by :mod:`artiq.language.units` and write ``1*kV`` instead of ``1000`` for the value.
+
+    In dashboards and clients these globals are not available. However, setting a dataset with a value of ``1`` and the unit ``kV`` simply represents the quantity ``1 kV``.
+
+    ``precision`` refers to the max number of decimal places to display. This parameter does not affect the underlying value, and is only used for display purposes.
 
-Datasets produced by an experiment run may be archived in the HDF5 output for that run.

doc/manual/faq.rst

@@ -1,32 +1,234 @@
-.. Copyright (C) 2014, 2015 Robert Jordens <jordens@gmail.com>
+FAQ (How do I...)
+=================
 
-FAQ
+use this documentation?
-###
+-----------------------
 
-How do I ...
+The content of this manual is arranged in rough reading order. If you start at the beginning and make your way through section by section, you should form a pretty good idea of how ARTIQ works and how to use it. Otherwise:
-============
+
+**If you are just starting out,** and would like to get ARTIQ set up on your computer and your core device, start with :doc:`installing`, :doc:`flashing`, and :doc:`configuring`, in that order.
+
+**If you have a working ARTIQ setup** (or someone else has set it up for you), start with the tutorials: read :doc:`rtio`, then progress to :doc:`getting_started_core`, :doc:`getting_started_mgmt`, and :doc:`using_data_interfaces`. If your system is in a DRTIO configuration, :doc:`DRTIO and subkernels <using_drtio_subkernels>` will also be helpful.
+
+Pages like :doc:`management_system` and :doc:`core_device` describe **specific components of the ARTIQ ecosystem** in more detail. If you want to understand more about device and dataset databases, for example, read the :doc:`environment` page; if you want to understand the ARTIQ Python dialect and everything it does or does not support, read the :doc:`compiler` page.
+
+Reference pages, like :doc:`main_frontend_tools` and :doc:`core_drivers_reference`, contain the detailed documentation of the individual methods and command-line tools ARTIQ provides. They are heavily interlinked throughout the rest of the documentation: whenever a method, tool, or exception is mentioned by name, like :class:`~artiq.frontend.artiq_run`, :meth:`~artiq.language.core.now_mu`, or :exc:`~artiq.coredevice.exceptions.RTIOUnderflow`, it can normally be clicked on to directly access the reference material. Notice also that the online version of this manual is searchable; see the 'Search docs' bar at left.
+
+.. _build-documentation:
+
+build this documentation?
+-------------------------
+
+To generate this manual from source, you can use ``nix build`` directives, for example: ::
+
+    $ nix build git+https://github.com/m-labs/artiq.git\?ref=release-[number]#artiq-manual-html
+
+Substitute ``artiq-manual-pdf`` to get the LaTeX PDF version. The results will be in ``result``.
+
+The manual is written in `reStructured Text <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_; you can find the source files in the ARTIQ repository under ``doc/manual``. If you spot a mistake, a typo, or something that's out of date or missing -- in particular, if you want to add something to this FAQ -- feel free to clone the repository, edit the source RST files, and make a pull request with your version of an improvement. (If you're not a fan of or not familiar with command-line Git, both GitHub and Gitea support making edits and pull requests directly in the web interface; tutorial materials are easy to find online.) The second best thing is to open an issue to make M-Labs aware of the problem.
+
+roll back to older versions of ARTIQ, or obtain it through other installation methods?
+--------------------------------------------------------------------------------------
+
+At all times, three versions of ARTIQ are actively supported by M-Labs, released through the beta, stable, and legacy channels. See :doc:`releases`.
+
+If you are trying to rollback to stable or legacy, the process should be accordingly simple. See the respective :doc:`installing` page in the respective version of the manual. If you've previously used the version you are rolling back to, you can likely use the rollback methods described in :ref:`installing-upgrading`; otherwise you can always treat it as a fresh install. Remember that it will also be necessary to reflash core devices with corresponding legacy binaries.
+
+Regarding pre-legacy releases, note that being actively supported simply means that M-Labs makes prebuilt packages and binaries for these versions available via the supported installation methods and through AFWS. Outdated versions aren't automatically built or offered over these channels, but their source code remains available in the Git repository, and you are free to use it or adapt it in accordance with the terms of the license, including building whatever packages you prefer. In general, though, newer releases of ARTIQ offer more features, more stability, better performance, and better support. The legacy release is supported simply as a convenience for users who haven't been able to upgrade yet. For normal purposes, it is recommended to use the current stable release of ARTIQ if at all possible, or the beta to gain access to new features and improvements that are still in development.
+
+For more details, see also `Clarifications regarding the ARTIQ release model and AFWS <https://forum.m-labs.hk/d/823-clarifications-regarding-the-artiq-release-model-and-afws>`_.
+
+.. tip::
+
+    If you're particularly concerned with being able to precisely reproduce older experiments, even when you've moved on to newer ARTIQ versions, upgrade carefully and make your own local backups to be able to rollback to older versions of your system. Make sure to keep copies of older firmware binaries in order to be able to reflash your hardware. Older versions of ARTIQ will always continue working if left untouched, and you won't need to worry about rebuilding from the source if you keep your own prebuilt versions around.
+
+.. _faq-networking:
+
+troubleshoot networking problems?
+---------------------------------
+
+Diagnosis aids:
+
+    - Can you ``ping`` the device?
+    - Is the Ethernet LED on?
+    - Is the ERROR LED on?
+    - Is there anything unusual recorded in :ref:`the UART log <connecting-UART>`?
+
+Some things to consider:
+
+    - Is the ``core_addr`` field of your ``device_db.py`` set correctly?
+    - Did your device flash and boot successfully? Were the binaries generated for the correct board hardware version?
+    - Are your core device's IP address and networking configurations definitely set correctly? Check the UART log to confirm, and talk to your network administrator about what the correct choices are.
+    - Is your core device configured for an external reference clock? If so, it cannot function correctly without one. Is the external reference clock plugged in?
+    - Are Ethernet and (on Kasli only) SFP0 plugged in all the way? Are they working? Try different cables and SFP adapters; M-Labs tests with CAT6 cables, but lower categories should be supported too.
+    - Are your PC and your crate in the same subnet?
+    - Is some other device in your network already using the configured IP address? Turn off the core device and try pinging the configured IP address; if it responds, you have a culprit. One of the two will need a different networking configuration.
+    - Are there restrictions or issues in your router or subnet that are preventing the core device from connecting? It may help to try connecting the core device to your PC directly.
+
+fix 'no startup kernel found' / 'no idle kernel found' in the core log?
+-----------------------------------------------------------------------
+
+Don't. Note that these are ``INFO`` messages, and not ``ERROR`` or even ``WARN``. If you haven't flashed an idle or startup kernel yet, this is normal, and will not cause any problems; between experiments the core device will simply do nothing. The same applies to most other messages in the style of 'no configuration found' or 'falling back to default'. Your system will generally run just fine on its defaults until you get around to setting these configurations, though certain features may be limited until properly set up. See :doc:`configuring` and the list of keys in :ref:`core-device-flash-storage`.
+
+fix 'Mismatch between gateware and software versions'?
+------------------------------------------------------
+
+Either reflash your core device with a newer version of ARTIQ (see :doc:`flashing`) or update your software (see :ref:`installing-upgrading`), depending on which is out of date.
+
+.. note::
+    You can check the specific versions you are using at any time by comparing the gateware version given in the core startup log and the output given by adding ``--version`` to any of the standard ARTIQ front-end commands. This is especially useful when e.g. seeking help in the forum or at the helpdesk, where your running ARTIQ version is often crucial information to diagnose a problem.
+
+    Minor version mismatches are common, even in stable ARTIQ versions, but should not cause any issues. The ARTIQ release system ensures breaking changes are strictly limited to new release versions, or to the beta branch (which explicitly makes no promises of stability.) Updates that *are* applied to the stable version are usually bug fixes, documentation improvements, or other quality-of-life changes. As long as gateware and software are using the same stable release version of ARTIQ, even if there is a minor mismatch, no warning will be displayed.
+
+change configuration settings of satellite devices?
+---------------------------------------------------
+
+Currently, it is not possible to reach satellites through ``artiq_coremgmt config``, although this is being worked on. On Kasli, use :class:`~artiq.frontend.artiq_mkfs` and :class:`~artiq.frontend.artiq_flash`; on Kasli-SoC, preload the SD card with a ``config.txt``, formatted as a list of ``key=value`` pairs, one per line.
+
+Don't worry about individually flashing idle or startup kernels. If your idle or startup kernel contains subkernels, it will automatically compile as a ``.tar``, which you only need to flash to the master.
+
+fix unreliable DRTIO master-satellite links?
+--------------------------------------------
+
+Inconsistent DRTIO connections, especially with odd or absent errors in the core logs, are often a symptom of overheating either in the master or satellite boards. Check the core device fans for failure or defects. Improve air circulation around the crate or attach additional fans to see if that improves or resolves the issue. In the long term, fan trays to be rack-mounted together with the crate are a clean solution to these kinds of problems.
+
+add or remove EEM peripherals or DRTIO satellites?
+--------------------------------------------------
+
+Adding new real-time hardware to an ARTIQ system almost always means reflashing the core device; if you are adding new satellite core devices, they will have to be flashed as well. If you have obtained your upgrades from M-Labs or QUARTIQ, updated binaries and reflashing support will normally be offered to you directly. In any other case, track down your JSON system description file(s), bring them up to date with the updated state of your system, and see :doc:`building_developing`.
+
+Once you have an updated set of binaries, reflash the core device, following the instructions in :doc:`flashing`. Be sure to update your device database before starting experimentation; run :mod:`~artiq.frontend.artiq_ddb_template` on your system description(s) to update the local devices, and copy over any aliases or entries for NDSP controllers you may have been using. Note that the device database is a Python file, and the generated file of local devices can also simply be imported into the final version, allowing for dynamic modifications, especially in complex systems that may have multiple device databases in use.
+
+see command-line help?
+----------------------
+
+Like most if not almost all terminal utilities, ARTIQ commands, tools and applets print their help messages directly into the terminal and exit when run with the flag ``--help`` or ``-h``: ::
+
+    $ artiq_run -h
+
+This is the simplest and most direct way of accessing the same usage and reference material that is replicated in this manual on the pages :doc:`main_frontend_tools` and :doc:`utilities`.
+
+.. _faq-find-examples:
 
 find ARTIQ examples?
 --------------------
 
-The examples are installed in the ``examples`` folder of the ARTIQ package. You can find where the ARTIQ package is installed on your machine with: ::
+The official examples are stored in the ``examples`` folder of the ARTIQ package. You can find the location of the ARTIQ package on your machine with: ::
 
   python3 -c "import artiq; print(artiq.__path__[0])"
 
-Copy the ``examples`` folder from that path into your home/user directory, and start experimenting!
+Copy the ``examples`` folder from that path into your home or user directory, and start experimenting! (Note that some examples have dependencies not included with a standard ARTIQ install, like matplotlib and numba. To run those examples properly, make sure those modules are accessible.)
 
-prevent my first RTIO command from causing an underflow?
+If you have progressed past this level and would like to see more in-depth code or real-life examples of how other groups have handled running experiments with ARTIQ, see the "Community code" directory on the M-labs `resources page <https://m-labs.hk/experiment-control/resources/>`_.
---------------------------------------------------------
 
-The first RTIO event is programmed with a small timestamp above the value of the timecounter when the core device is reset. If the kernel needs more time than this timestamp to produce the event, an underflow will occur. You can prevent it by calling ``break_realtime`` just before programming the first event, or by adding a sufficient delay.
+fix ``failed to connect to moninj`` in the dashboard?
+-----------------------------------------------------
 
-If you are not resetting the core device, the time cursor stays where the previous experiment left it.
+This and other similar messages almost always indicate that your device database lists controllers (for example, ``aqctl_moninj_proxy``) that either haven't been started or aren't reachable at the given host and port. See :ref:`mgmt-ctlmgr`, or simply run: ::
+
+    $ artiq_ctlmgr
+
+to let the controller manager start the necessary controllers automatically.
+
+fix ``address already in use`` when running ARTIQ commands?
+-----------------------------------------------------------
+
+A message like ``OSError: [Errno 98] error while attempting to bind on address ('127.0.0.1', 1067): [errno 98] address already in use`` indicates that the IP address and port number combination you're trying to use is already occupied by some other process. Often this simply means that the ARTIQ process you're trying to start is in fact already running. Note for example that trying to start a controller which is already being run by a controller manager will generally fail for this reason.
+
+.. note::
+    ARTIQ management system communications, whether distributed or local, run over TCP/IP, using TCP port numbers to identify their destinations. Generally speaking, client processes like the dashboard don't require fixed ports of their own, since they can simply reach out to the master when they want to establish a connection. Running multiple dashboards will never cause a port conflict. On the other hand, server processes like the ARTIQ master have to be 'listening' at a fixed, open port in order to be able to receive incoming connections. For more details, look into `ports in computer networking <https://en.wikipedia.org/wiki/Port_(computer_networking)>`_.
+
+    Most management system processes belong to the second category, and are bound to one or several fixed communication ports while they're running. See also :doc:`default_network_ports`.
+
+You can use the command ``netstat`` to list the ports currently in use on your system. To check the status of a specific port on Linux, try either of: ::
+
+    $ netstat -anp --inet | grep "<port-number>"
+    $ lsof -i:<port-number>
+
+On Windows, you can list ports with: ::
+
+    $ netstat -ano -p TCP
+
+Use your preferred method to search through the output; suitable commands will vary by environment (e.g. ``grep`` in an MSYS2 shell, ``Select-String`` in PowerShell, ``find`` in the Windows command line, etc.)
+
+In all cases, if there are no results, the port isn't in use and should be free for new processes.
+
+.. tip::
+    While it is possible to run, for example, two identical ARTIQ controllers on the same machine, they can't be bound to the same port numbers at the same time. If you're intentionally running multiple copies of the same ARTIQ processes, use the command-line ``--port`` options to set alternate ports for at least one of the two. See :doc:`main_frontend_tools` and :doc:`utilities` for exact flags to use. Controllers should have similar flags available and will also require updated :ref:`device database entries <ndsp-integration>`. Note that alternate ports must be consistent to be useful, e.g., a master and dashboard must have the same ``--port-notify`` set in order to communicate with each other!
+
+Otherwise, either the running process must be stopped, or you'll have to set different port numbers for the process you're trying to start. In some cases it might happen that a process is no longer accessible or has become unresponsive but is still occupying its ports. The easiest way to free the ports is to kill the process manually. On Linux, you can use the ``kill`` command with ``lsof``: ::
+
+    $ kill $(lsof -t -i:<port-number>)
+
+On Windows, use ``netstat`` again to identify the process ID, and then feed it into ``taskkill``, e.g.: ::
+
+    $ netstat -ano -p TCP
+    $ taskkill /F /PID <process-ID>
+
+diagnose and fix sequence errors?
+---------------------------------
+
+Go through your code, keeping manual track of SED lanes. See the following example: ::
+
+    @kernel
+    def run(self):
+        self.core.reset()
+        with parallel:
+            self.ttl0.on()  # lane0
+            self.ttl_sma.pulse(800*us)  # lane1(rising) lane1(falling)
+            with sequential:
+                self.ttl1.on()  # lane2
+                self.ttl2.on()  # lane3
+                self.ttl3.on()  # lane4
+                self.ttl4.on()  # lane5
+                delay(800*us)
+                self.ttl1.off() # lane5
+                self.ttl2.off() # lane6
+                self.ttl3.off() # lane7
+                self.ttl4.off() # lane0
+        self.ttl0.off()  # lane1 -> clashes with the falling edge of ttl_sma,
+                         # which is already at +800us
+
+In most cases, as in this one, it's relatively easy to rearrange the generation of events so that they will be better spread out across SED lanes without sacrificing actual functionality. One possible solution for the above sequence looks like: ::
+
+    @kernel
+    def run(self):
+        self.core.reset()
+        self.ttl0.on()     # lane0
+        self.ttl_sma.on()  # lane1
+        self.ttl1.on()     # lane2
+        self.ttl2.on()     # lane3
+        self.ttl3.on()     # lane4
+        self.ttl4.on()     # lane5
+        delay(800*us)
+        self.ttl1.off()    # lane5
+        self.ttl2.off()    # lane6
+        self.ttl3.off()    # lane7
+        self.ttl4.off()    # lane0  (no clash: new timestamp is higher than last)
+        self.ttl_sma.off() # lane1
+        self.ttl0.off()    # lane2
+
+In this case, the :meth:`~artiq.coredevice.ttl.TTLInOut.pulse` is split up into its component :meth:`~artiq.coredevice.ttl.TTLInOut.on` and  :meth:`~artiq.coredevice.ttl.TTLInOut.off` so that events can be generated more linearly. It can also be worth keeping in mind that delaying by even a single coarse RTIO cycle between events avoids switching SED lanes at all; in contexts where perfect simultaneity is not a priority, this is an easy way to avoid sequencing issues. See again :ref:`sequence-errors`.
+
+understand applet commands?
+---------------------------
+
+The 'Command' field contains the exact terminal command used to open and operate the applet. The default ``${artiq_applet}`` prefix simply translates to something to the effect of ``python -m artiq.applets.``, intended to be immediately followed by the applet module name. The options suffixed after the module name are the same used in the command line, and a list of them can be shown by using the standard command line ``-h`` help flag: ::
+
+    $ python -m artiq.applets.plot_xy -h
+
+in any terminal.
 
 organize datasets in folders?
 -----------------------------
 
 Use the dot (".") in dataset names to separate folders. The GUI will automatically create and delete folders in the dataset tree display.
 
+organize applets in groups?
+---------------------------
+
+Create groups by left-clicking within the applet list and selecting 'New Group'. Move applets in and out of groups by dragging them with the mouse. To unselect an applet or a group, use CTRL+click.
+
 organize experiment windows in the dashboard?
 ---------------------------------------------
 
@@ -37,74 +239,54 @@ Experiment windows can be organized by using the following hotkeys:
 
 The windows will be organized in the order they were last interacted with.
 
-write a generator feeding a kernel feeding an analyze function?
+fix errors when restarting management system after a crash?
----------------------------------------------------------------
+-----------------------------------------------------------
 
-  Like this::
+On Windows in particular, abnormal shutdowns such as power outages or bluescreens sometimes corrupt the organizational files used by the management system, resulting in errors to the tune of ``ValueError: source code string cannot contain null bytes`` when restarting. The easiest way to handle these problems is to delete the corrupted files and start from scratch. Note that GUI configuration ``.pyon`` files are kept in the user configuration directory, see below at :ref:`gui-config-files`
 
-    def run(self):
+create and use variable-length arrays in kernels?
-        self.parse(self.pipe(iter(range(10))))
+-------------------------------------------------
 
-    def pipe(self, gen):
+You can't, in general; see the corresponding notes under :ref:`compiler-types`. ARTIQ kernels do not support heap allocation, meaning in particular that lists, arrays, and strings must be of constant size. One option is to preallocate everything, as mentioned on the Compiler page; another option is to chunk it and e.g. read 100 events per function call, push them upstream and retry until the gate time closes.
-        for i in gen:
-            r = self.do(i)
-            yield r
 
-    def parse(self, gen):
+understand how best to send data between kernel and host?
-        for i in gen:
+---------------------------------------------------------
-            pass
 
-    @kernel
+See also :ref:`basic-artiq-python`. Let's run down the options for kernel-host data transfer:
-    def do(self, i):
-        return i
 
-create and use variable lengths arrays in kernels?
+    - Kernels can return single values directly. They *cannot* return lists, arrays or strings, because of the way these values are allocated, which prevents values of these types from outliving the kernel they are created in. This is still true when the values in question are wrapped in functions or objects, in which case they may be missed by lifetime tracking and accepted by the compiler, but will cause memory corruption when run.
---------------------------------------------------
 
-Don't. Preallocate everything. Or chunk it and e.g. read 100 events per
+    - Kernels can freely make changes to attributes of objects shared with the host, including ``self``. However, these changes will be made to a kernel-owned copy of the object, which is only synchronized with the host copy when the kernel completes. This means that host-side operations executed during the runtime of the kernel, including RPCs, will be handling an unmodified version of the object, and modifications made by those operations will simply be overwritten when the kernel returns.
-function call, push them upstream and retry until the gate time closes.
 
-execute multiple slow controller RPCs in parallel without losing time? 
+    .. note::
-----------------------------------------------------------------------
+        Attribute writeback happens *once per kernel*, that is, if your experiment contains many separate kernels called from the host, modifications will be written back when each separate kernel completes. This is generally not suitable for data transfer, however, as new kernels are costly to create, and experiments often try to avoid doing so. It is also important to specify that kernels called *from* a kernel will not write back to the host upon completion. Attribute writeback is only executed upon return to the host.
 
-Use ``threading.Thread``: portable, fast, simple for one-shot calls.
+    - Kernels can interact with datasets, either as attributes (if :meth:`~artiq.language.environment.HasEnvironment.setattr_dataset` is used) or by RPC of the get and set methods (:meth:`~artiq.language.environment.HasEnvironment.get_dataset`, :meth:`~artiq.language.environment.HasEnvironment.set_dataset`, etc.). In this case note that, like certain other host-side methods, :meth:`~artiq.language.environment.HasEnvironment.get_dataset` will not actually be accepted by the compiler, because its return type is not specified. To call it as an RPC, simply wrap it in another function which *does* specify a return type. :meth:`~artiq.language.environment.HasEnvironment.set_dataset` can be similarly wrapped to make it asynchronous.
+
+    - Kernels can of course also call arbitrary RPCs. When sending data to the host, these can be asynchronous, and this is normally the recommended way of transferring data back to the host, resulting in a relatively minor amount of delay in the kernel. Keep in mind however that asynchronous RPCs may still block execution for some time if the arguments are very large or if many RPCs are submitted in close succession. When receiving data from the host, RPCs must be synchronous, which is still considerably faster than starting a new kernel. Note that if data is being both (asynchronously) sent and received, there is a small possibility of minor race conditions (i.e. retrieved data may not yet show updates sent in an earlier RPC).
+
+Kernel attributes and data transfer remain somewhat of an open area of development. Many such developments are or will be implemented in `NAC3 <https://forum.m-labs.hk/d/392-nac3-new-artiq-compiler-3-prealpha-release>`_, the next-generation ARTIQ compiler. The overhead for starting new kernels, which is largely dominated by compile time, should be significantly reduced (NAC3 can be expected to complete compilations 6x - 30x faster than currently).
 
 write part of my experiment as a coroutine/asyncio task/generator?
 ------------------------------------------------------------------
 
-You can not change the API that your experiment exposes: ``build()``,
+You cannot change the API that your experiment exposes: :meth:`~artiq.language.environment.HasEnvironment.build`, :meth:`~artiq.language.environment.Experiment.prepare`, :meth:`~artiq.language.environment.Experiment.run` and :meth:`~artiq.language.environment.Experiment.analyze` need to be regular functions, not generators or asyncio coroutines. That would make reusing your own code in sub-experiments difficult and fragile. You can however wrap your own generators/coroutines/tasks in regular functions that you then expose as part of the API.
-``prepare()``, ``run()`` and ``analyze()`` need to be regular functions, not
-generators or asyncio coroutines. That would make reusing your own code in
-sub-experiments difficult and fragile. You can however wrap your own
-generators/coroutines/tasks in regular functions that you then expose as part
-of the API.
 
-determine the pyserial URL to attach to a device by its serial number?
+determine the pyserial URL to connect to a device by its serial number?
-----------------------------------------------------------------------
+-----------------------------------------------------------------------
 
-You can list your system's serial devices and print their vendor/product
+You can list your system's serial devices and print their vendor/product id and serial number by running::
-id and serial number by running::
 
     $ python3 -m serial.tools.list_ports -v
 
-It will give you the ``/dev/ttyUSBxx`` (or the ``COMxx`` for Windows) device
+This will give you the ``/dev/ttyUSBxx`` (or ``COMxx`` for Windows) device names. The ``hwid:`` field gives you the string you can pass via the ``hwgrep://`` feature of pyserial `serial_for_url() <https://pythonhosted.org/pyserial/pyserial_api.html#serial.serial_for_url>`_ in order to open a serial device.
-names.
-The ``hwid:`` field gives you the string you can pass via the ``hwgrep://``
-feature of pyserial
-`serial_for_url() <http://pyserial.sourceforge.net/pyserial_api.html#serial.serial_for_url>`_
-in order to open a serial device.
 
-The preferred way to specify a serial device is to make use of the ``hwgrep://``
+The preferred way to specify a serial device is to make use of the ``hwgrep://`` URL: it allows for selecting the serial device by its USB vendor ID, product
-URL: it allows to select the serial device by its USB vendor ID, product
+ID and/or serial number. These never change, unlike the device file name.
-ID and/or serial number. Those never change, unlike the device file name.
 
-For instance, if you want to specify the Vendor/Product ID and the USB Serial Number, you can do:
+For instance, if you want to specify the Vendor/Product ID and the USB Serial Number, you can do: ::
-
-``-d "hwgrep://<VID>:<PID> SNR=<serial_number>"``.
-for example:
-
-``-d "hwgrep://0403:faf0 SNR=83852734"``
 
+    $ -d "hwgrep://<VID>:<PID> SNR=<serial_number>"``.
 
 run unit tests?
 ---------------
@@ -124,9 +306,44 @@ The core device tests require the following TTL devices and connections:
 
 If TTL devices are missing, the corresponding tests are skipped.
 
-find the dashboard and browser configuration files are stored?
+.. _gui-config-files:
---------------------------------------------------------------
+
+find the dashboard and browser configuration files?
+---------------------------------------------------
 
 ::
 
   python -c "from artiq.tools import get_user_config_dir; print(get_user_config_dir())"
+
+Additional Resources
+====================
+
+Other related documentation
+---------------------------
+
+- the `Sinara wiki <https://github.com/sinara-hw/meta/wiki>`_
+- the `SiPyCo manual <https://m-labs.hk/artiq/sipyco-manual/>`_
+- the `Migen manual <https://m-labs.hk/migen/manual/>`_
+- in a pinch, the `M-labs internal docs <https://git.m-labs.hk/sinara-hw/assembly>`_
+
+For more advanced questions, sometimes the `list of publications <https://m-labs.hk/experiment-control/publications/>`_ about experiments performed using ARTIQ may be interesting. See also the official M-Labs `resources <https://m-labs.hk/experiment-control/resources/>`_ page, especially the section on community code.
+
+"Help, I've done my best and I can't get any further!"
+------------------------------------------------------
+
+- If you have an active M-Labs AFWS/support subscription, you can email helpdesk@ at any time for personalized assistance. Please include the following information:
+
+    - Your installed ARTIQ version (add ``--version`` to any of the standard ARTIQ commands)
+    - The variant name of your system (refer to the sticker on the crate if you aren't sure)
+    - The recent output of your core log, either through ``artiq_coremgmt`` (if you're able to contact your device by network), or over UART following :ref:`the guide here <connecting-UART>`
+    - How your problem happened, and what you've already tried to fix it
+
+- Compare your materials with the examples; see also :ref:`finding ARTIQ examples <faq-find-examples>` above.
+- Check the list of `active issues <https://github.com/m-labs/artiq/issues>`_ on the ARTIQ GitHub repository for possible known problems with ARTIQ. Search through the closed issues to see if your question or concern has been addressed before.
+- Search the `M-Labs forum <https://forum.m-labs.hk/>`_ for similar problems, or make a post asking for help yourself.
+- Look into the `Mattermost live chat <https://chat.m-labs.hk>`_ or the bridged IRC channel.
+- Read the open source code and its docstrings and figure it out.
+- If you're reasonably certain you've identified a bug, or if you'd like to suggest a feature that should be included in future ARTIQ releases, `file a GitHub issue <https://github.com/m-labs/artiq/issues/new/choose>`_ yourself, following one of the provided templates.
+- In some odd cases, you may want to see the `mailing list archive <https://www.mail-archive.com/artiq@lists.m-labs.hk/>`_; the ARTIQ mailing list was shut down at the end of 2020 and was last regularly used during the time of ARTIQ-2 and 3, but for some older ARTIQ features, or to understand a development thought process, you may still find relevant information there.
+
+In any situation, if you found the manual unclear or unhelpful, you might consider following the :ref:`directions for contribution <build-documentation>` and editing it to be more helpful for future readers.