doc: 'Core device' manual page overhaul (#2430)

doc/manual/core_device.rst

@@ -3,73 +3,51 @@ Core device
 
 The core device is a FPGA-based hardware component that contains a softcore or hardcore CPU tightly coupled with the so-called RTIO core, which runs in gateware and provides precision timing. The CPU executes Python code that is statically compiled by the ARTIQ compiler and communicates with peripherals (TTL, DDS, etc.) through the RTIO core, as described in :ref:`artiq-real-time-i-o-concepts`. This architecture provides high timing resolution, low latency, low jitter, high-level programming capabilities, and good integration with the rest of the Python experiment code. 
 
-While it is possible to use all the other parts of ARTIQ (controllers, master, GUI, dataset management, etc.) without a core device, many experiments require it.
+While it is possible to use the other parts of ARTIQ (controllers, master, GUI, dataset management, etc.) without a core device, many experiments require it.
 
 .. _core-device-flash-storage:
 
 Flash storage
-*************
+^^^^^^^^^^^^^
 
-The core device contains some flash storage space which is used to store configuration data. It is one sector (typically 64 kB) large and organized as a list of key-value records, accessible by using ``artiq_coremgmt`` (see: :ref:`core-device-management-tool`). 
-
-This area is used to store a variety of configurations, in particular the core device IP address and, if present, the startup and/or idle kernels (see also: :ref:`miscellaneous_config_core_device`).   
+The core device contains some flash storage space which is used to store configuration data. It is one sector (typically 64 kB) large and organized as a list of key-value records, accessible by using ``artiq_coremgmt`` (see :ref:`core-device-management-tool`). The core device IP and MAC addresses, as well as, if present, the startup and/or idle kernels (see :ref:`miscellaneous_config_core_device`) are stored here. 
 
 .. _board-ports:
 
 FPGA board ports
-****************
+^^^^^^^^^^^^^^^^
 
 All boards have a serial interface running at 115200bps 8-N-1 that can be used for debugging.
 
-Kasli
------
+Kasli and Kasli SoC
+^^^^^^^^^^^^^^^^^^^
 
-`Kasli <https://github.com/m-labs/sinara/wiki/Kasli>`_ is a versatile core device designed for ARTIQ as part of the `Sinara <https://github.com/sinara-hw/meta/wiki>`_ family of boards. All variants support interfacing to various EEM daughterboards (TTL, DDS, ADC, DAC...) connected directly to it.
+`Kasli <https://github.com/sinara-hw/Kasli/wiki>`_ and `Kasli-SoC <https://github.com/sinara-hw/Kasli-SOC/wiki>`_ are versatile core devices designed for ARTIQ as part of the open-source `Sinara <https://github.com/sinara-hw/meta/wiki>`_ family of boards. All support interfacing to various EEM daughterboards (TTL, DDS, ADC, DAC...) through twelve onboard EEM ports. Kasli-SoC, which runs on a separate `Zynq port <https://git.m-labs.hk/M-Labs/artiq-zynq>`_ of the ARTIQ firmware, is architecturally separate, among other things being capable of performing much heavier software computations quickly locally to the board, but provides generally similar features to Kasli. Kasli itself exists in two versions, of which the improved Kasli v2.0 is now in more common use, but the original Kasli v1.0 remains supported by ARTIQ. 
 
-Standalone variants
-+++++++++++++++++++
+Kasli can be connected to the network using a 10000Base-X SFP module, installed into the SFP0 cage. Kasli-SoC features a built-in Ethernet port to use instead. If configured as a DRTIO satellite, both boards instead reserve SFP0 for the upstream DRTIO connection; remaining SFP cages are available for downstream connections. Equally, if used as a DRTIO master, all free SFP cages are available for downstream connections (i.e. all but SFP0 on Kasli, all four on Kasli-SoC). 
 
-Kasli is connected to the network using a 1000Base-X SFP module. `No-name <https://www.fs.com>`_ BiDi (1000Base-BX) modules have been used successfully. The SFP module for the network should be installed into the SFP0 cage.
-The other SFP cages are not used.
-
-The RTIO clock frequency is 125MHz or 150MHz, which is generated by the Si5324.
-
-DRTIO master variants
-+++++++++++++++++++++
-
-Kasli can be used as a DRTIO master that provides local RTIO channels and can additionally control one DRTIO satellite.
-
-The RTIO clock frequency is 125MHz or 150MHz, which is generated by the Si5324. The DRTIO line rate is 2.5Gbps or 3Gbps.
-
-As with the standalone configuration, the SFP module for the Ethernet network should be installed into the SFP0 cage. The DRTIO connections are on SFP1 and SFP2, and optionally on the SATA connector.
-
-DRTIO satellite/repeater variants
-+++++++++++++++++++++++++++++++++
-
-Kasli can be used as a DRTIO satellite with a 125MHz or 150MHz RTIO clock and a 2.5Gbps or 3Gbps DRTIO line rate.
-
-The DRTIO upstream connection is on SFP0 or optionally on the SATA connector, and the remaining SFPs are downstream ports.
+The DRTIO line rate depends upon the RTIO clock frequency running, e.g., at 125MHz the line rate is 2.5Gbps, at 150MHz 3.0Gbps, etc. See below for information on RTIO clocks. 
 
 KC705
------
+^^^^^
 
 An alternative target board for the ARTIQ core device is the KC705 development board from Xilinx. It supports the NIST CLOCK and QC2 hardware (FMC).
 
 Common problems
-+++++++++++++++
+---------------
 
 * The SW13 switches on the board need to be set to 00001.
 * When connected, the CLOCK adapter breaks the JTAG chain due to TDI not being connected to TDO on the FMC mezzanine.
 * On some boards, the JTAG USB connector is not correctly soldered.
 
 VADJ
-++++
+----
 
 With the NIST CLOCK and QC2 adapters, for safe operation of the DDS buses (to prevent damage to the IO banks of the FPGA), the FMC VADJ rail of the KC705 should be changed to 3.3V. Plug the Texas Instruments USB-TO-GPIO PMBus adapter into the PMBus connector in the corner of the KC705 and use the Fusion Digital Power Designer software to configure (requires Windows). Write to chip number U55 (address 52), channel 4, which is the VADJ rail, to make it 3.3V instead of 2.5V.  Power cycle the KC705 board to check that the startup voltage on the VADJ rail is now 3.3V.
 
 
 NIST CLOCK
-++++++++++
+----------
 
 With the CLOCK hardware, the TTL lines are mapped as follows:
 
@@ -113,7 +91,7 @@ The DDS bus is on channel 27.
 
 
 NIST QC2
-++++++++
+--------
 
 With the QC2 hardware, the TTL lines are mapped as follows:
 
@@ -159,24 +137,28 @@ See :mod:`artiq.coredevice.i2c` for more details.
 .. _core-device-clocking:
 
 Clocking
-++++++++
+^^^^^^^^
 
-The KC705 in standalone variants supports an internal 125 MHz RTIO clock (based on its crystal oscillator, or external reference for PLL for DRTIO variants) and an external clock, that can be selected using the ``rtio_clock`` configuration entry. Valid values are:
+The core device generates the RTIO clock using a PLL locked either to an internal crystal or to an external frequency reference. If choosing the latter, external reference must be provided (via front panel SMA input on Kasli boards). Valid configuration options include: 
 
-    * ``int_125`` - internal crystal oscillator, 125 MHz output (default),
-    * ``ext0_bypass`` - external clock.
+  * ``int_100`` - internal crystal reference is used to synthesize a 100MHz RTIO clock,
+  * ``int_125`` - internal crystal reference is used to synthesize a 125MHz RTIO clock (default option), 
+  * ``int_150`` - internal crystal reference is used to synthesize a 150MHz RTIO clock.
+  * ``ext0_synth0_10to125`` - external 10MHz reference clock used to synthesize a 125MHz RTIO clock,
+  * ``ext0_synth0_80to125`` - external 80MHz reference clock used to synthesize a 125MHz RTIO clock,
+  * ``ext0_synth0_100to125`` - external 100MHz reference clock used to synthesize a 125MHz RTIO clock,
+  * ``ext0_synth0_125to125`` - external 125MHz reference clock used to synthesize a 125MHz RTIO clock. 
 
-KC705 in DRTIO variants and Kasli generate the RTIO clock using a PLL locked either to an internal crystal or to an external frequency reference. Valid values are:
+The selected option can be observed in the core device boot logs and accessed using ``artiq_coremgmt config`` with key ``rtio_clock``. 
 
-    * ``int_125`` - internal crystal oscillator using PLL, 125 MHz output (default),
-    * ``int_100`` - internal crystal oscillator using PLL, 100 MHz output,
-    * ``int_150`` - internal crystal oscillator using PLL, 150 MHz output,
-    * ``ext0_synth0_10to125`` - external 10 MHz reference using PLL, 125 MHz output,
-    * ``ext0_synth0_80to125`` - external 80 MHz reference using PLL, 125 MHz output,
-    * ``ext0_synth0_100to125`` - external 100 MHz reference using PLL, 125 MHz output,
-    * ``ext0_synth0_125to125`` - external 125 MHz reference using PLL, 125 MHz output,
-    * ``ext0_bypass``, ``ext0_bypass_125``, ``ext0_bypass_100`` - external clock - with explicit aliases available.
+As of ARTIQ 8, it is now possible for Kasli and Kasli-SoC configurations to enable WRPLL -- a clock recovery method using `DDMTD <http://white-rabbit.web.cern.ch/documents/DDMTD_for_Sub-ns_Synchronization.pdf>`_ and Si549 oscillators -- both to lock the main RTIO clock and (in DRTIO configurations) to lock satellites to master. This is set by the ``enable_wrpll`` option in the JSON description file. Because WRPLL requires slightly different gateware and firmware, it is necessary to re-flash devices to enable or disable it in extant systems. If you would like to obtain the firmware for a different WRPLL setting through ``awfs_client``, write to the helpdesk@ email. 
 
-The selected option can be observed in the core device boot logs.
+If phase noise performance is the priority, it is recommended to use ``ext0_synth0_125to125`` over other ``ext0`` options, as this bypasses the (noisy) MMCM.  
 
-Options ``rtio_clock=int_XXX`` and ``rtio_clock=ext0_synth0_XXXXX`` generate the RTIO clock using a PLL locked either to an internal crystal or to an external frequency reference (depending on exact option). ``rtio_clock=ext0_bypass`` bypasses that PLL and the user must supply the RTIO clock (typically 125 MHz) at the Kasli front panel SMA input. Bypassing the PLL ensures the skews between input clock, Kasli downstream clock outputs, and RTIO clock are deterministic accross reboots of the system. This is useful when phase determinism is required in situtations where the reference clock fans out to other devices before reaching Kasli.
+If not using WRPLL, PLL can also be bypassed entirely with the options 
+
+    * ``ext0_bypass`` (input clock used directly)
+    * ``ext0_bypass_125`` (explicit alias)
+    * ``ext0_bypass_100`` (explicit alias)
+
+Bypassing the PLL ensures the skews between input clock, downstream clock outputs, and RTIO clock are deterministic across reboots of the system. This is useful when phase determinism is required in situations where the reference clock fans out to other devices before reaching the master.

doc/manual/installing.rst

@@ -259,7 +259,7 @@ Run the command::
 
 Replace ``[username]`` with the login name that was given to you with the subscription, ``[variant]`` with the name of your system variant, and ``[afws_directory]`` with the name of an empty directory, which will be created by the command if it does not exist. Enter your password when prompted and wait for the build (if applicable) and download to finish. If you experience issues with the AFWS client, write to the helpdesk@ email.
 
-For certain configurations (KC705 or ZC705 only) it is also possible to source firmware from `the M-Labs Hydra server <https://nixbld.m-labs.hk/project/artiq>` (in ``main`` and ``zynq`` respectively).
+For certain configurations (KC705 or ZC705 only) it is also possible to source firmware from `the M-Labs Hydra server <https://nixbld.m-labs.hk/project/artiq>`_ (in ``main`` and ``zynq`` respectively).
 
 Without a subscription, you may build the firmware yourself from the open source code. See the section :ref:`Developing ARTIQ <developing-artiq>`.
 
@@ -333,7 +333,9 @@ If the Kasli-SoC won't boot due to nonexistent or corrupted firmware, extract th
 
   The SW13 switches need to be set to 00001.
 
-Flashing over network is also possible for Kasli and KC705, assuming IP networking has been set up. In this case, the ``-H HOSTNAME`` option is used; see the entry for ``artiq_flash`` in the :ref:`Utilities <flashing-loading-tool>` reference.  
+Flashing over network is also possible for Kasli and KC705, assuming IP networking has already been set up. In this case, the ``-H HOSTNAME`` option is used; see the entry for ``artiq_flash`` in the :ref:`Utilities <flashing-loading-tool>` reference.  
+
+.. _core-device-networking: 
 
 Setting up the core device IP networking
 ----------------------------------------
@@ -405,23 +407,12 @@ Select the RTIO clock source
 
 The core device may use any of: an external clock signal, its internal clock with external frequency reference, or its internal clock with internal crystal reference. Clock source and timing are set at power-up. To find out what clock signal you are using, check startup logs with ``artiq_coremgmt log``. 
 
-By default, an internal clock is used; to select another source, use a command of the form: 
+The default is to use an internal 125MHz clock. To select a source, use a command of the form: 
 
   $ artiq_coremgmt config write -s rtio_clock int_125  # internal 125MHz clock (default)
-  $ artiq_coremgmt config write -s rtio_clock ext0_bypass  # external clock (bypass)
+  $ artiq_coremgmt config write -s rtio_clock ext0_synth0_10to125  # external 10MHz reference used to synthesize internal 125MHz
 
-If set to ``ext0_bypass``, the Si5324 synthesizer is bypassed entirely in favor of an input clock, requiring that an input clock be present. 
-
-Other options include:
-  - ``ext0_synth0_10to125`` - external 10MHz reference clock used by Si5324 to synthesize a 125MHz RTIO clock,
-  - ``ext0_synth0_80to125`` - external 80MHz reference clock used by Si5324 to synthesize a 125MHz RTIO clock,
-  - ``ext0_synth0_100to125`` - external 100MHz reference clock used by Si5324 to synthesize a 125MHz RTIO clock,
-  - ``ext0_synth0_125to125`` - external 125MHz reference clock used by Si5324 to synthesize a 125MHz RTIO clock,
-  - ``int_100`` - internal crystal reference is used by Si5324 to synthesize a 100MHz RTIO clock,
-  - ``int_150`` - internal crystal reference is used by Si5324 to synthesize a 150MHz RTIO clock.
-  - ``ext0_bypass_125`` and ``ext0_bypass_100`` - explicit aliases for ``ext0_bypass``.
-
-Availability of these options depends on specific board and configuration - specific settings may or may not be supported. See also :ref:`core-device-clocking`. 
+See :ref:`core-device-clocking` for availability of specific options.    
 
 Set up resolving RTIO channels to their names
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

doc/manual/rtio.rst

@@ -121,6 +121,8 @@ To track down ``RTIOUnderflows`` in an experiment there are a few approaches:
     code.
   * The :any:`integrated logic analyzer <core-device-rtio-analyzer-tool>` shows the timeline context that lead to the exception. The analyzer is always active and supports plotting of RTIO slack. RTIO slack is the difference between timeline cursor and wall clock time (``now - rtio_counter``).
 
+.. _sequence_errors: 
+
 Sequence errors
 ---------------
 A sequence error happens when the sequence of coarse timestamps cannot be supported by the gateware. For example, there may have been too many timeline rewinds.