doc: Update + expansion of Environment manual page

doc/manual/developing_a_ndsp.rst

@@ -111,6 +111,8 @@ We see that the client has made a request to the server, which has, through the
 
 To access this driver in an experiment, we can retrieve the ``Client`` instance with the ``get_device`` and ``set_device`` methods of :class:`artiq.language.environment.HasEnvironment`, and then use it like any other device (provided the controller is running and accessible at the time). 
 
+.. _ndsp-integration:
+
 Integration with ARTIQ experiments
 ----------------------------------
 

doc/manual/environment.rst

@@ -1,53 +1,96 @@
 The environment
 ===============
 
-Experiments interact with an environment that consists of devices, arguments and datasets. Access to the environment is handled by the class :class:`artiq.language.environment.EnvExperiment` that experiments should derive from.
+ARTIQ experiments exist in an environment, which consists of devices, arguments, and datasets. Access to the environment is handled through the :class:`~artiq.language.environment.HasEnvironment` manager provided by the :class:`~artiq.language.environment.EnvExperiment` class that experiments should derive from. 
 
 .. _device-db:
 
 The device database
 -------------------
 
-The device database contains information about the devices available in a ARTIQ installation, what drivers to use, what controllers to use and on what machine, and where the devices are connected.
+Information about available devices is provided to ARTIQ through a file called the device database. It is generated by a Python script typically called ``device_db.py``, which many of the ARTIQ front-end tools require access to in order to run. The device database specifies: 
 
-The master (or :mod:`~artiq.frontend.artiq_run`) instantiates the device drivers (and the RPC clients in the case of controllers) for the experiments based on the contents of the device database.
+    * what devices are available to an ARTIQ installation 
+    * what drivers to use 
+    * what controllers to use
+    * how and where to contact each device, i.e. 
 
-The device database is stored in the memory of the master and is generated by a Python script typically called ``device_db.py``. That script must define a global variable ``device_db`` with the contents of the database. The device database is a Python dictionary whose keys are the device names, and values can have several types.
+      - at which RTIO channel each local device can be reached 
+      - at which network address each controller can be reached 
+
+as well as, if present, how and where to contact the core device itself (e.g., its IP address, often by a field named ``core_addr``). 
+
+This is stored in a Python dictionary whose keys are the device names, which the script must define as a global variable with the name ``device_db``. Examples for various system configurations can be found inside the subfolders of ``artiq/examples``.
+
+A typical device database entry looks like this: :: 
+
+    "led": {
+        "type": "local",
+        "module": "artiq.coredevice.ttl",
+        "class": "TTLOut",
+        "arguments": {"channel": 19}
+    },
+
+Note that the key (the name of the device) is ``led`` and the value is itself a Python dictionary. Names will later be used to gain access to a device through methods such as ``self.setattr_device("led")``. While in this case ``led`` can be replaced with another name, provided it is used consistently, some names (e.g. in particular ``core``) are used internally by ARTIQ and will cause problems if changed. It is often more convenient to use aliases for renaming purposes, see below. 
+
+.. note:: 
+    The device database is generated and stored in the memory of the master when the master is first started. Changes to the ``device_db.py`` file will not immediately affect a running master. In order to update the device database, right-click in the Explorer window and select 'Scan device database', or run the command ``artiq_client scan-devices``. 
+
+.. warning:: 
+    It is important to understand that the device database does not *set* your system configuration, only *describe* it. If you change the devices available to your system, it is usually necessary to edit the device database, but editing the database will not change what devices are available to your system. 
+
+    Remote (normally, non-realtime) devices must have accessible, suitable controllers and drivers; see :doc:`developing_a_ndsp` for more information, including how to add entries for new remote devices to your device database. Local devices (normally, realtime, e.g. your Sinara hardware) must be factually attached to your system, and more importantly, your gateware and firmware must have been compiled to account for them. 
+    
+    While controllers can be added and removed to your device database relatively easily, in order to make new real-time hardware accessible, it is often necessary to recompile and reflash your gateware and firmware. (If you purchase your hardware from M-Labs, you will generally be provided with new binaries and necessary assistance.) 
+
+    In other words, adding new real-time hardware is a difference in *system configuration,* which is specified by the JSON system description file used in compilation of gateware and firmware. The device database generally provides that information to ARTIQ which can change from instance to instance ARTIQ is run, e.g., device names and aliases, network addresses, clock frequencies, and so on. The system configuration file provides that information which is *not* permitted to change, e.g. what device is associated with what RTIO channel. Insofar as data is duplicated between the two, the device database is obliged to agree with the system description, not the other way around. 
 
 Local devices
-+++++++++++++
+^^^^^^^^^^^^^
 
-Local device entries are dictionaries that contain a ``type`` field set to ``local``. They correspond to device drivers that are created locally on the master (as opposed to going through the controller mechanism). The fields ``module`` and ``class`` determine the location of the Python class that the driver consists of. The ``arguments`` field is another (possibly empty) dictionary that contains arguments to pass to the device driver constructor.
+Local device entries are dictionaries which contain a ``type`` field set to ``local``. They correspond to device drivers that are created locally on the master as opposed to using the controller mechanism; this is normally the real-time hardware of the system, including the core, which is itself considered a local device. The ``led`` example above is a local device entry. 
+
+The fields ``module`` and ``class`` determine the location of the Python class of the driver. The ``arguments`` field is another (possibly empty) dictionary that contains arguments to pass to the device driver constructor. ``arguments`` is often used to specify the RTIO channel number of a peripheral, which must match the channel number in gateware (i.e., in the system description file). 
 
 Controllers
-+++++++++++
+^^^^^^^^^^^
 
-Controller entries are dictionaries whose ``type`` field is set to ``controller``. When an experiment requests such a device, a RPC client (see ``sipyco.pc_rpc``) is created and connected to the appropriate controller. Controller entries are also used by controller managers to determine what controllers to run.
+Controller entries are dictionaries which contain a ``type`` field set to ``controller``. When an experiment requests such a device, a RPC client (see ``sipyco.pc_rpc``) is created and connected to the appropriate controller. Controller entries are also used by controller managers to determine what controllers to run. For an example, see :ref:`ndsp-integration`.
 
-The ``best_effort`` field is a boolean that determines whether to use ``sipyco.pc_rpc.Client`` or ``sipyco.pc_rpc.BestEffortClient``. The ``host`` and ``port`` fields configure the TCP connection. The ``target`` field contains the name of the RPC target to use (you may use ``sipyco_rpctool`` on a controller to list its targets). Controller managers run the ``command`` field in a shell to launch the controller, after replacing ``{port}`` and ``{bind}`` by respectively the TCP port the controller should listen to (matches the ``port`` field) and an appropriate bind address for the controller's listening socket.
+The ``host`` and ``port`` fields configure the TCP connection. The ``target`` field contains the name of the RPC target to use (you may use ``sipyco_rpctool`` on a controller to list its targets). Controller managers run the ``command`` field in a shell to launch the controller, after replacing ``{port}`` and ``{bind}`` by respectively the TCP port the controller should listen to (matches the ``port`` field) and an appropriate bind address for the controller's listening socket.
+
+An optional ``best_effort`` boolean field determines whether to use ``sipyco.pc_rpc.Client`` or ``sipyco.pc_rpc.BestEffortClient``. ``BestEffortClient`` is very similar to ``Client``, but suppresses network errors and automatically retries connections in the background. If no ``best_effort`` field is present, ``Client`` is used by default. 
 
 Aliases
-+++++++
+^^^^^^^
 
-If an entry is a string, that string is used as a key for another lookup in the device database.
+If an entry is a string, that string is used as a key for another lookup in the device database.  
 
 Arguments
 ---------
 
-Arguments are values that parameterize the behavior of an experiment and are set before the experiment is executed.
+Arguments are values that parameterize the behavior of an experiment. ARTIQ supports both interactive arguments, requested and supplied at some point while an experiment is running, as well as submission-time arguments, requested in the build phase and set before the experiment is executed. For more on arguments in practice, see :ref:`mgmt-arguments`. For argument types and specific reference, see the relevant sections of :doc:`the core language reference <core_language_reference>`, as well as the example experiment ``examples/no_hardware/interactive.py``.    
-
-Requesting the values of arguments can only be done in the build phase of an experiment. The value requests are also used to define the GUI widgets shown in the explorer when the experiment is selected.
-
 
 Datasets
 --------
 
-Datasets are values (possibly arrays) that are read and written by experiments and live in a key-value store.
+Datasets are values that are read and written by experiments and principally live in a key-value store. They exist to facilitate exchange and storage of information between experiments, from experiments to the management system, and from experiments to storage. Datasets may be either scalars (``bool``, ``int``, ``float``, or NumPy scalar) or NumPy arrays. For basic use of datasets, see the :ref:`management system tutorial <getting-started-datasets>`. 
 
-A dataset may be broadcasted, that is, distributed to all clients connected to the master. For example, the ARTIQ GUI may plot it while the experiment is in progress to give rapid feedback to the user. Broadcasted datasets live in a global key-value store; experiments should use distinctive real-time result names in order to avoid conflicts. Broadcasted datasets may be used to communicate values across experiments; for example, a periodic calibration experiment may update a dataset read by payload experiments. Broadcasted datasets are replaced when a new dataset with the same key (name) is produced.
+A dataset may be broadcasted (``broadcast=True``), that is, distributed to all clients connected to the master. This is useful e.g. for the ARTIQ dashboard to plot results while an experiment is in progress and give rapid feedback to the user. Broadcasted datasets live in a global key-value store owned by the master. Care should be taken that experiments use distinctive real-time result names in order to avoid conflicts. Broadcasted datasets may be used to communicate values across experiments; for instance, a periodic calibration experiment may update a dataset read by payload experiments. 
 
-Datasets can attach metadata for numerical formatting with the ``unit``, ``scale`` and ``precision`` parameters. In experiment code, values are assumed to be in the SI unit. In code, setting a dataset with a value of `1000` and the unit `kV` represents the quantity `1 kV`. It is recommended to use the globals defined by `artiq.language.units` and write `1*kV` instead of `1000` for the value. In dashboards and clients these globals are not available. There, setting a dataset with a value of `1` and the unit `kV` simply represents the quantity `1 kV`. ``precision`` refers to the max number of decimal places to display. This parameter does not affect the underlying value, and is only used for display purposes.
+Broadcasted datasets are replaced when a new dataset with the same key (name) is produced. By default, they are erased when the master halts. Broadcasted datasets may be made persistent (``persistent=True``, which also implies ``broadcast=True``), in which case the master stores them in a LMDB database typically called ``dataset_db.mdb``, where they are saved across master restarts.   
 
-Broadcasted datasets may be persistent: the master stores them in a file typically called ``dataset_db.pyon`` so they are saved across master restarts.
+By default, datasets are archived in the HDF5 output for that run, although this can be opted against (``archive=False``). 
+
+Datasets and units 
+^^^^^^^^^^^^^^^^^^
+
+Datasets accept metadata for numerical formatting with the ``unit``, ``scale`` and ``precision`` parameters of ``set_dataset``. 
+
+.. warning:: 
+    In experiment code, values are assumed to be in the SI base unit. Setting a dataset with a value of ``1000`` and the unit ``kV`` represents the quantity ``1 kV``. It is recommended to use the globals defined by :mod:`artiq.language.units` and write ``1*kV`` instead of ``1000`` for the value. 
+    
+    In dashboards and clients these globals are not available. However, setting a dataset with a value of ``1`` and the unit ``kV`` simply represents the quantity ``1 kV``. 
+    
+    ``precision`` refers to the max number of decimal places to display. This parameter does not affect the underlying value, and is only used for display purposes.
 
-Datasets produced by an experiment run may be archived in the HDF5 output for that run.