doc: new dataset API

artiq/language/environment.py

@@ -207,6 +207,14 @@ class HasEnvironment:
 
     def set_dataset(self, key, value,
                     broadcast=False, persist=False, save=True):
+        """Sets the contents and handling modes of a dataset.
+
+        :param broadcast: the data is sent in real-time to the master, which
+            dispatches it. Returns a Notifier that can be used to mutate the dataset.
+        :param persist: the master should store the data on-disk. Implies broadcast.
+        :param save: the data is saved into the local storage of the current
+            run (archived as a HDF5 file).
+        """
         if self.__parent is not None:
             self.__parent.set_dataset(key, value, broadcast, persist, save)
             return
@@ -215,6 +223,15 @@ class HasEnvironment:
         return self.__dataset_mgr.set(key, value, broadcast, persist, save)
 
     def get_dataset(self, key, default=NoDefault):
+        """Returns the contents of a dataset.
+
+        The local storage is searched first, followed by the master storage
+        (which contains the broadcasted datasets from all experiments) if the
+        key was not found initially.
+
+        If the dataset does not exist, returns the default value. If no default
+        is provided, raises ``KeyError``.
+        """
         if self.__parent is not None:
             return self.__parent.get_dataset(key, default)
         if self.__dataset_mgr is None:
@@ -228,6 +245,8 @@ class HasEnvironment:
                 return default
 
     def setattr_dataset(self, key, default=NoDefault):
+        """Sets the contents of a dataset as attribute. The names of the
+        dataset and of the attribute are the same."""
         setattr(self, key, self.get_dataset(key, default))
 
 

doc/manual/environment.rst

@@ -3,7 +3,7 @@ The environment
 
 Experiments interact with an environment that consists of devices, parameters, arguments and results. Access to the environment is handled by the class :class:`artiq.language.environment.EnvExperiment` that experiments should derive from.
 
-.. _ddb:
+.. _device-db:
 
 The device database
 -------------------
@@ -12,7 +12,7 @@ The device database contains information about the devices available in a ARTIQ
 
 The master (or ``artiq_run``) instantiates the device drivers (and the RPC clients in the case of controllers) for the experiments based on the contents of the device database.
 
-The device database is stored in the memory of the master and is backed by a PYON file typically called ``ddb.pyon``.
+The device database is stored in the memory of the master and is backed by a PYON file typically called ``device_db.pyon``.
 
 The device database is a Python dictionary whose keys are the device names, and values can have several types.
 
@@ -33,11 +33,6 @@ Aliases
 
 If an entry is a string, that string is used as a key for another lookup in the device database.
 
-The parameter database
-----------------------
-
-The parameter database is a key-value store that is global to all experiments. It is stored in the memory of the master and is backed by a PYON file typically called ``pdb.pyon``. It may be used to communicate values across experiments; for example, a periodic calibration experiment may update a parameter read by payload experiments.
-
 Arguments
 ---------
 
@@ -45,7 +40,14 @@ Arguments are values that parameterize the behavior of an experiment and are set
 
 Requesting the values of arguments can only be done in the build phase of an experiment. The value requests are also used to define the GUI widgets shown in the explorer when the experiment is selected.
 
-Results
--------
 
-Results are the output of an experiment. They are archived after in the HDF5 format after the experiment is run. Experiments may define real-time results that are (additionally) distributed to all clients connected to the master; for example, the ARTIQ GUI may plot them while the experiment is in progress to give rapid feedback to the user. Real-time results are a global key-value store (similar to the parameter database); experiments should use distinctive real-time result names in order to avoid conflicts.
+Datasets
+--------
+
+Datasets are values (possibly arrays) that are read and written by experiments and live in a key-value store.
+
+A dataset may be broadcasted, that is, distributed to all clients connected to the master. For example, the ARTIQ GUI may plot it while the experiment is in progress to give rapid feedback to the user. Broadcasted datasets live in a global key-value store; experiments should use distinctive real-time result names in order to avoid conflicts. Broadcasted datasets may be used to communicate values across experiments; for example, a periodic calibration experiment may update a dataset read by payload experiments. Broadcasted datasets are replaced when a new dataset with the same key (name) is produced.
+
+Broadcasted datasets may be persistent: the master stores them in a file typically called ``dataset_db.pyon`` so they are saved across master restarts.
+
+Datasets produced by an experiment run may be archived in the HDF5 output for that run.

doc/manual/getting_started_core.rst

@@ -22,7 +22,7 @@ As a very first step, we will turn on a LED on the core device. Create a file ``
 
 The central part of our code is our ``LED`` class, that derives from :class:`artiq.language.environment.EnvExperiment`. Among other features, ``EnvExperiment`` calls our ``build`` method and provides the ``setattr_device`` method that interfaces to the device database to create the appropriate device drivers and make those drivers accessible as ``self.core`` and ``self.led``. The ``@kernel`` decorator tells the system that the ``run`` method must be executed on the core device (instead of the host). The decorator uses ``self.core`` internally, which is why we request the core device using ``setattr_device`` like any other.
 
-Copy the files ``ddb.pyon`` and ``pdb.pyon`` (containing the device and parameter databases) from the ``examples/master`` folder of ARTIQ into the same directory as ``led.py`` (alternatively, you can use the ``-d`` and ``-p`` options of ``artiq_run``). You can open the database files using a text editor - their contents are in a human-readable format. You will probably want to set the IP address of the core device in ``ddb.pyon`` so that the computer can connect to it (it is the ``host`` parameter of the ``comm`` entry). See :ref:`ddb` for more information. The example device database is designed for the NIST QC1 hardware on the KC705; see :ref:`board-ports` for RTIO channel assignments if you need to adapt the device database to a different hardware platform.
+Copy the files ``device_db.pyon`` and ``dataset_db.pyon`` (containing the device and dataset databases) from the ``examples/master`` folder of ARTIQ into the same directory as ``led.py`` (alternatively, you can use the ``--device-db`` and ``--dataset-db`` options of ``artiq_run``). You can open the database files using a text editor - their contents are in a human-readable format. You will probably want to set the IP address of the core device in ``device_db.pyon`` so that the computer can connect to it (it is the ``host`` parameter of the ``comm`` entry). See :ref:`device-db` for more information. The example device database is designed for the NIST QC1 hardware on the KC705; see :ref:`board-ports` for RTIO channel assignments if you need to adapt the device database to a different hardware platform.
 
 .. note::
     If the ``led`` device is a bidirectional TTL (i.e. ``TTLInOut`` instead of ``TTLOut``), you need to put it in output (driving) mode. Add the following at the beginning of ``run``: ::

doc/manual/getting_started_mgmt.rst

@@ -10,7 +10,7 @@ Starting your first experiment with the master
 
 In the previous tutorial, we used the ``artiq_run`` utility to execute our experiments, which is a simple stand-alone tool that bypasses the ARTIQ management system. We will now see how to run an experiment using the master (the central program in the management system that schedules and executes experiments) and the GUI client (that connects to the master and controls it).
 
-First, create a folder ``~/artiq-master`` and copy the ``ddb.pyon`` and ``pdb.pyon`` files (device and parameter databases) found in the ``examples/master`` directory from the ARTIQ sources. The master uses those files in the same way as ``artiq_run``.
+First, create a folder ``~/artiq-master`` and copy the ``device_db.pyon`` and ``dataset_db.pyon`` (containing the device and dataset databases) found in the ``examples/master`` directory from the ARTIQ sources. The master uses those files in the same way as ``artiq_run``.
 
 Then create a ``~/artiq-master/repository`` sub-folder to contain experiments. The master scans this ``repository`` folder to determine what experiments are available (the name of the folder can be changed using ``-r``).
 

doc/manual/utilities.rst

@@ -104,9 +104,7 @@ The artiq_coretool utility allows to perform maintenance on the core device:
     * as well as read, write and remove key-value records from the :ref:`core-device-flash-storage`;
     * erase the entire flash storage area.
 
-To use this tool, you need to specify a ``ddb.pyon`` DDB file which contains a ``comm`` device (an example is provided in ``artiq/examples/master/ddb.pyon``).
+To use this tool, you need to specify a ``device_db.pyon`` device database file which contains a ``comm`` device (an example is provided in ``artiq/examples/master/device_db.pyon``). This tells the tool how to connect to the core device (via serial or via TCP) and with which parameters (baudrate, serial device, IP address, TCP port). When not specified, the artiq_coretool utility will assume that there is a file named ``device_db.pyon`` in the current directory.
-This tells the tool how to connect to the core device (via serial or via TCP) and with which parameters (baudrate, serial device, IP address, TCP port).
-When not specified, the artiq_coretool utility will assume that there is a file named ``ddb.pyon`` in the current directory.
 
 
 To read the record whose key is ``mac``::