M-Labs
/
artiq
mirror of https://github.com/m-labs/artiq
2
0
Fork 1
Code Issues Projects Releases Wiki Activity
8,802 Commits
18 Branches
34 Tags
36 MiB
Commit Graph

272 Commits (master)

Author SHA1 Message Date
Sebastien Bourdeauducq 10267f39c9 log: use broadcast instead of sync_struct, filter on new messages only (#411) 2016-05-27 23:43:29 -05:00
Sebastien Bourdeauducq 597d7c389e dashboard: support reloading arguments from HDF5 2016-05-05 00:51:30 +08:00
Sebastien Bourdeauducq c50555e11c worker: use unix time for HDF5 start_time 2016-05-03 21:29:54 +08:00
Sebastien Bourdeauducq 0cf6df1933 master/experiments: log more details about experiment name conflicts 2016-04-16 21:36:21 +08:00
Sebastien Bourdeauducq b74aa12351 master/worker_db: rethrow DeviceError exception with device name on device creation failure. Closes #396 2016-04-16 19:55:08 +08:00
Sebastien Bourdeauducq caf774579a environment: refactor 2016-04-16 19:31:07 +08:00
Sebastien Bourdeauducq 12a8c76df9 environment,worker: remove enable_processors 2016-04-16 14:21:01 +08:00
Sebastien Bourdeauducq 848c86e8a4 master: group results by hour, not by minute 2016-04-15 01:13:53 +08:00
Sebastien Bourdeauducq 437b37b158 master/worker: pause/resume devices 2016-04-12 15:45:53 +08:00
Sebastien Bourdeauducq 3c70bc40a4 master/worker_db: add pause_devices and resume_devices 2016-04-12 15:45:30 +08:00
Robert Jördens 30d557a4f8 worker: run experiment in output directory 2016-04-08 10:53:13 +08:00
Robert Jördens 1ae8b8ffd1 worker_impl: save expid, rid, start_time 2016-04-08 10:21:53 +08:00
Robert Jördens de002ac811 worker_db: factor get_output_prefix() 2016-04-08 10:21:53 +08:00
Robert Jördens 4759ea337b worker_impl: style 2016-04-05 17:22:16 +08:00
Robert Jördens 05c61465e0 worker: trust that h5py encodes strings 2016-04-05 17:18:15 +08:00
Robert Jördens 587a0f4565 worker: trust that h5py maps all types as we want 2016-04-05 17:18:15 +08:00
Robert Jördens 4e7b004a4e worker, hdf5: move datasets to subgroup 2016-04-05 17:17:18 +08:00
Sebastien Bourdeauducq aa61c29efb transfer Python builtin exceptions over pc_rpc and master/worker 2016-04-04 22:02:42 +08:00
Sebastien Bourdeauducq 774ecceaba master/worker_impl: use ParentDatasetDB in examine mode. Closes #368 2016-04-02 23:06:14 +08:00
Sebastien Bourdeauducq a53d32c24e master: always expose full set of worker handlers (#368) 2016-04-02 23:05:16 +08:00
Sebastien Bourdeauducq 84d50c4caf environment,worker_db: mutate datasets from experiments via dedicated method instead of Notifier. Closes #345 2016-03-29 16:26:14 +08:00
Sebastien Bourdeauducq ca4e6e03fa master/worker: use only first line in short_exc_info 2016-03-25 19:43:12 +08:00
Sebastien Bourdeauducq 0d7d584f08 master/worker_db: add controller_aux_target device_db entry type 2016-03-24 00:48:12 +08:00
Sebastien Bourdeauducq 4bf0db32be master: publish repository status 2016-03-18 00:40:17 +08:00
Sebastien Bourdeauducq 790269eee7 master/worker_db: make arguments optional in DDB entries 2016-03-05 00:17:08 +08:00
Sebastien Bourdeauducq 785691ab98 fix indentation 2016-02-29 21:32:48 +08:00
Sebastien Bourdeauducq dda0e0846c worker: print core device tracebacks. Closes #292 2016-02-25 01:25:26 +08:00
Sebastien Bourdeauducq 251aed047f worker: do not print redundant exception information 2016-02-25 01:10:31 +08:00
Robert Jördens fef72506e4 ctlmgr/gui/master: start subprocesses in new pgroup 
This only makes a difference on POSIX. It prevents subprocesses
from receiving the signals that the parent receives. For ctlmgr
and master is cuts down on spam on the console (KeyboardInterrupt
tracebacks from all controllers) and enforces that proper
termination is followed.

This does not help if the parent gets SIGKILL (subprocesses
may linger).
 2016-02-18 23:51:12 +01:00
Sebastien Bourdeauducq 155c2ec2ef ctlmgr,worker: set PYTHONUNBUFFERED for subprocesses 2016-02-18 12:41:08 +01:00
Sebastien Bourdeauducq 6196aaf2f5 master/worker: increase timeouts. Windows VMs can be really slow. 2016-02-16 09:44:50 +01:00
Sebastien Bourdeauducq 72a993afe0 master: cache last RID. Closes #234 2016-02-15 18:20:50 +01:00
Robert Jördens 6434a9cd5f Merge branch 'master' into subprocess-termination 
* master: (44 commits)
  Revert "conda: restrict binutils-or1k-linux dependency to linux."
  manual/installing: refresh
  use https for m-labs.hk
  gui/log: top cell alignment
  master/log: do not break lines
  conda: fix pyqt package name
  gui/applets: log warning if IPC address not in command
  applets: make sure pyqtgraph imports qt5
  applets: avoid argparse subparser mess
  examples/histogram: artiq -> artiq.experiment
  gui/applets: save dock UID in state
  setup.py: give up trying to check for PyQt
  setup.py: fix PyQt5 package name
  Use Qt5
  applets: fix error message text
  applets: handle dataset mutations
  applets: properly name docks to support state save/restore
  applets: clean shutdown
  protocols/pyon: set support
  protocols/pyon: remove FlatFileDB
  ...
 2016-02-11 09:24:45 +01:00
Sebastien Bourdeauducq 2ce1e6a1e9 master/log: do not break lines 2016-02-10 06:33:41 +01:00
Robert Jördens 53e5d0a7bb worker: flake8 style cleanup 2016-02-02 15:32:40 -07:00
Robert Jördens 55006119c8 subprocesses: unify termination logic 2016-02-02 15:32:36 -07:00
Sebastien Bourdeauducq 5076c85ed6 worker: Windows VMs are slow, increase send_timeout 2016-01-27 21:15:22 +01:00
Sebastien Bourdeauducq be5162d60f worker: restore short exception info in first line of log 2016-01-26 22:07:54 +01:00
Sebastien Bourdeauducq 3cf67afeb1 generalize multiline logging to remote logs and controllers 2016-01-26 21:59:37 +01:00
Sebastien Bourdeauducq 1fed38a8dc worker: use MultilineFormatter 2016-01-26 21:30:28 +01:00
Sebastien Bourdeauducq 5aa4de8e89 refactor logging and implement in worker 2016-01-26 20:31:42 +01:00
Sebastien Bourdeauducq a583a923d8 worker: use pipe_ipc (no log) 2016-01-26 14:59:36 +01:00
Sebastien Bourdeauducq ae19f1c75d master: add filename in worker log entries. Closes #226 2016-01-23 21:43:24 -05:00
Sebastien Bourdeauducq cc6b808bf8 master: finer control of worker exception reporting. Closes #233 2016-01-23 21:23:02 -05:00
Sebastien Bourdeauducq f9323c3286 master/worker_db/get_last_rid: ignore improperly named files 2016-01-16 10:56:01 -07:00
Sebastien Bourdeauducq a808d26fc2 style 2016-01-16 10:48:09 -07:00
whitequark be560dbc63 Commit missing parts of 13e65c2a. 2016-01-16 03:00:17 +00:00
whitequark 13e65c2a0a scheduler: make sure worker exceptions are not unexpectedly hidden. 2016-01-16 02:20:32 +00:00
whitequark 6bf48e60ba worker: make parent errors readable in log. 2016-01-16 02:06:40 +00:00
whitequark 67d2e7a828 worker: display compile warnings and errors nicely (#227). 2016-01-16 01:28:26 +00:00
whitequark e6185e16d8 Commit missing parts of 127b117. 2016-01-16 00:06:18 +00:00
whitequark 127b117113 Add @host_only function decorator (#172). 2016-01-15 16:42:08 +00:00
whitequark 03dd1c3a43 Refactor the logic of printing diagnostics to solely rely on Engine. 2016-01-04 22:11:54 +08:00
Sebastien Bourdeauducq fc299ca918 language/environment: disable processors by default 2015-12-22 11:45:34 +08:00
Sebastien Bourdeauducq 93317d48c9 gui/explorer: file selector for experiments outside repos 2015-12-09 19:13:57 +08:00
Sebastien Bourdeauducq 7b2580583a master: allow remote listing of directories 2015-12-08 19:24:04 +08:00
Sebastien Bourdeauducq 5e14afde3e scheduler: use current (last scanned) repo revision instead of HEAD 2015-12-06 19:00:41 +08:00
Sebastien Bourdeauducq ea6dcb2743 master: repository -> experiment_db 2015-12-06 18:39:27 +08:00
Sebastien Bourdeauducq 8467013160 master,gui: support recomputation+reset of arguments 2015-12-06 17:27:15 +08:00
whitequark c6ccc87f54 worker: gracefully handle compile errors (fixes #179). 2015-11-24 22:34:33 +08:00
Sebastien Bourdeauducq de30a4b060 master/worker: print short exception info in first log entry of worker error 2015-11-22 23:26:32 +08:00
Sebastien Bourdeauducq 723ef71a87 master/repository: fix experiment name conflict handling 2015-11-16 22:46:40 +08:00
Sebastien Bourdeauducq d32c2c091f master: scan subdirectories in repos 2015-11-14 00:06:52 +08:00
Sebastien Bourdeauducq 44c0c617f3 master/worker: add ARTIQ version to HDF5 output 2015-11-09 12:32:29 +08:00
Sebastien Bourdeauducq 2c77c80b4f master: expose more scheduler APIs to the experiments 2015-10-30 13:41:18 +08:00
Sebastien Bourdeauducq 32c95f24d0 worker: reduce some logging levels 2015-10-29 09:34:41 +08:00
Sebastien Bourdeauducq 0d53f7ab0d ignore ProcessLookupError when killing subprocesses. Closes #167 2015-10-28 20:57:28 +08:00
Sebastien Bourdeauducq 40b4129c65 Only support scalars and numpy arrays in HDF5 output. Update documentation. Closes #145 2015-10-28 18:33:42 +08:00
Sebastien Bourdeauducq ec328cf5e1 master/worker_db: reduce exception backtrace clutter of get_dataset. Closes #168 2015-10-28 17:50:48 +08:00
Sebastien Bourdeauducq 828b48ad89 master/scheduler: reduce logging severity of worker exception backtraces to debug 2015-10-28 17:48:50 +08:00
Sebastien Bourdeauducq 1ada15ae5d master: simplify worker/parent RPC 2015-10-28 17:35:57 +08:00
Sebastien Bourdeauducq fcd29492f9 worker_db: support more types in HDF5 output. Closes #144. Closes #121 2015-10-24 10:54:59 +08:00
Sebastien Bourdeauducq 3044a053e1 databases: assume empty dataset_db if file not present. Closes #156 2015-10-24 10:15:20 +08:00
Sebastien Bourdeauducq d13b368a65 build logging into worker 2015-10-20 18:11:50 +08:00
Sebastien Bourdeauducq 03e317780b device_db: make best_effort parameter optional for controllers 2015-10-18 14:37:08 +08:00
Sebastien Bourdeauducq a5606768ad device_db: make target_name parameter optional for controllers 2015-10-18 14:35:53 +08:00
Sebastien Bourdeauducq 04a049b93d master: timestamp logfiles 2015-10-17 10:26:54 +08:00
Sebastien Bourdeauducq 60ed88f3d9 worker: separate stdout/stderr for forwarding 2015-10-17 10:22:20 +08:00
Sebastien Bourdeauducq f6fd7ecef2 logging: handle newlines in messages 2015-10-17 10:21:03 +08:00
Sebastien Bourdeauducq f332c1d3cc ctlmgr: forward controller logs 2015-10-16 20:08:11 +08:00
Sebastien Bourdeauducq 9bb94f0f59 master/log: move formatter into LogBufferHandler 2015-10-16 18:35:30 +08:00
Sebastien Bourdeauducq cbda753f44 master: TCP server for remote logging 2015-10-16 00:53:35 +08:00
Sebastien Bourdeauducq 9e2e233fef master: log file support 2015-10-15 23:47:31 +08:00
Sebastien Bourdeauducq 86e201c6a0 master/log: parse names, better filtering, cleanup 2015-10-14 16:30:57 +08:00
Sebastien Bourdeauducq 9f04af63e6 scheduler: raise logging severity of errors 2015-10-14 16:02:22 +08:00
Sebastien Bourdeauducq c00bce3967 master,client,gui: logging sources, timestamps, levels 2015-10-14 01:06:57 +08:00
Sebastien Bourdeauducq 1d14975bd5 worker: cleaner termination on exception in user code, improve unittest 2015-10-13 01:11:57 +08:00
Sebastien Bourdeauducq 3cec176318 get_ddb -> get_device_db 2015-10-12 19:46:31 +08:00
Sebastien Bourdeauducq 97accd2540 merge parameters and results into datasets 2015-10-12 17:18:23 +08:00
Sebastien Bourdeauducq 139072d402 Graceful experiment termination. Closes #76 2015-10-06 13:50:00 +08:00
Sebastien Bourdeauducq b3584bc190 language,master,run: support raw access to DDB from experiments. Closes #123 2015-10-04 18:29:39 +08:00
Sebastien Bourdeauducq 512bc79484 master: consistent db function argument names 2015-10-04 18:27:44 +08:00
Sebastien Bourdeauducq 0e3927b01a master: support DDB rescan 2015-10-04 17:38:07 +08:00
Sebastien Bourdeauducq f552d62b69 use Python 3.5 coroutines 2015-10-03 19:28:57 +08:00
Sebastien Bourdeauducq b117b9320d asyncio.async -> asyncio.ensure_future 2015-10-03 14:37:02 +08:00
Sebastien Bourdeauducq 125503139e remove workaround for Python bug in asyncio process.wait(). Requires Python 3.5. Closes #58 2015-10-03 14:33:18 +08:00
Sebastien Bourdeauducq cd3107ba75 do not use deprecated asyncio.JoinableQueue 2015-10-03 13:59:18 +08:00
Sebastien Bourdeauducq bb228b00ef gui,language,master: argument groups 2015-08-24 23:46:54 +08:00
Sebastien Bourdeauducq 06badd1dc1 scheduler: refactor, fix pipeline hazards 2015-08-10 21:58:11 +08:00
Sebastien Bourdeauducq f81b2eba43 master: better repo scan error handling 2015-08-08 23:36:12 +08:00
Sebastien Bourdeauducq 400b414885 master,client: support scanning specific repo revisions 2015-08-08 23:23:25 +08:00
Sebastien Bourdeauducq c80f0fa07a master: delete head repository checkout on exit 2015-08-08 11:44:19 +08:00
Sebastien Bourdeauducq 54d85efc2a master,gui: show Git commit messages in schedule 2015-08-08 11:08:04 +08:00
Sebastien Bourdeauducq 7ed8fe57fa Git support 2015-08-07 15:51:56 +08:00
Sebastien Bourdeauducq 9261254653 language/environment: support non-stored results 2015-08-06 18:43:27 +08:00
Sebastien Bourdeauducq 86fef7b53b master: do not scan experiments starting with '_' 2015-07-29 23:29:07 +08:00
Sebastien Bourdeauducq 3a06e22b67 master: handle logging while scanning repository 2015-07-23 23:06:15 +08:00
Sebastien Bourdeauducq 8402f1cdcd master,gui: basic log support 2015-07-22 05:13:50 +08:00
Sebastien Bourdeauducq 2576036ba1 language,worker: preserve order of arguments 2015-07-17 19:32:38 +02:00
Sebastien Bourdeauducq 6b36d93cc2 master,client: support repository rescan 2015-07-17 18:55:48 +02:00
Sebastien Bourdeauducq 39c6bc940c coreconfig: use new database API (closes #75) 2015-07-17 10:49:55 +02:00
Sebastien Bourdeauducq 84de2fb28b expid: experiment -> class_name 2015-07-15 11:08:12 +02:00
Sebastien Bourdeauducq 7de56666e3 worker,environment: support scanning of arguments with no default 2015-07-15 10:59:48 +02:00
Sebastien Bourdeauducq 9ed4dcd7d1 repository: load experiments in worker, list arguments 2015-07-15 10:54:44 +02:00
Sebastien Bourdeauducq 7770ab64f2 worker: factor timeouts 2015-07-14 23:43:08 +02:00
Sebastien Bourdeauducq 32d141f5ac refactor ddb/pdb/rdb 2015-07-13 22:21:32 +02:00
Sebastien Bourdeauducq 96a5d73c81 worker: split build stage from prepare 2015-07-09 13:18:12 +02:00
Sebastien Bourdeauducq c71fe29792 simplify unit system and use floats by default 2015-06-26 16:34:37 +02:00
Sebastien Bourdeauducq 9f9079589e gui: send monitor requests to core device 2015-06-05 14:52:41 +08:00
Sebastien Bourdeauducq a6a476593e worker: wait for process termination 
This prevents stray SIGCHLDs from crashing the program e.g. if the asyncio event loop is closed before the process actually terminates.
 2015-06-05 00:37:26 +08:00
Sebastien Bourdeauducq c843c353d7 worker: remove useless process wait 2015-06-05 00:05:38 +08:00
Yann Sionneau 60bdf74137 tests: use try/finally to close event loop + wait for process to die after killing it 2015-06-04 13:40:13 +02:00
Sebastien Bourdeauducq 78f9268277 worker: add note about correct use of close() 2015-06-04 11:30:34 +08:00
Sebastien Bourdeauducq aa242f7c66 scheduler: simplify priority policy 
Remove overdueness. User must submit calibration experiments with higher priority values for them to take precedence.
 2015-05-28 18:24:45 +08:00
Sebastien Bourdeauducq b0f8141018 scheduler: cancel flush when run is cancelled 2015-05-28 17:48:33 +08:00
Sebastien Bourdeauducq e752e57fa5 scheduler: do not duplicate 'run terminated' information 2015-05-28 17:37:08 +08:00
Sebastien Bourdeauducq 737f6d4485 scheduler: support pipeline flush 2015-05-28 17:20:58 +08:00
Yann Sionneau 4bf7875b87 flash_storage: refactor + unit tests + artiq_coreconfig.py CLI + doc 2015-05-27 18:06:12 +08:00
Sebastien Bourdeauducq fc449509b8 scheduler: pass priority to experiments 2015-05-24 20:37:47 +08:00
Sebastien Bourdeauducq a21373841c scheduler: catch worker exceptions in prepare and analyze stages 2015-05-24 20:23:49 +08:00
Sebastien Bourdeauducq d6ced1c780 scheduler: support priorities 2015-05-24 01:09:22 +08:00
Sebastien Bourdeauducq b74b8d5826 Scheduling TNG 2015-05-17 16:11:00 +08:00
Sebastien Bourdeauducq 283695e8aa language/core: integrate watchdogs 2015-04-28 23:23:59 +08:00
Sebastien Bourdeauducq ca89b6d0ed worker_db: support read-only mode 2015-04-07 15:40:57 +08:00
Sebastien Bourdeauducq 3257275782 worker_db: get_logger -> getLogger 2015-04-05 18:02:07 +08:00
Sebastien Bourdeauducq 71b7fe38bc worker_impl: add missing import 2015-04-05 17:50:48 +08:00
Sebastien Bourdeauducq 9b46bc623a dbhub: do not use as context manager, turn close exceptions into warnings, do not close devices early in worker 2015-04-05 17:49:41 +08:00
Robert Jördens 43893c6c1d worker_impl: use DBHub as context manager 2015-04-04 20:51:48 -06:00
Robert Jördens 7b86138fd2 DBHub: make it a context manager 2015-04-04 20:51:48 -06:00
Sebastien Bourdeauducq 200e20fb39 worker: close devices immediately after run and before analyze 2015-03-20 12:01:34 +01:00
Sebastien Bourdeauducq 2b3641ac0a db,worker: fix realtime result initialization 2015-03-13 15:12:55 +01:00
Sebastien Bourdeauducq 43a05c783d worker: split write_results action 2015-03-11 19:06:46 +01:00
Sebastien Bourdeauducq d5795fd619 master: watchdog support 
Introduces a watchdog context manager to use in the experiment code that
terminates the process with an error if it times out. The syntax is:

with self.scheduler.watchdog(20*s):
   ...

Watchdogs timers are implemented by the master process (and the worker
communicates the necessary information about them) so that they can be
enforced even if the worker crashes. They can be nested arbitrarily.
During yields, all watchdog timers for the yielding worker are
suspended [TODO]. Setting up watchdogs is not supported in kernels,
however, a kernel can be called within watchdog contexts (and terminating
the worker will terminate the kernel [TODO]).

It is possible to implement a heartbeat mechanism using a watchdog, e.g.:

for i in range(...):
    with self.scheduler.watchdog(...):
        ....

Crashes/freezes within the iterator or the loop management would not be
detected, but they should be rare enough.
 2015-03-11 16:43:14 +01:00
Sebastien Bourdeauducq f2134fa4b2 master,worker: split prepare/run/analyze 2015-03-09 23:34:09 +01:00
Sebastien Bourdeauducq 4c280d5fcc master: use a new worker process for each experiment 2015-03-09 16:22:41 +01:00
Sebastien Bourdeauducq ec1d082730 remove timeout from run_params (to be replaced by a better mechanism) 2015-03-09 10:51:32 +01:00
Sebastien Bourdeauducq d95a9cac9a move realtime result registration into dbh, simplify syntax 2015-03-08 17:27:27 +01:00
Sebastien Bourdeauducq f2e3dfb848 Experiment base class, replace __artiq_unit__ with docstring 2015-03-08 15:55:30 +01:00
Sebastien Bourdeauducq 651ed71b79 master: basic automatic experiment list 2015-02-21 20:34:31 -07:00
Sebastien Bourdeauducq 4267e0deab master,artiq_run: simplify unit autodetection 2015-02-21 20:26:58 -07:00