Introduces a watchdog context manager to use in the experiment code that
terminates the process with an error if it times out. The syntax is:
with self.scheduler.watchdog(20*s):
...
Watchdogs timers are implemented by the master process (and the worker
communicates the necessary information about them) so that they can be
enforced even if the worker crashes. They can be nested arbitrarily.
During yields, all watchdog timers for the yielding worker are
suspended [TODO]. Setting up watchdogs is not supported in kernels,
however, a kernel can be called within watchdog contexts (and terminating
the worker will terminate the kernel [TODO]).
It is possible to implement a heartbeat mechanism using a watchdog, e.g.:
for i in range(...):
with self.scheduler.watchdog(...):
....
Crashes/freezes within the iterator or the loop management would not be
detected, but they should be rare enough.
