Firmware is rather silent with faulty clocking. #198

New Issue

SingularitySurfer · 2022-08-24T22:28:55+08:00

SingularitySurfer commented

2022-08-24 22:28:55 +08:00

When the device is configured for an external clock and you do not provide that at boot, the firmware doesn't panic or turn on an LED. Only on the debug console you can see RTIO PLL failed to lock, without it beeing marked as a severe error.
An experiment will start as expexted but no RTIO events take place and it stops when it expects any return from the RTIOs.
Maybe a more definitive fault state that is consistent between all Kaslis (aka panic and turn on the Error LED) would be easier to debug for users.

When the device is configured for an external clock and you do not provide that at boot, the firmware doesn't panic or turn on an LED. Only on the debug console you can see `RTIO PLL failed to lock`, without it beeing marked as a severe error. An experiment will start as expexted but no RTIO events take place and it stops when it expects any return from the RTIOs. Maybe a more definitive fault state that is consistent between all Kaslis (aka panic and turn on the Error LED) would be easier to debug for users.

mwojcik commented

2022-08-25 13:37:14 +08:00

Good point.

Before recent changes, it would hang (with no indication at all), and any changes to config (e.g. changing clock source) or firmware with coremgmt were impossible - you had to take out the SD card, modify the config file.

Error LED has not been implemented yet, although it is present on the board.

How about we keep the behavior of not panicking (so config errors could be amended), but turn on the LED? The log line is already an error, too.

Good point. Before recent changes, it would hang (with no indication at all), and any changes to config (e.g. changing clock source) or firmware with coremgmt were impossible - you had to take out the SD card, modify the config file. Error LED has not been implemented yet, although it is present on the board. How about we keep the behavior of not panicking (so config errors could be amended), but turn on the LED? The log line is already an error, too.

sb10q commented

2022-08-25 13:39:12 +08:00

keep the behavior of not panicking

Alternatively, somehow keep the mgmt interface active in the panic handler. This way the crash messages can be retrieved (1) after the fact and (2) without connecting the UART cables.

> keep the behavior of not panicking Alternatively, somehow keep the mgmt interface active in the panic handler. This way the crash messages can be retrieved (1) after the fact and (2) without connecting the UART cables.

mwojcik referenced a pull request that will close this issue

2022-09-01 12:07:18 +08:00

Soft panic for RTIO PLL reasons #199

sb10q closed this issue

2022-10-21 17:56:35 +08:00

sb10q referenced this issue from a commit

2022-10-21 17:56:35 +08:00

enable network and mgmt during Rust panic, make RTIO PLL lock failure a panic

SingularitySurfer commented

2022-10-21 18:16:57 +08:00

Poster

thanks!