Test Report: Dual Sayma, DAC-to-DAC phase skew #3

New Issue

harry · 2021-11-24T09:42:11+08:00

harry commented

2021-11-24 09:42:11 +08:00

Description

Extended from #2, this test aims to validate the synchronisation of pre-defined waveforms generated on two Sayma cards. Repeated test runs are required for finding variation of phase alignment error across reboots.

The measured phase skew is outputted from plot_sayma_data.py, which is derived from @sb10q 's script. The statistics shown below are outputted from reports generated from analyze_sayma_data.py.

Specifications

DAC input data describes 9 MHz in-phase sinusoids of same amplitude on all DAC channels
6 dB attenuation on each BaseMod channel

Test Conditions & Results

Included and to be expanded in the comment section below. All phase skew values are in picoseconds.

## Description Extended from #2, this test aims to validate the synchronisation of pre-defined waveforms generated on two Sayma cards. Repeated test runs are required for finding variation of phase alignment error across reboots. The measured phase skew is outputted from [plot_sayma_data.py](https://git.m-labs.hk/harry/creotech-sayma-testsuite/src/branch/master/plot_sayma_data.py), which is derived from @sb10q 's script. The statistics shown below are outputted from reports generated from [analyze_sayma_data.py](https://git.m-labs.hk/harry/creotech-sayma-testsuite/src/branch/master/analyze_sayma_data.py). ## Specifications * DAC input data describes 9 MHz in-phase sinusoids of same amplitude on all DAC channels * 6 dB attenuation on each BaseMod channel ## Test Conditions & Results Included and to be expanded in the comment section below. All phase skew values are in picoseconds. * [2021-11-24](./issues/3#issuecomment-3290) * [2021-11-25](./issues/3#issuecomment-3298)

harry added the

report

label 2021-11-24 09:42:11 +08:00

harry referenced this issue

2021-11-24 09:54:07 +08:00

Test Report: Single Sayma, DAC-to-DAC phase skew #2

sb10q commented

2021-11-24 10:55:07 +08:00

DDMTD stability and HMC7043 phase slip failures need to be fully resolved before testing this.

👀 1

harry commented

2021-11-24 17:34:55 +08:00

2021-11-24

Conditions

Hardware: Metlino as DRTIO master, 2 sets of Sayma (Sayma-1, Sayma-2) as DRTIO satellites (via uTCA carrier hub fabric)
- Sayma-2 AD9154-1 is not usable due to hardware fault.
ARTIQ: custom branch ad9154-gth-fix-release-6 (04a4b8075a) based on release-6, with a nix-shell spawned with 575ef05cd5
Testsuite: 97cfcdf45d, except:
- In test_mlabs, all lines for RP power-cycling have been removed. The RP was powered-on well before this test run series began. This is to be considered a new revision of the test protocol.

Results

Sayma-1 AD9154-0 DAC1 -> Sayma-2 AD9154-1 DAC1

# Test Runs: 3
# Test Run with Extreme Data: 1
- Occurred in run # 3 - 0.1094 rad, 1934.71 ps.
# Test Run with Invalid/Missing Data: None
Statistics excluding extreme data:
- Mean: 442.7889
- Min: 355.7827
- Max: 529.7952
- Std. Dev.: 87.0063
- Abs. Dev. (mean): 87.0063

Discussion

In run # 3, where the skew is the highest, Sayma-1 reported failed to align SYSREF at FPGA: failed to reach SYSREF DDMTD phase target, while Sayma-2 did not report any related errors.
In all the other runs, no unexpected errors were reported on Metlino, Sayma-1 or Sayma-2.

### 2021-11-24 #### Conditions * Hardware: Metlino as DRTIO master, 2 sets of Sayma (**Sayma-1**, **Sayma-2**) as DRTIO satellites (via uTCA carrier hub fabric) * Sayma-2 AD9154-1 is not usable due to hardware fault. * ARTIQ: custom branch `ad9154-gth-fix-release-6` (https://github.com/HarryMakes/artiq/tree/04a4b8075a1ce7fcd2babff14cd6a7d97b2e843f) based on `release-6`, with a `nix-shell` spawned with https://git.m-labs.hk/M-Labs/nix-scripts/commit/575ef05cd554c239e4cc8cb97ae4611db458a80d * Testsuite: 97cfcdf45d4d8bde5519cb11c6c720cd5cb4f5ca, except: * In `test_mlabs`, all lines for RP power-cycling have been removed. The RP was powered-on well before this test run series began. This is to be considered a new revision of the test protocol. #### Results ##### Sayma-1 AD9154-0 DAC1 -> Sayma-2 AD9154-1 DAC1 * \# Test Runs: **3** * \# Test Run with Extreme Data: **1** * Occurred in run \# 3 - 0.1094 rad, 1934.71 ps. * \# Test Run with Invalid/Missing Data: None * Statistics excluding extreme data: * Mean: 442.7889 * Min: 355.7827 * Max: 529.7952 * Std. Dev.: 87.0063 * Abs. Dev. (mean): 87.0063 #### Discussion * In run \# 3, where the skew is the highest, Sayma-1 reported `failed to align SYSREF at FPGA: failed to reach SYSREF DDMTD phase target`, while Sayma-2 did not report any related errors. * In all the other runs, **no** unexpected errors were reported on Metlino, Sayma-1 or Sayma-2.

harry commented

2021-11-25 13:14:15 +08:00

2021-11-25

Conditions

Hardware: unchanged, Sayma-2 AD9154-1 remain unused.
ARTIQ: unchanged
Testsuite: unchanged

Results

Sayma-1 AD9154-0 DAC1 -> Sayma-2 AD9154-1 DAC1

# Test Runs: 30
# Test Run with Extreme Data: 11
- Occured in runs # 1, 3, 6, 5, 12, 13, 16, 19, 24, 25, 28
- Max: 1897.7273
- Min: -1722.2242
# Test Run with Invalid/Missing Data: None
Statistics excluding extreme data (# = 19):
- Mean: 577.8289
- Min: 388.0924
- Max: 753.7479
- Std. Dev.: 88.7857
- Abs. Dev. (mean): 62.7815

Discussion

SYSREF<>RTIO clock alignment error have been observed:
- failed to align SYSREF at FPGA: failed to reach SYSREF DDMTD phase target
  - Sayma-1: occurred in runs # 13, 19 & 28
  - Sayma-2: occurred in runs # 1, 3, 6, 7, 12, 16, 24 & 25
SYSREF<>TSC clock alignment error have been observed:
- failed to align SYSREF with TSC (SYSREF failed S/H timing)
  - Sayma-1: occurred in runs # 13 & 28
  - Sayma-2: occurred in runs # 1, 6, 12, 16 & 25
All occurrences of these SYSREF alignment errors exactly matches all the occurrences of extreme data.
- This implies that the extreme data were largely contributed by SYSREF alignment issues.
- Excluding the extreme data, the difference between max/min and the mean still shows certain degree of instability.
- We can perform another test series on the same day with the same setup and conditions, and then compare.
In all the other runs, no unexpected errors were reported on Metlino, Sayma-1 or Sayma-2.

### 2021-11-25 #### Conditions * Hardware: unchanged, Sayma-2 AD9154-1 remain unused. * ARTIQ: unchanged * Testsuite: unchanged #### Results ##### Sayma-1 AD9154-0 DAC1 -> Sayma-2 AD9154-1 DAC1 * \# Test Runs: **30** * \# Test Run with Extreme Data: **11** * Occured in runs \# 1, 3, 6, 5, 12, 13, 16, 19, 24, 25, 28 * Max: 1897.7273 * Min: -1722.2242 * \# Test Run with Invalid/Missing Data: None * Statistics excluding extreme data (\# = **19**): * Mean: 577.8289 * Min: 388.0924 * Max: 753.7479 * Std. Dev.: 88.7857 * Abs. Dev. (mean): 62.7815 #### Discussion * SYSREF<>RTIO clock alignment error have been observed: * `failed to align SYSREF at FPGA: failed to reach SYSREF DDMTD phase target` * Sayma-1: occurred in runs \# 13, 19 & 28 * Sayma-2: occurred in runs \# 1, 3, 6, 7, 12, 16, 24 & 25 * SYSREF<>TSC clock alignment error have been observed: * `failed to align SYSREF with TSC (SYSREF failed S/H timing)` * Sayma-1: occurred in runs \# 13 & 28 * Sayma-2: occurred in runs \# 1, 6, 12, 16 & 25 * All occurrences of these SYSREF alignment errors **exactly matches** all the occurrences of extreme data. * This implies that the extreme data were largely contributed by SYSREF alignment issues. * Excluding the extreme data, the difference between max/min and the mean still shows certain degree of instability. * We can perform another test series on the same day with the same setup and conditions, and then compare. * In all the other runs, **no** unexpected errors were reported on Metlino, Sayma-1 or Sayma-2.

harry commented

2021-12-02 11:06:12 +08:00

DDMTD stability and HMC7043 phase slip failures need to be fully resolved before testing this.

In all previous test runs based off ARTIQ-7 (e.g. single board DAC-DAC test on 2021-11-23), DDMTD peak-to-peak jitter and HMC7043 phase slip failure have been consistently replicated. However, this has never been replicable on any test runs based off ARTIQ-6 on the same hardware setup.

Right now I'm not confident to tell why it only detects DDMTD jitter on ARTIQ-7 but not ARTIQ-6, but I guess probably the change in CPU type and bus width (32 to 64 bits) might be related.

As for the HMC7043 phase slip error, I've actually tried reading the status (i.e. "clock outputs phases status") of the chip in all possible ways (SPI and GPO) on ARTIQ-7 when the error occured, but the error persists even though the chip has reported completion of each slip request.

Thus, I doubt there is any bug on our DDMTD gateware design or our firmware operations on HMC7043. We can prioritise our investigation for the SYSREF alignment procedures. No further debugging with ARTIQ-7 is needed at this moment.

> DDMTD stability and HMC7043 phase slip failures need to be fully resolved before testing this. In all previous test runs based off ARTIQ-7 (e.g. [single board DAC-DAC test on 2021-11-23](https://git.m-labs.hk/harry/creotech-sayma-testsuite/issues/2#issuecomment-3280)), DDMTD peak-to-peak jitter and HMC7043 phase slip failure have been consistently replicated. However, this has never been replicable on any test runs based off ARTIQ-6 on the same hardware setup. Right now I'm not confident to tell why it only detects DDMTD jitter on ARTIQ-7 but not ARTIQ-6, but I guess probably the change in CPU type and bus width (32 to 64 bits) might be related. As for the HMC7043 phase slip error, I've actually tried reading the status (i.e. "clock outputs phases status") of the chip in all possible ways (SPI and GPO) on ARTIQ-7 when the error occured, but the error persists even though the chip has reported completion of each slip request. Thus, I doubt there is any bug on our DDMTD gateware design or our firmware operations on HMC7043. We can prioritise our investigation for the SYSREF alignment procedures. No further debugging with ARTIQ-7 is needed at this moment.

Sign in to join this conversation.