harry
/
creotech-sayma-testsuite
1
0
Fork 1
Code Issues 3 Pull Requests Projects Releases Wiki Activity

Test Report: Single Sayma, DAC-to-DAC phase skew #2

New Issue
Open
opened 2021-05-25 16:42:02 +08:00 by harry · 8 comments
harry commented 2021-05-25 16:42:02 +08:00
Owner
Copy Link

Description

This test aims to validate the synchronisation of pre-defined waveforms generated on different AD9154 DAC chips on a single Sayma card. Repeated test runs are required for finding variation of phase alignment error across reboots.

The measured phase skew is outputted from plot_sayma_data.py, which is derived from @sb10q 's script.

Starting 2021-10-05, the statistics are outputted from reports generated from analyze_sayma_data.py.

Specifications

  • DAC input data describes 9 MHz in-phase sinusoids of same amplitude on all DAC channels
  • 6 dB attenuation on each BaseMod channel

Test Conditions & Results

Included and to be expanded in the comment section below. All phase skew values are in picoseconds.

## Description This test aims to validate the synchronisation of pre-defined waveforms generated on different AD9154 DAC chips on a single Sayma card. Repeated test runs are required for finding variation of phase alignment error across reboots. The measured phase skew is outputted from [plot_sayma_data.py](https://git.m-labs.hk/harry/creotech-sayma-testsuite/src/branch/master/plot_sayma_data.py), which is derived from @sb10q 's script. Starting 2021-10-05, the statistics are outputted from reports generated from [analyze_sayma_data.py](https://git.m-labs.hk/harry/creotech-sayma-testsuite/src/branch/master/analyze_sayma_data.py). ## Specifications * DAC input data describes 9 MHz in-phase sinusoids of same amplitude on all DAC channels * 6 dB attenuation on each BaseMod channel ## Test Conditions & Results Included and to be expanded in the comment section below. All phase skew values are in picoseconds. * [2021-04-23](./issues/2#issuecomment-2147) * [2021-04-28](./issues/2#issuecomment-2149) * [2021-05-24](./issues/2#issuecomment-2150) * [2021-05-25](./issues/2#issuecomment-2152) * [2021-10-05](./issues/2#issuecomment-2802) * [2021-10-06](./issues/2#issuecomment-2804) * [2021-10-07](./issues/2#issuecomment-2840) * [2021-11-23](./issues/2#issuecomment-3280)
harry added the
report
 label 2021-05-25 16:42:02 +08:00
harry commented 2021-05-25 16:42:18 +08:00
Author
Owner
Copy Link

2021-04-23

Conditions

  • Hardware: Metlino as DRTIO master, Sayma as DRTIO satellite (via uTCA carrier hub fabric)
  • ARTIQ: customized version (484b6c3b92) to fix GTH and AD9154 sync procedures and add hard-coded 9 MHz square wave outputs on Sayma MCX connectors (for DAC-TTL sync that is to be tested in a separate test)
  • Testsuite: 2cbf1f67bf, and in accordance to the test protocol

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 5
  • # Test Run with Extreme Data:
    • BaseMod outputs no signals: 1
  • Statistics excluding extreme data:
    • # Test Runs Included: 4
    • Min: 113.11
    • Max: 119.38
    • Std. Dev.: 2.933887731
    • Abs. Dev. (mean): 2.125059961

Discussion

  • BaseMod not outputting any signal could be a bug related to DRTIO link-up.
    • It has not re-appeared in future test runs.
### 2021-04-23 #### Conditions * Hardware: Metlino as DRTIO master, Sayma as DRTIO satellite (via uTCA carrier hub fabric) * ARTIQ: customized version (https://github.com/HarryMakes/artiq/commit/484b6c3b92017ae3fe3b226fc7147425b853a0f7) to fix GTH and AD9154 sync procedures and add hard-coded 9 MHz square wave outputs on Sayma MCX connectors *(for DAC-TTL sync that is to be tested in a separate test)* * Testsuite: 2cbf1f67bfc17189e76dc8eb57dd625578c21da0, and in accordance to the [test protocol](https://git.m-labs.hk/harry/creotech-sayma-testsuite/issues/1#issue-435) #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **5** * \# Test Run with Extreme Data: * BaseMod outputs no signals: **1** * Statistics excluding extreme data: * \# Test Runs Included: 4 * Min: 113.11 * Max: 119.38 * Std. Dev.: 2.933887731 * Abs. Dev. (mean): 2.125059961 #### Discussion * BaseMod not outputting any signal could be a bug related to DRTIO link-up. * It has not re-appeared in future test runs.
harry commented 2021-05-25 16:46:45 +08:00
Author
Owner
Copy Link

2021-04-28

Conditions

  • Hardware: unchanged
  • ARTIQ: unchanged
  • Testsuite: unchanged
  • Measurement: unchanged

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 13
  • # Test Run with Extreme Data:
    • BaseMod outputs no signals: 1
    • Absolute phase skew > π/2: 4
  • Statistics excluding extreme data:
    • # Test Runs Included: 8
    • Min: 150.79
    • Max: 159.16
    • Std. Dev.: 2.664442609
    • Abs. Dev. (mean): 1.959537157

Discussion

  • When the phase skew went extreme, there was 1 occurence where abs(skew) ≈ 2π/3, and 4 where abs(skew) ≈ π.
    • For abs(skew) ≈ 2π/3, skew was positive.
    • For abs(skew) ≈ π, 2 occurrences of positive skew, and 1 occurrence of negative skew.
    • This doesn't show any pattern.
### 2021-04-28 #### Conditions * Hardware: unchanged * ARTIQ: unchanged * Testsuite: unchanged * Measurement: unchanged #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **13** * \# Test Run with Extreme Data: * BaseMod outputs no signals: **1** * Absolute phase skew > π/2: **4** * Statistics excluding extreme data: * \# Test Runs Included: 8 * Min: 150.79 * Max: 159.16 * Std. Dev.: 2.664442609 * Abs. Dev. (mean): 1.959537157 #### Discussion * When the phase skew went extreme, there was 1 occurence where abs(skew) ≈ 2π/3, and 4 where abs(skew) ≈ π. * For abs(skew) ≈ 2π/3, skew was positive. * For abs(skew) ≈ π, 2 occurrences of positive skew, and 1 occurrence of negative skew. * This doesn't show any pattern.
harry commented 2021-05-25 16:57:59 +08:00
Author
Owner
Copy Link

2021-05-24

Conditions

  • Hardware: added one extra Sayma as satellite 2, sharing the same gateware as the first Sayma; new Sayma NOT involved in this test
  • ARTIQ: unchanged
  • Testsuite: ef463c32a5, which improves timing of the test protocol

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 5
  • # Test Run with Extreme Data:
    • Absolute phase skew > π/2: 2
  • Statistics excluding extreme data:
    • # Test Runs Included: 3
    • Min: 341.45
    • Max: 362.84
    • Std. Dev.: 11.31528325
    • Abs. Dev. (mean): 8.552765483

Discussion

  • Excluding extreme data, the skew values are generally at least doubled from previous data. The variation also exceeds previous records by quite a lot.
    • However, the variation should still be well within the tolerance we give on the cable length.
  • When the phase skew went extreme, there was 1 occurence where skew ≈ +π, and 1 where skew ≈ -π.
    • This doesn't show any pattern.
### 2021-05-24 #### Conditions * Hardware: added one extra Sayma as satellite 2, sharing the same gateware as the first Sayma; new Sayma NOT involved in this test * ARTIQ: unchanged * Testsuite: ef463c32a50fa0dc7e98182c4b910a8564222b0c, which improves timing of the [test protocol](https://git.m-labs.hk/harry/creotech-sayma-testsuite/issues/1#issue-435) #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **5** * \# Test Run with Extreme Data: * Absolute phase skew > π/2: **2** * Statistics excluding extreme data: * \# Test Runs Included: 3 * Min: 341.45 * Max: 362.84 * Std. Dev.: 11.31528325 * Abs. Dev. (mean): 8.552765483 #### Discussion * Excluding extreme data, the skew values are generally at least doubled from previous data. The variation also exceeds previous records by quite a lot. * However, the variation should still be well within the tolerance we give on the cable length. * When the phase skew went extreme, there was 1 occurence where skew ≈ +π, and 1 where skew ≈ -π. * This doesn't show any pattern.
harry commented 2021-05-25 17:14:41 +08:00
Author
Owner
Copy Link

2021-05-25

Conditions

  • Hardware: unchanged
  • ARTIQ: unchanged
  • Testsuite: unchanged

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 15
  • # Test Run with Extreme Data:
    • Absolute phase skew > π/2: 4
  • Statistics excluding extreme data:
    • # Test Runs Included: 11
    • Min: 136.60
    • Max: 156.07
    • Std. Dev.: 5.149441305
    • Abs. Dev. (mean): 3.124084702

Discussion

  • Excluding extreme data, the skew values and their variation are generally in line with measurements on or before 2021-04-28.
    • 2021-05-24 contained too few data points, but did show a different trend.
  • When the phase skew went extreme, there was 1 occurence where abs(skew) ≈ +π, and 3 where abs(skew) ≈ -π.
    • This doesn't show any pattern.
### 2021-05-25 #### Conditions * Hardware: unchanged * ARTIQ: unchanged * Testsuite: unchanged #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **15** * \# Test Run with Extreme Data: * Absolute phase skew > π/2: **4** * Statistics excluding extreme data: * \# Test Runs Included: 11 * Min: 136.60 * Max: 156.07 * Std. Dev.: 5.149441305 * Abs. Dev. (mean): 3.124084702 #### Discussion * Excluding extreme data, the skew values and their variation are generally in line with measurements on or before 2021-04-28. * 2021-05-24 contained too few data points, but did show a different trend. * When the phase skew went extreme, there was 1 occurence where abs(skew) ≈ +π, and 3 where abs(skew) ≈ -π. * This doesn't show any pattern.
harry commented 2021-10-06 17:39:04 +08:00
Author
Owner
Copy Link

2021-10-05

Conditions

  • Hardware: unchanged
  • ARTIQ: custom branch ad9154-gth-fix-release-6 (04a4b8075a) based on release-6 (d9b01ed81a)
  • Testsuite: unchanged
    • Statistics are calculated from analyze_sayma_data.py @ 97cfcdf45d

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 11
  • # Test Run with Extreme Data:
    • Absolute phase skew > π/2: None
  • Statistics:
    • Mean: 191.6758
    • Min: 164.3251
    • Max: 198.3547
    • Std. Dev.: 9.4752
    • Abs. Dev. (mean): 6.5418

Discussion

  • The ARTIQ custom branch in use addressed the link delay setup, involving steps to deduce new values for LMFCVar and LMFCDel by reading DYN_LINK_LATENCY_0, during which their reset values remain untouched.
    • New: LMFCVar = 4, LMFCDel = 10
  • With such a change, no more extreme data has been seen.
  • Interestingly, the min skew occurred at first run (at this point the uTCA crate has been powered-off more than 5 hours), and then jumped sharply close to the mean skew.
  • Subsequent runs resulted in little variation from the overall mean.
### 2021-10-05 #### Conditions * Hardware: unchanged * ARTIQ: custom branch `ad9154-gth-fix-release-6` (https://github.com/HarryMakes/artiq/tree/04a4b8075a1ce7fcd2babff14cd6a7d97b2e843f) based on `release-6` (https://github.com/m-labs/artiq/tree/d9b01ed81ad217d8c9d578432dbd2ebab45dc342) * Testsuite: unchanged * Statistics are calculated from analyze_sayma_data.py @ 97cfcdf45d4d8bde5519cb11c6c720cd5cb4f5ca #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **11** * \# Test Run with Extreme Data: * Absolute phase skew > π/2: **None** * Statistics: * Mean: 191.6758 * Min: 164.3251 * Max: 198.3547 * Std. Dev.: 9.4752 * Abs. Dev. (mean): 6.5418 #### Discussion * The ARTIQ custom branch in use addressed the link delay setup, involving steps to deduce new values for `LMFCVar` and `LMFCDel` by reading `DYN_LINK_LATENCY_0`, during which their reset values remain untouched. * New: `LMFCVar` = 4, `LMFCDel` = 10 * With such a change, no more extreme data has been seen. * Interestingly, the min skew occurred at first run (at this point the uTCA crate has been powered-off more than 5 hours), and then jumped sharply close to the mean skew. * Subsequent runs resulted in little variation from the overall mean.
harry commented 2021-10-06 17:44:53 +08:00
Author
Owner
Copy Link

2021-10-06

Conditions

  • Hardware: unchanged
  • ARTIQ: unchanged
  • Testsuite: 1688ceecde -- improved data collection & plotting using timestamps & logging
    • Statistics are calculated from analyze_sayma_data.py @ 97cfcdf45d

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 25
  • # Test Run with Extreme Data: None
  • # Test Run with Invalid/Missing Data: 1
  • Statistics:
    • Mean: 192.0353
    • Min: 165.6978
    • Max: 195.9422
    • Std. Dev.: 6.0150
    • Abs. Dev. (mean): 3.4889

Discussion

  • Same as previous day, no more extreme data has been seen.
  • Similar to previous day, the min skew occurred at first run (5-hour uTCA crate cooldown in prior), and then jumped sharply close to the mean skew. Subsequent runs resulted in little variation from the overall mean.
  • Worth noting that 1 data went missing because the RP was unable to be powered off with uhubctl.
    • This issue has occurred once in a long while, so perhaps we shouldn't power-cycle the RP between test runs?
### 2021-10-06 #### Conditions * Hardware: unchanged * ARTIQ: unchanged * Testsuite: 1688ceecde9a8dd87341b2d7beeb93ea27b7cb9e -- improved data collection & plotting using timestamps & logging * Statistics are calculated from analyze_sayma_data.py @ 97cfcdf45d4d8bde5519cb11c6c720cd5cb4f5ca #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **25** * \# Test Run with Extreme Data: **None** * \# Test Run with Invalid/Missing Data: **1** * Statistics: * Mean: 192.0353 * Min: 165.6978 * Max: 195.9422 * Std. Dev.: 6.0150 * Abs. Dev. (mean): 3.4889 #### Discussion * Same as previous day, no more extreme data has been seen. * Similar to previous day, the min skew occurred at first run (5-hour uTCA crate cooldown in prior), and then jumped sharply close to the mean skew. Subsequent runs resulted in little variation from the overall mean. * Worth noting that 1 data went missing because the RP was unable to be powered off with `uhubctl`. * This issue has occurred once in a long while, so perhaps we shouldn't power-cycle the RP between test runs?
harry commented 2021-10-07 17:09:33 +08:00
Author
Owner
Copy Link

2021-10-07

Conditions

  • Hardware: unchanged
  • ARTIQ: unchanged
  • Testsuite: 97cfcdf45d, except:
    • In test_mlabs, all lines for RP power-cycling have been removed. The RP was powered-on well before this test run series began. This is to be considered a new revision of the test protocol.

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 30
  • # Test Run with Extreme Data:
    • BaseMod outputs no signals: 1
  • # Test Run with Invalid/Missing Data: None
  • Statistics excluding extreme data:
    • Mean: 192.1416
    • Min: 154.7115 (run # 1)
    • Max: 197.0834
    • Std. Dev.: 7.6826
    • Abs. Dev. (mean): 4.2663

Discussion

  • The main purpose of this series of runs was to ensure RP oscilloscope stability when RP power-cycling is removed from the test protocol (ver. 1 is to be revised for upcoming test runs)
  • In run # 9, BaseMod did not output any signal.
    • Serial logs from Sayma AMC and RTM show connection problem between AMC and RTM. 
      [     7.328656s] ERROR(satman::repeater): [REP#0] received packet of an unknown type
[     7.789116s]  INFO(satman::repeater): [REP#0] link RX became up, pinging
[     7.794437s] ERROR(satman::repeater): [REP#0] received packet of an unknown type
[     7.801807s] ERROR(satman::repeater): [REP#0] link RX went down during ping
[     8.704338s]  INFO(satman::repeater): [REP#0] link RX became up, pinging
...
  • Similar as above, the min skew occurred at first run, and then jumped sharply close to the mean skew. Subsequent runs resulted in little variation from the overall mean.
### 2021-10-07 #### Conditions * Hardware: unchanged * ARTIQ: unchanged * Testsuite: 97cfcdf45d4d8bde5519cb11c6c720cd5cb4f5ca, except: * In `test_mlabs`, all lines for RP power-cycling have been removed. The RP was powered-on well before this test run series began. This is to be considered a new revision of the test protocol. #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **30** * \# Test Run with Extreme Data: * BaseMod outputs no signals: **1** * \# Test Run with Invalid/Missing Data: None * Statistics excluding extreme data: * Mean: 192.1416 * Min: 154.7115 (run # 1) * Max: 197.0834 * Std. Dev.: 7.6826 * Abs. Dev. (mean): 4.2663 #### Discussion * The main purpose of this series of runs was to ensure RP oscilloscope stability when RP power-cycling is removed from the test protocol **(ver. 1 is to be revised for upcoming test runs)** * In run # 9, BaseMod did not output any signal. * Serial logs from Sayma AMC and RTM show **connection problem between AMC and RTM**. ``` [ 7.328656s] ERROR(satman::repeater): [REP#0] received packet of an unknown type [ 7.789116s] INFO(satman::repeater): [REP#0] link RX became up, pinging [ 7.794437s] ERROR(satman::repeater): [REP#0] received packet of an unknown type [ 7.801807s] ERROR(satman::repeater): [REP#0] link RX went down during ping [ 8.704338s] INFO(satman::repeater): [REP#0] link RX became up, pinging ... ``` * Similar as above, the min skew occurred at first run, and then jumped sharply close to the mean skew. Subsequent runs resulted in little variation from the overall mean.
harry commented 2021-11-23 17:53:14 +08:00
Author
Owner
Copy Link

2021-11-23

Conditions

  • Hardware: unchanged
  • ARTIQ: 96128743dc -- patched version of ARTIQ-7 (from c940f104f1), plus preventing DDMTD-related errors from stopping SYSREF<>RTIO clock alignment
  • Testsuite: unchanged

Results

AD9154-0 DAC1 -> AD9154-1 DAC1
  • # Test Runs: 6
  • # Test Run with Extreme Data: None
  • # Test Run with Invalid/Missing Data: None
  • Statistics excluding extreme data:
    • Mean: 140.7851
    • Min: 124.9867 (run # 1)
    • Max: 146.0933
    • Std. Dev.: 7.1699
    • Abs. Dev. (mean): 5.2661
DDMTD errors
  • # reported DDMTD peak-to-peak jitter (raw=false, tolerance=1) failure: 5
    • Average # jitter failure per reboot (32 iterations): 14
  • # reported DDMTD slip failure: 5
    • Average # slip failure per reboot (1024 iterations): 46

Discussion

  • Since Sayma has been migrated to ARTIQ-7 with DRTIO patches (by @harry), there have been frequent DDMTD errors reported from calling test_ddmtd_stability(false, 1) or test_slip_ddmtd() in artiq/firmware/satman/jdcg.rs (residing in the RTM gateware). Sometimes, there is no error at all.
    • This kind of DDMTD errors were also independent from occurence of SYSREF<>TSC alignment errors, i.e. either, both or none of these two types of errors could happen in the same test run.
  • (Updated with correction) This series of test runs needs to verify DAC phase stability within the same Sayma system, as well as across multiple Sayma satellites connected to the same Metlino master with DRTIO.
    • This set of test results show that the phase stability between the Sayma AMC and RTM on the same system is unaffected by DDMTD errors.
    • However, instability of SYSREF<>RTIO alignment existing within a single Sayma system could propagate instability of DAC data alignment across multiple Sayma systems.
    • Therefore, test runs measuring DAC phases between two Sayma cards are to be done and reported in a separate Gitea issue (see #3).
  • Similar as before, the min skew occurred at first run, and then jumped sharply close to the mean skew. Subsequent runs resulted in little variation from the overall mean.
### 2021-11-23 #### Conditions * Hardware: unchanged * ARTIQ: https://github.com/HarryMakes/artiq/commit/96128743dc52c16b038a1a093b4b6da3f7032aec -- patched version of ARTIQ-7 (from https://github.com/m-labs/artiq/commit/c940f104f16286ae643ef59f38a20d59bde9a239), plus preventing DDMTD-related errors from stopping SYSREF<>RTIO clock alignment * Testsuite: unchanged #### Results ##### AD9154-0 DAC1 -> AD9154-1 DAC1 * \# Test Runs: **6** * \# Test Run with Extreme Data: **None** * \# Test Run with Invalid/Missing Data: **None** * Statistics excluding extreme data: * Mean: 140.7851 * Min: 124.9867 (run # 1) * Max: 146.0933 * Std. Dev.: 7.1699 * Abs. Dev. (mean): 5.2661 ##### DDMTD errors * \# reported DDMTD peak-to-peak jitter (raw=false, tolerance=1) failure: **5** * Average \# jitter failure per reboot (32 iterations): **14** * \# reported DDMTD slip failure: **5** * Average \# slip failure per reboot (1024 iterations): **46** #### Discussion * Since Sayma has been migrated to ARTIQ-7 with DRTIO patches (by @harry), there have been frequent DDMTD errors reported from calling `test_ddmtd_stability(false, 1)` or `test_slip_ddmtd()` in artiq/firmware/satman/jdcg.rs (residing in the RTM gateware). Sometimes, there is no error at all. * This kind of DDMTD errors were also independent from occurence of SYSREF<>TSC alignment errors, i.e. either, both or none of these two types of errors could happen in the same test run. * **(Updated with correction)** This series of test runs needs to verify DAC phase stability within the same Sayma system, *as well as* across multiple Sayma satellites connected to the same Metlino master with DRTIO. * This set of test results show that the phase stability between the Sayma AMC and RTM on the *same system* is unaffected by DDMTD errors. * However, instability of SYSREF<>RTIO alignment existing within a single Sayma system could propagate instability of DAC data alignment across multiple Sayma systems. * Therefore, test runs measuring DAC phases between two Sayma cards are to be done and reported in a separate Gitea issue (see #3). * Similar as before, the min skew occurred at first run, and then jumped sharply close to the mean skew. Subsequent runs resulted in little variation from the overall mean.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
Labels
Clear labels   
bug

Something is not working

  
discussion

This is a discussion that may or may not involve the codebase

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
notes

Descriptions, definitions, decisions etc. for the project

  
question

More information is needed

  
report

Results, analysis, etc.

  
wontfix

This won't be fixed

No Label
bug
discussion
duplicate
enhancement
help wanted
invalid
notes
question
report
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: harry/creotech-sayma-testsuite#2
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?