saymapp/pp.txt

This plan defines concrete performance goals, specifies which parties are responsible for specific deliverables and defines a project management structure. The aim is to provide sufficient specification and structure at the outset that all parties feel comfortable with their responsibilities and minimizes conflict. 

The plan includes design choices that simplify Sayma and lower costs. The github Sayma Issue tracking contains a long list of bug fixes and modifications to v1 (https://github.com/sinara-hw/sinara). The plan calls out explicit specifications in cases where there is not a clear consensus in the Issues. 

## Performers

Three developers will work together on Sayma v2.  
- Hardware Developer 
- Integration and Timing Developer 
- Software and Gateware Developer 
ARL will serve as Technical Point of Contact (TPOC) for the contracts.

## Tasks

Developer tasks are labeled HTj, Ojk and Mjk (jk integers) and are not divisible. If a task or feature is not assigned to a Developer then that Developer is not responsible for it.  

## Design Review

The design review consists of three steps: HT1, HT1.5 and HT2. The Hardware Developer has primary responsibility for the design review. Certain review tasks are delegated to other Developers as detailed below. Tracking of design review tasks is in a google docs spreadsheet communicated under separate cover.

## Handling Hardware Flaws

The Software and Gateware Developer and Integration and Timing Developer have tasks which depend on proper functionality of the Sayma v2 hardware. The Hardware Developer testing and reporting expectations are detailed in HT3. 

### Flaw found in HT3

If the HT3 report indicates a failing component or subsystem, a Developer working on a dependent task may at its option pursue the following options
    - cancel dependent task(s)
    - renegotiate contract terms for dependent task(s) with TPOC

### Flaw found after HT3
If a component or subsystem receives passing marks on the HT3 report but a Developer finds a hardware problem after receiving the Sayma v2 hardware the following applies. Said Developer shall 
     - report the problem and work with the Hardware Developer to identify a patch
     - budget 24 hours for hardware-flaw related debugging 
     - apply white wire patches  
     - budget for two shipments of PCBs back to Hardware Developer for rework 
If shipment budget or hardware-flaw budget is exhausted the Developer may
    - cancel dependent task(s) and receive payment for up to 50% task cost
    - renegotiate contract terms for dependent task(s) with TPOC

# Definitions
Let __TS-MTCA__ be a micro TCA crate configured as follows: 
- 1 NAT NATIVE-R5 (or -R9)
- 1 (or 2) NAT AC 600D 
- 1 NAT MCH-Basic v3.5

Let __TS1__ be a Sayma test setup consisting of the following
- 1 Sayma AMC+RTM 
- Powered by ATX connector from benchtop supply

Let __TS2__ be a Sayma test setup consisting of the following
- Single uTCA chassis configured as TS-MTCA
- 2 Sayma AMC+RTM
- 1 Metlino 
- All RF channels producing 400 MHz sinusoid
- Cards in adjacent slots

Let __TS7__ be a Sayma test setup consisting of the following
- Single uTCA chassis configured as TS-MTCA
- 6 Sayma AMC+RTM
- 1 Metlino 
- All RF channels producing 400 MHz sinusoid
- Cards in adjacent slots

__SAWG v1.0__ 
The original SAWG was developed by M-Labs for Sayma v1.0. Here, "v1" refers to SAWG with the same feature set but targeted at Sayma v2. 
- f_rtio = f_dac_ref = 150 MHz 
- dac data rate 0.6 GSPS
- f_dac_clk = 2.4 GHz (4X interpolation) 
- synchronization: none
- functionality in 1st Nyquist zone
    - 240 MHz is max RF out 
    - f1, f2: +/- 125 MHz modulation, +/- 100 MHz usable within anti-aliasing filter bandwidth

### SAWG v2.0
This is an extension of SAWG v1.0 with a 1 GSPS data rate and with board-board synchronization. 
- f_rtio = f_dac_ref = 125 MHz (same as for Kasli)
- dac data rate 1.0 GSPS
- f_dac_clk = 2.0 GHz (2X interpolation) 
- synchronization: new Sayma v2 sysref scheme
- support 2 RF channels per DAC
- functionality in 1st Nyquist
    - 400 MHz is max RF out 
    - f1, f2: +/- 125 MHz modulation, +/- 100 MHz usable within anti-aliasing filter bandwidth

### SAWG v2.1
This is an extension of SAWG v2.0 that includes improvements in performance and gateware utilization. 
- support for 4 RF channels per DAC
- spurious frequency components due to DSP: < -70 dBc at full amplitude, constant frequency 
- improvements detailed in MT1 


# Sayma v2 Diffs
When diff also applies for [Metlino](https://github.com/sinara-hw/Metlino/issues) it is noted. 



## Clock recovery and synthesizer

[TODO 
I don't know how to reconcile requests to freeze the design with  well motivated but invasive requests to change the design. Let's discuss how to proceed at a programatic level at the Friday conference call. 

https://github.com/sinara-hw/Sayma_RTM/issues/16#issuecomment-452661542
]

## DAC-FPGA Synchronization
Deterministic phase alignment between DACs on separate Sayma PCBs between power cycles is an important design criterion and should be considered at all stages of design. This implies a) SYSREF is aligned with RTIO clock b) DAC clocks are synchronized to local SYSREF edges. 
- Follow approach demonstrated in Issue #583 and summarized in Issue #594. 
- Remove HMC7043, add components needed to generate MGTREFCLK. 
- Test conditions
    - SAWG output on all channels on two Sayma boards, 400 MHz continuous RF 
    - Sayma-local TTL output at 75 MHz
    - using existing RTIO infrastructure and TTL PHY
    - laboratory temperature stability +/- 0.5 C
    - Cycle crate power 100 times
- Specification 
    - __ST1__: SAWG-SAWG phase alignment error across 
all power cycles: < 1/f_dac_clk 
    - __ST2__: SAWG to TTL alignment error across all power cycles: < 0.5 ns

## DAC Performance

### DAC Phase Stability
Phase stability between DACs on separate Sayma PCBs is an important design criterion and should be considered at all stages of the design. This assumes that DAC-FPGA synchronization.  
- Test conditions
    - SAWG output on all channels on two Sayma boards, 400 MHz continuous RF 
    - Assume laboratory temperature stability +/- 0.5 C
    - Duration 24 hours
- Target performance
    - __ST3__: SAWG-SAWG phase alignment variation < +/- 0.5 deg 
    - __ST4__: SAWG-TTL alignment variation < +/-250 ps

### DAC Noise

Low cross talk between adjacent analog channels on same AFE is an important design criterion and should be considered at all stages of the design (particularly layout). 
- cross talk goal: at 400 MHz < -80 dBc.

Phase noise at 400 MHz (3 dB margin on AD9154 specification):
- 10 Hz offset, < -94 dBc
- 100 Hz offset, < -103 dBc 
- 1000 Hz offset, < -120 dBc
- 10,000 Hz offset, < -126 dBc

spurs: < -60 dBc SFDR across operating range, max amplitude


## Reduce complexity of Sayma v2
- Remove fast JESD ADCs from RTM layout
- Reduce complexity of clocking options
    - Remove clock mezzanine
    - User must supply RTM with DAC reference clock (f_dac_ref) via back-panel SMA
    - f_rtio is obtained from front-panel SMA (if master) or fiber+CDR (if satellite)
    - User must ensure that the following are phase locked: f_dac_ref and f_rtio 
    - Synth on RTM generates f_dac from f_dac_ref
    - No support for analog backplane (LLRFBP, https://github.com/sinara-hw/sinara/issues/498). 
- Unambiguous mapping between control pin and IC naming semantics in schematics and corresponding ARTIQ source. 
- Remove 24-bit multi-channel ADC from Sayma RTM layout. 

## AFE Interface to RTM 
- Use FMC connector for RF, digital and power. This approach is validated by tests in https://github.com/sinara-hw/sinara/issues/386.
- Reduce number of AFE's to 2.

## BaseMod AFE v2 
- New form factor to accommodate switch to FMC and 4 DAC channels per AFE.
- FMC-type front panels, with connector holes determined by choice of AFE.
- AFE name and version number on FMC front panel. 
- No RF power detector.
- Add LVDS ADC. Copy design from Sampler. No development support for ADC in this contract. 
- Hardware Developer shall specify PCB material assuming maximum RF frequency is 1 GHz. 

## Smart Agile Waveform Generator 
Use SAWG v1 with minimal modification to demonstrate Sayma v2 hardware. SAWG v2 will be pursued once Sayma v2 hardware is working. 

## Ethernet
- Replace MAX24287 by M88E1512. This is same family IC as KC705 which is already supported by ARTIQ.
- Configuration of M88E1512  is by static pin strapping: 1 Gbps, GMII, 1000BASE-X.
- RGMII is connected only with FPGA -- no interface to MMC.
- Layout so that uTCA Port0 and SFP can be driven by M88E1512 or by GTH transceiver on FPGA. Use coupling capacitors to choose which.
- Also applies to Metlino. 



------------

# Hardware Developer
The Hardware Developer is responsible for hardware design and manufacturing. Testing responsibilities of the Hardware Developer are those detailed in HT3. 

The Hardware Developer has deliverables designated HTn. Written progress reports
    - on the 1st and 15th of the month

## __HT1__ high level design
High-level schematics for all PCBs to include component part numbers, high-level component interconnection, power budget, high-level layout of all PCBs including location of ICs, board-to-board headers, RF shielding, heat sinks, mechanical support and front panel design. Post design files to github and tag. Participate in design review on github. 

HT1 delivery is complete when the Software and Gateware Developer, Integration and Timing Developer and UMD/ARL sign off on the drawings. 

## __HT1.5__ high level design review of interconnections

HT1.5 is a review of connections between components. It consists of the following checks. 

- Are design change clearly articulated? (eg IC choice)? Confirm that schematic satisfies Sayma v2 Project Plan section Sayma v2 Diffs.
    - Clocks
    - GTx, Ethernet
    - RTM to AMC connectivity requirements
    - qty of LVDS links to AFE for future physicist needs
    - Are resources sufficient for WR implementation?
    - Front/rear panel connectivity
    - power consumption

- Check lead time for key ICs.

- Are any BOM items near end of life?

- Check of newly created symbols in layout software. Reviewer checks all pins against schematic.
https://github.com/sinara-hw/sinara/issues/605


## __HT2__ detailed design 
Detailed schematics for all PCBs, finished component interconnections, signal integrity analysis, thermal analysis, power analysis, final PCB routing. Post design files to github and tag. Participate in final design review, accept delegated responsibilities from Software and Gateware Developer Design. Output is layout ready for manufacture. 

Hardware Developer shall lead HT2 design review. This design review is in collaboration with scientists and other developers. The review will not be comprehensive. It will focus on differences between v1 and v2 of the hardware. Shall include validation of FPGA pinout. 

Interaction with Software and Gateware Developer:
- Hardware Developer delivers FPGA netlist(s) Software and Gateware Developer.

HT2 delivery is complete when Software and Gateware Developer and Integration and Timing Developer sign off on the drawings. 


## __HT3__ manufacturing, baseline testing
Fabricate PCBs in the following quantities. TPOC provides sign off prior to commencement of manufacturing. 
- Metlino AMC: print 10 PCBs, stuff 4
- Sayma AMC v2: print 10 PCBs, stuff 6
- Sayma RTM v2: print 10 PCBs, stuff 6
- BaseMod v2: print 20 PCBs, stuff 12
- TestMod v2: print 6 PCBs, stuff 6 (DAC outputs to SMAs via baluns)

Expectations for testing of all stuffed PCBs.
- Atomic documentation of tests and bugs using github Issue tracking system. 
    - https://github.com/sinara-hw/Sayma_AMC
    - https://github.com/sinara-hw/Sayma_RTM
    - https://github.com/sinara-hw/Metlino
    - https://github.com/sinara-hw/BaseMod
- thermal test in TS7
    - goal: maximum temperature of any IC 65 C
    - Monitor temperature of several representative ICs using MMC IPMI 
- crate power-on with all boards and with arbitrary RTM unplugged in TS7
- hot swap arbitrary AMC and RTM boards in TS7
- power-on with or without RTM unplugged in TS1 
- Signal integrity testing using any suitable IP (eg Xilinx) for all ICs. Tested individually. 
- Design and implement test to simulate heavy loading of a system configured as TS2. The test is expected to simultaneously exercise the following subsystems  
    - I2C, SPI, address all ICs round robin at full speed
    - 2 TestMod installed dissipating representative power 
    - AD built-in DAC JESD PRBS test at 10 Gb/s lane for both DACs
    - SDRAM PRBS write-then-read at [TODO ____ data rate]
    - AMC backplane ethernet PRBS at 1 GSPS
    - SFP loop-back PRBS at 1 Gb/s
    - FMC loop-back PRBS [TODO ____ data rate]
- MMC configuration including power supply sequencing and IPMI
    - Hardware Developer shall test MMC firmware on TS7 system prior to distribution.
    - If errors arise after initial distribution Hardware Developer shall debug and test on TS2.
- Publish MCH configuration configuration file used for these tests with TS-MTCA and NATIVE-R5. 



Distribute hardware to system integrators:
- Software and Gateware Developer: 2 AMC, 2 RTM, 4 AFE, 1 Metlino
- Integration and Timing Developer: 2 AMC, 2 RTM, 4 AFE, 1 Metlino
- The Hardware Developer retains the following for testing and as backups in case of system integrator hardware failure: 2 AMC, 2 RTM, 4 AFE, 1 Metlino 

HT3 delivery is complete when the Software and Gateware Developer and Integration and Timing Developer confirm receipt of hardware. 

## __HT4__ documentation and MMC
- Support system integrators by github Issue system. 
- Develop and document fixes for hardware bugs. 
    - If bug fixes are substantial (eg replacing multi-pin SMD), system integrators may at their option return boards to the Hardware Developer for repair. System integrator pays shipping inbound, Hardware Developer pays shipping outbound. 
    - Each board should have a serial number, and a wiki page where everyone who reworks the board has to document what they did. Other notes about that particular board and its specific bugs can be added there.
- MMC and Exar configuration
    - Publish source on github within sinara-hw domain.
    - Explicit instructions for building and flashing.
    - Mimic ARTIQ release structure: tag official releases, change log.
- Formal hardware documentation maintained in github
    - Follow ARTIQ model using Sphinx. 
    - Setup automatic rendering on readthedocs with webhook to git pushes.
    - Embed link in github that links to readthedocs.
    - scope: expected content is superset of existing Sayma v1 documentation
    - scope: documentation to also include the following github Issues
https://github.com/sinara-hw/sinara/issues?q=is%3Aissue+documentation+label%3Aarea%3Adocumentation+is%3Aopen

HT4 delivery is complete when documentation and source is committed and accepted by TPOC. 

## __HT5__ delivery to UMD
Build demonstration system that is fully assembled and tested. The system shall consist of the following components.
- TS-MTCA
- 1 Metlino serving as ARTIQ Master
- 2 Sayma v2 (AMC + RTM)
- 4 BaseMod v2
- Test jigs, clock hardware, documentation and ARTIQ Python code needed to demonstrate the following: 
    - ST1, ST2, ST3 and ST4 
- Ship demonstration system to UMD.
- UMD/ARL will return TS-MTCA and clock hardware to the Hardware Developer 1 month after delivery. All other parts will remain property of UMD. 

HT5 delivery is complete when UMD confirms receipt of the demonstration system /and/ TPOC successfully runs test code. TPOC has 7 business days to conduct tests. 

## __HT6__ round up
- Update schematics and layout to remedy bugs.
- Publish to github list of factory acceptance tests including expected results and failure criteria. 


# Integration and Timing Developer 
An Integration and Timing Developer is responsible for certain aspects of hardware integration and testing as detailed in this section. 

This developer has one deliverable OT1. Written progress reports  
    - on the 1st and 15th of the month

## __OT1__ integration and timing

Dependency: Software and Gateware Developer is responsible for supplying AD9154 support at the level of M25 and M32. Integration and Timing Developer will handle debugging of ADF4356 and AD9154 as relates to ST4. 

Code submitted for inclusion in ARTIQ shall comply with CONTRIBUTING.rst (https://github.com/m-labs/artiq/blob/master/CONTRIBUTING.rst). 

- __O11__ implement DAC-FPGA synchronization with 2 GHz DAC clock (1 GSPS data rate).

- __O12__ participate in HT1 and HT1.5 design reviews

- __O13__ participate in HT2 design review

- __O14__ Based on O11, demonstrate ST4 for a pair of AD9154 DACs on a single Sayma v2 board in configuration TS1.
    - Write and document code. Generate pull request.

- __O15__ write and test ADF4356 driver
    - Write and document code. Generate pull request.

- __O16__ in support of a future clock distribution scheme modeled after White Rabbit (WR), specify layout of Si549 and related components 
    [TODO: How to support this if at all? - Write and document code. Generate pull request.]



OT1 delivery is complete when the Integration and Timing Developer completes and documents tasks O11 to O16. 





















# Software and Gateware Developer

The Software and Gateware Developer is responsible for gateware and software needed to provide ARTIQ support and to implement SAWG. All software and gateware development items include:
- Test benches or unit tests for gateware and software
- Examples for runtime and kernel APIs
- Documentation for exposed APIs
- Documentation of design choices 
- Documentation of code functionality

All MTk items include support of code in ARTIQ for 3 years from start of contract including documentation of public APIs, maintenance of developed CI infrastructure, publication of CI results. 

The developer’s work consists of deliverables MTk below. Written progress reports are due 
    - on 1st and 15th of the month 
    - on 1st of the month after OT1 and HT5 are complete

## __MT1__ SAWG

MT1 is a port of SAWG v1.0 designed for Sayma v1 to Sayma v2. Additional improvements in SAWG are also specified. The improvements include consideration of future modulation by RTIO and by Modulation by local DSP. 

Each MTk includes a short report and option to implement. 

- __M11__ Support SAWG v2.0
    - Develop code for Sayma v2 SAWG+JESD+fpga clock tree at 1 GSPS. 
    - Implement new Sayma v2 sysref synchronization scheme.  

- __M12__ SAWG v2.1 element: Measure resource usage and limit number of channels if unable to fit.

- __M13__ SAWG v2.1 element: Evaluate clocking SAWG or parts of it at 250 MHz (2x f_rtio)

- __M14__ SAWG v2.1 element: Determine number of CORDIC stages, width, phase resolution to achieve a specified spur suppression under specified conditions

- __M15__ SAWG v2.1 element: Design transaction based SAWG RTIO protocol (rewrite protocol to stage interpolator settings over one or few non-concurrent RTIO output channels, then activate synchronously with a single RTIO output event). Discuss and determine data partitioning among interpolators and SAWG channels.

- __M16__ SAWG v2.1 element: Design fractional representation of high-order spline interpolator coefficients (i.e. instead of 64 bits, just 16 denominator + 16 numerator bits for the 3rd order spline coefficient). This also allows getting rid of the SAWG clock stretcher that was never officially supported.

- __M17__ SAWG v2.1 element: Evaluate SAWG DUC redesign with 3 or 4 bit frequency and phase resolution based on multipliers instead of parallel cordics.

- __M18__ SAWG v2.1 element: Evaluate resource consumption of modulation ports for after spline interpolators. 
    - Multiplicative for amplitude, additive and saturating for frequency and phase. 
    - No support for configurable clipping amplitude at modulation summing junctions. Clipping is always on.  




## __MT2__ Sayma v2 Planning

- __M21__ Develop fixed test pattern generator (4-point sine on odd channels and ramps with major carry toggles on even channels) as a compile time alternative to SAWG.

- __MT22__ Review and verify Sayma v2 design from the FPGA perspective. Do this by building a stub ARTIQ target and test compilation -- depends on obtaining netlist for FPGAs from Hardware Developer (HT2). This will verify IO assignments and usage patterns around the following subsystems
    - ethernet
    - transceivers
    - clocking
    - SDRAM
    - flash
    - RTM interface
    - AFE ports
    - FMC ports

- __MT23__ Review of the scheme for synchronization between pairs of Sayma boards.



## __MT3__ Sayma v2 ARTIQ Support

- __M31__ Review the code emerging from O14 and O15. Support merge into ARTIQ. Aspects included in review: DRTIO-on-RTM, DRTIO clock
recovery on RTM, JESD204B deterministic latency/synchronization.
[TODO: does this include DDMTD?]

- __M32__ Work with TPOC to develop test cases for SAWG v2.1 with ST1 and ST2 in mind. Split into manual test cases and tests amendable to continuous integration (CI). Procure hardware needed for hardware-in-the-loop tests.

- __M33__ Support for Sayma_AMC to include the following
    - power up
    - boot tasks including JTAG, loading flash & FPGA
    - UARTs
    - SDRAM
    - ethernet
    - I2C EEPROM (MAC address and board serial numbers) 
    - clocking (including DRTIO clock recovery)
    - RTIO transcievers
    - RTIO SERDES TTL
    - FMC connectors
    [- TODO anything missing?]
    - build infrastructure and packaging  

- __M34__ Support for Sayma_RTM to include the following
    - power up
    - boot tasks including JTAG (slave serial) loading of flash & FPGA
    - RTM clocking
    - develop DRTIO for AMC FPGA to RTM FPGA link
    - SPI to HMC830 and AD9154
    - AFE interfaces
    - build infrastructure and packaging
    [- TODO anything missing?] 
    - dependency: M33

- __M35__ Support for BaseMod to include the following
    - attenuator shift register
    - DRTIO control of RF switches using TTL PHY
    - dependency: M34

- __M36__ Validate RF output using SAWG v1.0. This tests operation of SAWG as developed for Sayma v1 works on Sayma v2. It tests the following elements.  
    - clocking of AD9154
    - SYSREF syncing of DAC and JESD core
    - JESD204B to AD9154
    - SAWG v1.0 integration
    - dependency: M34

- __M37__ SAWG v2.0
    - ensure passage of tests ST1, ST2 and ST4
    - Double check ST3 using code supplied under task O14
    - dependencies: M36, O14

- __M38__ SAWG v2.1  
    - ensure proper implementation of features in SAWG v2.1
    - confirm that ST1, ST2, ST3 and ST4 tests still pass
    - dependency: M37

- __M39__ Documentation 
    - integrate manual and automated CI tests into ARTIQ infrastructure
    - document manual and automated CI tests, hardware-in-the-loop infrastructure 
    - document CI infrastructure and how to run CI tests
    - support Hardware Developer and TPOC in reproducing CI tests
    - document Sayma v2 ARTIQ support 
        - SAWG v2.1 ARTIQ Python API
        - document underlying DSP architecture


----------

# Project Organization
ARL/UMD will fund and manage contracts with @jbqubit serving as technical point of contact (TPOC).  

New github repositories were setup for each component of the Sayma system. 
- https://github.com/sinara-hw/Sayma_AMC
- https://github.com/sinara-hw/Sayma_RTM
- https://github.com/sinara-hw/Metlino
- https://github.com/sinara-hw/BaseMod

Use github Issue tracking for test results and bug tracking. Link back to the old sinara-hw/sinara Issues as needed but don’t conduct new discussions in sinara-hw/sinara. 

Weekly discussions via google hangout during period of performance. Attendance of at least one team member for each Developer team is expected. Discussion limited to 30 minutes per week. UMD/ARL involvement if requested by MC. 

Master of Ceremonies (MC) expectations:
- Solicit agenda items from group. Select agenda topics that can be completed in 30 minutes. 
- Regulate discussion so that planned agenda items are covered. For technical matters that prove contentious or detailed, shift discussion to github Issue.
- Publish agenda > 4 hours prior to meeting start.
- Launch google hangout > 5 minutes in advance of meeting start.
- Schedule next meeting time using doodle; advertise meeting time in UTC
- Each week rotate responsibilities for MC and minutes-taking between Developers and UMD/ARL
Agenda items must include: 
    - vote on acceptance of last meeting's minutes
    - designate next meeting's MC
    - designate next meeting minutes-taker
    - ask participants to reflect on how the meeting went and what to do better next week