M-Labs/artiq-zynq
10
1
Fork 20
Code Issues 13 Pull Requests Releases Wiki Activity

Feature request: Make nix build logs permanently available. #355

New Issue
Closed
opened 2025-02-02 23:25:39 +08:00 by Dimitris · 3 comments
Dimitris commented 2025-02-02 23:25:39 +08:00
Copy Link

EDIT: If you want to get your Vivado build logs in a simple and deterministic way with just a few nix commands, skip this entire question and read the long answer at the bottom!

Feature request: Make nix build logs permanently available.

  • Whenever nix build [...] and in particular flake.nix -> "makeArtiqZynqPackage.kasli_soc-${variant}-sd" complete, we want to verify that the build log contains the magic phrase "All user specified timing constraints are met.".
  • In general, it would be nice to keep a copy of the build log permanently on file.

Problems

  1. If a variant has been built before, there is (almost) no build log. Solution: Place the log from the very first build as a file in /nix/store/$out and then never write to it again, only copy from it.
  2. If the name or the path of the json file (system description file) changes, then a new hash is generated for the final build result and a different /nix/store/$out is populated, but there is (almost) no build log, because the firmware and gateware are not affected and are thus not re-built. I don't know how to find the firmware and gateware programmatically because they do not appear in nix-store --query --references /nix/store/$out. Solution: ???

My own attempts

  • I have not found a way to call nix log recursively on /nix/store/...-paths so that the same build log is generated as from nix build [...].
  • I have added the code below to flake.nix -> "makeArtiqZynqPackage.kasli_soc-${variant}-sd", which places the log from the very first build as a file in /nix/store/$out, but this solution breaks when the json file (system description file) is simply renamed or moved to another directory, see problem 2. above.
      sd = pkgs.runCommand "${target}-${variant}-sd"
        {
          buildInputs = [ zynqpkgs.mkbootimage ];
        }
        ''
         [...]
          echo "--- NIX BUILD FINISHED IN \"$out\"; ATTEMPTING TO PLACE A COPY OF THE BUILD LOG THERE ---"
          log="/tmp/ARTIQ_ZYNQ_BUILD.log"
          if [ -f "$out/first_ever_build.log" ]; then
            echo "[INFO] \"$out/first_ever_build.log\" already exists. Nothing more to do."
          else
            echo "[INFO] \"$out/first_ever_build.log\" does not exist yet."
            if [ ! -f "$log" ]; then
              echo "[ERROR] Log file \"$log\" does not exist. Aborting."
            else
              echo "[INFO] Log file \"$log\" exists."
              echo "[INFO] \"$log\" copied to \"$out/first_ever_build.log\". Success!"
              cp "$log" "$out/first_ever_build.log"
            fi
          fi
        '';
_**EDIT:**_ If you want to get your Vivado build logs in a simple and deterministic way with just a few nix commands, skip this entire question and read the long answer at the bottom! ------- # Feature request: Make `nix build` logs permanently available. - Whenever `nix build [...]` and in particular [`flake.nix` -> `"makeArtiqZynqPackage.kasli_soc-${variant}-sd"`](https://git.m-labs.hk/M-Labs/artiq-zynq/src/commit/63157588bb8ed214bc9f24ade2d09b4ddefada03/flake.nix#L185) complete, we want to verify that the build log contains the magic phrase `"All user specified timing constraints are met."`. - In general, it would be nice to **keep a copy of the build log permanently on file**. ## Problems 1. If a variant has been built before, there is (almost) no build log. Solution: Place the log from the very first build as a file in `/nix/store/$out` and then never write to it again, only copy from it. 2. If the name or the path of the json file (system description file) changes, then a new hash is generated for the final build result and a different `/nix/store/$out` is populated, but there is (almost) no build log, because the firmware and gateware are not affected and are thus not re-built. I don't know how to find the firmware and gateware programmatically because they do not appear in `nix-store --query --references /nix/store/$out`. Solution: ??? ## My own attempts - I have not found a way to call `nix log` recursively on `/nix/store/...`-paths so that the same build log is generated as from `nix build [...]`. - I have added the code below to [`flake.nix` -> `"makeArtiqZynqPackage.kasli_soc-${variant}-sd"`](https://git.m-labs.hk/M-Labs/artiq-zynq/src/commit/63157588bb8ed214bc9f24ade2d09b4ddefada03/flake.nix#L185), which places the log from the very first build as a file in `/nix/store/$out`, but this solution breaks when the json file (system description file) is simply renamed or moved to another directory, see problem 2. above. ``` sd = pkgs.runCommand "${target}-${variant}-sd" { buildInputs = [ zynqpkgs.mkbootimage ]; } '' [...] echo "--- NIX BUILD FINISHED IN \"$out\"; ATTEMPTING TO PLACE A COPY OF THE BUILD LOG THERE ---" log="/tmp/ARTIQ_ZYNQ_BUILD.log" if [ -f "$out/first_ever_build.log" ]; then echo "[INFO] \"$out/first_ever_build.log\" already exists. Nothing more to do." else echo "[INFO] \"$out/first_ever_build.log\" does not exist yet." if [ ! -f "$log" ]; then echo "[ERROR] Log file \"$log\" does not exist. Aborting." else echo "[INFO] Log file \"$log\" exists." echo "[INFO] \"$log\" copied to \"$out/first_ever_build.log\". Success!" cp "$log" "$out/first_ever_build.log" fi fi ''; ```
sb10q commented 2025-02-02 23:27:29 +08:00
Owner
Copy Link

What's your issue with using the existing nix log feature?

What's your issue with using the existing ``nix log`` feature?
sb10q commented 2025-02-02 23:28:57 +08:00
Owner
Copy Link

This is just what Hydra is using and it is working fine for the purposes you mention. https://nixbld.m-labs.hk/build/173401/log

This is just what Hydra is using and it is working fine for the purposes you mention. https://nixbld.m-labs.hk/build/173401/log
👍 1
Dimitris commented 2025-02-03 02:14:39 +08:00
Author
Copy Link

@sb10q Thank you, Sebastien! I learned more about nix than I ever wanted to.

Edit from a few days later: See Automated firmware building for the Kasli SoC and sinara-firmware-compiler.

@ future visitors: Given the system description file on my computer and before running nix build, the python code further below outputs:

{'firmware.drv': '/nix/store/1x6ygq5g87csisxmjdnammwjs62h5i7d-firmware.drv',
 'gateware.drv': '/nix/store/2va05vm5id4q34xfkf0i5nmal9n4jim7-kasli_soc-satellite-gateware.drv',
 'sd.drv': '/nix/store/xarv6arh0hplqngkibcv2sxffxj4bhjn-kasli_soc-satellite-sd.drv',
 'firmware': '/nix/store/vc289nh603xa8qz1a81ax7v3w20wn90i-firmware',
 'gateware': '/nix/store/1sxaa5703vn2rqvpqsdyl0zs3665m21m-kasli_soc-satellite-gateware',
 'sd': '/nix/store/j597kma8q00wlbx9s2cli69infnx0q0m-kasli_soc-satellite-sd'}

After nix build completes, one can use nix log "/nix/store/1sxaa5703vn2rqvpqsdyl0zs3665m21m-kasli_soc-satellite-gateware" (for example) to get the original build log of the gateware (for example).

import os
import subprocess
import sys
import json

def from_json(json_file_path : str):
    json_file_path = os.path.realpath(json_file_path, strict=True)
    with open(json_file_path, "r") as file:
        data = file.read()
    return json.loads(data)

def shell_wrapper(
    command : str,
    strict  : bool, # if True, raise exception when subprocess.run().returncode != 0
    cwd     : str = None, # can be a relative path, command will be executed inside that directory
    output_to_terminal : bool = False, # output will not be captured by python and will only appear
                       # in the terminal, provided that you ran a python script from the terminal
):
    if cwd != None:
        cwd = os.path.realpath(cwd, strict=True)
    opt = {"capture_output" : True}
    if output_to_terminal:
        opt = {"stdout" : sys.stdout, "stderr" : sys.stderr}
    r = subprocess.run(
        args  = command,
        cwd   = cwd,
        shell = True,
        text  = True,
        **opt
    )
    if strict and r.returncode != 0:
        if cwd == None:
            cwd = os.path.realpath(".", strict=True)
        message = f"Error when running the following shell command with sinara.shell_wrapper() -> subprocess.run():" \
        + f"\ncd \"{cwd}\" && {command}" \
        + f"\nreturncode: {r.returncode}\nstdout: {r.stdout}\nstderr: {r.stderr}"
        raise Exception(message)
    return r

def nix_build_expression_kasli_soc(
    artiq_zynq_path : str, # path of top-level directory inside git repository, where `flake.nix` is
    json_file_path  : str, # path of system description file
):
    artiq_zynq_path = os.path.realpath(artiq_zynq_path, strict=True)
    json_file_path = os.path.realpath(json_file_path, strict=True)
    drtio_role = from_json(json_file_path)["drtio_role"]
    return f"""--impure --expr '
    let
        fl = builtins.getFlake "{artiq_zynq_path}";
    in
        (fl.makeArtiqZynqPackage {{
            target  = "kasli_soc";
            variant = "{drtio_role}";
            json    = "{json_file_path}";
        }}).kasli_soc-{drtio_role}-sd
'"""

def get_nix_derivations_and_outputs(
    artiq_zynq_path : str, # path of top-level directory inside git repository, where `flake.nix` is
    json_file_path  : str, # path of system description file
):
    nix_build_expression = nix_build_expression_kasli_soc(artiq_zynq_path, json_file_path)
    r = shell_wrapper(f"nix develop --command nix derivation show {nix_build_expression}",
                      cwd=artiq_zynq_path, strict=True)
    d = json.loads(r.stdout)
    def get_unique_key(keys, unique_pattern):
        matching_keys = [k for k in keys if unique_pattern in k]
        assert len(matching_keys) == 1
        return matching_keys[0]
    sd_drv = get_unique_key(d.keys(), "-sd.drv")
    gateware_drv = get_unique_key(d[sd_drv]["inputDrvs"].keys(), "-gateware.drv")
    firmware_drv = get_unique_key(d[sd_drv]["inputDrvs"].keys(), "-firmware.drv")
    def get_nix_store_output(nix_store_drv):
        d = json.loads(shell_wrapper(f'nix derivation show "{nix_store_drv}"', strict=True).stdout)
        return d[nix_store_drv]["outputs"]["out"]["path"]
    return {
        "firmware.drv" : firmware_drv,
        "gateware.drv" : gateware_drv,
        "sd.drv" : sd_drv,
        "firmware" : get_nix_store_output(firmware_drv),
        "gateware" : get_nix_store_output(gateware_drv),
        "sd" : get_nix_store_output(sd_drv),
    }

artiq_zynq_path = "/opt/m1/workspace/artiq-zynq-original-untouched"
json_file_path = "/opt/m1/workspace/sinara-firmware-compiler/examples/\
system_description_files/crate_mqva_dds2_kasli_soc_v1.1_artiq_zynq_master\
_788529ccdf5c293079ab7c5f97d7e59a52ebd50b_drtio_satellite_wrpll_True_sed_32\
_periph_urukul_6_addr_10.0.0.50/crate_mqva_dds2_kasli_soc_v1.1_artiq_zynq_master\
_788529ccdf5c293079ab7c5f97d7e59a52ebd50b_drtio_satellite_wrpll_True_sed_32_periph\
_urukul_6_addr_10.0.0.50.json"

get_nix_derivations_and_outputs(artiq_zynq_path, json_file_path)
@sb10q Thank you, Sebastien! I learned more about nix than I ever wanted to. **Edit from a few days later:** See [Automated firmware building for the Kasli SoC](https://forum.m-labs.hk/d/897-automated-firmware-building-for-the-kasli-soc) and [sinara-firmware-compiler](https://github.com/dimitsev/sinara-firmware-compiler). @ future visitors: Given the system description file on my computer and before running `nix build`, the python code further below outputs: ``` {'firmware.drv': '/nix/store/1x6ygq5g87csisxmjdnammwjs62h5i7d-firmware.drv', 'gateware.drv': '/nix/store/2va05vm5id4q34xfkf0i5nmal9n4jim7-kasli_soc-satellite-gateware.drv', 'sd.drv': '/nix/store/xarv6arh0hplqngkibcv2sxffxj4bhjn-kasli_soc-satellite-sd.drv', 'firmware': '/nix/store/vc289nh603xa8qz1a81ax7v3w20wn90i-firmware', 'gateware': '/nix/store/1sxaa5703vn2rqvpqsdyl0zs3665m21m-kasli_soc-satellite-gateware', 'sd': '/nix/store/j597kma8q00wlbx9s2cli69infnx0q0m-kasli_soc-satellite-sd'} ``` After `nix build` completes, one can use `nix log "/nix/store/1sxaa5703vn2rqvpqsdyl0zs3665m21m-kasli_soc-satellite-gateware"` (for example) to get the original build log of the gateware (for example). ```python import os import subprocess import sys import json def from_json(json_file_path : str): json_file_path = os.path.realpath(json_file_path, strict=True) with open(json_file_path, "r") as file: data = file.read() return json.loads(data) def shell_wrapper( command : str, strict : bool, # if True, raise exception when subprocess.run().returncode != 0 cwd : str = None, # can be a relative path, command will be executed inside that directory output_to_terminal : bool = False, # output will not be captured by python and will only appear # in the terminal, provided that you ran a python script from the terminal ): if cwd != None: cwd = os.path.realpath(cwd, strict=True) opt = {"capture_output" : True} if output_to_terminal: opt = {"stdout" : sys.stdout, "stderr" : sys.stderr} r = subprocess.run( args = command, cwd = cwd, shell = True, text = True, **opt ) if strict and r.returncode != 0: if cwd == None: cwd = os.path.realpath(".", strict=True) message = f"Error when running the following shell command with sinara.shell_wrapper() -> subprocess.run():" \ + f"\ncd \"{cwd}\" && {command}" \ + f"\nreturncode: {r.returncode}\nstdout: {r.stdout}\nstderr: {r.stderr}" raise Exception(message) return r def nix_build_expression_kasli_soc( artiq_zynq_path : str, # path of top-level directory inside git repository, where `flake.nix` is json_file_path : str, # path of system description file ): artiq_zynq_path = os.path.realpath(artiq_zynq_path, strict=True) json_file_path = os.path.realpath(json_file_path, strict=True) drtio_role = from_json(json_file_path)["drtio_role"] return f"""--impure --expr ' let fl = builtins.getFlake "{artiq_zynq_path}"; in (fl.makeArtiqZynqPackage {{ target = "kasli_soc"; variant = "{drtio_role}"; json = "{json_file_path}"; }}).kasli_soc-{drtio_role}-sd '""" def get_nix_derivations_and_outputs( artiq_zynq_path : str, # path of top-level directory inside git repository, where `flake.nix` is json_file_path : str, # path of system description file ): nix_build_expression = nix_build_expression_kasli_soc(artiq_zynq_path, json_file_path) r = shell_wrapper(f"nix develop --command nix derivation show {nix_build_expression}", cwd=artiq_zynq_path, strict=True) d = json.loads(r.stdout) def get_unique_key(keys, unique_pattern): matching_keys = [k for k in keys if unique_pattern in k] assert len(matching_keys) == 1 return matching_keys[0] sd_drv = get_unique_key(d.keys(), "-sd.drv") gateware_drv = get_unique_key(d[sd_drv]["inputDrvs"].keys(), "-gateware.drv") firmware_drv = get_unique_key(d[sd_drv]["inputDrvs"].keys(), "-firmware.drv") def get_nix_store_output(nix_store_drv): d = json.loads(shell_wrapper(f'nix derivation show "{nix_store_drv}"', strict=True).stdout) return d[nix_store_drv]["outputs"]["out"]["path"] return { "firmware.drv" : firmware_drv, "gateware.drv" : gateware_drv, "sd.drv" : sd_drv, "firmware" : get_nix_store_output(firmware_drv), "gateware" : get_nix_store_output(gateware_drv), "sd" : get_nix_store_output(sd_drv), } artiq_zynq_path = "/opt/m1/workspace/artiq-zynq-original-untouched" json_file_path = "/opt/m1/workspace/sinara-firmware-compiler/examples/\ system_description_files/crate_mqva_dds2_kasli_soc_v1.1_artiq_zynq_master\ _788529ccdf5c293079ab7c5f97d7e59a52ebd50b_drtio_satellite_wrpll_True_sed_32\ _periph_urukul_6_addr_10.0.0.50/crate_mqva_dds2_kasli_soc_v1.1_artiq_zynq_master\ _788529ccdf5c293079ab7c5f97d7e59a52ebd50b_drtio_satellite_wrpll_True_sed_32_periph\ _urukul_6_addr_10.0.0.50.json" get_nix_derivations_and_outputs(artiq_zynq_path, json_file_path) ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
priority:critical

   
priority:high

   
priority:low

   
priority:medium
No Label
priority:critical
priority:high
priority:low
priority:medium
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/artiq-zynq#355
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?