M-Labs
/
nix-scripts
9
0
Fork 7
Code Issues 6 Pull Requests 3 Releases Wiki Activity

artiq-full evaluation is a memory hog #25

New Issue
Closed
opened 2020-06-10 21:58:54 +08:00 by sb10q · 5 comments
sb10q commented 2020-06-10 21:58:54 +08:00
Owner

Slow and makes hydra-eval-job use more than 20GB and sometimes crash with OOM.

Slow and makes hydra-eval-job use more than 20GB and sometimes crash with OOM.
astro commented 2020-06-11 01:45:19 +08:00

Does 8f7308b improve the situation?

A migen patch may not be the most durable. I'm open for suggestions.

Does 8f7308b improve the situation? A migen patch may not be the most durable. I'm open for suggestions.
sb10q commented 2020-06-11 08:16:46 +08:00
Poster
Owner

It's the evaluation (i.e. hydra-eval-job process, as I mentioned) that is the problem. Not the Vivado runs, where you actually want the little parallelism it supports to reduce build time.

It's the *evaluation* (i.e. hydra-eval-job process, as I mentioned) that is the problem. Not the Vivado runs, where you actually want the little parallelism it supports to reduce build time.
astro commented 2020-06-18 05:21:32 +08:00

This improves memory usage and duration a lot for me: f1325dce29

This improves memory usage and duration a lot for me: f1325dce2941dbc127654e13b8acffdc151e5ffa
astro closed this issue 2020-06-18 05:21:56 +08:00
sb10q commented 2020-06-18 12:20:38 +08:00
Poster
Owner

Thanks, also much better on nixbld.m-labs.hk.
I'm curious what was happening - lots of copies of the entire nixpkgs object?

Thanks, also much better on nixbld.m-labs.hk. I'm curious what was happening - lots of copies of the entire nixpkgs object?
astro commented 2020-06-18 17:29:57 +08:00

For each of the board jobs, it was loading and evaluating the Vivado, Artiq and associated declarations over and over. I think it's putting import statements inside much-called functions (artiq-board) even when none of the function parameters are actually passed along.

For each of the board jobs, it was loading and evaluating the Vivado, Artiq and associated declarations over and over. I think it's putting `import` statements inside much-called functions (`artiq-board`) even when none of the function parameters are actually passed along.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/nix-scripts#25
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?