M-Labs
/
zynq-rs
10
0
Fork 12
Code Issues 6 Pull Requests 1 Releases Wiki Activity

poor Ethernet performance #30

New Issue
Closed
opened 2020-05-25 17:44:41 +08:00 by sb10q · 8 comments
sb10q commented 2020-05-25 17:44:41 +08:00
Owner

IIRC just a couple MB/s from @astro's tests.

IIRC just a couple MB/s from @astro's tests.
astro commented 2020-05-29 03:35:57 +08:00

2 MB/s

2 MB/s
astro commented 2020-06-11 08:09:20 +08:00

First approach: set the DDR pages bufferable (writeback) in the MMU. Unfortunately, the eth descriptors (not the buffers) need to be placed into non-bufferable pages as they're smaller than a cacheline. That means they cannot be invalidated from L1 individually.

First approach: set the DDR pages bufferable (writeback) in the MMU. Unfortunately, the eth descriptors (not the buffers) need to be placed into non-bufferable pages as they're smaller than a cacheline. That means they cannot be invalidated from L1 individually.
sb10q commented 2020-06-11 09:02:21 +08:00
Poster
Owner

What is the Xilinx code doing?

What is the Xilinx code doing?
astro commented 2020-06-12 06:10:53 +08:00

The Xilinx embeddedsw doesn't contain cache/barrier instructions. It probably runs w/o bufferable pages.

The Linux driver uses only barriers which I am so far unable to adopt towards reliable behavior.

One non-bufferable MMU page just for the descriptors seems to be a promising solution...

The Xilinx embeddedsw doesn't contain cache/barrier instructions. It probably runs w/o bufferable pages. The Linux driver uses only barriers which I am so far unable to adopt towards reliable behavior. One non-bufferable MMU page just for the descriptors seems to be a promising solution...
astro commented 2020-06-18 07:28:51 +08:00

Enabling MMU bufferable pages has doubled throughput.

Enabling MMU bufferable pages has doubled throughput.
pca006132 commented 2020-08-13 12:41:59 +08:00

M-Labs/artiq-zynq#55

The L2 cache would completely break the ethernet driver. That is probably related to cache invalidation issue, as a lot of the polling returns buffer with 0 length. There are several seconds of delay.

By adding dcci_slice for rx buffer, and return Err(None) when len is 0 for recv_next function, the delay is reduced, but there is still occasional latency spike of up to ~500ms which would not occur with L2 cache turned off.

Also, there is still problem with RTIO analyzer buffer transmission, the stream is not closed even if I flush the data at the end of the transmission. The stream would close if I open another connection though.

https://git.m-labs.hk/M-Labs/artiq-zynq/issues/55 The L2 cache would completely break the ethernet driver. That is probably related to cache invalidation issue, as a lot of the polling returns buffer with 0 length. There are several seconds of delay. By adding `dcci_slice` for rx buffer, and return `Err(None)` when len is 0 for `recv_next` function, the delay is reduced, but there is still occasional latency spike of up to ~500ms which would not occur with L2 cache turned off. Also, there is still problem with RTIO analyzer buffer transmission, the stream is not closed even if I flush the data at the end of the transmission. The stream would close if I open another connection though.
pca006132 commented 2020-08-14 15:48:50 +08:00

The L2 cache problem is fixed by modifying the MMU setting for uncached slice. However, the rx speed is still pretty slow comparing to tx speed. There are also a few changes, like modified the cache flush for rx and prevented duplicates in the waker queue.

Branch:

> python -m unittest test_performance -v
test_kernel_overhead (test_performance.KernelOverheadTest) ... 0.017314874350558965 s
ok
test_device_to_host (test_performance.TransferTest) ... 31.725000159885067 MiB/s
ok
test_device_to_host_array (test_performance.TransferTest) ... 5.5078113984377195 MiB/s
ok
test_host_to_device (test_performance.TransferTest) ... 3.6111348770332103 MiB/s
ok
The L2 cache problem is fixed by modifying the MMU setting for uncached slice. However, the rx speed is still pretty slow comparing to tx speed. There are also a few changes, like modified the cache flush for rx and prevented duplicates in the waker queue. Branch: * https://git.m-labs.hk/pca006132/zynq-rs/src/branch/l2-cache (note that L2 cache is not enabled by default) * https://git.m-labs.hk/pca006132/artiq-zynq/src/branch/l2-cache ``` > python -m unittest test_performance -v test_kernel_overhead (test_performance.KernelOverheadTest) ... 0.017314874350558965 s ok test_device_to_host (test_performance.TransferTest) ... 31.725000159885067 MiB/s ok test_device_to_host_array (test_performance.TransferTest) ... 5.5078113984377195 MiB/s ok test_host_to_device (test_performance.TransferTest) ... 3.6111348770332103 MiB/s ok ```
pca006132 commented 2020-08-21 14:28:45 +08:00

We get good performance by changing memcpy implementation and optimization level to 's'/2. Those would be done in artiq-zynq, closing now

We get good performance by changing memcpy implementation and optimization level to 's'/2. Those would be done in artiq-zynq, closing now
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
esavkin/reboot_support
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/zynq-rs#30
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?