From: Jesper Dangaard Brouer <brouer@redhat.com>
To: "Federico Parola" <federico.parola@polito.it>,
"Björn Töpel" <bjorn.topel@gmail.com>
Cc: brouer@redhat.com, xdp-newbies@vger.kernel.org
Subject: Re: XDP redirect max rate on Intel XL710
Date: Mon, 12 Apr 2021 17:02:56 +0200 [thread overview]
Message-ID: <20210412170256.4a62fea1@carbon> (raw)
In-Reply-To: <8f897fd2-5ac1-a86f-2db1-6c4a45e144ad@polito.it>
On Mon, 12 Apr 2021 15:48:10 +0200
Federico Parola <federico.parola@polito.it> wrote:
> Hello everybody,
> when I redirect packets between two ports of an Intel XL710 40Gb card
> (e.g. with the xdp_redirect_map example fo the kernel) I can not achieve
> throughputs higher than ~31 Mpps.
> This NIC is not able to reach the theoretical ~60 Mpps with small
1/(84*8)*40*10^9 = 59,523,810 pps
> packets ([1] p. 23) but with DPDK I'm able to achieve ~40 Mpps with the
> testpmd application and 2 cores.
Lets calculate the processing or arrival rate per packet (or between
packets arriving).
31 Mpps = 32.25 nanosec (1/31*(10^9/10^6))
40 Mpps = 25 nanosec
Difference = 7.26 nanosec.
Thus, you only have to optimize XDP redirect with 7 nanosec to catch up
with DPDK. Assuming a 4GHz CPU (7*4) you get 28 cycles more with DPDK
that I'm sure your application really needs and use with carefully ;-)
> In XDP when dropping packets I achieve more or less the same throughput
> with 3 cores, but I'm not able to exceed 31 Mpps when forwarding, no
> matter how many cores I use.
> I tried tuning the size of the RX/TX rings and the DDIO occupancy but
> with no success. I can scale with the number of cores more linearly but
> as soon as I reach the 31 threshold cores usage decreases and throughput
> remains the same.
If the CPU usage start do decrease (e.g. CPU gets idle cycles) then the
limitation is likely elsewhere, likely in the PCIe layer. Or a limit
in the NIC hardware, but given DPDK is 7 nanosec faster then it might
not be the NIC hardware, but it could as DPDK could have disabled some
flow-dissector HW feature. It can also be that XDP/kernel is doing more
PCIe operations than DPDK.
> I don't know if this is just related to my setup or my specific NIC (I
> exprimented on kernels 5.11 and 5.9), does anybody know what the reason
> could be?
Bjørn have recently optimized the kernel XDP redirect code path with
around 3-4 nanosec. You can try with net-next kernel (Bjørn correct me
if this isn't true).
In your userspace AF_XDP application also make sure you don't use these
28 cycles / 7 nanosec for something that the DPDK testpmd doesn't.
> [1]
> https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwicv__o6_jvAhWUwQIHHQtpBu8QFjABegQIBhAD&url=https%3A%2F%2Fwww.intel.com%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Fspecification-updates%2Fxl710-10-40-controller-spec-update.pdf&usg=AOvVaw27anCuRwRX25h8Dg3epgoy
>
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
prev parent reply other threads:[~2021-04-12 15:03 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-12 13:48 XDP redirect max rate on Intel XL710 Federico Parola
2021-04-12 15:02 ` Jesper Dangaard Brouer [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210412170256.4a62fea1@carbon \
--to=brouer@redhat.com \
--cc=bjorn.topel@gmail.com \
--cc=federico.parola@polito.it \
--cc=xdp-newbies@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).