* How to debug XDP_REDIRECT errors
@ 2019-10-20 21:07 Zvi Effron
2019-10-21 4:04 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 2+ messages in thread
From: Zvi Effron @ 2019-10-20 21:07 UTC (permalink / raw
To: Xdp
Hello,
I'm trying to get some additional performance out of XDP on the i40e
driver (under kernel 5.0.7, Fedora 28) by adjusting the queues to be
one per desired core. I have several cores isolated for network
traffic, so I'm adjusting the queues to be one per isolated core (with
IRQ adjustments to match) as opposed to the default of one per online
core. I've noticed that when I adjust the number of queues down, I see
packet loss proportional to the number of queues missing. For example,
if I drop the queues from the default of 36 down to 34, I see ~6%
packet loss.
Using the trace_pipe, I've managed to confirm that my program is
successfully calling bpf_redirect_map() for all of the packet (no
losst). I found the xdp_monitor tutorial[1] and ran it, and it shows
the loss showing up as XDP_REDIRECT errors (which I believe means the
raw_tracepoint xdp:xdp_redirect_map_err).
At this point, I'm a bit stuck on how to proceed further to determine
what is causing the redirect error and where the packet is being
dropped.
Any help in pointing me in the right direction for debugging
techniques would be greatly appreciated.
Thanks!
--Zvi
[1]: https://github.com/xdp-project/xdp-tutorial/blob/master/tracing02-xdp-monitor
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: How to debug XDP_REDIRECT errors
2019-10-20 21:07 How to debug XDP_REDIRECT errors Zvi Effron
@ 2019-10-21 4:04 ` Jesper Dangaard Brouer
0 siblings, 0 replies; 2+ messages in thread
From: Jesper Dangaard Brouer @ 2019-10-21 4:04 UTC (permalink / raw
To: Zvi Effron; +Cc: Xdp, brouer
On Sun, 20 Oct 2019 16:07:13 -0500
Zvi Effron <zeffron@riotgames.com> wrote:
> Hello,
>
> I'm trying to get some additional performance out of XDP on the i40e
> driver (under kernel 5.0.7, Fedora 28) by adjusting the queues to be
> one per desired core. I have several cores isolated for network
> traffic, so I'm adjusting the queues to be one per isolated core (with
> IRQ adjustments to match) as opposed to the default of one per online
> core. I've noticed that when I adjust the number of queues down, I see
> packet loss proportional to the number of queues missing. For example,
> if I drop the queues from the default of 36 down to 34, I see ~6%
> packet loss.
>
> Using the trace_pipe, I've managed to confirm that my program is
> successfully calling bpf_redirect_map() for all of the packet (no
> losst). I found the xdp_monitor tutorial[1] and ran it, and it shows
> the loss showing up as XDP_REDIRECT errors (which I believe means the
> raw_tracepoint xdp:xdp_redirect_map_err).
I usually use standard perf-record:
$ perf record -e xdp:xdp_redirect_map_err sleep 1
And then look at detailed results with:
$ perf script
As this is a standard tracepoint, you can lookup docs for howto use
tracepoints. To debug closer you likely want to extract other info,
and maybe aggregate. I will recommend looking closer at bpftrace
tool[2].
[2] https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
> At this point, I'm a bit stuck on how to proceed further to determine
> what is causing the redirect error and where the packet is being
> dropped.
>
> Any help in pointing me in the right direction for debugging
> techniques would be greatly appreciated.
>
> Thanks!
> --Zvi
>
> [1]: https://github.com/xdp-project/xdp-tutorial/blob/master/tracing02-xdp-monitor
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2019-10-21 4:04 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-10-20 21:07 How to debug XDP_REDIRECT errors Zvi Effron
2019-10-21 4:04 ` Jesper Dangaard Brouer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.