From: Magnus Karlsson <magnus.karlsson@gmail.com>
To: Joshua Coutinho <souichirosano@gmail.com>
Cc: xdp-newbies@vger.kernel.org
Subject: Re: XSK tx on loopback drops packets.
Date: Mon, 11 Dec 2023 11:26:09 +0100 [thread overview]
Message-ID: <CAJ8uoz1fLNLS7Lyt3o6f+0+L5Yh1QEha-buji2sRVGNE_LzNCg@mail.gmail.com> (raw)
In-Reply-To: <CAOafi9EZ8L150jw_pC_DZj4_VO6r-BxCMDZzsC_X-V06o73org@mail.gmail.com>
On Mon, 4 Dec 2023 at 16:42, Joshua Coutinho <souichirosano@gmail.com> wrote:
>
> Hi All,
>
> I'm trying to transmit udp packets via an xsk socket. (XDP receives
> work just fine). The xdp program is irrelevant/unused, I'm trying to
> simply leverage the xsk socket write.
>
> (Kernel: Linux fedora 6.5.12-300.fc39.x86_64)
> (OS: "Fedora Linux 39 (Workstation Edition)"
>
> I want a minimal working example of sending packets via an XSK socket
> over loopback in user space land. I want to be able to fill in the
> required memory regions and trigger the kernel to send the packet and
> capture the sent packets on the other side with 'nc -lu 127.0.0.1
> <port>' This seems to happen partly successfully but on the ingress
> part of the loopback it is dropped somewhere after reaching the kernel
> function ip_rcv and then nf_hook_slow. Specifically, I simply want to
> write a packet into a UMEM region, fill in the TX descriptor and then
> submit that descriptor like so.
>
> u32 txIdx = -1;
> const u32 txSlotsRecvd = xsk_ring_prod__reserve(&qs.txQ, 1, &txIdx);
> u32 addr = umem.txState.nextSlot();
>
> xdp_desc* txDescr = xsk_ring_prod__tx_desc(&qs.txQ, txIdx);
> txDescr->addr = addr;
> txDescr->len = sizeof(OrderFrame);
> txDescr->options = 0;
>
> u8* outputBuf = umem.buffer + addr;
>
> TimeNs submitTime = currentTimeNs();
> OrderFrame& frame = *reinterpret_cast<OrderFrame *>(outputBuf);
>
> std::array<u8, ETH_ALEN> sourceMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
> std::array<u8, ETH_ALEN> destMac = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0};
> std::copy(sourceMac.begin(), sourceMac.end(), frame.eth.h_source);
> std::copy(destMac.begin(), destMac.end(), frame.eth.h_dest);
>
> frame.eth.h_proto = htons(ETH_P_IP);
> frame.ip.ihl = 5;
> frame.ip.version = 4;
> frame.ip.tos = 0;
> frame.ip.tot_len = htons(sizeof(OrderFrame) - sizeof(ethhdr));
> frame.ip.id = orderId;
>
> frame.ip.frag_off = 0x0;
> frame.ip.ttl = static_cast<u8>(255);
> frame.ip.protocol = 17;
> frame.ip.check = 0;
> constexpr u8 sourceIPBytes[4] = {127, 0, 0, 1};
> constexpr u8 destIPBytes[4] = {127, 0, 0, 1};
> const u32 sourceIP = *reinterpret_cast<const u32*>(sourceIPBytes);
> const u32 destIP = *reinterpret_cast<const u32*>(destIPBytes);
> frame.ip.saddr = sourceIP;
> frame.ip.daddr = destIP;
> const u8* dataptr = reinterpret_cast<u8 *>(&frame.ip);
> const u16 kernelcsum = ip_fast_csum(dataptr, frame.ip.ihl);
> frame.ip.check = kernelcsum;
> constexpr int udpPacketSz = sizeof(OrderFrame) -
> sizeof(ethhdr) - sizeof(iphdr);
> frame.udp.len = htons(udpPacketSz);
> frame.udp.check = 0;
> frame.udp.dest = htons(OE_PORT);
> frame.udp.source = htons(1234);
> ... // application packet logic
> frame.udp.check = 0;
>
> xsk_ring_prod__submit(&qs.txQ, 1);
> if (xsk_ring_prod__needs_wakeup(&qs.txQ)) {
> assert((socket.cfg.bind_flags & XDP_COPY) != 0);
> const ssize_t ret = sendto(socket.xskFD, nullptr, 0,
> MSG_DONTWAIT, nullptr, 0);
> }
>
> This is a relevant stacktrace from the kernel indicating the path of
> my packet after the above sendto is called.
>
> __netif_receive_skb_one_core+0x3c/0xa0
> process_backlog+0x85/0x120
> __napi_poll+0x28/0x1b0
> net_rx_action+0x2a4/0x380
> __do_softirq+0xd1/0x2c8
> do_softirq.part.0+0x3d/0x60
> __local_bh_enable_ip+0x68/0x70
> __dev_direct_xmit+0x152/0x210
> __xsk_generic_xmit+0x3e4/0x710
> xsk_sendmsg+0x12f/0x1f0
> __sys_sendto+0x1d6/0x1e0
> __x64_sys_sendto+0x24/0x30
> do_syscall_64+0x5d/0x90
> entry_SYSCALL_64_after_hwframe+0x6e/0xd8
>
> My socket is bound to localhost using xdpgeneric. I see the
> transmitted packets in tcpdump and via bpftrace I see that ip_rcv is
> invoked for the packets. nf_hook_slow is also invoked with 1 active
> prerouting hook. On kfree_skb I see the reason for the drop is 'reason
> not specified'. Examining the packet in tcpdump I see no errors with
> the checksums or packet lengths and ports. Listeners for the
> corresponding udp ports never receieve the packets.This is how I
> create my socket
>
> cfg.rx_size = XSKQueues::NUM_READ_DESC;
> cfg.tx_size = XSKQueues::NUM_WRITE_DESC;
> cfg.libxdp_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD;
> cfg.xdp_flags = XDP_FLAGS_SKB_MODE;
> cfg.bind_flags = XDP_USE_NEED_WAKEUP | XDP_COPY;
>
> if (xsk_socket__create(&socket, iface.c_str(), QUEUE, umem.umem,
> &qs.rxQ, &qs.txQ, &cfg)) {
> perror("XSK: ");
> exit(EXIT_FAILURE);
> }
>
> What could be the issue?
It is the right forum. Sorry, just forgot to answer your mail.
What happens if you instead of opening an AF_INET socket, open an
AF_PACKET socket (or an AF_XDP socket for that matter) at the
receiving end? Do you get the packet? If so, might it just be that
your packet is wrong and it is being discarded by the stack? Use
tcpdump or wireshark at the receiving end and see what it says. That
is if you get a packet out of the AF_PACKET socket.
prev parent reply other threads:[~2023-12-11 10:26 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-04 15:41 XSK tx on loopback drops packets Joshua Coutinho
2023-12-11 10:12 ` Joshua Coutinho
2023-12-11 10:26 ` Magnus Karlsson [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJ8uoz1fLNLS7Lyt3o6f+0+L5Yh1QEha-buji2sRVGNE_LzNCg@mail.gmail.com \
--to=magnus.karlsson@gmail.com \
--cc=souichirosano@gmail.com \
--cc=xdp-newbies@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).