From: Jaroslav Pulchart <jaroslav.pulchart@gooddata.com>
To: Jakub Kicinski <kuba@kernel.org>
Cc: Jason Wang <jasowang@redhat.com>, Igor Raits <igor@gooddata.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
kvm@vger.kernel.org, virtualization@lists.linux.dev,
netdev@vger.kernel.org, Stefano Garzarella <sgarzare@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>
Subject: Re: REGRESSION: RIP: 0010:skb_release_data+0xb8/0x1e0 in vhost/tun
Date: Mon, 8 Apr 2024 08:35:30 +0200 [thread overview]
Message-ID: <CAK8fFZ74tj=u5HKfpFue1ejy_9V3xWdf-ekC0gLt8BmJs7Y5ZQ@mail.gmail.com> (raw)
In-Reply-To: <CAK8fFZ5LHFMPAOFCKu-vr7JQJHKo9jshrgvCCP50d596nFiXUQ@mail.gmail.com>
čt 4. 4. 2024 v 20:17 odesílatel Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> napsal:
>
> čt 4. 4. 2024 v 15:37 odesílatel Jakub Kicinski <kuba@kernel.org> napsal:
> >
> > On Thu, 4 Apr 2024 07:42:45 +0200 Jaroslav Pulchart wrote:
> > > We do not have much progress
> >
> > Random thought - do you have KFENCE enabled?
> > It's sufficiently low overhead to run in production and maybe it could
> > help catch the bug? You also hit some inexplicable bug in the Intel
> > driver, IIRC, there may be something odd going on.. (it's not all
> > happening on a single machine, right?)
>
> We have KFENCE enabled.
>
> Issue was observed at multiple servers. It is not a problem to reproduce it
> everywhere where we deploy Loki service. The trigger is: I click
> once/twice "run query" (LogQL) button by Grafana UI. the Loki is
> starting to load data from the minio cluster at a speed of ~2GB/s and
> almost immediately it crashes.
>
> The Intel ICE driver is in my suspicion as well, it will not be for
> the first time when we are hitting some bugs there. I will try one
> testing server where we have different NIC vendor later.
I run the setup on a server with a different network card than E810, I
used BCM57414 NetXtreme-E + driver bnxt_en. The issue is not
reproducible there. So it looks to be connected with Intel's ice
driver for E810 network card and introduced in 6.3.
prev parent reply other threads:[~2024-04-08 6:35 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-19 9:00 REGRESSION: RIP: 0010:skb_release_data+0xb8/0x1e0 in vhost/tun Igor Raits
2024-03-19 12:29 ` Igor Raits
2024-03-19 13:08 ` Willem de Bruijn
2024-03-19 13:22 ` Igor Raits
2024-03-19 13:41 ` Willem de Bruijn
2024-03-19 13:52 ` Igor Raits
2024-03-19 13:12 ` Stefan Hajnoczi
2024-03-19 13:15 ` Igor Raits
2024-03-20 9:33 ` Jason Wang
2024-03-21 9:44 ` Igor Raits
2024-03-22 5:30 ` Jason Wang
2024-03-22 8:39 ` Igor Raits
2024-03-22 11:19 ` Igor Raits
2024-03-22 11:29 ` Stefano Garzarella
2024-03-25 8:44 ` Igor Raits
2024-03-26 4:46 ` Jason Wang
2024-03-26 13:25 ` Jaroslav Pulchart
2024-03-27 3:11 ` Jason Wang
2024-03-27 7:08 ` Jaroslav Pulchart
2024-04-04 5:42 ` Jaroslav Pulchart
2024-04-04 13:37 ` Jakub Kicinski
2024-04-04 18:17 ` Jaroslav Pulchart
2024-04-08 6:35 ` Jaroslav Pulchart [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAK8fFZ74tj=u5HKfpFue1ejy_9V3xWdf-ekC0gLt8BmJs7Y5ZQ@mail.gmail.com' \
--to=jaroslav.pulchart@gooddata.com \
--cc=igor@gooddata.com \
--cc=jasowang@redhat.com \
--cc=kuba@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=sgarzare@redhat.com \
--cc=stefanha@redhat.com \
--cc=virtualization@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).