From: Eric Dumazet <edumazet@google.com>
To: "David S . Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com,
Eric Dumazet <edumazet@google.com>
Subject: [PATCH net-next 4/8] net: make softnet_data.dropped an atomic_t
Date: Thu, 28 Mar 2024 17:03:05 +0000 [thread overview]
Message-ID: <20240328170309.2172584-5-edumazet@google.com> (raw)
In-Reply-To: <20240328170309.2172584-1-edumazet@google.com>
If under extreme cpu backlog pressure enqueue_to_backlog() has
to drop a packet, it could do this without dirtying a cache line
and potentially slowing down the target cpu.
Move sd->dropped into a separate cache line, and make it atomic.
In non pressure mode, this field is not touched, no need to consume
valuable space in a hot cache line.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/linux/netdevice.h | 3 ++-
net/core/dev.c | 13 +++++++++----
net/core/net-procfs.c | 3 ++-
3 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 70775021cc269e0983f538619115237b0067d408..1c31cd2691d32064613836141fbdeeebc831b21f 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -3236,10 +3236,11 @@ struct softnet_data {
unsigned int input_queue_tail;
#endif
unsigned int received_rps;
- unsigned int dropped;
struct sk_buff_head input_pkt_queue;
struct napi_struct backlog;
+ atomic_t dropped ____cacheline_aligned_in_smp;
+
/* Another possibly contended cache line */
spinlock_t defer_lock ____cacheline_aligned_in_smp;
int defer_count;
diff --git a/net/core/dev.c b/net/core/dev.c
index af7a34b0a7d6683c6ffb21dd3388ed678473d95e..be2392ba56bc57bed456e2748b332d4971c83a4e 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4790,17 +4790,22 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
struct softnet_data *sd;
unsigned long flags;
unsigned int qlen;
+ int max_backlog;
reason = SKB_DROP_REASON_DEV_READY;
if (!netif_running(skb->dev))
goto bad_dev;
+ reason = SKB_DROP_REASON_CPU_BACKLOG;
sd = &per_cpu(softnet_data, cpu);
+ qlen = skb_queue_len_lockless(&sd->input_pkt_queue);
+ max_backlog = READ_ONCE(net_hotdata.max_backlog);
+ if (unlikely(qlen > max_backlog))
+ goto cpu_backlog_drop;
backlog_lock_irq_save(sd, &flags);
qlen = skb_queue_len(&sd->input_pkt_queue);
- if (qlen <= READ_ONCE(net_hotdata.max_backlog) &&
- !skb_flow_limit(skb, qlen)) {
+ if (qlen <= max_backlog && !skb_flow_limit(skb, qlen)) {
if (qlen) {
enqueue:
__skb_queue_tail(&sd->input_pkt_queue, skb);
@@ -4816,11 +4821,11 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu,
napi_schedule_rps(sd);
goto enqueue;
}
- reason = SKB_DROP_REASON_CPU_BACKLOG;
- sd->dropped++;
backlog_unlock_irq_restore(sd, &flags);
+cpu_backlog_drop:
+ atomic_inc(&sd->dropped);
bad_dev:
dev_core_stats_rx_dropped_inc(skb->dev);
kfree_skb_reason(skb, reason);
diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c
index a97eceb84e61ec347ad132ff0f22c8ce82f12e90..fa6d3969734a6ec154c3444d1b25ee93edfc5588 100644
--- a/net/core/net-procfs.c
+++ b/net/core/net-procfs.c
@@ -144,7 +144,8 @@ static int softnet_seq_show(struct seq_file *seq, void *v)
seq_printf(seq,
"%08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x %08x "
"%08x %08x\n",
- sd->processed, sd->dropped, sd->time_squeeze, 0,
+ sd->processed, atomic_read(&sd->dropped),
+ sd->time_squeeze, 0,
0, 0, 0, 0, /* was fastroute */
0, /* was cpu_collision */
sd->received_rps, flow_limit_count,
--
2.44.0.478.gd926399ef9-goog
next prev parent reply other threads:[~2024-03-28 17:03 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-28 17:03 [PATCH net-next 0/8] net: rps: misc changes Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 1/8] net: move kick_defer_list_purge() to net/core/dev.h Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 2/8] net: move dev_xmit_recursion() helpers " Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 3/8] net: enqueue_to_backlog() change vs not running device Eric Dumazet
2024-03-29 3:21 ` Jason Xing
2024-03-29 6:31 ` Eric Dumazet
2024-03-29 8:44 ` Jason Xing
2024-03-28 17:03 ` Eric Dumazet [this message]
2024-03-28 17:03 ` [PATCH net-next 5/8] net: enqueue_to_backlog() cleanup Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 6/8] net: rps: change input_queue_tail_incr_save() Eric Dumazet
2024-03-29 12:07 ` kernel test robot
2024-03-29 12:29 ` kernel test robot
2024-03-28 17:03 ` [PATCH net-next 7/8] net: rps: add rps_input_queue_head_add() helper Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 8/8] net: rps: move received_rps field to a better location Eric Dumazet
2024-03-29 5:45 ` [PATCH net-next 0/8] net: rps: misc changes Jakub Kicinski
2024-03-29 6:44 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240328170309.2172584-5-edumazet@google.com \
--to=edumazet@google.com \
--cc=davem@davemloft.net \
--cc=eric.dumazet@gmail.com \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.