From: Eric Dumazet <edumazet@google.com>
To: "David S . Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Paolo Abeni <pabeni@redhat.com>
Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com,
Eric Dumazet <edumazet@google.com>
Subject: [PATCH net-next 7/8] net: rps: add rps_input_queue_head_add() helper
Date: Thu, 28 Mar 2024 17:03:08 +0000 [thread overview]
Message-ID: <20240328170309.2172584-8-edumazet@google.com> (raw)
In-Reply-To: <20240328170309.2172584-1-edumazet@google.com>
process_backlog() can batch increments of sd->input_queue_head,
saving some memory bandwidth.
Also add READ_ONCE()/WRITE_ONCE() annotations around sd->input_queue_head
accesses.
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/net/rps.h | 9 +++++++--
net/core/dev.c | 13 ++++++++-----
2 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/include/net/rps.h b/include/net/rps.h
index c13f829b8556fda63e76544c332f2c089f0d6ea4..135427bc6fcd29b9dad92a671c9a9f4efc975dec 100644
--- a/include/net/rps.h
+++ b/include/net/rps.h
@@ -51,13 +51,18 @@ static inline void rps_input_queue_tail_save(u32 *dest, u32 tail)
#endif
}
-static inline void rps_input_queue_head_incr(struct softnet_data *sd)
+static inline void rps_input_queue_head_add(struct softnet_data *sd, int val)
{
#ifdef CONFIG_RPS
- sd->input_queue_head++;
+ WRITE_ONCE(sd->input_queue_head, sd->input_queue_head + val);
#endif
}
+static inline void rps_input_queue_head_incr(struct softnet_data *sd)
+{
+ rps_input_queue_head_add(sd, 1);
+}
+
/*
* The rps_dev_flow_table structure contains a table of flow mappings.
*/
diff --git a/net/core/dev.c b/net/core/dev.c
index 1fe7c6b10793d45a03461ee581d240d2442f9e17..59e7fc30e8f03880340bfbeda0fa9e9ac757a168 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4518,7 +4518,7 @@ set_rps_cpu(struct net_device *dev, struct sk_buff *skb,
out:
#endif
rflow->last_qtail =
- per_cpu(softnet_data, next_cpu).input_queue_head;
+ READ_ONCE(per_cpu(softnet_data, next_cpu).input_queue_head);
}
rflow->cpu = next_cpu;
@@ -4600,7 +4600,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb,
*/
if (unlikely(tcpu != next_cpu) &&
(tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
- ((int)(per_cpu(softnet_data, tcpu).input_queue_head -
+ ((int)(READ_ONCE(per_cpu(softnet_data, tcpu).input_queue_head) -
READ_ONCE(rflow->last_qtail))) >= 0)) {
tcpu = next_cpu;
rflow = set_rps_cpu(dev, skb, rflow, next_cpu);
@@ -4655,7 +4655,7 @@ bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index,
rflow = &flow_table->flows[flow_id];
cpu = READ_ONCE(rflow->cpu);
if (rflow->filter == filter_id && cpu < nr_cpu_ids &&
- ((int)(per_cpu(softnet_data, cpu).input_queue_head -
+ ((int)(READ_ONCE(per_cpu(softnet_data, cpu).input_queue_head) -
READ_ONCE(rflow->last_qtail)) <
(int)(10 * flow_table->mask)))
expire = false;
@@ -6035,9 +6035,10 @@ static int process_backlog(struct napi_struct *napi, int quota)
rcu_read_lock();
__netif_receive_skb(skb);
rcu_read_unlock();
- rps_input_queue_head_incr(sd);
- if (++work >= quota)
+ if (++work >= quota) {
+ rps_input_queue_head_add(sd, work);
return work;
+ }
}
@@ -6060,6 +6061,8 @@ static int process_backlog(struct napi_struct *napi, int quota)
backlog_unlock_irq_enable(sd);
}
+ if (work)
+ rps_input_queue_head_add(sd, work);
return work;
}
--
2.44.0.478.gd926399ef9-goog
next prev parent reply other threads:[~2024-03-28 17:03 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-28 17:03 [PATCH net-next 0/8] net: rps: misc changes Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 1/8] net: move kick_defer_list_purge() to net/core/dev.h Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 2/8] net: move dev_xmit_recursion() helpers " Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 3/8] net: enqueue_to_backlog() change vs not running device Eric Dumazet
2024-03-29 3:21 ` Jason Xing
2024-03-29 6:31 ` Eric Dumazet
2024-03-29 8:44 ` Jason Xing
2024-03-28 17:03 ` [PATCH net-next 4/8] net: make softnet_data.dropped an atomic_t Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 5/8] net: enqueue_to_backlog() cleanup Eric Dumazet
2024-03-28 17:03 ` [PATCH net-next 6/8] net: rps: change input_queue_tail_incr_save() Eric Dumazet
2024-03-29 12:07 ` kernel test robot
2024-03-29 12:29 ` kernel test robot
2024-03-28 17:03 ` Eric Dumazet [this message]
2024-03-28 17:03 ` [PATCH net-next 8/8] net: rps: move received_rps field to a better location Eric Dumazet
2024-03-29 5:45 ` [PATCH net-next 0/8] net: rps: misc changes Jakub Kicinski
2024-03-29 6:44 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240328170309.2172584-8-edumazet@google.com \
--to=edumazet@google.com \
--cc=davem@davemloft.net \
--cc=eric.dumazet@gmail.com \
--cc=kuba@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.