From: Soheil Hassas Yeganeh <soheil.kdev@gmail.com>
To: torvalds@linux-foundation.org, viro@zeniv.linux.org.uk,
linux-fsdevel@vger.kernel.org
Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org,
dave@stgolabs.net, edumazet@google.com, willemb@google.com,
khazhy@google.com, guantaol@google.com,
Soheil Hassas Yeganeh <soheil@google.com>
Subject: [PATCH 5/8] epoll: simplify and optimize busy loop logic
Date: Fri, 6 Nov 2020 18:16:32 -0500 [thread overview]
Message-ID: <20201106231635.3528496-6-soheil.kdev@gmail.com> (raw)
In-Reply-To: <20201106231635.3528496-1-soheil.kdev@gmail.com>
From: Soheil Hassas Yeganeh <soheil@google.com>
ep_events_available() is called multiple times around the busy loop
logic, even though the logic is generally not used.
ep_reset_busy_poll_napi_id() is similarly always called, even when
busy loop is not used.
Eliminate ep_reset_busy_poll_napi_id() and inline it inside
ep_busy_loop(). Make ep_busy_loop() return whether there are any
events available after the busy loop. This will eliminate unnecessary
loads and branches, and simplifies the loop.
Signed-off-by: Soheil Hassas Yeganeh <soheil@google.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Khazhismel Kumykov <khazhy@google.com>
---
fs/eventpoll.c | 42 +++++++++++++++++++-----------------------
1 file changed, 19 insertions(+), 23 deletions(-)
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 5226b0cb1098..28c1d341d2e6 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -391,19 +391,26 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time)
* busy loop will return if need_resched or ep_events_available.
*
* we must do our busy polling with irqs enabled
+ *
+ * Returns whether new events are available after a successful busy loop.
*/
-static void ep_busy_loop(struct eventpoll *ep, int nonblock)
+static bool ep_busy_loop(struct eventpoll *ep, int nonblock)
{
unsigned int napi_id = READ_ONCE(ep->napi_id);
- if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on())
+ if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) {
napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep);
-}
-
-static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep)
-{
- if (ep->napi_id)
+ if (ep_events_available(ep))
+ return true;
+ /*
+ * Busy poll timed out. Drop NAPI ID for now, we can add
+ * it back in when we have moved a socket with a valid NAPI
+ * ID onto the ready list.
+ */
ep->napi_id = 0;
+ return false;
+ }
+ return false;
}
/*
@@ -444,12 +451,9 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
#else
-static inline void ep_busy_loop(struct eventpoll *ep, int nonblock)
-{
-}
-
-static inline void ep_reset_busy_poll_napi_id(struct eventpoll *ep)
+static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock)
{
+ return false;
}
static inline void ep_set_busy_poll_napi_id(struct epitem *epi)
@@ -1857,21 +1861,13 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
}
fetch_events:
-
- if (!ep_events_available(ep))
- ep_busy_loop(ep, timed_out);
-
eavail = ep_events_available(ep);
+ if (!eavail)
+ eavail = ep_busy_loop(ep, timed_out);
+
if (eavail)
goto send_events;
- /*
- * Busy poll timed out. Drop NAPI ID for now, we can add
- * it back in when we have moved a socket with a valid NAPI
- * ID onto the ready list.
- */
- ep_reset_busy_poll_napi_id(ep);
-
do {
if (signal_pending(current))
return -EINTR;
--
2.29.1.341.ge80a0c044ae-goog
next prev parent reply other threads:[~2020-11-06 23:18 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-06 23:16 [PATCH 0/8] simplify ep_poll Soheil Hassas Yeganeh
2020-11-06 23:16 ` [PATCH 1/8] epoll: check for events when removing a timed out thread from the wait queue Soheil Hassas Yeganeh
2020-11-10 6:05 ` Davidlohr Bueso
2020-11-06 23:16 ` [PATCH 2/8] epoll: simplify signal handling Soheil Hassas Yeganeh
2020-11-06 23:16 ` [PATCH 3/8] epoll: pull fatal signal checks into ep_send_events() Soheil Hassas Yeganeh
2020-11-06 23:16 ` [PATCH 4/8] epoll: move eavail next to the list_empty_careful check Soheil Hassas Yeganeh
2020-11-06 23:16 ` Soheil Hassas Yeganeh [this message]
2020-11-06 23:16 ` [PATCH 6/8] epoll: pull all code between fetch_events and send_event into the loop Soheil Hassas Yeganeh
2020-11-06 23:16 ` [PATCH 7/8] epoll: replace gotos with a proper loop Soheil Hassas Yeganeh
2020-11-06 23:16 ` [PATCH 8/8] epoll: eliminate unnecessary lock for zero timeout Soheil Hassas Yeganeh
2020-11-06 23:35 ` [PATCH 0/8] simplify ep_poll Linus Torvalds
2020-11-08 1:43 ` Andrew Morton
2020-11-08 4:45 ` Soheil Hassas Yeganeh
2020-11-09 18:59 ` Davidlohr Bueso
2020-11-09 19:28 ` Soheil Hassas Yeganeh
2020-11-10 22:05 ` Andrew Morton
2020-11-11 3:37 ` Soheil Hassas Yeganeh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201106231635.3528496-6-soheil.kdev@gmail.com \
--to=soheil.kdev@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=dave@stgolabs.net \
--cc=edumazet@google.com \
--cc=guantaol@google.com \
--cc=khazhy@google.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=soheil@google.com \
--cc=torvalds@linux-foundation.org \
--cc=viro@zeniv.linux.org.uk \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).