From: Lokesh Gidra <lokeshgidra@google.com>
To: akpm@linux-foundation.org
Cc: lokeshgidra@google.com, linux-fsdevel@vger.kernel.org,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
selinux@vger.kernel.org, surenb@google.com,
kernel-team@android.com, aarcange@redhat.com, peterx@redhat.com,
david@redhat.com, axelrasmussen@google.com, bgeffon@google.com,
willy@infradead.org, jannh@google.com, kaleshsingh@google.com,
ngeoffray@google.com, timmurray@google.com, rppt@kernel.org,
Liam.Howlett@oracle.com
Subject: [PATCH v6 0/3] per-vma locks in userfaultfd
Date: Tue, 13 Feb 2024 13:57:38 -0800 [thread overview]
Message-ID: <20240213215741.3816570-1-lokeshgidra@google.com> (raw)
Performing userfaultfd operations (like copy/move etc.) in critical
section of mmap_lock (read-mode) causes significant contention on the
lock when operations requiring the lock in write-mode are taking place
concurrently. We can use per-vma locks instead to significantly reduce
the contention issue.
Android runtime's Garbage Collector uses userfaultfd for concurrent
compaction. mmap-lock contention during compaction potentially causes
jittery experience for the user. During one such reproducible scenario,
we observed the following improvements with this patch-set:
- Wall clock time of compaction phase came down from ~3s to <500ms
- Uninterruptible sleep time (across all threads in the process) was
~10ms (none in mmap_lock) during compaction, instead of >20s
Changes since v5 [5]:
- Use abstract function names (like uffd_mfill_lock/uffd_mfill_unlock)
to avoid using too many #ifdef's, per Suren Baghdasaryan and Liam
Howlett
- Use 'unlikely' (as earlier) to anon_vma related checks, per Liam Howlett
- Eliminate redundant ptr->err->ptr conversion, per Liam Howlett
- Use 'int' instead of 'long' for error return type, per Liam Howlett
Changes since v4 [4]:
- Fix possible deadlock in find_and_lock_vmas() which may arise if
lock_vma() is used for both src and dst vmas.
- Ensure we lock vma only once if src and dst vmas are same.
- Fix error handling in move_pages() after successfully locking vmas.
- Introduce helper function for finding dst vma and preparing its
anon_vma when done in mmap_lock critical section, per Liam Howlett.
- Introduce helper function for finding dst and src vmas when done in
mmap_lock critical section.
Changes since v3 [3]:
- Rename function names to clearly reflect which lock is being taken,
per Liam Howlett.
- Have separate functions and abstractions in mm/userfaultfd.c to avoid
confusion around which lock is being acquired/released, per Liam Howlett.
- Prepare anon_vma for all private vmas, anonymous or file-backed,
per Jann Horn.
Changes since v2 [2]:
- Implement and use lock_vma() which uses mmap_lock critical section
to lock the VMA using per-vma lock if lock_vma_under_rcu() fails,
per Liam R. Howlett. This helps simplify the code and also avoids
performing the entire userfaultfd operation under mmap_lock.
Changes since v1 [1]:
- rebase patches on 'mm-unstable' branch
[1] https://lore.kernel.org/all/20240126182647.2748949-1-lokeshgidra@google.com/
[2] https://lore.kernel.org/all/20240129193512.123145-1-lokeshgidra@google.com/
[3] https://lore.kernel.org/all/20240206010919.1109005-1-lokeshgidra@google.com/
[4] https://lore.kernel.org/all/20240208212204.2043140-1-lokeshgidra@google.com/
[5] https://lore.kernel.org/all/20240213001920.3551772-1-lokeshgidra@google.com/
Lokesh Gidra (3):
userfaultfd: move userfaultfd_ctx struct to header file
userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx
userfaultfd: use per-vma locks in userfaultfd operations
fs/userfaultfd.c | 86 ++-----
include/linux/userfaultfd_k.h | 75 ++++--
mm/userfaultfd.c | 438 +++++++++++++++++++++++++---------
3 files changed, 405 insertions(+), 194 deletions(-)
--
2.43.0.687.g38aa6559b0-goog
next reply other threads:[~2024-02-13 21:57 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-13 21:57 Lokesh Gidra [this message]
2024-02-13 21:57 ` [PATCH v6 1/3] userfaultfd: move userfaultfd_ctx struct to header file Lokesh Gidra
2024-02-13 21:57 ` [PATCH v6 2/3] userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx Lokesh Gidra
2024-02-13 21:57 ` [PATCH v6 3/3] userfaultfd: use per-vma locks in userfaultfd operations Lokesh Gidra
2024-02-14 22:12 ` Ryan Roberts
2024-02-14 22:20 ` Suren Baghdasaryan
2024-02-14 22:27 ` Lokesh Gidra
2024-02-14 22:33 ` Ryan Roberts
2024-02-14 15:17 ` [PATCH v6 0/3] per-vma locks in userfaultfd Liam R. Howlett
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240213215741.3816570-1-lokeshgidra@google.com \
--to=lokeshgidra@google.com \
--cc=Liam.Howlett@oracle.com \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=axelrasmussen@google.com \
--cc=bgeffon@google.com \
--cc=david@redhat.com \
--cc=jannh@google.com \
--cc=kaleshsingh@google.com \
--cc=kernel-team@android.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ngeoffray@google.com \
--cc=peterx@redhat.com \
--cc=rppt@kernel.org \
--cc=selinux@vger.kernel.org \
--cc=surenb@google.com \
--cc=timmurray@google.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).