From: Chenliang Li <cliang01.li@samsung.com>
To: axboe@kernel.dk, asml.silence@gmail.com
Cc: io-uring@vger.kernel.org, peiwei.li@samsung.com,
joshi.k@samsung.com, kundan.kumar@samsung.com,
anuj20.g@samsung.com, gost.dev@samsung.com,
Chenliang Li <cliang01.li@samsung.com>
Subject: [PATCH v4 4/4] io_uring/rsrc: enable multi-hugepage buffer coalescing
Date: Tue, 14 May 2024 15:54:44 +0800 [thread overview]
Message-ID: <20240514075444.590910-5-cliang01.li@samsung.com> (raw)
In-Reply-To: <20240514075444.590910-1-cliang01.li@samsung.com>
Modify the original buffer registration and enable the coalescing for
buffers with more than one hugepages.
Signed-off-by: Chenliang Li <cliang01.li@samsung.com>
---
io_uring/rsrc.c | 44 ++++++++------------------------------------
1 file changed, 8 insertions(+), 36 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index 53fac5f27bbf..5e5c1d6f3501 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -1047,7 +1047,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
unsigned long off;
size_t size;
int ret, nr_pages, i;
- struct folio *folio = NULL;
+ struct io_imu_folio_data data;
*pimu = (struct io_mapped_ubuf *)&dummy_ubuf;
if (!iov->iov_base)
@@ -1062,30 +1062,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
goto done;
}
- /* If it's a huge page, try to coalesce them into a single bvec entry */
- if (nr_pages > 1) {
- folio = page_folio(pages[0]);
- for (i = 1; i < nr_pages; i++) {
- /*
- * Pages must be consecutive and on the same folio for
- * this to work
- */
- if (page_folio(pages[i]) != folio ||
- pages[i] != pages[i - 1] + 1) {
- folio = NULL;
- break;
- }
- }
- if (folio) {
- /*
- * The pages are bound to the folio, it doesn't
- * actually unpin them but drops all but one reference,
- * which is usually put down by io_buffer_unmap().
- * Note, needs a better helper.
- */
- unpin_user_pages(&pages[1], nr_pages - 1);
- nr_pages = 1;
- }
+ /* If it's huge page(s), try to coalesce them into fewer bvec entries */
+ if (io_sqe_buffer_try_coalesce(pages, nr_pages, &data)) {
+ ret = io_coalesced_imu_alloc(ctx, iov, pimu, last_hpage,
+ pages, &data);
+ goto done;
}
imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL);
@@ -1109,10 +1090,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov,
*pimu = imu;
ret = 0;
- if (folio) {
- bvec_set_page(&imu->bvec[0], pages[0], size, off);
- goto done;
- }
for (i = 0; i < nr_pages; i++) {
size_t vec_len;
@@ -1218,23 +1195,18 @@ int io_import_fixed(int ddir, struct iov_iter *iter,
* we know that:
*
* 1) it's a BVEC iter, we set it up
- * 2) all bvecs are PAGE_SIZE in size, except potentially the
+ * 2) all bvecs are the same in size, except potentially the
* first and last bvec
*
* So just find our index, and adjust the iterator afterwards.
* If the offset is within the first bvec (or the whole first
* bvec, just use iov_iter_advance(). This makes it easier
* since we can just skip the first segment, which may not
- * be PAGE_SIZE aligned.
+ * be folio_size aligned.
*/
const struct bio_vec *bvec = imu->bvec;
if (offset < bvec->bv_len) {
- /*
- * Note, huge pages buffers consists of one large
- * bvec entry and should always go this way. The other
- * branch doesn't expect non PAGE_SIZE'd chunks.
- */
iter->bvec = bvec;
iter->nr_segs = bvec->bv_len;
iter->count -= offset;
--
2.34.1
next prev parent reply other threads:[~2024-05-14 7:55 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CGME20240514075453epcas5p17974fb62d65a88b1a1b55b97942ee2be@epcas5p1.samsung.com>
2024-05-14 7:54 ` [PATCH v4 0/4] io_uring/rsrc: coalescing multi-hugepage registered buffers Chenliang Li
[not found] ` <CGME20240514075457epcas5p10f02f1746f957df91353724ec859664f@epcas5p1.samsung.com>
2024-05-14 7:54 ` [PATCH v4 1/4] io_uring/rsrc: add hugepage buffer coalesce helpers Chenliang Li
2024-05-16 14:07 ` Anuj gupta
[not found] ` <CGME20240514075459epcas5p2275b4c26f16bcfcea200e97fc75c2a14@epcas5p2.samsung.com>
2024-05-14 7:54 ` [PATCH v4 2/4] io_uring/rsrc: store folio shift and mask into imu Chenliang Li
2024-05-16 14:08 ` Anuj gupta
[not found] ` <CGME20240514075500epcas5p1e638b1ae84727b3669ff6b780cd1cb23@epcas5p1.samsung.com>
2024-05-14 7:54 ` [PATCH v4 3/4] io_uring/rsrc: add init and account functions for coalesced imus Chenliang Li
[not found] ` <CGME20240514075502epcas5p10be6bef71d284a110277575d6008563d@epcas5p1.samsung.com>
2024-05-14 7:54 ` Chenliang Li [this message]
2024-05-16 14:09 ` [PATCH v4 4/4] io_uring/rsrc: enable multi-hugepage buffer coalescing Anuj gupta
2024-05-16 14:01 ` [PATCH v4 0/4] io_uring/rsrc: coalescing multi-hugepage registered buffers Anuj gupta
2024-05-16 14:58 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240514075444.590910-5-cliang01.li@samsung.com \
--to=cliang01.li@samsung.com \
--cc=anuj20.g@samsung.com \
--cc=asml.silence@gmail.com \
--cc=axboe@kernel.dk \
--cc=gost.dev@samsung.com \
--cc=io-uring@vger.kernel.org \
--cc=joshi.k@samsung.com \
--cc=kundan.kumar@samsung.com \
--cc=peiwei.li@samsung.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).