From: Zhu Yanjun <zyjzyj2000@gmail.com>
To: "Håkon Bugge" <haakon.bugge@oracle.com>,
linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
netdev@vger.kernel.org, rds-devel@oss.oracle.com
Cc: Jason Gunthorpe <jgg@ziepe.ca>, Leon Romanovsky <leon@kernel.org>,
Saeed Mahameed <saeedm@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>,
"David S . Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Tejun Heo <tj@kernel.org>, Lai Jiangshan <jiangshanlai@gmail.com>,
Allison Henderson <allison.henderson@oracle.com>,
Manjunath Patil <manjunath.b.patil@oracle.com>,
Mark Zhang <markzhang@nvidia.com>,
Chuck Lever <chuck.lever@oracle.com>,
Shiraz Saleem <shiraz.saleem@intel.com>,
Yang Li <yang.lee@linux.alibaba.com>
Subject: Re: [PATCH v2 3/6] RDMA/cma: Brute force GFP_NOIO
Date: Thu, 16 May 2024 09:37:19 +0200 [thread overview]
Message-ID: <82bf9e5f-b798-4d29-8473-c074a34f15b0@linux.dev> (raw)
In-Reply-To: <20240515125342.1069999-4-haakon.bugge@oracle.com>
On 15.05.24 14:53, Håkon Bugge wrote:
> In cma_init(), we call memalloc_noio_{save,restore} in a parenthetic
> fashion when enabled by the module parameter force_noio.
>
> This in order to conditionally enable rdma_cm to work aligned with
> block I/O devices. Any work queued later on work-queues created during
> module initialization will inherit the PF_MEMALLOC_{NOIO,NOFS}
> flag(s), due to commit ("workqueue: Inherit NOIO and NOFS alloc
> flags").
>
> We do this in order to enable ULPs using the RDMA stack to be used as
> a network block I/O device. This to support a filesystem on top of a
> raw block device which uses said ULP(s) and the RDMA stack as the
> network transport layer.
>
> Under intense memory pressure, we get memory reclaims. Assume the
> filesystem reclaims memory, goes to the raw block device, which calls
> into the ULP in question, which calls the RDMA stack. Now, if
> regular GFP_KERNEL allocations in the ULP or the RDMA stack require
> reclaims to be fulfilled, we end up in a circular dependency.
>
> We break this circular dependency by:
>
> 1. Force all allocations in the ULP and the relevant RDMA stack to use
> GFP_NOIO, by means of a parenthetic use of
> memalloc_noio_{save,restore} on all relevant entry points.
>
> 2. Make sure work-queues inherits current->flags
> wrt. PF_MEMALLOC_{NOIO,NOFS}, such that work executed on the
> work-queue inherits the same flag(s).
>
> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
> ---
> drivers/infiniband/core/cma.c | 20 +++++++++++++++++---
> 1 file changed, 17 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 1e2cd7c8716e8..23a50cc3e81cb 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -50,6 +50,10 @@ MODULE_LICENSE("Dual BSD/GPL");
> #define CMA_IBOE_PACKET_LIFETIME 16
> #define CMA_PREFERRED_ROCE_GID_TYPE IB_GID_TYPE_ROCE_UDP_ENCAP
>
> +static bool cma_force_noio;
> +module_param_named(force_noio, cma_force_noio, bool, 0444);
> +MODULE_PARM_DESC(force_noio, "Force the use of GFP_NOIO (Y/N)");
> +
> static const char * const cma_events[] = {
> [RDMA_CM_EVENT_ADDR_RESOLVED] = "address resolved",
> [RDMA_CM_EVENT_ADDR_ERROR] = "address error",
> @@ -5424,6 +5428,10 @@ static struct pernet_operations cma_pernet_operations = {
> static int __init cma_init(void)
> {
> int ret;
> + unsigned int noio_flags;
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/maintainer-netdev.rst?h=v6.9#n376
"
Netdev has a convention for ordering local variables in functions.
Order the variable declaration lines longest to shortest, e.g.::
struct scatterlist *sg;
struct sk_buff *skb;
int err, i;
If there are dependencies between the variables preventing the ordering
move the initialization out of line.
"
Zhu Yanjun
> +
> + if (cma_force_noio)
> + noio_flags = memalloc_noio_save();
>
> /*
> * There is a rare lock ordering dependency in cma_netdev_callback()
> @@ -5439,8 +5447,10 @@ static int __init cma_init(void)
> }
>
> cma_wq = alloc_ordered_workqueue("rdma_cm", WQ_MEM_RECLAIM);
> - if (!cma_wq)
> - return -ENOMEM;
> + if (!cma_wq) {
> + ret = -ENOMEM;
> + goto out;
> + }
>
> ret = register_pernet_subsys(&cma_pernet_operations);
> if (ret)
> @@ -5458,7 +5468,8 @@ static int __init cma_init(void)
> if (ret)
> goto err_ib;
>
> - return 0;
> + ret = 0;
> + goto out;
>
> err_ib:
> ib_unregister_client(&cma_client);
> @@ -5469,6 +5480,9 @@ static int __init cma_init(void)
> unregister_pernet_subsys(&cma_pernet_operations);
> err_wq:
> destroy_workqueue(cma_wq);
> +out:
> + if (cma_force_noio)
> + memalloc_noio_restore(noio_flags);
> return ret;
> }
>
next prev parent reply other threads:[~2024-05-16 7:37 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-15 12:53 [PATCH v2 0/6] rds: rdma: Add ability to force GFP_NOIO Håkon Bugge
2024-05-15 12:53 ` [PATCH v2 1/6] workqueue: Inherit NOIO and NOFS alloc flags Håkon Bugge
2024-05-15 16:54 ` Tejun Heo
2024-05-16 15:27 ` Haakon Bugge
2024-05-16 16:29 ` Tejun Heo
2024-05-21 14:02 ` Haakon Bugge
2024-05-15 12:53 ` [PATCH v2 2/6] rds: Brute force GFP_NOIO Håkon Bugge
2024-05-15 12:53 ` [PATCH v2 3/6] RDMA/cma: " Håkon Bugge
2024-05-16 7:37 ` Zhu Yanjun [this message]
2024-05-16 15:49 ` Haakon Bugge
2024-05-16 19:07 ` Greg Sword
2024-05-17 9:28 ` Haakon Bugge
2024-05-26 9:27 ` Leon Romanovsky
2024-05-15 12:53 ` [PATCH v2 4/6] RDMA/cm: " Håkon Bugge
2024-05-15 12:53 ` [PATCH v2 5/6] RDMA/mlx5: " Håkon Bugge
2024-05-15 12:53 ` [PATCH v2 6/6] net/mlx5: " Håkon Bugge
2024-05-21 14:24 ` [PATCH v2 0/6] rds: rdma: Add ability to " Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=82bf9e5f-b798-4d29-8473-c074a34f15b0@linux.dev \
--to=zyjzyj2000@gmail.com \
--cc=allison.henderson@oracle.com \
--cc=chuck.lever@oracle.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=haakon.bugge@oracle.com \
--cc=jgg@ziepe.ca \
--cc=jiangshanlai@gmail.com \
--cc=kuba@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=manjunath.b.patil@oracle.com \
--cc=markzhang@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=rds-devel@oss.oracle.com \
--cc=saeedm@nvidia.com \
--cc=shiraz.saleem@intel.com \
--cc=tariqt@nvidia.com \
--cc=tj@kernel.org \
--cc=yang.lee@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).