Re: BUG: scheduling while atomic when nvmet_rdma_queue_response fails in posting a request

From: Keith Busch <kbusch@kernel.org>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Michal Kalderon <mkalderon@marvell.com>,
	Christoph Hellwig <hch@lst.de>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
	Shai Malin <smalin@marvell.com>, Ariel Elior <aelior@marvell.com>
Subject: Re: BUG: scheduling while atomic when nvmet_rdma_queue_response fails in posting a request
Date: Tue, 8 Jun 2021 11:41:34 -0700	[thread overview]
Message-ID: <20210608184134.GA339600@dhcp-10-100-145-180.wdc.com> (raw)
In-Reply-To: <e059fc90-81e7-dbe8-86f4-87b4bdbd5bb0@grimberg.me>

On Tue, Jun 08, 2021 at 10:43:45AM -0700, Sagi Grimberg wrote:
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index 7d607f435e36..6d2eea322779 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -16,6 +16,7 @@
>  #include <linux/wait.h>
>  #include <linux/inet.h>
>  #include <asm/unaligned.h>
> +#include <linux/async.h>
> 
>  #include <rdma/ib_verbs.h>
>  #include <rdma/rdma_cm.h>
> @@ -712,6 +713,12 @@ static void nvmet_rdma_send_done(struct ib_cq *cq,
> struct ib_wc *wc)
>         }
>  }
> 
> +static void nvmet_rdma_async_release_rsp(void *data, async_cookie_t cookie)
> +{
> +       struct nvmet_rdma_rsp *rsp = data;
> +       nvmet_rdma_release_rsp(rsp);
> +}
> +
>  static void nvmet_rdma_queue_response(struct nvmet_req *req)
>  {
>         struct nvmet_rdma_rsp *rsp =
> @@ -745,7 +752,12 @@ static void nvmet_rdma_queue_response(struct nvmet_req
> *req)
> 
>         if (unlikely(ib_post_send(cm_id->qp, first_wr, NULL))) {
>                 pr_err("sending cmd response failed\n");
> -               nvmet_rdma_release_rsp(rsp);
> +               /*
> +                * We might be in atomic context, hence release
> +                * the rsp in async context in case we need to
> +                * process the wr_wait_list.
> +                */
> +               async_schedule(nvmet_rdma_async_release_rsp, rsp);
>         }
>  }

Just FYI, async_schedule() has conditions where it may execute your
callback synchronously. Your suggestion is probably fine for testing,
but it sounds like you require something that can guarantee a non-atomic
context for nvmet_rdma_release_rsp().

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme