* [PATCH v3 RESEND 0/2] fuse: Add support for resend pending requests
@ 2023-12-11 12:06 Zhao Chen
2023-12-11 12:06 ` [PATCH v3 RESEND 1/2] fuse: Introduce a new notification type " Zhao Chen
2023-12-11 12:06 ` [PATCH v3 RESEND 2/2] fuse: Use the high bit of request ID for indicating resend requests Zhao Chen
0 siblings, 2 replies; 4+ messages in thread
From: Zhao Chen @ 2023-12-11 12:06 UTC (permalink / raw)
To: linux-fsdevel; +Cc: miklos
After the FUSE daemon crashes, the fuse mount point becomes inaccessible.
In some production environments, a watchdog daemon is used to preserve
the FUSE connection's file descriptor (fd). When the FUSE daemon crashes,
a new FUSE daemon is started and takes over the fd from the watchdog
daemon, allowing it to continue providing services.
However, if any inflight requests are lost during the crash, the user
process becomes stuck as it does not receive any replies.
To resolve this issue, this patchset introduces a new notification type
that enable resending these pending requests to the FUSE daemon again,
allowing the stuck user process to recover.
When using the resend API, FUSE daemon needs to ensure avoidance of
processing duplicate non-idempotent requests to prevent potential
consistency issues. The high bit of the fuse request id is utilized for
indicating the resend request.
---
v2->v3:
- use notification instead of sysfs API to trigger resend
- simplify FUSE_REQ_ID_MASK related code
- rename some related macro names
v1->v2:
- remove flush sysfs API in the original mail
- add using high bit of request ID for indicating resend requests
- add wakeup in fuse_resend_pqueue()
Zhao Chen (2):
fuse: Introduce a new notification type for resend pending requests
fuse: Use the high bit of request ID for indicating resend requests
fs/fuse/dev.c | 69 ++++++++++++++++++++++++++++++++++++++-
fs/fuse/inode.c | 3 +-
include/uapi/linux/fuse.h | 12 +++++++
3 files changed, 82 insertions(+), 2 deletions(-)
--
2.32.0.3.g01195cf9f
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH v3 RESEND 1/2] fuse: Introduce a new notification type for resend pending requests
2023-12-11 12:06 [PATCH v3 RESEND 0/2] fuse: Add support for resend pending requests Zhao Chen
@ 2023-12-11 12:06 ` Zhao Chen
2023-12-11 12:06 ` [PATCH v3 RESEND 2/2] fuse: Use the high bit of request ID for indicating resend requests Zhao Chen
1 sibling, 0 replies; 4+ messages in thread
From: Zhao Chen @ 2023-12-11 12:06 UTC (permalink / raw)
To: linux-fsdevel; +Cc: miklos
When a FUSE daemon panics and failover, we aim to minimize the impact on
applications by reusing the existing FUSE connection. During this process,
another daemon is employed to preserve the FUSE connection's file
descriptor. The new started FUSE Daemon will takeover the fd and continue
to provide service.
However, it is possible for some inflight requests to be lost and never
returned. As a result, applications awaiting replies would become stuck
forever. To address this, we can resend these pending requests to the
new started FUSE daemon.
This patch introduces a new notification type "FUSE_NOTIFY_RESEND", which
can trigger resending of the pending requests, ensuring they are properly
processed again.
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
---
fs/fuse/dev.c | 64 +++++++++++++++++++++++++++++++++++++++
include/uapi/linux/fuse.h | 1 +
2 files changed, 65 insertions(+)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 1a8f82f478cb..a5a874b2f2e2 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -1775,6 +1775,67 @@ static int fuse_notify_retrieve(struct fuse_conn *fc, unsigned int size,
return err;
}
+/*
+ * Resending all processing queue requests.
+ *
+ * During a FUSE daemon panics and failover, it is possible for some inflight
+ * requests to be lost and never returned. As a result, applications awaiting
+ * replies would become stuck forever. To address this, we can use notification
+ * to trigger resending of these pending requests to the FUSE daemon, ensuring
+ * they are properly processed again.
+ *
+ * Please note that this strategy is applicable only to idempotent requests or
+ * if the FUSE daemon takes careful measures to avoid processing duplicated
+ * non-idempotent requests.
+ */
+static void fuse_resend(struct fuse_conn *fc)
+{
+ struct fuse_dev *fud;
+ struct fuse_req *req, *next;
+ struct fuse_iqueue *fiq = &fc->iq;
+ LIST_HEAD(to_queue);
+ unsigned int i;
+
+ spin_lock(&fc->lock);
+ if (!fc->connected) {
+ spin_unlock(&fc->lock);
+ return;
+ }
+
+ list_for_each_entry(fud, &fc->devices, entry) {
+ struct fuse_pqueue *fpq = &fud->pq;
+
+ spin_lock(&fpq->lock);
+ list_for_each_entry_safe(req, next, &fpq->io, list) {
+ spin_lock(&req->waitq.lock);
+ if (!test_bit(FR_LOCKED, &req->flags)) {
+ __fuse_get_request(req);
+ list_move(&req->list, &to_queue);
+ }
+ spin_unlock(&req->waitq.lock);
+ }
+ for (i = 0; i < FUSE_PQ_HASH_SIZE; i++)
+ list_splice_tail_init(&fpq->processing[i], &to_queue);
+ spin_unlock(&fpq->lock);
+ }
+ spin_unlock(&fc->lock);
+
+ list_for_each_entry_safe(req, next, &to_queue, list) {
+ __set_bit(FR_PENDING, &req->flags);
+ }
+
+ spin_lock(&fiq->lock);
+ /* iq and pq requests are both oldest to newest */
+ list_splice(&to_queue, &fiq->pending);
+ fiq->ops->wake_pending_and_unlock(fiq);
+}
+
+static int fuse_notify_resend(struct fuse_conn *fc)
+{
+ fuse_resend(fc);
+ return 0;
+}
+
static int fuse_notify(struct fuse_conn *fc, enum fuse_notify_code code,
unsigned int size, struct fuse_copy_state *cs)
{
@@ -1800,6 +1861,9 @@ static int fuse_notify(struct fuse_conn *fc, enum fuse_notify_code code,
case FUSE_NOTIFY_DELETE:
return fuse_notify_delete(fc, size, cs);
+ case FUSE_NOTIFY_RESEND:
+ return fuse_notify_resend(fc);
+
default:
fuse_copy_finish(cs);
return -EINVAL;
diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
index e7418d15fe39..277dc25b7863 100644
--- a/include/uapi/linux/fuse.h
+++ b/include/uapi/linux/fuse.h
@@ -635,6 +635,7 @@ enum fuse_notify_code {
FUSE_NOTIFY_STORE = 4,
FUSE_NOTIFY_RETRIEVE = 5,
FUSE_NOTIFY_DELETE = 6,
+ FUSE_NOTIFY_RESEND = 7,
FUSE_NOTIFY_CODE_MAX,
};
--
2.32.0.3.g01195cf9f
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH v3 RESEND 2/2] fuse: Use the high bit of request ID for indicating resend requests
2023-12-11 12:06 [PATCH v3 RESEND 0/2] fuse: Add support for resend pending requests Zhao Chen
2023-12-11 12:06 ` [PATCH v3 RESEND 1/2] fuse: Introduce a new notification type " Zhao Chen
@ 2023-12-11 12:06 ` Zhao Chen
1 sibling, 0 replies; 4+ messages in thread
From: Zhao Chen @ 2023-12-11 12:06 UTC (permalink / raw)
To: linux-fsdevel; +Cc: miklos
Some FUSE daemons want to know if the received request is a resend
request. The high bit of the fuse request ID is utilized for indicating
this, enabling the receiver to perform appropriate handling.
The init flag "FUSE_HAS_RESEND" is added to indicate this feature.
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
---
fs/fuse/dev.c | 5 ++++-
fs/fuse/inode.c | 3 ++-
include/uapi/linux/fuse.h | 11 +++++++++++
3 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index a5a874b2f2e2..65febd013ce9 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -28,6 +28,7 @@ MODULE_ALIAS("devname:fuse");
/* Ordinary requests have even IDs, while interrupts IDs are odd */
#define FUSE_INT_REQ_BIT (1ULL << 0)
#define FUSE_REQ_ID_STEP (1ULL << 1)
+#define FUSE_REQ_ID_MASK (~(FUSE_INT_REQ_BIT | FUSE_UNIQUE_RESEND))
static struct kmem_cache *fuse_req_cachep;
@@ -194,7 +195,7 @@ EXPORT_SYMBOL_GPL(fuse_len_args);
u64 fuse_get_unique(struct fuse_iqueue *fiq)
{
- fiq->reqctr += FUSE_REQ_ID_STEP;
+ fiq->reqctr = (fiq->reqctr + FUSE_REQ_ID_STEP) & FUSE_REQ_ID_MASK;
return fiq->reqctr;
}
EXPORT_SYMBOL_GPL(fuse_get_unique);
@@ -1822,6 +1823,8 @@ static void fuse_resend(struct fuse_conn *fc)
list_for_each_entry_safe(req, next, &to_queue, list) {
__set_bit(FR_PENDING, &req->flags);
+ /* mark the request as resend request */
+ req->in.h.unique |= FUSE_UNIQUE_RESEND;
}
spin_lock(&fiq->lock);
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 2a6d44f91729..a4f1f539d4d9 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -1330,7 +1330,8 @@ void fuse_send_init(struct fuse_mount *fm)
FUSE_NO_OPENDIR_SUPPORT | FUSE_EXPLICIT_INVAL_DATA |
FUSE_HANDLE_KILLPRIV_V2 | FUSE_SETXATTR_EXT | FUSE_INIT_EXT |
FUSE_SECURITY_CTX | FUSE_CREATE_SUPP_GROUP |
- FUSE_HAS_EXPIRE_ONLY | FUSE_DIRECT_IO_ALLOW_MMAP;
+ FUSE_HAS_EXPIRE_ONLY | FUSE_DIRECT_IO_ALLOW_MMAP |
+ FUSE_HAS_RESEND;
#ifdef CONFIG_FUSE_DAX
if (fm->fc->dax)
flags |= FUSE_MAP_ALIGNMENT;
diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
index 277dc25b7863..c0e38acee083 100644
--- a/include/uapi/linux/fuse.h
+++ b/include/uapi/linux/fuse.h
@@ -410,6 +410,8 @@ struct fuse_file_lock {
* symlink and mknod (single group that matches parent)
* FUSE_HAS_EXPIRE_ONLY: kernel supports expiry-only entry invalidation
* FUSE_DIRECT_IO_ALLOW_MMAP: allow shared mmap in FOPEN_DIRECT_IO mode.
+ * FUSE_HAS_RESEND: kernel supports resending pending requests, and the high bit
+ * of the request ID indicates resend requests
*/
#define FUSE_ASYNC_READ (1 << 0)
#define FUSE_POSIX_LOCKS (1 << 1)
@@ -449,6 +451,7 @@ struct fuse_file_lock {
#define FUSE_CREATE_SUPP_GROUP (1ULL << 34)
#define FUSE_HAS_EXPIRE_ONLY (1ULL << 35)
#define FUSE_DIRECT_IO_ALLOW_MMAP (1ULL << 36)
+#define FUSE_HAS_RESEND (1ULL << 37)
/* Obsolete alias for FUSE_DIRECT_IO_ALLOW_MMAP */
#define FUSE_DIRECT_IO_RELAX FUSE_DIRECT_IO_ALLOW_MMAP
@@ -961,6 +964,14 @@ struct fuse_fallocate_in {
uint32_t padding;
};
+/**
+ * FUSE request unique ID flag
+ *
+ * Indicates whether this is a resend request. The receiver should handle this
+ * request accordingly.
+ */
+#define FUSE_UNIQUE_RESEND (1ULL << 63)
+
struct fuse_in_header {
uint32_t len;
uint32_t opcode;
--
2.32.0.3.g01195cf9f
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH v3 RESEND 2/2] fuse: Use the high bit of request ID for indicating resend requests
2023-12-20 8:49 [PATCH v3 RESEND 0/2] fuse: Add support for resend pending requests Zhao Chen
@ 2023-12-20 8:49 ` Zhao Chen
0 siblings, 0 replies; 4+ messages in thread
From: Zhao Chen @ 2023-12-20 8:49 UTC (permalink / raw)
To: linux-fsdevel; +Cc: miklos
Some FUSE daemons want to know if the received request is a resend
request. The high bit of the fuse request ID is utilized for indicating
this, enabling the receiver to perform appropriate handling.
The init flag "FUSE_HAS_RESEND" is added to indicate this feature.
Signed-off-by: Zhao Chen <winters.zc@antgroup.com>
---
fs/fuse/dev.c | 5 ++++-
fs/fuse/inode.c | 3 ++-
include/uapi/linux/fuse.h | 11 +++++++++++
3 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index a5a874b2f2e2..65febd013ce9 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -28,6 +28,7 @@ MODULE_ALIAS("devname:fuse");
/* Ordinary requests have even IDs, while interrupts IDs are odd */
#define FUSE_INT_REQ_BIT (1ULL << 0)
#define FUSE_REQ_ID_STEP (1ULL << 1)
+#define FUSE_REQ_ID_MASK (~(FUSE_INT_REQ_BIT | FUSE_UNIQUE_RESEND))
static struct kmem_cache *fuse_req_cachep;
@@ -194,7 +195,7 @@ EXPORT_SYMBOL_GPL(fuse_len_args);
u64 fuse_get_unique(struct fuse_iqueue *fiq)
{
- fiq->reqctr += FUSE_REQ_ID_STEP;
+ fiq->reqctr = (fiq->reqctr + FUSE_REQ_ID_STEP) & FUSE_REQ_ID_MASK;
return fiq->reqctr;
}
EXPORT_SYMBOL_GPL(fuse_get_unique);
@@ -1822,6 +1823,8 @@ static void fuse_resend(struct fuse_conn *fc)
list_for_each_entry_safe(req, next, &to_queue, list) {
__set_bit(FR_PENDING, &req->flags);
+ /* mark the request as resend request */
+ req->in.h.unique |= FUSE_UNIQUE_RESEND;
}
spin_lock(&fiq->lock);
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 2a6d44f91729..a4f1f539d4d9 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -1330,7 +1330,8 @@ void fuse_send_init(struct fuse_mount *fm)
FUSE_NO_OPENDIR_SUPPORT | FUSE_EXPLICIT_INVAL_DATA |
FUSE_HANDLE_KILLPRIV_V2 | FUSE_SETXATTR_EXT | FUSE_INIT_EXT |
FUSE_SECURITY_CTX | FUSE_CREATE_SUPP_GROUP |
- FUSE_HAS_EXPIRE_ONLY | FUSE_DIRECT_IO_ALLOW_MMAP;
+ FUSE_HAS_EXPIRE_ONLY | FUSE_DIRECT_IO_ALLOW_MMAP |
+ FUSE_HAS_RESEND;
#ifdef CONFIG_FUSE_DAX
if (fm->fc->dax)
flags |= FUSE_MAP_ALIGNMENT;
diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h
index 277dc25b7863..c0e38acee083 100644
--- a/include/uapi/linux/fuse.h
+++ b/include/uapi/linux/fuse.h
@@ -410,6 +410,8 @@ struct fuse_file_lock {
* symlink and mknod (single group that matches parent)
* FUSE_HAS_EXPIRE_ONLY: kernel supports expiry-only entry invalidation
* FUSE_DIRECT_IO_ALLOW_MMAP: allow shared mmap in FOPEN_DIRECT_IO mode.
+ * FUSE_HAS_RESEND: kernel supports resending pending requests, and the high bit
+ * of the request ID indicates resend requests
*/
#define FUSE_ASYNC_READ (1 << 0)
#define FUSE_POSIX_LOCKS (1 << 1)
@@ -449,6 +451,7 @@ struct fuse_file_lock {
#define FUSE_CREATE_SUPP_GROUP (1ULL << 34)
#define FUSE_HAS_EXPIRE_ONLY (1ULL << 35)
#define FUSE_DIRECT_IO_ALLOW_MMAP (1ULL << 36)
+#define FUSE_HAS_RESEND (1ULL << 37)
/* Obsolete alias for FUSE_DIRECT_IO_ALLOW_MMAP */
#define FUSE_DIRECT_IO_RELAX FUSE_DIRECT_IO_ALLOW_MMAP
@@ -961,6 +964,14 @@ struct fuse_fallocate_in {
uint32_t padding;
};
+/**
+ * FUSE request unique ID flag
+ *
+ * Indicates whether this is a resend request. The receiver should handle this
+ * request accordingly.
+ */
+#define FUSE_UNIQUE_RESEND (1ULL << 63)
+
struct fuse_in_header {
uint32_t len;
uint32_t opcode;
--
2.32.0.3.g01195cf9f
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-12-20 8:49 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-12-11 12:06 [PATCH v3 RESEND 0/2] fuse: Add support for resend pending requests Zhao Chen
2023-12-11 12:06 ` [PATCH v3 RESEND 1/2] fuse: Introduce a new notification type " Zhao Chen
2023-12-11 12:06 ` [PATCH v3 RESEND 2/2] fuse: Use the high bit of request ID for indicating resend requests Zhao Chen
-- strict thread matches above, loose matches on Subject: below --
2023-12-20 8:49 [PATCH v3 RESEND 0/2] fuse: Add support for resend pending requests Zhao Chen
2023-12-20 8:49 ` [PATCH v3 RESEND 2/2] fuse: Use the high bit of request ID for indicating resend requests Zhao Chen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).