* [PATCH] [RFC] Fix data missing when reusing bh which is ready to be checkpointed
@ 2022-12-20 15:05 Zhihao Cheng
2022-12-21 10:13 ` Jan Kara
0 siblings, 1 reply; 3+ messages in thread
From: Zhihao Cheng @ 2022-12-20 15:05 UTC (permalink / raw
To: tytso, jack
Cc: linux-ext4, linux-kernel, chengzhihao1, yi.zhang, libaokun1,
zhanchengbin1, yangerkun
From: zhanchengbin <zhanchengbin1@huawei.com>
Following process will make data lost and could lead to a filesystem
corrupted problem:
1. jh(bh) is inserted into T1->t_checkpoint_list, bh is dirty, and
jh->b_transaction = NULL
2. T1 is added into journal->j_checkpoint_transactions.
3. Get bh prepare to write while doing checkpoing:
PA PB
do_get_write_access jbd2_log_do_checkpoint
spin_lock(&jh->b_state_lock)
if (buffer_dirty(bh))
clear_buffer_dirty(bh) // clear buffer dirty
set_buffer_jbddirty(bh)
transaction =
journal->j_checkpoint_transactions
jh = transaction->t_checkpoint_list
if (!buffer_dirty(bh))
__jbd2_journal_remove_checkpoint(jh)
// bh won't be flushed
jbd2_cleanup_journal_tail
__jbd2_journal_file_buffer(jh, transaction, BJ_Reserved)
4. Aborting journal/Power-cut before writing latest bh on journal area.
In this way we get a corrupted filesystem with bh'data lost.
Fix it by wrapping clear_buffer_dirty(bh) and jh->b_transaction setting
into journal->j_list_lock, so that jbd2_log_do_checkpoint() will wait
until jh's new transaction fininshed even bh is currently not dirty.
Cc: <stable@kernel.org>
Signed-off-by: zhanchengbin <zhanchengbin1@huawei.com>
Signed-off-by: Baokun Li <libaokun1@huawei.com>
Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com>
---
This is a quick fix, I need some suggestions about this patch, whether
it will import new problems if this patch is applied.
Yi suggests that the formal solution could be splitting
journal->j_list_lock into two locks: one protects checkpoint list and
the other one for other lists. Besides, jh->b_state_lock should be
held while traversing transaction->t_checkpoint_list in
jbd2_log_do_checkpoint()/journal_shrink_one_cp_list().
fs/jbd2/transaction.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c
index 6a404ac1c178..d22460001d6b 100644
--- a/fs/jbd2/transaction.c
+++ b/fs/jbd2/transaction.c
@@ -990,6 +990,7 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
start_lock = jiffies;
lock_buffer(bh);
spin_lock(&jh->b_state_lock);
+ spin_lock(&journal->j_list_lock);
/* If it takes too long to lock the buffer, trace it */
time_lock = jbd2_time_diff(start_lock, jiffies);
@@ -1039,6 +1040,7 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
error = -EROFS;
if (is_handle_aborted(handle)) {
+ spin_unlock(&journal->j_list_lock);
spin_unlock(&jh->b_state_lock);
goto out;
}
@@ -1049,8 +1051,10 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
* b_next_transaction points to it
*/
if (jh->b_transaction == transaction ||
- jh->b_next_transaction == transaction)
+ jh->b_next_transaction == transaction) {
+ spin_unlock(&journal->j_list_lock);
goto done;
+ }
/*
* this is the first time this transaction is touching this buffer,
@@ -1073,11 +1077,11 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
* Paired with barrier in jbd2_write_access_granted()
*/
smp_wmb();
- spin_lock(&journal->j_list_lock);
__jbd2_journal_file_buffer(jh, transaction, BJ_Reserved);
spin_unlock(&journal->j_list_lock);
goto done;
}
+ spin_unlock(&journal->j_list_lock);
/*
* If there is already a copy-out version of this buffer, then we don't
* need to make another one
--
2.31.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] [RFC] Fix data missing when reusing bh which is ready to be checkpointed
2022-12-20 15:05 [PATCH] [RFC] Fix data missing when reusing bh which is ready to be checkpointed Zhihao Cheng
@ 2022-12-21 10:13 ` Jan Kara
2023-01-07 9:28 ` Zhihao Cheng
0 siblings, 1 reply; 3+ messages in thread
From: Jan Kara @ 2022-12-21 10:13 UTC (permalink / raw
To: Zhihao Cheng
Cc: tytso, jack, linux-ext4, linux-kernel, yi.zhang, libaokun1,
zhanchengbin1, yangerkun
On Tue 20-12-22 23:05:51, Zhihao Cheng wrote:
> From: zhanchengbin <zhanchengbin1@huawei.com>
>
> Following process will make data lost and could lead to a filesystem
> corrupted problem:
>
> 1. jh(bh) is inserted into T1->t_checkpoint_list, bh is dirty, and
> jh->b_transaction = NULL
> 2. T1 is added into journal->j_checkpoint_transactions.
> 3. Get bh prepare to write while doing checkpoing:
> PA PB
> do_get_write_access jbd2_log_do_checkpoint
> spin_lock(&jh->b_state_lock)
> if (buffer_dirty(bh))
> clear_buffer_dirty(bh) // clear buffer dirty
> set_buffer_jbddirty(bh)
> transaction =
> journal->j_checkpoint_transactions
> jh = transaction->t_checkpoint_list
> if (!buffer_dirty(bh))
> __jbd2_journal_remove_checkpoint(jh)
> // bh won't be flushed
> jbd2_cleanup_journal_tail
> __jbd2_journal_file_buffer(jh, transaction, BJ_Reserved)
> 4. Aborting journal/Power-cut before writing latest bh on journal area.
>
> In this way we get a corrupted filesystem with bh'data lost.
>
> Fix it by wrapping clear_buffer_dirty(bh) and jh->b_transaction setting
> into journal->j_list_lock, so that jbd2_log_do_checkpoint() will wait
> until jh's new transaction fininshed even bh is currently not dirty.
>
> Cc: <stable@kernel.org>
> Signed-off-by: zhanchengbin <zhanchengbin1@huawei.com>
> Signed-off-by: Baokun Li <libaokun1@huawei.com>
> Signed-off-by: Zhihao Cheng <chengzhihao1@huawei.com>
> ---
> This is a quick fix, I need some suggestions about this patch, whether
> it will import new problems if this patch is applied.
> Yi suggests that the formal solution could be splitting
> journal->j_list_lock into two locks: one protects checkpoint list and
> the other one for other lists. Besides, jh->b_state_lock should be
> held while traversing transaction->t_checkpoint_list in
> jbd2_log_do_checkpoint()/journal_shrink_one_cp_list().
>
> fs/jbd2/transaction.c | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
Good catch! Did you find it by code inspection or were you able to actually
trigger this problem?
I think there might be a simpler fix of the problem. Move the clearing
of buffer_dirty bit just before the call to __jbd2_journal_file_buffer().
We'll need to keep the buffer locked somewhat longer but that should not be
a huge deal.
Honza
> diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c
> index 6a404ac1c178..d22460001d6b 100644
> --- a/fs/jbd2/transaction.c
> +++ b/fs/jbd2/transaction.c
> @@ -990,6 +990,7 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
> start_lock = jiffies;
> lock_buffer(bh);
> spin_lock(&jh->b_state_lock);
> + spin_lock(&journal->j_list_lock);
>
> /* If it takes too long to lock the buffer, trace it */
> time_lock = jbd2_time_diff(start_lock, jiffies);
> @@ -1039,6 +1040,7 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
>
> error = -EROFS;
> if (is_handle_aborted(handle)) {
> + spin_unlock(&journal->j_list_lock);
> spin_unlock(&jh->b_state_lock);
> goto out;
> }
> @@ -1049,8 +1051,10 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
> * b_next_transaction points to it
> */
> if (jh->b_transaction == transaction ||
> - jh->b_next_transaction == transaction)
> + jh->b_next_transaction == transaction) {
> + spin_unlock(&journal->j_list_lock);
> goto done;
> + }
>
> /*
> * this is the first time this transaction is touching this buffer,
> @@ -1073,11 +1077,11 @@ do_get_write_access(handle_t *handle, struct journal_head *jh,
> * Paired with barrier in jbd2_write_access_granted()
> */
> smp_wmb();
> - spin_lock(&journal->j_list_lock);
> __jbd2_journal_file_buffer(jh, transaction, BJ_Reserved);
> spin_unlock(&journal->j_list_lock);
> goto done;
> }
> + spin_unlock(&journal->j_list_lock);
> /*
> * If there is already a copy-out version of this buffer, then we don't
> * need to make another one
> --
> 2.31.1
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] [RFC] Fix data missing when reusing bh which is ready to be checkpointed
2022-12-21 10:13 ` Jan Kara
@ 2023-01-07 9:28 ` Zhihao Cheng
0 siblings, 0 replies; 3+ messages in thread
From: Zhihao Cheng @ 2023-01-07 9:28 UTC (permalink / raw
To: Jan Kara
Cc: tytso, jack, linux-ext4, linux-kernel, yi.zhang, libaokun1,
zhanchengbin1, yangerkun
在 2022/12/21 18:13, Jan Kara 写道:
> On Tue 20-12-22 23:05:51, Zhihao Cheng wrote:
>> From: zhanchengbin <zhanchengbin1@huawei.com>
>>
>> Following process will make data lost and could lead to a filesystem
>> corrupted problem:
>>
>> 1. jh(bh) is inserted into T1->t_checkpoint_list, bh is dirty, and
>> jh->b_transaction = NULL
>> 2. T1 is added into journal->j_checkpoint_transactions.
>> 3. Get bh prepare to write while doing checkpoing:
>> PA PB
>> do_get_write_access jbd2_log_do_checkpoint
>> spin_lock(&jh->b_state_lock)
>> if (buffer_dirty(bh))
>> clear_buffer_dirty(bh) // clear buffer dirty
>> set_buffer_jbddirty(bh)
>> transaction =
>> journal->j_checkpoint_transactions
>> jh = transaction->t_checkpoint_list
>> if (!buffer_dirty(bh))
>> __jbd2_journal_remove_checkpoint(jh)
>> // bh won't be flushed
>> jbd2_cleanup_journal_tail
>> __jbd2_journal_file_buffer(jh, transaction, BJ_Reserved)
>> 4. Aborting journal/Power-cut before writing latest bh on journal area.
[...]
>>
>> fs/jbd2/transaction.c | 8 ++++++--
>> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> Good catch! Did you find it by code inspection or were you able to actually
> trigger this problem?
By code inspection.
Reproducer: https://bugzilla.kernel.org/show_bug.cgi?id=216898
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2023-01-07 9:29 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-12-20 15:05 [PATCH] [RFC] Fix data missing when reusing bh which is ready to be checkpointed Zhihao Cheng
2022-12-21 10:13 ` Jan Kara
2023-01-07 9:28 ` Zhihao Cheng
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).