* [PATCH] cifs: fix race between call_async() and reconnect()
@ 2015-12-23 6:32 Rabin Vincent
[not found] ` <1450852361-26556-1-git-send-email-rabin.vincent-VrBV9hrLPhE@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Rabin Vincent @ 2015-12-23 6:32 UTC (permalink / raw
To: sfrench-eUNUBHrolfbYtjvyW6yDsg
Cc: linux-cifs-u79uwXL29TY76Z2rM5mHXA, rabin-66gdRtMMWGc,
Rabin Vincent
cifs_call_async() queues the MID to the pending list and calls
smb_send_rqst(). If smb_send_rqst() performs a partial send, it sets
the tcpStatus to CifsNeedReconnect and returns an error code to
cifs_call_async(). In this case, cifs_call_async() removes the MID
from the list and returns to the caller.
However, cifs_call_async() releases the server mutex _before_ removing
the MID. This means that a cifs_reconnect() can race with this function
and manage to remove the MID from the list and delete the entry before
cifs_call_async() calls cifs_delete_mid(). This leads to various
crashes due to the use after free in cifs_delete_mid().
Task1 Task2
cifs_call_async():
- rc = -EAGAIN
- mutex_unlock(srv_mutex)
cifs_reconnect():
- mutex_lock(srv_mutex)
- mutex_unlock(srv_mutex)
- list_delete(mid)
- mid->callback()
cifs_writev_callback():
- mutex_lock(srv_mutex)
- delete(mid)
- mutex_unlock(srv_mutex)
- cifs_delete_mid(mid) <---- use after free
Fix this by removing the MID in cifs_call_async() before releasing the
srv_mutex. Also hold the srv_mutex in cifs_reconnect() until the MIDs
are moved out of the pending list.
Signed-off-by: Rabin Vincent <rabin.vincent-VrBV9hrLPhE@public.gmane.org>
---
fs/cifs/connect.c | 2 +-
fs/cifs/transport.c | 6 ++++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index ecb0803..3c194ff 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -368,7 +368,6 @@ cifs_reconnect(struct TCP_Server_Info *server)
server->session_key.response = NULL;
server->session_key.len = 0;
server->lstrp = jiffies;
- mutex_unlock(&server->srv_mutex);
/* mark submitted MIDs for retry and issue callback */
INIT_LIST_HEAD(&retry_list);
@@ -381,6 +380,7 @@ cifs_reconnect(struct TCP_Server_Info *server)
list_move(&mid_entry->qhead, &retry_list);
}
spin_unlock(&GlobalMid_Lock);
+ mutex_unlock(&server->srv_mutex);
cifs_dbg(FYI, "%s: issuing mid callbacks\n", __func__);
list_for_each_safe(tmp, tmp2, &retry_list) {
diff --git a/fs/cifs/transport.c b/fs/cifs/transport.c
index 2a24c52..87abe8e 100644
--- a/fs/cifs/transport.c
+++ b/fs/cifs/transport.c
@@ -576,14 +576,16 @@ cifs_call_async(struct TCP_Server_Info *server, struct smb_rqst *rqst,
cifs_in_send_dec(server);
cifs_save_when_sent(mid);
- if (rc < 0)
+ if (rc < 0) {
server->sequence_number -= 2;
+ cifs_delete_mid(mid);
+ }
+
mutex_unlock(&server->srv_mutex);
if (rc == 0)
return 0;
- cifs_delete_mid(mid);
add_credits_and_wake_if(server, credits, optype);
return rc;
}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] cifs: fix race between call_async() and reconnect()
[not found] ` <1450852361-26556-1-git-send-email-rabin.vincent-VrBV9hrLPhE@public.gmane.org>
@ 2015-12-23 15:21 ` Shirish Pargaonkar
[not found] ` <CADT32eL9JHxq-xNQgtgeLgFm9HNKGX32uSH2cVnhrA0bi4uZ8g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Shirish Pargaonkar @ 2015-12-23 15:21 UTC (permalink / raw
To: Rabin Vincent; +Cc: Steve French, linux-cifs, rabin-66gdRtMMWGc, Rabin Vincent
Looks correct. I think only cifs_call_async() is where
DeletedMidQEntry() is called without holding srv_mutex lock.
Acked-by: Shirish Pargaonkar <shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
On Wed, Dec 23, 2015 at 12:32 AM, Rabin Vincent <rabin.vincent-VrBV9hrLPhE@public.gmane.org> wrote:
> cifs_call_async() queues the MID to the pending list and calls
> smb_send_rqst(). If smb_send_rqst() performs a partial send, it sets
> the tcpStatus to CifsNeedReconnect and returns an error code to
> cifs_call_async(). In this case, cifs_call_async() removes the MID
> from the list and returns to the caller.
>
> However, cifs_call_async() releases the server mutex _before_ removing
> the MID. This means that a cifs_reconnect() can race with this function
> and manage to remove the MID from the list and delete the entry before
> cifs_call_async() calls cifs_delete_mid(). This leads to various
> crashes due to the use after free in cifs_delete_mid().
>
> Task1 Task2
>
> cifs_call_async():
> - rc = -EAGAIN
> - mutex_unlock(srv_mutex)
>
> cifs_reconnect():
> - mutex_lock(srv_mutex)
> - mutex_unlock(srv_mutex)
> - list_delete(mid)
> - mid->callback()
> cifs_writev_callback():
> - mutex_lock(srv_mutex)
> - delete(mid)
> - mutex_unlock(srv_mutex)
>
> - cifs_delete_mid(mid) <---- use after free
>
> Fix this by removing the MID in cifs_call_async() before releasing the
> srv_mutex. Also hold the srv_mutex in cifs_reconnect() until the MIDs
> are moved out of the pending list.
>
> Signed-off-by: Rabin Vincent <rabin.vincent-VrBV9hrLPhE@public.gmane.org>
> ---
> fs/cifs/connect.c | 2 +-
> fs/cifs/transport.c | 6 ++++--
> 2 files changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
> index ecb0803..3c194ff 100644
> --- a/fs/cifs/connect.c
> +++ b/fs/cifs/connect.c
> @@ -368,7 +368,6 @@ cifs_reconnect(struct TCP_Server_Info *server)
> server->session_key.response = NULL;
> server->session_key.len = 0;
> server->lstrp = jiffies;
> - mutex_unlock(&server->srv_mutex);
>
> /* mark submitted MIDs for retry and issue callback */
> INIT_LIST_HEAD(&retry_list);
> @@ -381,6 +380,7 @@ cifs_reconnect(struct TCP_Server_Info *server)
> list_move(&mid_entry->qhead, &retry_list);
> }
> spin_unlock(&GlobalMid_Lock);
> + mutex_unlock(&server->srv_mutex);
>
> cifs_dbg(FYI, "%s: issuing mid callbacks\n", __func__);
> list_for_each_safe(tmp, tmp2, &retry_list) {
> diff --git a/fs/cifs/transport.c b/fs/cifs/transport.c
> index 2a24c52..87abe8e 100644
> --- a/fs/cifs/transport.c
> +++ b/fs/cifs/transport.c
> @@ -576,14 +576,16 @@ cifs_call_async(struct TCP_Server_Info *server, struct smb_rqst *rqst,
> cifs_in_send_dec(server);
> cifs_save_when_sent(mid);
>
> - if (rc < 0)
> + if (rc < 0) {
> server->sequence_number -= 2;
> + cifs_delete_mid(mid);
> + }
> +
> mutex_unlock(&server->srv_mutex);
>
> if (rc == 0)
> return 0;
>
> - cifs_delete_mid(mid);
> add_credits_and_wake_if(server, credits, optype);
> return rc;
> }
> --
> 1.7.10.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] cifs: fix race between call_async() and reconnect()
[not found] ` <CADT32eL9JHxq-xNQgtgeLgFm9HNKGX32uSH2cVnhrA0bi4uZ8g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-23 20:37 ` Steve French
[not found] ` <CAH2r5mtwbAh7gjVVcbcwe72LXnt_NcCZJUUSfTZGTg2+q=JUhA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Steve French @ 2015-12-23 20:37 UTC (permalink / raw
To: Shirish Pargaonkar
Cc: Rabin Vincent, Steve French, linux-cifs, rabin-66gdRtMMWGc,
Rabin Vincent
Looks like a stable candidate.
Opinions?
On Wed, Dec 23, 2015 at 9:21 AM, Shirish Pargaonkar
<shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Looks correct. I think only cifs_call_async() is where
> DeletedMidQEntry() is called without holding srv_mutex lock.
>
> Acked-by: Shirish Pargaonkar <shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
>
> On Wed, Dec 23, 2015 at 12:32 AM, Rabin Vincent <rabin.vincent-VrBV9hrLPhE@public.gmane.org> wrote:
>> cifs_call_async() queues the MID to the pending list and calls
>> smb_send_rqst(). If smb_send_rqst() performs a partial send, it sets
>> the tcpStatus to CifsNeedReconnect and returns an error code to
>> cifs_call_async(). In this case, cifs_call_async() removes the MID
>> from the list and returns to the caller.
>>
>> However, cifs_call_async() releases the server mutex _before_ removing
>> the MID. This means that a cifs_reconnect() can race with this function
>> and manage to remove the MID from the list and delete the entry before
>> cifs_call_async() calls cifs_delete_mid(). This leads to various
>> crashes due to the use after free in cifs_delete_mid().
>>
>> Task1 Task2
>>
>> cifs_call_async():
>> - rc = -EAGAIN
>> - mutex_unlock(srv_mutex)
>>
>> cifs_reconnect():
>> - mutex_lock(srv_mutex)
>> - mutex_unlock(srv_mutex)
>> - list_delete(mid)
>> - mid->callback()
>> cifs_writev_callback():
>> - mutex_lock(srv_mutex)
>> - delete(mid)
>> - mutex_unlock(srv_mutex)
>>
>> - cifs_delete_mid(mid) <---- use after free
>>
>> Fix this by removing the MID in cifs_call_async() before releasing the
>> srv_mutex. Also hold the srv_mutex in cifs_reconnect() until the MIDs
>> are moved out of the pending list.
>>
>> Signed-off-by: Rabin Vincent <rabin.vincent-VrBV9hrLPhE@public.gmane.org>
>> ---
>> fs/cifs/connect.c | 2 +-
>> fs/cifs/transport.c | 6 ++++--
>> 2 files changed, 5 insertions(+), 3 deletions(-)
>>
>> diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
>> index ecb0803..3c194ff 100644
>> --- a/fs/cifs/connect.c
>> +++ b/fs/cifs/connect.c
>> @@ -368,7 +368,6 @@ cifs_reconnect(struct TCP_Server_Info *server)
>> server->session_key.response = NULL;
>> server->session_key.len = 0;
>> server->lstrp = jiffies;
>> - mutex_unlock(&server->srv_mutex);
>>
>> /* mark submitted MIDs for retry and issue callback */
>> INIT_LIST_HEAD(&retry_list);
>> @@ -381,6 +380,7 @@ cifs_reconnect(struct TCP_Server_Info *server)
>> list_move(&mid_entry->qhead, &retry_list);
>> }
>> spin_unlock(&GlobalMid_Lock);
>> + mutex_unlock(&server->srv_mutex);
>>
>> cifs_dbg(FYI, "%s: issuing mid callbacks\n", __func__);
>> list_for_each_safe(tmp, tmp2, &retry_list) {
>> diff --git a/fs/cifs/transport.c b/fs/cifs/transport.c
>> index 2a24c52..87abe8e 100644
>> --- a/fs/cifs/transport.c
>> +++ b/fs/cifs/transport.c
>> @@ -576,14 +576,16 @@ cifs_call_async(struct TCP_Server_Info *server, struct smb_rqst *rqst,
>> cifs_in_send_dec(server);
>> cifs_save_when_sent(mid);
>>
>> - if (rc < 0)
>> + if (rc < 0) {
>> server->sequence_number -= 2;
>> + cifs_delete_mid(mid);
>> + }
>> +
>> mutex_unlock(&server->srv_mutex);
>>
>> if (rc == 0)
>> return 0;
>>
>> - cifs_delete_mid(mid);
>> add_credits_and_wake_if(server, credits, optype);
>> return rc;
>> }
>> --
>> 1.7.10.4
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Thanks,
Steve
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] cifs: fix race between call_async() and reconnect()
[not found] ` <CAH2r5mtwbAh7gjVVcbcwe72LXnt_NcCZJUUSfTZGTg2+q=JUhA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2015-12-24 16:07 ` Rabin Vincent
[not found] ` <20151224160700.GA6309-VrBV9hrLPhE@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Rabin Vincent @ 2015-12-24 16:07 UTC (permalink / raw
To: Steve French
Cc: Shirish Pargaonkar, Steve French, linux-cifs, rabin-66gdRtMMWGc
On Wed, Dec 23, 2015 at 02:37:34PM -0600, Steve French wrote:
> Looks like a stable candidate.
>
> Opinions?
Yes, please. I intended to include the CC:stable tag in the patch
submission but forgot to do so. We've seen the crashes in our testing
so the race is not just theoretical.
Thanks.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] cifs: fix race between call_async() and reconnect()
[not found] ` <20151224160700.GA6309-VrBV9hrLPhE@public.gmane.org>
@ 2016-01-14 20:39 ` Steve French
0 siblings, 0 replies; 5+ messages in thread
From: Steve French @ 2016-01-14 20:39 UTC (permalink / raw
To: Rabin Vincent
Cc: Shirish Pargaonkar, Steve French, linux-cifs, rabin-66gdRtMMWGc
merged into cifs-2.6.git for-next (and added cc:stable)
Let me know if others for 4.5
On Thu, Dec 24, 2015 at 10:07 AM, Rabin Vincent <rabin.vincent-VrBV9hrLPhE@public.gmane.org> wrote:
> On Wed, Dec 23, 2015 at 02:37:34PM -0600, Steve French wrote:
>> Looks like a stable candidate.
>>
>> Opinions?
>
> Yes, please. I intended to include the CC:stable tag in the patch
> submission but forgot to do so. We've seen the crashes in our testing
> so the race is not just theoretical.
>
> Thanks.
--
Thanks,
Steve
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-01-14 20:39 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-12-23 6:32 [PATCH] cifs: fix race between call_async() and reconnect() Rabin Vincent
[not found] ` <1450852361-26556-1-git-send-email-rabin.vincent-VrBV9hrLPhE@public.gmane.org>
2015-12-23 15:21 ` Shirish Pargaonkar
[not found] ` <CADT32eL9JHxq-xNQgtgeLgFm9HNKGX32uSH2cVnhrA0bi4uZ8g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-23 20:37 ` Steve French
[not found] ` <CAH2r5mtwbAh7gjVVcbcwe72LXnt_NcCZJUUSfTZGTg2+q=JUhA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2015-12-24 16:07 ` Rabin Vincent
[not found] ` <20151224160700.GA6309-VrBV9hrLPhE@public.gmane.org>
2016-01-14 20:39 ` Steve French
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.