From: "Subash Abhinov Kasiviswanathan (KS)" <quic_subashab@quicinc.com>
To: Eric Dumazet <edumazet@google.com>
Cc: <soheil@google.com>, <ncardwell@google.com>, <yyd@google.com>,
<ycheng@google.com>, <quic_stranche@quicinc.com>,
<davem@davemloft.net>, <kuba@kernel.org>,
<netdev@vger.kernel.org>
Subject: Re: Potential impact of commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale")
Date: Wed, 15 May 2024 20:32:27 -0600 [thread overview]
Message-ID: <60b04b0a-a50e-4d4a-a2bf-ea420f428b9c@quicinc.com> (raw)
In-Reply-To: <CANn89iKRuxON3pWjivs0kU-XopBiqTZn4Mx+wOKHVmQ97zAU5A@mail.gmail.com>
On 5/15/2024 1:10 AM, Eric Dumazet wrote:
> On Wed, May 15, 2024 at 6:47 AM Subash Abhinov Kasiviswanathan (KS)
> <quic_subashab@quicinc.com> wrote:
>>
>> We recently noticed that a device running a 6.6.17 kernel (A) was having
>> a slower single stream download speed compared to a device running
>> 6.1.57 kernel (B). The test here is over mobile radio with iperf3 with
>> window size 4M from a third party server.
>
> Hi Subash
>
> I think you gave many details, but please give us more of them :
Hi Eric
Thanks for getting back. Hope the information below is useful.
>
> 1) What driver is used on the receiver side.
rmnet
> 2) MTU
1372
> 3) cat /proc/sys/net/ipv4/tcp_rmem
4096 6291456 16777216
>
> Ideally, you could snapshot "ss -temoi dst <otherpeer>" on receive
> side while the transfer is ongoing,
> and possibly while stopping the receiver thread (kill -STOP `pidof iperf`)
>
192.0.0.2 is the device side address. I've listed the output of "ss
-temoi dst 223.62.236.10" mid transfer and one around the end of transfer.
I believe iperf3 makes a control connection prior to triggering the data
connection so it will list two flows. The transfer between
192.0.0.2:42278 <-> 223.62.236.10:5215 is the main data connection in
this case.
//mid transfer
State Recv-Q Send-Q Local Address:Port Peer
Address:Port
ESTAB 0 0 192.0.0.2:42278 223.62.236.10:5215
ino:129232 sk:3218 fwmark:0xc0078 <->
skmem:(r0,rb8388608,t0,tb8388608,f0,w0,o0,bl0,d1) ts sack
cubic wscale:7,6 rto:236 rtt:34.249/16.545 ato:40 mss:1320 rcvmss:1320
advmss:1320 cwnd:10 ssthresh:1400 bytes_acked:38
bytes_received:211495680 segs_out:46198 segs_in:160290 data_segs_out:1
data_segs_in:160287 send 3.1Mbps lastsnd:3996 pacing_rate 6.2Mbps
delivery_rate 452.4Kbps app_limited busy:24ms rcv_rtt:26.542
rcv_space:3058440 minrtt:23.34
ESTAB 0 0 192.0.0.2:42270 223.62.236.10:5215
ino:128718 sk:4273 fwmark:0xc0078 <->
skmem:(r0,rb6291456,t0,tb2097152,f0,w0,o0,bl0,d0) ts sack
cubic wscale:10,9 rto:528 rtt:144.931/93.4 ato:40 mss:1320 rcvmss:536
advmss:1320 cwnd:10 ssthresh:1400 bytes_acked:223 bytes_received:4
segs_out:9 segs_in:8 data_segs_out:3 data_segs_in:4 send 728.6Kbps
lastsnd:6064 lastrcv:3948 lastack:3948 pacing_rate 1.5Mbps delivery_rate
351.8Kbps app_limited busy:156ms rcv_space:13200 minrtt:30.021
//close to end of transfer
State Recv-Q Send-Q Local Address:Port Peer
Address:Port
ESTAB 4324072 0 192.0.0.2:42278 223.62.236.10:5215
ino:129232 sk:3218 fwmark:0xc0078 <->
skmem:(r4511016,rb8388608,t0,tb8388608,f2776,w0,o0,bl0,d1) ts
sack cubic wscale:7,6 rto:236 rtt:34.249/16.545 ato:40 mss:1320
rcvmss:1320 advmss:1320 cwnd:10 ssthresh:1400 bytes_acked:38
bytes_received:608252040 segs_out:133117 segs_in:460963 data_segs_out:1
data_segs_in:460960 send 3.1Mbps lastsnd:10104 pacing_rate 6.2Mbps
delivery_rate 452.4Kbps app_limited busy:24ms rcv_rtt:25.111
rcv_space:3871560 minrtt:23.34
ESTAB 0 294 192.0.0.2:42270 223.62.236.10:5215
timer:(on,412ms,0) ino:128718 sk:4273 fwmark:0xc0078 <->
skmem:(r0,rb6291456,t0,tb2097152,f2010,w2086,o0,bl0,d0) ts
sack cubic wscale:10,9 rto:512 rtt:129.796/94.265 ato:40 mss:1320
rcvmss:536 advmss:1320 cwnd:10 ssthresh:1400 bytes_acked:224
bytes_received:5 segs_out:12 segs_in:9 data_segs_out:5 data_segs_in:5
send 813.6Kbps lastsnd:48 lastrcv:52 lastack:52 pacing_rate 1.6Mbps
delivery_rate 442.8Kbps app_limited busy:228ms unacked:1 rcv_space:13200
notsent:290 minrtt:23.848
> TCP is sensitive to the skb->len/skb->truesize ratio.
> Some drivers are known to provide 'bad skbs' in this regard.
>
> Commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale") is
> simply a step for dynamic
> probing of skb->len/skb->truesize ratio, and give incentive for better
> memory use.
>
> Ultimately, TCP RWIN derives from effective memory usage.
>
> Sending a too big RWIN can cause excessive memory usage or packet drops.
> If you say RWIN was 6MB+ before the patch, this looks like a bug to me,
> because tcp_rmem[2] = 6MB by default. There is no way a driver can
> pack 6MB of TCP payload in 6MB of memory (no skb/headers overhead ???)
> This would only work well in lossless networks, and if receiving
> application drains TCP receive queue fast enough.
>
> Please take a look at these relevant patches.
> Note they are not perfect patches, because usbnet can still provide
> 'bad skbs', forcing TCP to send small RWIN.
rmnet is not updating the truesize directly in the receive path. There
is no cloning and there is an explicit copy of the data content to a
freshly allocated skb similar to your commits shared below.
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/net/ethernet/qualcomm/rmnet/rmnet_map_data.c?h=v6.6.17#n385
From netif_receive_skb_entry tracing, I see that the truesize is around
~2.5K for ~1.5K packets.
>
> d50729f1d60bca822ef6d9c1a5fb28d486bd7593 net: usb: smsc95xx: stop
> lying about skb->truesize
> 05417aa9c0c038da2464a0c504b9d4f99814a23b net: usb: sr9700: stop lying
> about skb->truesize
> 1b3b2d9e772b99ea3d0f1f2252bf7a1c94b88be6 net: usb: smsc75xx: stop
> lying about skb->truesize
> 9aad6e45c4e7d16b2bb7c3794154b828fb4384b4 usb: aqc111: stop lying about
> skb->truesize
> 4ce62d5b2f7aecd4900e7d6115588ad7f9acccca net: usb: ax88179_178a: stop
> lying about skb->truesize
I reviewed many of the tcpdumps from other tests internally and I
consistently see the receiver window size scale to roughly half of what
is specified in iperf3 regardless of whatever radio configurations or
MTU. There was no download speed issue reported for any of these cases.
I believe this particular download test is failing as the RTT is likely
higher in this network than the other cases.
next prev parent reply other threads:[~2024-05-16 2:32 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-15 4:46 Potential impact of commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale") Subash Abhinov Kasiviswanathan (KS)
2024-05-15 7:10 ` Eric Dumazet
2024-05-16 2:32 ` Subash Abhinov Kasiviswanathan (KS) [this message]
2024-05-16 5:36 ` Eric Dumazet
2024-05-16 7:16 ` Subash Abhinov Kasiviswanathan (KS)
2024-05-16 7:57 ` Eric Dumazet
2024-05-16 8:31 ` Eric Dumazet
2024-05-16 18:49 ` Subash Abhinov Kasiviswanathan (KS)
2024-05-17 7:08 ` Subash Abhinov Kasiviswanathan (KS)
2024-05-19 2:13 ` Subash Abhinov Kasiviswanathan (KS)
2024-05-20 15:12 ` Eric Dumazet
2024-05-20 17:09 ` Subash Abhinov Kasiviswanathan (KS)
2024-05-20 17:20 ` Eric Dumazet
2024-05-20 17:33 ` Subash Abhinov Kasiviswanathan (KS)
2024-05-20 17:42 ` Eric Dumazet
2024-05-16 8:04 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=60b04b0a-a50e-4d4a-a2bf-ea420f428b9c@quicinc.com \
--to=quic_subashab@quicinc.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=kuba@kernel.org \
--cc=ncardwell@google.com \
--cc=netdev@vger.kernel.org \
--cc=quic_stranche@quicinc.com \
--cc=soheil@google.com \
--cc=ycheng@google.com \
--cc=yyd@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).