* Trying to understand a weird timeout when using fio and multipath IO
@ 2015-11-03 21:22 Todd Lawall
2015-11-04 5:47 ` Sitsofe Wheeler
2015-11-04 9:10 ` Andrey Kuzmin
0 siblings, 2 replies; 3+ messages in thread
From: Todd Lawall @ 2015-11-03 21:22 UTC (permalink / raw
To: fio@vger.kernel.org
Hello,
I'm trying to understand a behavior, and I am hoping to further my
understanding of what Fio is doing. In the specific case in question,
I'm seeing a seven minute wait in resuming IO after a failover. In other
variations on this job file, the seven minute wait disappears, and it
drops back down to the 40 second wait that I see with usual IO
loads that are run.
The setup:
- I have one Windows 2012 R2 host, with two NICs.
- I have one storage array, with 2 controllers A and B, with 2 10GbE ports
for each side, making 4 ports, with failover capability between the two
sides.
- I have iSCSI and MPIO setup so that there is one login from each NIC to
each side, so four sessions total for each volume. The map looks
something like this:
nic1 nic2
/ \ / \
/ \ / \
side A side B side A side B
port 0 port 0 port 1 port 1
- I have the fio job below. It is basically 256k blocks, 1 iodepth, one
worker with 48 drives.
[global]
do_verify=0
ioengine=windowsaio
numjobs=1
iodepth=1
offset=0
direct=1
thread
[fio-0]
blocksize=256k
readwrite=rw
filename=\\.\PHYSICALDRIVE19
filename=\\.\PHYSICALDRIVE20
<snipped out the other 44 drives>
filename=\\.\PHYSICALDRIVE13
filename=\\.\PHYSICALDRIVE14
size=100%
If I alter the job in any of the following ways, IO keeps going after the
failover period which is about 40 seconds. To summarize:
Doesn't work:
- multiple disks, single job, 1 iodepth
Works:
- Single disk, one job, 1 iodepth
- multiple disks, one job with all disks, same iodepth as # of disks (e.g.
if there's 48 disks, iodepth is set to 48)
- multiple disks, one job per disk, 1 iodepth
Would anyone have any idea why that one arrangement causes a
significant delay before IO is resumed?
Thanks in advance,
Todd
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Trying to understand a weird timeout when using fio and multipath IO
2015-11-03 21:22 Trying to understand a weird timeout when using fio and multipath IO Todd Lawall
@ 2015-11-04 5:47 ` Sitsofe Wheeler
2015-11-04 9:10 ` Andrey Kuzmin
1 sibling, 0 replies; 3+ messages in thread
From: Sitsofe Wheeler @ 2015-11-04 5:47 UTC (permalink / raw
To: Todd Lawall; +Cc: fio@vger.kernel.org
Hi Todd,
What path policy have you set Windows to and what does the does the
Windows event log have to say when things stall? Can you reproduce
this error if you readwrite to only writes or only reads? Does halving
the number of disks still show the problem? Does reducing the
blocksize make the problem less likely to happen? If you run two fio
binaries with the same job do they both pause at the same time?
On 3 November 2015 at 21:22, Todd Lawall <tlawall@nexgenstorage.com> wrote:
> Hello,
>
> I'm trying to understand a behavior, and I am hoping to further my
> understanding of what Fio is doing. In the specific case in question,
> I'm seeing a seven minute wait in resuming IO after a failover. In other
> variations on this job file, the seven minute wait disappears, and it
> drops back down to the 40 second wait that I see with usual IO
> loads that are run.
>
> The setup:
> - I have one Windows 2012 R2 host, with two NICs.
> - I have one storage array, with 2 controllers A and B, with 2 10GbE ports
> for each side, making 4 ports, with failover capability between the two
> sides.
> - I have iSCSI and MPIO setup so that there is one login from each NIC to
> each side, so four sessions total for each volume. The map looks
> something like this:
>
> nic1 nic2
> / \ / \
> / \ / \
> side A side B side A side B
> port 0 port 0 port 1 port 1
>
> - I have the fio job below. It is basically 256k blocks, 1 iodepth, one
> worker with 48 drives.
>
> [global]
> do_verify=0
> ioengine=windowsaio
> numjobs=1
> iodepth=1
> offset=0
> direct=1
> thread
>
> [fio-0]
> blocksize=256k
> readwrite=rw
> filename=\\.\PHYSICALDRIVE19
> filename=\\.\PHYSICALDRIVE20
> <snipped out the other 44 drives>
> filename=\\.\PHYSICALDRIVE13
> filename=\\.\PHYSICALDRIVE14
> size=100%
>
> If I alter the job in any of the following ways, IO keeps going after the
> failover period which is about 40 seconds. To summarize:
>
> Doesn't work:
> - multiple disks, single job, 1 iodepth
>
> Works:
> - Single disk, one job, 1 iodepth
> - multiple disks, one job with all disks, same iodepth as # of disks (e.g.
> if there's 48 disks, iodepth is set to 48)
> - multiple disks, one job per disk, 1 iodepth
>
> Would anyone have any idea why that one arrangement causes a
> significant delay before IO is resumed?
>
> Thanks in advance,
> Todd
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Sitsofe | http://sucs.org/~sits/
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Trying to understand a weird timeout when using fio and multipath IO
2015-11-03 21:22 Trying to understand a weird timeout when using fio and multipath IO Todd Lawall
2015-11-04 5:47 ` Sitsofe Wheeler
@ 2015-11-04 9:10 ` Andrey Kuzmin
1 sibling, 0 replies; 3+ messages in thread
From: Andrey Kuzmin @ 2015-11-04 9:10 UTC (permalink / raw
To: Todd Lawall; +Cc: fio@vger.kernel.org
On Wed, Nov 4, 2015 at 12:22 AM, Todd Lawall <tlawall@nexgenstorage.com> wrote:
> Hello,
>
> I'm trying to understand a behavior, and I am hoping to further my
> understanding of what Fio is doing. In the specific case in question,
> I'm seeing a seven minute wait in resuming IO after a failover. In other
> variations on this job file, the seven minute wait disappears, and it
> drops back down to the 40 second wait that I see with usual IO
> loads that are run.
>
> The setup:
> - I have one Windows 2012 R2 host, with two NICs.
> - I have one storage array, with 2 controllers A and B, with 2 10GbE ports
> for each side, making 4 ports, with failover capability between the two
> sides.
> - I have iSCSI and MPIO setup so that there is one login from each NIC to
> each side, so four sessions total for each volume. The map looks
> something like this:
>
> nic1 nic2
> / \ / \
> / \ / \
> side A side B side A side B
> port 0 port 0 port 1 port 1
>
> - I have the fio job below. It is basically 256k blocks, 1 iodepth, one
> worker with 48 drives.
>
> [global]
> do_verify=0
> ioengine=windowsaio
> numjobs=1
> iodepth=1
> offset=0
> direct=1
> thread
>
> [fio-0]
> blocksize=256k
> readwrite=rw
> filename=\\.\PHYSICALDRIVE19
> filename=\\.\PHYSICALDRIVE20
> <snipped out the other 44 drives>
> filename=\\.\PHYSICALDRIVE13
> filename=\\.\PHYSICALDRIVE14
> size=100%
>
If my understanding of how multi-pathing works doesn't fool me, you
problem with the above setup is fairly straightforward. You are
running an asynchronous engine with single I/O queue depth
single-threaded against multiple files, which means running
synchronous I/O against one file at a time (FIO switches files for you
round-robin, unless specifically instructed to do it any other way).
So when a path fails, your job does roughly 11 path failovers
(slightly more, depending on whether a device the current I/O was
running against is on the failed path) one by one, which should take
11x40s = 440 s or roughly 7 minutes.
> If I alter the job in any of the following ways, IO keeps going after the
> failover period which is about 40 seconds. To summarize:
>
> Doesn't work:
> - multiple disks, single job, 1 iodepth
>
> Works:
> - Single disk, one job, 1 iodepth
> - multiple disks, one job with all disks, same iodepth as # of disks (e.g.
> if there's 48 disks, iodepth is set to 48)
> - multiple disks, one job per disk, 1 iodepth
All above setups differ from the original in that failover for each
drive run in parallel, taking roughly 40 s.
Hope this helps,
Andrey
>
> Would anyone have any idea why that one arrangement causes a
> significant delay before IO is resumed?
>
> Thanks in advance,
> Todd
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2015-11-04 9:11 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-03 21:22 Trying to understand a weird timeout when using fio and multipath IO Todd Lawall
2015-11-04 5:47 ` Sitsofe Wheeler
2015-11-04 9:10 ` Andrey Kuzmin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.