* I/O wait problem with hardware raid
@ 2008-08-29 1:27 Eric S. Johansson
2008-08-30 18:47 ` Bill Davidsen
0 siblings, 1 reply; 5+ messages in thread
From: Eric S. Johansson @ 2008-08-29 1:27 UTC (permalink / raw
To: linux-raid
I'm not sure if this mailing lists is just for the Linux kernel raid so this
query does not belong, feel free to tell me where to go.
I'm running Ubuntu 8.04 on an HP ProLiant ml350 with an E200i sata raid card.
The problem is the I/O wait percentage increases during data transfer. A simple
rsync transfer will bring the percent I/O wait value up to the mid-20s to 40s
and a little more stress causes the wait percentage to climb to upper 90% range.
it appears that most of the I/O wait time is from the disk array because a copy
has a similar effect. However, I cannot eliminate some contribution by the
networking code.
I'm not sure how to debug this. vmstat doesn't show anything extraordinary so
I'm left scratching my head.
what should I be looking for?
Thanks
---eric
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O wait problem with hardware raid
2008-08-29 1:27 I/O wait problem with hardware raid Eric S. Johansson
@ 2008-08-30 18:47 ` Bill Davidsen
2008-08-31 14:50 ` Eric S. Johansson
0 siblings, 1 reply; 5+ messages in thread
From: Bill Davidsen @ 2008-08-30 18:47 UTC (permalink / raw
To: Eric S. Johansson; +Cc: linux-raid
Eric S. Johansson wrote:
> I'm not sure if this mailing lists is just for the Linux kernel raid so this
> query does not belong, feel free to tell me where to go.
>
> I'm running Ubuntu 8.04 on an HP ProLiant ml350 with an E200i sata raid card.
>
> The problem is the I/O wait percentage increases during data transfer. A simple
> rsync transfer will bring the percent I/O wait value up to the mid-20s to 40s
> and a little more stress causes the wait percentage to climb to upper 90% range.
> it appears that most of the I/O wait time is from the disk array because a copy
> has a similar effect. However, I cannot eliminate some contribution by the
> networking code.
>
> I'm not sure how to debug this. vmstat doesn't show anything extraordinary so
> I'm left scratching my head.
>
> what should I be looking for?
>
iowait means that there is a program waiting for I/O. That's all. Of
course when you do a copy (regardless of software) the CPU is waiting
for disk transfers. I'm not sure what you think you should debug, i/o
takes time, and if the program is blocked until the next input comes in
it will enter the waitio state. If there is no other process to use the
available CPU it becomes waitio, which is essentially available CPU
cycles similar to idle.
What exactly do you think is wrong?
--
Bill Davidsen <davidsen@tmr.com>
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O wait problem with hardware raid
2008-08-30 18:47 ` Bill Davidsen
@ 2008-08-31 14:50 ` Eric S. Johansson
2008-08-31 15:48 ` Roger Heflin
2008-08-31 21:08 ` Bill Davidsen
0 siblings, 2 replies; 5+ messages in thread
From: Eric S. Johansson @ 2008-08-31 14:50 UTC (permalink / raw
To: Bill Davidsen; +Cc: linux-raid
Bill Davidsen wrote:
> iowait means that there is a program waiting for I/O. That's all.
I was under the impression that I/O wait created a blocking condition.
> Of
> course when you do a copy (regardless of software) the CPU is waiting
> for disk transfers. I'm not sure what you think you should debug, i/o
> takes time, and if the program is blocked until the next input comes in
> it will enter the waitio state. If there is no other process to use the
> available CPU it becomes waitio, which is essentially available CPU
> cycles similar to idle.
>
> What exactly do you think is wrong?
As I run rsync which increases the I/O wait state, the first thing I notice is
that IMAP starts getting slow, users start experiencing failures in sending
e-mail, and the initial exchange for ssh gets significantly longer.
All of these problems have both networking and file I/O in common and I'm trying
to separate out where the problem is coming from. I have run netcat which has
shown that the network throughput is not wonderful but that's a different
problem for me to solve. When I run netcat, there is no degradation of ssh,
IMAP or SMTP response times. the problem shows up if I run CP or rsync internal
source and target. the problem becomes the worst when I'm doing rsync within
the local filesystem and another rsync to an external rsync server. At that
point, the system becomes very close to unusable.
Of course, I can throttle back rsync and regain some usability but I'm backing
up a couple terabytes of information and it's a time-consuming process even with
rsync and would like it to run as quickly as possible. I should probably point
out that the disk array is a relatively small raid five set up with six 1 TB
drives. Never did like raid five especially when it's on a bit of firmware.
Can't wait for ZFS (or its equivalent) on linux to reach production quality.
from where I stand right now, this might be "it sucks but it's perfectly
normal". In a situation with heavy disk I/O, I would expect anything that
accesses the disc to run slowly and in a more naïve moment, I thought that the
GUI wouldn't be hurt by heavy disk I/O and then I remembered that gnome and its
kindred have lots of configuration files to read every time you move the mouse. :-)
Any case, the people that sign my check aren't happy because they spent all his
money on an HP server and performs no better than an ordinary PC. I'm hoping I
can learn enough to give them a cogent explanation if I can't give them a solution.
I appreciate the help.
---eric
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O wait problem with hardware raid
2008-08-31 14:50 ` Eric S. Johansson
@ 2008-08-31 15:48 ` Roger Heflin
2008-08-31 21:08 ` Bill Davidsen
1 sibling, 0 replies; 5+ messages in thread
From: Roger Heflin @ 2008-08-31 15:48 UTC (permalink / raw
To: Eric S. Johansson; +Cc: Bill Davidsen, linux-raid
Eric S. Johansson wrote:
> Bill Davidsen wrote:
>
>> iowait means that there is a program waiting for I/O. That's all.
>
> I was under the impression that I/O wait created a blocking condition.
>
>> Of
>> course when you do a copy (regardless of software) the CPU is waiting
>> for disk transfers. I'm not sure what you think you should debug, i/o
>> takes time, and if the program is blocked until the next input comes in
>> it will enter the waitio state. If there is no other process to use the
>> available CPU it becomes waitio, which is essentially available CPU
>> cycles similar to idle.
>>
>> What exactly do you think is wrong?
>
> As I run rsync which increases the I/O wait state, the first thing I notice is
> that IMAP starts getting slow, users start experiencing failures in sending
> e-mail, and the initial exchange for ssh gets significantly longer.
>
> All of these problems have both networking and file I/O in common and I'm trying
> to separate out where the problem is coming from. I have run netcat which has
> shown that the network throughput is not wonderful but that's a different
> problem for me to solve. When I run netcat, there is no degradation of ssh,
> IMAP or SMTP response times. the problem shows up if I run CP or rsync internal
> source and target. the problem becomes the worst when I'm doing rsync within
> the local filesystem and another rsync to an external rsync server. At that
> point, the system becomes very close to unusable.
>
> Of course, I can throttle back rsync and regain some usability but I'm backing
> up a couple terabytes of information and it's a time-consuming process even with
> rsync and would like it to run as quickly as possible. I should probably point
> out that the disk array is a relatively small raid five set up with six 1 TB
> drives. Never did like raid five especially when it's on a bit of firmware.
> Can't wait for ZFS (or its equivalent) on linux to reach production quality.
>
> from where I stand right now, this might be "it sucks but it's perfectly
> normal". In a situation with heavy disk I/O, I would expect anything that
> accesses the disc to run slowly and in a more naïve moment, I thought that the
> GUI wouldn't be hurt by heavy disk I/O and then I remembered that gnome and its
> kindred have lots of configuration files to read every time you move the mouse. :-)
>
> Any case, the people that sign my check aren't happy because they spent all his
> money on an HP server and performs no better than an ordinary PC. I'm hoping I
> can learn enough to give them a cogent explanation if I can't give them a solution.
>
>
> I appreciate the help.
>
IO wait really means.
You are asking the IO subsystem to do more than it can, and one can do this *NO*
matter how fast ones disks are, in this case throttle the IO processes somewhat
to allow other things to still function.
The only real way out of this is to either throttle things, or get faster disks
and/or raid subsystem.
I would suggest that you (with the machine otherwise unused) test sequential
reads and writes off the raid unit and post the speeds of those test, you will
need to make sure things break the cache to get real numbers, and also post the
number and type of disks that you have, that will give some idea of what the IO
system can do/should be able to do.
If the parts are picked incorrectly things will suck, there are several
different general classes of raid controller and some of them run alot faster
than the others (and cost more). There are at least 3 classes of disks (SATA,
SAS-10k, SAS-15k) and the more expensive ones run much faster.
Even the best of disks/raid controllers will run at full speed on any of the
dual socket machines (from anyone) that has enough PCI-X and/or PCI-e busses to
keep the contollers supplied with data.
How much one can push is related to how many PCI-X and/or PCI-e busses and how
much is shared between the different busses. On most desktop boards *ALL* of
the pci busses are shared (there is only 132MB/second between all PCI slots),
this got better with PCI-e but usually on desktop boards there aren't very many
of them. On the higher end boards (usually 2+ socket, but some single socket
enterprise boards that have PCI-X) often there are several different PCI-X
busses that are not shared, and won't interfere with each other. And on some
higher end AMD boards there are slots/chipsets connected to *BOTH* cpus, so that
increases possible bandwidth if done correctly.
Roger
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O wait problem with hardware raid
2008-08-31 14:50 ` Eric S. Johansson
2008-08-31 15:48 ` Roger Heflin
@ 2008-08-31 21:08 ` Bill Davidsen
1 sibling, 0 replies; 5+ messages in thread
From: Bill Davidsen @ 2008-08-31 21:08 UTC (permalink / raw
To: Eric S. Johansson; +Cc: linux-raid
Eric S. Johansson wrote:
> Bill Davidsen wrote:
>
>
>> iowait means that there is a program waiting for I/O. That's all.
>>
>
> I was under the impression that I/O wait created a blocking condition.
>
I means a process is waiting for I/O, so that one is blocked. And often
it means that heavy disk access will slow down other disk i/o. But the
CPU involved in iowait is available for CPU-bound process (any process
which needs it).
One of the things Linux does poorly is to balance reads and writes. If
you are doing heavy writes you don't have reads jumping the queue and
getting donw in reasonable time. Use of the deadline i/o scheduler may
help this, as may making the dirty ratio _smaller_ to slow writes to let
reads get run.
I played with allowing the reads to bypayy a certain number of writes to
balance performance. It worked beautifully, and I could tune it for any
load, but I never got it to tune itself, so there was a always a
"jackpot case" which weorked WAY worse than standard. Needless to say I
never followed up on it, I haven't had inspiration.
>
>> Of
>> course when you do a copy (regardless of software) the CPU is waiting
>> for disk transfers. I'm not sure what you think you should debug, i/o
>> takes time, and if the program is blocked until the next input comes in
>> it will enter the waitio state. If there is no other process to use the
>> available CPU it becomes waitio, which is essentially available CPU
>> cycles similar to idle.
>>
>> What exactly do you think is wrong?
>>
>
> As I run rsync which increases the I/O wait state, the first thing I notice is
> that IMAP starts getting slow, users start experiencing failures in sending
> e-mail, and the initial exchange for ssh gets significantly longer.
>
> All of these problems have both networking and file I/O in common and I'm trying
> to separate out where the problem is coming from. I have run netcat which has
> shown that the network throughput is not wonderful but that's a different
> problem for me to solve. When I run netcat, there is no degradation of ssh,
> IMAP or SMTP response times. the problem shows up if I run CP or rsync internal
> source and target. the problem becomes the worst when I'm doing rsync within
> the local filesystem and another rsync to an external rsync server. At that
> point, the system becomes very close to unusable.
>
> Of course, I can throttle back rsync and regain some usability but I'm backing
> up a couple terabytes of information and it's a time-consuming process even with
> rsync and would like it to run as quickly as possible. I should probably point
> out that the disk array is a relatively small raid five set up with six 1 TB
> drives. Never did like raid five especially when it's on a bit of firmware.
> Can't wait for ZFS (or its equivalent) on linux to reach production quality.
>
> from where I stand right now, this might be "it sucks but it's perfectly
> normal". In a situation with heavy disk I/O, I would expect anything that
> accesses the disc to run slowly and in a more naïve moment, I thought that the
> GUI wouldn't be hurt by heavy disk I/O and then I remembered that gnome and its
> kindred have lots of configuration files to read every time you move the mouse. :-)
>
> Any case, the people that sign my check aren't happy because they spent all his
> money on an HP server and performs no better than an ordinary PC. I'm hoping I
> can learn enough to give them a cogent explanation if I can't give them a solution.
>
Some of the tuning I suggested may help, perhaps a lot. If you have a
lot of memory you can fill it with dirty buffers and the response will
be a problem.
>
> I appreciate the help.
>
Then let me know if any of the things I suggested help. Someone else may
have other ideas, I would cut the dirty ratio by half and see if it
makes any difference and for better or worse.
>
> ---eric
>
>
--
Bill Davidsen <davidsen@tmr.com>
"Woe unto the statesman who makes war without a reason that will still
be valid when the war is over..." Otto von Bismark
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2008-08-31 21:08 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-08-29 1:27 I/O wait problem with hardware raid Eric S. Johansson
2008-08-30 18:47 ` Bill Davidsen
2008-08-31 14:50 ` Eric S. Johansson
2008-08-31 15:48 ` Roger Heflin
2008-08-31 21:08 ` Bill Davidsen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.