* NFS Performance on Linux 2.2.19 (RedHat version) -- lstat64() ?
@ 2001-12-05 0:27 Joe.Pranevich
2001-12-05 12:11 ` Trond Myklebust
0 siblings, 1 reply; 2+ messages in thread
From: Joe.Pranevich @ 2001-12-05 0:27 UTC (permalink / raw
To: linux-kernel
Hello,
I'm currently involved in porting one of our properties over to Linux from
Solaris and have noticed some disparities in NFS client performance between
Linux and Solaris and I thought I'd ask about it. (The property is almost
100% NFS-based with a couple terabytes of storage on the backend in a sort
of hashed directory structure.) The NFS servers that we are using are all
NetApps.
I have run quick benchmarks (dd'ing from /dev/zero to files on the nfs
mount, then reading them back on other hosts) and have seen good
performance for Linux all around. (The servers that we are using are IBM
dual 800 mhz jobs w/ 2 Gig RAM vs a Sun E450) Under these simple
circumstances, Linux ranges from almost as good as Solaris (writing to NFS)
to almost twice as good (reading from NFS)-- at least how we have things
setup, but not in a precise testing environment. Where my confusion comes
in is that a simple "ls" against a directory with a couple files takes
approximately 4-5 times as long under Linux as on Solaris. Doing a "strace
-c" on the ls process shows that nearly all of the time is being spent in
lstat64(). (And yes, I was using "ls --color=none" :) ) (Our application
does a lot of similar operations, so this is a valid test.)
Is this a known issue or is there are workaround that might allow me to
improve performance? Although I'm running a RedHat kernel (which I
understand is based on one of the AC kernels, partly), there may be a newer
version of the NFS client software that I could try. (I found the
SourceForge site for the server software, but not the client.) Does the 2.4
kernel have significantly better NFS performance or any advantage that
might be worth the effort of switching for benchmarking? (You'd think I
might know the answer to that, wouldn't you...)
(And yes, I know that this is an apples-to-oranges comparison, but since
we're mostly network bound I don't think it's likely to be much of a deal.)
Thanks so much for your help, I've appended some additional information to
this mail that may make things clearer.
Joe Pranevich (wearing his Lycos hat)
Production Service Manager
Lycos Personal Publishing
--
NFS options:
hard,nfsvers=3,intr,rsize=8192,wsize=8192,bg
(similar trials were performed with NFS v2, and both TCP and UDP modes.)
Ethernet cards are eepro100 cards using the Linux "eepro100.o" driver. I
did a benchmark with Intel's e100.o driver and it was twice as slow, even
with hardware checksumming enabled. Note that I'm forcing duplex and speed,
but not doing any other trickery.
strace -c -f ls -alF --color=none
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.82 4.184893 3532 1185 lstat64
0.12 0.005008 218 23 5 open
0.04 0.001773 253 7 getdents
0.01 0.000292 14 21 write
0.00 0.000116 10 12 read
0.00 0.000109 5 24 old_mmap
0.00 0.000074 4 21 close
0.00 0.000057 6 9 munmap
0.00 0.000046 5 10 brk
0.00 0.000029 15 2 socket
0.00 0.000025 5 5 fcntl
0.00 0.000025 2 13 fstat
0.00 0.000019 10 2 2 connect
0.00 0.000016 3 5 mprotect
0.00 0.000014 2 6 fstat64
0.00 0.000010 2 5 lseek
0.00 0.000008 4 2 2 ioctl
0.00 0.000006 6 1 getpid
0.00 0.000005 5 1 time
0.00 0.000002 2 1 personality
------ ----------- ----------- --------- --------- ----------------
100.00 4.192527 1355 9 total
Same directory on Sun: (although their truss adds two seconds to this
command. When done un-trussed using "time", it's only one second. Linux
isn't as affected as such.)
syscall seconds calls errors
_exit .00 1
read .00 1
write .02 11
open .00 6 1
close .00 6
time .00 1
brk .00 60
stat .00 1
fstat .00 3
ioctl .01 2 2
execve .00 1
fcntl .00 2
mmap .00 7
munmap .00 2
memcntl .00 1
llseek .00 1
acl .14 1185
door .00 4
getdents64 .00 53
lstat64 .40 1185
fstat64 .00 2
open64 .00 2
---- --- ---
sys totals: .57 2537 3
usr time: .33
elapsed: 3.05
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: NFS Performance on Linux 2.2.19 (RedHat version) -- lstat64() ?
2001-12-05 0:27 NFS Performance on Linux 2.2.19 (RedHat version) -- lstat64() ? Joe.Pranevich
@ 2001-12-05 12:11 ` Trond Myklebust
0 siblings, 0 replies; 2+ messages in thread
From: Trond Myklebust @ 2001-12-05 12:11 UTC (permalink / raw
To: linux-kernel
>>>>> " " == Joe Pranevich <Joe.Pranevich@corp.terralycos.com> writes:
> but not in a precise testing environment. Where my confusion
> comes in is that a simple "ls" against a directory with a
> couple files takes approximately 4-5 times as long under Linux
> as on Solaris. Doing a "strace -c" on the ls process shows that
> nearly all of the time is being spent in lstat64(). (And yes, I
> was using "ls --color=none" :) ) (Our application does a lot of
> similar operations, so this is a valid test.)
Hi,
Solaris is currently more efficient than we are when it comes to
filehandle and attribute caching, but together with Chuck Lever, we
worked hard over the summer to even the odds.
My main suspect for your report is the NFSv3 operation
READDIRPLUS. Whereas ordinary NFSv2/v3 READDIR returns only the
filename + inode number for each directory entry, READDIRPLUS also
returns the NFS filehandle and full file attributes. With proper
caching, this is sufficient to save an entire LOOKUP RPC call per file
when you get to the stat() calls...
I do have patches that you can test, though unfortunately only for
the 2.4. series. A backport to the 2.2.x series should be possible if
you're able and willing, but my own schedule unfortunately won't
permit it.
As far as I know, the 2.4 patches can be considered `production
quality' (meaning that I haven't seen any bugs reported over the last
couple of months), and I've been using them in production on our own
machines.
If you are interested, please first apply the attribute cache
improvements in
http://www.fys.uio.no/~trondmy/src/2.4.16/linux-2.4.15-cto.dif
and then the readdirplus implementation in
http://www.fys.uio.no/~trondmy/src/2.4.16/linux-2.4.15-rdplus.dif
Cheers,
Trond
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2001-12-05 12:12 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2001-12-05 0:27 NFS Performance on Linux 2.2.19 (RedHat version) -- lstat64() ? Joe.Pranevich
2001-12-05 12:11 ` Trond Myklebust
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.