Docs
Notes
04/28/2006 - jasonn
On this page... (hide)
The NetApp Filer has support for many NFSv4 features buried in its options. But does it play well with Linux?
The following features are of particular interest:
Some useful links:
Connect to NetApp web-interface (can also use ssh or telnet):
https://netapp/na_admin/
Set or synchronize time. If you will be using kerberos, this is very important and can save you hours of trouble-shooting later (not that it happened to me... ahem):
Filer -> Set Date/Time
If SSH access is not enabled, enable it:
Secure Admin -> SSH-> Enable/Disable
Create desired volume:
Volumes -> Add
FlexVol type is recommended.
Choose size, etc, then commit.
Modify NFS exports:
NFS -> Manage Exports Select volume created previously Enable: Read-Write Access, Root Access, Security Export Path: use default (remember for mounting later) Read-Write Hosts: 192.168.10.0/24 (host or subnet) Root Hosts: 192.168.10.0/24 (host or subnet) Security: Unix Style/sys, krb5, krb5i, krb5p
Login to NetApp Filer and view NFS options:
ssh root@netapp options nfs
Check that tcp and nfs.v4 are enabled, otherwise turn them on:
options nfs.tcp.enable on options nfs.v4.enable on options nfs.v4.id.domain localdomain
Exit:
<ctrl-d>
Notes:
If using a modern distribution and kernel, standard NFSv4 mounts should work. New features may require a very current distribution. Fedora stays fairly current with CITI NFSv4 patches.
As of these notes:
| distro | kernel version | nfs-utils version |
| CITI builds | 2.6.17-rc2 | nfs-utils-1.0.8-rc4 |
| Fedora Core 5 | 2.6.16-1 | nfs-utils-1.0.8.rc2 |
| Gentoo | 2.6.15-r1 | nfs-utils-1.0.6-r6 |
| Debian Stable | 2.6.12 | nfs-common-1.0.6 |
| Debian Etch | 2.6.15-1 | nfs-common-1.0.7-9 |
Fedora Core 5 works out-of-the-box with newer NFSv4 features. For others, you may need to download and compile nfs-utils from source.
To mount using nfsv4, specify 'nfs4' as the filesystem type instead of 'nfs'. Also, note that the export paths may be different due to nfsv4's pseudo-filesystem.
Mount from command line:
mount -t nfs4 -o intr,proto=tcp netapp:/vol/volname /mnt/nfs4
Or, create an entry in /etc/fstab:
netapp:/vol/volname /mnt/nfs4 nfs4 hard,intr,proto=tcp,noauto 0 0
Check your mount options:
mount (will show options you passed, including sec) cat /proc/mounts (shows your options and defaults, doesn't show sec)
(./testserver.py --maketree netapp:/vol/nfsv4 all -v --paddednull)
| Skipped | Failed | Warned | Passed | |
| 13 | 169 | 20 | 376 | |
This test seems to show some differences between Linux and NetApp NFSv4. Newpynfs is considered very noisy in general so...
Kerberos authentication will work for nfs2, nfs3, and nfs4. We just care about nfs4.
First, setup a KDC:
http://cryptnet.net/fdp/admin/kerby-infra/en/kerby-infra.html.
KDC setup should go something like this:
install mit-krb5 modify /etc/krb5.conf modify /etc/krb5kdc/kdc.conf kdb5_util -r <realm> create -s /etc/init.d/mit-krb5kdc start kadmin.local -q 'addprinc root/admin' echo "root/admin *" > /etc/krb5kdc/kadm5.acl
Then, make sure kdc, servers and clients have the same time. NTP or ntpdate via cronjob is good for this:
install ntp modify /etc/ntp.conf
On KDC:
Create krb5.keytabs for the NetApp Filer and NFS client using kadmin.local:
kadmin.local addprinc -randkey nfs/netapp.osdl.org ktadd -e des-cbc-crc:normal -k /tmp/netapp-krb5.keytab nfs/netapp.osdl.org addprinc -randkey nfs/nfs01.osdl.org ktadd -e des-cbc-crc:normal -k /tmp/nfs01-krb5.keytab nfs/nfs01.osdl.org
Notes:
Put krb5.conf and correct krb5.keytab in /etc/ for server and client.
For the NetApp, nfs mount /vol/vol0 (or use ftp after enabling):
mount -t nfs netapp:/vol/vol0 /mnt/netapp cp /etc/krb5.conf /mnt/netapp/etc/krb5.conf cp /tmp/netapp-krb5.keytab /mnt/netapp/etc/krb5.keytab
Enable Kerberos on NetApp Filer:
ssh root@netapp
nfs setup
use: unix kdc, realm=NFSV4, principal=netapp.osdl.org
exit: <ctrl-d>
On the client, you will also want to make sure the needed services start correctly:
rpc.idmapd rpc.gssd portmap (if you have older nfs-utils)
mount from command line:
mount -t nfs4 -o intr,tcp,sec=krb5 netapp:/vol/volname /mnt/nfs4
Or, create an entry in /etc/fstab:
netapp:/vol/volname /mnt/nfs4 nfs4 defaults,hard,intr,tcp,sec=krb5,noauto 0 0
Notes:
(dd if=/dev/zero of=./test.out bs=1k count 2000000)
| Mode | MB/s | NetApp CPU % | |
| sys | 68.8 | ~80% | |
| krb5 | 68.2 | ~80% | |
| krb5i | 41.6 | >90% | |
| krb5p | 15.8 | >90% | |
Krb5 doesn't show a performance hit versus sys. However, krb5i and krb5p have an obvious penalty.
Both appear quite processor-intensive.
(300 mb tarball, 30895 files)
| Mode | Extract MB/s | Create MB/s | |
| sys | 5.80 | 9.74 | |
| krb5 | 4.90 | 8.33 | |
| krb5i | 3.99 | 5.88 | |
| krb5p | 2.46 | crash | |
A little more difference between krb5 and sys on the tar test.
Krb5p support on linux looks pretty buggy. A kernel crash occurred on tarball creation, as well as during other tests. Problems only occurred client-side.
(./server -a -t netapp-2 -o hard,intr,proto=tcp -p /vol/nfsv4)
| Mode | Parent Pass | Child Pass | |
| sys | 49/49 | 64/64 | |
| krb5 | 49/49 | 64/64 | |
| krb5i | 49/49 | 64/64 | |
| krb5p | 49/49 | 64/64 | |
Same results for each security mode (everything passed), which isn't too surprising considering the popularity of connectathon as a testing tool.
(iozone -+q 1 -i 0 -i 1 -g 1G -Race -f /mnt/netapp/iozone.tmp)
iozone sys results
iozone krb5 results
iozone krb5i results
iozone krb5p results
As expected, sys and krb5 performed well while krb5i and krb5p had an obvious performance impact. Performace is similar to the dd tests, but the iozone results provide more detail.
NFSv4 ACLs are a new feature. The idea is to be (semi-)compatibile with with both Posix and Windows ACLs.
Enable ACLs on NetApp Filer:
ssh root@netapp options nfs.v4.acl.enable on <ctrl-d>
I'm not yet sure how to set ACLs on files stored on a NetApp Filer. Setting via an NFS mount doesn't work:
[root@nfs09 netapp]# setfacl -m u:root:rw test setfacl: test: Operation not supported
Linux client-side NFSv4 ACL tools are still a work-in-progress.
Further testing pending more knowledge...
Delegations are a feature new with NFSv4.
By allowing the client to cache files with guaranteed read or
exclusive write access, performance should improve.
Enable delegations on NetApp Filer:
ssh root@netapp options nfs.v4.read_delegation on options nfs.v4.write_delegation on <ctrl-d>
Possible methods of testing delegations might include:
Another suggestion was a kernel compile, the thinking being that reading header files might be sped up with delegations. Even though a kernel compile is more processor intensive, maybe there will be some difference?
(make w/ default config on 2.4.17-rc2)
| Mode | Time |
| normal | 49m38s |
| read deleg | 49m08s |
| read/write deleg | 49m08s |
Any benefit seems negligible in this test.
I've also tried writing a small perl script to open and read a file repeatedly, but didn't find a difference. The file just gets cached in memory so same without delegations. Opening, reading, and closing doesn't cache in memory, but also seems to give the same results with or without delegations.
Further testing pending...