DNFS Configuration process
This process is based on the setup required to attach a ZFS-BA to an Exadata. Unlike the ZFS-7320 a ZFS-BA has more infiniband links connected to the system and so can support greater throughput.
On the ZFS appliance
Create a new project to hold the backup destination ‘MyCompanyBackuptest’
Edit project ‘MyCompanyBackuptest’
General Tab
→ Set ‘Synchronous write bias’ to Throughput
→ Set ‘Mountpoint’ to /export/mydb
Protocols Tab
→ Add nfs exceptions for all of ‘MyCompany’ servers for read/write and root access, using ‘Network’ and giving the individual IP addresses.
192.168.28.7/32 192.168.28.6/32 192.168.28.3/32 192.168.28.2/32
Shares Tab
→ Create filesystems backup1 to backup8
On SPARC node
As root
Check the required kernel parameters are set in /etc/system (done automatically by ssctuner service)
set rpcmod:clnt_max_conns = 8 set nfs:nfs3_bsize = 131072
Set suggested ndd parameters, by creating a script in /etc/rc2.d so they are set after every boot.
root@sc5acn01-d1:/etc/rc2.d# cat S99ndd /usr/sbin/ndd -set /dev/tcp tcp_max_buf 4194304 /usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 2097152 /usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 2097152 /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 16384 /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 16384
Create mountpoints for the backup directories
root@sc5acn01-d1:/# for i in 1 2 3 4 5 6 7 8
do
mkdir /backup${i}
done
Add /etc/vfstab entries for the mountpoints
sc5a-storIB:/export/mydb/backup1 - /backup1 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup2 - /backup2 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup3 - /backup3 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup4 - /backup4 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup5 - /backup5 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup6 - /backup6 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup7 - /backup7 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio sc5a-storIB:/export/mydb/backup8 - /backup8 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
Mount the filesystems and set ownership to oracle:dba
root@sc5acn01-d1:/# for i in 1 2 3 4 5 6 7 8
do
mount /backup${i}
done
root@sc5acn01-d1:/# for i in 1 2 3 4 5 6 7 8
do
chown oracle:dba
/backup${i}
done
As Oracle
Stop any databases running from the ORACLE_HOME where you want to enable DNFS.
Ensure you can remotely authenticate as sysdba, creating a password file using orapwd if required.
Relink for dnfs support
oracle@sc5acn01-d1:/u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk dnfs_on
I was a little uncertain about the oradnfstab entries as most examples relate to a ZFS-BA which has many IB connections and 2 active heads, whereas the 7320 in this case was set in Active/Passive. I created $ORACLE_HOME/dbs/oradnfstab with the following entries.
server:sc5a-storIB path:192.168.28.1 export: /export/mydb/backup1 mount:/backup1 export: /export/mydb/backup2 mount:/backup2 export: /export/mydb/backup3 mount:/backup3 export: /export/mydb/backup4 mount:/backup4 export: /export/mydb/backup5 mount:/backup5 export: /export/mydb/backup6 mount:/backup6 export: /export/mydb/backup7 mount:/backup7 export: /export/mydb/backup8 mount:/backup8
Restart you database and check the alertlog to see if DNFS has been enabled by grepping for NFS.
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0 Wed Mar 26 16:50:43 2014
Backup and restore scripts will need to be adjusted to set suggested underscore parameters and to use the new locations.
oracle@sc5acn01-d1:~/mel$ cat dnfs_backup.rman
startup mount
run
{
sql 'alter system set "_backup_disk_bufcnt"=64';
sql 'alter system set "_backup_disk_bufsz"=1048576';
ALLOCATE CHANNEL ch01 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup1/mydb/%U';
ALLOCATE CHANNEL ch02 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup2/mydb/%U';
ALLOCATE CHANNEL ch03 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup3/mydb/%U';
ALLOCATE CHANNEL ch04 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup4/mydb/%U';
ALLOCATE CHANNEL ch05 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup5/mydb/%U';
ALLOCATE CHANNEL ch06 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup6/mydb/%U';
ALLOCATE CHANNEL ch07 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup7/mydb/%U';
ALLOCATE CHANNEL ch08 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup8/mydb/%U';
backup database TAG='dnfs-backup';
backup current controlfile format '/backup/dnfs-backup/backup-controlfile';
}
oracle@sc5acn01-d1:~/mel$ cat dnfs_restore.rman
startup nomount
restore controlfile from '/backup/dnfs-backup/backup-controlfile';
alter database mount;
configure device type disk parallelism 2;
run
{
sql 'alter system set "_backup_disk_bufcnt"=64';
sql 'alter system set "_backup_disk_bufsz"=1048576';
ALLOCATE CHANNEL ch01 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup1/mydb/%U';
ALLOCATE CHANNEL ch02 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup2/mydb/%U';
ALLOCATE CHANNEL ch03 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup3/mydb/%U';
ALLOCATE CHANNEL ch04 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup4/mydb/%U';
ALLOCATE CHANNEL ch05 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup5/mydb/%U';
ALLOCATE CHANNEL ch06 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup6/mydb/%U';
ALLOCATE CHANNEL ch07 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup7/mydb/%U';
ALLOCATE CHANNEL ch08 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup8/mydb/%U';
restore database from TAG='dnfs-backup';
}
Results of the changes
The timings are based on the longest running backup piece, rather than the wall clock time as this could include other RMAN operations such as re-cataloging files.
| Standard NFS | DNFS | |
| Backup | 2:32:09 | 44:58 |
| Restore | 33:42 | 24:46 |
So, it’s clear from these results that DNFS can have a huge impact on the backup performance and also a positive effect on restore performance.
If you look at the ZFS analytics for the backup, you can see that we were writing approximately 2 GB/s
Also we were seeing approximately 1.2 GB/s read for the restore.
Oracle, Unix and the world at large 