Category Archives: storage

Factory resetting a ZFS Appliance when you can’t login to the system.

Note – this process will completely destroy all configuration and data on the ZFS Appliance. I only need to do this when a system is returned to me with an unknown IP and password, but I can get onto the ILOM. Please contact Oracle Support before doing this and truly understand what you are doing.

Normally, if you can login to a system you can issue the command ‘maintenance system factoryreset’ to get this result.  DO NOT DO THIS IF YOU HAVE ANY DATA YOU NEED ON THE APPLIANCE.

First – ensure that you are on the current version of Firmware/BIOS on the ZFS Appliance. This can be checked using MOS document 1174698.1 Oracle ZFS Storage Appliance: How to check the SP BIOS revision level as you could encounter problems editing the grub menu.

Login to the SP/ILOM


   -> start /SYS

   -> start /SP/console

 

Wait for the GRUB menu which will be editable for 10 seconds.

Within the 10 seconds, Press ‘e’ on the keyboard

Select the line kernel … To navigate, use ‘v’ to go down and ‘^’ to go up.

Press “e” on keyboard to edit this line

Append ” -c” to this line (spell as “space minus c”)

change this  :

 kernel$ /platform/i86pc/kernel/$ISADIR/unix -B zfs-bootfs=system/368,console=ttya,ttya-mode="9600,8,n,1,-"

to this      :

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B zfs-bootfs=system/368,console=ttya,ttya-mode="9600,8,n,1,-" -c

Press <return>

Finally press “b” on keyboard to reboot.

 

This will print the following lines :

SunOS Release 5.11 Version ak/generic@2013.06.05.6.8,1-1.1 64-bit
Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.

   Use is subject to license terms.

>  Clearing appliance configuration ...... done.

   Configuring network devices ... done.

The system will then wipe all of the previous configuration,  reboot, and allow you to reconfigure the networking and root password.

Note: The change you made to the grub boot menu is temporary, and so you should not need to go back in and edit it again.

 

Advertisements

Enabling DNFS and configuring a RMAN backup to a ZFS 7320

DNFS Configuration process

This process is based on the setup required to attach a ZFS-BA to an Exadata. Unlike the ZFS-7320 a ZFS-BA has more infiniband links connected to the system and so can support greater throughput.

On the ZFS appliance

Create a  new project to hold the backup destination ‘MyCompanyBackuptest’

Edit project ‘MyCompanyBackuptest’
General Tab

→ Set ‘Synchronous write bias’ to Throughput
→ Set ‘Mountpoint’ to /export/mydb

Protocols Tab

→ Add nfs exceptions for all of ‘MyCompany’ servers for read/write and root access, using ‘Network’ and giving the individual IP addresses.

192.168.28.7/32
192.168.28.6/32
192.168.28.3/32
192.168.28.2/32

Shares Tab

→ Create filesystems backup1 to backup8

On SPARC node

As root

Check the required kernel parameters are set in /etc/system (done automatically by ssctuner service)

set rpcmod:clnt_max_conns = 8
set nfs:nfs3_bsize = 131072

Set suggested ndd parameters, by creating a script in /etc/rc2.d so they are set after every boot.

root@sc5acn01-d1:/etc/rc2.d# cat S99ndd
/usr/sbin/ndd -set /dev/tcp tcp_max_buf 4194304
/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 2097152
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 2097152
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 16384
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q0 16384

Create mountpoints for the backup directories

root@sc5acn01-d1:/# for i in 1 2 3 4 5 6 7 8 
do 
mkdir /backup${i} 
done

Add /etc/vfstab entries for the mountpoints

sc5a-storIB:/export/mydb/backup1 - /backup1 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup2 - /backup2 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup3 - /backup3 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup4 - /backup4 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup5 - /backup5 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup6 - /backup6 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup7 - /backup7 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio
sc5a-storIB:/export/mydb/backup8 - /backup8 nfs - yes rw,bg,hard,nointr,rsize=1048576,wsize=1048576,proto=tcp,vers=3,forcedirectio

Mount the filesystems and set ownership to oracle:dba

root@sc5acn01-d1:/# for i in 1 2 3 4 5 6 7 8 
do 
mount /backup${i} 
done
root@sc5acn01-d1:/# for i in 1 2 3 4 5 6 7 8 
do 
chown oracle:dba 
/backup${i} 
done

As Oracle

Stop any databases running from the ORACLE_HOME where you want to enable DNFS.
Ensure you can remotely authenticate as sysdba, creating a password file using orapwd if required.
Relink for dnfs support

oracle@sc5acn01-d1:/u01/app/oracle/product/11.2.0.3/dbhome_1/rdbms/lib$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk dnfs_on

I was a little uncertain about the oradnfstab entries as most examples relate to a ZFS-BA which has many IB connections and 2 active heads, whereas the 7320 in this case was set in Active/Passive. I created $ORACLE_HOME/dbs/oradnfstab with the following entries.

server:sc5a-storIB path:192.168.28.1
export: /export/mydb/backup1 mount:/backup1
export: /export/mydb/backup2 mount:/backup2
export: /export/mydb/backup3 mount:/backup3
export: /export/mydb/backup4 mount:/backup4
export: /export/mydb/backup5 mount:/backup5
export: /export/mydb/backup6 mount:/backup6
export: /export/mydb/backup7 mount:/backup7
export: /export/mydb/backup8 mount:/backup8

Restart you database and check the alertlog to see if DNFS has been enabled by grepping for NFS.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0
 Wed Mar 26 16:50:43 2014

Backup and restore scripts will need to be adjusted to set suggested underscore parameters and to use the new locations.

oracle@sc5acn01-d1:~/mel$ cat dnfs_backup.rman
startup mount
run
{
sql 'alter system set "_backup_disk_bufcnt"=64';
sql 'alter system set "_backup_disk_bufsz"=1048576';
ALLOCATE CHANNEL ch01 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup1/mydb/%U';
ALLOCATE CHANNEL ch02 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup2/mydb/%U';
ALLOCATE CHANNEL ch03 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup3/mydb/%U';
ALLOCATE CHANNEL ch04 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup4/mydb/%U';
ALLOCATE CHANNEL ch05 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup5/mydb/%U';
ALLOCATE CHANNEL ch06 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup6/mydb/%U';
ALLOCATE CHANNEL ch07 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup7/mydb/%U';
ALLOCATE CHANNEL ch08 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup8/mydb/%U';
backup database TAG='dnfs-backup';
backup current controlfile format '/backup/dnfs-backup/backup-controlfile';
}
oracle@sc5acn01-d1:~/mel$ cat dnfs_restore.rman
startup nomount
restore controlfile from '/backup/dnfs-backup/backup-controlfile';
alter database mount;
configure device type disk parallelism 2;
run
{
sql 'alter system set "_backup_disk_bufcnt"=64';
sql 'alter system set "_backup_disk_bufsz"=1048576';
ALLOCATE CHANNEL ch01 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup1/mydb/%U';
ALLOCATE CHANNEL ch02 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup2/mydb/%U';
ALLOCATE CHANNEL ch03 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup3/mydb/%U';
ALLOCATE CHANNEL ch04 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup4/mydb/%U';
ALLOCATE CHANNEL ch05 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup5/mydb/%U';
ALLOCATE CHANNEL ch06 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup6/mydb/%U';
ALLOCATE CHANNEL ch07 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup7/mydb/%U';
ALLOCATE CHANNEL ch08 DEVICE TYPE DISK connect 'sys/welcome1@mydb' format '/backup8/mydb/%U';
restore database from TAG='dnfs-backup';
}

Results of the changes

The timings are based on the longest running backup piece, rather than the wall clock time as this could include other RMAN operations such as re-cataloging files.

Standard NFS DNFS
Backup 2:32:09 44:58
Restore 33:42 24:46

So, it’s clear from these results that DNFS can have a huge impact on the backup performance and also a positive effect on restore performance.

If you look at the ZFS analytics for the backup, you can see that we were writing approximately 2 GB/s

backup

Also we were seeing approximately 1.2 GB/s read for the restore.
restore

ZFS Appliance NFS exceptions

I had a situation where I wanted to restrict access to a project on my ZFS storage appliance (7320) to a small list of hosts on a private network. The project needs to be accessible r/w, with root permissions from 4 hosts that I need to specify by IP address.

192.168.28.2     
192.168.28.3    
192.168.28.6   
192.168.28.7

However, other hosts in the 192.168.28.X/22 range must not be able to mount the share.
The way to achieve this is to lock down the permissions and then explicitly grant access to the systems you need. You have 3 ways of specifying the names of hosts for exceptions:-

  • Host(FQDN) or Netgroup – This requires you to have your private hostnames registered in DNS, which was not possible in my case. You CANNOT enter an IP address in this field.
  • DNS Domain – all of my hosts are in the same domain, so this was not fine grained enough.
  • Network – Counter-intuitively, it is network that will allow me to specify individual IP addresses, using a CIDR netmask that allows only 1 host (the netmask does not have to match that of the underlying interface)

First thing – set the default NFS share mode to ‘NONE’ so that non-excepted hosts cannot mount the share.

Then add exception for each host, using a /32 netmask which limits it to a single IP.

zfs

So, a quick test. This one should work

root@myhost-d1:/stage# ifconfig stor_ipmp0
stor_ipmp0: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 65520 index 3
        inet 192.168.28.2 netmask fffffc00 broadcast 192.168.31.255
        groupname stor_ipmp0
root@myhost-d1:/# mount -f nfs -o rw 192.168.28.1:/export/stage /mnt
root@myhost-d1:/# df -k /mnt
Filesystem           1024-blocks        Used   Available Capacity  Mounted on
192.168.28.1:/export/stage
                     10737418209          31 10737418178     1%    /mnt

This one should fail

root@myhost-d3:~# ifconfig stor_ipmp0
stor_ipmp0: flags=8001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 65520 index 3
        inet 192.168.28.4 netmask fffffc00 broadcast 192.168.31.255
        groupname stor_ipmp0
root@myhost-d3:~#  mount -f nfs -o rw 192.168.28.1:/export/stage /mnt
nfs mount: mount: /mnt: Permission denied

Zoning a Brocade Switch Using WWNs

Way back in the mists of time I used to use port based zoning on Brocade switches, however, I started having problems with this and newer storage systems (almost certainly pilot error!). I needed to zone some switches for a customer’s piece of work and this time I thought I’d get with the future and use WWN based zoning.

So, in my setup I have 2 hosts, each with 2 connections per switch, and 2 storage arrays with 1 connection to the switch.

swd77:admin> switchshow
switchName:     swd77
switchType:     34.0
switchState:    Online
switchMode:     Native
switchRole:     Principal
switchDomain:   1
switchId:       fffc01
switchWwn:      10:00:00:05:1e:02:a2:08
zoning:         OFF
switchBeacon:   OFF

Area Port Media Speed State
==============================
  0   0   id    N4   Online    F-Port  20:14:00:a0:b8:29:f5:56 <- Storage Array 1
  1   1   id    N4   Online    F-Port  20:16:00:a0:b8:29:cd:b4 <- Storage Array 2
  2   2   id    N4   Online    F-Port  21:00:00:24:ff:20:3a:f6 <- Host A
  3   3   id    N4   Online    F-Port  21:00:00:24:ff:20:3a:e0 <- Host A
  4   4   --    N4   No_Module
  5   5   --    N4   No_Module
  6   6   id    N4   No_Light
  7   7   id    N4   No_Light
  8   8   id    N4   Online    F-Port  21:00:00:24:ff:20:3b:92 <- Host B
  9   9   id    N4   Online    F-Port  21:00:00:24:ff:25:6d:ac <- Host B
 10  10   id    N4   No_Light
 11  11   id    N4   No_Light
 12  12   id    N4   No_Light
 13  13   id    N4   No_Light
 14  14   --    N4   No_Module
 15  15   --    N4   No_Module

Create aliases for your hosts and storage arrays

swd77:admin> alicreate host1_a,"21:00:00:24:ff:20:3b:92"
swd77:admin> alicreate host1_b,"21:00:00:24:ff:25:6d:ac"
swd77:admin> alicreate host2_a,"21:00:00:24:ff:20:3a:f6"
swd77:admin> alicreate host2_b,"21:00:00:24:ff:20:3a:e0"
swd77:admin> alicreate "a6140","20:14:00:a0:b8:29:f5:56"
swd77:admin> alicreate "b6140","20:16:00:a0:b8:29:cd:b4"

Create Zones to include your alias

swd77:admin> zonecreate "port2","host1_a; a6140; b6140"
swd77:admin> zonecreate "port3","host1_b; a6140; b6140"
swd77:admin> zonecreate "port8","host2_a;  a6140; b6140"
swd77:admin> zonecreate "port9","host2_b;  a6140; b6140"

Create a configuration for your zones and save it

swd77:admin> cfgcreate "customer1","port2; port3; port8; port9"
swd77:admin> cfgsave
You are about to save the Defined zoning configuration. This
action will only save the changes on Defined configuration.
Any changes made on the Effective configuration will not
take effect until it is re-enabled.
Do you want to save Defined zoning configuration only?  (yes, y, no, n): [no] yes

When you’re happy with your configuration, enable it.

swd77:admin> cfgenable customer1
You are about to enable a new zoning configuration.
This action will replace the old zoning configuration with the
current configuration selected.
Do you want to enable 'customer1' configuration  (yes, y, no, n): [no] y
zone config "customer1" is in effect
Updating flash ...

Check at the OS level to see if you can see all your required volumes.

Setting the vtoc on multiple disks in Solaris

You know how it is, you get a rack of storage which you’ve turned into multiple identical  LUNs. Then you need to feed these to  ASM, which requires you to start your partitions at cylinder > 0

Firstly you need to make sure all of your disks have a valid label. You can do this either by manually selecting each disk you need to label in format and answering ‘yes’ when prompted, or you could create a command file to feed into format.

The file should list the disk number (from format) and issue the command to label the disk

disk 12
label
disk 14
label
disk 72
label

I have a simple script to generate this file, but it does assume you have something distinctive about the disks so you can grep them out of the format output. In this case it is all the devices on c26, but it could be something like the cylinder count or disk manufacturer.

#!/bin/sh
# Generate disk list and label command for format
# creates a file called mylabel.cmd
# Assumes that you have something distinctive about the disks to be labelled
# e.g controller number (c26)
# feed the output file from this (after review) to format
# format -f mylabel.cmd
for i in `echo "quit" |format 2>&1 | grep c26 | awk -f. '{print $1}'|sed 's/\(.*\)./\1/'`
do
        echo "disk $i" >> mylabel.cmd
        echo "label" >> mylabel.cmd
done

You then take this file (after reviewing it – don’t go labelling the wrong disks!)  and feed it into format

# format -f mylabel.cmd

Use format to partition your first disk (in this example /dev/rdsk/c0t0d0s2) and then use fmthard to put this partition onto another disk.

# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2

Wrap it up in while loop and you can have your entire rack setup with the correct vtoc in seconds.

Formatting a 2540 using SSCS

Sometimes to speed up the setup of identical disk systems you might decide to use SSCS rather than the Common Array Manager GUI to configure them.

All syntax mentioned here can be found in http://dlc.sun.com/pdf/820-4192-12/820-4192-12.pdf

First login to your CAMS server using sccs

root@c14-48 # sscs login -h c14-48 -u root

This only tells you if the login has failed, if you don’t get a response, everything is ok and you’re now connected to your CAMS server until it times out.

Select the profile for the storage you are setting up. You can get a list of profiles on your array (esal-2540-2 in my example) using

root@c14-48 # sscs list -a esal-2540-2 profile
Profile: Oracle_OLTP_HA
Profile: Oracle_DSS
Profile: Oracle_9_VxFS_HA
Profile: Sun_SAM-FS
Profile: High_Performance_Computing
Profile: Oracle_9_VxFS
Profile: Oracle_10_ASM_VxFS_HA
Profile: Random_1
Profile: Sequential
Profile: Sun_ZFS
Profile: Sybase_OLTP_HA
Profile: Sybase_DSS
Profile: Oracle_8_VxFS
Profile: Mail_Spooling
Profile: Microsoft_NTFS_HA
Profile: Microsoft_Exchange
Profile: Sybase_OLTP
Profile: Oracle_OLTP
Profile: VxFS
Profile: Default
Profile: NFS_Mirroring
Profile: NFS_Striping
Profile: Microsoft_NTFS
Profile: High_Capacity_Computing

You can get more detail on a profile using

 root@c14-48 #  sscs list -a esal-2540-2 profile Oracle_9_VxFS_HA
 Profile: Oracle_9_VxFS_HA
Profile In Use: no
Factory Profile: yes
Description:
Oracle 9 over VxFS (High Availability)
RAID Level: 1
Segment Size: 128 KB
Read Ahead: on
Optimal Number of Drives: variable
Disk Type: SAS
Dedicated Hot Spare: no

The next step is to create my pool ‘bt-poc’

The syntax for the command is

sscs create -a  -p  pool  
root@c14-48 #   sscs create -a  esal-2540-2 -p Oracle_9_VxFS_HA pool bt-poc

Logically, at this point you want to create a logical disk, however you can’t create one directly but you can create one implicitly as part of the volume creation

sscs create -a  -p  -s 

-n volume
root@c14-48 # sscs create -a  esal-2540-2 -p bt-poc -s 15gb -n 6 volume vol1 

Check the name of you new virtual disk using

root@c14-48 # sscs list -a esal-2540-2 vdisk
Virtual Disk: 1

root@c14-48 # sscs list -a esal-2540-2 vdisk 1
Virtual Disk: 1
Status: Optimal
State: Ready
Number of Disks: 6
RAID Level: 1
Total Capacity: 836.690 GB
Configured Capacity: 15.000 GB
Available Capacity: 821.690 GB
Array Name: esal-2540-2
Array Type: 2540
Disk Type: SAS
Maximal Volume Size: 821.690 GB
Associated Disks:
Disk: t85d01
Disk: t85d02
Disk: t85d03
Disk: t85d04
Disk: t85d05
Disk: t85d06
Associated Volumes:
Volume: vol1

Since I need another 4 identical volumes on this virtual disk I can be lazy just script up the creation.

root@c14-48 # for i in 2 3 4 5
do
sscs create -a esal-2540-2 -p bt-poc -s 15gb -v 1 volume vol${i}
done

The sccs commands are asynchronous – they return before the action is completed, so long running tasks like creating a RAID5 volume will still be running while you create your volumes on it.

Can delete any mix ups simply

root@c14-48 # sscs delete -a  esal-2540-2 volume vol3

At this point, you can either map your volumes to the default storage domain, and all hosts connected to the storage will be able to see all the volumes, or you can do LUN mapping and limit which hosts can see which volumes.

Map to the default storage domain

for i in 1 2 3 4 5
do
sscs map -a esal-2540-2 volume vol${i}
done

Create host based mappings

Create your hosts

root@c14-48 # sscs create -a esal-2540-2 host dingo 
root@c14-48 # sscs create -a esal-2540-2 host chief 

Create the initators that map to the World Wide Number (WWN) for the Host Bus Adaptor (HBA) of each machine.

First find your WWN – you can do this either by looking on the storage switch if you have one, or on the hosts that will be accessing the storage.

Looking on the host you issue the command fcinfo hba-port, and look for the HBA port WWN associated with the correct fibre channel devices. I’ve highlighted the entries in red for clarity.

dingo # fcinfo hba-port
HBA Port WWN: 21000003ba9b3679
OS Device Name: /dev/cfg/c1
Manufacturer: QLogic Corp.
Model: 2200
Firmware Version: 2.01.145
FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.15 04/03/22
Type: L-port
State: online
Supported Speeds: 1Gb
Current Speed: 1Gb
Node WWN: 20000003ba9b3679
HBA Port WWN: 210000e08b09965e
OS Device Name: /dev/cfg/c3
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Firmware Version: 3.03.27
FCode/BIOS Version: fcode: 1.13;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b09965e
HBA Port WWN: 210100e08b29965e
OS Device Name: /dev/cfg/c4
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Firmware Version: 3.03.27
FCode/BIOS Version: fcode: 1.13;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200100e08b29965e

root@c14-48 # sscs create -a esal-2540-2 -w 210000e08b09965e -h dingo initiator dingo-1 
root@c14-48 # sscs create -a esal-2540-2 -w 210000e08b29965e -h dingo initiator dingo-2 

root@c14-48 # sscs map -a esal-2540-2 -v vol1,vol3 host dingo

And there you have it – you’ve formatted and mapped your volumes without needing the web interface.