Tag Archives: exadata

Changing attributes on ASM Diskgroups

You can see the attributes set on your ASM diskgroups in 2 ways, via the view  v$asm_attribute

SQL> select a.name, b.name, b.value from v$asm_attribute b, v$asm_diskgroup a where a.group_number=b.group_number and a.name='DATAX6C1' ;

Or via asmcmd and the lsattr

grid@sc7ach00pd00-d1:~$ asmcmd

ASMCMD> cd DATAX6C1
ASMCMD> lsattr -G DATAXC1 -l

Attributes you might want to pay attention to in an Exadata environment are

  • compatible.advm
  • cell.smart_scan_capable
  • appliance.mode
  • compatible.asm
  • compatible.rdbms

If you manually create a diskgroup via asmca these attributes will not normally be set, and so you may want to go manually set them.

SQL> ALTER DISKGROUP DATAX6C1 SET ATTRIBUTE 'appliance.mode'='TRUE';

The attribute is set immediately, but based on my experience, it does not come into effect until the disk group has  been rebalanced.

SQL> alter diskgroup DATAX6C1 rebalance power 2;

Allowing a non-privileged user to view LDOM configuration (updated for EM13.2)

In Solaris 11.2 (and probably other releases – I haven’t checked) if you try to view the ldom configuration as a non-privileged user you will get the following message

emagent @sscadb01:~$ ldm ls
Authorization failed

You can grant the user the ability to view the LDOM config  using the built in ‘LDoms Review’ profile

# usermod -P 'LDoms Power Mgmt Observability' emagent
# usermod -P 'LDoms Review' emagent

Login and out again and it should work.

emagent@sscadb01:~$ ldm ls
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-cv- UART 128 523776M 3.7% 3.7% 41d 6h 9m
ssccn1-app1 active -n---- 5001 128 512G 0.4% 0.2% 41d 6h 32m

Enterprise Manager 13 is LDOMs aware, and you will need to add this privilege to the agent software owner if you want the virtualization to be shown in this tool.

The Enterprise Manager 13.2 manual now lists slightly different privileges as required for the Enterprise Manager Systems Integration plugin. http://docs.oracle.com/cd/E73210_01/EMADM/GUID-62C91671-4F42-40A0-B929-22CBFEE73672.htm#EMADM15592

# usermod -P 'LDoms Power Mgmt Observability' emagent
# usermod -A solaris.ldoms.read,solaris.ldoms.ldmpower emagent

 

Creating an Infiniband listener on SuperCluster

There is a common IB network between all nodes in a SuperCluster. This is primarily used for access to the ZFS storage appliance, but it can also be used to provide IB access to the database.

I have a 2 node RAC cluster, and both nodes have a network adapter similar to this

# ifconfig stor_ipmp0 
stor_ipmp0: flags=108001000843 mtu 65520 index 2
        inet 192.168.28.2 netmask fffffc00 broadcast 192.168.31.255
        groupname stor_ipmp0

First create entries for the IB VIP in /etc/hosts on all nodes in the cluster and on the hosts who want to access the IB Listener (e.g. Exalogic, application nodes in the SuperCluster)

192.168.28.102  sc5a01-d1-ibvip sc5a01-d1-ibvip.osc.uk.oracle.com
192.168.28.103  sc5a02-d1-ibvip sc5a02-d1-ibvip.osc.uk.oracle.com

Create the required oracle network resources on one node of the cluster. If you’re uncertain about the subnet to specify you can use one of the online calculators such as http://www.subnet-calculator.com/

Define the network interface as a global public cluster interface as root

# /u01/app/11.2.0.3/grid/bin/oifcfg setif -global stor_ipmp0/192.168.28.0:public


# /u01/app/11.2.0.3/grid/bin/oifcfg getif                                       
bondeth0  138.3.16.0  global  public
bondib0  192.168.0.0  global  cluster_interconnect
bondib1  192.168.0.0  global  cluster_interconnect
bondib2  192.168.0.0  global  cluster_interconnect
bondib3  192.168.0.0  global  cluster_interconnect
stor_ipmp0  192.168.28.0  global  public

Create the new network resource in the grid infrastructure as root

# /u01/app/11.2.0.3/grid/bin/srvctl add network -k 2 -S 192.168.28.0/255.255.252.0/stor_ipmp0

# ./srvctl config network
Network exists: 1/138.3.16.0/255.255.254.0/bondeth0, type static
Network exists: 2/192.168.28.0/255.255.252.0/stor_ipmp0, type static

Create your VIP resources as root

# ./srvctl add vip -n sc5acn01-d1 -A sc5a01-d1-ibvip/255.255.252.0/stor_ipmp0 -k 2
# ./srvctl add vip -n sc5acn02-d1 -A sc5a02-d1-ibvip/255.255.252.0/stor_ipmp0 -k 2

Create IB Listener as the grid home owner

$ srvctl add listener -l LISTENER_IB -k 2 -p "TCP:1522,/SDP:1522" -s

Add entries to the grid home tnsnames.ora. There are slightly different configuartions per node, as we will explicitly name the ‘local’ and ‘remote’ nodes. If we had more than 2 nodes, then our entries for LISTENER_IBREMOTE and LISTENER_IPREMOTE would list all of the non-local nodes.

Node 1

## IB Listener configuration


DBM01_IB =
  (DESCRIPTION =
    (LOAD_BALANCE=on)
    (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a01-d1-ibvip)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a02-d1-ibvip)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dbm01)
    )
  )

LISTENER_IBREMOTE =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a02-d1-ibvip)(PORT = 1522))
    )
  )  

LISTENER_IBLOCAL =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a01-d1-ibvip)(PORT = 1522))
            (ADDRESS = (PROTOCOL = SDP)(HOST = sc5a01-d1-ibvip)(PORT = 1522))
    )
  )
LISTENER_IPLOCAL =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a01-d1-vip)(PORT = 1521))
    )
  )
LISTENER_IPREMOTE =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a02-d1-vip)(PORT = 1521))
    )
  )

Node 2


## IB Listener configuration


DBM01_IB =
  (DESCRIPTION =
    (LOAD_BALANCE=on)
    (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a01-d1-ibvip)(PORT = 1522))
    (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a02-d1-ibvip)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dbm01)
    )
  )

LISTENER_IBREMOTE =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a01-d1-ibvip)(PORT = 1522))
    )
  )

LISTENER_IBLOCAL =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a02-d1-ibvip)(PORT = 1522))
            (ADDRESS = (PROTOCOL = SDP)(HOST = sc5a02-d1-ibvip)(PORT = 1522))
    )
  )
LISTENER_IPLOCAL =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a02-d1-vip)(PORT = 1521))
    )
  )
LISTENER_IPREMOTE =
  (DESCRIPTION =
        (ADDRESS_LIST = 
            (ADDRESS = (PROTOCOL = TCP)(HOST = sc5a01-d1-vip)(PORT = 1521))
    )
  )

Start your new listener as the grid home owner

grid@sc5acn01-d1:/u01/app/11.2.0.3/grid/network/admin$ srvctl start listener -l LISTENER_IB

Copy your new tnsnames.ora entries to your $ORACLE_HOME/network/admin/tnsnames.ora

Login as to your database as sysdba and set the listener_networks parameter

SQL> alter system set listener_networks='((NAME=network2)(LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))','((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both sid='*';

This will cause the database to register with the new infiniband listener.