Strange behaviour of listener_networks and scan listener

I had an Exalogic and a Sparc SuperCluster T4-4 connected together by infiniband for a set of tests. This meant that I was able to enable SDP and IP over the infiniband network.

To configure it, I had followed the instructions in the Exalogic Manual.

After setting the listener_networks parameter I checked whether the services had registered correctly with the scan listener. Expected behaviour is to see all instances registered with all 3 scan listeners

– Set your environment to the GRID_HOME

- Check which nodes are running the scan listener as you can only interrogate the listener from that node

 oracle@ssca01:~$ srvctl status scan
 SCAN VIP scan1 is enabled
 SCAN VIP scan1 is running on node ssca03
 SCAN VIP scan2 is enabled
 SCAN VIP scan2 is running on node ssca04
 SCAN VIP scan3 is enabled
 SCAN VIP scan3 is running on node ssca01

So on ssca01, I can check the status of LISTENER_SCAN1 ..
And it had  no services registered. Strange. Checked all of my listeners and only LISTENER_SCAN3 had any services registered

oracle@ssca01:~$ /u01/app/11.2.0.3/grid/bin/lsnrctl status LISTENER_SCAN3
LSNRCTL for Solaris: Version 11.2.0.3.0 - Production on 06-SEP-2012 15:20:43
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
 STATUS of the LISTENER
 ------------------------
 Alias                     LISTENER_SCAN3
 Version                   TNSLSNR for Solaris: Version 11.2.0.3.0 - Production
 Start Date                30-AUG-2012 15:59:53
 Uptime                    6 days 23 hr. 20 min. 49 sec
 Trace Level               off
 Security                  ON: Local OS Authentication
 SNMP                      OFF
 Listener Parameter File   /u01/app/11.2.0.3/grid/network/admin/listener.ora
 Listener Log File         /u01/app/11.2.0.3/grid/log/diag/tnslsnr/ssca01/listener_scan3/alert/log.xml
 Listening Endpoints Summary...
 (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=137.3.16.87)(PORT=1521)))
 Services Summary...
 Service "ibs" has 3 instance(s).
 Instance "IBS2", status READY, has 1 handler(s) for this service...
 Instance "IBS3", status READY, has 1 handler(s) for this service...
 Instance "IBS4", status READY, has 1 handler(s) for this service...
 The command completed successfully

I was confident my scan address was registered correctly in the DNS

oracle@ssca01:~$ nslookup ssca-scan
 Server:         138.4.34.5
 Address:        138.4.34.5#53
Name:   ssca-scan.blah.com
 Address: 137.3.16.89
 Name:   ssca-scan.blah.com
 Address: 137.3.16.88
 Name:   ssca-scan.blah.com
 Address: 137.3.16.87

I looked on Oracle Support and I could find no other reports of this problem, but then only a small proportion of customers will be running in this configuration.

However, I did find a note 1448717.1 that documented a similar problem with the remote_listener parameter.

So, I amended my tnsnames.ora file so that my LISTENER_IPREMOTE alias included the 3 scan ip addresses

#LISTENER_IPREMOTE =
#  (DESCRIPTION =
#    (ADDRESS = (PROTOCOL = TCP)(HOST = ssca-scan.blah.com)(PORT = 1521))
#  )

LISTENER_IPREMOTE =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 137.3.16.87)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = 137.3.16.88)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = 137.3.16.89)(PORT = 1521))
  )

You can trace the PMON registration process by setting the following database event

alter system set events=’immediate trace name listener_registration level 3′;

and then issue a alter system register; to force pmon to re-register to listeners.

This will produce a trace file in background_dump dest

Looking through this logfile I saw it was still trying to register with the SCAN address.

 Remote listeners:
  0 - (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ssca-scan.osc.uk.oracle.com)(PORT=1521)))
       state=1, err=0
       nse[0]=0, nse[1]=0, nte[0]=0, nte[1]=0, nte[2]=0
       ncre=0
       endp=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ssca-scan.osc.uk.oracle.com)(PORT=1521)))
         flg=0x0 nse=12545

At this point I realised that the tnsnames.ora must only be checked at certain times, such as database startup. So, I restarted my database.

Success! On checkng all of my scan listeners they all had services registered.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s