Enabling access to EM Express in Oracle Cloud Infrastructure

By default EM Express is not available on the DB Systems (12c or 18.1) in Oracle Cloud Infrastructure, but it can be enabled.

In this example I will be allowing access to it over the public internet, from a Subnet that has an internet gateway defined.

https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/monitoringDB.htm#Ports

First enable the EM Express application on your chosen port. I am going to use 5500 (the default)

As the oracle user, set your environment and connect to your database. Then set the https port to the required value.

SQL> exec DBMS_XDB_CONFIG.SETHTTPSPORT(5500);

PL/SQL procedure successfully completed.

SQL> quit

Then check HTTP is running on the listener , and also note the location of the xdb_wallet directory

$ lsnrctl status |grep HTTP

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=tb.sub############.vcn######.oraclevcn.com)(PORT=5500))(Security=(my_wallet_directory=/u01/app/oracle/admin/poc1_fra2zn/xdb_wallet))(Presentation=HTTP)(Session=RAW))

 

Check permissions on wallet files – the  asmadmin group needs to be able to read the these. If you do not do this you will receive a strange error in the web browser saying that the website cannot be authenticated (firefox) or a message about outdated TLS (IE)
https://docs.cloud.oracle.com/iaas/Content/knownissues.htm#Unable

The location of the wallet files is listed in the lsnrctl status command above.

[oracle@tb ~]$ cd /u01/app/oracle/admin/poc1_fra2zn/xdb_wallet
[oracle@tb xdb_wallet]$ ls -latr
total 16
drwxr-x--- 6 oracle oinstall 4096 Jul 10 09:35 ..
-rw------- 1 oracle asmadmin 3867 Jul 10 09:35 ewallet.p12
-rw------- 1 oracle asmadmin 3912 Jul 10 09:35 cwallet.sso
drwxr-x--- 2 oracle asmadmin 4096 Jul 10 09:35 .

[oracle@tb xdb_wallet]$ chmod 640 *

Open the iptables firewall port as root

# iptables -I INPUT 8 -p tcp -m state --state NEW -m tcp --dport 5500 -j ACCEPT -m comment --comment "Required for EM Express"

Save the iptables configuration, reload the firewall and check that the port 5500 is open.

# service iptables save
# service iptables reload
# service iptables status |grep 5500

 

In the cloud, navigate to your VCN, and open up the security rule for port 5500

Add a new stateful ingress rule for port 5500. It would be better to limit the possible source CIDR to the network used by your customer but this will work.

blah

 

 

Advertisements

Hacking SLOB to run on Solaris

Kevin Closson’s Silly Little Oracle Benchmark https://kevinclosson.net/slob/ is not ported to Solaris. This means that to get it working is not as easy as it should be. Kevin himself suggests that you use a small linux host to run the tool if your database is running on a non-supported operating system.

I’m going to track the changes I make here.

setup.sh

First problem – grep!

./setup.sh test 16
SLOB 2.4.0
FATAL : 2018.05.15-14:40:57 : Usage : ./setup.sh.orig: <tablespace name> <number of SLOB schemas to create and load>
FATAL : 2018.05.15-14:40:57 : Option 2 must be an integer

This is caused by the function f_is_int requiring the gnu grep command behaviour. Simplest way to this is to change the grep to point to /usr/gnu/bin/grep in this script.

Next problem..

Once this change was made, I was then able to create the first schema ( yay!) but then when it came to make the remaining 15, it failed.

NOTIFY : 2018.05.15-14:37:27 : Waiting for background batch 1. Loading up to user11
FATAL : 2018.05.15-14:37:29 : 
FATAL : 2018.05.15-14:37:29 : f_flag_abort: Triggering abort
FATAL : 2018.05.15-14:37:29 : 
FATAL : 2018.05.15-14:37:30 : 
FATAL : 2018.05.15-14:37:30 : f_flag_abort: Triggering abort
FATAL : 2018.05.15-14:37:30 : 
FATAL : 2018.05.15-14:37:40 : 
FATAL : 2018.05.15-14:37:40 : f_flag_abort: Triggering abort
FATAL : 2018.05.15-14:37:40 : 
FATAL : 2018.05.15-14:37:40 : 
FATAL : 2018.05.15-14:37:40 : f_check_abort_flag: discovered abort flag
FATAL : 2018.05.15-14:37:40 : 
FATAL : 2018.05.15-14:37:40 : Aborting SLOB setup. See /export/home/oracle/mel/SLOB/cr_tab_and_load.out

 

Handily there is an error message in the logfile

ALTER TABLE cf1 MINIMIZE RECORDS_PER_BLOCK
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-00039: error during periodic action
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT




Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
FATAL : 2018.05.15-14:37:40 :
FATAL : 2018.05.15-14:37:40 : f_setup: Failed to load user4 SLOB table
FATAL : 2018.05.15-14:37:40 :

I’m going to assume the clue is in the error and increase the PGA aggregate limit

SQL> alter system set pga_aggregate_limit=36G scope=both sid='*';

After this change, the script ran to the end successfully.

 

$ diff setup.sh setup.sh.orig
30,31d29
< 
< 
86c84
< if ( ! echo $s | /usr/gnu/bin/grep -q "^-\?[0-9]*$" )
---
> if ( ! echo $s | grep -q "^-\?[0-9]*$" )
250c248
< if ( echo "$cdb" | /usr/gnu/bin/grep -q "YES" > /dev/null 2>&1 )
---
> if ( echo "$cdb" | grep -q "YES" > /dev/null 2>&1 )
254c252
< if ( echo "$cdb" | /usr/gnu/bin/grep -q "NO" > /dev/null 2>&1 )
---
> if ( echo "$cdb" | grep -q "NO" > /dev/null 2>&1 )
410c408
< done | sqlplus -s "$constring" 2>&1 | tee -a $fname | /usr/gnu/bin/grep -i "dropped" | wc -l | while read num_processed
---
> done | sqlplus -s "$constring" 2>&1 | tee -a $fname | grep -i "dropped" | wc -l | while read num_processed

runit.sh

So – we already know that the scripts need gnu grep, especially in the function f_is_int. Safest option will be  to change every occurrence of grep with /usr/gnu/bin/grep.

Try running it with a single schema

 ./runit.sh 1

The duration of the run is based on the slob.conf variable RUN_TIME. The default is 300 seconds, but I am dropping it to 90 while I am doing debugging.

This ran successfully (I think!). Certainly AWR reports were created.

There was one obvious failure – the mpstat output was not generated.

$ cat mpstat.out 
mpstat: -P expects a number
Usage: mpstat [-aqm] [-A core|soc|bin] [-k key1,[key2,...]] [-o num] [-p | -P processor_set] [-T d|u] [-I statfile | -O statfile ] [interval [count]]

Ok – so this is the line causing the problem.

 ( mpstat -P ALL 3 > mpstat.out 2>&1) &

On Linux mpstat without the -P flag will give you summary output, -P separates the output by processor.  On Solaris the default is to show one line per CPU, so I just need to change the line to

 ( mpstat 3 > mpstat.out 2>&1) &

also check for other invocations of mpstat

Next - try with 2 sessions

./runit.sh 2

Arrgh. That fails with

: List of monitored sqlplus PIDs written to /tmp/.SLOB.2018.05.15.153524/13174.f_wait_pids.out.
usage: ps [ -aAdefHlcjLPyZ ] [ -o format ] [ -t termlist ]
 [ -u userlist ] [ -U userlist ] [ -G grouplist ]
 [ -p proclist ] [ -g pgrplist ] [ -s sidlist ] [ -z zonelist ] [-h lgrplist]
 'format' is one or more of:
 user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid ctid
 pri opri pcpu pmem vsz rss rssprivate rssshared osz nice class time etime stime zone zoneid env
 f s c lwp nlwp psr tty addr wchan fname comm args projid project pset lgrp

This wasn’t obvious where the fault was happening. So lets add the -x flag to the first line of our script to try and find the line causing our problems.

++ sed /PID/d
++ wc -l
+++ cat /tmp/.SLOB.2018.05.15.153852/15451.f_wait_pids.out
++ sed 's/[^0-9]//g'
++ ps -p 32526 32527
usage: ps [ -aAdefHlcjLPyZ ] [ -o format ] [ -t termlist ]
 [ -u userlist ] [ -U userlist ] [ -G grouplist ]
 [ -p proclist ] [ -g pgrplist ] [ -s sidlist ] [ -z zonelist ] [-h lgrplist]
 'format' is one or more of:
 user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid ctid
 pri opri pcpu pmem vsz rss rssprivate rssshared osz nice class time etime stime zone zoneid env
 f s c lwp nlwp psr tty addr wchan fname comm args projid project pset lgrp
++ return 0
+ tmp=0

So there is something with the ps -p command that is not happy in solaris. I  can reproduce the problem at my command line, by providing ps -p with multiple pids.


# ps -p 7 8
usage: ps [ -aAdefHlcjLPyZ ] [ -o format ] [ -t termlist ].....

If we look at the ps man page, on solaris it says

Some options accept lists as arguments. Items in a list can be either
separated by commas or else enclosed in quotes and separated by commas
or spaces. Values for proclist and grplist must be numeric.

so this means the following will work, but not the format within runit.sh

# ps -p 7,8
 PID TTY TIME CMD
 7 ? 127:47 zpool-rp
 8 ? 1:39 kmem_tas
# ps -p "7 8"
 PID TTY TIME CMD
 7 ? 127:47 zpool-rp
 8 ? 1:39 kmem_tas

 

So – I need to find the line with ps command, and try to change the format presented.

while ( ps -p $pidstring > /dev/null 2>&1 )

 ps -fp $pidstring

both occur in function f_wait_pids

but, they are actually being presented with the list of pids built up in line 1440

sqlplus_pids="${sqlplus_pids} $!"

So – how do I go about getting either a comma or a quote in the string that is passed to the function f_wait_pids?

Well, rather than destroying something that Kevin may rely on later, I add a line below 1440 to generate the pid list with commas

 melsqlplus_pids="${melsqlplus_pids},$!"

This is pretty ugly, as it means my string leads with a comma. However it seems that Solaris doesn’t care about this.

Now I need to add this to the function call on line 1493

if ( ! f_wait_pids "$(( SCHEMAS * THREADS_PER_SCHEMA ))" "$RUN_TIME" "$WORK_LOOP" "$sqlplus_pids"  "$melsqlplus_pids")

Of course I need to add something to pick up that new 5th argument to the function, so within the function itself

local melpidstring="$5"

while ( ps -p "$melpidstring" > /dev/null 2>&1 )

ps -fp "$melpidstring"

So this all seems ok.. but then I discover some more little rats in f_count_pid.

So once again – create my own custom usage of the pid list. There probably is a MUCH simpler way to do this, I’m just building it up as I go along. So – here are the changes I have made to runit.sh to get it to execute without error

 

oracle@sc8avm-25:~/mel/SLOB$ diff runit.sh runit.sh.orig
1c1
< #!/bin/bash 
---
> #!/bin/bash
45c45
< if ( ! echo "$s" | /usr/gnu/bin/grep -q "^-\?[0-9]*$" ) 
---
> if ( ! echo "$s" | grep -q "^-\?[0-9]*$" ) 
275c275
< for string in 'iostat -xm 3' 'mpstat 3' 'vmstat 3'
---
> for string in 'iostat -xm 3' 'mpstat -P ALL 3' 'vmstat 3'
300c300
< ls -l /proc/${tmp}/fd | /usr/gnu/bin/grep deleted
---
> ls -l /proc/${tmp}/fd | grep deleted
416c416
< sqlplus $user/${user}${non_admin_connect_string} <<EOF 2>/dev/null | sed 's/^.* FATAL/FATAL/g' | /usr/gnu/bin/grep FATAL > $tmpfile
---
> sqlplus $user/${user}${non_admin_connect_string} <<EOF 2>/dev/null | sed 's/^.* FATAL/FATAL/g' | grep FATAL > $tmpfile
457c457
< if ( /usr/gnu/bin/grep FATAL "$tmpfile" > /dev/null 2>&1 )
---
> if ( grep FATAL "$tmpfile" > /dev/null 2>&1 )
485d484
< local melpidstring="$5"
488d486
< local meltmpfile="${SLOB_TEMPDIR}/${RANDOM}.MEL${FUNCNAME}.out"
502d499
< echo "$melpidstring" > $meltmpfile 2>&1
508d504
< f_msg NOTIFY "List of monitored sqlplus PIDs with commas written to ${meltmpfile}."
514c510
< tmp=`f_count_pids "$meltmpfile"`
---
> tmp=`f_count_pids "$tmpfile"`
542,544c538,539
< echo "This is the pidstring value $pidstring"
< echo "This is the melpidstring value $melpidstring"
< while ( ps -p "$melpidstring" > /dev/null 2>&1 )
---
> 
> while ( ps -p $pidstring > /dev/null 2>&1 )
551c546
< ps -fp "$melpidstring"
---
> ps -fp $pidstring
875c870
< if ( ! echo "$tmp" | /usr/gnu/bin/grep -q '\-' > /dev/null 2>&1 )
---
> if ( ! echo "$tmp" | grep -q '\-' > /dev/null 2>&1 )
1397c1392
< ( mpstat 3 > mpstat.out 2>&1) &
---
> ( mpstat -P ALL 3 > mpstat.out 2>&1) &
1446d1440
< melsqlplus_pids="${melsqlplus_pids},$!"
1496c1490,1491
< if ( ! f_wait_pids "$(( SCHEMAS * THREADS_PER_SCHEMA ))" "$RUN_TIME" "$WORK_LOOP" "$sqlplus_pids" "$melsqlplus_pids" )
---
> 
> if ( ! f_wait_pids "$(( SCHEMAS * THREADS_PER_SCHEMA ))" "$RUN_TIME" "$WORK_LOOP" "$sqlplus_pids" )

Tomorrow – time to get someone else to run it and see if it behaves how they expect.

 

What packages did that incorporation just install?

So you just installed a Solaris package incorporation and you want to work out what it actually included..

First .. find out when your package was installed

root@host-8-200:/var/log/pkg# pkg history 
START OPERATION CLIENT OUTCOME
2017-10-12T14:51:03 set-property transfer module Succeeded
2017-10-12T14:51:03 image-create transfer module Succeeded
2017-10-12T14:51:04 refresh-publishers transfer module Succeeded
2017-10-12T14:51:20 rebuild-image-catalogs transfer module Succeeded
2017-10-12T14:51:27 install transfer module Succeeded
2017-10-12T15:33:18 install pkg Succeeded
2017-10-12T15:39:07 install pkg Succeeded

We are going to dig into the install that occurred at 15:39

root@host-8-200:/var/log/pkg# pkg history -t 2017-10-12T15:39:07 -l

This gives a  really long listing.. but the key part for me was headed

 

Package version changes:
None -> pkg://solaris/developer/build/make@0.5.11,5.11-0.175.2.0.0.34.0:20140303T132010Z
None -> pkg://solaris/developer/assembler@0.5.11,5.11-0.175.3.9.0.2.0:20160528T012706Z
None -> pkg://solaris/group/prerequisite/oracle/oracle-rdbms-server-12-1-preinstall@0.5.11,5.11-0.175.3.11.0.4.0:20160804T020607Z

This shows that 3 packages were installed as they went from version ‘None’ to an actual version number.

What solaris packages do I need to use runinstaller?

If you are running runinstaller on Solaris 11, and you have not installed the solaris-large-server incorporation you are likely to be missing some packages that you need to install the Oracle database.

oracle@host-8-200:/var/tmp/database$ pkg list |grep server
 group/system/solaris-small-server 0.5.11-0.175.3.11.0.4.0 i--

So – let’s start at what you need to run runinstaller..

oracle@host-8-200:/var/tmp/database$ ./runInstaller
 Starting Oracle Universal Installer...

Checking Temp space: must be greater than 547 MB. Actual 1956079 MB Passed
 Checking swap space: must be greater than 150 MB. Actual 1964769 MB Passed
 Checking monitor: must be configured to display at least 256 colors
 >>> Could not execute /usr/bin/xdpyinfo Failed <<<<

Some requirement checks failed. You must fulfill these requirements before

continuing with the installation,

Continue? (y/n) [n] n

You could just install solaris -large-server or just a minimal packages (if you prefer) to allow runinstaller to run.

# pkg install xauth x11/diagnostic/x11-info-clients library/motif terminal/xterm

 

If you receive an error

VM warning: PICL (libpicl.so.1) is missing. Performance will not be optimal.

you can resolve this by installing the package picl

# pkg install /system/picl

Daxstat in Solaris 11.3/SuperCluster problems

One of the big challenges with the Software in Silicon features is actually monitoring them to see if they are actually doing anything. To monitor the DAX on the SPARC M7 chip you used to have to use busstat commands, and then interpret them (not easy! the documentation is not very thorough).

In later releases of Solaris 11.3 (SRU 19 onwards?) there is a command called daxstat which you can use to see in a much more human readable form the activity on your DAX.https://docs.oracle.com/cd/E86824_01/html/E54764/daxstat-1m.html

However, when I went to use it on my SuperCluster I hit a problem… it was failing with an error I couldn’t understand.

# daxstat
Traceback (most recent call last):
 File "/usr/bin/daxstat", line 969, in <module>
 sys.exit(main())
 File "/usr/bin/daxstat", line 962, in main
 return process_opts()
 File "/usr/bin/daxstat", line 905, in process_opts
 dax_ids, dax_queue_ids = derive_dax_opts(args, parser)
 File "/usr/bin/daxstat", line 844, in derive_dax_opts
 dax_ids = find_ids(query, parser, None)
 File "/usr/bin/daxstat", line 683, in find_ids
 all_dax_kstats = RCU.list_objects(kbind.Kstat(), query)
 File "/usr/lib/python2.7/vendor-packages/rad/connect.py", line 391, in list_objects
 a RADInterface object
 File "/usr/lib/python2.7/vendor-packages/rad/client.py", line 213, in _raise_error
 packer.pack_int((timestamp % 1000000) * 1000)
rad.client.NotFoundError: Error listing com.oracle.solaris.rad.kstat:type=Kstat: not found (3)

It *should* have worked on my current version of Solaris

# pkg list entire
NAME (PUBLISHER) VERSION IFO
entire 0.5.11-0.175.3.22.0.3.0 i--

 

So, I did some tweaking. I am not sure which of steps 1 or 2 actually fixed my problem, as it seemed to need the reboot to activate my ‘fix’

Step 1 – Make sure you have the Remote Administration Daemon packages installed. https://docs.oracle.com/cd/E53394_01/html/E54825/index.html I installed the package group group/system/management/rad/rad-server-interfaces to make sure I wasn’t missing anything.

 # pkg list |grep rad
group/system/management/rad/rad-server-interfaces 0.5.11-0.175.3.0.0.30.0 i--
system/management/rad 0.5.11-0.175.3.21.0.4.0 i--
system/management/rad/client/rad-c 0.5.11-0.175.3.21.0.3.0 i--
system/management/rad/client/rad-java 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/client/rad-python 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-dlmgr 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-files 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-kstat 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-network 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-panels 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-smf 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-time 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-usermgr 0.5.11-0.175.3.17.0.4.0 i--
system/management/rad/module/rad-zfsmgr 0.5.11-0.175.3.17.0.1.0 i--
system/management/rad/module/rad-zonemgr 0.5.11-0.175.3.22.0.1.0 i--

Step 2 – Make sure the RAD service is running (mine was disabled)

# svcs -a |grep rad

online 9:22:10 svc:/system/rad:local
online 9:22:10 svc:/system/logadm-upgrade:default
online 9:22:10 svc:/system/rad:local-http

It was still failing to work though.. and I couldn’t work out why. In desperation I tried a reboot and the command stopped failing.

# daxstat 1
No data available to display.

Run workload that uses the DAX and then it will populate the output.

# daxstat 1
DAX commands fallbacks input output %busy
 0 466 9 4.0M 0.0M 0
 1 469 13 4.0M 0.0M 0
 2 462 12 2.0M 0.0M 0
 3 461 16 4.0M 0.0M 0
 4 473 9 4.0M 0.0M 0
 5 457 8 2.0M 0.0M 0
 6 459 6 2.0M 0.0M 0
 7 465 10 4.0M 0.0M 0

Changing Exadata Cell access parameters

One challenge is with Exadata cells in a lab environment is that they are secure! This means that it has long lock out times in the event of an incorrect login and tough lock settings. You can manually change these.. but every time you update your cell there is a chance they will be reset.

A more permanent way is to use /opt/oracle.cellos/host_access_control on each storage cell. https://docs.oracle.com/cd/E58626_01/html/E58630/z40036a01393423.html#scrolltoc

For example, if you want to drop the lock time in the event of a failed login from 10 minutes to a more manageable 60 seconds  you would issue the command

/opt/oracle.cellos/host_access_control pam-auth --lock=60

You can combine multiple pam-auth commands on the same line.. e.g. if I also want to say that the cell only remembers one previous password I could say

/opt/oracle.cellos/host_access_control pam-auth --lock=60 --remember=1

 

There are a lot of options for this tool – you can set the system back to secure defaults, or make it even more secure, such as locking an account after a single failed login!

Factory resetting a ZFS Appliance when you can’t login to the system.

Note – this process will completely destroy all configuration and data on the ZFS Appliance. I only need to do this when a system is returned to me with an unknown IP and password, but I can get onto the ILOM. Please contact Oracle Support before doing this and truly understand what you are doing.

Normally, if you can login to a system you can issue the command ‘maintenance system factoryreset’ to get this result.  DO NOT DO THIS IF YOU HAVE ANY DATA YOU NEED ON THE APPLIANCE.

First – ensure that you are on the current version of Firmware/BIOS on the ZFS Appliance. This can be checked using MOS document 1174698.1 Oracle ZFS Storage Appliance: How to check the SP BIOS revision level as you could encounter problems editing the grub menu.

Login to the SP/ILOM


   -> start /SYS

   -> start /SP/console

 

Wait for the GRUB menu which will be editable for 10 seconds.

Within the 10 seconds, Press ‘e’ on the keyboard

Select the line kernel … To navigate, use ‘v’ to go down and ‘^’ to go up.

Press “e” on keyboard to edit this line

Append ” -c” to this line (spell as “space minus c”)

change this  :

 kernel$ /platform/i86pc/kernel/$ISADIR/unix -B zfs-bootfs=system/368,console=ttya,ttya-mode="9600,8,n,1,-"

to this      :

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B zfs-bootfs=system/368,console=ttya,ttya-mode="9600,8,n,1,-" -c

Press <return>

Finally press “b” on keyboard to reboot.

 

This will print the following lines :

SunOS Release 5.11 Version ak/generic@2013.06.05.6.8,1-1.1 64-bit
Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.

   Use is subject to license terms.

>  Clearing appliance configuration ...... done.

   Configuring network devices ... done.

The system will then wipe all of the previous configuration,  reboot, and allow you to reconfigure the networking and root password.

Note: The change you made to the grub boot menu is temporary, and so you should not need to go back in and edit it again.

 

Installing Enterprise Manager agent on Oracle Linux 6.7 SPARC

I have access to an EM13 Enterprise Manager server, and I am going to add my Oracle Linux 6.7 SPARC to this system for monitoring.

First – check that you have the latest plugins and agents installed for the platform.

Screenshot-Self Update: Agent Software - Oracle Enterprise Manager - Mozilla Firefox

Next, on the hosts, create a user to ‘own’ the agent software

[root@host-8-160 ~]# groupadd -g 10001 oinstall
[root@host-8-160 ~]# useradd -g oinstall -s /bin/bash -d /home/agent13 -m agent13
[root@host-8-160 ~]# passwd agent13
Changing password for user agent13.

Create a directory structure for the software

[root@host-8-160 ~]# mkdir -p /u01/app
[root@host-8-160 ~]# chgrp -R oinstall /u01
[root@host-8-160 ~]# chmod g+rwx /u01

Now, back in Enterprise Manager go.. Setup -> Add Target -> Add Target Manually -> Install Agent on Host.

Enter the fully qualified domain name of your host, and the correct Platform

Screenshot-Add Host Targets : Host and Platform - Mozilla Firefox

 

Enter the installation location

Enter the credentials for agent13 and root user and hit next.

Then you can hit deploy agent.

 

Post install configuration/Worries

 

The agent installed successfully – but the host target is not being marked as available.

Looking at the output of emctl status agent I have 2 concerns.

[agent13@host-8-160 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.2.0.0.0
OMS Version : 13.2.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/agent13/ngc13/agent_inst
Agent Log Directory : /u01/app/agent13/ngc13/agent_inst/sysman/log
Agent Binaries : /u01/app/agent13/ngc13/agent_13.2.0.0.0
Core JAR Location : /u01/app/agent13/ngc13/agent_13.2.0.0.0/jlib
Agent Process ID : 20636
Parent Process ID : 20497
Agent URL : https://host-8-160.blah.com:3876/emd/main/
Local Agent URL in NAT : https://host-8-160.blah.com:3876/emd/main/
Repository URL : https://ngc13c.blah.com:4901/empbs/upload
Started at : 2017-06-13 10:52:26
Started by user : agent13
Operating System : Linux version 4.1.12-94.3.4.el6uek.sparc64 (sparcv9)
Number of Targets : (none)
Last Reload : (none)
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 98.09%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2017-06-13 10:57:31
Last successful heartbeat to OMS : 2017-06-13 10:57:31
Next scheduled heartbeat to OMS : 2017-06-13 10:58:31

 

There are no targets, and there has not been a successful upload.

[agent13@host-8-160 bin]$ ./emctl pingOMS
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD pingOMS completed successfully

[agent13@host-8-160 bin]$ ./emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

If I look in my targets.xml it is pretty empty

[agent13@host-8-160 ngc13]$ cat ./agent_inst/sysman/emd/targets.xml
<Targets AGENT_TOKEN="67DBE4C8ECBA03FA5DC991893B75619C55C9B1CEACAA6ED68074AB9C65CFF973"/>

[agent13@host-8-160 bin]$ ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
[agent13@host-8-160 bin]$

On the enterprise manager server I had errors similar to this

 
Metric evaluation error start - Unable to connect to the agent at https://host-8-161.blah.com:3876/emd/main/ [No route to host]

Tried putting that URL into my browser… cannot connect to it.

Firewall! DOH! Of course!

Temporarily disabled the iptables firewall

[root@host-8-161 /]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]

Now I can connect to the agent address in my browser.

 

So.. the choice is disable the firewall, or alter the rules. As I’m in a lab, I’m going straight to disabling the firewall.

[root@host-8-161 /]# chkconfig iptables off

Now, try to get the agent to generate the internal target list (host, ORACLE_HOME)

[agent13@host-8-161 bin]$ ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

[agent13@host-8-161 bin]$ ./emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
2017-06-13 12:00:37,234 [main] WARN oracle.sysman.gcagent.comm.agent.http.SSLInit - User requested cipher suite SSL_RSA_WITH_RC4_128_MD5, which is not supported for SSLContext TLSv1.2
2017-06-13 12:00:37,242 [main] WARN oracle.sysman.gcagent.comm.agent.http.SSLInit - User requested cipher suite SSL_RSA_WITH_RC4_128_SHA, which is not supported for SSLContext TLSv1.2

[agent13@host-8-161 bin]$ ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

Now when I look at my targets.xml it has entries

[agent13@host-8-161 agent_inst]$ cat ./sysman/emd/targets.xml
<Targets AGENT_TOKEN="6A415CAF76EC952756AE3BC675B0080ADAEE066B3F9B10B1B4A6410870130843">
 <Target TYPE="host" NAME="host-8-161.blah.com" DISPLAY_NAME="host-8-161.osc.uk.oracle.com" EMD_URL="https://host-8-161.blah.com:3876/emd/main/" TIMEZONE_REGION="" IDENTIFIER="TARGET_GUID=51D4595ED5982DE8E0539011038AD7DB"/>
 <Target TYPE="oracle_emd" NAME="host-8-161.blah.com:3876" DISPLAY_NAME="host-8-161.blah.com:3876" EMD_URL="https://host-8-161.blah.com:3876/emd/main/" TIMEZONE_REGION="" IDENTIFIER="TARGET_GUID=0958C84AFB17CE4D3F9FB85C81250615"/>
 <Target TYPE="oracle_home" NAME="agent13c1_1_host-8-161.blah.com_1639" DISPLAY_NAME="agent13c1_1_host-8-161.blah.com_1639" EMD_URL="https://host-8-161.blah.com:3876/emd/main/" TIMEZONE_REGION="" IDENTIFIER="TARGET_GUID=C13E4BCE40F4509C3FC788A3C08EED68">
 <Property NAME="HOME_TYPE" VALUE="O"/>
 <Property NAME="INVENTORY" VALUE="/u01/app/oraInventory"/>
 <Property NAME="INSTALL_LOCATION" VALUE="/u01/app/agent13/ngc13/agent_13.2.0.0.0"/>
 </Target>
</Targets>

When I look at the hosts in Enterprise Manager they are now marked as up.

 

linux

Thoughts and other questions..

The agent13 user on the primary domain has automatically been given the permission to run read only ldm commands (similar to the privileges that need to be manually applied to the user on Solaris).

Unlike on other platforms (e.g. SuperCluster) the hierachy of LDOMs does not seem to be recorded.

Installing and configuring DTRACE on Oracle Linux SPARC

DTRACE is one of the killer features of Solaris, and allows you to programmatically monitor system statistics and diagnose performance issues.  See https://github.com/opendtrace/toolkit for toolkit scripts so you do not have to write your own.

Dtrace is not shipped with the install media. You need to manually download the rpms from

http://www.oracle.com/technetwork/server-storage/linux/downloads/linux-dtrace-2800968.html

 

 

Dtrace is very kernel version dependent. Do not yum update your kernel without checking that dtrace is available for that release or you will have problems!

You can use yum to install the rpms

[root@host-8-161 sfw]# yum localinstall dtrace*
Loaded plugins: downloadonly, ulninfo
Setting up Local Package Process
Examining dtrace-utils-0.6.0-3.el6.sparc64.rpm: dtrace-utils-0.6.0-3.el6.sparc64
Marking dtrace-utils-0.6.0-3.el6.sparc64.rpm to be installed
Examining dtrace-utils-devel-0.6.0-3.el6.sparc64.rpm: dtrace-utils-devel-0.6.0-3.el6.sparc64
Marking dtrace-utils-devel-0.6.0-3.el6.sparc64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package dtrace-utils.sparc64 0:0.6.0-3.el6 will be installed
---> Package dtrace-utils-devel.sparc64 0:0.6.0-3.el6 will be installed
--> Processing Dependency: libdtrace-ctf-devel > 0.4.0 for package: dtrace-utils-devel-0.6.0-3.el6.sparc64
--> Running transaction check
---> Package libdtrace-ctf-devel.sparc64 0:0.5.0-3.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 dtrace-utils sparc64 0.6.0-3.el6 /dtrace-utils-0.6.0-3.el6.sparc64 766 k
 dtrace-utils-devel
 sparc64 0.6.0-3.el6 /dtrace-utils-devel-0.6.0-3.el6.sparc64 77 k
Installing for dependencies:
 libdtrace-ctf-devel
 sparc64 0.5.0-3.el6 public_ol6_latest 15 k

Transaction Summary
================================================================================
Install 2 Packages (+1 Dependent package)

Total size: 857 k
Total download size: 15 k
Installed size: 877 k
Is this ok [y/N]: y
Downloading Packages:
libdtrace-ctf-devel-0.5.0-3.el6.sparc64.rpm | 15 kB 00:00 
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Public key for libdtrace-ctf-devel-0.5.0-3.el6.sparc64.rpm is not installed
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
 Userid : "Oracle OSS group (Open Source Software group) <build@oss.oracle.com>"
 Fingerprint: 4214 4123 fecf c55b 9086 313d 72f9 7b74 ec55 1f03
 Package : 6:oraclelinux-release-6Server-7.0.8.sparc64 (@anaconda-OracleLinuxServer-201705232044.sparc64/6.7)
 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Is this ok [y/N]: y
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
 Installing : dtrace-utils-0.6.0-3.el6.sparc64 1/3 
 Installing : libdtrace-ctf-devel-0.5.0-3.el6.sparc64 2/3 
 Installing : dtrace-utils-devel-0.6.0-3.el6.sparc64 3/3 
 Verifying : dtrace-utils-devel-0.6.0-3.el6.sparc64 1/3 
 Verifying : libdtrace-ctf-devel-0.5.0-3.el6.sparc64 2/3 
 Verifying : dtrace-utils-0.6.0-3.el6.sparc64 3/3

Installed:
 dtrace-utils.sparc64 0:0.6.0-3.el6 dtrace-utils-devel.sparc64 0:0.6.0-3.el6

Dependency Installed:
 libdtrace-ctf-devel.sparc64 0:0.5.0-3.el6

Complete!

 

 

At this point when you run dtrace it doesn’t show anything useful and has no probes available.

[root@host-8-160 sfw]# dtrace -l
dtrace: module license 'CDDL' taints kernel.
Disabling lock debugging due to kernel taint
 ID PROVIDER MODULE FUNCTION NAME
 1  dtrace                   BEGIN
 2  dtrace                   END
 3  dtrace                   ERROR

 

You need to manually load the kernel modules for the probes and providers you want to use.  There is a list of providers in the the Oracle Linux Dtrace Tutorial manual and the Oracle Linux Dtrace Guide

A summary of what is available at the time of writing (June 2017) is below.

Provider Kernel Module Description
dtrace dtrace Provides probes that relate to DTrace itself, such as BEGIN, ERROR, and END. You can use these probes to initialize DTrace’s state before tracing begins, process its state after tracing has completed, and handle unexpected execution errors in other probes.
fasttrap fasttrap Supports user-space tracing of DTrace-enabled applications.
io sdt Provides probes that relate to data input and output. The io provider enables quick exploration of behavior observed through I/O monitoring tools such as iostat.
proc sdt Provides probes for monitoring process creation and termination, LWP creation and termination, execution of new programs, and signal handling.
profile profile Provides probes associated with an interrupt that fires at a fixed, specified time interval. These probes are associated with the asynchronous interrupt event rather than with any particular point of execution. You can use these probes to sample some aspect of a system’s state.
sched sdt Provides probes related to CPU scheduling. Because CPUs are the one resource that all threads must consume, the sched provider is very useful for understanding systemic behavior.
syscall systrace Provides probes at the entry to and return from every system call. Because system calls are the primary interface between user-level applications and the operating system kernel, these probes can offer you an insight into the interaction between applications and the system.

You can manually load the probes

[root@host-8-160 log]# modprobe -a dtrace profile systrace sdt dt_test fasttrap

However, you may want to write  startup script to automatically load the probes at boot time if the dtrace device exists.

[root@host-8-160 sfw]# cat /etc/sysconfig/modules/dtrace.modules
 

#!/bin/sh
if [ ! -c /dev/dtrace/dtrace ] ; then
         exec /sbin/modprobe -a dtrace profile systrace sdt dt_test
 fi

[root@host-8-160 sfw]# chmod 755 /etc/sysconfig/modules/dtrace.modules

 

Once the module has been loaded into the kernel, you can list all probes using

[root@host-8-160 sfw]# dtrace -l

or for just a single provider

[root@host-8-160 etc]# dtrace -l -P io
 ID  PROVIDER MODULE    FUNCTION NAME
 266 io       vmlinux   end_bio_bh_io_sync done
 267 io       vmlinux   _submit_bh start
 269 io       vmlinux   __wait_on_buffer wait-start
 270 io       vmlinux   __wait_on_buffer wait-done


You may also want to look at the information in the manual about setting the permissions on the dtrace helper device to allow code that runs as a user other than root to be recorded.

Scribbled notes on installing the Oracle database on Oracle Linux 6.7 SPARC

I had a very short time to play with my Oracle Linux SPARC box before I handed it to my customers, so I only had a very quick attempt to install the Oracle RDBMS and start a database. I did only a very basic install using database storage on filesystem, and allowed the installer to create the DB. So these notes are even more scrappy than usual.

While not yet a certified platform, you can download the Oracle Database 12.1.0.2 for Oracle Linux 6.7 SPARC images on e-delivery. There is not a publicly available install document, so I’m going to follow the install guide for Linux

Preparing for the Install

https://docs.oracle.com/database/121/LADBI/olinrpm.htm#LADBI7477

I couldn’t find the pre-installation rpm for Linux-SPARC on ULN. So I am going to have to follow the documentation and hope I have all the packages.

Verify openssh is installed

[root@host-8-161 yum.repos.d]# rpm -qa |grep ssh
 openssh-clients-5.3p1-117.el6.sparc64
 openssh-5.3p1-117.el6.sparc64
 openssh-server-5.3p1-117.el6.sparc64
 libssh2-1.4.2-2.el6_7.1.sparc64

Check that the required packages are installed.

https://docs.oracle.com/database/121/LADBI/pre_install.htm#LADBI7534

I added the following packages..

  • compat-libcap1
  • compat-libstdc++-33

On the installation media the is an additional rpm to install cvuqdisk-1.0.9-1.rpm – but this requires an oracle user….

So.. lets create my user and groups for now

[root@host-8-161 rpm]# groupadd -g 1001 oinstall
 [root@host-8-161 rpm]# groupadd -g 1002 dba
 [root@host-8-161 rpm]# useradd -g dba -G oinstall -s /bin/bash -d /home/oracle/ oracle
 [root@host-8-161 rpm]# passwd oracle

Now retry installing the package

[root@host-8-161 rpm]# rpm -i cvuqdisk-1.0.9-1.rpm

Using default group oinstall to install package

Create directories

[root@host-8-161 rpm]# mkdir /u01
[root@host-8-161 rpm]# mkdir -p /u01/app/oracle
[root@host-8-161 rpm]# chown -R oracle:dba /u01

 

Set the Oracle user resource limits

[root@host-8-161 rpm]# cat /etc/security/limits.conf

# End of file
 oracle soft nofile 1024
 oracle hard nofile 65536
 oracle soft nproc 2047
 oracle hard nproc 16384
 oracle soft stack 10240
 oracle hard stack 10240
 oracle soft memlock 3145728
 oracle hard memlock 3145728

 

  • Set the display and try running runinstaller
  • .. fails with PRVF-0002 Unable to retrieve local node name
  • Added the hostname and IP to the local /etc/hosts and the install continued.
  • Pre-installation checks give warning about kernel parameters and swap size… but it does offer me a fixit script for the kernel parameters. Need to ensure that these changes to parameters are added to /etc/sysctl.conf
  • Ran the fixit, and still some parameters giving a warning – semms etc – I guess these would need a reboot. So the kernel parameters will need reviewing

Things I might want to consider adding to the /etc/sysctl.conf (stolen from another system)


kernel.msgmni = 2878
kernel.msgmax = 8192
kernel.msgmnb = 65536
kernel.shmmni = 4096
kernel.shmmax = 229916494233
kernel.shmall = 28065978

 

After the install completed, the database started ok.