Display HBA and WWN in Solaris

First display the available HBA's installed in your system:

# fcinfo hba-port
HBA Port WWN: 2100001b4377092a
        OS Device Name: /dev/cfg/c1
        Manufacturer: QLogic Corp.
        Model: 375-3355-02
        Firmware Version: 05.04.03
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-0851658438
        Driver Name: qlc
        Driver Version: 20110321-3.05
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 2000001b4377092a
...

Then run fcinfo again with the remote-port option to see the zoned WWN's to the above HBA:

# fcinfo remote-port -p 2100001b3288092a
Remote Port WWN: 600608790221db92
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 600608790211db92
Remote Port WWN: 315511a1b93032ed
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 311511a1b93032ed
...

Finally you can display the avaialable Lun's:

# fcinfo remote-port -s -p 2100001b3288092a
Remote Port WWN: 600608790221db92
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 500507680100ca81
        LUN: 0
          Vendor: IBM
          Product: 2145
          OS Device Name: Unknown
        LUN: 9
          Vendor: IBM
          Product: 2145
          OS Device Name: /dev/rdsk/c1t500507681100ca81d9s2
...

Where the target represent the WWN of the LUN.

VxHA: set a new monitor timeout

Today I had a customer with a cluster and a Orcale database. The customer called in because his database always shutsdown once a day. At first I took a look at the alert log for the database (SID DB):

# less alert_DB.log
...
Instance terminated by USER, pid = 15764
...

Obviously the database was shutdown manually. The next log I investigated was engine_A.log and Oracle_A.log. One messages repeated itself in both log files:

# less engine_A.log
...
VCS ERROR V-16-2-13027 (NODE_2) Resource(Oracle_DB) - monitor procedure did not complete within the expected time.
...

So I deceided to set the timeouts for the Oracle resource higher. The first thing I needed to know was the type for the resource:

# hares -display Oracle_DB
...
Oracle_DB Type             global     Oracle
...

The type is Oracle and this type holds the values for the timeout:

# hatype -display Oracle
...
Oracle       MonitorTimeout          60
...
Oracle       MonitorInterval         60
...

There is no sense in setting only the timeout higher, the interval should also be updated to a higher value. To update the values the config must be in rw mode (-makerw):

# haconf -makerw

When the config is in rw mode then the new values for the timeout and the interval can be set:

# hatype -modify Oracle MonitorTimeout 180
# hatype -modify Oracle MonitorInterval 180

After that the config must be spread across all nodes (-dump) and set to be in ro mode (-makero) again:

# haconf -dump -makero
Finally make sure that the values are set:

# hatype -value Oracle MonitorTimeout
180
# hatype -value Oracle MonitorInterval
180

The values for the monitor timeout and the monitor interval were successfully set from 60 to 180 seconds.

Adding Nagiosgraph to Nagios

This article is an update to my prior released article about installing Nagios in Slackware.
Nagiosgraph allows you to add very nice graphs to your Nagios which is extremely helpful. With these graphs you'll be able to look into the past a little and probably do some trending for the future. You can act before it is too late.
Before you can begin to install Nagiosgraph you need an additional package rrdtool. First create a directory where you can store the sources and change into it:

# mkdir -p /usr/src/nagios && cd /usr/src/nagios

Then download rrdtool and Nagiosgraph:

# wget -c "http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.4.7.tar.gz"
# wget -c "http://downloads.sourceforge.net/project/nagiosgraph/nagiosgraph/1.4.4/nagiosgraph-1.4.4.tar.gz"

Extract the rrdtool package and change into the new directory:

# tar xf rrdtool-1.4.7.tar.gz
# cd rrdtool-1.4.7

Run configure, make and make install to compile and install rrdtool. If you're on a 32bit system then ignore the libdir option:

# ./configure --prefix=/usr --libdir=/usr/lib64 --with-perl-options='INSTALLDIRS=vendor'
...
# make
...
# make install
...

When rrdtool is installed then run the following perl command to check if the rrdtool perl module was build and installed correct:

# perl -e 'use RRDs; print "OK\n";'
OK

Now you can begin to install Nagiosgraph it self. First extract the source package and change into the new directory:

# cd /usr/src/nagios
# tar xf nagiosgraph-1.4.4.tar.gz
# cd nagiosgraph-1.4.4

Then create the Makefile with perl and run make:

# perl Makefile.PL
...
# make
...

Beofre you continue: note that I've changed some pathes below. These settings are for my Nagios installation. If you use this instruction on your very own Nagios installation then take care of the following settings:

# make install
...
Destination directory (prefix)? [/usr/local/nagiosgraph] /opt/nagios/latest/nagiosgraph
Location of configuration files (etc-dir)? [/opt/nagios/latest/nagiosgraph/etc]
Location of executables? [/opt/nagios/latest/nagiosgraph/bin]
Location of CGI scripts? [/opt/nagios/latest/nagiosgraph/cgi] /opt/nagios/latest/sbin
Location of documentation (doc-dir)? [/opt/nagios/latest/nagiosgraph/doc]
Location of examples? [/opt/nagios/latest/nagiosgraph/examples]
Location of CSS and JavaScript files? [/opt/nagios/latest/nagiosgraph/share] /opt/nagios/latest/share/nagiosgraph
Location of utilities? [/opt/nagios/latest/nagiosgraph/util]
Location of state files (var-dir)? [/opt/nagios/latest/nagiosgraph/var]
Location of RRD files? [/opt/nagios/latest/nagiosgraph/var/rrd]
Location of log files (log-dir)? [/opt/nagios/latest/nagiosgraph/var]
Path of log file? [/opt/nagios/latest/nagiosgraph/var/nagiosgraph.log]
Path of CGI log file? [/opt/nagios/latest/nagiosgraph/var/nagiosgraph-cgi.log]
URL of CGI scripts? [/nagiosgraph/cgi-bin] /nagios/cgi-bin
URL of CSS file? [/nagiosgraph/nagiosgraph.css] /nagios/share/nagiosgraph
URL of JavaScript file? [/nagiosgraph/nagiosgraph.js] /nagios/share/nagiosgraph
Path of Nagios performance data file? [/tmp/perfdata.log] /opt/nagios/latest/var/perfdata.log
URL of Nagios CGI scripts? [/nagios/cgi-bin]
username or userid of Nagios user? [nagios]
username or userid of web server user? [apache]
Modify the Nagios configuration? [n]
Modify the Apache configuration? [n]
...
Continue with this configuration? [y]
...

Whan Nagiosgraph is installed then change the user permissions for the nagios user:

# chown -R nagios:nagios /opt/nagios/latest/nagiosgraph

Next configure Nagios to process performance data:

# su - nagios
$ vi /opt/nagios/latest/etc/nagios.cfg
...
#process_performance_data=0
process_performance_data=1
service_perfdata_file=/opt/nagios/latest/var/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30
service_perfdata_file_processing_command=process-service-perfdata
...

Add insert.pl to commands.cfg (remove/comment out prior process-service-perfdata command definition):

$ vi /opt/nagios/latest/etc/objects/commands.cfg
...
# 'process-service-perfdata' command definition
#define command{
#       command_name    process-service-perfdata
#       command_line    /usr/bin/printf "%b" "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SE
RVICEOUTPUT$\t$SERVICEPERFDATA$\n" >> /opt/nagios/3.5.0/var/service-perfdata.out
#       }

define command {
        command_name  process-service-perfdata
        command_line  /opt/nagios/latest/nagiosgraph/bin/insert.pl
}

Next configure Nagiosgraph and correct the path to the perfdata.log file:

$ cd /opt/nagios/latest/nagiosgraph/etc
$ vi nagiosgraph.log
...
perfdata = /opt/nagios/latest/var/perfdata.log
...

You can restart Nagios now:

# /etc/rc.d/rc.nagios restart
...

After a few minutes a new directory should be available for storing the rrd performance data:

# ls /opt/nagios/latest/nagiosgraph/var/rrd
localhost
...
# ls /opt/nagios/latest/nagiosgraph/var/rrd/localhost
...

Before you can access these performance data you need to configure Nagios again. The graphed-service template will add a small graph icon to each service (if enabled):

# su - nagios
$ cd /opt/nagios/latest/etc/objects
$ vi templates.cfg
...
define service {
        name graphed-service
        action_url                      /nagios/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$' onMouseOver='showGraphPopup(this)' onMouseOut='hideGraphPopup()' rel='/nagios/cgi-bin/showgraph.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&period=week&rrdopts=-w+450+-j
        }

You can enable the graph icon by adding the above template to each service for each host, eg. for the PING service it could look like this:


$ cd /opt/nagios/latest/etc/objects
$ vi localhost.cfg
...
define service{
        use                    local-service,graphed-service
        service_description    PING
...

When you restart Nagios again:

# /etc/rc.d/nagios restart

Then a little graph icon will be available for the PING service for the host localhost. When you click on it then a new page should open with some graphs to watch (it could take some minutes before enough data is gathered).

Some errors I bumped into:

1. Cannot write to nagiosgraph-cgi.log

The error.log for Apache show the following entry:

$ less /var/log/http/error.log
...
Sat Apr  6 12:51:47 2013 showgraph.cgi critical Cannot write to '/opt/nagios/latest/nagiosgraph/var/nagiosgraph-cgi.log', using STDERR instead, referer: http://xbmc/nagios/cgi-bin/show.cgi?host=localhost&service=Current%20Load
...

Change permissions to the nagiosgraph-cgi.log file so that every user in the group nagios can access that file. This implies that the apache user is in the nagios group:

# chmod 664 /opt/nagios/latest/nagiosgraph/var/nagiosgraph-cgi.log
# ls -la /opt/nagios/latest/nagiosgraph/var/nagiosgraph-cgi.log
-rw-rw-r-- 1 nagios nagios 0 Apr  6 11:43 /opt/nagios/latest/nagiosgraph/var/nagiosgraph-cgi.log

2. nagiosgraph.js is not installed or wrong version.

When you open any graph then you might be confronted with the following message:

nagiosgraph.js is not installed or wrong version.

To fix this error check nagiosgraph.conf first:

$ cd /opt/nagios/latest/nagiosgraph/etc
$ less nagiosgraph.conf
...
# JavaScript: URL to the nagiosgraph javascript file.
javascript = /nagios/share/nagiosgraph
# Stylesheet: URL to the nagiosgraph stylesheet.
stylesheet = /nagios/share/nagiosgraph
...

Then check the above path (note that share is the path for you web files), eg:

$ ls -la /opt/nagios/latest/share/nagiosgraph/
...
-rw-r--r--  1 nagios nagios  2656 Apr  6 12:20 nagiosgraph.css
-rw-r--r--  1 nagios nagios 40944 Apr  6 12:20 nagiosgraph.js

Try to open the js or css file in your browser. For me the URL http://mon01/nagios/share/nagiosgraph/nagiosgraph.js failed but the following url http://mon01/nagios/nagiosgraph/nagiosgraph.js worked. Check the apapche logs and the path to these files until you get the correct URL.
When you have found the correct URL then adjust the above path in the nagiosgraph.conf file and add the file to the path for the javascript and css file:

$ cd /opt/nagios/latest/nagiosgraph/etc
$ vi nagiosgraph.conf
...
# JavaScript: URL to the nagiosgraph javascript file.
# javascript = /nagios/share/nagiosgraph
javascript = /nagios/nagiosgraph/nagiosgraph.js
# Stylesheet: URL to the nagiosgraph stylesheet.
# stylesheet = /nagios/share/nagiosgraph
stylesheet = /nagios/nagiosgraph/nagiosgraph.css
...

The error shoudn't appear anymore when you open a page with some graphs.

ORA-02082: a loopback database link must have a connection qualifier

I have two Oracle databases, one 9.2.0.8 (SID test9) and another 10.2.0.4 (SID test10). I needed to create a database link from test9 to test10 but I always got the following error:

$ sqlplus "sys/password@test9"
SQL> create database link test connect to sys identified by password using 'test10';
create database link test connect to sys identified by password using 'test10'
                         *
ERROR at line 1:
ORA-02082: a loopback database link must have a connection qualifier

After some searching I finally got the solution. Both database global names are test and the link name is also test which ends in the above error:

$ sqlplus "sys/password@test9"
SQL> select * from global_name;
GLOBAL_NAME
-------------
test.karellen
SQL> exit

$ sqlplus "sys/password@test10"
SQL> select * from global_name;
GLOBAL_NAME
-------------
test.karellen
SQL> exit

Specifying another link name did it:

$ sqlplus "sys/password@test9"
SQL> create database link test10 connect to sys identified by password using 'test10';
Database link created.
SQL> select version from v$instance;
VERSION
-----------------
9.2.0.8.0
SQL> select version from v$instance@test10;
VERSION
-----------------
10.2.0.4.0

Basic Nagios installation

This article is about basic Nagios installation in Slackware. It is not about configure Nagios completely, adding hosts etc. It is just a basic tutorial howto install Nagios from sources.
Before you begin create a directory where you can store the Nagios sources:

# mkdir -p /usr/src/nagios && cd /usr/src/nagios

Then download the nagios and the nagios-plugins sources:

# wget -c "http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.5.0.tar.gz"
# wget -c "http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.16.tar.gz"

Before you continue create a nagios group and a nagios user:

# groupadd -g 96 nagios
# useradd -c "Nagios User" -d /opt/nagios -g nagios -m nagios -s /bin/bash -u 96

Now you can begin to compile the nagios sources. First extract the nagios source package:

# tar xf nagios-3.5.0.tar.gz
# cd nagios

Then run the configure script. I have added a few specific Slackware options for the Apache configuration and rc scripts:

# ./configure --prefix=/opt/nagios/3.5.0 --with-httpd-conf=/etc/httpd/extra --with-init-dir=/etc/rc.d
...
*** Configuration summary for nagios 3.5.0 03-15-2013 ***:

 General Options:
 -------------------------
        Nagios executable:  nagios
        Nagios user/group:  nagios,nagios
       Command user/group:  nagios,nagios
            Embedded Perl:  no
             Event Broker:  yes
        Install ${prefix}:  /opt/nagios/3.5.0
                Lock file:  ${prefix}/var/nagios.lock
   Check result directory:  ${prefix}/var/spool/checkresults
           Init directory:  /etc/rc.d
  Apache conf.d directory:  /etc/httpd/extra
             Mail program:  /usr/bin/mail
                  Host OS:  linux-gnu
 Web Interface Options:
 ------------------------
                 HTML URL:  http://localhost/nagios/
                  CGI URL:  http://localhost/nagios/cgi-bin/
 Traceroute (used by WAP):  /usr/bin/traceroute

If you see an output like above then run make to compile the sources:

# make all
...

Then install the binaries, the rc script, the Apache configuration and the basic configuration - step by step. Begin with the binaries:

# make install
...

Then link the new /opt/nagios/3.5.0 to /opt/nagios/latest:

# ln -s /opt/nagios/3.5.0 /opt/nagios/latest

Install the rc script and move it to /etc/rc.d/rc.nagios:

# make install-init
...
# mv /etc/rc.d/nagios /etc/rc.d/rc.nagios

Install the Apache configuration for Nagios:

# make install-webconf
...

Install the basic Nagios configuration and link it to /etc

# make install-config
...
# ln -s /opt/nagios/latest/etc /etc/nagios

Before you can start Nagios you need to create one additional directory. Use the prior created Nagios user for creating the directory:

# su - nagios
$ mkdir -p /opt/nagios/latest/var/rw/

Next continue and configure Apache. You need to enable cgi, php, add index.php as DirectoryIndex diective and add the new Nagios configuration file for Apache (I assume you have an untouched Apache configuration shipped with Slackware):

# vi /etc/httpd/httpd.conf
...
LoadModule cgi_module lib64/httpd/modules/mod_cgi.so
...
#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>
...
# Uncomment the following line to enable PHP:
#
Include /etc/httpd/mod_php.conf
...
# Nagios
Include /etc/httpd/extra/nagios.conf

Before you can navigate to your new Nagios in your browser you need to create a user for authentication. A user nagiosadmin is already defined in the cgi.cfg for Nagios, the only thing you need to do is to create a htpasswd file and set a password for the nagiosadmin user (run htpasswd as nagios user):

# su - nagios
$ htpasswd -c /opt/nagios/latest/etc/htpasswd.users nagiosadmin
New password:
Re-type new password:
Adding password for user nagios

Since the Apache user needs access to the Nagios web files it is a wise choice to add the Apache user to the Nagios group:

# usermod -a -G nagios apache

Then run chmod on the rc.https script to make it executable and start Apache daemon httpd:

# chmod 755 /etc/rc.d/rc.httpd
# /etc/rc.d/rc.httpd start

Do the same with the rc script for nagios:

# chmod 755 /etc/rc.d/rc.nagios
# /etc/rc.d/rc.nagios start

At this point you should be able to navigate your browser to the Nagios URL, eg. http://hq01/nagios/. The first thing you need to do is to apply your username (nagiosadmin) and password you have choosen for the htpasswd command. You can navigate a little through the Nagios web interface but currently there is not too much to see since only localhost is configured. More worse, if you list all the hosts (click on 'Services' on the left navigation bar) then localhost seems to be down. That's because the ping plugin is missing but I'll show you how to install the plugins in the next section.

Change back to the location where you've stored the Nagios sources:

# cd /usr/src/nagios

Then extract the nagios-plugins sources and change in the new directory:

# tar xf nagios-plugins-1.4.16.tar.gz
# cd nagios-plugins-1.4.16

Run the configure script with the same prefix as used for Nagios configure script:

# ./configure --prefix=/opt/nagios/3.5.0
...
            --with-apt-get-command:
              --with-ping6-command: /bin/ping6 -n -U -w %d -c %d %s
               --with-ping-command: /bin/ping -n -U -w %d -c %d %s
                       --with-ipv6: yes
                      --with-mysql: /usr/bin/mysql_config
                    --with-openssl: yes
                     --with-gnutls: no
               --enable-extra-opts: no
                       --with-perl: /usr/bin/perl
             --enable-perl-modules: no
                     --with-cgiurl: /nagios/cgi-bin
               --with-trusted-path: /bin:/sbin:/usr/bin:/usr/sbin
                   --enable-libtap: no

No more action needed, just run make followed by make install:

# make
...
# make install
...

When the plugins are installed (/opt/nagios/latest/libexec) then navigate your browser back to localhost in the Nagios web interface. After some time localhost should be appear as 'up'.
That's it. Nagios is up and running and ready for more hosts, services etc.
If you want to start Nagios during boot then add the following line to your rc.local script:

echo "/etc/rc.d/rc.nagios start" >> /etc/rc.d/rc.local

With this line Nagios will always start when you boot your system.

Converting mpc to wav

Today I was confronted with some mpc files which I needed to convert into wav files. There is a tool called mpcdec but it has to be compiled first. I have tested the following instruction successfully on my Slackware 14 machine (64bit).
To start create a directory where you can store the sources and change into it:

# mkdir -p /usr/src/mpc
# cd /usr/src/mpc

Then check out the sources:

# svn export http://svn.musepack.net/libcuefile libcuefile
...
# svn export http://svn.musepack.net/libreplaygain libreplaygain
...
# svn export http://svn.musepack.net/libmpc/trunk libmpc
...

When all sources are checked out then you can begin to compile them. Start with configuring libcuefile:

# cd /usr/src/mpc/libcuefile/trunk/
# cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX:PATH=/usr -DLIB_SUFFIX=64
...

Run make to compile and install the sources:

# make
...
# make install
...

The header files must be copied manually:

# cp -r include/cuetools /usr/include

The next library to compile is libreplaygain:

# cd /usr/src/mpc/libreplaygain/
# cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX:PATH=/usr -DLIB_SUFFIX=64
...
# make
...
# make install

Again, the header files have to be copied manually:

# cp -r include/replaygain /usr/include

Finally compile libmpc. This package contains the library as well as the binary mpcdec to convert mpc files into wav files:

# cd /usr/src/mpc/libmpc/
# cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX:PATH=/usr
...
# make
...
# make install
...

Check that mpcdec was installed:

# which mpcdec
/usr/bin/mpcdec

When run mpcdec the first time then you might notice the following error:

# mpcdec
mpcdec: error while loading shared libraries: libmpcdec.so: cannot open shared object file: No such file or directory

The issue is that libmpcdec.so is not available on your system:

# ldd /usr/bin/mpcdec
        ...
        libmpcdec.so => not found
        ...

OK, it is available but it was not installed by make before. Just copy libmpcdec.so manually and run ldd again:

# cp /usr/src/mpc/libmpc/libmpcdec/libmpcdec.so /usr/lib64/
# ldd /usr/bin/mpcdec
        ...
        libmpcdec.so => /usr/lib64/libmpcdec.so (0x00007ff7478f9000)
        ...

Finally you can convert a mpc file into a wav file:

# mpcdec track-01.mpc track-01.wav
mpcdec - Musepack (MPC) decoder v1.0.0 (C) 2006-2009 MDT
Built Mar 24 2013 14:48:34
1978620 samples decoded in 70 ms (640.95x)

Links:

http://trac.musepack.net/wiki/SV8Build

su: cannot set user id: Resource temporarily unavailable

Today I had a customer complaining about his DB that nobody can login anymore. The number of max sessions (in the DB) was set to 4096 and there were about 2048 current seesion on the DB. So, what was wrong?
I decided to login as root into the DB server which worked wihtout any issue. The system it self was OK, enough memory free, no filled volumes etc. Then I tried to switch to the oracle user:

# su - oracle
su: cannot set user id: Resource temporarily unavailable

OK, it seems that the system cannot create any more sessions for the oracle user. A quick check about the number of operating system processes gave me the final clue:

# ps -U ora10 | wc -l
2048

The number of processes was 2048 and this value can be set in the file /etc/security/limits.conf:

# cat /etc/security/limits.conf
...
oracle soft nproc 2048
...

I just changed the value for the number of processes (soft nproc) to 4096 (the same value for the maximum number of sessions inside the DB) and the customer could work normally again.