Detecting redo size increase

AWR tables (require Diagnostics license) can be accessed to identify the redo size increase. Following query spools the daily rate of redo size. You can easily open the output file redosize.lst in an Excel spreadsheet and graph the data to visualize the redo size change. Use pipe symbol as the delimiter while opening the file in excel spreadsheet.

spool redosize.lst
REM  You need Diagnostic Pack licence to execute this query!
REM  Author: Riyaj Shamsudeen
col begin_interval_time format a30
set lines 160 pages 1000
col end_interval_time format a30
set colsep '|'
alter session set nls_date_format='DD-MON-YYYY';
with redo_sz as (
SELECT  sysst.snap_id, sysst.instance_number, begin_interval_time ,end_interval_time ,  startup_time,
VALUE - lag (VALUE) OVER ( PARTITION BY  startup_time, sysst.instance_number
                ORDER BY begin_interval_time, startup_time, sysst.instance_number) stat_value,
EXTRACT (DAY    FROM (end_interval_time-begin_interval_time))*24*60*60+
            EXTRACT (HOUR   FROM (end_interval_time-begin_interval_time))*60*60+
            EXTRACT (MINUTE FROM (end_interval_time-begin_interval_time))*60+
            EXTRACT (SECOND FROM (end_interval_time-begin_interval_time)) DELTA
  FROM sys.wrh$_sysstat sysst , DBA_HIST_SNAPSHOT snaps
WHERE (sysst.dbid, sysst.stat_id) IN ( SELECT dbid, stat_id FROM sys.wrh$_stat_name WHERE  stat_name='redo size' )
AND snaps.snap_id = sysst.snap_id
AND snaps.dbid =sysst.dbid
AND sysst.instance_number=snaps.instance_number
and begin_interval_time > sysdate-90
)
select instance_number, 
  to_date(to_char(begin_interval_time,'DD-MON-YYYY'),'DD-MON-YYYY') dt 
, sum(stat_value) redo1
from redo_sz
group by  instance_number,
  to_date(to_char(begin_interval_time,'DD-MON-YYYY'),'DD-MON-YYYY') 
order by instance_number, 2
/
spool off

Visualizing the data will help you to quickly identify any pattern anomalies in redo generation. Here is an example graph created from the excel spreadsheet and see that redo size increased recently.

Guess the object using ‘db block changes’ statistics

A quick method to guess the objects generating higher redo size is to use ‘db block changes’ statistics. The philosophy behind this technique is that, if the object is modified heavily then that object will probably generate more redo. But, it is not an entirely accurate statement as less frequently modified objects can generate more redo and vice versa. If you are lucky, one or two objects will stand out as a problem and you can review those segments further to reduce redo size.

@segment_stats.sql
To show all segment level statistics in one screen

Enter value for statistic_name: db block changes
old   6:           where value >0  and statistic_name like '%'||'&&statistic_name' ||'%'
new   6:           where value >0  and statistic_name like '%'||'db block changes' ||'%'
   INST_ID STATISTIC_NAME                 OWNER        OBJECT_NAME        OBJECT_TYP        VALUE   PERC
---------- ------------------------------ ------------ ----------------- ---------- ------------ ------
         1 db block changes               INV          SALES_TEMP_N1     INDEX        3831599856  48.66
         3                                INV          MTL_RESERV        TABLE        3794818912  23.78
         3                                ZX           DET_FACTORS_      INDEX        2468120576  15.47
         2                                APPLSYS      FND_TAB           TABLE        2346839248  16.33
….
  

Segment_stats.sql script can be found in segment_stats.

Identify objects using logminer

Scientific method to identify the object generating higher redo uses log mining package. Objects can be identified by the following steps:

Step 1: Start log miner from sys or system user in SQL*Plus. Example given here is for finprod2 instance archivelog file.

begin
  sys.dbms_logmnr.ADD_LOGFILE ('/opt/app/prod/finprod2/arch/finprod_212_2_1212121221.arch');
end;
/
begin
  sys.dbms_logmnr.START_LOGMNR;
end;
/

Step 2: Create a table by querying the data from v$logmnr_contents dynamic performance view. I tend to create a separate table for each archive log file for two reasons: (a) to improve the query performance (b)I haven’t tested thoroughly with multiple archivelog files. Following SQL statement finds the length of redo record by subtracting the RBA (Redo Byte Address) of the current record from the RBA of next record. Redo byte address provides the physical location of a redo record in a redo log file. Using the physical location of current redo record and the next redo record, we can find the length of current redo record.

Update 1: As Greg pointed out in comments section, I was using hard-coded 512 bytes for redo block size in my script, which is true in Solaris and Linux platform. But, in HP platform, redo block size is 1024 bytes. You can use the following SQL statement to identify the redo block size. I have modified the create table script to query redo block size dynamically.

SQL>select max(lebsz) from x$kccle;
MAX(LEBSZ)
----------
       512

drop table redo_analysis_212_2;
CREATE TABLE redo_analysis_212_2 nologging AS
SELECT data_obj#, oper,
  rbablk * le.bsz + rbabyte curpos,
  lead(rbablk*le.bsz+rbabyte,1,0) over (order by rbasqn, rbablk, rbabyte) nextpos
FROM
  ( SELECT DISTINCT data_obj#, operation oper, rbasqn, rbablk, rbabyte
  FROM v$logmnr_contents
  ORDER BY rbasqn, rbablk, rbabyte
  ) ,
  (SELECT MAX(lebsz) bsz FROM x$kccle ) le 
/

Step 3: Query the table to identify the object_name: In this step, we join the table created and obj$ table to identify the objects inducing redo size. Outer join is needed as the object may have been dropped recently. START indicates the redo record for the start of a transaction and COMMIT indicates the redo record for the end of a transaction.

set lines 120 pages 40
column data_obj# format  9999999999
column oper format A15
column object_name format A60
column total_redo format 99999999999999
compute sum label 'Total Redo size' of total_Redo on report
break on report
spool /tmp/redo_212_2.lst
select data_obj#, oper, obj_name, sum(redosize) total_redo
from
(
select data_obj#, oper, obj.name obj_name , nextpos-curpos-1 redosize
from redo_analysis_212_2 redo1, sys.obj$ obj
where (redo1.data_obj# = obj.obj# (+) )
and  nextpos !=0 -- For the boundary condition
and redo1.data_obj#!=0
union all
select data_obj#, oper, 'internal ' , nextpos-curpos  redosize
from redo_analysis_212_2 redo1
where  redo1.data_obj#=0 and  redo1.data_obj# = 0
and nextpos!=0
)
group by data_obj#, oper, obj_name
order by 4
/
...
      46346 INSERT          WSH_EXCEPTIONS                        87006083
   12466144 INTERNAL        MSII_N9                               95800577
   12427363 INTERNAL        MSII_N1                               96445137
          0 START           internal                             125165844
          0 COMMIT          internal                             205600756
   12960642 UPDATE          XLA_GLT_1234567890                   243625297
                                                           ---------------
Total Redo                                                      3681252096
spool off

Notice that objects identified using log miner tool is not matching with the objects from db block changes statistics. In this example, the discrepancy is probably because, I am looking at segment stats from the start of instance which may not be accurate.

In summary, log miner utility can be used to identify the objects generating higher redo. This will help you to understand why the redo generation is higher and may be, gives you a mechanism to reduce redo.

S96ohasd -> /etc/init.d/ohasd

/etc/rc3.d/S96ohasd start  # init daemon starting ohasd.

Similarly, a server shutdown will call scripts in rcX.d directories, for example, ohasd is shut down by calling K15ohasd script:

K15ohasd -> /etc/init.d/ohasd
/etc/rc3.d/K15ohasd stop  #UNIX daemons stopping ohasd

In Summary, server startup will call files matching the pattern of S* in the /etc/rcX.d directories. Calling sequence of the scripts is in the lexical order of script name. For example, S10cscape will be called prior to S96ohasd, as the script S10cscape occurs earlier in the lexical sequence.

Google if you want to learn further about RC startup sequence. Of course, Linux 6 introduces Upstart feature and the mechanism is a little different: http://en.wikipedia.org/wiki/Upstart

That’s not the whole story!

Have you ever thought why the ‘crsctl start crs’ returns immediately? You can guess that Clusterware is started in the background as the command returns to UNIX prompt almost immediately. Executing the crsctl command just modifies the ohasdrun file content to ‘restart’. It doesn’t actually perform the task of starting the clusterware. Daemon init.ohasd reads the ohasdrun file every few seconds and starts the Clusterware if the file content is changed to ‘restart’.

# cat /etc/oracle/scls_scr/oel6rac1/root/ohasdrun
restart

If you stop has using ‘crsctl stop has’ , then the ohasdstr file content is modified to stop and so, init.ohasd daemon will not restart Clusterware. However, stop command is synchronous and executes the stop of clusterware too.

# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel6rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel6rac1'
..

The content of ohasdrun is modified to stop:

# cat  /etc/oracle/scls_scr/oel6rac1/root/ohasdrun
stop # 

In a nutshell, init.ohasd daemon is monitoring the ohasdrun file and starts the Clusterware stack if the value in the file is modified to restart.

Inittab

Init.ohasd daemon is an essential daemon for Clusterware startup. Even if the Clusterware is not running on a node, you can start the Clusterware from a different node. How does that work? Init.ohasd is the reason.

The init.ohasd daemon is started from /etc/inittab. Entries in the inittab is monitored by the init daemon (pid=1) and init daemon will react if the inittab file is modified. The init daemon monitors all processes listed in the inittab file and reacts according to the configuration in the inittab file. For example, if init.ohasd fails for some reason, it is immediately restarted by init daemon.

Following is an example entry in the inittab file. Fields are separated a colon, second field indicates that init.ohasd will be started in run level 3, and the third field indicates an action field. Restart in the action field means that, if the target process exist, just continue scanning inittab file; if the target process does not exist, then restart the process.

#cat /etc/inittab
…
h1:3:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

If you issue a clusterware startup command from a remote node, that a message sent to init.ohasd daemon in the target node, and the daemon initates the clusterware startup. So, init.ohasd will be always running irrespective of whether the Clusterware is running or not.

You can use strace on init.ohasd to verify this behavior. Following are a few relevant lines from the output of strace command of init.ohasd process:

…
5641  1369083862.828494 open("/etc/oracle/scls_scr/oel6rac1/root/ohasdrun", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
5641  1369083862.828581 dup2(3, 1)      = 1
5641  1369083862.828606 close(3)        = 0
5641  1369083862.828631 execve("/bin/echo", ["/bin/echo", "restart"], [/* 12 vars */]) = 0
…

Just for fun!

So, what happens if I manually modify that ohasdrun to restart? I copied the ohasdrun to a temporary file (/tmp/a1.lst) and stopped the clusterware.

cp /etc/oracle/scls_scr/oel6rac1/root/ohasdrun /tmp/a1.lst 
# crsctl stop has

I verified that Clusterware is completely stopped. Now, I will copy the file again overlaying ohasdrun:

# cat /tmp/a1.lst
restart
# cp /tmp/a1.lst  /etc/oracle/scls_scr/oel6rac1/root/ohasdrun

After a minute or so, I see that Clusterware processes are started. Not that, you would use this type of hack in a Production cluster, but this test proves my point.

It’s also important not to remove the files in the scls_scr directories. Any removal of the files underneath the scls_scr directory structure can lead to an invalid configuration.

There are also two more files in the scls_scr directory structure. Ohasdstr file decides if the HAS daemon should be started automatically or not. For example, if you execute ‘crsctl disable has’, that command modifies ohasdstr file contents to ‘disable’. Similarly, crsstart file controls CRS daemon startup. Again, you should use recommended commands to control the startup, rather than modifying any of these files directly.

11.2.0.1 and HugePages

If you tried to configure hugepages in 11.2.0.1 clusterware, by increasing memlock kernel parameter for GRID and database owner, you would have realized that database doesn’t use hugepages if started by the clusterware. Database startup using sqlplus will use hugepages, but the database startup using srvctl may not use hugepages.

As the new processes are cloned from the init.ohasd daemon, until init.ohasd is restarted, user level memlock limit changes are not correctly reflected in an already running process. Only recommended way to resolve the problem is to restart the node completely (not just the clusterware), as init.ohasd daemon must be restarted to reflect the user level limits.

Version 11.2.0.2 fixes this issue by explicitly calling ulimit command from /etc/init.d/ohasd files.

Summary
In Summary, init.ohasd process is an important process. Files underneath scls_scr directory is controlling the startup behavior. This also means that if a server is restarted, you don’t need to explicitly stop the Clusterware. You can let the server startup to restart the Clusterware.

PS: Some of you have asked my why my blogging frequency has decreased. I have been extremely busy co-authoring a book on RAC titled “Expert RAC Practices 12c”. We are covering lots of interesting stuff in the book. As soon as 12c release is Production, we can release the book also.

Rough outline:
(i) Ability to query access path from memory, AWR repository
(ii) Ability to use cardinality feedback method to understand access plan issues. Few tips from a real world experience will be provided too.
(iii) Ability to understand issues with database links etc.
(iv) Options such as ADVANCED, ALLSTATS etc
(v) Why should you choose dbmx_xplan over tkprof+sql_trace combination?
(vi) Disadvantages of dbms_xplan and a quick introduction to dbms_monitor.

Refreshments sponsored by me

Update: Uploading the presentation pdf files. Enjoy

dbms_xplan

ASMLIB is applicable only to Linux platform and does not apply to any other platform.

Now, you might ask why bother and why not just use OEL and UK? Well, not every Linux server is used as a database server. In a typical company, there are hundreds of Linux servers and just few percent of those servers are used as Database servers. Linux system administrators prefer to keep one flavor of Linux distribution for management ease and so, asking clients to change the distribution from RHEL to OEL or OEL to RHEL is always not a viable option.

Do you need to use ASMLIB in Linux?

Short answer is No. Long answer is possibly No. ASMLIB is an optional support library and eases the administration of ASM devices. Especially, it is helpful while adding new devices to the nodes in a cluster. ASMLIB essentially stamps the devices and so, it is easily visible in other nodes of a cluster in the next asm scandisk. asmlib also provides device persistence, which is the important benefit of ASM (see the discussion below for more details about device persistence).

But, how many times do you add disks to the servers? How many times, do we change the server or disk architectures in a given year? In my opinion, ASMLIB is an additional software layer. It is possible to setup RAC without ASMLIB and that’s the discussion of this blog.

Problem definition

Problem with devices in Linux is that the device name can change after a server reboot. That means that ASM might not come up since the device names may not be matching with asm_diskstring parameter after the reboot. Especially, since OCR and Voting disks can be stored in ASM devices from 11.2 onwards, debugging GI startup problems are painful if the device persistence is not setup properly.

How do you resolve that?

Option #1: UDEV only

UDEV eliminates the device persistence problem, provides ability to create user defined aliases, and setup device permissions. While it might seem like another new concept to learn, UDEV is quite easy to use. I will explain basic UDEV setup. Of course, I am not covering numerous options available under UDEV, and covering just necessary items.

During server startup, when the kernel detects a device ( or when a new device is added), kernel sends an event to udevd daemon. udevd daemon uses rules to match the incoming event and takes action depending upon the rule (such as remove device node, add device node etc). udevd rules have many attributes and one such attribute is that, an arbitrary program can be used in the rule to process incoming events. and that can be used to return human-friendly device names. Essentially, to setup udev we need to write udev rules.

( Since you are probably a DBA if you are reading this blog, it is easier to imagine the UDEV rule as a function. That function accepts the scsi_id and sets up human friendly aliases.)

Method to setup udev rule

1. Modify /etc/scsi_id.config and add the following line at the end. Essentially, UDEV will assume that all SCSI devices will provide unique UUIDs.

options=-g

2. Identify the unique SCSI id from scsi_id command. Command scsi_id will return unique value for a given SCSCI device. (This unique value is also called UUID (or WWID) if you use SAN arrays such as EMC or Hitachi etc) Size of the device can be identified using blockdev command.

( For RHEL6, use –gud as options for scsi_id command. Looks like, option –s is replaced by –d.)

For example, for the device /dev/sdd:
# /sbin/scsi_id -gus /block/sdd
3600254567259abde00006000004c0000

To get device size, use blockdev (output in bytes):
# /sbin/blockdev –getsize64 /dev/sdd

3. Now, setup rules: vi /etc/udev/rules.d/99-asmdevices.rules

Add a rule for the above device. A rule is essentially an if-then-else logic. In the rule, we specify is satisfied for an event, actions will be taken to setup the device. Refer to the rule printed below. If the event is a scsi device (KERNEL, BUS attributes), then call the /sbin/scsi_id -g -u -s program, passing the block device as first argument (PROGRAM attribute and %p in the rule definitin). If the RESULT of the program call matches with a value of 3600143801259abde00006000004c0000 (RESULT attribute in the rule), then create a device entry as “asmcrs01″, with owner as grid, group owner as oinstall, and permissions as 0660.

( one rule must be in a single line, but output below is wrapped. Make sure that rule doesn’t wrap aound in the .rules file).

KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”, RESULT==”3600254567259abde00006000004c0000″, NAME=”asmcrs01″,
OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″

4. Test the rules using udevtest

# udevtest /block/sdd

This would show that udev might create three symlinks, a symlink named/dev/asmcrs01, one symlink in /dev/disk/by-id/, and third symlink in /dev/disk/by-path/. We will use /dev/asmcrs01 symlink for ASM setup.

5. Reload rules and start udev

This should create the symlink in /dev/asmcrs01.

# /sbin/udevcontrol reload_rules
# /sbin/start_udev

6. Now, setup asm_diskstring parameter to ‘/dev/asmcrs*’ so that ASM will identify these devices. Repeat the above steps for all devices that you are planning to add to ASM. You could potentially decide to perform start_udev after all rules have been setup.
7. Once you are happy with one node setup, copy the file /etc/udev/rules.d/99-asmdevices.rules to all nodes of RAC cluster and restart udev.

Option #2: Multipathing feature

Multipathing feature provides fault tolerance for paths to storage devices and uses device mapper framework to map block devices to aliases. Even if you have just one path to the device, you could potentially setup this feature. I prefer this method at this time as it provides easier migration to multipathed devices in future.

Setup is very similar to UDEV. Here is the step-by-step instruction.

1. Verify that device mapper rpm version is compatible.

$ rpm –qa|grep device-mapper
device-mapper-multipath-0.4.7-46.el5

2. Verify and configure devices
3. Verify that all SCSI devices are seen in all nodes. Note that some devices will be seen multiple times through different HBAs. Identify the SCSI devices for the database.

   # lsscsi
   # fdisk -l

4. Modify /etc/scsi_id.config and add the following line at the end.This is for scsi_id to assume all sCSI devices will provide unique scsi id.

options=-g

5. Identify the unique SCSI Id from scsi_id command. Command scsi_id will return unique value for a given SCSI device . (This ID is also called UUID or WWID if you use SAN arrays such as EMC or Hitachi etc) Size of the device can be identified using blockdev command.

( For RHEL6, use –gud for scsi_id command. Looks like, option –s is replaced by –d.)

For example, for the device /dev/sdd:
# /sbin/scsi_id -gus /block/sdd
3600254567259abde00006000004c0000

To get device size, use blockdev (output in bytes):
# /sbin/blockdev –getsize64 /dev/sdd

6. Edit /etc/multipath.conf (of course take a backup of the file)

a. Comment out this stanza.

# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices. 
#blacklist {
#        devnode "*"
#}

b. Blacklist all local devices. Devices such as raw, loop, floppy disk etc doesn’t need to have multipathing configured. ( Remember that if you use raw device, you need to modify this procedure little bit as we are blacklisting raw devices here) .

# Blacklist all local devices

blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z][[0-9]*]"
        devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
        devnode "dasd[a-z]+[0-9]*"
}

c. Add this stanza for specific to SAN array.

##
## Essentially, you can setup attributes specific to disk array.
##   This would require you to check with vendor documentation. 
##    In this case, we setting up for HSV200/300 HP array.
## This stanza defines how multipathing should behave. This is specific to a disk array but
## can allow to use default values too.
devices {
        device {
                vendor "HP"
                product "HSV2[01]0|HSV300|HSV4[05]0"
                getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
                prio_callout "/sbin/mpath_prio_alua /dev/%n"
                hardware_handler "0"
                path_selector "round-robin 0"
                path_grouping_policy group_by_prio
                failback immediate
                rr_weight uniform
                no_path_retry 18
                rr_min_io 100
                path_checker tur
        }
}

d. Add following stanza. In this stanza, within a multipaths block, you would specify a device contained within another multipath block. Notice that UUID is what we got from scsi_id command earlier is used in the first part of the stanza for the device asmcrs01. We want to setup that SCSI device to have a name of asmcrs01 for the UUID 3600254567259abde00006000004c0000. This stanza will allow device mapper to create symlink as /dev/mapper/asmcrs01 for that device. Further device mapper, sets up permissions using uid/gid combination. ( Use correct uid/gid combination matching with your environment.) In this example, uid= 1100 =grid, gid=1000=oinstall. So, first multiblock stanza will create a device named /dev/mapper/asmcrs01 with permissions owned by grid:oinstall with 0660 permissions( i.e. read, write for owner and group, no permissions for other group).

I am also setting up multiple devices below to provide an example.

##
## Multipathing for SCSI devices from storage array
##
multipaths {
  multipath {
    wwid 3600254567259abde00006000004c0000
    alias asmcrs01
    uid 1100
    gid 1000
    mode 660
   }
  multipath {
    wwid 3601213101259abde0000600000500000
    alias asmcrs02
    uid 1100
    gid 1000
    mode 660
   }
…
  multipath {
    wwid 3601212111259abde0000600000c00000
    alias asmdev15
    uid 1100
    gid 1000
    mode 660
   }
}

e. Enable multipath daemons and make sure that they are enabled at startup.

# modprobe dm-multipath
# service multipathd start
# multipath –d
# multipath –v2
# multipath -v2
create: asmcrs01 (3600254567259abde00006000004c0000) HP,HSV300
[size=2.0G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=100][undef]
\_ 1:0:0:2 sdap 66:144 [undef][ready]
\_ 0:0:1:2 sdv 65:80 [undef][ready]
\_ round-robin 0 [prio=10][undef]
\_ 1:0:1:2 sdbj 67:208 [undef][ready]
..
# chkconfig multipathd on
# chkconfig –list multipathd
multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

f. Copy /etc/multipath.conf to all remaining cluster nodes in the DB cluster. Repeat step 6 in all nodes.

g. At this point, we have setup /dev/mapper/asm* entries. asm_diskstring should be setup to match /dev/mapper/asm*.

In essence, we can either use UDEV or Multipathing facilities to implement device persistence, and permissions without requiring ASMLIB to be setup.

Update 1:
In RHEL6/OEL6, as uid/gid permissions through multipath.conf does not work (even though documentation supports these attributes), you can overcome the issue with an udev rule:
For example:

# dmsetup ls|grep p1
asmcrs11p1 (253, 23)
asmcrs01p1 (253, 15)
asmcrs02p1 (253, 14)

#cat /etc/udev/rules.d/12-dm-permissions.rules
ENV{DM_NAME}==”asmcrs01p1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”660″
ENV{DM_NAME}==”asmcrs02p1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”660″
ENV{DM_NAME}==”asmcrs03p1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”660″
ENV{DM_NAME}==”asmcrs04p1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”660″
ENV{DM_NAME}==”asmcrs05p1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”660″
ENV{DM_NAME}==”asmcrs06p1″, OWNER:=”oracle”, GROUP:=”oinstall”, MODE:=”660″

]]>