Posted by Riyaj Shamsudeen on November 12, 2013
It is easier to create one or two AWR reports quickly using OEM. But, what if you have to create AWR reports for many snapshots? For example, your Oracle support analyst wants you to supply 10 1-hour AWR reports from 10AM to 8PM in a 8 node cluster? That’s about 80 AWR reports to create! Okay, okay, I may(!) be overselling it, but you get the point. It is useful to have a script to create AWR report for all instances for a given range of snapshot IDs. Following scripts are handy:
|1. To create one AWR report per instance, for the last snap duration :
|2. Same as (1) but in html format :
|3. To create one AWR report per instance, for a range of snap IDs :
|4. To create one AWR report, per instance, per snap ID :
Zip file: awrrpt_scripts
These scripts do not modify anything in the database, just retrieves the data using dbms_workload_repository package. Test the scripts to understand further. Of course, you need access to dbms_workload_repository and access to gv$instance.
Posted in Oracle database internals, Performance tuning, RAC | Tagged: AWR reports, awrrpt.sql, awrrpt_all_gen.sql, awrrpt_all_range_gen.sql | 1 Comment »
Posted by Riyaj Shamsudeen on September 9, 2013
I blogged about DFS lock handle contention in an earlier blog entry. SV resources in Global Resource Directory (GRD) is used to maintain the cached sequence values. I will further probe the internal mechanics involved in the cached sequences. I will also discuss minor changes in the resource names to support pluggable databases (version 12c).
Let’s create an ordered sequence in rs schema and then query values from the sequence few times.
create sequence rs.test_seq order cache 100;
select rs.test_seq.nextval from dual; -- repeated a few times.
Sequence values are permanently stored in the seq$ dictionary table. Cached sequence values are maintained in SV resources in GRD and SV resource names follows the naming convention to include object_id of the sequence. I will generate a string using a small helper script and we will use that resource name to search in the GRD.
SELECT DISTINCT '[0x'
|| '],[SV]' res
FROM dba_objects WHERE object_name=upper('&objname')
AND owner=upper('&owner') AND object_type LIKE 'SEQUENCE%'
Enter value for objname: TEST_SEQ
Enter value for owner: RS
Read the rest of this entry »
Posted in 12c, Oracle database internals, Performance tuning, RAC, weird stuff | Tagged: oracle performance, pluggable database, RAC internals, RAC performance, SV resource, weird stuff | 2 Comments »
Posted by Riyaj Shamsudeen on September 8, 2013
A quick note, Expert Oracle RAC book co-written by me is available now: Expert Oracle RAC 12c. I have written about 6 chapters covering the RAC internals that you may want to learn I even managed to discuss the network internals in deep, after all, network is one of the most important component of a RAC cluster.
Posted in 12c, Oracle database internals, Performance tuning, RAC | Tagged: oracle performance, performance, RAC internals, RAC performance | Leave a Comment »
Posted by Riyaj Shamsudeen on June 12, 2013
This blog entry is to discuss a method to identify the objects inducing higher amount of redo. First,we will establish that redo size increased sharply and then identify the objects generating more redo. Unfortunately, redo size is not tracked at a segment level. However, you can make an educated guess using ‘db block changes’ statistics. But, you must use logminer utility to identify the objects generating more redo scientifically.
Detecting redo size increase
AWR tables (require Diagnostics license) can be accessed to identify the redo size increase. Following query spools the daily rate of redo size. You can easily open the output file redosize.lst in an Excel spreadsheet and graph the data to visualize the redo size change. Use pipe symbol as the delimiter while opening the file in excel spreadsheet.
REM You need Diagnostic Pack licence to execute this query!
REM Author: Riyaj Shamsudeen
col begin_interval_time format a30
set lines 160 pages 1000
col end_interval_time format a30
set colsep '|'
alter session set nls_date_format='DD-MON-YYYY';
with redo_sz as (
SELECT sysst.snap_id, sysst.instance_number, begin_interval_time ,end_interval_time , startup_time,
VALUE - lag (VALUE) OVER ( PARTITION BY startup_time, sysst.instance_number
ORDER BY begin_interval_time, startup_time, sysst.instance_number) stat_value,
EXTRACT (DAY FROM (end_interval_time-begin_interval_time))*24*60*60+
EXTRACT (HOUR FROM (end_interval_time-begin_interval_time))*60*60+
EXTRACT (MINUTE FROM (end_interval_time-begin_interval_time))*60+
EXTRACT (SECOND FROM (end_interval_time-begin_interval_time)) DELTA
FROM sys.wrh$_sysstat sysst , DBA_HIST_SNAPSHOT snaps
WHERE (sysst.dbid, sysst.stat_id) IN ( SELECT dbid, stat_id FROM sys.wrh$_stat_name WHERE stat_name='redo size' )
AND snaps.snap_id = sysst.snap_id
AND snaps.dbid =sysst.dbid
and begin_interval_time > sysdate-90
, sum(stat_value) redo1
group by instance_number,
order by instance_number, 2
Visualizing the data will help you to quickly identify any pattern anomalies in redo generation. Here is an example graph created from the excel spreadsheet and see that redo size increased recently.
Read the rest of this entry »
Posted in 11g, Oracle database internals, Performance tuning, RAC | Tagged: identify objects redo, redo internals, segment_stats.sql, v$logmnr_contents, v$segment_stats | 8 Comments »
Posted by Riyaj Shamsudeen on October 22, 2012
Please join us at the DOUG (DALLAS ORACLE USERS GROUP) Oracle Database Forum meeting on Thursday, October 25, 2012 from 5 pm – 7 pm.
Presented by Riyaj Shamsudeen, OraInternals, & Sahil Thapar:
“Out with the old way, Enter dbms_xplan: A Swiss army knife for performance engineers”
(i) Ability to query access path from memory, AWR repository
(ii) Ability to use cardinality feedback method to understand access plan issues. Few tips from a real world experience will be provided too.
(iii) Ability to understand issues with database links etc.
(iv) Options such as ADVANCED, ALLSTATS etc
(v) Why should you choose dbmx_xplan over tkprof+sql_trace combination?
(vi) Disadvantages of dbms_xplan and a quick introduction to dbms_monitor.
Refreshments sponsored by me
Update: Uploading the presentation pdf files. Enjoy
Posted in Performance tuning, Presentations | Tagged: dbms_xplan, dbms_xplan advanced, dbms_xplan allstats last, display_awr, display_cursor | 5 Comments »
Posted by Riyaj Shamsudeen on August 18, 2012
Just a quick note, I will be presenting on “Truss, pstack, pmap, and more” talking about advanced UNIX utilities and how it can be utilized to understand inner working of an application or even Oracle Database Engine.
My timeslot is between 2:15 and 3:15 in Room 2016.
Uploading presentation files. Thanks for attending at OOW12.
Posted in Oracle database internals, Performance tuning | Tagged: oracle performance, pmap, pstack, truss | 1 Comment »
Posted by Riyaj Shamsudeen on June 15, 2012
Quick note about Jonathan Lewis trip to Dallas: Jonathan Lewis will be presenting two day seminar on two topics, “Beating the Oracle Optimizer” (June 28) and “Troubleshooting and tuning” (June 29th).
The event will be held June 28-29, 2012 at SMU-in-Legacy in Plano, TX.
This is a must-attend event for experienced DBAs and Developers. Especially, if you are planning to upgrade your database/application in the near-future or if you are in the middle of an upgrade, you must attend these two seminars. This seminar series provide enormous value resolving complex Production performance issues.
Click Here for details.
Posted in Performance tuning, Presentations | Tagged: cost based optimizer presentations, oracle performance | Leave a Comment »
Posted by Riyaj Shamsudeen on June 1, 2012
This is a quick note about reverse path filtering and impact of that feature to RAC. I encountered an interesting problem recently with a client and it is worth blogging about it, with a strong hope that it might help one of you in the future.
Environment is 22.214.171.124 GI, Linux 5.6. In a 3 node cluster, Grid Infrastructure (GI) comes up cleanly in just one node, but never comes up in other nodes. If we shutdown GI in first node, we can start the GI in second node with no issues. Meaning, GI can be up in just one node at any time.
System Admins indicated that there are no major changes, only few bug fixes. Seemingly, problem started after those bug fixes. But there were few other changes to the environment /init.ora parameter change etc. So, the problem was not immediately attributable to just OS changes.
Read the rest of this entry »
Posted in Oracle database internals, Performance tuning, RAC | Tagged: "has Disk HB, advanced RAC training, but no Network HB", cssd not joining cluster, RAC performance, reverse path filtering, rp_filter | 10 Comments »
Posted by Riyaj Shamsudeen on May 22, 2012
Let’s first discuss how RAC traffic works before continuing. Environment for the discussion is: 2 node cluster with 8K database block size, UDP protocol is used for cache fusion. (BTW, UDP and RDS protocols are supported in UNIX platform; whereas Windows uses TCP protocol).
UDP protocol, fragmentation, and assembly
UDP Protocol is an higher level protocol stack, and it is implemented over IP Protocol ( UDP/IP). Cache Fusion uses UDP protocol to send packets over the wire (Exadata uses RDS protocol though).
MTU defines the Maximum Transfer Unit of an IP packet. Let us consider an example of MTU set to 1500 in a network interface. One 8K block transfer can not be performed with just one IP packet as the IP packet size (1500 bytes) is less than 8K. So, one transfer of UDP packet of 8K size is fragmented to 6 IP packets and sent over the wire. In the receiving side, those 6 packets are reassembled to create one UDP buffer of size 8K. After the assembly, that UDP buffer is delivered to an UDP port of a UNIX process. Usually, a foreground process will listen on that port to receive the UDP buffer.
Read the rest of this entry »
Posted in 11g, Oracle database internals, Performance tuning, Presentations, RAC, video | Tagged: cache fusion mtu, fragmentation and reassembly, gc lost packets, ipfrag_high_thres, ipfrag_low_thres, ipfrag_time, Jumbo frames, MTU, MTU=9000, oracle performance, RAC internals, RAC performance, RAC presentations, RAC training, RAC video, RAC videos, RDS, UDP vs tcp, wireshark | 10 Comments »
Posted by Riyaj Shamsudeen on April 29, 2012
We know that database blocks are transferred between the nodes through the interconnect, aka cache fusion traffic. Common misconception is that packet transfer size is always database block size for block transfer (Of course, messages are smaller in size). That’s not entirely true. There is an optimization in the cache fusion code to reduce the packet size (and so reduces the bits transferred over the private network). Don’t confuse this note with Jumbo frames and MTU size, this note is independent of MTU setting.
In a nutshell, if free space in a block exceeds a threshold (_gc_fusion_compression) then instead of sending the whole block, LMS sends a smaller packet, reducing private network traffic bits. Let me give an example to illustrate my point. Let’s say that the database block size is 8192 and a block to be transferred is a recently NEWed block, say, with 4000 bytes of free space. Transfer of this block over the interconnect from one node to another node in the cluster will result in a packet size of ~4200 bytes. Transfer of bytes representing free space can be avoided completely, just a symbolic notation of free space begin offset and free space end offset is good enough to reconstruct the block in the receiving side without any loss of data.This optimization makes sense as there is no need to clog the network unnecessarily.
Read the rest of this entry »
Posted in 11g, Oracle database internals, Performance tuning, RAC | Tagged: RAC internals, RAC performance, RAC performance myths, _gc_fusion_compression | 9 Comments »