Posted by Riyaj Shamsudeen on January 19, 2012
In this blog entry, we will explore the wonderful world of SCNs and how Oracle database uses SCN internally. We will also explore few new bugs and clarify few misconceptions about SCN itself.
What is SCN?
SCN (System Change Number) is a primary mechanism to maintain data consistency in Oracle database. SCN is used primarily in the following areas, of course, this is not a complete list:
- Every redo record has an SCN version of the redo record in the redo header (and redo records can have non-unique SCN). Given redo records from two threads (as in the case of RAC), Recovery will order them in SCN order, essentially maintaining a strict sequential order. As explained in my paper, every redo record has multiple change vectors too.
- Every data block also has block SCN (aka block version). In addition to that, a change vector in a redo record also has expected block SCN. This means that a change vector can be applied to one and only version of the block. Code checks if the target SCN in a change vector is matching with the block SCN before applying the redo record. If there is a mismatch, corruption errors are thrown.
- Read consistency also uses SCN. Every query has query environment which includes an SCN at the start of the query. A session can see the transactional changes only if that transaction commit SCN is lower then the query environment SCN.
- Commit. Every commit will generate SCN, aka commit SCN, that marks a transaction boundary. Group commits are possible too.
SCN is a huge number with two components to it: Base and wrap. Wrap is a 16 bit number and base is a 32 bit number. It is of the format wrap.base. When the base exceeds 4 billion, then the wrap is incremented by 1. Essentially, wrap counts the number of times base wrapped around 4 billion. Few simple SQL script will enumerate this better:
But wait, there’s more!
Posted in 11g, corruption, Oracle database internals, Performance tuning, RAC, recovery | Tagged: get_system_change_number, hot backup scn growth, kcmgas calls, kcvblg, ORA-600 , oracle performance, SCN bug, tracefile_name | 28 Comments »
Posted by Riyaj Shamsudeen on January 13, 2012
You might encounter RAC wait event ‘gc cr disk read’ in 11.2 while tuning your applications in RAC environment. Let’s probe this wait event to understand why a session would wait for this wait event.
Understanding the wait event
Let’s say that a foreground process running in node 1, is trying to access a block using a SELECT statement and that block is not in the local cache. To maintain the read consistency, foreground process will require the block consistent with the query SCN. Then the sequence of operation is(simplified):
- Foreground session calculates the master node of the block; Requests a LMS process running in the master node to access the block.
- Let’s assume that block is resident in the master node’s buffer cache. If the block is in a consistent state (meaning block version SCN is lower (or equal?) to query SCN), then LMS process can send the block to the foreground process immediately. Life is not that simple, so, let’s assume that requested block has an uncommitted transaction.
- Since the block has uncommitted changes, LMS process can not send the block immediately. LMS process must create a CR (Consistent Read) version of the block: clones the buffer, applies undo records to the cloned buffer rolling back the block to the SCN consistent with the requested query SCN.
- Then the CR block is sent to the foreground process.
LMS is a light weight process
Global cache operations must complete quickly, in the order of milli-seconds, to maintain the overall performance of RAC database. LMS is a critical process and does not do heavy lifting tasks such as disk I/O etc. If LMS process has to initiate I/O, instead of initiating I/O, LMS will downgrade the block mode and send the block to the requesting foreground process (this is known as Light Works rule). Foreground process will apply undo records to the block to construct CR version of the block.
Posted in 11g, Oracle database internals, Performance tuning, Presentations, RAC | Tagged: gc cr disk read, RAC performance | 6 Comments »
Posted by Riyaj Shamsudeen on January 10, 2012
On February 14-16, I’ll be at the Colorado Convention Center in Denver, Colorado for RMOUG’s Training Days Conference. This is the largest regional Oracle User Conference in North America and attracts presenters from all around the country and the globe. I’ll be presenting:
Presentation Name: Troubleshooting RAC Background Process
Abstract: RAC background process performance is critical to keep the application performance. This session will demo techniques to review the performance of RAC background processes such as LMS, LMD, LMON, etc. using various statistics and UNIX tools. The presentation will also discuss why certain background processes must run in higher priority to maintain the application performance in RAC.
Presentation Name: A Kind and Gentle Introduction to RAC
Abstract: This session will introduce basic concepts such as cache fusion, conversion to RAC, protocols for interconnect, general architectural overview, GES layer locks, clusterware, etc. The session will also discuss the srvctl command and demo a few of these commands to improve the understanding.
Presentation Name: Parallel Execution in RAC
Abstract: This presentation will start to discuss and demo parallel server allocation, intra, and inter node parallelism aspects. The session will discuss the new parallelism features such as parallel statement queuing, parallel auto dop, and discuss the interaction of those features with RAC. The session will probe a few critical parameters to improve PQ performance in RAC.
Click here for more information or to register for RMOUG’s Training Days.
Posted in Oracle database internals, Performance tuning, Presentations, RAC | Tagged: oracle performance, performance, RAC performance | Leave a Comment »
Posted by Riyaj Shamsudeen on November 8, 2011
Waits for ‘DFS lock handle’ can cause massive performance issues in a busy RAC cluster. In this blog entry, we will explore the DFS lock handle wait event, and understand how to troubleshoot the root cause of these waits. I am also going to use locks and resources interchangeably in this blog, but internally, they are two different types of structures.
A little background
DFS (stands for Distributed File System) is an ancient name, associated with cluster file system operations, in a Lock manager supplied by vendors in Oracle Parallel Server Environment (prior name for RAC). But, this wait event has morphed and is now associated with waits irrelevant to database files also. Hence, it is imperative to understand the underlying details to debug the ‘DFS lock handle’ waits.
How does it work?
I have no access to the code, so read this paragraph with caution, as I may have misunderstood my test results: A process trying to acquire a lock on a global GES resource sends a AST(Asynchronous Trap) or BAST (Blocking Asynchronous Trap) message to LCK process, constructing the message with (lock pointer, resource pointer, and resource name) information. If the resource is not available, then the LCK process sends a message to the lock holder for a lock downgrade.
Posted in 11g, Oracle database internals, Performance tuning, RAC | Tagged: AST, BAST, BB enqueue, CI enqueue, DFS lock handle, GES, gv$ges_resource, oracle performance, RAC, RAC performance, SV enqueue, v$lock_type | 10 Comments »
Posted by Riyaj Shamsudeen on September 28, 2011
I will be talking about “Administering Parallel Execution in RAC” with demos on Sunday morning 9AM-10AM (session id 28060). This is part of IOUG RAC SIG presentation series. You would enjoy the content and demos I have prepared.
I know, it is too early, but hoping to see you there!
BTW, if you have attended my RAC Advanced Troubleshooting class series, please don’t hesitate to introduce yourself when we meet.
Session ID: 28060
Session Title: IOUG: Administering Parallel Execution in Oracle RAC
Venue / Room: Moscone West- 2005
Date and Time: 10/2/11, 9:00 – 10:00
Update: I just completed this session in IOUG. Thank you for coming, if you were in the room.
You can download the pdf file from
PX execution in RAC
Posted in 11g, Performance tuning, Presentations, RAC | Tagged: 11g, oracle performance, parallel_instance_group, RAC, RAC performance | 4 Comments »
Posted by Riyaj Shamsudeen on July 11, 2011
I will be conducting a 1-hour deep dive session about RAC LMS process (and about LGWR processes too if time permits) using advanced UNIX utilities. Read Tanel’s blog entry for details:
RAC hack session
See you there!
Posted in Performance tuning, Presentations, RAC | Tagged: LGWR, LMS, log file sync, oracle performance, performance logfile dump, pstack, RAC, RAC performance | 5 Comments »
Posted by Riyaj Shamsudeen on July 6, 2011
You probably aware of 1 day event happening on July 7th Thursday 9AM-5PM EDT (virtually and physically). I will be talking about advanced UNIX tools to debug issues. You can find details of 1-day event here
Here is the outline of my presentation:
Advanced tools and techniques in Unix environment – Riyaj Shamsudeen – 07/07/2011, 3:15pm – 4:00pm EDT
Unix environments provides rich set of tools to debug performance issues. Even if the issue is complex to understand, if we use right tool for the job, we can identify the root cause of an issue quickly. In this presentation, the speaker will demo tools such as truss/strace, pfiles, pmap, prstat, vmstat, mpstat etc to show how you can debug complex issues. The speaker will also introduce Solaris/dtrace to understand performance issues.
Why not join us?
Posted in Performance tuning, Presentations | Leave a Comment »
Posted by Riyaj Shamsudeen on July 5, 2011
I will be delivering an intense, advanced 2-week RAC training seminar in Aug 22-26 and Sep 19-23 with numerous demos, dumps and scripts. We will meet 4 hours per day, in a virtual world, 8AM-12 Noon Pacific time for those two weeks. Tanel has written has a great blog post about this seminar series Advanced RAC training.
If you like my presentations about RAC and performance tuning, you will love my seminar series. You will gain better understanding about RAC internals, and you will be able to advance your debugging and performance tuning skills attending my seminar series.
Join me and Let’s explore RAC further.
Posted in Oracle database internals, Performance tuning, Presentations, RAC | Tagged: oracle performance, rac online training, RAC performance, RAC training, seminar | 5 Comments »
Posted by Riyaj Shamsudeen on June 27, 2011
I will be presenting in IOUG Webinar on Thursday June 30, 2011 11AM-12Noon CDT. Hope you can join me.
Register at Private interconnect .
I know, I have not been blogging more actively, but I have been working on books and few presentations lately. Hopefully, you will see more blog entries soon.
Thanks for reading my blog.
Posted in Performance tuning, Presentations, RAC | Tagged: private interconnect, RAC | Leave a Comment »
Posted by Riyaj Shamsudeen on January 20, 2011
As you probably know that my first co-authored book Expert Oracle Practices was released in 2009. I have co-authored one more book Pro Oracle SQL with my esteemed colleagues. This books covers many aspects of better SQL development. Have fun reading :-)
I also will be presenting in few conferences in the upcoming months: RMOUG Training days 2011 , Hotsos symposium ’11, and IOUG Collob . Hopefully, I will see you in one of the conference (or all of the conferences)
Posted in Performance tuning, Presentations | 2 Comments »