However by the following lines in the trace file it appears that the session is suffering from ‘transaction table consistent reads – undo records applied’.

WAIT #2: nam=’db file sequential read’ ela= 4278 file#=5 block#=1351837 blocks=1 obj#=0 tim=5088627255019 –> here block being waited for is from an UNDO segment.

WAIT #2: nam=’gc cr disk read’ ela= 593 p1=5 p2=1351834 p3=252 obj#=0 tim=5088627255928 –> again blocks are read from UNDO segment for UNDO apply for transaction table consistent reads.

I will also dump the blocks next time when this issue occurs again( I hope it is better to dump the data block( here UNDO block) when this issue is happening)

Thanks again for sparing your valuable,

Bheemasen

For commit cleanouts:- undo header blocks (transaction table entries) and undo blocks associated with the transaction table control block changes must be read. The transaction table in the undo header block will indicate the transaction state. But, if the transaction table slot is overwritten for that transaction (and so commit SCN is not immediately available), then Database code will follow undo chain of the transaction table control blocks to create a prior version of transaction table recursively, in a quest to find commit SCN of the transaction.

For consistent reads:- undo chain of a transaction (a transaction that is either active or commit SCN of that transaction is greater than query SCN) must be followed for consistent read buffer generation.

First thing is to differentiate between two events: Does the session suffer from ‘data blocks consistent reads – undo records applied’ or ‘transaction table consistent reads – undo records applied” issue? (Stats names can be little different between versions) Check the session level stats to identify that:

undefine sid
undefine search_string
accept search_string prompt 'Enter the wildcard character (Null=All):'
accept value_threshold prompt 'Enter value threshold :'
accept sid prompt 'Enter sid :'
select  stat.name, ses.value from
v$sesstat ses , v$statname stat
where stat.statistic#=ses.statistic# and
ses.sid=&sid
and lower(stat.name) like  lower('%'||nvl('&&search_string','%')||'%')
and ses.value >= to_number(nvl('&&value_threshold',0))
/

As an additional check, review the block contents by dumping the blocks. Check if the undo records are indeed application table changes. You may have to dump the blocks to see contents. alter system dump datafile block min block max is the syntax.

(1) If the blocks are undo header blocks and stats indicate high ‘transaction table consistent reads – undo records applied’, then it is possible that you are running in to a side effect of auto undo retention tuning feature. Essentially, auto tuning of undo retention tries to reduce probability of ORA-1555 errors, by keeping as much undo possible tuning undo_retention upwards. But, this can negatively affect commit cleanouts since the code is seemingly trying to find exact commit SCN by rolling back the transaction table header changes numerous times.

In this scenario, you can setup max undo_retention and reduce the effects of auto tuned undo retention feature. you can also setup a job to select from the table, serially, and perform commit cleanouts in the same node.

(2) If the changes are for CR generation: (not undo header blocks), then it would imply that these blocks are read for Consistent Read generation. Meaning, few processes have updated objects heavily in one node,and sessions in other nodes are reading the blocks, but can not use the blocks since there is a pending transaction (or that commit SCN is later then query env SCN), and so rolling back the transactions to create a Consistent Read copies.

You could solve this problem with two approaches. If node 3 is modifying blocks heavily by (a) either schedule those intensive updates during less active period, so that, you can avoid these CR storm (b) Modify the application affinity such a way that programs modifying the objects aggressively and programs reading the blocks aggressively are executed in the same node. Features such as “in memory undo” etc doesn’t work well with RAC and so, other node sessions might have to read the blocks physically.

I do see that light_works in node 3 is much higher than other nodes, this would indicate that node 3 is performing massive changes and other nodes are reading the blocks from node 3.

I thought, I was going to write few words

HTH

Cheers,
Riyaj

WAIT #2: nam=’db file sequential read’ ela= 7208 file#=5 block#=1352468 blocks=1 obj#=0 tim=5088625409951
WAIT #2: nam=’gc cr disk read’ ela= 446 p1=5 p2=1352466 p3=252 obj#=0 tim=5088625410501
WAIT #2: nam=’db file sequential read’ ela= 6460 file#=5 block#=1352466 blocks=1 obj#=0 tim=5088625417000
WAIT #2: nam=’gc cr disk read’ ela= 507 p1=5 p2=1352463 p3=252 obj#=0 tim=5088625417585
WAIT #2: nam=’db file sequential read’ ela= 4773 file#=5 block#=1352463 blocks=1 obj#=0 tim=5088625422392
WAIT #2: nam=’gc cr disk read’ ela= 681 p1=5 p2=1352460 p3=252 obj#=0 tim=5088625423231
WAIT #2: nam=’db file sequential read’ ela= 9462 file#=5 block#=1352460 blocks=1 obj#=0 tim=5088625432742

Here you can see that only UNDO blocks are read and there is no request for data block. Is this due to commit clean out? This is THREE node 11gR1 RAC. We kill the session and rerun the job then it goes through.

We find this issue happening when ever this select query runs from either node1 or node2 and UNDO blocks are from node3( file#=5 is datafile for UNDO tablespace for node3)

Here is some more information on traffic between the nodes.

SQL> select INST_ID,CR_REQUESTS,CURRENT_REQUESTS,UNDO_REQUESTS,DISK_READ_RESULTS,FAIRNESS_DOWN_CONVERTS,LIGHT_WORKS,ERRORS from gv$cr_block_server order by 1;

INST_ID CR_REQUESTS CURRENT_REQUESTS UNDO_REQUESTS DISK_READ_RESULTS FAIRNESS_DOWN_CONVERTS LIGHT_WORKS     ERRORS
------- ----------- ---------------- ------------- ----------------- ---------------------- ----------- ----------
      1    54693551          6422699       1676129           7381302                6125434     4608074          0
      2     2948493          3134301       2661336           2829933                 352236      157720          0
      3    58158519         25104616      18365357          19676544                6631054    16605103          0

To avoid this commit clean out we need to find which process did modify the blocks in the node3 so that it can be run on the other nodes. We are not able to find it.

Thanks for stopping by.

A good whole chapter could be written about preparation of a tablespace to be made read only for a substantial period of time.

In extreme cases someone might also need to hermitize* sequences in support of application affinity so that blocks of seq$ are spared from excessive instance migration as well.

*hermitize meaning to isolate rows one per block (like hermits, one per dwelling place). Oracle normally stores about 75 sequence definitions per 8K block. The block grouping tends to be by order of definition and so may cross application affinity boundaries. Listing out the sequence names grouped by block can be quite illuminating.