Oracle database internals by Riyaj

Discussions about Oracle performance tuning, RAC, Oracle internal & E-business suite.

Dynamic Resource Mastering in 12c

Posted by Riyaj Shamsudeen on February 28, 2014

I blogged about Dynamic Resource Mastering (DRM) in RAC here . DRM freezes the global resources during the reconfiguration event and no new resources can be allocated during the reconfiguration. This freeze has a dramatic effect of inducing huge amount of waits for gc buffer busy [acquire|release] events and other gcs drm freeze release, gcs remaster events. In database version 12c, DRM has been improved further.

A major improvement I see is that not all resources are frozen at any time. Essentially, resources are broken down in to partitions and only a resource partition is frozen. This improvement should decrease the impact of DRM related waits tremendously.

LMON Trace file

Following shows the snippet from the LMON trace file. As you see, only one resource partition is frozen, at-a-time. Resources in the first partition is frozen, completes the resource remastering tasks, and unfreezes that resource partition. Then freezes next resource partition and continues until all resources are remastered.

* drm quiesce
2014-02-15 16:03:45.131840 : DRM(3) resources quiesced [0-1023], rescount 1719  <--- Note this line
2014-02-15 16:03:45.132173 : DRM(3) local converts quiesced [0-1023], lockcount 0, bucket 0
* drm sync 1
* drm freeze
* drm cleanup
* drm sync 2
* drm replay
* drm sync 3
* drm fix writes
* drm sync 4
* drm end

* DRM RCFG called (swin 0)
  CGS recovery timeout = 85 sec
* drm quiesce
2014-02-15 16:03:45.172867 : DRM(3) resources quiesced [1024-2047], rescount 1754 <-- and this line.
2014-02-15 16:03:45.172932 : DRM(3) local converts quiesced [1024-2047], lockcount 0, bucket 0
* drm sync 1
* drm freeze
* drm cleanup
* drm sync 2
* drm replay
* drm sync 3
* drm fix writes
* drm sync 4
* drm end
...
2014-02-15 16:03:45.191899 : DRM(3) resources quiesced [2048-3071], rescount 1732 <-- and this line
..
2014-02-15 16:03:45.211919 : DRM(3) resources quiesced [3072-4095], rescount 1746 <-- and this line

In another LMON trace file, number of partitions were 8 and resource count in a resource partition was much higher. So, partition count doesn’t seem to be a fixed number, probably, derived from the number of configured resources.

* drm quiesce
2014-02-28 11:14:54.701259 : DRM(3) resources quiesced [0-8191], rescount 10064
2014-02-28 11:14:54.701298 : DRM(3) local converts quiesced [0-8191], lockcount 0, bucket 0
* drm sync 1
* drm freeze
* drm cleanup
* drm sync 2
* drm replay
* drm sync 3
* drm fix writes
* drm sync 4
* drm end

* DRM RCFG called (swin 0)
CGS recovery timeout = 85 sec
* drm quiesce
2014-02-28 11:14:54.762057 : DRM(3) resources quiesced [8192-16383], rescount 10067
2014-02-28 11:14:54.762097 : DRM(3) local converts quiesced [8192-16383], lockcount 0, bucket 0
* drm sync 1
* drm freeze
* drm cleanup
* drm sync 2
* drm replay
* drm sync 3
* drm fix writes
* drm sync 4
* drm end

LMS Trace file

As we know already, multiple LMS processes will work on the resources in parallel, completing the reconfiguration quickly. In 11g and 12c beta, this information was written to LMS trace file. However, it looks like, this information is written by a brand new RMV0 background process. I still think that work is done by the LMS process, just the trace file writes are handled by RMV0 process (I can’t glean much information about RMV0 process either).

In 12c beta:
===========
* lms 0 finished parallel drm freeze in DRM(3) window 1, pcount 61
2014-02-15 16:03:45.165509 :  1512 GCS shadows traversed, 0 replayed in drm replay
DRM(3) win(1) lms 0 finished replaying gcs resources
DRM(3) win(1) lms 0 finished fixing gcs write protocol
DRM(3) quiesced basts [1024-2047]
2014-02-15 16:03:45.175816 :
* lms 0 finished parallel drm freeze in DRM(3) window 2, pcount 61
2014-02-15 16:03:45.182015 :  1517 GCS shadows traversed, 0 replayed in drm replay
DRM(3) win(2) lms 0 finished replaying gcs resources
DRM(3) win(2) lms 0 finished fixing gcs write protocol
DRM(3) quiesced basts [2048-3071]
...

In 12.1.0.1, following lines are written to RMV0 trace file. Point is that if you are trying to review the DRM speed, you will have to review RMV trace files from 12.1.0.1 onwards, not just LMS trace files.

<pre
2014-02-28 11:14:54.722162 :
* lms 0 finished parallel drm freeze in DRM(3) window 1, pcount 80
2014-02-28 11:14:54.746119 : 457 GCS shadows traversed, 0 replayed in drm replay
DRM(3) win(1) rmslv 0 finished replaying gcs resources
DRM(3) win(1) rmslv 0 finished fixing gcs write protocol
DRM(3) quiesced basts [8192-16383]
2014-02-28 11:14:54.792176 :
* lms 0 finished parallel drm freeze in DRM(3) window 2, pcount 80
2014-02-28 11:14:54.816115 : 453 GCS shadows traversed, 0 replayed in drm replay
DRM(3) win(2) rmslv 0 finished replaying gcs resources
DRM(3) win(2) rmslv 0 finished fixing gcs write protocol
DRM(3) quiesced basts [16384-24575]

Also, in 12c, more processes are designed to run in elevated priority. Here is the value of the _high_priority_processes parameter:

Parameter: _high_priority_processes
Value : LMS*|VKTM|LM*|LCK0|GCR*|DIAG

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 215 other followers

%d bloggers like this: