Comments on: Log file synch tuning #2

By: Log File Write And Waits « Vishal Gupta's Blog

Log File Write And Waits « Vishal Gupta's Blog — Sat, 18 Jun 2011 20:02:37 +0000

[...] Riyaj Shamsudeen – Log file sync tuning [...]

By: qihua

qihua — Thu, 08 Apr 2010 09:10:11 +0000

I also have a question about the redo write.

the statistics of “redo write” is the number of time lgwr writes the data, right? For our database, the times of write on OS level (vxstat) is about 6 times as the “redo write”, we didn’t multiplex the redo, and redo file is quick io file. The redo volume is separated from other data volume, so the vxstat reported on redo volume should be equal to the write IO from lgwr. I can’t understand why one “redo write” was broken into 6 OS IO.

By: qihua

qihua — Thu, 08 Apr 2010 09:05:43 +0000

For your following dtruss output
6924/5: 240044 230587 pwrite(0×103, “”, 0×100000) = 1048576 0 ……….(1)
6924/5: 240091 39 kaio(0×4, 0×0, 0x1038E9DD0) = 0 0
6924/1: 2447794 232035 kaio(0×2, 0xFFFFFFFF7FFFD400, 0×0) = 1 0

We can’t see whether the pwrite and kaio is on the same file or not. When I trussed a lgwr on my database, I found pwrite is for controfile, and kaio is for online redo log.

Can you also add the file handler information to pwrite and kaio so it is clear which file was written?

By: Log File Write And Waits « Vishal Gupta's Blog

Log File Write And Waits « Vishal Gupta's Blog — Sun, 14 Feb 2010 22:46:06 +0000

[...] Riyaj Shamsudeen – Log file sync tuning [...]

By: Darshan

Darshan — Thu, 04 Feb 2010 16:55:54 +0000

Riyaj,

Thanks for fast response. I will try and let you what all we see.

Regards,
Darshan

By: orainternals

orainternals — Thu, 04 Feb 2010 16:47:43 +0000

Darshan
Try this script:

http://www.orainternals.com/scripts_perf1.php

Look for “dtrace script to trace Oracle processes”.

http://www.orainternals.com/viewfile.php?id=46

Beware that, in kernel land, functions are not well documented and can be misleading if you don’t understand it properly. Many function calls are overloaded.

Cheers
Riyaj

By: Darshan

Darshan — Thu, 04 Feb 2010 16:25:50 +0000

Riyaj,

Thanks for your response. We do not have CPU issue. It is around 10% only.
We are using Vxfs veritas volume manager on netapp. I provided v$active_session__history waits in my first comment, where we do not see it is spending much time on I/o “log file parallel write”.

Could you please provide dtrace script for lgwr? That should help to figure where it is spending time.

By: orainternals

orainternals — Thu, 04 Feb 2010 15:34:22 +0000

Darshan
Sorry for the late reply. Looks like, replying automatic email from wordpress does not show up in the comment!

Have you checked bugs with ‘log file sync’ event waits in 10.2.0.3. There are couple of bugs such as bug 8842553. It is quite possible that you are hitting one of these bugs.
Also, Is LGWR getting enough CPU? Is the file system properly mounted with direct I/O? There are many things that can go wrong, so, it may be essential to dtrace LGWR to see why LGWR is not performing. What is LGWR waiting as per v$active_session__history?
dtrace is, by design, safe. So, there is no downside to it. Let me know, if you want a script to dtrace LGWR writes.
Cheers
Riyaj

By: Darshan

Darshan — Sat, 30 Jan 2010 19:49:38 +0000

Hi Riyaj,

I have read couple of your articles. All are well composed and lots of good information.

I have similar issue on one of the DB on Solaris 10, Oracle 10.2.0.3.

From ASH details I see sometimes Log file sync time approching more then second even though there is no much load on the system, or lots of redo generation. At the same time I do not see “log file parallel write” taking too much time. I do not want try out kernal tracing tools like truss or dtrace on very critical production DB. Any other ways to go further in debugging. Here is the ASH for that spike with LGWR and other sessions waits.

SAMPLE_TIME SID# ————— 29-JAN-10 08.01.27.685 AM 4401,1 29-JAN-10 08.01.29.705 AM 4401,1 29-JAN-10 08.01.39.855 AM 4401,1 29-JAN-10 08.01.41.875 AM 4401,1 29-JAN-10 08.01.43.905 AM 1297,356 29-JAN-10 08.01.43.905 AM 1301,551 29-JAN-10 08.01.43.905 AM 1327,60 29-JAN-10 08.01.43.905 AM 1329,1945 29-JAN-10 08.01.43.905 AM 1330,4 29-JAN-10 08.01.43.905 AM 1345,314 29-JAN-10 08.01.43.905 AM 1357,3 29-JAN-10 08.01.43.905 AM 1369,114 29-JAN-10 08.01.43.905 AM 1377,199 29-JAN-10 08.01.43.905 AM 1437,20533 29-JAN-10 08.01.43.905 AM 1468,54740 29-JAN-10 08.01.43.905 AM 1881,30949 29-JAN-10 08.01.43.905 AM 2105,63152 29-JAN-10 08.01.43.905 AM 2136,54766 29-JAN-10 08.01.43.905 AM 3500,298 29-JAN-10 08.01.43.905 AM 3514,2490 29-JAN-10 08.01.43.905 AM 3534,1 29-JAN-10 08.01.43.905 AM 3591,796 29-JAN-10 08.01.43.905 AM 3597,296 29-JAN-10 08.01.43.905 AM 3894,35906 29-JAN-10 08.01.43.905 AM 4401,1 29-JAN-10 08.01.45.545 AM 1297,356 29-JAN-10 08.01.45.545 AM 1301,551 29-JAN-10 08.01.45.545 AM 1324,15 29-JAN-10 08.01.45.545 AM 1327,60 29-JAN-10 08.01.45.545 AM 1329,1945 29-JAN-10 08.01.45.545 AM 1330,4 29-JAN-10 08.01.45.545 AM 1345,314 29-JAN-10 08.01.45.545 AM 1357,3 29-JAN-10 08.01.45.545 AM 1359,101 29-JAN-10 08.01.45.545 AM 1369,114 29-JAN-10 08.01.45.545 AM 1377,199 29-JAN-10 08.01.45.545 AM 1410,405 29-JAN-10 08.01.45.545 AM 1437,20533 29-JAN-10 08.01.45.545 AM 1468,54740 29-JAN-10 08.01.45.545 AM 1878,971 29-JAN-10 08.01.45.545 AM 1881,30949 29-JAN-10 08.01.45.545 AM 2105,63152 29-JAN-10 08.01.45.545 AM 2136,54766 29-JAN-10 08.01.45.545 AM 3260,11235 29-JAN-10 08.01.45.545 AM 3495,632 29-JAN-10 08.01.45.545 AM 3500,298 29-JAN-10 08.01.45.545 AM 3501,34 29-JAN-10 08.01.45.545 AM 3502,218 29-JAN-10 08.01.45.545 AM 3514,2490 29-JAN-10 08.01.45.545 AM 3534,1 29-JAN-10 08.01.45.545 AM 3556,69 29-JAN-10 08.01.45.545 AM 3591,796 29-JAN-10 08.01.45.545 AM 3596,444 29-JAN-10 08.01.45.545 AM 3597,296 29-JAN-10 08.01.45.545 AM 3602,1017 29-JAN-10 08.01.45.545 AM 3622,232 29-JAN-10 08.01.45.545 AM 3813,61056 29-JAN-10 08.01.45.545 AM 3894,35906 29-JAN-10 08.01.45.545 AM 4401,1 29-JAN-10 08.01.48.575 AM 4401,1 29-JAN-10 08.01.57.725 AM 4401,1 29-JAN-10 08.01.58.735 AM 4401,1 29-JAN-10 08.02.02.775 AM 4401,1 SEQ# EVENT P1 P2 P3 WAIT_TIME TIME_WAITED mic sec
———– ————— ———- ————————- ———- ———- ———- ———- ———–
44462 log file parallel write 2 6 2 0 767
44904 0 0 0 1104 0
47010 0 0 0 1250 0
47457 0 0 0 775 0
52089 log file sync 5116 0 0 0 983094
23265 log file sync 5116 0 0 0 977351
16461 log file sync 5116 0 0 0 985489
34813 log file sync 5116 0 0 0 979476
32679 log file sync 5116 0 0 0 979009
23435 log file sync 5116 0 0 0 982434
59819 log file sync 5116 0 0 0 986190
58771 log file sync 5116 0 0 0 982554
42979 log file sync 5116 0 0 0 983632
17258 log file sync 5116 0 0 0 982394
44957 log file sync 5116 0 0 0 985416
51016 log file sync 5116 0 0 0 985110
30301 log file sync 5116 0 0 0 979693
60438 log file sync 5116 0 0 0 977066
34960 log file sync 5116 0 0 0 979268
12034 log file sync 5109 0 0 0 986379
2637 log file sync 5116 0 0 0 978855
32231 log file sync 5116 0 0 0 986208
19271 log file sync 5116 0 0 0 977616
21366 log file sync 5116 0 0 0 981770
47716 LGWR wait for redo copy 9 0 0 0 19655
52089 log file sync 5116 0 0 0 986282
23265 log file sync 5116 0 0 0 943796
65205 log file sync 5116 0 0 0 977774
16461 log file sync 5116 0 0 0 777708
34813 log file sync 5116 0 0 0 914354
32679 log file sync 5116 0 0 0 787733
23435 log file sync 5116 0 0 0 357618
59819 log file sync 5116 0 0 0 986153
7231 log file sync 5116 0 0 0 980302
58771 log file sync 5116 0 0 0 337973
42979 log file sync 5116 0 0 0 986484
31493 log file sync 5116 0 0 0 985918
17258 log file sync 5116 0 0 0 533481
44957 log file sync 5116 0 0 0 494599
3871 log file sync 5116 0 0 0 866648
51016 log file sync 5116 0 0 0 866072
30301 log file sync 5116 0 0 0 581489
60438 log file sync 5116 0 0 0 986590
24180 log file sync 5116 0 0 0 619846
38628 log file sync 5116 0 0 0 442214
34960 log file sync 5116 0 0 0 986723
63119 log file sync 5116 0 0 0 480076
42923 log file sync 5116 0 0 0 978969
12034 log file sync 5109 0 0 0 442955
2637 log file sync 5116 0 0 0 318910
2417 log file sync 5116 0 0 0 592775
32231 log file sync 5116 0 0 0 406718
26541 log file sync 5116 0 0 0 651510
19271 log file sync 5116 0 0 0 495818
42070 log file sync 5116 0 0 0 771974
51276 log file sync 5116 0 0 0 872151
37110 log file sync 5116 0 0 0 601072
21366 log file sync 5116 0 0 0 826675
47716 LGWR wait for redo copy 9 0 0 0 19473
48286 log file parallel write 2 2 2 0 772
50746 0 0 0 592 0
51050 0 0 0 472 0
51917 log file parallel write 2 10 2 0 1002

By: orainternals

orainternals — Thu, 11 Dec 2008 02:25:59 +0000

Hi Hans-Peter
Thanks for providing more information here.
I saw your excellent dtrace script and discussion in this open solaris thread . Excellent work. As visible in that thread, Hans-Peter, Jim Mauro and others were able to pinpoint that, problem is really due to ufs single writer lock and read-modify-write issue.
Further, David Miller identified this as bug 5752399 and there is a mandatory patch for 10.2.0.3 version for veritas or solsite disk. One of the line in the bug text clearly indicate:
“You may also observe that direct I/O is not working, despite having the parameter filesystemio_options set to an appropriate value in your init.ora or spfile.ora file”

Essentially, file systems need to be (re)mounted with forcedirectio for direct I/O to work correctly.

Interesting issue.

Cheers
Riyaj