Cheers
Riyaj

Thanks for the quick update. I’m trying to address the concurrent request that took forever due to the ‘log file sync’ wait event. I referred the other request set and it’s CPU usage (which’s not what ppl. are worrying about), because you had asked to check the CPU usage (guideline # 2).

Here’re the answers for your questions :

This is the standard Oracle report (Journal Entries Report). It’s a good point to drop/rebuild the indices to reduce the redo generation. I’ve never thought about that. Since this is not a real production, I think, we can think about it.

It’s a straight Insert statement with bind variables.

We’ve 4 log members (4 group with one each) of 1.5 Gb each. Ours is a JFS2 type of file system.

The issue here is that, this is the break fix environment that’s being refreshed every night from the copy from production.

Thanks once again and cheers
Muthu

Cheers
Riyaj

If you are trying to tune the request, I guess, redo size need to be reduced (or) LGWR & its components need to be tuned. Since there are 5 requests running, all of them may be generating much redo. Is this a custom program or standard seeded program? Is there a possibility of reducing redo size by dropping few indices on that table and then rebuild them later? Look for the opportunities to tune them.

Also, all these inserts, are they inserts with bind variables? or ‘insert into table select * from table1′ form of SQLS?

How many Log members you have? What type of file system? How big are the log files?

Sorry to ask you more questions, but I would like to understand the root cause before giving you some suggestions.

Cheers
Riyaj

Wonderful article and happy to read you. We’ve an issue with the same wait event. Here’s our scenario :

We’re running Oracle EBS 11i in AIX 5.3 (16 CPU and 60 Gb RAM) SGA is 3G and log_buffer is 10 M. Database is running in no archive log mode.

This is our breakfix environment and only one concurrent request is running for hours, waiting on this and ‘db file sequential read’ on an Insert statement.

I collected the stats as follows :

  SID  STAT# NAME                                      VALUE
----- ------ ------------------------------ ----------------
   95    115 redo size                          -708,973,184
   95      5 user rollbacks                                0
   95    117 redo wastage                                  0
   95    118 redo writer latching time                     0
   95    121 redo write time                               0
   95    125 redo ordering marks                           0
   95    124 redo log switch interrupts                    0
   95    120 redo blocks written                           0
   95    119 redo writes                                   0
   95      0 logons cumulative                             1
   95      1 logons current                                1
   95      4 user commits                                  3
   95    116 redo buffer allocation retries               30
   95    122 redo log space requests                      38
   95      6 user calls                                   75
   95    123 redo log space wait time                    425
   95     73 redo synch time                       2,265,549
   95     72 redo synch writes                    18,909,918
   95    114 redo entries                         60,812,763

Output from Tanel's tool : (edited for clarity, of course)

----------------------------------------------------------------
 SID, TYPE, STATISTIC                  DELTA, HDELTA/SEC,  %TIME
----------------------------------------------------------------
  95, WAIT, log file sync            2076179,   692.06ms,  69.2%
  95, WAIT, db file sequential read    58498,     19.5ms,   1.9%
--  End of snap 1, end=2009-08-07 18:06:06, seconds=3

User commits is low; so is user calls and redo blocks written is 0. Only redo synch time, redo synch writes and redo entries are huge.

(And the redo size is a negative number. Metalink says this is a bug in the pre-10g releases (ours is 9.2.0.7) and asked to refer the ‘redo blocks written’)

I had checked the CPU usage. In the morning, there was a request set that laucnhed 5 child threads and they all sucked up the CPU to the tune of more than 50% and I waited for their completion. But even after the completion of them and the CPU usage came down to under 10 %, the issue remained the same.

I had discussed with our OS sysadmin and he had involved the SAN guys and confirmed that, though the filesystem is active, it’s quite normal.

Can you help me what to look for, further ?

Thanks
Muthu

How to know semnum 118 is the user process traced? Any method to map a process with a set id and semnum? I am trying to map the semnum to oracle pid from v$process, but it can’t map correctly.

What is the duration of your AWR report? It may be for 1 hour or 30 minutes and so, aggregation might be hiding the details. Even ASH report of 15 minutes is not useful.

This might needs little bit more research using v$active_session_history.

(i) Try to find what sessions were waiting for using the script: http://www.orainternals.com/scripts_perf1.php – ash_top20.sql (or) use this query:

select * from (
select event,  inst_id,
   sum(decode(ash.session_state,'WAITING',1,0)) cnt_waiting
from  gv$active_session_history ash
where sample_time between sysdate - (&&start_min /( 60*24) ) and sysdate-(&&end_min/(60*24))
group by event,inst_id
order by 3 desc
) where rownum <=30
/

(ii) If data in v$active_session_history is still available, review LGWR activity using ash_lgwr.sql script.

(iii) Create ash report for very specific duration:

select output from table(dbms_workload_repository.ash_report_text( (select dbid from v$database),
                             1,
                             sysdate- &&start_min/(24*60),
                             sysdate-&&end_min,
                             0))
/

We had slow performance on the database and the requests from app server are queued up. I looked at AWR and ASH reports. I saw log file sync in one top top 5 wait event in AWR and it is listed as Blocking session in ASH report. THe blocking session with log file sync by LGWR was there the same time period of problem. I do not see high CPU usage or user commits during that time.

Top 5 Timed Events                                         Avg %Total
~~~~~~~~~~~~~~~~~~                                        wait   Call
Event                                 Waits    Time (s)   (ms)   Time Wait Class
------------------------------ ------------ ----------- ------ ------ ----------
CPU time                                          6,458          90.2
db file sequential read             318,605         644      2    9.0   User I/O
log file sync                        22,952          96      4    1.3     Commit
log file parallel write              23,407          80      3    1.1 System I/O
read by other session                15,400          53      3    0.7   User I/O

Statistic                                       Total
-------------------------------- --------------------
AVG_BUSY_TIME                                 236,562
AVG_IDLE_TIME                                 125,022
AVG_IOWAIT_TIME                                 7,748
AVG_SYS_TIME                                   44,124
AVG_USER_TIME                                 192,347
BUSY_TIME                                     946,610
IDLE_TIME                                     500,474
IOWAIT_TIME                                    31,357
SYS_TIME                                      176,865
USER_TIME                                     769,745
LOAD                                                3
OS_CPU_WAIT_TIME                              133,200
RSRC_MGR_CPU_WAIT_TIME                              0
VM_IN_BYTES                                   999,424
VM_OUT_BYTES                                        0
PHYSICAL_MEMORY_BYTES                  33,569,898,496
NUM_CPUS                                            4
          -------------------------------------------------------------

Statistic                                     Total     per Second     per Trans
-------------------------------- ------------------ -------------- -------------
total number of times SMON poste                 20            0.0           0.0
transaction rollbacks                            77            0.0           0.0
transaction tables consistent re                  0            0.0           0.0
transaction tables consistent re                  0            0.0           0.0
undo change vector size                 110,010,444       30,452.7       6,225.5
user I/O wait time                           69,807           19.3           4.0
user calls                               12,892,591        3,568.9         729.6
user commits                                 17,437            4.8           1.0
user rollbacks                                  234            0.1           0.0

But I see log file sync under Blocking session in ASH

Top Blocking Sessions        DB/Inst: COMMON5/common5  (May 10 15:15 to 15:30)
-> Blocking session activity percentages are calculated with respect to
      waits on enqueues, latches and "buffer busy" only
-> '% Activity' represents the load on the database caused by
      a particular blocking session
-> '# Samples Active' shows the number of ASH samples in which the
      blocking session was found active.
-> 'XIDs' shows the number of distinct transaction IDs sampled in ASH
      when the blocking session was found active.

   Blocking Sid % Activity Event Caused                      % Event
--------------- ---------- ------------------------------ ----------
User                 Program                          # Samples Active     XIDs
-------------------- ------------------------------ ------------------ --------
      771,    1       4.12 log file sync                        4.12
                                                           0/90 [  0%]        0

Did it cause the performance issue on the database?

What is that you are trying to tune? A specific business process?

I will try to help.
1. It looks like write performance to log files are not great. 12ms average for log file parallel write is not sufficient. Couple of things you can do to improve throughput to log files:
a) Try using direct and asynch I/O. I am not sure, whether underlying file system supports direct I/O or not. You might want to research that. I do see that filesystemio_options is set to setall, which means that Oracle will try to use asynch and direct I/O if the file system supports it.
Avg
%Time Total Wait wait Waits
Event Waits -outs Time (s) (ms) /txn
—————————- ————– —— ———– ——- ———
..
log file parallel write 90,505 .0 1,114 12 1.2
log file sequential read 82,154 .0 1,057 13 1.1
..
control file parallel write 48,544 .0 326 7 0.6
…
control file single write 8,255 .0 154 19 0.1
..
LGWR-LNS wait on channel 4,380 100.0 47 11 0.1
..

2. Same with control files too. control file single writes are taking an average of 19ms. Much of the control file writes are performed under the protection of CF enqueue and so, CF enqueue waits may be related to this issue.
3. Looks like LGWR ASYNC is used for log archive to standby. You might want to consider tuning LNS process also. Simply even increasing buffer size might be a good start.
4. _log_io_size is set to 256. I am not sure why. In most platforms that will result in triggering LGWR after 128KB of redo. That might trigger hyperactive LGWR. I don’t know your application well enough to recommend something here. Check the ratio between redo size and user commits + user rollbacks to see if that parameter value is warranted.
5. Last but not least, there are bugs in 10.2.0.3 with LGWR and log file sync events. In some cases, LGWR is not efficient enough when the workload is not so high. Read 34592.1 for further information. You might want to think about 10.2.0.4 upgrade. Of course, test your upgrade thoroughly

In summary, this goes to basic things. Improve write throughput to redo log files and control files. Eliminate bugs with better software etc.

It is also critical to review this again with an AWR report of 30 minutes or so.