Cheers
Riyaj

Recently Oracle introduced a new “Best Practice” which is turning off NUMA optimization. May be in future patchsets we might see these default values:

_enable_NUMA_optimization=false
_db_block_numa=1

Thanks for reading my blog and for your kind words. Thank you for pointing out that bug. Our problem (Early September) actually predates that bug date ( 23-OCT-2008) and I didn’t see that bug report earlier either. I am printing a summary of the bug below.

Bug summary: In 10.2.0.4 NUMA features were enabled by default and this can cause issues in platforms/hardware without NUMA capability. Workaround is to disable NUMA.

In this specific case though, hardware had NUMA capability as shown in lgrpinfo output. Just that we were hitting a side effect of NUMA capability.

Cheers
Riyaj

Ryan Matteson, An excellent solaris system administrator [ with our client] said: ” That would make sense, since each processor has a built-in memory controller that has direct access to one or more banks of memory. CPUs are: Dual-Core AMD Opteron(tm) Processor 8220 CPU ”

Cheers
Riyaj

Hi Tanel
Thanks for the idea about LOGOFF trigger. That’s a cool idea to explore and use global context variables to keep track of rate of disconnects. Yes, -c flag might have been more accurate in that truss output.

Hi Christian
Let me check on hardware configuration. It’s a Solaris zone and I don’t have much access in client environment.

Hi Freek
Yes, that was a mistake..Should be 16 CPUs and will correct it. I think, they are dual core CPUs.

Cheers
Riyaj

very good article.

I just have 1 question: you say the system has 12 cpu’s, but isn’t the mpstat output showing 14 cpu’s?

regards,

Freek

I wonder if you are running on a Solaris/AMD Opteron: Oracle on Solaris considers each Opteron processor as a unit on par with a SPARC domain. An eight 2-cores system with a 32GB RAM will hold eight Oracle ccNUMA nodes. Oracle’s SGA will be split in small 2-4 GB chunks depending on the SGA allocation. Small chunks relative to the allocated SGA lower the probability for a thread to access its “local” memory. This can make the whole scheme irrelevant and possibly creates overhead and bugs (such as bug 5173642 references in Metalink) for non-partitioned servers.

_enable_NUMA_optimization was available (default value: FALSE) until Oracle 10.1.0 and was made obsolete in 10.2.0.
_db_block_numa is since Oracle 10.2.0.2 the number of Numa “nodes”. The “node” is the ccNUMA awareness unit.

Christian

Another option is to use a BEFORE LOGOFF trigger to throttle number of logouts when certain conditions are met. You can use global context variables to keep track of logoff rates instance-wide..

Btw, the truss -d option in Solaris just gives you timestamps when system calls completed, not the time spent in them. The truss -c should give you time spent *inside* syscalls.