What is a Latch?

Latches are “lightweight” mechanisms that are used to serialise access to memory structures. It is not possible to queue for a latch – a process either gets the latch or does not. Memory structures protected by latches include things like the buffer cache, the java pool and the library cache.

Behaviour on Latch Get Failure

When a process tries to obtain a latch, it may succeed or fail. On most hardware, the latch get operation is implemented as an atomic “test-and-set” instruction. Unlike failure to obtain a lock (which may result in a process sleeping or failing), failure to obtain a latch does not result in the process giving up the CPU. Exactly what happens when a latch is not obtained depends on the mode in which the latch was requested. If it was requested in “immediate” mode then the requesting process resumes exectuion immediately; it may then fail or carry out another action. If the latch was requested in “willing to wait” mode, then the process does not resume immediately if the latch request fails. Instead, the process starts the “spin and sleep” routine (as it’s named in Conquering).

Because of the speed at which latches are obtained and released, the latch-seeking process can remain on the CPU, and is said to be “spin-locked”. Spin-locked processes are consuming CPU, and my be preventing other processes from executing. If a spin-locked process still has not obtained the latch after _spin_count attempts, it will relinquish the CPU and sleep, allowing other processes to resume execution. The sleeping process will post the “latch free” wait.

When the sleeping process is resumed on the CPU, it will again request the latch. If it fails, it will spin again, and request again, until it has once more made _spin_count attempts to acquire the latch, at which point it will sleep again. The first sleep while waiting for a particular latch will be 10ms long, as will the second. Sleep lengths follow an exponential pattern, so the third and fourth are 20ms long, the fifth and sixth are 40ms long, and so on.

Parent and Child Latches

Some latches, such as the cache buffers chains latch protecting cached data blocks, have multiple children. In these cases the parent latch is not, as far as I can see, a useful object. It exists to aggregate statistics in dynamic performance views. Where child latches exist, they offer finer granularity of locking than would otherwise be available.
For instance, if there are N child latches protecting blocks in the buffer cache, then there can be up to N processes altering these blocks concurrently. Because each block “belongs” with a particular latch, it is still possible for running processes to block each other, but this becomes less likely as the granularity of the latching increases.

Oracle Wait Interface[4] is confusing on the exact behaviour of processes trying to obtain latches. On page 145, it says:

If a process fails to get one of the child latches in the first attempt, it will ask for the next, also with the no-wait mode. The willing-to-wait mode is used only on the last latch an when all no-wait requests against other child latches have failed.

That behaviour wouldn’t make sense if the latch requested was protecting a block in the buffer cache. After all, if any of the N child latches was sufficient, then there could be N processes concurrently changing the same block! It’s been suggested to me that the above is wrong in the general case, but may be true in some specific cases like the redo copy latch. This seems plausible to me, but I can’t verify it.

Latch-Related System Information

There are a number of latch-related dynamic performance views. These include:

• v$latch – lists all latches available in the system
• v$latchname – a straightforward translation from number to name
• v$latch_parent /v$latch_children – has nearly identical columns to v$latch, but de-aggregated
• v$latch_misses – details on where latch acquisition attempts have been missed (but see Ixora article)
• v$latchholder – contains a SID and PID for each held latch
• v$event_histogram – contains a histogram of waits, sorted by the event that caused them (including latches) and how many milliseconds they lasted
  

Dealing With Contention

Resolving makes the point that latch contention must be considered in the context of the whole system. If latch free waits are only a small proportion of the total waits, then there’s little to be gained by trying to eliminate them. Furthermore, it’s not the latch miss ratio that matters, but the total number of misses.

Conquering takes an understanding-led approach. When dealing with latch contention, it suggests three questions:

1. What is the use of the memory structure protected by this latch?
2. Why do processes want to access this latch so often?
3. Why do they hold the latch so long?

After understanding the cause of the problem, possible solutions should present themsleves.

_spin_count: Up or Down?

Resolving points out that _spin_count has been an undocumented parameter since version 8i (released in 1999). Its default value, 2000, has apparently not changed in that period. Since CPUs are much faster today than they were in 1999, the time taken by a spinning process to request a latch 2000 times is much lower today than it was when the default was established. So a process that cannot obtain a much latch spends a much greater proportion of its time on modern hardware sleeping than it would have on an older hardware. This is demonstrated with the results of an experiment, where the optimum value of _spin_count was found to be approximately 12000.

Before advising against raising _spin_count, Conquering notes that by the time the DBA sees a latch free wait, the posting process has already consumed CPU time waiting for the latch. The step in the author’s reasoning that I don’t quite get is the next:

“With this in mind, you may be able to see why it is very common to have latch contention when there is an operating system CPU bottleneck.”

This seems back-to-front to me. I can see that:

more latch contention => more CPU consumption by spinning processes => CPU bottleneck

What I can’t see is that a CPU bottleneck would lead to more latch contention, unless a lot of sleeping CPU-starved processes are holding latches required by processes on the CPU. In any case, I do agree that you probably won’t want to increase _spin_count when there is a CPU bottleneck: this certainly would increase CPU demand, but may not increase the rate at which useful work is done.

Reading Material

1. Resolving Oracle Latch Contention, Guy Harrison
2. Conquering Oracle Latch Contention, Craig Shallahamer
3. Oracle Data Dictionary Pocket Reference, D. C. Kreines. O’Reilly, 2003
4. Oracle Wait Interface: A Practical Guide to Performance Diagnostics & Tuning, R. Shee, K. Deshpande, K. Gopalakrishnan, McGraw-Hill Osborne, 2004

A package body needed to be recompiled on production. The large package body took about ten seconds to compile in QA. Obviously, nothing else went invalid or needed to be recompiled.

On production, the compile seemed to hang, as it was still going on after 30s. OEM showed a spike in sessions working or waiting, all of which were concurrency waits. Investigating further, these were all “library cache pin” waits. We killed the rebuild session from its terminal, and about two-thirds of the sessions waiting disappeared fairly quickly.

The remaining hundred or so sessions were all on the same node. Each was trying to execute one of two stored procedures from the package we had tried to recompile. I looked in GV$SQL to see if there was anything significantly different about the statements in each instances memory, but this was slightly out of my depth and nothing was immediately obvious to me.

The waiting sessions were made to go away by being manually killed. The available literature on how to “fix” library cache pins generally refers to x$ views to which I didn’t have access. Nor was DBA_BLOCKERS available.

What I would like to know is, what was the correct way of dealing with this? Was a lock or latch held on the package body by one of the sessions? If so, was it for a good reason, or was there a process failure of some sort? Where could I have looked for this information?

What’s a transaction table? I don’t recall them being mentioned before this point, and there’s no forward link from here either. It’s not in the glossary, and the index only points to a description of how they are recovered, not what they are.

I think this is wrong, or at least badly expressed. Consider part of the description of figure 2-5, “[A]fter the amount of used space falls below 40%, new rows can again be inserted into this block.” This makes me think that PCTUSED is a maximum, rather than a minimum. The idea is quite hard to express clearly in writing; I think the diagrams are much more useful here.

I always had trouble keeping PCTUSED and PCTFREE apart in my mind, until I read the best sentence in this section, “[U]pdates can reduce the available space of a block to less than PCTFREE, the space reserved for updates but not accesible to inserts.“

Why would Oracle want to write something about tables into every data block? Why not allow this information to remain in the data dictionary or the segment header? Is it here for efficiency purposes?

For this to work, I guess that each instance must keep its redo logs on the shared storage (no use of locally-attached storage, which I believe is possible in 11g). Each instance must also know which log files belong to which other instance.

As far as I can see, there also must be a way of distinguishing how much of the logged changes have already been stored “properly”, and how many remain to be written. This would be true in a RAC or a single-node environment. The first possibility that occurs is that the process of writing changes back to disk is also logged.

A table will not be regarded as key-preserved by Oracle in a view if analytic functions are invoked on it in a sub-select:

SQL> drop table analytic_test_a;

Table dropped

SQL> drop table analytic_test_b;

Table dropped

SQL> drop view analytic_test_view_1;

View dropped

SQL> drop view analytic_test_view_2;

View dropped

SQL> drop view analytic_test_view_1s;

View dropped

SQL> drop view analytic_test_view_2s;

View dropped

SQL> create table analytic_test_a as
2 select o.object_id, o.object_name, o.created
3 from user_objects o
4 where o.object_type in (‘TABLE’,'VIEW’);

Table created

SQL> alter table ANALYTIC_TEST_A
2 add constraint ANALYTIC_TEST_A_PK primary key (OBJECT_NAME);

Table altered

SQL> create table analytic_test_b as
2 select t.table_name, t.tablespace_name, t.pct_free
3 from all_tables t;

Table created

SQL> create view analytic_test_view_1 as
2 select *
3 from analytic_test_a a,
4 analytic_test_b b
5 where b.table_name = a.object_name;

View created

SQL> create view analytic_test_view_2 as
2 select a.*, b.*, min(OBJECT_ID) over (partition by TABLESPACE_NAME) min_id
3 from analytic_test_a a,
4 analytic_test_b b
5 where b.table_name = a.object_name;

View created

SQL> select *
2 from user_updatable_columns c
3 where lower(c.table_name) = ‘analytic_test_view_1′;

OWNER TABLE_NAME COLUMN_NAME UPDATABLE INSERTABLE DELETABLE
—————————— —————————— —————————— ——— ———- ———
BARTOJ_DEV ANALYTIC_TEST_VIEW_1 OBJECT_ID NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_1 OBJECT_NAME NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_1 CREATED NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_1 TABLE_NAME YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_1 TABLESPACE_NAME YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_1 PCT_FREE YES YES YES

6 rows selected

SQL> select *
2 from user_updatable_columns c
3 where lower(c.table_name) = ‘analytic_test_view_2′;

OWNER TABLE_NAME COLUMN_NAME UPDATABLE INSERTABLE DELETABLE
—————————— —————————— —————————— ——— ———- ———
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 OBJECT_ID NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 OBJECT_NAME NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 CREATED NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 TABLE_NAME YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 TABLESPACE_NAME YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 PCT_FREE YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_2 MIN_ID NO NO NO

7 rows selected

SQL> create view analytic_test_view_1s as
2 select *
3 from analytic_test_view_1;

View created

SQL> select *
2 from user_updatable_columns c
3 where lower(c.table_name) = ‘analytic_test_view_1s’;

OWNER TABLE_NAME COLUMN_NAME UPDATABLE INSERTABLE DELETABLE
—————————— —————————— —————————— ——— ———- ———
BARTOJ_DEV ANALYTIC_TEST_VIEW_1S OBJECT_ID NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_1S OBJECT_NAME NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_1S CREATED NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_1S TABLE_NAME YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_1S TABLESPACE_NAME YES YES YES
BARTOJ_DEV ANALYTIC_TEST_VIEW_1S PCT_FREE YES YES YES

6 rows selected

SQL> create view analytic_test_view_2s as
2 select *
3 from analytic_test_view_2;

View created

SQL> select *
2 from user_updatable_columns c
3 where lower(c.table_name) = ‘analytic_test_view_2s’;

OWNER TABLE_NAME COLUMN_NAME UPDATABLE INSERTABLE DELETABLE
—————————— —————————— —————————— ——— ———- ———
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S OBJECT_ID NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S OBJECT_NAME NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S CREATED NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S TABLE_NAME NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S TABLESPACE_NAME NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S PCT_FREE NO NO NO
BARTOJ_DEV ANALYTIC_TEST_VIEW_2S MIN_ID NO NO NO

7 rows selected

SQL>