Scops
Docs Download

Oracle problems

RAC Lost Blocks: gc cr block lost, the Interconnect Checks, and the Fix

3 November 2026

On Oracle RAC a block that one instance needs and another holds crosses the interconnect, and gc cr block lost and gc current block lost are the waits of a block that did not arrive. Oracle's reference says the request resulted in a potentially lost block and is retried automatically, so nothing fails; everything just takes longer, the global cache waits climb to the top of the report, and the cause is a network that drops packets, a host whose buffers overflow, or a workload that sends far more blocks across than it should. The queries below say which, and the fix for each is different.

How many blocks were lost, and how long transfers take

SELECT inst_id, name, value
FROM   gv$sysstat
WHERE  name IN ('gc blocks lost', 'gc blocks corrupt',
                'gc cr blocks received', 'gc current blocks received',
                'gc cr block receive time',
                'gc current block receive time')
ORDER  BY inst_id, name;

The counters run since instance start, so two readings ten minutes apart give the rate. The receive times are in hundredths of a second, so 10 × receive time ÷ blocks received is the average transfer in milliseconds; on a healthy private network it is around a millisecond or less and steady, and the sign of trouble is a change, not a value. Lost blocks should be zero. The last minute, from the metrics:

SELECT inst_id, metric_name, ROUND(value, 2) AS value
FROM   gv$sysmetric
WHERE  group_id = 2
AND    metric_name LIKE 'Global Cache%'
ORDER  BY inst_id, metric_name;

The waits, and what each one means

SELECT inst_id, event, total_waits,
       ROUND(time_waited_micro/1000)                        AS ms,
       ROUND(time_waited_micro/NULLIF(total_waits,0)/1000,2) AS avg_ms
FROM   gv$system_event
WHERE  event IN ('gc cr block lost', 'gc current block lost',
                 'gc cr block busy', 'gc buffer busy acquire',
                 'gc buffer busy release', 'gc cr block 2-way',
                 'gc current block 2-way', 'gc cr block 3-way',
                 'gc current block 3-way')
ORDER  BY inst_id, time_waited_micro DESC;

The lost waits are the network. The 2-way and 3-way waits are normal transfers, and their average time is the interconnect's speed. The busy waits are contention, not loss: Oracle's reference describes gc buffer busy acquire as a session that cannot pin the buffer because another session is reading it from the cache of another instance, and gc buffer busy release as another instance taking the buffer out of this cache so it can pin it. Lost blocks with slow transfers on every instance is a network problem; zero lost blocks with busy waits on a few objects is a hot-block problem, and the fix is in the application.

The network the cluster is using

SELECT inst_id, name, ip_address, is_public, source
FROM   gv$cluster_interconnects
ORDER  BY inst_id;

Oracle's reference says it expects cluster traffic to be run on private interconnects only, and IS_PUBLIC at YES is the misconfiguration the RAC guide names first: message and block traffic on the public network. SOURCE says where the interface came from, the cluster registry, the operating system, or the CLUSTER_INTERCONNECTS parameter. On most platforms the blocks travel as UDP datagrams, and a lost block is a dropped datagram, so the rest of the evidence is on the host: the receive errors and buffer errors of UDP in the kernel's statistics, and the dropped and overrun counters of the interconnect interface and the switch port behind it.

The objects that cross the interconnect most

SELECT owner, object_name, statistic_name, value
FROM   v$segment_statistics
WHERE  statistic_name IN ('gc buffer busy', 'gc cr blocks received',
                          'gc current blocks received')
AND    value > 0
ORDER  BY value DESC
FETCH  FIRST 20 ROWS ONLY;

Run on each instance, this is the list of tables and indexes whose blocks are shared across the cluster. The RAC guide puts the root cause of most Cache Fusion problems in one sentence: either interconnect or load issues, or SQL execution against a large shared working set. The segments at the top of this list are that working set.

The fix, in order

  1. Decide whether it is the network or the load. Lost blocks above zero, receive times up on every instance, and UDP or interface errors on the hosts: the network, and nothing in the database fixes it. Lost blocks at zero, busy waits on a few segments: the load, and the network is fine.
  2. Put the interconnect on a private network, and nothing else on it. IS_PUBLIC must be NO on every instance. Backups, replication and file copies over the same interface are the usual uninvited traffic.
  3. Stop the drops. The RAC guide says that once the interconnect is operative you cannot significantly influence its performance, but you can influence the protocol's efficiency by adjusting the IPC buffer sizes: the kernel's socket receive buffers at least at the values the installation guide requires, a NIC and switch port that agree on speed, duplex and MTU end to end, and a driver and firmware that are current. An interface that drops packets under load is replaced, not tuned.
  4. Shrink the shared working set. Partition the work so that each instance works on its own data, with services that send an application to one instance; fix the statement that scans a table the other instance is updating; give a sequence a large CACHE and NOORDER; and spread an index whose rightmost block every instance inserts into, with a reverse key or a hash partition. The busy waits fall with the traffic.
  5. Do not chase normal numbers. The RAC guide says most global cache wait events that show a high total time in the reports are normal and may present themselves as the top time consumers without indicating a problem. The comparison that matters is against the same hour last week, which is what Radar Engine does.
  6. Read the cluster alert log. A node evicted, a lost global cache block message or an interconnect failover in the logs is the event behind the numbers, and it has a time.

See it coming

Radar Engine, the detection layer of Scops, watches the lost-block counters and the transfer times of every RAC database live, against what is usual for that database, and reads the alert logs for the cluster messages, so the drop that starts at ten past two is a finding at twenty past, with the instance and the numbers, instead of a slow Monday:

DetectorClockWhen it fires
RAC interconnect troubleLive, RAC only. Lost blocks, or transfer times three times the usual, for 10 minutes.Warning. Both: Critical.
Cluster waits surgingLive. Twice the hour before, well above its usual spread.Watch. With a second sign, such as 5 blocked sessions or commits three times slower: Warning. Above max(4, 4 × usual) active sessions, with an effect: Critical.
Logs: a node evicted, lost global cache blocksSerious errors and patterns in the alert logs, such as an evicted node or lost global cache blocks.A serious error for the first time in 24 hours: Critical.

The finding names the database, the instance, the blocks lost, the transfer time against the usual one, and the segments that crossed the interconnect most in those minutes. Scops samples every session every 5 seconds on every instance, so the Cluster-class waits are in the Activity chart by session and statement, with nothing installed on the cluster and no Diagnostics Pack.

Questions

What does gc cr block lost mean?
A consistent-read block requested from another instance did not arrive over the interconnect. Oracle retries the request automatically, so the wait is the time lost, and the statistic gc blocks lost counts how often it happened. It is a network or host buffer problem, not a database one.

Is a high global cache wait time always a problem?
No. Oracle's RAC guide says most global cache wait events with a high total time are normal and may be the top time consumers without indicating a problem. Lost blocks, transfer times far above the usual, and busy waits on a few hot objects are the ones that matter.

How do I check that the interconnect is private?
V$CLUSTER_INTERCONNECTS lists the interfaces in use with IS_PUBLIC; it must be NO on every instance. Oracle expects cluster traffic to be run on private interconnects only.

What is the difference between gc block lost and gc buffer busy?
Lost is a transfer that failed on the network. Busy is a buffer that another session or instance holds while this session needs it, which is contention on a hot block, fixed in the application and the schema rather than in the network.

Related problems

Sources