EBS 12.2 ADOP Patching — Common Failures and How to Fix Them

Before You Start — Checking Session State

Before running adop, check whether the database is in a clean state from the previous patch cycle.

SELECT adop_session_id, prepare_status, apply_status, finalize_status,
       cutover_status, cleanup_status, abort_status, status, node_name
FROM ad_adop_sessions
ORDER BY adop_session_id DESC
FETCH FIRST 5 ROWS ONLY;

Status codes: N = not started, R = running, C = complete, F = failed, X = not applicable.

A healthy baseline before starting a new cycle: all statuses are C or X except abort_status which should be N. If the previous session shows F in any phase, investigate before proceeding.

-- Check for orphaned patch edition objects from a failed session
SELECT count(*) FROM dba_objects
WHERE edition_name IS NOT NULL AND status = 'INVALID';

Also confirm fs_clone ran after the last cycle. If it did not, the patch filesystem is out of sync and apply will behave unpredictably. There is no built-in query for this; check your patch log history or compare timestamps under $APPL_TOP/../fs1 and $APPL_TOP/../fs2.

Prepare Phase Failures

Edition Already Exists

The most common prepare failure is:

Error: Edition ORA$BASE already exists in an invalid state

This happens when a previous prepare phase failed midway and left an orphaned edition. Find and drop it:

-- Find editions in the database
SELECT edition_name, usable FROM dba_editions ORDER BY 1;
-- Drop the orphaned patch edition (not ORA$BASE)
DROP EDITION <patch_edition_name> CASCADE;

Then re-run prepare:

adop phase=prepare

Worker Timeouts During Prepare

Prepare spawns parallel workers that compile code in the patch edition. On large or busy databases, workers can time out:

Error: Worker process timed out after 3600 seconds

Increase the worker count and bump the timeout:

adop phase=prepare workers=8 prepare_timeout=7200

Finding Prepare Logs

Prepare log location: $APPL_TOP/../log/adop//. For database-side errors, query AD_ZD_LOGS:

SELECT log_sequence, log_type, log_text, log_date
FROM ad_zd_logs
WHERE adop_session_id = (SELECT MAX(adop_session_id) FROM ad_adop_sessions)
  AND log_type = 'E'
ORDER BY log_date DESC
FETCH FIRST 20 ROWS ONLY;

Apply Phase Hangs

When apply stops making progress, check worker states:

SELECT count(*), status
FROM ad_parallel_workers
WHERE adop_session_id = (SELECT MAX(adop_session_id) FROM ad_adop_sessions)
GROUP BY status;

A healthy apply shows most workers R (running) or C (complete). All workers in W (waiting) with none running means the job queue is blocked — possibly waiting for a lock held by another session.

Find which jobs are assigned but stuck:

SELECT worker_id, assigned_job_id, status, start_date
FROM ad_parallel_workers
WHERE adop_session_id = (SELECT MAX(adop_session_id) FROM ad_adop_sessions)
  AND status = 'W'
ORDER BY start_date;

Restarting Apply Without Losing Progress

ADOP apply is restartable. Kill the hung process and restart with restart=yes:

adop phase=apply restart=yes workers=8

ADOP skips jobs already marked complete and resumes from where it left off. You will not lose work already done.

apply-mode=downtime as a Last Resort

For a patch that repeatedly hangs in online mode, switch to downtime mode. This requires taking all application services down and disables the dual-edition mechanism:

adop phase=apply apply_mode=downtime workers=16

Use this only when online mode is blocking the patch indefinitely and you have a maintenance window.

Cutover Timeout

Cutover is the point of no return in a patch cycle. The default timeout is 30 minutes. On busy systems this can expire before all user sessions drain from the patch edition.

Error: Cutover did not complete within the timeout period

Increase the timeout:

adop phase=cutover cutover_timeout=120

Before running cutover, find what is holding up session drain:

SELECT sid, serial#, username, status, program, sql_id,
       ROUND((SYSDATE - last_call_et / 86400)  24  60, 0) minutes_active
FROM v$session
WHERE status = 'ACTIVE'
  AND username IS NOT NULL
  AND username != 'SYS'
ORDER BY last_call_et DESC
FETCH FIRST 20 ROWS ONLY;

Kill stuck sessions that are not real user work, then retry cutover.

fs_clone Failures

fs_clone synchronizes the run filesystem from the patch filesystem after a successful cutover. It is the most disk-intensive phase and the one most likely to fail due to space or NFS issues.

Disk Space Check

The patch filesystem needs roughly 1.3x the run filesystem size. Check all app tier nodes before starting:

df -h $APPL_TOP $INST_TOP $COMMON_TOP

NFS Mount Issues on Multi-Node

In multi-node configurations, the patch filesystem is typically NFS-shared. A stale mount causes cryptic copy errors. Check on all nodes:

df -h | grep nfs
mount | grep $APPL_TOP

Remount if needed, then retry:

adop phase=fs_clone restart=yes

Abort and Full Recovery

When to Abort

Abort when apply has failed irreversibly and restart is not recovering, or when the patch introduced a regression you cannot fix before cutover:

adop phase=abort

Abort rolls back the patch edition changes and returns the database to the pre-patch state. It does not affect the run filesystem — EBS continues operating normally.

Full Cleanup

After abort (or after a successful cycle), run cleanup to reclaim disk space used by the patch filesystem:

adop phase=cleanup cleanup_mode=full_cleanup

Verify cleanup completed — every status should be C or X:

SELECT adop_session_id, prepare_status, apply_status, finalize_status,
       cutover_status, cleanup_status, abort_status
FROM ad_adop_sessions
ORDER BY adop_session_id DESC
FETCH FIRST 3 ROWS ONLY;

Any status showing F after cleanup means cleanup itself failed. Check $APPL_TOP/../log/adop//cleanup/ for the cause.

Multi-Node Common Mistakes

Running adop on the Wrong Node

In multi-node EBS, adop must be initiated from the master (run) node — the node where the WebLogic AdminServer runs. Running from a secondary node causes inconsistent patch application across nodes.

Identify the master node:

SELECT node_name, server_address, support_cp, support_web, support_admin, support_db
FROM fnd_nodes
WHERE support_admin = 'Y';

SSH Trust Between Nodes

ADOP requires passwordless SSH between all application tier nodes as the applmgr OS user. Test before patching:

# From master node, as applmgr
ssh applmgr@<secondary_node_hostname> hostname

If this prompts for a password, fix SSH trust before starting a patch cycle. A missing SSH trust discovered mid-apply forces an abort.

Patch Filesystem Sync After Multi-Node Apply

After fs_clone, the patch filesystem on all nodes should mirror the run filesystem. Verify the patch timestamp matches across nodes:

ls -lt $APPL_TOP/../fs2/EBSapps/appl/ad/12.0.0/patch/115/ | head -5
Run on each node; timestamps should match within seconds