EBS 12.2 ADOP Patching — Common Failures and How to Fix Them

EBS 12.2 introduced online patching via ADOP (AD Online Patching) — patches apply to a patch edition while users stay on the run edition, then a brief cutover switches them. In theory, zero downtime. In practice, ADOP failures are some of the most stressful incidents an EBS DBA faces. This article covers the most common failures at each phase and exactly how to fix them.

ADOP Phase Overview

prepare → apply → finalize → cutover → cleanup

Each phase must complete before the next starts. A failure in any phase leaves the patch cycle in a partially-applied state that must be resolved before normal operations resume.

-- Check current ADOP session status
SELECT adop_session_id, status, prepare_status, apply_status,
       finalize_status, cutover_status, cleanup_status,
       TO_CHAR(start_date,'DD-MON-YY HH24:MI') start_date
FROM ad_adop_sessions
ORDER BY adop_session_id DESC
FETCH FIRST 5 ROWS ONLY;

Phase 1: Prepare Failures

The prepare phase creates the patch edition and synchronizes the patch file system. It's the most complex phase and has the most failure modes.

Edition creation failure

ERROR: ORA-38807: Implementation restriction: editions not supported

Editions must be enabled on the database. Check:

SELECT property_value
FROM database_properties
WHERE property_name = 'EDITIONS_ENABLED';

If FALSE, enable it:

ALTER DATABASE ENABLE EDITIONS;

Patch file system sync failure

ERROR: Could not copy file to patch file system
Patch file system is not accessible

The prepare phase copies the run file system to the patch file system. If fs_clone fails:

# Check available space on both file systems
df -h $RUN_BASE $PATCH_BASE

# Check NFS mounts if using shared storage
mount | grep -i nfs

# Re-run prepare with cleanup flag
adop phase=prepare cleanup_mode=full

Cutover pending from previous session

ERROR: An ADOP session is already in progress
Previous session must be abandoned or completed

-- Check for stuck sessions
SELECT adop_session_id, status
FROM ad_adop_sessions
WHERE status NOT IN ('C', 'F');  -- C=complete, F=failed

-- Abandon a stuck prepare (if no patches applied yet)
adop phase=abort

Phase 2: Apply Failures

Worker failures

The apply phase runs patch workers in parallel. A worker failure doesn't immediately abort — other workers continue, then ADOP reports how many failed.

# Check worker log files
ls -lt $NE_BASE/EBSapps/log/adop/<session_id>/apply*/
tail -100 $NE_BASE/EBSapps/log/adop/<session_id>/apply*/<context>/log/adworker*.log

# Re-run apply with restart (picks up where it left off)
adop phase=apply restart=yes

ORA-00060 deadlock during apply

ERROR: ORA-00060: deadlock detected while waiting for resource

Object locking during apply is common when users are actively using the system. The patch worker retries automatically, but if it exceeds the retry limit:

-- Find blocking sessions
SELECT s.sid, s.serial#, s.username, s.status,
       s.event, s.blocking_session
FROM v$session s
WHERE s.blocking_session IS NOT NULL;

-- Kill the blocker if it's safe to do so
ALTER SYSTEM KILL SESSION 'sid,serial#' IMMEDIATE;

Missing prerequisite patch

ERROR: Prerequisite patch <patch_number> is not applied

# Check which prereqs are missing
adop phase=apply patches=<patch_number> prereq_only=yes

# Apply the prerequisite first, then the main patch
adop phase=apply patches=<prereq_number>
adop phase=apply patches=<main_patch_number>

Invalid objects blocking apply

ERROR: Object <object_name> is invalid in APPS schema

-- Recompile before retrying apply
EXEC UTL_RECOMP.RECOMP_PARALLEL(4, 'APPS');

-- Check what's still invalid
SELECT object_name, object_type, last_ddl_time
FROM dba_objects
WHERE owner = 'APPS'
  AND status = 'INVALID'
ORDER BY last_ddl_time DESC;

Phase 3: Finalize Failures

Finalize compiles invalid objects in the patch edition and prepares for cutover.

Compilation failures

ERROR: PL/SQL compilation errors in patch edition

# Check finalize log
tail -200 $NE_BASE/EBSapps/log/adop/<session_id>/finalize*/log/adfinalize*.log

# Recompile manually in the patch edition
sqlplus apps/<pwd>
ALTER SESSION SET EDITION = <patch_edition_name>;
EXEC UTL_RECOMP.RECOMP_PARALLEL(4);

-- Find the patch edition name
SELECT edition_name, usable
FROM dba_editions
ORDER BY create_time DESC;

Seed data errors

ERROR: Error loading seed data
FNDLOAD failed with exit code 1

FNDLOAD errors during finalize are usually data conflicts — the patch is trying to load a menu or concurrent program definition that conflicts with a customization.

# Check the FNDLOAD log
grep -i "error\|warning\|failed" $NE_BASE/EBSapps/log/adop/<session_id>/finalize*/log/FNDLOAD*.log

# Re-run finalize after resolving the conflict
adop phase=finalize

Phase 4: Cutover Failures

Cutover is the brief downtime window — it switches the run and patch file systems and moves users to the new edition. Failures here are the most critical because the system is mid-switch.

Cutover timeout

ERROR: Cutover timed out waiting for users to disconnect

-- Check active sessions during cutover
SELECT COUNT(*), machine, program
FROM v$session
WHERE type = 'USER'
  AND status = 'ACTIVE'
GROUP BY machine, program
ORDER BY COUNT(*) DESC;

-- Force disconnect if needed (after warning users)
SELECT 'ALTER SYSTEM KILL SESSION ''' || sid || ',' || serial# || ''' IMMEDIATE;'
FROM v$session
WHERE type = 'USER'
  AND username NOT IN ('SYS','SYSTEM','APPS');

File system switch failure

ERROR: Could not switch file systems
Run edition is still active

# Check which edition is currently active
cat $APPL_TOP/admin/<context_name>/adovars.env | grep FS_CLONE_STATUS

# If the switch failed mid-way, check the cutover log
tail -500 $NE_BASE/EBSapps/log/adop/<session_id>/cutover*/log/adcutover*.log

# Retry cutover
adop phase=cutover

WebLogic not restarting after cutover

After cutover, WebLogic must restart against the new run file system. If it fails to start:

# Check WLS startup log
tail -100 $EBS_DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log

# Common fix: AutoConfig wasn't run before cutover
# Run AutoConfig on both tiers, then retry
cd $ADMIN_SCRIPTS_HOME
./adautocfg.sh apps/<pwd>

# Then restart WebLogic
./adadminsrvctl.sh start apps/<pwd>

Phase 5: Cleanup Failures

Cleanup drops the old patch edition and removes obsolete objects. Failures here are less critical — the system is running on the new edition — but must be resolved before the next patch cycle.

Edition drop failure

ERROR: Cannot drop edition — objects still reference it

-- Find what's referencing the old edition
SELECT object_name, object_type, edition_name
FROM dba_objects_ae
WHERE edition_name = '<old_edition_name>'
  AND object_type NOT IN ('NON-EXISTENT');

-- Force cleanup
adop phase=cleanup cleanup_mode=full

ADOP Status Reference

-- Full status of current and recent sessions
SELECT s.adop_session_id,
       s.status,
       s.prepare_status,
       s.apply_status,
       s.finalize_status,
       s.cutover_status,
       s.cleanup_status,
       TO_CHAR(s.start_date,'DD-MON-YY HH24:MI') started,
       TO_CHAR(s.end_date,'DD-MON-YY HH24:MI') ended
FROM ad_adop_sessions s
ORDER BY s.adop_session_id DESC;

Status codes: N=not started, R=running, C=complete, F=failed, X=abandoned

Emergency: Abandoning a Patch Cycle

If a patch cycle must be abandoned (incorrect patch applied, critical production issue):

# Abandon current session and restore run file system
adop phase=abort

# This returns you to the pre-patch state
# Any changes applied to the patch edition are discarded

Note: abort is only available if cutover has not completed. After cutover, the patch is live — you must apply a rollback patch or use FNDLOAD to revert data changes.

Preventing ADOP Failures

Most ADOP failures are preventable:

1. Always run in a test environment first — apply the patch to a clone before production
2. Check ETCC before every patch cycle — missing FMW/WLS patches cause apply failures
3. Verify disk space — prepare needs ~2x the run file system size
4. Schedule during low-usage windows — fewer active sessions means fewer locking conflicts
5. Keep APPS schema clean — recompile invalid objects before starting any patch cycle

TuneVault's ADOP Status tab shows the current patch cycle phase, session status, and any active ADOP errors directly in the browser — no SSH required. The Patching & ADOP tab under EBS Ops also lets you run all 9 ADOP phases from the UI.