EBS Upgrade / Technology Stack Upgrade Triage Runbook

1. EBS 12.1 → 12.2 Upgrade Path Overview

EBS 12.1→12.2 is a major upgrade involving both the application technology stack (WebLogic replaces Oracle AS, Forms 11g, ADOP replaces adpatch) and the patching model (online patching with dual filesystem). It is not a patch — it is a full upgrade requiring a maintenance window of typically 8–24 hours.

Phase	What happens	Duration (typical)
Pre-upgrade steps	AD/TXK delta, DB upgrade to 12c/19c, preinstall patches	2–4 hrs
Upgrade start	Run u12201061.drv (12.2 upgrade driver) — schema changes, data migration	4–12 hrs
Technology stack install	Install WebLogic, Forms 11g, OHS; configure domains	1–2 hrs
AutoConfig	First AutoConfig under new tech stack	15–30 min
ADOP cutover model	Switch to dual-filesystem (run/patch editions)	30 min
Post-upgrade patches	Mandatory post-upgrade patches per MOS 1594222.1	1–3 hrs

2. ADOP Phase Failures

ADOP (AD Online Patching) replaced adpatch in EBS 12.2. It operates in five sequential phases. A failure in any phase requires targeted recovery — not a full restart.

-- Check the status of the current ADOP cycle
SELECT
  patch_run_id,
  phase,
  status,
  start_date,
  end_date,
  workers,
  log_file
FROM ad_adop_session_patches
ORDER BY patch_run_id DESC
FETCH FIRST 10 ROWS ONLY;

-- Check current ADOP session
SELECT
  adop_session_id,
  node_name,
  status,
  start_date,
  end_date
FROM ad_adop_sessions
ORDER BY start_date DESC
FETCH FIRST 5 ROWS ONLY;

prepare phase failure

-- prepare creates the patch edition and synchronizes filesystems
-- Log: $APPL_TOP/../log/adop_<session>/prepare_<node>.log

-- Common causes: stale patch edition from prior failed cycle
-- Fix: cleanup first, then re-prepare
adop phase=cleanup
adop phase=prepare

apply phase failure

-- apply runs patch workers; a failed worker does NOT stop the whole apply
-- Find failed workers in log directory
-- ls $APPL_TOP/../log/adop_<session>/apply_<node>/

-- Restart a failed apply from where it left off
adop phase=apply patches=<patch_number> restart=yes

-- If a specific worker keeps failing, check its log
-- cat $APPL_TOP/../log/adop_<session>/apply_<node>/adworker_<N>.log

-- Check for worker errors in DB
SELECT
  w.worker_id,
  w.status,
  w.log_file,
  j.unit_id,
  j.action_code,
  j.message
FROM ad_parallel_workers w
JOIN ad_parallel_jobs j ON w.worker_id = j.worker_id
WHERE j.status IN ('F', 'R')
ORDER BY j.last_update_date DESC;

finalize phase failure

-- finalize compiles invalid objects and runs post-patch tasks
-- Log: $APPL_TOP/../log/adop_<session>/finalize_<node>.log

-- Re-run finalize (safe to re-run)
adop phase=finalize

-- If finalize fails on a specific object compilation, find it in DB
SELECT owner, object_name, object_type, status
FROM dba_objects
WHERE status = 'INVALID'
  AND owner NOT IN ('SYS', 'SYSTEM')
ORDER BY owner, object_type, object_name;

cutover phase failure

-- cutover switches the run edition to the patched edition
-- It requires all users to be out of the system (check active sessions)

-- Check active DB sessions before cutover
SELECT username, COUNT(*) AS sessions
FROM v$session
WHERE username IS NOT NULL
  AND username NOT IN ('SYS', 'SYSTEM', 'DBSNMP')
GROUP BY username
ORDER BY sessions DESC;

-- Run cutover
adop phase=cutover

-- If cutover fails midway: check the log, restart is NOT supported
-- You must abort and cleanup, then restart the entire patch cycle
-- See Section 3 for abort recovery

cleanup phase failure

-- cleanup drops the old patch edition objects and removes stale filesystem content
-- Safe to re-run if it fails
adop phase=cleanup

-- If cleanup hangs on edition drop (large schemas take time)
-- Check for blocking locks
SELECT
  s.sid, s.serial#, s.username, s.status,
  l.type, l.request, l.block
FROM v$session s
JOIN v$lock l ON s.sid = l.sid
WHERE l.block = 1;

-- Force cleanup if it's been stuck > 2 hours
adop phase=cleanup mode=full

3. Failed Cutover Recovery

-- Step 1: Abort the failed adop session
adop phase=abort

-- Step 2: If abort also fails (stuck sessions), kill them in DB
SELECT sid, serial#, username, status, schemaname
FROM v$session
WHERE schemaname LIKE '%_PATCH%'
   OR schemaname LIKE '%_NEXT%';

ALTER SYSTEM KILL SESSION '&sid,&serial' IMMEDIATE;

-- Step 3: Run cleanup to remove the failed patch edition
adop phase=cleanup

-- Step 4: Verify no patch editions remain
SELECT edition_name, usable
FROM dba_editions
ORDER BY edition_name;
-- Should show only one usable edition (the run/base edition)

-- Step 5: Verify filesystem is clean
-- ls $APPL_TOP_NE  (should be minimal/empty after cleanup)

-- Step 6: Now safe to start a new patch cycle
adop phase=prepare

Cleaning stuck ADOP sessions in the database

-- ADOP session state in DB (check ad_adop_sessions)
SELECT
  adop_session_id,
  node_name,
  status,
  start_date
FROM ad_adop_sessions
WHERE status NOT IN ('C', 'F')
ORDER BY start_date DESC;

-- Mark stuck session as aborted (last resort if adop phase=abort fails)
UPDATE ad_adop_sessions
SET status = 'F'  -- F = Failed/Aborted
WHERE status = 'R'  -- R = Running
  AND adop_session_id = &session_id;
COMMIT;

4. Concurrent Manager Not Starting Post-Upgrade

-- Check CM status
SELECT
  concurrent_queue_name,
  manager_type,
  running_processes,
  max_processes,
  target_node,
  last_refresh_date
FROM fnd_concurrent_queues_vl
WHERE enabled_flag = 'Y'
ORDER BY concurrent_queue_name;

ICM (Internal Concurrent Manager) won't start

-- Check APPS password is correct in FND_USER
SELECT user_name, encrypted_user_password, last_logon_date
FROM fnd_user
WHERE user_name = 'APPS';

-- Reset APPS password if needed (requires SYSADMIN login)
-- Navigate to: Security → User → Define → APPS user → change password
-- OR use FNDCPASS utility:
FNDCPASS apps/apps_password 0 Y system/manager_password ALLORACLE APPS new_password

-- Verify $FND_TOP/bin/FNDLIBR is executable
-- ls -la $FND_TOP/bin/FNDLIBR

-- Check service startup log
-- cat $APPLCSF/$APPLLOG/STARTCM.log (11.5) or check admin/log/

-- Force CM restart
-- As applmgr: adcmctl.sh start apps/<password>

CM starts but workers fail immediately

-- Check for environment issues — AutoConfig may not have run after upgrade
-- Verify correct ORACLE_HOME and ORACLE_SID are set in the Apps tier env
-- source $APPL_TOP/envshell.env (or APPS<SID>_<host>.env)

-- Check for stale NLS_CHARACTERSET mismatches
SELECT parameter, value
FROM nls_database_parameters
WHERE parameter IN ('NLS_CHARACTERSET', 'NLS_NCHAR_CHARACTERSET');

-- Verify FND_INSTALL_PROCESSES table is clean
SELECT * FROM fnd_install_processes
WHERE status IN ('I', 'R')
ORDER BY last_updated_date DESC;

5. AutoConfig Failures During Technology Stack Patching

-- AutoConfig log location
-- $APPL_TOP/admin/<SID>/log/<date>/adconfig.txt
-- For apps tier: $INST_TOP/admin/log/<date>/adconfig.txt

-- Run AutoConfig manually (as applmgr)
perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

-- If AutoConfig fails on database tier
perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE run=DB

Common AutoConfig failure: context file out of date

-- After major upgrade, context file may have missing or wrong values
-- Run adbldxml.pl to rebuild context file
perl $AD_TOP/bin/adbldxml.pl \
  appspass=apps_password \
  contextfile=$CONTEXT_FILE \
  xmlfile=/path/to/old_context.xml

-- Validate context file
perl $AD_TOP/bin/adchkutl.pl contextfile=$CONTEXT_FILE

AutoConfig fails on WebLogic domain config

-- Check MW_HOME and WL_HOME are set correctly in context
-- grep 'MW_HOME\|WL_HOME' $CONTEXT_FILE

-- WebLogic domain config backup before AutoConfig
-- cp -r $INST_TOP/ora/10.1.3/ $INST_TOP/ora/10.1.3.bak/

-- If AutoConfig corrupted domain: restore from backup and re-run
-- rm -rf $INST_TOP/ora/10.1.3/
-- cp -r $INST_TOP/ora/10.1.3.bak/ $INST_TOP/ora/10.1.3/
perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

6. WebLogic / Forms / OHS Upgrade Issues

AdminServer won't start after upgrade

-- Check AdminServer log
-- tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer.log
-- tail -100 $DOMAIN_HOME/servers/AdminServer/logs/AdminServer-diagnostic.log

-- Common cause: port conflict (AdminServer default: 7001)
-- lsof -i :7001 (or netstat -tlnp | grep 7001)

-- Common cause: boot.properties missing or corrupted
-- Recreate boot.properties:
-- mkdir -p $DOMAIN_HOME/servers/AdminServer/security/
-- cat > $DOMAIN_HOME/servers/AdminServer/security/boot.properties <<EOF
-- username=weblogic
-- password=<plain_text_password>
-- EOF
-- WLS will encrypt it on next start

-- Start AdminServer manually
-- $DOMAIN_HOME/bin/startWebLogic.sh > /tmp/admin.log 2>&1 &

NodeManager failures

-- NodeManager manages Managed Server lifecycle
-- Log: $MW_HOME/oracle_common/common/nodemanager/nodemanager.log

-- Check if NodeManager is running
-- ps -ef | grep NodeManager

-- Restart NodeManager
-- cd $MW_HOME/oracle_common/common/bin
-- ./startNodeManager.sh > /tmp/nm.log 2>&1 &

-- If NodeManager refuses to enroll a machine
-- Check nodemanager.properties: ListenPort, ListenAddress
-- cat $WL_HOME/server/lib/nodemanager.properties | grep Listen

OHS (Oracle HTTP Server) won't start

-- OHS log location
-- $INST_TOP/ora/10.1.3/Apache/Apache/logs/error_log

-- Check port conflicts (default OHS: 8000 / 443)
-- netstat -tlnp | grep -E '8000|443|7777'

-- Restart OHS
$ADMIN_SCRIPTS_HOME/adopmnctl.sh start
$ADMIN_SCRIPTS_HOME/adohactl.sh start

-- If OHS config is corrupt after AutoConfig: check httpd.conf
-- diff $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf \
--      $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf.bkXXXX

7. EBS R12.2 Online Patching Troubleshooting

EBS 12.2 uses an online patching model (Edition-Based Redefinition) — patches apply against the patch edition while users work in the run edition. The two editions share the same data but have separate code objects.

-- Check current run and patch editions
SELECT
  property_name,
  property_value
FROM fnd_product_installations
WHERE property_name LIKE '%EDITION%';

-- Check edition objects are in sync
SELECT
  object_name, object_type,
  edition_name,
  status
FROM dba_edition_obj_audit_trail
WHERE status = 'INVALID'
ORDER BY object_name;

Edition drift — run edition has invalid objects

-- Check run edition objects for invalids
SELECT owner, object_name, object_type, status
FROM dba_objects
WHERE status = 'INVALID'
  AND owner IN ('APPS', 'APPLSYS')
ORDER BY object_type, object_name;

-- Force recompile in APPS schema
EXEC DBMS_UTILITY.COMPILE_SCHEMA(schema=>'APPS', compile_all=>FALSE);
EXEC DBMS_UTILITY.COMPILE_SCHEMA(schema=>'APPLSYS', compile_all=>FALSE);

8. AD Utilities — Common Failure Modes

adadmin — Maintain Applications Database Objects

-- adadmin log: $APPL_TOP/admin/<SID>/log/adadmin<N>.log

-- Run adadmin to recompile invalid objects (menu option)
-- adadmin -> Maintain Applications Database Objects ->
--           Compile Invalid Objects in the APPS Schema

-- adadmin fails with ORA-01017 (invalid username/password)
-- Usually APPS schema password needs reset
-- alter user apps identified by <password>;

-- Check adadmin worker log for specific errors
-- ls $APPL_TOP/admin/<SID>/log/adwork*.log | xargs grep "Error\|ORA-" | tail -20

adpatch — applying patches on 12.1 or older 12.2

-- adpatch log: $APPL_TOP/admin/<SID>/log/adpatch.log

-- Check if previous adpatch session needs to be completed
SELECT
  patch_run_id,
  phase,
  status,
  start_date
FROM ad_patch_runs
ORDER BY start_date DESC
FETCH FIRST 5 ROWS ONLY;

-- If adpatch stopped mid-run, restart it
-- adpatch will detect the incomplete run and offer to restart
adpatch

-- Check for failed workers in a previous run
SELECT
  w.worker_id, w.status,
  j.action_code, j.message
FROM ad_parallel_workers w
JOIN ad_parallel_jobs j ON w.worker_id = j.worker_id
WHERE j.status = 'F'
ORDER BY j.last_update_date DESC;

9. Post-Upgrade Validation

-- 1. Verify FND_INSTALL_PROCESSES shows successful upgrade steps
SELECT
  install_group_num,
  status,
  last_update_date,
  product_short_name,
  creation_date
FROM fnd_install_processes
WHERE status NOT IN ('S')
ORDER BY last_update_date DESC
FETCH FIRST 20 ROWS ONLY;

-- 2. Verify AD_BUGS for applied patches
SELECT
  bug_number,
  bug_name,
  last_update_date
FROM ad_bugs
WHERE bug_number IN (
  '12345678'  -- replace with your critical patch numbers
)
ORDER BY last_update_date;

-- 3. Verify key product versions post-upgrade
SELECT
  product_short_name,
  release_name,
  patch_level,
  install_status
FROM fnd_product_installations
WHERE install_status != 'N'
ORDER BY product_short_name;

-- 4. Check AD patch history for missing drivers
SELECT
  patch_name,
  patch_type,
  creation_date
FROM ad_applied_patches
ORDER BY creation_date DESC
FETCH FIRST 20 ROWS ONLY;

-- 5. Verify AutoConfig ran with no errors after upgrade
-- grep "ERROR\|FAILED" $INST_TOP/admin/log/*/adconfig.txt | tail -20

10. Apps Tier Cloning Issues Post-Upgrade

-- After upgrading, cloning requires updated clone scripts

-- Step 1: On source (post-upgrade) system, run adpreclone
-- perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier
-- perl $ADMIN_SCRIPTS_HOME/adpreclone.pl dbTier

-- Step 2: Copy filesystem to target

-- Step 3: On target, run adcfgclone
-- perl $CLONE_TOP/bin/adcfgclone.pl appsTier /path/to/context_template.xml

-- Common post-upgrade clone failure: outdated $CLONE_TOP scripts
-- The clone scripts in $AD_TOP/clone must be updated to EBS 12.2 versions
-- Verify: grep 'version' $AD_TOP/clone/bin/adcfgclone.pl | head -3

-- If adcfgclone fails on context file: rebuild context first
perl $AD_TOP/bin/adbldxml.pl \
  contextfile=$NEW_CONTEXT_FILE \
  templatefile=$AD_TOP/admin/template/<template>.xml

11. Common Errors

AD-A-01022: Unable to determine the version of file

-- adop/adpatch can't determine a file's version
-- Root cause: file header missing, binary file, or encoding issue

-- Identify the problem file from adpatch/adop log
-- grep "AD-A-01022" $APPL_TOP/admin/<SID>/log/adpatch.log

-- Option 1: Force-apply by skipping version check
-- adop phase=apply patches=<N> options=nofileversioncheck
-- adpatch: use 'options=nofileversioncheck' flag

-- Option 2: Manually update the file version in FND tables
SELECT * FROM fnd_application_files
WHERE file_name = '&filename';

-- Option 3: Copy clean file from another server running same patch level

ADZDCLN errors during cleanup

-- ADZDCLN is the PL/SQL cleanup package for edition objects
-- Errors usually mean dependent objects are blocking edition drop

-- Check for objects dependent on the patch edition
SELECT
  owner, name, type,
  referenced_owner, referenced_name, referenced_type
FROM dba_dependencies
WHERE referenced_owner LIKE '%_NEXT'
   OR referenced_owner LIKE '%_PATCH'
FETCH FIRST 20 ROWS ONLY;

-- If cleanup keeps failing on the same object, check MOS for a bug
-- Search: "ADZDCLN [object_name] 12.2" on support.oracle.com

-- Force cleanup with verbose logging
adop phase=cleanup loglevel=statement

OUI-67076: OPatch failed — inventory not found

-- OPatch can't find the Oracle Inventory
-- Check inventory pointer
-- cat /etc/oraInst.loc

-- Verify inventory exists
-- ls -la $ORACLE_INVENTORY/ContentsXML/inventory.xml

-- Re-attach Oracle home to inventory
$ORACLE_HOME/oui/bin/runInstaller -silent -ignoreSysPrereqs \
  -attachHome \
  ORACLE_HOME=$ORACLE_HOME \
  ORACLE_HOME_NAME=OraDB19Home1

-- Validate the home is now visible to OPatch
$ORACLE_HOME/OPatch/opatch lsinventory -oh $ORACLE_HOME