Documentation · Database Operations

Oracle EBS 12.2 + 19c RAC Clone Using RMAN Duplicate

An end-to-end runbook to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c RAC database using RMAN Active Database Duplicate within the same datacenter — including single-instance-to-RAC conversion, EBS Rapid Clone, post-clone tasks, and adding a new RAC node.

Version 2.0 · Oracle 19c + EBS 12.2.x · Same-datacenter clone

Contents

Overview and Architecture
Pre-Clone Checklist
Target Cluster Preparation
RMAN Active Database Duplicate
Convert Single Instance to 2-Node RAC
EBS 12.2 Application Tier Clone
Post-Clone Tasks
Adding a New Node to Existing RAC Cluster
Troubleshooting Common Issues
Post-Clone Validation Checklist

1. Overview and Architecture

This runbook covers the end-to-end procedure to clone an Oracle E-Business Suite 12.2 environment backed by a 2-node Oracle 19c Real Application Clusters (RAC) database using RMAN Duplicate within the same datacenter. It also covers the process of adding a new RAC node to an existing cluster.

1.1 Scope

Source: 2-node Oracle 19c RAC (CDB with EBS PDB), ASM storage, EBS 12.2 application tier (2 app nodes)
Target: New 2-node RAC cluster + 2 new EBS app tier nodes — all within the same datacenter
Database clone method: RMAN Duplicate (Active Database Duplicate — no backup required)
EBS clone method: Rapid Clone (adcfgclone.pl) after DB duplicate
New node addition: Oracle Clusterware + Grid Infrastructure extension to a third/fourth node

1.2 Environment Reference

Source DB Cluster

prod-rac1, prod-rac2

Target DB Cluster

clone-rac1, clone-rac2

Source EBS App Nodes

prod-app1, prod-app2

Target EBS App Nodes

clone-app1, clone-app2

Oracle DB Version

19c (19.x.x.x)

EBS Version

12.2.x (with latest patches)

DB Name (CDB)

PRODCDB → CLNECDB

EBS PDB Name

EBSPDB → CLNEPDB

ASM Disk Groups

+DATA, +REDO, +FRA

Grid Home

/u01/app/19.0.0/grid

Oracle Home

/u01/app/oracle/product/19.0.0/dbhome_1

EBS Base

/u01/app/ebsapps

OS User (DB)

oracle / oinstall, dba

OS User (Grid)

grid / oinstall, asmadmin

2. Pre-Clone Checklist

2.1 Hardware and OS Requirements

Identical CPU architecture (x86_64)
Same OS version and patch level as source nodes
Sufficient RAM: minimum 32GB per DB node (same as source)
ASM disk groups provisioned and visible on target nodes
Network: Public, Private (interconnect), and SCAN IPs configured
SSH equivalence set up between all source and target nodes (oracle and grid users)
NTP synchronized across all nodes

2.2 Oracle Software Prerequisites

Oracle Grid Infrastructure 19c installed and configured on target cluster (clusterware up, SCAN listener running)
Oracle Database 19c software installed on target nodes (ORACLE_HOME same path as source)
ASM disk groups created on target: +DATA, +REDO, +FRA (same names as source, size ≥ source usage)
Oracle Inventory consistent across all nodes

2.3 Network Prerequisites

Public IPs

One per node (clone-rac1, clone-rac2)

Virtual IPs (VIP)

One per node (clone-rac1-vip, clone-rac2-vip)

SCAN Name

clone-scan (resolves to 3 IPs via DNS/GNS)

Private Interconnect

bond0 (10GbE recommended, MTU 9000)

/etc/hosts

All nodes must resolve each other by hostname

2.4 Source Database Checks

Step	Task	Command / Action	Run As
1	Check DB open mode	`SELECT name, open_mode, db_unique_name FROM v$database;`	oracle/sqlplus
2	Check CDB/PDB status	`SELECT name, open_mode FROM v$pdbs;`	oracle/sqlplus
3	Check archive log mode	`ARCHIVE LOG LIST;`	oracle/sqlplus
4	Check FRA space	`SELECT * FROM v$recovery_file_dest;`	oracle/sqlplus
5	Check RMAN config	`SHOW ALL;`	rman target /
6	Verify ASM disk groups	`SELECT name, state, total_mb, free_mb FROM v$asm_diskgroup;`	grid/sqlplus
7	Check active sessions on source	`SELECT count(*) FROM v$session WHERE status='ACTIVE';`	oracle/sqlplus

⚠ NOTE: Place the source database in archive log mode if it is not already. EBS 12.2 requires archive log mode for online patching (ADOP).

3. Target Cluster Preparation

3.1 Grid Infrastructure Verification

Step	Task	Command / Action	Run As
1	Verify clusterware status	`crsctl stat res -t`	grid
2	Check SCAN listener	`srvctl status scan_listener`	grid
3	Verify ASM disk groups	`asmcmd lsdg`	grid
4	Check OCR and Voting Disk	`ocrcheck` · `crsctl query css votedisk`	root
5	Verify interconnect	`oifcfg getif`	grid

3.2 ASM Disk Group Preparation

Shell — clone-rac1 as grid

# On clone-rac1 as grid: asmcmd ASMCMD> lsdg ASMCMD> mkdir +DATA/CLNECDB ASMCMD> mkdir +REDO/CLNECDB ASMCMD> mkdir +FRA/CLNECDB ASMCMD> exit

3.3 Oracle Password File on Target

Shell — as oracle

# On clone-rac1 as oracle: orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB1 password=<sys_password> entries=10 # On clone-rac2 as oracle: orapwd file=$ORACLE_HOME/dbs/orapwCLNECDB2 password=<sys_password> entries=10

3.4 Static Listener Configuration on Target

Create/update $TNS_ADMIN/listener.ora on both target nodes:

listener.ora

LISTENER_CLNECDB = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = clone-rac1-vip)(PORT = 1521)))) SID_LIST_LISTENER_CLNECDB = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = CLNECDB) (ORACLE_HOME = /u01/app/oracle/product/19.0.0/dbhome_1) (SID_NAME = CLNECDB1))) # Start the listener: lsnrctl start LISTENER_CLNECDB

3.5 Auxiliary Init Parameter File

Create /tmp/init_aux.ora on clone-rac1:

/tmp/init_aux.ora

db_name=CLNECDB db_unique_name=CLNECDB enable_pluggable_database=TRUE sga_target=8G pga_aggregate_target=2G control_files='+DATA/CLNECDB/control01.ctl','+REDO/CLNECDB/control02.ctl' db_file_name_convert='+DATA/PRODCDB','+DATA/CLNECDB','+REDO/PRODCDB','+REDO/CLNECDB' log_file_name_convert='+REDO/PRODCDB','+REDO/CLNECDB' db_recovery_file_dest='+FRA' db_recovery_file_dest_size=500G audit_file_dest='/u01/app/oracle/admin/CLNECDB/adump' diagnostic_dest='/u01/app/oracle' cluster_database=FALSE instance_number=1 thread=1

⚠ NOTE: Set cluster_database=FALSE initially. It will be set to TRUE after the duplicate completes and RAC is configured.

4. RMAN Active Database Duplicate

4.1 Start Auxiliary Instance

Shell — clone-rac1 as oracle

# On clone-rac1 as oracle: export ORACLE_SID=CLNECDB1 mkdir -p /u01/app/oracle/admin/CLNECDB/adump sqlplus / as sysdba SQL> STARTUP NOMOUNT PFILE='/tmp/init_aux.ora'; SQL> EXIT;

4.2 Configure RMAN Channels and Start Duplicate

Run the following RMAN script from clone-rac1 as oracle. This performs an Active Database Duplicate — no backup needed, data streams directly from source to target:

RMAN — connect

rman target sys/<sys_pwd>@PRODCDB auxiliary sys/<sys_pwd>@CLNECDB_AUX

Then at the RMAN prompt:

RMAN — DUPLICATE

DUPLICATE TARGET DATABASE TO CLNECDB FROM ACTIVE DATABASE USING COMPRESSED BACKUPSET NOFILENAMECHECK SPFILE PARAMETER_VALUE_CONVERT 'PRODCDB','CLNECDB', '+DATA/PRODCDB','+DATA/CLNECDB', '+REDO/PRODCDB','+REDO/CLNECDB' SET db_unique_name='CLNECDB' SET db_recovery_file_dest='+FRA' SET db_recovery_file_dest_size='500G' SET cluster_database='FALSE' SET instance_number='1' SET thread='1' ALLOCATE AUXILIARY CHANNEL c1 DEVICE TYPE DISK ALLOCATE AUXILIARY CHANNEL c2 DEVICE TYPE DISK ALLOCATE AUXILIARY CHANNEL c3 DEVICE TYPE DISK ALLOCATE AUXILIARY CHANNEL c4 DEVICE TYPE DISK;

⚠ NOTE: The duplicate may take several hours depending on source database size. Monitor the alert log on the target: tail -f $ORACLE_BASE/diag/rdbms/clnecdb/CLNECDB1/trace/alert_CLNECDB1.log

4.3 Post-Duplicate Verification

Step	Task	Command / Action	Run As
1	Verify DB is open	`SELECT name, open_mode FROM v$database;`	oracle/sqlplus
2	Verify PDB status	`SELECT name, open_mode FROM v$pdbs;`	oracle/sqlplus
3	Open EBS PDB	`ALTER PLUGGABLE DATABASE CLNEPDB OPEN;`	oracle/sqlplus
4	Check datafiles	`SELECT name FROM v$datafile;`	oracle/sqlplus
5	Verify redo logs	`SELECT group#, status FROM v$log;`	oracle/sqlplus
6	Check temp tablespace	`SELECT name FROM v$tempfile;`	oracle/sqlplus

5. Convert Single Instance to 2-Node RAC

5.1 Create SPFILE in ASM

SQL — clone-rac1 as oracle

# On clone-rac1 as oracle: sqlplus / as sysdba SQL> CREATE SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora' FROM MEMORY; SQL> SHUTDOWN IMMEDIATE; SQL> EXIT;

5.2 Create RAC-Enabled PFILE

Create /tmp/init_rac.ora on clone-rac1:

/tmp/init_rac.ora

SPFILE='+DATA/CLNECDB/spfileCLNECDB.ora'

5.3 Modify SPFILE for RAC

SQL

sqlplus / as sysdba SQL> STARTUP PFILE='/tmp/init_rac.ora'; SQL> ALTER SYSTEM SET cluster_database=TRUE SCOPE=SPFILE; SQL> ALTER SYSTEM SET instance_number=1 SID='CLNECDB1' SCOPE=SPFILE; SQL> ALTER SYSTEM SET instance_number=2 SID='CLNECDB2' SCOPE=SPFILE; SQL> ALTER SYSTEM SET thread=1 SID='CLNECDB1' SCOPE=SPFILE; SQL> ALTER SYSTEM SET thread=2 SID='CLNECDB2' SCOPE=SPFILE; SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS1' SID='CLNECDB1' SCOPE=SPFILE; SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SID='CLNECDB2' SCOPE=SPFILE; SQL> SHUTDOWN IMMEDIATE; SQL> EXIT;

5.4 Create Undo Tablespace for Node 2

SQL

sqlplus / as sysdba SQL> STARTUP; SQL> CREATE UNDO TABLESPACE UNDOTBS2 DATAFILE '+DATA/CLNECDB/undotbs2_01.dbf' SIZE 2G AUTOEXTEND ON; SQL> SHUTDOWN IMMEDIATE; SQL> EXIT;

5.5 Enable Redo Thread for Node 2

SQL

sqlplus / as sysdba SQL> STARTUP MOUNT; SQL> ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 4 ('+REDO/CLNECDB/redo04a.log','+DATA/CLNECDB/redo04b.log') SIZE 512M, GROUP 5 ('+REDO/CLNECDB/redo05a.log','+DATA/CLNECDB/redo05b.log') SIZE 512M, GROUP 6 ('+REDO/CLNECDB/redo06a.log','+DATA/CLNECDB/redo06b.log') SIZE 512M; SQL> ALTER DATABASE ENABLE PUBLIC THREAD 2; SQL> ALTER DATABASE OPEN; SQL> EXIT;

5.6 Add Database to Clusterware (srvctl)

Shell — clone-rac1 as oracle

# On clone-rac1 as oracle: srvctl add database -d CLNECDB \ -o $ORACLE_HOME \ -p +DATA/CLNECDB/spfileCLNECDB.ora \ -r PRIMARY \ -s OPEN \ -t IMMEDIATE \ -i CLNECDB1,CLNECDB2 srvctl add instance -d CLNECDB -i CLNECDB1 -n clone-rac1 srvctl add instance -d CLNECDB -i CLNECDB2 -n clone-rac2 srvctl start database -d CLNECDB srvctl status database -d CLNECDB

5.7 Register PDB with Clusterware

Shell

srvctl add service -d CLNECDB -s EBSPDB_SERVICE \ -pdb CLNEPDB \ -preferred CLNECDB1,CLNECDB2 \ -failovertype SELECT \ -failovermethod BASIC \ -failoverretry 5 \ -failoverdelay 3 srvctl start service -d CLNECDB -s EBSPDB_SERVICE srvctl status service -d CLNECDB -s EBSPDB_SERVICE

6. EBS 12.2 Application Tier Clone

6.1 Prerequisites — Database Side

Step	Task	Command / Action	Run As
1	Connect to EBS PDB	`ALTER SESSION SET CONTAINER=CLNEPDB;`	oracle/sqlplus
2	Unlock APPS schema (if locked)	`ALTER USER APPS ACCOUNT UNLOCK;`	oracle/sqlplus
3	Reset APPS password	`EXEC APPS.FND_USER_PKG.CHANGE_USER_NAME('APPS','APPS');`	oracle/sqlplus
4	Verify APPLSYS	`SELECT username, account_status FROM dba_users WHERE username='APPLSYS';`	oracle/sqlplus
5	Check concurrent manager tables	`SELECT COUNT(*) FROM apps.fnd_concurrent_queues;`	oracle/sqlplus

6.2 Source Application Tier — Pre-Clone Steps

Run on prod-app1 as applmgr:

Shell — prod-app1 as applmgr

# Source the EBS environment source /u01/app/ebsapps/EBSapps.env run # Prepare source for cloning perl $ADMIN_SCRIPTS_HOME/adpreclone.pl appsTier

⚠ NOTE: adcfgclone.pl is used for full environment clones (new environment, different hostname/instance name). For adding a node to an existing EBS installation, use adclonectx.pl addnode instead — see Section 8.7.

6.3 Copy Application Tier to Target Nodes

From prod-app1, copy the entire EBS base to both target app nodes:

Shell — prod-app1

# Copy filesystem to clone-app1 tar czf - /u01/app/ebsapps | ssh clone-app1 "tar xzf - -C /" # Copy filesystem to clone-app2 tar czf - /u01/app/ebsapps | ssh clone-app2 "tar xzf - -C /"

⚠ NOTE: This may take significant time depending on EBS installation size (typically 100GB–300GB). Consider using rsync for subsequent syncs.

6.4 Configure Application Tier on clone-app1

Run on clone-app1 as applmgr:

Shell — clone-app1 as applmgr

cd $COMMON_TOP/clone/bin perl adcfgclone.pl appsTier

When prompted, provide:

Target System Hostname

clone-app1

Target Instance Name

CLNE (new environment name)

Target DB Host

clone-scan (SCAN hostname)

Target DB Port

1521

Target DB Service Name

EBSPDB_SERVICE (or PDB service)

Target ORACLE_SID (CDB)

CLNECDB

APPS Password

<apps_password>

SYSTEM Password

<system_password>

WebLogic Admin Password

<wls_password>

6.5 Configure Application Tier on clone-app2

Run on clone-app2 as applmgr:

Shell — clone-app2 as applmgr

cd $COMMON_TOP/clone/bin perl adcfgclone.pl appsTier

Provide the same parameters as clone-app1 but with clone-app2 as the hostname.

6.6 Post-Clone EBS Configuration

Step	Task	Command / Action	Run As
1	Source env on clone-app1	`source /u01/app/ebsapps/EBSapps.env run`	applmgr
2	Start Apache/OHS	`adapcctl.sh start`	applmgr
3	Start OPMN	`adopmnctl.sh start`	applmgr
4	Start Node Manager	`adnodemgrctl.sh start`	applmgr
5	Start WLS Admin Server	`adstrtal.sh`	applmgr
6	Start all Managed Servers	`adadminsrvctl.sh start all`	applmgr
7	Start Concurrent Manager	`adcmctl.sh start apps/<apps_pwd>`	applmgr
8	Verify EBS URL	`http://clone-app1.domain.com:8000/OA_HTML/AppsLocalLogin.jsp`	Browser

7. Post-Clone Tasks

7.1 EBS Database Profile Updates

SQL — connect to EBS PDB

-- Connect to EBS PDB: ALTER SESSION SET CONTAINER=CLNEPDB; -- Update application node hostname in FND_NODES UPDATE apps.fnd_nodes SET node_name='clone-app1' WHERE node_name='prod-app1'; UPDATE apps.fnd_nodes SET node_name='clone-app2' WHERE node_name='prod-app2'; COMMIT; -- Update server host in WF_AGENTS UPDATE apps.wf_agents SET address=REPLACE(address,'prod-app1','clone-app1') WHERE address LIKE '%prod-app1%'; COMMIT;

7.2 Disable/Modify Email Notifications

SQL

-- Disable all concurrent program email notifications on clone UPDATE apps.fnd_svc_components SET component_status='STOPPED' WHERE component_type='WF_MAILER'; COMMIT;

⚠ NOTE: Always disable email notifications on cloned/non-production environments to prevent accidental emails to real users.

7.3 Autoconfig

Run Autoconfig on both app nodes and the DB node to synchronize configuration:

Shell

# On clone-app1 and clone-app2 as applmgr: perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE # On clone-rac1 as oracle (DB tier autoconfig): perl $AD_TOP/bin/adconfig.pl contextfile=$CONTEXT_FILE

7.4 ETCC Verification

Run the EBS Technology Codelevel Checker to confirm patch levels:

Shell

perl $AD_TOP/patch/115/bin/adpatch.pl options='nocopypfile' \ patchtop=$PATCH_TOP appspass=$APPS_PWD \ workers=8 logfile=etcc.log

7.5 Verify ADOP Cycle is Clean

SQL

-- Check for stuck ADOP sessions: SELECT session_id, prepare_status, apply_status, finalize_status, cutover_status, cleanup_status FROM apps.ad_adop_sessions WHERE session_id = (SELECT MAX(session_id) FROM apps.ad_adop_sessions);

⚠ NOTE: If the source had an in-progress ADOP session, it will be replicated to the clone. Abort it on the clone: adop phase=abort.

8. Adding a New Node to Existing RAC Cluster

8.1 Overview

This section covers extending an existing 2-node RAC cluster by adding a third node (clone-rac3). The same process applies to adding a fourth node.

8.2 Prepare the New Node

Step	Task	Command / Action	Run As
1	Install same OS version	Match patch level with existing nodes	root
2	Configure kernel parameters	`/etc/sysctl.conf` — match existing nodes	root
3	Create OS users/groups	oracle, grid, oinstall, dba, asmadmin etc.	root
4	Configure /etc/hosts	Add all node hostnames + VIPs + SCAN	root
5	Set up SSH equivalence	`ssh-keygen; ssh-copy-id` to all existing nodes	oracle/grid
6	Verify ASM disks visible	`ls /dev/oracleasm/disks/`	root
7	Configure NTP	Same NTP server as other nodes	root

8.3 Extend Grid Infrastructure to New Node

Run from an existing node (clone-rac1) as grid:

Shell — clone-rac1 as grid

cd $GRID_HOME/addnode ./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}" \ "CLUSTER_NEW_VIRTUAL_HOSTNAMES={clone-rac3-vip}"

⚠ NOTE: This will copy Grid Infrastructure to the new node, configure CSS/CRS/OCR on the new node, and start clusterware. Monitor progress via $GRID_HOME/cfgtoollogs/addNode/.

8.4 Verify Clusterware on New Node

Shell

# Verify from any node: crsctl stat res -t # Verify from new node: crsctl check cluster -all olsnodes -n

8.5 Extend Oracle Database Software to New Node

Run from clone-rac1 as oracle:

Shell — clone-rac1 as oracle

cd $ORACLE_HOME/addnode ./addnode.sh "CLUSTER_NEW_NODES={clone-rac3}"

8.6 Add New Instance to Database

Shell + SQL — clone-rac1

# Add instance 3 as oracle on clone-rac1: srvctl add instance -d CLNECDB -i CLNECDB3 -n clone-rac3 # Configure instance-specific parameters: sqlplus / as sysdba SQL> ALTER SYSTEM SET instance_number=3 SID='CLNECDB3' SCOPE=SPFILE; SQL> ALTER SYSTEM SET thread=3 SID='CLNECDB3' SCOPE=SPFILE; SQL> ALTER SYSTEM SET undo_tablespace='UNDOTBS3' SID='CLNECDB3' SCOPE=SPFILE; SQL> EXIT; # Create undo tablespace for node 3: sqlplus / as sysdba SQL> CREATE UNDO TABLESPACE UNDOTBS3 DATAFILE '+DATA/CLNECDB/undotbs3_01.dbf' SIZE 2G AUTOEXTEND ON; # Add redo thread for node 3: SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 7 ('+REDO/CLNECDB/redo07a.log','+DATA/CLNECDB/redo07b.log') SIZE 512M, GROUP 8 ('+REDO/CLNECDB/redo08a.log','+DATA/CLNECDB/redo08b.log') SIZE 512M, GROUP 9 ('+REDO/CLNECDB/redo09a.log','+DATA/CLNECDB/redo09b.log') SIZE 512M; SQL> ALTER DATABASE ENABLE PUBLIC THREAD 3; SQL> EXIT; # Start instance on new node: srvctl start instance -d CLNECDB -i CLNECDB3 srvctl status database -d CLNECDB

8.7 Add New EBS App Node

In standard EBS 12.2 deployments, the application filesystem (APPL_TOP, COMMON_TOP, OA_HTML) is on shared NFS/NAS storage mounted by all app nodes. Adding a new app node does NOT require copying files — the new node mounts the existing shared filesystem. Only INST_TOP (instance-specific config, logs, temp) is local to each node.

⚠ NOTE: If your EBS is deployed on local disk (non-shared, e.g. DMZ environments), you will need to rsync/tar the filesystem to the new node first. For standard shared NFS deployments, skip directly to Step 1 below.

Run on clone-app3 as applmgr:

Shell — clone-app3

# Step 1: Mount shared EBS filesystems on new node (as root): mount -t nfs nfs-server:/exports/ebsapps /u01/app/ebsapps mount -t nfs nfs-server:/exports/common /u01/app/common # Add permanent entries to /etc/fstab # Step 2: Create local INST_TOP directory: mkdir -p /u01/app/inst/apps/CLNE_clone-app3 # Step 3: Source environment from existing node context: source /u01/app/ebsapps/EBSapps.env run # Step 4: Create node-specific context file using adclonectx.pl: perl $AD_TOP/bin/adclonectx.pl addnode \ contextfile=$CONTEXT_FILE \ pairsfile=/tmp/clone-app3_pairs.txt

The pairs file (/tmp/clone-app3_pairs.txt) contains node-specific overrides:

/tmp/clone-app3_pairs.txt

s_hostname=clone-app3 s_local_hostname=clone-app3 s_webentryhost=clone-app3 s_active_webport=8000 s_tools_oracle_home=/u01/app/ebsapps/fs1/EBSapps/10.1.2

Shell — clone-app3

# Step 5: Run Autoconfig on new node using new context file: perl $AD_TOP/bin/adconfig.pl contextfile=/u01/app/inst/apps/CLNE_clone-app3/appl/admin/CLNE_clone-app3.xml # Step 6: Start services on new node: adapcctl.sh start adopmnctl.sh start adnodemgrctl.sh start adstrtal.sh # Step 7: Register new app node in FND_NODES (if not auto-registered): INSERT INTO apps.fnd_nodes (node_name, server_id, support_cp, support_forms, support_web, platform_code) VALUES ('clone-app3', apps.fnd_nodes_s.nextval, 'Y', 'Y', 'Y', 'LINUX'); COMMIT;

9. Troubleshooting Common Issues

9.1 RMAN Duplicate Failures

Symptom	Resolution
ORA-17628: Oracle error 19505	ASM directory for target does not exist. Create dirs with `asmcmd mkdir`.
RMAN-05001: auxiliary channel not started	Auxiliary listener not running or TNS not reachable. Check `tnsnames.ora` and `listener.ora` on target.
ORA-12154: TNS could not resolve	Add PRODCDB and CLNECDB_AUX to `tnsnames.ora` on source and target.
Duplicate hangs after datafile copy	Check network bandwidth. Use fewer parallel channels if bandwidth limited.

9.2 RAC Startup Issues

Symptom	Resolution
ORA-29701: unable to connect to Cluster Manager	Clusterware not running. Run: `crsctl start cluster -all`
Instance stuck in MOUNT state	Redo thread not enabled. `ALTER DATABASE ENABLE PUBLIC THREAD N;`
UNDO errors on node 2	Wrong undo tablespace assigned. Check SPFILE parameter `undo_tablespace` for SID.
CSS daemon not starting on new node	OCR or voting disk not accessible. Check ASM disk group visibility.

9.3 EBS Clone Issues

Symptom	Resolution
adcfgclone.pl fails on context file	Run `adpreclone.pl` on source first. For node addition use `adclonectx.pl addnode` instead of `adcfgclone.pl`.
WLS Admin Server not starting	Check `$DOMAIN_HOME/servers/AdminServer/logs/`. Often Java heap or port conflict.
Login page not loading after clone	Run Autoconfig. Check Apache/OHS config in `$INST_TOP/ora/10.1.3/Apache/Apache/conf/`.
FND_NODES mismatch	Update `fnd_nodes` with new hostnames. Run Autoconfig after.
Concurrent Manager not starting	Check ICM log in `$APPLCSF/$APPLLOG`. Usually APPS password or FND_NODES issue.

10. Post-Clone Validation Checklist

✓	Task	Command / Action	Run As
✓	DB instances running on both nodes	`srvctl status database -d CLNECDB`	oracle
✓	All PDBs open	`SELECT name, open_mode FROM v$pdbs;`	oracle/sqlplus
✓	SCAN listener working	`tnsping EBSPDB_SERVICE`	oracle
✓	EBS login page accessible	`http://clone-app1:8000/OA_HTML/AppsLocalLogin.jsp`	Browser
✓	Concurrent Manager running	`adcmctl.sh status apps/<pwd>`	applmgr
✓	No invalid objects	`SELECT COUNT(*) FROM dba_objects WHERE status='INVALID';`	oracle/sqlplus
✓	Archive log mode verified	`ARCHIVE LOG LIST;`	oracle/sqlplus
✓	Email notifications disabled	Check `fnd_svc_components` WF_MAILER status	oracle/sqlplus
✓	Autoconfig completed on all nodes	Check autoconfig log for errors	applmgr
✓	ADOP session clean (no stuck sessions)	Check `ad_adop_sessions`	oracle/sqlplus
✓	Tablespace usage normal	Check `dba_tablespace_usage_metrics`	oracle/sqlplus
✓	Alert log clean on all nodes	`tail -100 alert_CLNECDB*.log`	oracle

After a clone, run a TuneVault health check to validate the environment automatically — invalid objects, tablespace usage, ADOP state, listener health, and EBS operations. Add a connection →