OPP Tuning for Oracle EBS Runbook

1. What is OPP and Why It Fails

The Output Post Processor (OPP) is a specialised Concurrent Manager service responsible for post-processing output from Concurrent Request runs — primarily PDF and XML Publisher (BI Publisher) output generation. When a request completes, OPP converts XML output to the final delivery format (PDF, HTML, Excel) using BI Publisher templates.

OPP failures affect any report using BI Publisher / XML Publisher in EBS. Users see requests completing with "Warning" or "Error" status even though the underlying concurrent request ran successfully.

Common causes of OPP failures:

• Thread starvation: OPP is configured with too few worker threads (default: 1). During report bursts, the queue backs up.
• Memory exhaustion: Large reports cause the OPP JVM to run out of heap memory (OOM), killing worker threads.
• Template corruption: A bad BI Publisher template causes OPP to hang on that request indefinitely.
• APPS schema issues: OPP cannot connect to APPS schema (password change, locked account).
• Temp directory full: OPP writes intermediate files to $APPLPTMP — if the filesystem is full, all processing stops.

2. Symptoms

• Concurrent requests complete with "Warning: Post Processing failed" or "Error" status
• Users cannot view output files for completed requests
• Output files show as "Unavailable" in the Request Completion window
• OPP queue depth climbing in TuneVault EBS health check
• CM log shows: OPP service deactivated or Post Processing Agent not responding
• In FNDOPP log: java.lang.OutOfMemoryError

3. Diagnose: OPP Status and Queue

3a. Check OPP service status in EBS

-- OPP manager status (check ACTIVE_PROCESSES)
SELECT
  cm.concurrent_queue_name,
  cm.description,
  cm.max_processes,
  cm.running_processes,
  cm.worker_count
FROM apps.fnd_concurrent_queues cm
WHERE cm.concurrent_queue_name = 'FNDOPP';

3b. Check OPP queue depth

-- Count requests waiting for OPP post-processing
SELECT COUNT(*)
FROM apps.fnd_concurrent_requests
WHERE phase_code = 'C'   -- Completed
  AND status_code = 'C'   -- Completed (waiting for post-processing)
  AND print_group IS NOT NULL;

-- Detail: top requests in OPP queue
SELECT
  r.request_id,
  r.concurrent_program_id,
  p.user_concurrent_program_name,
  r.requested_start_date,
  r.actual_completion_date,
  r.status_code,
  r.completion_text
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
  AND p.application_id = r.program_application_id
WHERE r.status_code IN ('D', 'Z')  -- Pending post-processing
ORDER BY r.actual_completion_date
FETCH FIRST 20 ROWS ONLY;

3c. Check OPP log for errors

-- Find latest OPP log file on the apps tier
-- As applmgr OS user:
-- ls -lt $APPLLOG/FNDOPP*.log | head -5
-- tail -200 $APPLLOG/FNDOPP_PID.log

-- Look for these key patterns:
-- "java.lang.OutOfMemoryError" → heap issue, increase JVM memory
-- "ORA-01017" → APPS password authentication failed
-- "UNABLE_TO_GET_DATA" → template not found or corrupted
-- "Deactivated by Administrator" → manually deactivated

3d. Check temp directory space

-- Check APPLPTMP filesystem usage (as applmgr OS user)
-- echo $APPLPTMP
-- df -h $APPLPTMP

-- Clean old temp files if disk is full
-- find $APPLPTMP -name "*.tmp" -mtime +1 -delete
-- find $APPLPTMP -name "*.pdf" -mtime +7 -delete

4. Clear Stuck OPP Requests

-- Find requests stuck in post-processing for more than 30 minutes
SELECT
  r.request_id,
  p.user_concurrent_program_name,
  r.actual_completion_date,
  ROUND((SYSDATE - r.actual_completion_date) * 1440, 0) AS mins_waiting,
  r.status_code
FROM apps.fnd_concurrent_requests r
JOIN apps.fnd_concurrent_programs_vl p
  ON p.concurrent_program_id = r.concurrent_program_id
  AND p.application_id = r.program_application_id
WHERE r.status_code = 'Z'
  AND (SYSDATE - r.actual_completion_date) * 1440 > 30
ORDER BY r.actual_completion_date;

-- Mark a specific stuck request as errored to unblock the queue
-- Replace 12345 with the actual request_id
UPDATE apps.fnd_concurrent_requests
SET status_code = 'E', completion_text = 'Manually cleared - OPP recovery'
WHERE request_id = 12345
  AND status_code = 'Z';
COMMIT;

5. Restart OPP

5a. Restart via EBS System Administrator

Preferred method for production:

1. Log in to EBS as System Administrator
2. Navigate to: Concurrent → Manager → Administer
3. Find Output Post Processor in the list
4. Click Deactivate, wait for processes to stop
5. Click Activate

5b. Restart via SQL (when UI is inaccessible)

-- Deactivate OPP via SQL
UPDATE apps.fnd_concurrent_queues
SET running_processes = 0, max_processes = 0
WHERE concurrent_queue_name = 'FNDOPP';
COMMIT;

-- Wait 30 seconds for processes to stop, then reactivate
UPDATE apps.fnd_concurrent_queues
SET max_processes = 1  -- or your target thread count
WHERE concurrent_queue_name = 'FNDOPP';
COMMIT;

5c. Bounce OPP from OS (last resort)

-- As applmgr OS user, source EBS environment first
-- . /u01/oracle/ebs/EBSapps.env run

-- Stop OPP
-- cmctl stop FNDOPP

-- Verify it stopped
-- ps -ef | grep FNDOPP

-- Start OPP
-- cmctl start FNDOPP

6. Tune OPP Thread Count

The default OPP thread count of 1 is insufficient for environments with heavy BI Publisher usage. Recommended values:

• Low volume (fewer than 50 XML Publisher reports/day): 1–2 threads
• Medium volume (50–200 reports/day): 3–5 threads
• High volume (200+ reports/day): 5–10 threads

-- Increase OPP max processes
UPDATE apps.fnd_concurrent_queues
SET max_processes = 5,    -- target thread count
    worker_count = 5
WHERE concurrent_queue_name = 'FNDOPP';
COMMIT;

-- Restart OPP for the change to take effect (use method from Step 5)

6b. Increase OPP JVM heap (for OOM errors)

-- Find OPP JVM args profile option
SELECT profile_option_value
FROM apps.fnd_profile_option_values
WHERE profile_option_id = (
  SELECT profile_option_id
  FROM apps.fnd_profile_options
  WHERE profile_option_name = 'FND_OPP_USE_JVM'
);

-- In EBS: System Admin → Profiles → System
-- Profile: FND: Output Post Processor JVM Parameters
-- Add: -Xmx1024m (increase from default 512m to 1024m)

7. Verify and Monitor

-- Confirm OPP is running with correct thread count
SELECT
  concurrent_queue_name,
  running_processes,
  max_processes,
  worker_count
FROM apps.fnd_concurrent_queues
WHERE concurrent_queue_name = 'FNDOPP';

-- Test: submit a simple BI Publisher report and verify it completes
-- Check: status_code should move from Z to C (completed) within a minute
SELECT request_id, status_code, completion_text
FROM apps.fnd_concurrent_requests
WHERE request_id = &YOUR_TEST_REQUEST_ID;