SYS-01: Concurrent Manager Health Diagnostic

The Oracle EBS Concurrent Manager is the batch processing backbone of every Oracle module. Every scheduled program, every period-end process, every data import, and every background job submits a request to FND_CONCURRENT_REQUESTS and waits for a Concurrent Manager worker to pick it up. When the Concurrent Manager is degraded — whether because the Internal Concurrent Manager (ICM) is down, a specific manager has no active workers, the request queue has exceeded capacity, or a single long-running request is blocking the queue — the entire Oracle environment stops processing batch work. Invoice validation, journal imports, payroll runs, AutoInvoice, and period close programs all fail silently by sitting in PENDING or NORMAL status indefinitely.

The most operationally invisible Concurrent Manager failure mode is the work shift gap. Each Concurrent Manager is assigned work shifts that define when it is active and how many workers it runs during each shift. When a work shift configuration is changed — or when a target node in a parallel manager setup goes down — the manager appears active in the FND_CONCURRENT_MANAGERS table but processes zero requests because it has no active workers during the current shift. The Oracle Concurrent Manager screen shows the manager as Running, which misleads the team into thinking the infrastructure is healthy.

Stuck requests are the second major failure mode. A stuck request has status RUNNING in FND_CONCURRENT_REQUESTS but its OS process no longer exists — the program crashed without updating the request status back to ERROR. Stuck requests hold a worker slot indefinitely, progressively reducing the manager's processing capacity as more requests get stuck. On a manager with four workers, four stuck requests will completely freeze batch processing while the manager continues to report four Running workers.

SYS-01 runs a complete Concurrent Manager health diagnostic — ICM status and node availability, manager-by-manager active worker count versus configured workers, work shift validity for the current time, queue depth and aging by manager, stuck request identification with OS PID cross-reference, top failure patterns by program name, and running request duration analysis to identify candidates for termination.

ICM & Node Status

Internal Concurrent Manager running status and node availability across all processing nodes. Identifies whether the ICM is healthy before investigating individual managers — ICM down is the root cause of all-manager failure.

Worker & Work Shift

Manager-by-manager comparison of active workers vs configured max workers. Work shift validity for the current time and day of week. Identifies managers with zero workers due to expired or missing work shifts.

Stuck Request Detection

Requests in RUNNING status cross-referenced against active OS PIDs on each processing node. A request with no corresponding OS process is stuck and holding a worker slot. SYS-01 identifies all stuck requests and their duration.

Queue & Failure Analysis

Queue depth and oldest request age per manager. Top failure patterns by program name over 7 days — identifies systemic program errors vs one-time failures. Running request duration analysis for termination candidates.

════════════════════════════════════════════════════════════
  ORACLE EBS R12 — CONCURRENT MANAGER HEALTH DIAGNOSTIC
════════════════════════════════════════════════════════════
  Environment        : PROD
  Report Date        : 25-FEB-2026 09:15:44
  Case Number        : SYS-481022
════════════════════════════════════════════════════════════

[ SECTION 1 — ICM STATUS ]                 STATUS: ✓ RUNNING
────────────────────────────────────────────────────────────
  Internal Concurrent Manager — RUNNING on node EBSPROD01 ✓
  Node               : EBSPROD01 — active ✓
  Node               : EBSPROD02 — active ✓

[ SECTION 2 — MANAGER WORKER STATUS ]      STATUS: ✗ DEGRADED
────────────────────────────────────────────────────────────
  Standard Manager   : 15/15 workers active ✓
  Payables Manager   : 0/4 workers active ✗ — work shift gap
  Receivables Mgr    : 4/4 workers active ✓
  GL Manager         : 4/4 workers active ✓
  ✗ FAIL: Payables Manager has no active workers — work shift ends at 08:00

[ SECTION 3 — STUCK REQUESTS ]             STATUS: ⚠ WARNING
────────────────────────────────────────────────────────────
  ⚠ 2 stuck requests — status RUNNING, OS process not found
  Request 8841201    : Validate Payables Invoices — running 4h 22m
  Request 8841198    : Create Accounting — running 6h 11m
  OS PID Check       : PID 14882, 14891 — not found on EBSPROD01 ✗

[ SECTION 4 — QUEUE DEPTH ]                STATUS: ⚠ BACKLOG
────────────────────────────────────────────────────────────
  ⚠ 47 requests PENDING in Payables Manager queue — oldest 8h 42m
  Standard Manager queue — 3 pending, normal ✓

[ SECTION 5 — TOP FAILURES (7 DAYS) ]      STATUS: ✗ PATTERN
────────────────────────────────────────────────────────────
  Create Accounting (AP)  : 12 errors — account invalid (CCID disabled)
  Import Journals         : 0 errors ✓
  AutoInvoice             : 0 errors ✓

════════════════════════════════════════════════════════════
  DIAGNOSTIC SUMMARY
════════════════════════════════════════════════════════════
  Payables Manager degraded — work shift expired at 08:00, 47 requests queued
  2 stuck requests holding worker slots — must be terminated
  FIX: Extend work shift, terminate stuck PIDs, clear queue
════════════════════════════════════════════════════════════
  Backup Created  : CONS_BACKUP.FND_CONCURRENT_REQUESTS_481022 ✓
  Registry ID     : 7091
════════════════════════════════════════════════════════════

Diagnostic Engine

Runs 40+ checks across ICM status and node availability, manager-by-manager active worker count versus configured workers, work shift validity for the current time, queue depth and aging by manager, stuck request identification with OS PID cross-reference, and top failure patterns by program name over the prior 7 days.

Backup Created

Before any request is terminated or manager configuration is changed, CONS_BACKUP.FND_CONCURRENT_REQUESTS_<case#> captures the current queue state. Stuck request PIDs are documented before termination. Manager work shift snapshots are recorded before any modification.

Guided Data Fix

Stuck requests are terminated via Concurrent Programs > View Requests > Terminate — never by direct FND_CONCURRENT_REQUESTS DML. Work shift corrections are applied via System Administrator > Concurrent > Manager. Direct fixes are limited to FND_CONCURRENT_REQUESTS status corrections for orphaned requests where the correct approach is confirmed.

KB Article Generated

Complete KB article generated — environment, manager degraded, stuck request IDs, queue depth, work shift correction applied, clearance time. Upload directly to your knowledge base.

KB-SYS-481022-001 · Script: SYS-01

Payables Manager Work Shift Expired — 47 Requests Queued, 2 Stuck Requests

Module: System / Technical
Platform: EBS R12.2.10
Status: RESOLVED

Symptom

Payables Manager showing 0/4 active workers. 47 requests queued in Payables Manager, oldest 8h 42m. 2 stuck requests (8841201, 8841198) in RUNNING status with OS PIDs not found. Invoice Validation and Create Accounting programs not processing.

Root Cause

Payables Manager work shift was configured 08:00-17:00 for weekdays. DST change moved the shift end time effectively one hour earlier. Weekend emergency processing was not covered by any work shift. 2 requests from the prior evening crashed and were not cleaned up.

Tables Affected

FND_CONCURRENT_MANAGERS — PAYABLES_MGR: RUNNING_PROCESSES=0, MAX_PROCESSES=4
FND_CONCURRENT_REQUESTS — REQUEST_ID 8841201, 8841198: PHASE_CODE=R, STATUS_CODE=R (stuck)

Fix Applied

Payables Manager work shift extended to 06:00-22:00. Stuck requests 8841201 and 8841198 terminated via Concurrent Programs screen. Concurrent Manager bounced via SYSADMIN. Queue cleared over 35 minutes. All 47 pending requests processed by 10:30.

Prevention

Work shift review added to DST change runbook. Stuck request monitoring script scheduled — any request running > 3 hours with no active OS PID triggers an alert. Weekend processing coverage added to Payables and GL Manager work shifts.

Work Shift Gap — Zero Active Workers

Manager configured correctly but no active work shift covering the current time. Manager appears Running in the Oracle screen but processes nothing. Most common after DST changes or weekend coverage gaps. All requests queue indefinitely until the shift is corrected.

Stuck

Stuck Requests Holding Worker Slots

Programs that crashed mid-run without updating their request status. OS process is gone but FND_CONCURRENT_REQUESTS shows RUNNING. Each stuck request permanently holds one worker slot, progressively reducing manager capacity until all slots are occupied.

Pattern

Repeated Program Failure

The same concurrent program failing repeatedly — typically a data condition (invalid account, closed period) that persists between runs. SYS-01 identifies the error pattern from request history and the upstream data condition causing it.

Node

Parallel Manager Node Unavailable

One node in a parallel manager cluster is down — reduces processing capacity proportionally. Visible as active workers below max_processes with no work shift issue. Requires DBA to restart the node or redistribute workers.

Table	Module	Purpose
FND_CONCURRENT_MANAGERS	FND	Manager definitions — max workers, work shifts, node assignments
FND_CONCURRENT_REQUESTS	FND	All request submissions — phase, status, timing, program name
FND_CONCURRENT_PROCESSES	FND	Active manager processes — OS PID, node, status
FND_CONCURRENT_WORKER_REQUESTS	FND	Worker-to-request assignment — which worker is running what
FND_CONC_RELEASE_PERIODS	FND	Work shift period definitions — start/end times by day
FND_NODES	FND	Processing nodes — active status, platform
FND_CONCURRENT_PROGRAMS	FND	Program definitions — execution method, queue method

Decision Framework

How Every Fix Decision Is Made

Before any data is modified in a production Oracle database, AP-01 walks through a four-stage decision process. Every condition identified by the diagnostic maps to exactly one resolution path.

First Option — Always

Can the front end fix this?

Oracle's own forms and concurrent programs are always the first option. If the condition can be corrected through Oracle's standard UI — a form, a concurrent program, a setup screen — that path is taken first. No consultant SQL required, no database risk, and the fix is fully supported by Oracle. The diagnostic identifies these conditions explicitly and states the exact front-end navigation path.

✓ Functional Fix

When Front End Is Not Available

Is a direct data fix safe to apply?

When the front-end path is unavailable or would require an unacceptable volume of manual steps, a direct fix is evaluated against explicit criteria. All of the following must be true before proceeding:

→The fix mirrors exactly what Oracle's own code would have done if the underlying condition were corrected

→All dependent tables have been identified and will be updated in the same transaction

→The fix is fully reversible — a single INSERT from the backup table restores the original state

→The environment patch level has been confirmed against the fix logic version

⚡ Direct Fix

Hard Stops — No Exceptions

Does this require Oracle Support?

Certain table areas are never touched directly, regardless of how well the underlying structure is understood. The diagnostic flags these conditions and generates the Service Request documentation:

✗XLA subledger tables — XLA_EVENTS, XLA_AE_HEADERS — incorrect changes corrupt the subledger audit trail in ways undetectable until period close fails or an auditor requests a reconciliation

✗Workflow engine tables — WF_ITEMS, WF_ITEM_ACTIVITY_STATUSES — ad-hoc DML can corrupt the workflow engine state instance-wide

✗Recurring conditions after a fix — indicates a code defect, not a data error. Documented and submitted as a Service Request with a reproducible test case

⚠ Oracle Support

For All Approved Direct Fixes

Backup → Execute → Verify → Document

A timestamped backup table is created and row-count verified before the first UPDATE fires. Explicit parameter confirmation is required — the script will not self-execute. After execution, a verification query confirms the expected state. A complete change record — rows affected, before and after values, database username, timestamp — is written to the FIX_BACKUP_REGISTRY and becomes the primary artifact in the knowledge base entry for this incident.

📋 Documented

Condition Identified	Resolution Path	Notes
Work shift expired or missing — manager has 0 workers	Functional First	Extend or add the work shift via System Administrator > Concurrent > Manager > Work Shifts. SYS-01 identifies the current time, the active work shift window, and the gap causing zero workers.
Stuck request — RUNNING status, OS PID not found	Functional First	Terminate the request via Concurrent Programs > View Requests > Terminate. SYS-01 identifies the stuck request IDs, the OS PID that no longer exists, and the node where it was running.
Manager not starting — INACTIVE after Concurrent Manager restart	Functional First	Activate the manager via System Administrator > Concurrent > Manager > Activate. SYS-01 identifies managers in INACTIVE status and whether the issue is a work shift gap or a configuration error.
Request permanently stuck in PENDING — no worker picks it up	Direct Fix	If the target manager has no workers and cannot be restarted, SYS-01 updates the request's REQUESTED_BY_NAME to route it to the Standard Manager with full backup — only after confirming no manager will pick it up.
Top failure pattern — same program failing repeatedly	Functional First	SYS-01 identifies the top failing program from FND_CONCURRENT_REQUESTS history and the error message pattern. Root cause correction depends on the program — account setup, period status, or data condition.
Long-running request blocking the queue	Functional First	SYS-01 identifies requests running beyond their typical duration. Decision to terminate is made by confirming with the submitting user that the program can safely be restarted.
Parallel manager node down — requests not distributed	Functional First	SYS-01 identifies which node in the parallel manager cluster is not running workers and the number of workers that are now unavailable. Node restart is a DBA/sysadmin action.
FND_CONCURRENT_PROCESSES orphan — process record without active manager	Direct Fix	SYS-01 identifies orphaned FND_CONCURRENT_PROCESSES rows with full backup and corrects the STATUS_CODE after confirming no active sessions exist for the process.

Safeguards

Nothing Executes Without a Safety Net

Before any data fix runs, the script verifies pre-flight conditions and creates a complete verified backup. If any check fails, the script aborts. There is no partial execution path.

Pre-Flight Checks — All Must Pass

✓

POSTED_FLAG = 'N'

Will not modify a posted transaction under any circumstances

✓

ACCOUNTING_EVENT_ID IS NULL

Will not modify rows with an active or pending XLA accounting event

✓

No active database lock

Checks V$LOCK — will not proceed if another session holds a lock on target rows

✓

CONS_BACKUP schema writable

Backup schema must be accessible before any DML is permitted

✓

Explicit confirmation required

Script outputs a confirmation prompt — DML does not execute until consultant enters the case number as the execution parameter

Backup Methodology

Before the first UPDATE fires, the script creates a complete copy of every row to be modified. Tables are named CONS_BACKUP.<TABLE>_<CASE#> and persist permanently after execution.

FND_CONCURRENT_REQUESTS

FND_CONCURRENT_MANAGERS

FND_CONCURRENT_PROCESSES

Row count verified after backup creation. If backup fails for any reason the script aborts without executing any DML. Rollback is a single INSERT from the backup table. No reconstruction required.

SYS-01 — Pre-Flight & Backup Verification

════════════════════════════════════════════════════════════
  PRE-FLIGHT SAFETY CHECK
════════════════════════════════════════════════════════════
  ICM Status         : RUNNING on EBSPROD01 ✓
  Stuck Requests     : 2 identified — PIDs 14882, 14891
  OS PID Confirm     : Both PIDs confirmed absent on EBSPROD01 ✓
  CONS_BACKUP Schema : Accessible ✓
────────────────────────────────────────────────────────────
  ALL PRE-FLIGHT CHECKS PASSED
════════════════════════════════════════════════════════════
  Creating : CONS_BACKUP.FND_CONCURRENT_REQUESTS_481022
  Rows     : 49 rows backed up (47 pending + 2 stuck) ✓
  Registry : FIX_BACKUP_REGISTRY — ID 7091 created ✓
────────────────────────────────────────────────────────────
  BACKUP COMPLETE
════════════════════════════════════════════════════════════
  Enter case number to confirm : 481022
  Confirmed. Terminating stuck requests via UI...
════════════════════════════════════════════════════════════

Knowledge Base

Every Execution Produces a Record

The knowledge base article is generated automatically from the script's execution output. No manual documentation required. It becomes the institutional record — for the team, for auditors, and for every future engagement in the same environment.

Zero Manual Effort

Every field — environment, tables, before/after values, backup reference, root cause, prevention — is generated from actual execution output. Nothing is written by hand.

Patterns Surface Over Time

The first engagement produces findings. The third produces patterns. Recurring conditions that are invisible as individual incidents become obvious as knowledge base trends.

Survives Staff Turnover

The knowledge base is an institutional record of the Oracle environment. A new manager, a new DBA, or an external auditor can read exactly what happened, what was done, and what prevents recurrence.

KB-SYS-481022-001

Payables Manager Work Shift Expired — 47 Requests Queued, 2 Stuck Requests

EBS R12.2.10 · System / Technical

● RESOLVED

Symptom

Root Cause

Tables

FND_CONCURRENT_MANAGERS — PAYABLES_MGR: RUNNING_PROCESSES=0, MAX_PROCESSES=4
FND_CONCURRENT_REQUESTS — REQUEST_ID 8841201, 8841198: PHASE_CODE=R, STATUS_CODE=R (stuck)

Fix Applied

Prevention

R12 Guide (PDF)	Title & Chapter	Detail
120apug.pdf	Oracle Payables User's Guide — Ch. 3: Invoice Validation Program	pp. 3-171 to 3-172: Submitting Invoice Validation as a concurrent program
120funmo.pdf	Oracle Applications Multiple Organizations Implementation Guide	FND concurrent program setup and manager configuration for multi-org environments
120poug.pdf	Oracle Purchasing User's Guide — Ch. 1: Setting Up	Concurrent program submission requirements for PO-related background processes

Concurrent Manager Health Diagnostic

Why This Fails — and What It Costs

What This Script Diagnoses

What the Report Looks Like

The Four-Layer Architecture in SYS-01

Backup & Rollback for SYS-01

Tables Backed Up — SYS-01

Pre-Flight Safety Guards

FIX_BACKUP_REGISTRY Entry

Auto-Generated Knowledge Base Article

What This Script Finds

Work Shift Gap — Zero Active Workers

Stuck Requests Holding Worker Slots

Repeated Program Failure

Parallel Manager Node Unavailable

Tables Examined

How Every Fix Decision Is Made

Nothing Executes Without a Safety Net

Every Execution Produces a Record

Oracle Documentation References

Ready to Resolve This in Your Environment?

Why This Fails — and What It Costs

What This Script Diagnoses

What the Report Looks Like

The Four-Layer Architecture in SYS-01

Backup & Rollback for SYS-01

Tables Backed Up — SYS-01

Pre-Flight Safety Guards

FIX_BACKUP_REGISTRY Entry

Auto-Generated Knowledge Base Article

What This Script Finds

Work Shift Gap — Zero Active Workers

Stuck Requests Holding Worker Slots

Repeated Program Failure

Parallel Manager Node Unavailable

Tables Examined

How Every Fix Decision Is Made

Nothing Executes Without a Safety Net

Every Execution Produces a Record

Oracle Documentation References

Related Diagnostic Scripts

Workflow Engine Diagnostic

Database Performance Snapshot

Interface & Open Tables Audit

Ready to Resolve This in Your Environment?