SYS-02: Workflow Engine Diagnostic

Oracle Workflow is the process automation engine underneath Oracle EBS — PO approvals, AP invoice holds, AR collections, HR self-service actions, and dozens of other business processes are all orchestrated by the Workflow engine through activity transitions stored in WF_ITEM_ACTIVITY_STATUSES and WF_ITEMS. When the Workflow engine stalls — because the Background Engine is not running, the Notification Mailer is down, or an activity has errored and blocked the transition — the business process it controls stops completely. Users see approvals that never arrive, notifications that are never sent, and statuses that never change.

The Background Engine is the heartbeat of Oracle Workflow. It processes deferred activities (activities set to run asynchronously rather than inline) and timed-out activities (activities that have exceeded their timeout threshold). When the Background Engine is not running as a concurrent program, deferred activities accumulate in WF_DEFERRED and never execute. This produces the symptom of a workflow that appears to be running — the item exists, it has a current activity — but the activity never transitions because it is waiting for the Background Engine to pick it up.

Stuck workflow items are the most operationally impactful Workflow failure. A workflow item in ERROR status has an activity that threw an unhandled exception. The item is frozen at the error point — it will not transition forward or backward without intervention. Depending on the item type, a stuck item blocks a PO approval, prevents an invoice hold from being released, or stops a customer account from being activated. SYS-02 identifies all items in ERROR status across all item types, the specific activity that errored, and the error message stored in WF_ITEM_ACTIVITY_STATUSES.RESULT_CODE.

SYS-02 runs a complete Workflow engine health diagnostic — Background Engine running status, agent listener status for inbound notifications, Notification Mailer status, deferred activity queue depth, item-level ERROR status analysis across all item types, stuck item duration, and notification aging for items awaiting response.

Engine Services

Background Engine running status and last execution time. Agent Listener status for inbound message processing. Notification Mailer health — outbound email delivery confirmation. Three services must all be running for the Workflow engine to function.

Deferred Queue

WF_DEFERRED queue depth and oldest activity age. Timed-out activity count. Identifies whether the Background Engine is keeping pace with activity volume or falling behind — a growing queue indicates the engine needs more frequent scheduling or higher activity thresholds.

ERROR Item Analysis

All items in ERROR status across every active item type — POAPPRV, APINV, HRSSA, and others. Error message from WF_ITEM_ACTIVITY_STATUSES per item. Duration in ERROR status. Identifies the upstream data condition causing each error before retry.

Notification Health

Notification aging by item type — identifies notifications awaiting response beyond the expected approval time. Notifications in inactive FND_USER queues. Escalation rule configuration review for notifications that should have auto-escalated.

════════════════════════════════════════════════════════════
  ORACLE EBS R12 — WORKFLOW ENGINE DIAGNOSTIC
════════════════════════════════════════════════════════════
  Environment        : PROD
  Report Date        : 24-FEB-2026 14:30:22
  Case Number        : SYS-492810
════════════════════════════════════════════════════════════

[ SECTION 1 — BACKGROUND ENGINE ]          STATUS: ✗ NOT RUNNING
────────────────────────────────────────────────────────────
  ✗ FAIL: Background Engine not running — last run 6h 14m ago
  Deferred Queue     : 284 activities in WF_DEFERRED ✗
  Timed Out Queue    : 12 timed-out activities ⚠

[ SECTION 2 — NOTIFICATION MAILER ]        STATUS: ✓ RUNNING
────────────────────────────────────────────────────────────
  Workflow Notification Mailer — RUNNING ✓
  Outbound: 18 notifications sent in last hour ✓

[ SECTION 3 — ITEMS IN ERROR ]             STATUS: ✗ ERRORS
────────────────────────────────────────────────────────────
  ✗ 7 items in ERROR across 3 item types
  POAPPRV (PO Approval)  : 4 items — activity APPROVE_PO errored ✗
  APINV (AP Invoice)     : 2 items — activity RELEASE_HOLD errored ✗
  HRSSA (HR Self-Service) : 1 item — activity NOTIFY_MANAGER errored ✗
  Oldest Error       : POAPPRV item WF-PO-2026-8841 — 3 days 4 hours

[ SECTION 4 — NOTIFICATIONS AGING ]        STATUS: ⚠ AGING
────────────────────────────────────────────────────────────
  ⚠ 14 notifications awaiting response > 48 hours
  ⚠ 3 notifications sent to inactive FND_USER accounts

════════════════════════════════════════════════════════════
  DIAGNOSTIC SUMMARY
════════════════════════════════════════════════════════════
  Background Engine stopped — 284 deferred activities queued
  7 workflow items in ERROR — 4 PO approvals, 2 AP hold releases blocked
  FIX: Restart Background Engine, retry ERROR items after root cause fixed
════════════════════════════════════════════════════════════
  Backup Created  : CONS_BACKUP.WF_ITEM_ACTIVITY_STATUSES_492810 ✓
  Registry ID     : 7108
════════════════════════════════════════════════════════════

Diagnostic Engine

Runs 35+ checks across Background Engine running status, agent listener status, Notification Mailer health, deferred activity queue depth in WF_DEFERRED, item-level ERROR status across all active item types, stuck item duration, notification aging, and notifications sent to inactive FND_USER accounts.

Backup Created

Before any workflow item is retried or corrected, CONS_BACKUP.WF_ITEM_ACTIVITY_STATUSES_<case#> and WF_ITEMS_<case#> are created. WF tables (WF_ITEMS, WF_ITEM_ACTIVITY_STATUSES) are treated as read-only — corrections are made through the Workflow Administrator interface, never by direct DML on core WF tables.

Guided Data Fix

ERROR items are retried via Workflow Administrator > Workflow Monitor > Retry. Deferred activities are processed by restarting the Background Engine concurrent program. Notification reassignments go through the Notifications inbox reassign function. WF_NOTIFICATIONS status corrections are the only permitted direct fix.

KB Article Generated

Complete KB article generated — item type, item key, error activity, error message, resolution path, Background Engine restart time, queue clearance time. Upload directly to your knowledge base.

KB-SYS-492810-001 · Script: SYS-02

Background Engine Stopped — 284 Deferred Activities Queued, 7 Items in ERROR

Module: System / Technical
Platform: EBS R12.2.10
Status: RESOLVED

Symptom

Background Engine concurrent program not running — last completed 6h 14m ago. 284 activities in WF_DEFERRED queue. 7 workflow items in ERROR across POAPPRV, APINV, HRSSA item types. 4 PO approvals and 2 AP invoice hold releases blocked.

Root Cause

Background Engine concurrent program was terminated as part of a planned maintenance window on 23-FEB-2026 but was not restarted when maintenance completed. Deferred activities accumulated over 6 hours. 7 items that errored before the maintenance window were not identified and retried.

Tables Affected

WF_ITEM_ACTIVITY_STATUSES — STATUS=ERROR, ERROR_MESSAGE: 7 rows across POAPPRV (4), APINV (2), HRSSA (1)
FND_CONCURRENT_REQUESTS — Background Engine: last COMPLETE at 08:16, no subsequent submission

Fix Applied

Background Engine restarted via System Administrator > Concurrent > Run. 284 deferred activities processed over 22 minutes. 7 ERROR items diagnosed — POAPPRV errors due to inactive approver accounts (same root cause as SYS-492810). Items retried via Workflow Monitor after approver accounts corrected. All 7 items completed successfully.

Prevention

Background Engine added to maintenance window restart checklist. Monitoring alert configured — if Background Engine has not run for > 2 hours, alert sent to DBA team. Workflow ERROR item count check added to daily operations review.

Background Engine Not Running

The most common cause of broad Workflow stalling. Deferred activities accumulate in WF_DEFERRED and never execute. Typically missed after maintenance windows or server restarts where the restart checklist is incomplete. Queue clears automatically once the engine is restarted.

Error

Workflow Items Stuck in ERROR

An unhandled exception in a workflow activity freezes the item. PO approvals, invoice hold releases, and HR actions all stop. The error message in WF_ITEM_ACTIVITY_STATUSES identifies the root cause — typically an invalid approver, a missing setup, or a data condition that must be corrected before retry.

Notification

Notifications to Inactive Users

Approval notifications sent to FND_USER accounts that have been end-dated or suspended. The notification exists and is counted as delivered but no one can respond to it. Most common when employees leave without workflow reassignment configuration being updated.

Mailer

Notification Mailer Down

Users receive no email notifications — approvals appear to be pending but no request has arrived. Workflow items are actually processing correctly and notifications exist in WF_NOTIFICATIONS, but no email is being sent. Resolved by restarting the Mailer concurrent program.

Table	Module	Purpose
WF_ITEMS	WF	Workflow item instances — item type, item key, status
WF_ITEM_ACTIVITY_STATUSES	WF	Activity-level status — ERROR, COMPLETE, DEFERRED, NOTIFIED
WF_NOTIFICATIONS	WF	Notification records — recipient, status, sent date
WF_DEFERRED	WF	Deferred activity queue — processed by Background Engine
WF_NOTIFICATION_ATTRIBUTES	WF	Notification content — message text, response options
FND_USER	FND	User account active status — cross-reference for notification routing
FND_CONCURRENT_REQUESTS	FND	Background Engine and Mailer concurrent program run history
WF_ITEM_TYPES	WF	Item type definitions — for ERROR item categorization

Decision Framework

How Every Fix Decision Is Made

Before any data is modified in a production Oracle database, AP-01 walks through a four-stage decision process. Every condition identified by the diagnostic maps to exactly one resolution path.

First Option — Always

Can the front end fix this?

Oracle's own forms and concurrent programs are always the first option. If the condition can be corrected through Oracle's standard UI — a form, a concurrent program, a setup screen — that path is taken first. No consultant SQL required, no database risk, and the fix is fully supported by Oracle. The diagnostic identifies these conditions explicitly and states the exact front-end navigation path.

✓ Functional Fix

When Front End Is Not Available

Is a direct data fix safe to apply?

When the front-end path is unavailable or would require an unacceptable volume of manual steps, a direct fix is evaluated against explicit criteria. All of the following must be true before proceeding:

→The fix mirrors exactly what Oracle's own code would have done if the underlying condition were corrected

→All dependent tables have been identified and will be updated in the same transaction

→The fix is fully reversible — a single INSERT from the backup table restores the original state

→The environment patch level has been confirmed against the fix logic version

⚡ Direct Fix

Hard Stops — No Exceptions

Does this require Oracle Support?

Certain table areas are never touched directly, regardless of how well the underlying structure is understood. The diagnostic flags these conditions and generates the Service Request documentation:

✗XLA subledger tables — XLA_EVENTS, XLA_AE_HEADERS — incorrect changes corrupt the subledger audit trail in ways undetectable until period close fails or an auditor requests a reconciliation

✗Workflow engine tables — WF_ITEMS, WF_ITEM_ACTIVITY_STATUSES — ad-hoc DML can corrupt the workflow engine state instance-wide

✗Recurring conditions after a fix — indicates a code defect, not a data error. Documented and submitted as a Service Request with a reproducible test case

⚠ Oracle Support

For All Approved Direct Fixes

Backup → Execute → Verify → Document

A timestamped backup table is created and row-count verified before the first UPDATE fires. Explicit parameter confirmation is required — the script will not self-execute. After execution, a verification query confirms the expected state. A complete change record — rows affected, before and after values, database username, timestamp — is written to the FIX_BACKUP_REGISTRY and becomes the primary artifact in the knowledge base entry for this incident.

📋 Documented

Condition Identified	Resolution Path	Notes
Background Engine not running — deferred queue backing up	Functional First	Submit the Background Engine concurrent program via System Administrator > Concurrent > Run. SYS-02 identifies the last run time, the queue depth, and the program parameters to use for resubmission.
Workflow item in ERROR — activity threw an exception	Functional First	Retry the item via Workflow Administrator > Workflow Monitor after the root cause of the error is corrected. SYS-02 identifies the specific activity, the error message, and the upstream data condition to fix before retrying.
Notification sent to inactive FND_USER — response never comes	Functional First	Reassign the notification to an active user via the Notifications inbox Reassign function. SYS-02 identifies notifications in inactive user queues and the appropriate active user to reassign to based on the item context.
Notification Mailer not running — no email notifications sent	Functional First	Restart the Notification Mailer via Workflow Administrator > Notification Mailer. SYS-02 identifies the mailer status and the last successful notification send time.
Agent listener not running — inbound notifications not processed	Functional First	Restart the Agent Listener concurrent program. SYS-02 identifies the listener status and the inbound notification queue depth that has accumulated.
WF_DEFERRED queue depth excessive — engine cannot clear backlog	Functional First	Increase the Background Engine concurrent program parameters — specifically the number of activities per run and the threshold for deferred vs inline execution. SYS-02 identifies the queue depth and the estimated clearance time at the current throughput rate.
Notification aging > configured timeout	Direct Fix	SYS-02 corrects the NOTIFICATION_ID expiration date in WF_NOTIFICATIONS with full backup when a notification should not have expired but did due to incorrect timeout configuration. Standard path is Workflow Administrator > notification timeout adjustment.
WF_PURGE needed — completed item accumulation degrading performance	Functional First	Submit the Purge Obsolete Workflow Runtime Data concurrent program for completed items older than the retention threshold. SYS-02 identifies the count of purgeable items and the safe age threshold based on audit requirements.

Safeguards

Nothing Executes Without a Safety Net

Before any data fix runs, the script verifies pre-flight conditions and creates a complete verified backup. If any check fails, the script aborts. There is no partial execution path.

Pre-Flight Checks — All Must Pass

✓

POSTED_FLAG = 'N'

Will not modify a posted transaction under any circumstances

✓

ACCOUNTING_EVENT_ID IS NULL

Will not modify rows with an active or pending XLA accounting event

✓

No active database lock

Checks V$LOCK — will not proceed if another session holds a lock on target rows

✓

CONS_BACKUP schema writable

Backup schema must be accessible before any DML is permitted

✓

Explicit confirmation required

Script outputs a confirmation prompt — DML does not execute until consultant enters the case number as the execution parameter

Backup Methodology

Before the first UPDATE fires, the script creates a complete copy of every row to be modified. Tables are named CONS_BACKUP.<TABLE>_<CASE#> and persist permanently after execution.

WF_ITEMS

WF_ITEM_ACTIVITY_STATUSES

WF_NOTIFICATIONS

Row count verified after backup creation. If backup fails for any reason the script aborts without executing any DML. Rollback is a single INSERT from the backup table. No reconstruction required.

SYS-02 — Pre-Flight & Backup Verification

════════════════════════════════════════════════════════════
  PRE-FLIGHT SAFETY CHECK
════════════════════════════════════════════════════════════
  Background Engine  : NOT RUNNING — last run 6h 14m ago ✗
  WF_DEFERRED Count  : 284 activities
  Items in ERROR     : 7 items — POAPPRV, APINV, HRSSA
  CONS_BACKUP Schema : Accessible ✓
  WF Tables          : Read-only — no DML on WF_ITEMS or WF_ITEM_ACTIVITY_STATUSES ✓
────────────────────────────────────────────────────────────
  ALL PRE-FLIGHT CHECKS PASSED
════════════════════════════════════════════════════════════
  Creating : CONS_BACKUP.WF_ITEM_ACTIVITY_STATUSES_492810
  Rows     : 7 ERROR rows backed up ✓
  Creating : CONS_BACKUP.WF_NOTIFICATIONS_492810
  Rows     : 17 aging notifications backed up ✓
  Registry : FIX_BACKUP_REGISTRY — ID 7108 created ✓
────────────────────────────────────────────────────────────
  BACKUP COMPLETE — Restart Background Engine, then retry ERROR items
════════════════════════════════════════════════════════════

Knowledge Base

Every Execution Produces a Record

The knowledge base article is generated automatically from the script's execution output. No manual documentation required. It becomes the institutional record — for the team, for auditors, and for every future engagement in the same environment.

Zero Manual Effort

Every field — environment, tables, before/after values, backup reference, root cause, prevention — is generated from actual execution output. Nothing is written by hand.

Patterns Surface Over Time

The first engagement produces findings. The third produces patterns. Recurring conditions that are invisible as individual incidents become obvious as knowledge base trends.

Survives Staff Turnover

The knowledge base is an institutional record of the Oracle environment. A new manager, a new DBA, or an external auditor can read exactly what happened, what was done, and what prevents recurrence.

KB-SYS-492810-001

Background Engine Stopped — 284 Deferred Activities Queued, 7 Items in ERROR

EBS R12.2.10 · System / Technical

● RESOLVED

Symptom

Root Cause

Tables

WF_ITEM_ACTIVITY_STATUSES — STATUS=ERROR, ERROR_MESSAGE: 7 rows across POAPPRV (4), APINV (2), HRSSA (1)
FND_CONCURRENT_REQUESTS — Background Engine: last COMPLETE at 08:16, no subsequent submission

Fix Applied

Prevention

R12 Guide (PDF)	Title & Chapter	Detail
120poug.pdf	Oracle Purchasing User's Guide — Ch. 1: Choosing Workflow Options	pp. 1-83 to 1-88: Workflow options for PO and requisition approval routing
120apug.pdf	Oracle Payables User's Guide — Ch. 3: Invoice Approval Workflow	pp. 3-172 to 3-190: Invoice approval workflow program and stuck activity resolution
120funmo.pdf	Oracle Applications Multiple Organizations Implementation Guide	Workflow business event system and operating unit context for multi-org workflows

Workflow Engine Diagnostic

Why This Fails — and What It Costs

What This Script Diagnoses

What the Report Looks Like

The Four-Layer Architecture in SYS-02

Backup & Rollback for SYS-02

Tables Backed Up — SYS-02

Pre-Flight Safety Guards

FIX_BACKUP_REGISTRY Entry

Auto-Generated Knowledge Base Article

What This Script Finds

Background Engine Not Running

Workflow Items Stuck in ERROR

Notifications to Inactive Users

Notification Mailer Down

Tables Examined

How Every Fix Decision Is Made

Nothing Executes Without a Safety Net

Every Execution Produces a Record

Oracle Documentation References

Ready to Resolve This in Your Environment?

Why This Fails — and What It Costs

What This Script Diagnoses

What the Report Looks Like

The Four-Layer Architecture in SYS-02

Backup & Rollback for SYS-02

Tables Backed Up — SYS-02

Pre-Flight Safety Guards

FIX_BACKUP_REGISTRY Entry

Auto-Generated Knowledge Base Article

What This Script Finds

Background Engine Not Running

Workflow Items Stuck in ERROR

Notifications to Inactive Users

Notification Mailer Down

Tables Examined

How Every Fix Decision Is Made

Nothing Executes Without a Safety Net

Every Execution Produces a Record

Oracle Documentation References

Related Diagnostic Scripts

Concurrent Manager Health

PO Approval Workflow Diagnostic

AP Invoice Validation Diagnostic

Ready to Resolve This in Your Environment?