This blog post was originally released on Pythian blog

Concurrent processing is one of the key elements of any e-Business Suite system. It provides scheduling and queueing functionality for background jobs and it’s used by most of the applications modules. As many things depend on concurrent processing it’s important to make sure the configuration is tuned for your requirements and hardware specification.

This is the first article in a series about performance of concurrent processing. We’ll take a closer look at the internals of concurrent managers, the settings that affect their performance and the ways of diagnosing performance and configuration issues. Today we’ll start with an overview of the internal workflow of a concurrent manager process. Enjoy the reading.

There are few things that need to be clear before we start:

  1. This is only about how the concurrent processing framework (the concurrent managers) works and it’s not about the concurrent requests executed in the framework.
  2. There are multiple types of concurrent managers in eBS - internal manager, conflict resolution manager, workflow agent listener service, standard manager, etc. Their roles are different and in this post I’ll discuss only managers that pick up scheduled concurrent requests from the “queue” and execute them - or specifically all managers that have type of “Concurrent Manager” set in Concurrent managers’ definition form. Typical examples of these managers are “Standard Manager”, “Inventory Manager” and probably your own custom concurrent managers created for processing specific types of concurrent requests.
Concurrent Managers Form

Concurrent Managers Form

It is important to understand the internal workflow of a concurrent manager because otherwise it’s hard to realize how a configuration change actually affects the system. Several years ago I had to implement an online change of a specialization rule and it triggered a bounce of all Standard Manager processes - that’s when I realized I had to understand how it worked and since then I have spent lots of hours looking into internals of concurrent managers. I’m not saying everything is 100% clear for me now - there are too many little things that matter in certain situations. This series of posts will be more about concepts and I hope you’ll find it useful.

So how does a concurrent manager process work? Here is a diagram I created to explain it:

Internal workflow of a concurrent manager process

Internal workflow of a concurrent manager process

I’ve numbered each step of the diagram to provide a bit more details about them:

  1. This is where the story begins. There is no EXIT state in the diagram as the managers normally process requests in an infinite loop. Obviously, there is a way for a concurrent manager process to receive the command to quit when the managers need to be shut down, but that’s not included here for simplicity.
  2. Internal Concurrent Manager (ICM) requests the Service Manager (FNDSM) to start up the Concurrent Manager process. For the Standard Manager processes the binary executable FNDLIBR is started, for the Inventory Manager it’s INVLIBR, and there are others too.
  3. The manager process connects to the database and reads the settings (e.g profile options, sleep seconds, cache size).
  4. The process saves information about itself in FND_CONCURRENT_PROCESSES table (os process id, database name, instance name, DB session identifiers, logfile path and name and others). It also updates FND_CONCURRENT_QUEUES by increasing the value of RUNNING_PROCESSES.
  5. The concurrent manager process collects information from the database to build the SQL for querying the FND_CONCURRENT_REQUESTS table. The query will be used every time the manager process will look for scheduled concurrent requests. This is the only time the manager process reads the Specialization Rules (which programs it is allowed to execute and which not) from the database. Keep in mind, if the specialization rules are changed while the managers are running, they are bounced without warning as that is the only way to update the specialization rules cached by the manager process.
  6. The SQL (from step 4) is executed to collect information about pending concurrent requests from FND_CONCURRENT_REQUESTS table.
  7. The results are checked to verify if any requests are pending for execution.
  8. If no requests are pending for execution the manager process sleeps and then goes to step 5. The “Sleep Seconds” parameter of the “Work Shifts” settings of the concurrent manager determines how long the process sleeps before FND_CONCURRENT_REQUESTS table is queried again. This is the only time the “sleep seconds” setting is used.
  9. If there is at least one concurrent request pending for execution the concurrent manager process caches rowids for the FND_CONCURRENT_REQUESTS rows of pending concurrent requests. The “Cache Size” setting of the concurrent manager specifies how many rowids to cache.
  10. The cached list of rowids is checked to verify if there are any unprocessed concurrent requests (rows in FND_CONCURRENT_REQUESTS table) left .If none are left - the processing returns to step 5 and the FND_CONCURRENT_REQUESTS table is queried again.
  11. The next unprocessed rowid is picked from the process cache and the processing starts.
  12. Concurrent manager process executes a SELECT-for-UPDATE statement to lock the STATUS_CODE in FND_CONCURRENT_PROCESSES for the request it’s about to process. This is the mechanism to ensure that each concurrent request is executed only once and only by one manager process even if many processes are running simultaneously. The SELECT-for-UPDATE statement can complete with “ORA-00054: resource busy and acquire with NOWAIT specified” or “0 rows updated” if another manager process has started processing the request already.
  13. If the STATUS_CODE of the request was locked successfully the concurrent manager executes the concurrent request. The processing moves to step 9 where the cached list of concurrent requests (rowids) is being checked again.

The workflow is not very complex, but it’s important to remember there are normally multiple concurrent manager processes running at the same time and they are competing for the requests to run. This competition introduces another dimension of tuning for settings, like number of concurrent manager processes, sleep seconds or cache size. Stay tuned for the next post in the series to find out more!