Maris Elsins’ blog

Achieving Rock-solid Maintenance with Template-based Action Plans

2019-02-07T13:05:00-06:00

A while ago, I created and started using a tool that helps advancing the maintenance action plans from one testing iteration to another, and finally - to the maintenance activity in production reliably and without introduction of copy/paste or text replacement mistakes. It’s main idea is to allow creating action plan templates that have placeholders instead of environment-specific values, and a spreadsheet that defines all the different values for these placeholders. The final action plan is created automatically with a single click of a button. Interested in doing the same? Read more here on Pythian blog!

Just a Serverless Blog Experiment!

2019-01-10T08:46:52-06:00

This is not really a propper blog post. I’m just experimenting with serverless blog ideas on GCP.

This page is served from GCP App Engine.

The Code for the blogs are stored in a private Google Source Repository and the conversion to static HTML is done by running jekyll build in a docker container that’s initiated with a Cloud Build trigger on each commit to the master branch of the repository. The trigger also redeploys the application automatically to App Engine.

If you see this page, it means it worked fine :)

The getMOSPatch V2 is Here!!!

2016-07-08T07:46:52-05:00

Just wanted to let you know that I’ve released the 2nd version of getMOSPatch - a tool for downloading Oracle patches directly from My Oracle Support to your server without having to visit the support site. It’s now written in java and thus it’s platform independent! Read more about it on my Pythian blog post here, and get it directly from GitHub here!

getMOSPatch

Simple Script for Real-Time Wait Event Monitoring

2016-04-14T17:52:53-05:00

What do you do when you’re dealing with an ongoing performance issue and somebody’s watching over your shoulder and asking something like “Hey, has our commits’ response time become any better by now?”. Or, if you’re observing a spike in IO requests that doesn’t last for too long, and you want to see how much worse the response times of db file sequential read events gets? Writing queries at these moments require time which one doesn’t have, so I’ve written a script that I already use in situations like these

As the title suggests, the script is very simple, it relies on SQL*Plus variables to preserve the retrieved values and uses them to calculate deltas when the query is executed again.

The latest version of the script will always be here: rt_wait_perf.sql

The script contains a single SQL statement and nothing else:

set lines 300 ver off feed off
def v_tst=''
def v_wts='0'
def v_twm='0'
def v_event='&1';
col TIME_WAITED_MICRO for a15 new_value v_twm
col TOTAL_WAITS for a15 new_value v_wts
col TSTAMP for a26 new_value v_tst
col D_WTS for 9999999999999
col D_TWM for 9999999999999
col D_AVG_MS_WT for 999999999.999
col INTERVAL_SECS for 999999999.999
col EVENT for a40
select to_char(systimestamp,'DD.MM.YYYY HH24:MI:SSXFF') TSTAMP,
       round(sysdate+(86400*(systimestamp-to_timestamp('&&v_tst','DD.MM.YYYY HH24:MI:SSXFF')))-sysdate,3) interval_secs,
       event,
       to_char(TOTAL_WAITS) TOTAL_WAITS,
       to_char(TIME_WAITED_MICRO) TIME_WAITED_MICRO,
       TOTAL_WAITS-&&v_wts D_WTS,
       TIME_WAITED_MICRO-&&v_twm D_TWM,
       round((TIME_WAITED_MICRO-&&v_twm)/decode(TOTAL_WAITS-&&v_wts,0,null,TOTAL_WAITS-&&v_wts)/1000,3) D_AVG_MS_WT
from v$system_event
where event='&&v_event';

Here’s how you use it:

It requires one argument - the name of the wait event that you want to monitor, i.e SQL> @rt_wait_perf.sql "log file sync"
Simply execute the SQL statement again (by using /) to get the delta values and milliseconds/wait since the previous execution of the SQL
Keep in mind it shows the info for the current instance only (it queries from v$system_event)
Most of the times you’ll be interested in outputs for “D_WTS” and “D_AVG_MS_WT” columns to see how many times the wait occurred and how long they were during the reference interval.

This is how it works:

SQL> @rt_wait_perf "log file sync"

TSTAMP                      INTERVAL_SECS EVENT           TOTAL_WAITS     TIME_WAITED_MIC       D_WTS         D_TWM D_AVG_MS_WT
-------------------------- -------------- --------------- --------------- --------------- ----------- ------------- -----------
14.04.2016 18:07:25.055984                log file sync   137478504       1096070215896     137478504 1096070215896       7.973
SQL> /

TSTAMP                      INTERVAL_SECS EVENT           TOTAL_WAITS     TIME_WAITED_MIC       D_WTS         D_TWM D_AVG_MS_WT
-------------------------- -------------- --------------- --------------- --------------- ----------- ------------- -----------
14.04.2016 18:07:45.189085         20.133 log file sync   137478672       1096070457766           168        241870       1.440
SQL> /

TSTAMP                      INTERVAL_SECS EVENT           TOTAL_WAITS     TIME_WAITED_MIC       D_WTS         D_TWM D_AVG_MS_WT
-------------------------- -------------- --------------- --------------- --------------- ----------- ------------- -----------
14.04.2016 18:08:00.585761         15.397 log file sync   137478819       1096070666101           147        208335       1.417

Columns reference:

TSTAMP - current timestamp
INTERVAL_SECS - elapsed time in seconds since previous execution. This is the wall clock time interval between the “snapshots” used to calculate the deltas
EVENT - name of the wait event
TOTAL_WAITS - total number of waits
TIME_WAITED_MIC - total time spent on the wait event in microseconds
D_WTS - delta number of waits (since the previous execution)
D_TWM - delta time spent on the wait event in microseconds (since the previous execution)
D_AVG_MS_WT - average milliseconds per wait event (since the previous execution)

Enjoy!

Internals of Querying the Concurrent Requests Queue - Revisited for R12.2

2016-04-11T19:36:00-05:00

Once upon a time I wrote about the Internal Workflow of an E-Business Suite Concurrent Manager Process. Many things have changed since that blog post, from which the most obvious change was the release of Oracle e-Business Suite R12.2. So, I decided to check if the way how the concurrent manager queues were processed by concurrent manager processes were still the same. My main goal was to see if the manager processes still don’t attempt any way of coordination to distribute the requests among them.

This is how I did the testing:

I used the VM templates provided by Oracle to build my R12.2.4 test environment. By the way, I didn’t expect that the process of getting the environment up would be so simple! Downloading the media files from (edelivery.oracle.com) was the most time-consuming step, once done - it took me just 1 hour to un-compress everything, import the Virtual Assembly file and bring up the R12.2.4 environment on my laptop.
3 Standard managers are defined by default
“Sleep seconds” were left as is = 30 seconds
“Cache size” was increased from 1 to 5.
Identified the 3 DB processes that belong to the Standard managers: select sid, serial# from v$session where module='e:FND:cp:STANDARD';
I enabled tracing with binds and waits for each of them like this: exec dbms_monitor.session_trace_enable(sid,serial#,true,true);
Once that was done I submitted one concurrent program - “Active users” and waited for it to complete.
I disabled the tracing like this: exec dbms_monitor.session_trace_disable(sid,serial#);.
Collected the trace files

I found 2 of the trace files to be very interesting. To make things more simple, the manager process “A” will be the one that executed the concurrent request, and process “B” will be the one that didn’t.

Before the “Active Users” request was submitted

No other requests were running at the time I did the testing, so I clearly observed how both Managers A and B queried the FND_CONCURRENT_REQUESTS table BOTH of the trace files displayed the same method of how requests are picked up from the queue. Note, I’m showing only the lines relevant to display the main query only, and I have formatted the query text to make it more readable

PARSING IN CURSOR #139643743645920 len=1149 dep=0 uid=100 oct=3 lid=100 tim=1460211399835915 hv=3722997734 ad='d275f750' sqlid='cd23u4zfyhvz6'
SELECT R.Rowid
FROM Fnd_Concurrent_Requests R
WHERE R.Hold_Flag                             = 'N'
AND R.Status_Code                             = 'I'
AND R.Requested_Start_Date                   <= Sysdate
AND (R.Node_Name1                            IS NULL
OR (R.Node_Name1                             IS NOT NULL
AND FND_DCP.target_node_mgr_chk(R.request_id) = 1))
AND (R.Edition_Name                          IS NULL
OR R.Edition_Name                            <= sys_context('userenv', 'current_edition_name'))
AND EXISTS
  (SELECT NULL
  FROM Fnd_Concurrent_Programs P
  WHERE P.Enabled_Flag         = 'Y'
  AND R.Program_Application_Id = P.Application_Id
  AND R.Concurrent_Program_Id  = P.Concurrent_Program_Id
  AND EXISTS
    (SELECT NULL
    FROM Fnd_Oracle_Userid O
    WHERE R.Oracle_Id = O.Oracle_Id
    AND EXISTS
      (SELECT NULL
      FROM Fnd_Conflicts_Domain C
      WHERE P.Run_Alone_Flag = C.RunAlone_Flag
      AND R.CD_Id            = C.CD_Id
      )
    )
  AND (P.Execution_Method_Code                          != 'S'
  OR (R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN ((0,98),(0,100),(0,31721),(0,31722),(0,31757)))
  )
AND ((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN ((510,40032),(510,40033),(510,42156),(510,42157),(530,43793),(530,43794),(535,42626),(535,42627),(535,42628)))
ORDER BY NVL(R.priority, 999999999),
  R.Priority_Request_ID,
  R.Request_ID
END OF STMT
EXEC #139643743645920:c=0,e=33,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=3984653669,tim=1460211399835910
FETCH #139643743645920:c=0,e=546,p=0,cr=106,cu=0,mis=0,r=0,dep=0,og=1,plh=3984653669,tim=1460211399836507
WAIT #139643743645920: nam='SQL*Net message to client' ela= 3 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=1460211399836572

*** 2016-04-09 10:17:09.837
WAIT #139643743645920: nam='SQL*Net message from client' ela= 30000367 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=1460211429836965
...
EXEC #139643743645920:c=0,e=59,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=3984653669,tim=1460211429838767
FETCH #139643743645920:c=0,e=689,p=0,cr=106,cu=0,mis=0,r=0,dep=0,og=1,plh=3984653669,tim=1460211429839587
WAIT #139643743645920: nam='SQL*Net message to client' ela= 4 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=1460211429839652

*** 2016-04-09 10:17:39.840
WAIT #139643743645920: nam='SQL*Net message from client' ela= 30000325 driver id=1952673792 #bytes=1 p3=0 obj#=-1 tim=1460211459840003
...

It’s important to observe that:

All manager’s processes still compete for the same requests. If the query is executed at the same time - the same list of concurrent requests will be retrieved by all processes.
The constants literals used in lines 30-32 mean that the query for checking the queue is still built when the concurrent manager process starts up. These constants are mainly used to implement the specializations rules in the query.
Only rowid for the pending requests’ rows in FND_CONCURRENT_REQUESTS are fetched.
The sleep time is clearly visible on lines 41,42 and 48,50

After the “Active Users” request was submitted - Starting the concurrent request.

The manager process A was the first to pick up the submitted requests and it could be observed by the r=1 (1 row fetched) in the FETCH call for the query we just reviewed: FETCH #139643743645920:c=0,e=437,p=0,cr=113,cu=0,mis=0,r=1,dep=0,og=1,plh=3984653669,tim=1460211519844640

Immediately after that, the manager process A locked the row in FND_CONCURRENT_REQUESTS table, this way, the request got assigned to this process. Notice the similar where predicates used in this query, these are actually required to make sure that the request is still not picked up by another manager process. However the main thing here is the fact that the request row is accessed by the rowid retrieved earlier (row 45, the value of the bind variable :reqname is AAAjnSAA/AAAyn1AAH in this case). Locking of the row is done by the FOR UPDATE OF R.status_code NoWait clause on line 49

PARSING IN CURSOR #139643743640368 len=4530 dep=0 uid=100 oct=3 lid=100 tim=1460211519864113 hv=4239777398 ad='cde86338' sqlid='6ya6bzgybbrmq'
SELECT R.Conc_Login_Id,
  R.Request_Id,
  ... excluded other 156 columns for brevity...
FROM fnd_concurrent_requests R,
  fnd_concurrent_programs P,
  fnd_application A,
  fnd_user U,
  fnd_oracle_userid O,
  fnd_conflicts_domain C,
  fnd_concurrent_queues Q,
  fnd_application A2,
  fnd_executables E,
  fnd_conc_request_arguments X
WHERE R.Status_code             = 'I'
AND (R.Edition_Name            IS NULL
OR R.Edition_Name              <= sys_context('userenv', 'current_edition_name'))
AND R.Request_ID                = X.Request_ID(+)
AND R.Program_Application_Id    = P.Application_Id(+)
AND R.Concurrent_Program_Id     = P.Concurrent_Program_Id(+)
AND R.Program_Application_Id    = A.Application_Id(+)
AND P.Executable_Application_Id = E.Application_Id(+)
AND P.Executable_Id             = E.Executable_Id(+)
AND P.Executable_Application_Id = A2.Application_Id(+)
AND R.Requested_By              = U.User_Id(+)
AND R.Cd_Id                     = C.Cd_Id(+)
AND R.Oracle_Id                 = O.Oracle_Id(+)
AND Q.Application_Id            = :q_applid
AND Q.Concurrent_Queue_Id       = :queue_id
AND (P.Enabled_Flag            IS NULL
OR P.Enabled_Flag               = 'Y')
AND R.Hold_Flag                 = 'N'
AND R.Requested_Start_Date     <= Sysdate
AND ( R.Enforce_Seriality_Flag  = 'N'
OR ( C.RunAlone_Flag            = P.Run_Alone_Flag
AND (P.Run_Alone_Flag           = 'N'
OR NOT EXISTS
  (SELECT NULL
  FROM Fnd_Concurrent_Requests Sr
  WHERE Sr.Status_Code         IN ('R', 'T')
  AND Sr.Enforce_Seriality_Flag = 'Y'
  AND Sr.CD_id                  = C.CD_Id
  ))))
AND Q.Running_Processes                                     <= Q.Max_Processes
AND R.Rowid                                                  = :reqname
AND ((P.Execution_Method_Code                               != 'S'
OR (R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID)       IN ((0,98),(0,100),(0,31721),(0,31722),(0,31757))))
AND ((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN ((510,40032),(510,40033),(510,42156),(510,42157),(530,43793),(530,43794),(535,42626),(535,42627),(535,42628)))
FOR UPDATE OF R.status_code NoWait

The behavior of the manager process B was a little bit more interesting. It too managed to fetch the same rowid from FND_CONCURRENT_PROCESSES table belonging to the submitted “Active Users” processes. However, when it tried to lock the row in FND_CONCURRENT_REQUESTS (By using exactly the same query), this happened:

PARSING IN CURSOR #139690311998256 len=4530 dep=0 uid=100 oct=3 lid=100 tim=1460211519900924 hv=4239777398 ad='cde86338' sqlid='6ya6bzgybbrmq'
...
BINDS #139690311998256:
...
Bind#2
  oacdty=01 mxl=32(18) mxlc=00 mal=00 scl=00 pre=00
  oacflg=20 fl2=1000001 frm=01 csi=873 siz=0 off=64
  kxsbbbfp=7f0c2f713f20  bln=32  avl=18  flg=01
  value='AAAjnSAA/AAAyn1AAH'
EXEC #139690311998256:c=1000,e=1525,p=0,cr=25,cu=1,mis=0,r=0,dep=0,og=1,plh=4044729389,tim=1460211519902727
ERROR #139690311998256:err=54 tim=1460211519902750

The query failed with ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired. This is how the access to pending concurrent requests is serialized to make sure only one of the manager processes can run it. And, I think, relying on the well-tuned and highly efficient locking mechanism of Oracle Database is a very very smart idea.

Conclusions

the coordination between manager processes is still not happening to distribute the requests, but the managers all query the queue the same way and then compete between themselves to lock the requests’ entries on the table 1st. The process that gets the lock also gets to execute the concurrent request.
The cache size variable couldn’t be observed in the trace files, but as far as I remember from my previous research the process would only fetch “cache size”-number of rowids using the 1st query in this post. This could be tested by submitting larger volume of requests simultaneously.
The “sleep seconds” kicks in only when the manager process didn’t fetch any rowids from the queue. After all the cached requests are attempted/executed by the manager process, the queue is checked again immediately without waiting for the “sleep seconds” (Not explained in detail in this post, but it’s revealed in the trace files)
The DMLs used to query the FND_CONCURRENT_REQUESTS and to lock the row are very very similar to Pre-R12.2 releases of e-Business Suite (Another sign that the process hasn’t changed). (One change that I see is the addition of where clause predicates for Checking the Editions)

SQL Profiles vs. SQL Plan Baselines

2016-03-24T20:26:56-05:00

Recently one of my customers asked, what would happen if we had SQL Profile set up for a particular query to make sure it uses the correct execution plan, as well as an existing SQL Plan Baseline for the same query? What would happen if these two features would like the SQL statement to use different execution plans? Which one would win? I hesitated to answer immediately, although the answer seams obvious now. Look how I came to the answer!

I decided to build a quick test case in my 12.1.0.2 lab. The plan is to observe how the preferred execution plan changes after each of these steps:

Create query that uses full table access to query some table, create SQL Plan Baseline for it
Create an index on the table so that it allowed more efficient access to the data
Observe the new baseline for indexed access being created, evolve it to get it accepted
Create SQL Profile to make the full table access favored
Remove the SQL Plan Baseline that allows the full table access
Remove the SQL Plan Baseline for the indexed access

The Test

This is the table I’m building the test case on:

SQL> desc all_ash
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 DAT                                                DATE
 MODULE                                             VARCHAR2(64)
 PROGRAM                                            VARCHAR2(64)
 ACTION                                             VARCHAR2(64)
 EVENT                                              VARCHAR2(64)
 MACHINE                                            VARCHAR2(64)
 CNT                                                NUMBER

And here’s a very simple query that I’ll be using!

SQL> select count(*) from all_ash where module='TEST';

  COUNT(*)
----------
         0

SQL> explain plan for select count(*) from all_ash where module='TEST'

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------
Plan hash value: 3460552988

------------------------------------------------------------------------------
| Id  | Operation          | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |         |     1 |    17 |   189   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |         |     1 |    17 |            |          |
|*  2 |   TABLE ACCESS FULL| ALL_ASH |     1 |    17 |   189   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):

   2 - filter('MODULE'='TEST')

Creating the SQL Plan Baseline for the query using the Full Table Scan. I’ll load the Plan in to the SQL Management Base from the cursor cache and then will verify if the baseline is there:

SQL> select sql_id, sql_text, plan_hash_value from v$sql where plan_hash_value=3460552988 and sql_text like 'select%';

SQL_ID
-------------
SQL_TEXT
--------------------------------------------------------------------------------
PLAN_HASH_VALUE
---------------
9kt35fkxmk1qw
select count(*) from all_ash where module='TEST'
     3460552988


SQL> var n number
SQL> exec :n:=DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE('9kt35fkxmk1qw',3460552988);

PL/SQL procedure successfully completed.

SQL> print n

         N
----------
         1

SQL> explain plan for select count(*) from all_ash where module='TEST';

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------
Plan hash value: 3460552988

------------------------------------------------------------------------------
| Id  | Operation          | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |         |     1 |    17 |   189   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |         |     1 |    17 |            |          |
|*  2 |   TABLE ACCESS FULL| ALL_ASH |     1 |    17 |   189   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):

   2 - filter('MODULE'='TEST')

Note
-----
   - SQL plan baseline 'SQL_PLAN_6bnp3kg5khwxa275ca563' used for this statement

The SQL Plan Baseline SQL_PLAN_6bnp3kg5khwxa275ca563 is now in place for the plan using TABLE ACCESS FULL. Let’s create the index, after which we’ll verify it’s not used immediately (because we have the SQL Plan Baseline), but then we’ll evolve the Baseline to start using the index

SQL> create index ix_all_ash_mod on all_ash(module);

Index created.

SQL> explain plan for select count(*) from all_ash where module='TEST';

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 3460552988

------------------------------------------------------------------------------
| Id  | Operation          | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |         |     1 |    17 |   189   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |         |     1 |    17 |            |          |
|*  2 |   TABLE ACCESS FULL| ALL_ASH |     1 |    17 |   189   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------

   2 - filter('MODULE'='TEST')

Note
-----
   - SQL plan baseline 'SQL_PLAN_6bnp3kg5khwxa275ca563' used for this statement

18 rows selected.

SQL> select count(*) from all_ash where module='TEST';

  COUNT(*)
----------
         0

SQL> select sql_handle from dba_sql_plan_baselines where plan_name='SQL_PLAN_6bnp3kg5khwxa275ca563';

SQL_HANDLE
--------------------------------------------------------------------------------------------------------------------------------
SQL_65d2a393cb2873aa

1 rows selected.

SQL> col r for a200
SQL> set lines 200
SQL> set pages 50000
SQL> set long 999999
SQL> var r clob
SQL> exec :r:=DBMS_SPM.evolve_sql_plan_baseline(sql_handle => 'SQL_65d2a393cb2873aa');

PL/SQL procedure successfully completed.

SQL> print r

R
---------------------------------------------------------------------------------------------
GENERAL INFORMATION SECTION
---------------------------------------------------------------------------------------------

 Task Information:
 ---------------------------------------------
 Task Name            : TASK_83
 Task Owner           : C##MARIS
 Execution Name       : EXEC_115
 Execution Type       : SPM EVOLVE
 Scope                : COMPREHENSIVE
 Status               : COMPLETED
 Started              : 03/23/2016 07:50:34
 Finished             : 03/23/2016 07:50:35
 Last Updated         : 03/23/2016 07:50:35
 Global Time Limit    : 2147483646
 Per-Plan Time Limit  : UNUSED
 Number of Errors     : 0
---------------------------------------------------------------------------------------------

SUMMARY SECTION
---------------------------------------------------------------------------------------------
  Number of plans processed  : 1
  Number of findings         : 2
  Number of recommendations  : 1
  Number of errors           : 0
---------------------------------------------------------------------------------------------

DETAILS SECTION
---------------------------------------------------------------------------------------------
 Object ID          : 2
 Test Plan Name     : SQL_PLAN_6bnp3kg5khwxa81486eb4
 Base Plan Name     : SQL_PLAN_6bnp3kg5khwxa275ca563
 SQL Handle         : SQL_65d2a393cb2873aa
 Parsing Schema     : C##MARIS
 Test Plan Creator  : C##MARIS
 SQL Text           : select count(*) from all_ash where module='TEST'

Execution Statistics:
-----------------------------
                    Base Plan                     Test Plan
                    ----------------------------  ----------------------------
 Elapsed Time (s):  .000302                       .000003
 CPU Time (s):      .000289                       0
 Buffer Gets:       67                            0
 Optimizer Cost:    189                           1
 Disk Reads:        0                             0
 Direct Writes:     0                             0
 Rows Processed:    0                             0
 Executions:        10                            10


FINDINGS SECTION
---------------------------------------------------------------------------------------------

Findings (2):
-----------------------------
 1. The plan was verified in 0.06000 seconds. It passed the benefit criterion
    because its verified performance was 338.14440 times better than that of
    the baseline plan.
 2. The plan was automatically accepted.

Recommendation:
-----------------------------
 Consider accepting the plan.


EXPLAIN PLANS SECTION
---------------------------------------------------------------------------------------------

Baseline Plan
-----------------------------
 Plan Id          : 203
 Plan Hash Value  : 660383075

-------------------------------------------------------------------------
| Id  | Operation            | Name    | Rows | Bytes | Cost | Time     |
       prompt 'Enter value for force_matching (FALSE): ' -
-------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |         |    1 |    17 |  189 | 00:00:01 |
|   1 |   SORT AGGREGATE     |         |    1 |    17 |      |          |
| * 2 |    TABLE ACCESS FULL | ALL_ASH |    1 |    17 |  189 | 00:00:01 |
-------------------------------------------------------------------------

Predicate Information (identified by operation id):
------------------------------------------
* 2 - filter('MODULE'='TEST')


Test Plan
-----------------------------
 Plan Id          : 204
 Plan Hash Value  : 2169007796

-------------------------------------------------------------------------------
| Id  | Operation           | Name           | Rows | Bytes | Cost | Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT    |                |    1 |    17 |    1 | 00:00:01 |
|   1 |   SORT AGGREGATE    |                |    1 |    17 |      |          |
| * 2 |    INDEX RANGE SCAN | IX_ALL_ASH_MOD |    1 |    17 |    1 | 00:00:01 |
-------------------------------------------------------------------------------

Predicate Information (identified by operation id):
------------------------------------------
* 2 - access('MODULE'='TEST')

---------------------------------------------------------------------------------------------


SQL> explain plan for select count(*) from all_ash where module='TEST';

Explained.

SQL> select * from table(dbms_xplan.display)
  2  ;

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 2623139211

------------------------------------------------------------------------------------
| Id  | Operation         | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |                |     1 |    17 |     1   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE   |                |     1 |    17 |            |          |
|*  2 |   INDEX RANGE SCAN| IX_ALL_ASH_MOD |     1 |    17 |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access('MODULE'='TEST')

Note
-----
   - SQL plan baseline 'SQL_PLAN_6bnp3kg5khwxa81486eb4' used for this statement

So the new plan was accepted and baseline SQL_PLAN_6bnp3kg5khwxa81486eb4 became the preferred one.

In the next step I’ll use create_sql_profile.sql script that I found on this post on Kerry Osborne’s Oracle Blog. The script extracts the outline hints from the execution plan and packages them into a SQL Profile.

SQL> select child_number, plan_hash_value from v$sql where sql_id='9kt35fkxmk1qw';

CHILD_NUMBER PLAN_HASH_VALUE
------------ ---------------
           1      3460552988

SQL> @create_sql_profile.sql
Enter value for sql_id: 9kt35fkxmk1qw
Enter value for child_no (0): 1
Enter value for profile_name (PROF_sqlid_planhash):
Enter value for category (DEFAULT):
Enter value for force_matching (FALSE):
old  19: sql_id = '&&sql_id'
new  19: sql_id = '9kt35fkxmk1qw'
old  20: and child_number = &&child_no
new  20: and child_number = 1
old  27: decode('&&profile_name','X0X0X0X0','PROF_&&sql_id'||'_'||plan_hash_value,'&&profile_name')
new  27: decode('X0X0X0X0','X0X0X0X0','PROF_9kt35fkxmk1qw'||'_'||plan_hash_value,'X0X0X0X0')
old  33: sql_id = '&&sql_id'
new  33: sql_id = '9kt35fkxmk1qw'
old  34: and child_number = &&child_no;
new  34: and child_number = 1;
old  39: category => '&&category',
new  39: category => 'DEFAULT',
old  41: force_match => &&force_matching
new  41: force_match => false
old  52:   dbms_output.put_line('ERROR: sql_id: '||'&&sql_id'||' Child: '||'&&child_no'||' not found in v$sql.');
new  52:   dbms_output.put_line('ERROR: sql_id: '||'9kt35fkxmk1qw'||' Child: '||'1'||' not found in v$sql.');
SQL>
SQL>
SQL> explain plan for select count(*) from all_ash where module='TEST';

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 3460552988

------------------------------------------------------------------------------
| Id  | Operation          | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |         |     1 |    17 |   189   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |         |     1 |    17 |            |          |
|*  2 |   TABLE ACCESS FULL| ALL_ASH |     1 |    17 |   189   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter('MODULE'='TEST')

Note
-----
   - SQL profile 'PROF_9kt35fkxmk1qw_3460552988' used for this statement
   - SQL plan baseline 'SQL_PLAN_6bnp3kg5khwxa275ca563' used for this statement

19 rows selected.

I think the result above is one of the most interesting ones in this test, because we see both the SQL Profile and the SQL Plan baseline being used by the SQL statement. It’s also interesting to observe that after the profile was created, the SQL Plan Baselines that suggests using the full table scan was preferred, although before that the indexed access was preferred. Does this mean that SQL Profiles are stronger than SQL Plan Baselines? No, because at this moment both of the SQL Plan Baselines are there and the optimizer is allowed to pick any of them. Next, let’s drop SQL Plan Baseline SQL_PLAN_6bnp3kg5khwxa275ca563, and the only remaining baseline would allow the indexed access.

SQL> exec :n:=dbms_spm.drop_sql_plan_baseline(sql_handle=>'SQL_65d2a393cb2873aa',plan_name=>'SQL_PLAN_6bnp3kg5khwxa275ca563');

PL/SQL procedure successfully completed.

SQL> print n

         N
----------
         1

SQL> explain plan for select count(*) from all_ash where module='TEST';

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 2623139211

------------------------------------------------------------------------------------
| Id  | Operation         | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |                |     1 |    17 |     1   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE   |                |     1 |    17 |            |          |
|*  2 |   INDEX RANGE SCAN| IX_ALL_ASH_MOD |     1 |    17 |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access('MODULE'='TEST')

Note
-----
   - SQL profile 'PROF_9kt35fkxmk1qw_3460552988' used for this statement
   - SQL plan baseline 'SQL_PLAN_6bnp3kg5khwxa81486eb4' used for this statement

19 rows selected.

Again, both SQL Profile and the SQL Plan Baseline is reported in the Notes section of the dbms_xplan output, but this time, the index range scan was used although the SQL Profile packages hints that ask the optimizer to use the full table scan. What changes if the remaining SQL Plan Baseline is removed, but the index and the Profile still remains in place?

SQL> exec :n:=dbms_spm.drop_sql_plan_baseline(sql_handle=>'SQL_65d2a393cb2873aa',plan_name=>'SQL_PLAN_6bnp3kg5khwxa81486eb4');

PL/SQL procedure successfully completed.

SQL> print n

         N
----------
         1

SQL> explain plan for select count(*) from all_ash where module='TEST';

Explained.

SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 3460552988

------------------------------------------------------------------------------
| Id  | Operation          | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |         |     1 |    17 |   189   (0)| 00:00:01 |
|   1 |  SORT AGGREGATE    |         |     1 |    17 |            |          |
|*  2 |   TABLE ACCESS FULL| ALL_ASH |     1 |    17 |   189   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter('MODULE'='TEST')

Note
-----
   - SQL profile 'PROF_9kt35fkxmk1qw_3460552988' used for this statement

The SQL Profile is tolerated.

Conclusions

I actually needed this test so that the result stuck better in my head :) The interaction of SQL Profiles and SQL Plan Baselines was already very well explained by Maria Colgan in her blog post “What is the difference between SQL Profiles and SQL Plan Baselines”? And I confirmed what was stated in the blog post - the SQL Profiles and SQL Plan Baselines don’t work against each other, but they complement each other. While the Baselines define the set of execution plans that can be used by each query, the SQL Profiles only provide additional information to “push” the optimizer to favor one or another plan, but they don’t guarantee the plan would be used.

That was well visible during the test - while we had both SQL Plan Baselines accepted, the profile was tolerated and the optimizer chose the full table scan plan as “suggested” by the profile. As soon as the FTS plan was removed from Sql Plan Baselines, the “suggestion” from the SQL Profile didn’t have any effect, because there was only one allowed execution plan for the SQL Statement - the one using INDEX RANGE SCAN, and no SQL Profile can override that. However, as soon as the remaining SQL Plan Baseline was removed, the SQL Profile was again tolerated as at that moment there were no restrictions on which plans were allowed to be used, so again the FULL TABLE SCAN was used.

Do AWR Reports Show the Whole Picture?

2014-01-26T16:00:00-06:00

AWR report is a great source of aggregated information on top activities happening in our databases. I use data collected in AWR quite often, and obviously the easiest way of getting the data out from the AWR is by running the AWR report. In most cases that’s not an issue, but there are certain scenarios when it hides the information one is looking for, just because of how it’s designed.

If I’m trying to collect information about top queries by physical reads, I would normally look at the “SQL ordered by Reads” section and this is what I’d see:

I have the top SQLs by physical reads – just what I’ve been looking for (except the fact that AWR report covers only one of my RAC nodes).

But wait a second, what if there are queries that don’t use bind variables? This might be a problem as each query would have it’s own SQL_ID and probably they wouldn’t make it into the TOP10 just because each of them is treated separately. Nothing to worry about – AWR also collects FORCE_MATCHING_SIGNATURE values (read this blog post to understand why I know they would help) and we can use them to identify and group “similar” statements, we just need a custom script to do that.

Here I use my custom script to report TOP 20 SQL_IDs by physical reads in last 7 days (and I’m reporting data from both RAC nodes in the same list) – you can see the few TOP SQLs are the same as reported in AWR report, but because I’m reporting database-wide statistics instead of instance-wide as AWR does, I have other SQLs on the list too. I’ve also included 2 additional columns:

DIFF_PLANS – number of different PLAN_HASH_VALUE values reported for this SQL_ID, and if only one is found – it shows the actual PLAN_HASH_VALUE
DIFF_FMS – number of different FORCE_MATCHING_SIGNATURE values reported for this SQL_ID, and if only one is found – it shows the actual FORCE_MATCHING_SIGNATURE

Now, I can adjust the custom script to aggregate the data by FORCE_MATCHING_SIGNATURE, instead of SQL_ID. I’ll still keep the DIFF_PLANS column and will add a new one – DIFF_SQLID.

The situation is a little bit different now. Notice how the second row reports FORCE_MATCHING_SIGNATURE = 0, this typically shows PL/SQL blocks that execute the SQL statements and aggregate statistics from them, so we’re not interested in them. Otherwise the original report by SQL_ID showed quite accurate data in this situation and my suspicions regarding the misuse of literal values where binds should be used, didn’t materialize. Could I be missing anything else? Yes — even the FORCE_MATCHING_SIGNATURE could be misleading in identification of TOP resource consumers, you can write two completely different SQLs (i.e. select * from dual a and select * from dual b) that will do the same thing and will use the same execution plan. Let’s query the top consumers by PLAN_HASH_VALUE to check this theory!

I’ve highlighted the third row as the same PLAN_HASH_VALUE is reported for 20 different SQL_IDs, which allowed it to take the third place in the TOP list by physical reads (actually it’s the second place as PLAN_HASH_VALUE=0 is ignorable). The next query expands the third row:

And here are All the SQL statements:

What I have here is 20 different views generated by Oracle Discoverer that query the database by using exactly the same execution plan. Closer look revealed the views included hardcoded query parameters (date intervals for reporting), but in the end, this was the same query! It’s the TOP2 query by physical reads. in the database and if I tune it – all 20 discoverer views will benefit.

I think one of the drawbacks of AWR reports is that it is not able to identify such situations, it would be great if user could choose the column by which he aggregation is done. In the situation I described I was able to identify one of the top queries by physical reads only when I aggregated data by PLAN_HASH_VALUE.

My Experience At UKOUG Tech13

2013-12-09T18:08:03-06:00

This blog post was originally released on Pythian blog

UKOUG Tech13 was great! Not only because of the amount of interesting presentations to choose from, but also because of the surrounding events. I felt this conference was quite different from previous UKOUG Oracle Technology & E-Business Suite events in Birmingham (I’ve attended six previous events so you can trust me when I say that.) I decided to take a few notes about my experience at the conference to reveal what made it such a great event to attend.

The Changes

This was the first year Oracle Apps and Oracle Tech streams were separated in two different conferences. This is a great change for most Oracle DBAs, because the conference is able to concentrate more on Oracle Tech topics. However, I personally missed the opportunity to attend a couple of presentations for Apps DBAs. I usually looked for these presentations at previous events, as UKOUG conferences had been the only events (that I knew of) in Europe that gave a voice to Apps DBAs too. This year, it was no longer possible — on the other hand, I’m not interested in Apps Tech stuff only. I like learning new things about the database too, like performance tuning, internals and advanced troubleshooting. This conference had the widest selection of topics to chose from.

“Six presentations per company” is another change introduced this year. Alex Gorbachev discusses it more in his blog post here. I think this rule makes the conference worse, and not because Pythian had to remove some of the presentations and I couldn’t meet a few other colleagues. Mainly because the overall quality of the topics presented at the conference was artificially decreased. If I was able to suggest an experiment I’d ask UKOUG to remove that rule for the conference next year to check if the average marks from the session feedback surveys increase or decrease when the rule is not applied. I think the score would increase.

Meeting And Making Friends And Having Fun

The is the most important part of the conference for me. Nothing attracts so many great people working with the same technology as a good conference. It was great to finally meet some of the community activists that I knew from twitter and blogs only: Øyvind Isene, Osama Mustafa, David Kurtz, Philippe Fierens, Tim Hall, Alex Nuijten, Martin Bach, Elliot Zissman, Alex Zaballa, Fahd Mirza, Marcin Przepiorowski and others I forgot to put on the list. I checked and confirmed the rumors that a great way of making new friends is volunteering for some work. This time I volunteered to be one of the RAC Attack Ninjas and in the end, it was more fun and less work than I expected.

Me and Andrejs Karpovs practicing some useful RAC Attack Ninja moves.

Having fun is undeniably an important part of the conference. The brain needs to rest from the intense learning experience, and trust me, there were plenty of options to chose from. There were the official social events, and usually some unofficial events too. Take a look! These are my colleagues Michael McKee and Luke Davies discussing how useless Twitter is at one of the unofficial events.

Luke, Twitter is useless!

There are ways to spend quality time, even if you’re tired by the end of the conference. The city can offer something you won’t find anywhere else.

Attending Manchester United vs Everton

The Twitter

Oh man, this was intense. I’m glad I had prepared for this by buying an extra battery for my phone – the money was well spent. The statistics – 150 tweets in 5 days. This doesn’t sound too much, just 30 tweets a day, but for me, who had been doing an average of ~1 tweet a day, it was a lot. I even saw my name in twitter analytics board in place 3 ranked by twitter mentions, but this most likely is because @UKOUG retweeted almost all of my tweets!

Twitter Analytics by Rittman Mead

At one point, I almost quit tweeting from the conference but one tweet kept me going:

@MarisElsins seems to be the only dba in #ukoug_tech13 who tweets about contents of sessions, rather than their title or praise for speaker!
— Babak Tourani (@2ndhalf_oracle) December 2, 2013

If you asked me which of my tweets highlights the experiences in the conference best, I’d choose this one, which also brings me to the next topic – my presentation.

OMG! This is the room for my session on Wednesday. :o #Panic #ukoug_tech13 pic.twitter.com/rXDqtCkjH3
— Maris Elsins (@MarisDBA) December 2, 2013

My Presentation

OK, this was not really my presentation. The topic was submitted by a good friend of mine – Yury Velikanov, who was not able to attend the conference. I volunteered to present the paper as I thought it would be easy – topic by Yury, slides by Yury, speech by Maris. I was not a first-time presenter. I actually counted around 15 presentations I had delivered at international conferences all done in English (not my native language as you might have noticed by reading this blog), so I felt confident. I had seen Yury present the topic, I had checked the slides carefully and changed a few things, I understood the topic, so what could possibly go wrong? Anything. There were two things that went wrong for me.

The Auditorium. This was the first time I presented in the largest room at the conference. I found out which room it was on Monday. The presentation was on Wednesday, so I had two full days to panic. Having the biggest room had put additional pressure on the task and yes I was worried.
I and two other Pythian speakers – Luke Davies and Michael McKee – decided to rehearse the presentations in a hotel room on Tuesday afternoon. And I’m glad we did, because I struggled a lot – I was not able to formulate the thought fluently, and I was not able to do much more than reading the bullet points. There were couple of complicated slides that I couldn’t explain at all. My morale went down to zero, and panic levels skyrocketed. But I’m extremely glad we did the rehearsal, as I found out I was not ready at all by that time. That day I spent 6 hours preparing for the presentation. I reviewed all the slides again, and again. I put down some notes, which I never do for my own presentations. I tested few things that I wasn’t 100% sure would work, as I had to present them. I exchanged a number of thoughts with Luke and at 2AM I closed the laptop.

My presentation “10 Ways to improve your RMAN script” was scheduled for 10:05 on Wednesday. I got up early to rehearse it one more time in front of the hotel mirror — I felt I was much more fluent, but I was still not satisfied. At that time there was nothing more to do and the auditorium was waiting for me. I can’t say much about my own performance. I only remember all panic disappeared in first 3 minutes of the presentation; In fact, I panicked much more in front of the mirror than on the stage — I think I did OK. Not perfect, but OK. The most important thing for me was to come to these conclusions:

Presenting a topic which I didn’t write is much more difficult. I usually think a lot about the topic before submitting it, and I know large part of the content I’m going to include in the presentation. I missed this stage of preparation and therefore had to spend way more time on working through the slides to prepare the speech.
Rehearse the presentation at least a day before the show. This reveals how ready I am and I can still use the time left to improve the presentations.
Rehearse the presentation shortly before they “go live” on the day of the speech. This reduces my panic levels and puts my brain on track for the topic aI present.

I’m looking forward to receiving the session evaluations as this was a unique experience for me and I really hope I didn’t disappoint anyone. I appreciate the support and feedback I received on twitter. Thank you for that. And thanks to Yury for trusting the delivery of his presentation to me.

Learning

There were many great presentations but these are the ones I enjoyed the most. I’ve already managed to use some of the lessons I learned from these sessions in my day-to-day work.

Jonathan Lewis: “Compression: Index, basic and OLTP”
James Morle: “Optimal Oracle Configuration for Efficient Table Scanning”
David Kurtz: “Partition, Archive, Compress, Purge – Keep your ERP on the Road”
Christo Kutrovsky: “Maximise Data Warehouse Performance with Parallel Queries”
Frits Hoogland: “Hacking session: Advanced profiling of Oracle using function calls” – this was the only presentation I attended at “OakTable World UK 2013” conference.

This is the short summary of my experiences. I’m really happy I was able to attend the conference and meet all the people I usually meet virtually. I’m already waiting for UKOUG Tech14. See you there!

getMOSPatch.sh - Downloading Patches From My Oracle Support

2013-11-11T18:08:03-06:00

This blog post was originally released on Pythian blog

How to download patches from My Oracle Support (MOS) directly to the server? This has bothered me since the ftp access was closed a few years ago. Of course, I’ve been given some options by Oracle, like, I could access MOS from the server using a browser (probably from a VNC desktop - thank you very much), or I could look up the patches on my workstation to download the WGET script from MOS, which I uploaded to the server, adjusted with the username and the password of my MOS account and then started the downloads. Not too convenient, is it?

Then, back in 2009 my teammate John Piwowar published a blog post on [Retrieving Oracle patches with wget(http://only4left.jpiwowar.com/2009/02/retrieving-oracle-patches-with-wget). This eliminated the need to upload the wget script from MOS to the server and I only had to get the URLs of the patches and pass them to a shell function. While this was so much easier, I still needed to open the browser to find those URLs.

I think it’s time to get rid of browser dependency. So I’ve written a shell script [getMOSPatch.sh(http://bit.ly/getMOSPatch) that can be used to download patches directly to the server using only the patch number.

I’ve tested the tool on Linux, and there is a good chance it won’t work on some other platforms as it utilizes tools like awk, sed, grep, egrep and wget with options that probably only work on Linux, but if there’s much interest in this tool and I get many comments on this blog post I promise to change that :)

You can use wget to download the script to the server directly:

[oracle@mel1 Patches]$ wget --no-check-certificate -nv https://raw.github.com/MarisElsins/TOOLS/master/Shell/getMOSPatch.sh
WARNING: cannot verify raw.github.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3':
  Unable to locally verify the issuer's authority.
2013-11-10 17:42:17 URL:https://raw.github.com/MarisElsins/TOOLS/master/Shell/getMOSPatch.sh [4021/4021] -> 'getMOSPatch.sh' [1]
[oracle@mel1 Patches]$ chmod u+x getMOSPatch.sh

First time you run the script (or when you run it with parameter reset=yes) it will let you choose which platforms and languages the patches need to be downloaded for and the choices will be saved in a configuration file. The available platforms and languages are fetched from MOS.

[oracle@mel1 Patches]$ ./getMOSPatch.sh reset=yes
Oracle Support Userid: elsins@pythian.com
Oracle Support Password:

Getting the Platform/Language list
Available Platforms and Languages:
527P - Acme Packet OS
293P - Apple Mac OS X (Intel) (32-bit)
522P - Apple Mac OS X (Intel) (64-bit)
...
226P - Linux x86-64
912P - Microsoft Windows (32-bit)
...
7L - Finnish (SF)
2L - French (F)
4L - German (D)
104L - Greek (EL)
107L - Hebrew (IW)
...
39L - Ukrainian (UK)
43L - Vietnamese (VN)
999L - Worldwide Spanish (ESW)
Comma-delimited list of required platform and language codes: 226P,4L
[oracle@mel1 Patches]$

After this you simply have to run the script with parameter patch=patchnr1,patchnr2,... and the listed patches will be downloaded. This is how it happens:

The script looks up each of the patches for each platform and language and:
- if one patch is found - it is automatically downloaded
- if multiple patches are found (this can happen if the same patch is available for multiple releases) - the tool will ask you to choose which patches to download.
you can also specify parameter download=all to download all found patches without being asked to choose ones from the list.
you can also specify parameter regexp to apply filters to the filenames of the looked up patches. This is especially useful for Apps DBAs as filter regexp=".*A_R12.*" would be helpful for e-Business Suite Release 12.0 and regexp=".*B_R12.*" - for R12.1.
if you set environment variables mosUser and mosPass before running the script you won’t be asked to enter the user credentials.

Take a look at the following examples:

downloading the latest CPU patch (patch 16902043, OCT2013) for 11gR2.

[oracle@mel1 Patches]$ ./getMOSPatch.sh patch=16902043
Oracle Support Userid: elsins@pythian.com
Oracle Support Password:

Getting patch 16902043 for 'Linux x86-64';
p16902043_112030_Linux-x86-64.zip completed with status: 0

Getting patch 16902043 for 'German (D)';
no patch available

Downloading multiple patches at the same time without prompting user to specify which files to download if multiple files are found. (don’t be confused that files with “LINUX” and not “Linux-x86-64” in the filename are downloaded here. These are e-Business Suite patches and both 32b and 64b platforms have the same patch):

[oracle@mel1 Patches]$ ./getMOSPatch.sh patch=10020251,10141333 download=all
Oracle Support Userid: elsins@pythian.com
Oracle Support Password:

Getting patch 10020251 for 'Linux x86-64'
p10020251_R12.AR.B_R12_LINUX.zip completed with status: 0
p10020251_R12.AR.A_R12_LINUX.zip completed with status: 0

Getting patch 10020251 for 'German (D)'
p10020251_R12.AR.A_R12_d.zip completed with status: 0

Getting patch 10141333 for 'Linux x86-64'
p10141333_R12.AR.A_R12_LINUX.zip completed with status: 0
p10141333_R12.AR.B_R12_LINUX.zip completed with status: 0

Getting patch 10141333 for 'German (D)'
p10141333_R12.AR.B_R12_d.zip completed with status: 0
p10141333_R12.AR.A_R12_d.zip completed with status: 0

Downloading the same patches as in the previous example with an additional filter for e-Business Suite 12.1 patches only:

[oracle@mel1 Patches]$ ./getMOSPatch.sh regexp='.*B_R12.*' patch=10020251,10141333 download=all
Oracle Support Userid: elsins@pythian.com
Oracle Support Password:

Getting patch 10020251 for 'Linux x86-64'
p10020251_R12.AR.B_R12_LINUX.zip completed with status: 0

Getting patch 10020251 for 'German (D)'
no patch available

Getting patch 10141333 for 'Linux x86-64'
p10141333_R12.AR.B_R12_LINUX.zip completed with status: 0

Getting patch 10141333 for 'German (D)'
p10141333_R12.AR.B_R12_d.zip completed with status: 0

Mining the AWR to Identify Performance Trends

2013-11-04T12:56:57-06:00

This blog post was originally released on Pythian blog

Sometimes it’s useful to check how performance of a SQL statement changes over time. The diagnostic pack features provide some really useful information to answer these questions. The data is there, but it not always easy to retrieve it, especially if you want to see how the performance changes over time. I’ve been using three really simple scripts to retrieve this information from the AWR. These scripts help me answer the following questions:

How does the performance of a particular SQL change over time?
How do wait times of a particular wait event change over time?
How does a particular statistic change over time?

_ Please note, the scripts provided here require diagnostic pack licenses and it’s your task to make sure you have them before running the scripts._

SQL performance

I use script <a awr_sqlid_perf_trend.sql to check how performance of the SQL changes over time. The script summarizes the data from DBA_HIST_SQLSTAT and reports the average statistics for a single execution of the query during the reporting interval. It requires 3 input parameters:

SQL ID
Days to report. It will summarize all AWR snapshots starting with “trunc(sysdate)-{days to report}+1”, so if you pass “1”, it will summarize all snapshots from today, if “2” - than it’s yesterday and today are included.
Interval in hours. “24” will provide one row for each day, “6” will give 4 rows a day.

Nothing shows it better than an example. Below you see how I’m checking execution statistics for sql_id fd7rrqkn1k2xb by summarizing the AWR information captured in last 2 weeks and reporting the average values for 2-day intervals. Then I’m taking a little closer look at the last 4 days for the same SQL by summarizing data over 6hour intervals. Note, the time column shows the beginning of the interval.

SQL> @awr_sqlid_perf_trend.sql fd7rrqkn1k2xb 14 48

TIME                 EXECUTIONS ELAPSED_TIME_S_1EXEC CPU_TIME_S_1EXEC IOWAIT_S_1EXEC CLWAIT_S_1EXEC APWAIT_S_1EXEC CCWAIT_S_1EXEC ROWS_PROCESSED_1EXEC BUFFER_GETS_1EXEC  DISK_READS_1EXEC DIRECT_WRITES_1EXEC
------------------- ----------- -------------------- ---------------- -------------- -------------- -------------- -------------- -------------------- ----------------- ----------------- -------------------
10.2013 00:00:00         351              195.571           74.995           .097           .000           .000           .000           134417.570      21319182.291        293731.556          304434.305
10.2013 00:00:00         364               91.225           47.474          1.687           .000           .000           .002           141140.228      20364053.544        270107.745          273343.709
10.2013 00:00:00         542               20.686            9.378           .004           .000           .000           .000           146436.875       4597922.220             3.168                .000
10.2013 00:00:00         531               25.060           12.086           .161           .000           .000           .000           146476.605       6026729.224         23999.684           23998.859
10.2013 00:00:00         542               51.611           40.675          1.880           .000           .000           .000           146814.220      21620264.039        287994.862          287994.701
10.2013 00:00:00         534               39.949           26.688          1.050           .000           .000           .000           147099.275      14081016.607        159704.463          159704.418
10.2013 00:00:00         245               37.837           29.384          1.150           .000           .000           .000           147135.216      15505533.959        179244.437          179244.367

rows selected.

SQL> @awr_sqlid_perf_trend.sql fd7rrqkn1k2xb 4 6

TIME                 EXECUTIONS ELAPSED_TIME_S_1EXEC CPU_TIME_S_1EXEC IOWAIT_S_1EXEC CLWAIT_S_1EXEC APWAIT_S_1EXEC CCWAIT_S_1EXEC ROWS_PROCESSED_1EXEC BUFFER_GETS_1EXEC  DISK_READS_1EXEC DIRECT_WRITES_1EXEC
------------------- ----------- -------------------- ---------------- -------------- -------------- -------------- -------------- -------------------- ----------------- ----------------- -------------------
10.2013 00:00:00          72               19.209            9.439           .000           .000           .000           .000           147076.000       4623816.597              .111                .000
10.2013 06:00:00          72               15.391            9.401           .000           .000           .000           .000           147086.403       4624153.819              .000                .000
10.2013 12:00:00          72               14.022            9.351           .000           .000           .000           .000           147099.000       4624579.639              .000                .000
10.2013 18:00:00          55               48.174           35.723          1.575           .000           .000           .000           147099.000      19192781.582        243584.055          243584.055
10.2013 00:00:00          72               76.723           43.350          2.116           .000           .000           .000           147099.000      23258326.875        314445.111          314445.111
10.2013 06:00:00          72               64.921           43.914          2.084           .000           .000           .000           147107.542      23258506.028        315673.000          315673.000
10.2013 12:00:00          72               52.567           43.383          2.041           .000           .000           .000           147116.000      23258739.403        315673.000          315673.000
10.2013 18:00:00          47               25.522           18.095           .523           .000           .000           .000           147117.532       9382873.851         80597.702           80597.362
10.2013 00:00:00          65               17.645            9.384           .000           .000           .000           .000           147120.000       4625354.262              .000                .000
10.2013 06:00:00          19               17.571            9.451           .000           .000           .000           .000           147122.421       4625411.263              .000                .000
10.2013 12:00:00           6               14.083            9.645           .000           .000           .000           .000           147208.167       4629315.167              .000                .000
10.2013 18:00:00          48               42.173           35.208          1.509           .000           .000           .000           147236.375      18606643.833        229433.750          229433.750
10.2013 00:00:00          72               53.015           43.517          2.022           .000           .000           .000           147245.125      23265547.847        314507.319          314507.083
10.2013 06:00:00          30               52.181           43.638          1.932           .000           .000           .000           147250.300      23265839.767        303949.000          303949.000
10.2013 12:00:00           5               59.576           43.836          1.177           .000           .000           .000           144049.800      23267109.200        227814.000          227814.000

rows selected.

I’ve checked this SQL because the users reported inconsistent performance. It can also be observed in the outputs above. Take a look! The number of rows processed during each execution of the SQL doesn’t change - it’s always around 147K, but look at the disk reads and the direct writes! These values can be around zero, but then they suddenly jump up to 300K, and when they do, the buffer gets increase too and the CPU time goes up from 9 seconds to 43. Based on the information above it looks like there could be two different execution plans involved and bind variable peeking could be causing one or the other plan to become the active plan. Additionally you can use the same script to check how execution statistics for the same SQL change over time. Does the elapsed time increase? Do the number of processed rows or number of buffer gets per execution change?

Wait event performance

Script awr_wait_trend.sql can be used to show the changes in wait counts and wait durations for a particular event over time. Similarly to the previous script it also requires 3 parameters, only instead of SQL ID you pass the name of the wait event. This time the data comes from DBA_HIST_SYSTEM_EVENT. I typically use this script in two situations:

To check if a particular wait event performs worse when an overall performance problem is reported (usually I’m looking at IO events)
Illustrate how the implemented change improved the situation.

The example below shows how the performance of log file parallel write event changed over 3 weeks. On october 19th we moved the redo logs to dedicated high performance LUNs. Before that the 2 members of each redo log group were located on a saturated LUN together with all the data files.

SQL> @awr_wait_trend.sql 'log file parallel write' 21 24

TIME                EVENT_NAME                       TOTAL_WAITS   TOTAL_TIME_S    AVG_TIME_MS
------------------- ---------------------------- --------------- -------------- --------------
10.2013 00:00:00 log file parallel write              4006177      31667.591          7.905
10.2013 00:00:00 log file parallel write              3625342      28296.640          7.805
10.2013 00:00:00 log file parallel write              3483249      31032.324          8.909
10.2013 00:00:00 log file parallel write              3293462      33351.490         10.127
10.2013 00:00:00 log file parallel write              2871091      36413.925         12.683
10.2013 00:00:00 log file parallel write              3763916      30262.718          8.040
10.2013 00:00:00 log file parallel write              3018760      28262.172          9.362
10.2013 00:00:00 log file parallel write              3303205      31062.276          9.404
10.2013 00:00:00 log file parallel write              3012105      31831.491         10.568
10.2013 00:00:00 log file parallel write              2692697      26981.966         10.020
10.2013 00:00:00 log file parallel write              1038399        512.950           .494
10.2013 00:00:00 log file parallel write               959443        427.554           .446
10.2013 00:00:00 log file parallel write              1520444        606.580           .399
10.2013 00:00:00 log file parallel write              1618490        655.873           .405
10.2013 00:00:00 log file parallel write              1889845        751.216           .398
10.2013 00:00:00 log file parallel write              1957384        760.656           .389
10.2013 00:00:00 log file parallel write              2204260        853.691           .387
10.2013 00:00:00 log file parallel write              2205783        856.731           .388
10.2013 00:00:00 log file parallel write              2033199        785.785           .386
10.2013 00:00:00 log file parallel write              2439092        923.368           .379
10.2013 00:00:00 log file parallel write              2233614        840.628           .376

rows selected.

Visualizing the data from output like that is easy too!

System Statistics

The last script from this set is awr_stat_trend.sql. It does the same thing with the system statistics collected in DBA_HIST_SYSSTAT as previous scripts did to the performance of SQLs and wait events. The parameters are similar again - the name of the system statistic, days to report and the interval. I usually use the query to check how the redo size or the number of physical reads change over time, but there’s huge number of statistics available (638 different statistics in 11.2.0.3) and that’s why I’m sure you’ll find your own reasons to use this script.

SQL> @awr_stat_trend.sql 'redo size' 3 4

TIME                STAT_NAME                             VALUE
------------------- ------------------------- -----------------
10.2013 00:00:00 redo size                        1739466208
10.2013 04:00:00 redo size                        2809857936
10.2013 08:00:00 redo size                         648511376
10.2013 12:00:00 redo size                         533287888
10.2013 16:00:00 redo size                         704832684
10.2013 20:00:00 redo size                         819854908
10.2013 00:00:00 redo size                        2226799060
10.2013 04:00:00 redo size                        3875182764
10.2013 08:00:00 redo size                        1968024072
10.2013 12:00:00 redo size                        1125339352
10.2013 16:00:00 redo size                        1067175300
10.2013 20:00:00 redo size                         936404908
10.2013 00:00:00 redo size                        1758952428
10.2013 04:00:00 redo size                        3949193948
10.2013 08:00:00 redo size                        1715444632
10.2013 12:00:00 redo size                        1008385144
10.2013 16:00:00 redo size                         544946804

rows selected.

AWR is a gold mine, but you need the right tools for digging. I hope the scripts will be useful for you too! P.S. You might have noticed the scripts are published on GitHub, let me know if you find any issues using them and perhaps one day you’ll find new versions for the script.

Update (4-Nov-2013)

I’ve added the instance numbers to the outputs in all three scripts. This is how it looks now:

SQL> @awr_sqlid_perf_trend.sql 0fxqsqwhkn03b 14 48

 INST TIME                 EXECUTIONS ELAPSED_TIME_S_1EXEC CPU_TIME_S_1EXEC IOWAIT_S_1EXEC CLWAIT_S_1EXEC APWAIT_S_1EXEC CCWAIT_S_1EXEC ROWS_PROCESSED_1EXEC BUFFER_GETS_1EXEC  DISK_READS_1EXEC DIRECT_WRITES_1EXEC
----- ------------------- ----------- -------------------- ---------------- -------------- -------------- -------------- -------------- -------------------- ----------------- ----------------- -------------------
28.10.2013 00:00:00         840                 .611             .014           .595           .007           .000           .000                1.000          1085.583           128.724                .000
10.2013 00:00:00        1466                 .491             .011           .479           .005           .000           .000                1.000           976.001            88.744                .000
11.2013 00:00:00         542                 .798             .023           .760           .025           .000           .000                1.000           896.978           114.196                .000
11.2013 00:00:00         544                 .750             .021           .719           .017           .000           .000                1.000          1098.213           134.941                .000

28.10.2013 00:00:00        1638                 .498             .017           .474           .013           .000           .000                1.001           953.514            96.287                .000
10.2013 00:00:00        1014                 .745             .022           .712           .019           .000           .000                1.000          1034.249           131.057                .000
11.2013 00:00:00        1904                 .633             .011           .624           .002           .000           .000                1.000          1045.668           104.568                .000
11.2013 00:00:00         810                 .602             .017           .581           .010           .000           .000                1.000           929.778           108.998                .000


rows selected.

Meaning of “Disk Reads” Values in DBA_HIST_SQLSTAT

2013-11-01T15:38:58-05:00

This blog post was originally released on Pythian blog

This post relates to my previous writing on mining the AWR. I noticed that it’s very easy to misinterpret the DISK_READS_TOTAL and DISK_READS_DELTA columns in DBA_HIST_SQLSTAT. Let’s see what the documentation says:

DISK_READS_TOTAL - Cumulative number of disk reads for this child cursor

DISK_READS_DELTA - Delta number of disk reads for this child cursor

You might think it’s clear enough and that’s exactly what I thought too. The number of disk reads is the number of IO requests to the storage. But is it really true?

I started suspecting something was not right after using my own awr_sqlid_perf_trend.sql script (see more details on this script here. I noticed the DISK_READS_DELTA values were too close to BUFFER_GETS_DELTA values for queries that use full table scans, which are normally executed using multi-block IO requests to the storage. I was expecting disk reads to be at least two times lower than the buffer gets, but it was something closer to 90% in a few cases. So was I looking at the number of IO requests or the number of blocks read from disks? The best way to find it out was a test case.

The following testing was done in an 11.2.0.3 database:

I created a new AWR snapshot and enabled tracing for my session. I made sure the db_file_multiblock_read_count parameter was set to a high value and then executed a SQL that was forced to use a full table scan (FTS) to read the data from disks. Another AWR snapshot was taken after that.

SQL> alter session set tracefile_identifier='TEST1';
Session altered.

SQL> show parameter multiblock
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_file_multiblock_read_count        integer     26

SQL> alter system set db_file_multiblock_read_count=128;
System altered.

SQL> show parameter multiblock
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_file_multiblock_read_count        integer     128

SQL> exec dbms_workload_repository.create_snapshot();
PL/SQL procedure successfully completed.

SQL> alter session set max_dump_file_size=unlimited;
Session altered.

SQL> alter system set events '10046 trace name context forever, level 12';
System altered.

SQL> select /*+ full(a) */ count(I_DATA) from tpcc.item;
COUNT(I_DATA)
-------------
       100000

SQL> exec dbms_workload_repository.create_snapshot();
PL/SQL procedure successfully completed.

I found the sql_id in the trace file (it was 036c3dmx2u3x9) and executed the awr_sqlid_perf_trend.sql to find out how many disk reads were made (I removed a few columns that are not important here).

SQL> @awr_sqlid_perf_trend.sql 036c3dmx2u3x9 20 0.001

 INST TIME                BUFFER_GETS_1EXEC  DISK_READS_1EXEC DIRECT_WRITES_1EXEC  EXECUTIONS ROWS_PROCESSED_1EXEC
----- ------------------- ----------------- ----------------- ------------------- ----------- --------------------
    1 24.10.2013 03:53:06          1092.000          1073.000                .000           1                1.000

It was a single execution and look at the numbers! 1073 disk reads and 1092 buffer gets. Could it be the DISK_READS_DELTA is actually the number of blocks read from disks? I need to check the raw trace file to find out.

I found the following lines in the trace file. I’ve highlighted all lines that report waits on physical IO. Notice the first query (sqlid=’96g93hntrzjtr’) is a recursive SQL (dep=1) for the query I executed (sqlid=’036c3dmx2u3x9’, and it was executed during the PARSE phase “PARSE #7904600” for my query. There were few other recursive statements, but they didn’t do any disk IOs (you’ll have to trust me here). It’s good to know the lines are written to the trace file after the corresponding event completes, this is why the recursive statements of the parse phase are reported before the line describing the whole parse operation.

PARSING IN CURSOR #25733316 len=210 dep=1 uid=0 oct=3 lid=0 tim=1382576068532471 hv=864012087 ad='3ecd4b88' sqlid='96g93hntrzjtr'
select /*+ rule */ bucket_cnt, row_cnt, cache_cnt, null_cnt, timestamp#, sample_size, minimum, maximum, distcnt, lowval, hival, density, col#, spare1, spare2, avgcln from hist_head$ where obj#=:1 and intcol#=:2
END OF STMT
PARSE #25733316:c=0,e=240,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=3,plh=0,tim=1382576068532470
EXEC #25733316:c=0,e=404,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=3,plh=2239883476,tim=1382576068532919
WAIT #25733316: nam='db file sequential read' ela= 885 file#=1 block#=64857 blocks=1 obj#=427 tim=1382576068533833
WAIT #25733316: nam='db file sequential read' ela= 996 file#=1 block#=58629 blocks=1 obj#=425 tim=1382576068534935
FETCH #25733316:c=0,e=2092,p=2,cr=3,cu=0,mis=0,r=1,dep=1,og=3,plh=2239883476,tim=1382576068535022
STAT #25733316 id=1 cnt=1 pid=0 pos=1 obj=425 op='TABLE ACCESS BY INDEX ROWID HIST_HEAD$ (cr=3 pr=2 pw=0 time=2079 us)'
STAT #25733316 id=2 cnt=1 pid=1 pos=1 obj=427 op='INDEX RANGE SCAN I_HH_OBJ#_INTCOL# (cr=2 pr=1 pw=0 time=989 us)'
CLOSE #25733316:c=0,e=58,dep=1,type=3,tim=1382576068535146
=====================
PARSING IN CURSOR #7904600 len=50 dep=0 uid=0 oct=3 lid=0 tim=1382576068535413 hv=4197257129 ad='3618a2a4' sqlid='036c3dmx2u3x9'
select /*+ full(a) */ count(I_DATA) from tpcc.item
END OF STMT
PARSE #7904600:c=8001,e=12985,p=2,cr=19,cu=0,mis=1,r=0,dep=0,og=1,plh=1537583476,tim=1382576068535411
EXEC #7904600:c=0,e=29,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1537583476,tim=1382576068535500
WAIT #7904600: nam='SQL*Net message to client' ela= 3 driver id=1650815232 #bytes=1 p3=0 obj#=-1 tim=1382576068535530
WAIT #7904600: nam='db file sequential read' ela= 1960 file#=4 block#=113050 blocks=1 obj#=65019 tim=1382576068537566
WAIT #7904600: nam='direct path read' ela= 1203 file number=4 first dba=113051 block cnt=5 obj#=65019 tim=1382576068539309
WAIT #7904600: nam='direct path read' ela= 1531 file number=4 first dba=123392 block cnt=8 obj#=65019 tim=1382576068541567
WAIT #7904600: nam='direct path read' ela= 1047 file number=4 first dba=123401 block cnt=15 obj#=65019 tim=1382576068542719
WAIT #7904600: nam='direct path read' ela= 1081 file number=4 first dba=123417 block cnt=15 obj#=65019 tim=1382576068543895
WAIT #7904600: nam='direct path read' ela= 956 file number=4 first dba=123433 block cnt=15 obj#=65019 tim=1382576068544997
WAIT #7904600: nam='direct path read' ela= 950 file number=4 first dba=123449 block cnt=15 obj#=65019 tim=1382576068546096
WAIT #7904600: nam='direct path read' ela= 1168 file number=4 first dba=123465 block cnt=15 obj#=65019 tim=1382576068547425
WAIT #7904600: nam='direct path read' ela= 1151 file number=4 first dba=123481 block cnt=15 obj#=65019 tim=1382576068548784
WAIT #7904600: nam='direct path read' ela= 1279 file number=4 first dba=123497 block cnt=15 obj#=65019 tim=1382576068550229
WAIT #7904600: nam='direct path read' ela= 9481 file number=4 first dba=123522 block cnt=126 obj#=65019 tim=1382576068559912
WAIT #7904600: nam='direct path read' ela= 6872 file number=4 first dba=123650 block cnt=126 obj#=65019 tim=1382576068566997
WAIT #7904600: nam='direct path read' ela= 5562 file number=4 first dba=123778 block cnt=126 obj#=65019 tim=1382576068573516
WAIT #7904600: nam='direct path read' ela= 7524 file number=4 first dba=123906 block cnt=126 obj#=65019 tim=1382576068582195
WAIT #7904600: nam='direct path read' ela= 5858 file number=4 first dba=124034 block cnt=126 obj#=65019 tim=1382576068589263
WAIT #7904600: nam='direct path read' ela= 5326 file number=4 first dba=124162 block cnt=126 obj#=65019 tim=1382576068595750
WAIT #7904600: nam='direct path read' ela= 5788 file number=4 first dba=124290 block cnt=126 obj#=65019 tim=1382576068602627
WAIT #7904600: nam='direct path read' ela= 2446 file number=4 first dba=124418 block cnt=70 obj#=65019 tim=1382576068607337
FETCH #7904600:c=4000,e=73444,p=1071,cr=1073,cu=0,mis=0,r=1,dep=0,og=1,plh=1537583476,tim=1382576068608996
STAT #7904600 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE (cr=1073 pr=1071 pw=0 time=73444 us)'
STAT #7904600 id=2 cnt=100000 pid=1 pos=1 obj=65019 op='TABLE ACCESS FULL ITEM (cr=1073 pr=1071 pw=0 time=49672 us cost=198 size=3900000 card=100000)'
WAIT #7904600: nam='SQL*Net message from client' ela= 148 driver id=1650815232 #bytes=1 p3=0 obj#=65019 tim=1382576068609235
FETCH #7904600:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=1537583476,tim=1382576068609261
WAIT #7904600: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=65019 tim=1382576068609276
WAIT #7904600: nam='SQL*Net message from client' ela= 940 driver id=1650815232 #bytes=1 p3=0 obj#=65019 tim=1382576068610226
CLOSE #7904600:c=0,e=17,dep=0,type=0,tim=1382576087713173

The next task was to count the “blocks” for db file sequential reads and “block cnt” for direct path reads. The recursive SQL (96g93hntrzjtr) read 2 data blocks from disks and the main SQL (036c3dmx2u3x9) read 1071 data blocks from disks. The total number is 1073! Hey, this is exactly what DISK_READS_DELTA (DISK_READS_1EXEC in the script outputs above) reported - so it’s the number of data blocks, and not the number of IO requests!

The investigation resulted in two obvious conclusions:

DISK_READS_TOTAL and DISK_READS_DELTA in DBA_HIST_SQLSTAT report the number of blocks read from disks.
The query statistics in DBA_HIST_SQLSTAT also include the data from execution of the recursive statements.

P.S. Later I found another column - DBA_HIST_SQLSTAT.PHYSICAL_READ_REQUESTS_DELTA - that was introduced in 11.2 along with a large number of additional columns. PHYSICAL_READ_REQUESTS_DELTA and PHYSICAL_READ_REQUESTS_TOTAL represent the number of IO requests that were executed. You can compare the numbers by counting the highlighted rows above to the value I found in DBA_HIST_SQLSTAT below.

SQL> select DISK_READS_DELTA, PHYSICAL_READ_REQUESTS_DELTA from dba_hist_sqlstat where sql_id='036c3dmx2u3x9';

DISK_READS_DELTA PHYSICAL_READ_REQUESTS_DELTA
---------------- ----------------------------
            1073                           20

There’s Always Another Bug Hiding Just Around the Corner

2013-08-15T04:01:51-05:00

This blog post was originally released on Pythian blog

We were using a 10.2.0.3 database, and it had been running without any issues for several years. What could possibly go wrong? Anything! Suddenly, we started getting ORA-07445: exception encountered: core dump [qercoStart()+156] [SIGSEGV] [Address not mapped to object] a few times a minute in the alert log. A closer investigation revealed that one of the popular SQLs in the application couldn’t complete anymore. It looked like a bug, since only the SQL was failing.

We found a few references for various releases with the same conditions: ORA-07445 + qercoStart(). This list summarizes the possible causes for the error I found on My Oracle Support:

Using ROWNUM < x condition in the where clause
Using ROWNUM condition and FULL OUTER joins
Using ROWNUM condition with UNION ALL set operation

The strange thing was that this started suddenly; no changes to the code were made. Moreover, the SQL didn’t contain FULL OUTER join operations or UNION ALL set operations:

SELECT CBMD.CBMD_BASE_MDL_NUMBER,
 MFG_GROUP MFG_ID,
 MFG_NAME,
 CBMD_CATALOG_MODEL_NUMBER,
 CBMD_CATALOG_MODEL_SHORT_DESC,
 CBMD_IMAGE_PATH,
 SUM (CSMD_AVAILABLE_QTY) QTY
FROM CATALOG_BASE_MODEL_DATA CBMD,
 CATALOG_BASE_MODEL_CATEGORY CBMC,
 CATALOG_SUB_MODEL_DATA CSMD
WHERE CBMD.CBMD_BASE_MDL_NUMBER = CSMD.CBMD_BASE_MDL_NUMBER
 AND CBMD.CBMD_BASE_MDL_NUMBER = CBMC.CBMD_BASE_MDL_NUMBER
 AND (
 (CSMD.CSMD_AVAILABLE_FOR_SALE_FLAG = 'N'
 AND CBMC.DC_DIVISION_CODE = 1)
 OR (CBMC.DC_DIVISION_CODE = 2)
 )
 AND CBMD_PUT_ON_WEB_FLAG = 'Y'
 AND CSMD_AVAIL_FOR_WEB_DISP_FLAG = 'Y'
 AND CBMC.DC_DIVISION_CODE = :B1
 AND ROWNUM < 5
GROUP BY CBMD.CBMD_BASE_MDL_NUMBER,
 MFG_GROUP,
 MFG_NAME,
 CBMD_CATALOG_MODEL_NUMBER,
 CBMD_CATALOG_MODEL_SHORT_DESC,
 CBMD_IMAGE_PATH,
 CBMD_IMAGE_NAME
ORDER BY QTY DESC

We also tried all the possible workarounds listed in the bug descriptions, but nothing helped:

Flushed the shared pool
setting "_complex_view_merging"=false
bouncing the database

As raising a SR for our 10.2.0.3 DB was unlikely to help, I decided to dig deeper. I knew something had changed, and that change was what triggered the bug. I didn’t know where to start, so I decided to look more closely in the bug descriptions in My Oracle Support. All the bugs listed examples of SQL statements containing “ROWNUM < X” condition. The second similarity was harder to notice. Here are some examples - I’ve highlighted the interesting lines:

from bug 7704557 on 10.2.0.4

select jsp1.name name , jsp1.value value
from 'SYSJCS'.jcs_scheduler_parameters jsp1
where jsp1.name in ('database_name', 'global_names', 'scheduler_hostname', 'remote_start_port', 'scheduler_connect_string', 'oracle_sid', 'listener_port', 'remote_http_output','remote_http_port')
and rownum <= 9
and scheduler_name = nvl (:scheduler, scheduler_name);

from bug 7528596 on 10.2.0.3

SELECT /*+ FIRST_ROWS(200) */ rv.STATUS_NAME H_STATUS_ID
,rv.DESCRIPTION H_DESCRIPTION  ,rv.PRIORITY_MEANING H_PRIORITY_CODE
,rv.CREATED_BY_NAME H_CREATED_BY  , rv.CREATED_BY_EMAIL H_CREATED_BY_E ,
H_CREATED_BY_N  , rv.CREATION_DATE H_CREATION_DATE  ,rv.ASSIGNED_TO_NAME
H_ASSIGNED_TO_USER_ID  , rv.ASSIGNED_TO_EMAIL H_ASSIGNED_TO_USER_ID_E ,
rv.ASSIGNED_TO_USERNAME H_ASSIGNED_TO_USER_ID_N  ,rv.REQUEST_ID
H_REQUEST_ID
FROM itgadm.kcrt_requests_v rv,
     itgadm.kcrt_req_header_details rh WHERE (1=1
AND (rv.batch_number =1 OR rv.batch_number is null)
AND rv.REQUEST_TYPE_ID in (30593)
AND rv.REQUEST_TYPE_ID in (30593)
AND ( rv.STATUS_CODE NOT LIKE 'CLOSED%'
AND rv.STATUS_CODE NOT LIKE 'CANCEL%' )
AND exists(SELECT /*+ NO_UNNEST */
pcv.REQUEST_ID FROM itgadm.KCRT_PARTICIPANT_CHECK_V pcv WHERE
pcv.request_id = rv.request_id and pcv.user_id = 30481)
AND rh.request_id = rv.request_id )
AND ( rh.PARAMETER22 = 30481 OR rv.ASSIGNED_TO_USER_ID = 30481 )
AND ROWNUM <= 200
ORDER BY rv.REQUEST_ID DESC;

from bug 7416171 on 10.2.0.3

SELECT COUNT(*) FROM (
  SELECT (
    SELECT DECODE(COUNT(*),0,0,1) isStemData
    FROM ccd2.vw_policy_admins q
    WHERE q.system_id = a.system_id
      AND q.policy_no = a.policy_no
      AND ((q.policy_admin_id in (440502405,440502499)))
      AND rownum < 2)isStemData
  FROM ccd2.vw_ordf_partners_cnt a
  WHERE a.PARTNER_ID = 36977489
  /*AND a.PARTNER_ID2 = a.PARTNER_ID */
  AND a.ORDF_ID = 2
  AND 1=1) b
WHERE isStemData=1
AND rownum < 300

from bug 3211315 on 9.2.0.4

select dummy  from
  (SELECT dummy from dual where rownum < 2)  FULL OUTER JOIN
  (SELECT dummy from dual where rownum < 2)
using (dummy)

The first 3 SQLs contain “IN” or “OR” operators, and the last one contains a FULL OUTER JOIN set operation that was said to have issues. Knowing a bit of theory helped me identify some similarities:

Oracle introduced native FULL OUTER JOIN operation in 10.2.0.5. Before that, it was implemented using the UNION ALL operation. (Cristian Antognini explains it here and gives some examples.)
“OR” and “IN” predicates can sometimes be optimized by applying the “OR Expansion” transformation, which acquires the result set of each disjunction condition separately and then combines them using the set operations, i.e. UNION ALL. (Maria Colgan explains it here better than anyone else could.)

At that moment, I started suspecting this could be our case too because the SQL had an “OR” predicate. It was easy to check and confirm by looking at the execution plan. The highlighted line contained the CONCATENATION operation, which is the same as UNION ALL:

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------
Plan hash value: 132832423

--------------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name                    | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                   |                         |   434 | 69874 |    61   (4)| 00:00:01 |
|   1 |  SORT ORDER BY                     |                         |   434 | 69874 |    61   (4)| 00:00:01 |
|   2 |   HASH GROUP BY                    |                         |   434 | 69874 |            |          |
|*  3 |    COUNT STOPKEY                   |                         |       |       |            |          |
|   4 |     CONCATENATION                  |                         |       |       |            |          |
|*  5 |      FILTER                        |                         |       |       |            |          |
|   6 |       NESTED LOOPS                 |                         |     1 |   161 |    30   (0)| 00:00:01 |
|   7 |        NESTED LOOPS                |                         |     7 |  1008 |    23   (0)| 00:00:01 |
|*  8 |         TABLE ACCESS FULL          | CATALOG_SUB_MODEL_DATA  |    21 |   420 |     2   (0)| 00:00:01 |
|*  9 |         TABLE ACCESS BY INDEX ROWID| CATALOG_BASE_MODEL_DATA |     1 |   124 |     1   (0)| 00:00:01 |
|* 10 |          INDEX UNIQUE SCAN         | CBMD_C1_1_PK            |     1 |       |     0   (0)| 00:00:01 |
|* 11 |        INDEX RANGE SCAN            | CBMC_C1_1_PK            |     1 |    17 |     1   (0)| 00:00:01 |
|* 12 |      FILTER                        |                         |       |       |            |          |
|  13 |       NESTED LOOPS                 |                         |     2 |   322 |    29   (0)| 00:00:01 |
|  14 |        NESTED LOOPS                |                         |     7 |  1008 |    22   (0)| 00:00:01 |
|* 15 |         TABLE ACCESS FULL          | CATALOG_SUB_MODEL_DATA  |    20 |   400 |     2   (0)| 00:00:01 |
|* 16 |         TABLE ACCESS BY INDEX ROWID| CATALOG_BASE_MODEL_DATA |     1 |   124 |     1   (0)| 00:00:01 |
|* 17 |          INDEX UNIQUE SCAN         | CBMD_C1_1_PK            |     1 |       |     0   (0)| 00:00:01 |
|* 18 |        INDEX RANGE SCAN            | CBMC_C1_1_PK            |     1 |    17 |     1   (0)| 00:00:01 |
--------------------------------------------------------------------------------------------------------------

A quick Google search gave me a NO_EXPAND hint, which disables the OR expansion. However, I couldn’t use it since it required a code change. I knew that the behavior of the optimizer was controlled by a large number of hidden parameters that are also listed in the 10053 trace:

SQL> ALTER SESSION SET EVENTS '10053 trace name context forever,level 1';

Session altered.

SQL> alter session set tracefile_identifier=CR758708_2;

Session altered.

SQL> alter session set max_dump_file_size=unlimited;

Session altered.

SQL> explain plan for
 SELECT CBMD.CBMD_BASE_MDL_NUMBER,
 /*removed some lines for readability*/
Explained.

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

$ more test_ora_17805_CR758708_2.trc
/*removed some lines for readability*/
...
***************************************
PARAMETERS USED BY THE OPTIMIZER
********************************
...
 *************************************
 PARAMETERS WITH DEFAULT VALUES
 ******************************
...
 _fast_full_scan_enabled = true
 _optim_enhance_nnull_detection = true
 _parallel_broadcast_enabled = true
 _px_broadcast_fudge_factor = 100
 _ordered_nested_loop = true
 _no_or_expansion = false
 optimizer_index_cost_adj = 100
 optimizer_index_caching = 0
 _system_index_caching = 0
 _disable_datalayer_sampling = false
...

I disabled the OR expansion by setting the parameter "_no_or_expansion" = true, checked the execution plan, and confirmed that the query transformation didn’t happen:

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------------------
Plan hash value: 1045847658

-----------------------------------------------------------------------------------------------------
| Id  | Operation                 | Name                    | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |                         |     4 |   644 |   324   (3)| 00:00:04 |
|   1 |  SORT ORDER BY            |                         |     4 |   644 |   324   (3)| 00:00:04 |
|   2 |   HASH GROUP BY           |                         |     4 |   644 |   324   (3)| 00:00:04 |
|*  3 |    COUNT STOPKEY          |                         |       |       |            |          |
|*  4 |     HASH JOIN             |                         |   434 | 69874 |   322   (2)| 00:00:04 |
|*  5 |      TABLE ACCESS FULL    | CATALOG_SUB_MODEL_DATA  |  3777 | 75540 |    62   (4)| 00:00:01 |
|*  6 |      HASH JOIN            |                         |  3087 |   425K|   260   (2)| 00:00:04 |
|*  7 |       INDEX FAST FULL SCAN| CBMC_C1_1_PK            |  1759 | 29903 |    13   (8)| 00:00:01 |
|*  8 |       TABLE ACCESS FULL   | CATALOG_BASE_MODEL_DATA |  2685 |   325K|   247   (2)| 00:00:03 |
-----------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter(ROWNUM<5)
   4 - access('CBMD'.'CBMD_BASE_MDL_NUMBER'='CSMD'.'CBMD_BASE_MDL_NUMBER')
       filter('CSMD'.'CSMD_AVAILABLE_FOR_SALE_FLAG'='N' AND 'CBMC'.'DC_DIVISION_CODE'=1 OR
              'CBMC'.'DC_DIVISION_CODE'=2)
   5 - filter('CSMD_AVAIL_FOR_WEB_DISP_FLAG'='Y')
   6 - access('CBMD'.'CBMD_BASE_MDL_NUMBER'='CBMC'.'CBMD_BASE_MDL_NUMBER')
   7 - filter(('CBMC'.'DC_DIVISION_CODE'=1 OR 'CBMC'.'DC_DIVISION_CODE'=2) AND
              'CBMC'.'DC_DIVISION_CODE'=TO_NUMBER(:B1))
   8 - filter('CBMD_PUT_ON_WEB_FLAG'='Y')

28 rows selected.

In our case, the optimizer had changed the execution plan after fresh statistics were collected - this was the change that triggered the bug. We set the parameter to disable the OR expansion until we upgrade to 11.2.

I wanted to share this story with you because it’s interesting how different things (IN and OR predicates, UNION ALL and FULL OUTER JOIN set operations, etc.) transform behind the scenes into the same conditions to trigger the same bug. I think this incident has also changed the way I’ll read bug descriptions on My Oracle Support in the future - there is information hidden between the lines.

Be Warned: cmclean.sql is Dangerous!

2013-07-18T12:53:06-05:00

This blog post was originally released on Pythian blog

I’m sure one of the most popular scripts for Apps DBAs on My Oracle Support is cmclean.sql from MOS Article ID 134007.1 “Concurrent Processing - CMCLEAN.SQL - Non Destructive Script to Clean Concurrent Manager Tables”. DBAs usually use the script to clean up stale data from concurrent processing tables (FND_CONCURRENT_%) after incidents like a crash of the database or concurrent processing node. This script sets correct completion phase and status codes for terminated concurrent requests and sets correct control codes for terminated concurrent manager processes. Despite the assuring “Non Destructive” claim in the title of the MOS Article there is a possibility to lose concurrent request schedules when cmclean.sql is executed.

First of all it’s important to understand how scheduled concurrent requests are executed and resubmitted. A simplified process of the execution is:

Concurrent manager process (e.g. FNDLIBR in case of Standard Manager) queries the FND_CONCURRENT_REQUESTS table for pending requests.
When a pending request is found, the manager process updates the PHASE_CODE=R (Running) and STATUS_CODE=R (Running).
The next step is to start the executable of the concurrent program. If it’s a PL/SQL procedure – FNDLIBR connects to the DB and executes the PL/SQL code, if it’s a java program – FNDLIBR starts up a java process to execute the java class, etc.
FNDLIBR catches the exit codes from the executable of the concurrent program and updates the statuses in FND_CONCURRENT_REQUESTS accordingly - PHASE_CODE=C (Completed) and STATUS_CODE = C (Normal), G (Warning) or E (Error).
FNDLIBR checks if the concurrent request has a schedule and needs to be resubmitted. If yes – it resubmits a new concurrent request with the same parameters.

But what happens if the FNDLIBR process crashes, terminates or gets killed while it’s running a concurrent request? Who takes care of the statuses in FND_CONCURRENT_REQUESTS table and how the request is resubmitted if the concurrent manager process is not there anymore?

It appears the Internal Concurrent Manager (ICM) takes care of these tasks. It checks the running requests periodically (every two minutes by default) and if it finds any that’s missing the concurrent manager process and the DB session, it updates the statuses for the concurrent request and also resubmits it if it has a schedule. This action is followed by a log entry in the ICM log file:

                   Process monitor session started : 17-JUL-2013 04:24:24

Found running request 5829148 attached to dead manager process.
Setting request status to completed.

Found dead process: spid=(15160), cpid=(2032540), ORA pid=(35), manager=(0/0)

Starting STANDARD Concurrent Manager               : 17-JUL-2013 04:24:25

                     Process monitor session ended : 17-JUL-2013 04:24:25

Interesting to note, if the Internal Concurrent Manager is terminated at the same time with the manager process and is restarted later by the reviver process or by running “adcmctl.sh start” manually, the ICM performs the same check of running requests as part of the startup sequence, but this time it restarts the request instead of terminating and resubmitting it. The log of the ICM contains the following lines:

Found running request 5829146 attached to dead manager process.
Attempting to restart request.

The concurrent request is started again with exactly the same request_id as previous time when it was terminated, also the log file of the request will contain information from 2 executions – the first, which didn’t complete and then the secon which probably completed. I think this scenario is very confusing and instead of restarting the request it should better be terminated and a new one should be submitted.

Let’s get back to the problem with cmclean.sql! The worst thing that can be done is running cmclean.sql after crash of the concurrent processing node before starting up the concurrent managers. Why? Because cmclean.sql cleans up data in FND_CONCURRENT_REQUESTS by executing one simple update statement to change the phase and status of any “Running” or “Terminating” request to “Completed/Error”:

UPDATE fnd_concurrent_requests
SET phase_code = 'C', status_code = 'E'
WHERE status_code ='T' OR phase_code = 'R';

cmclean.sql does not resubmit the request if it has a schedule. Execute it and you risk to lose some scheduled programs without any warning.

Similarly – never run cmclean.sql if you stopped the concurrent managers using “adcmctl.sh abort” or “kill -9” on concurrent manager processes to speed up the shutdown procedure. There’s the same risk to lose some scheduled requests.

Despite the risks, cmclean.sql is still a useful tool in case concurrent managers don’t come up after a failure or there are some stale data that is otherwise not cleaned up. But please, be careful when you run it! Check closely the list of requests reported in the following section of the outputs from cmclean.sql, because these requests have to be resubmitted manually if they had schedules.

-- Updating any Running or Terminating requests to Completed/Error

Request ID Phase  Status
---------- ------ ------
6607       R      W
6700       R      W
893534056  R      R

3 rows updated.

“Concurrent Manager Recovery” wizard is even worse! (Added on Jul 21, 2013)

After posting this article I started thinking if the “Concurrent Manager Recovery” Wizard available from Oracle Applications Manager in e-Business Suite was any better then cmclean.sql. As I didn’t have much experience with it I decided to give it a try. This is what I did:

Scheduled 2 concurrent programs (“CP Java Regression Test” and “CP PLSQL Regression Test”) to restart in 1 minute after the previous execution completes. These are simple test concurrent programs which sleep for some time and then complete.
I made sure both of the programs were running and terminated all concurrent manager process and DB sessions for these concurrent programs.
Termination of the processes and sessions left the rows in FND_CONCURRENT_REQUESTS with PHASE_CODE=R and STATUS_CODE=R
I executed the “Concurrent Manager Recovery” wizard which fixed the status codes of the concurrent manager processes, but didn’t touch the statuses of the concurrent requests - I thought this was a good thing (I expected the ICM would clean up the statuses and would resubmit the requests at it’s startup phase)
I started up the concurrent managers, but ICM didn’t clean up the 2 stale records in FND_CONCURRENT_REQUESTS table. The 2 requests appeared as they would be running, while in fact they didn’t have any OS processes or DB sessions.

I didn’t have much time to look into the details, but it looks like the ICM is only cleaning up requests attached to dead managers (“Active” status in the FND_CONCURRENT_PROCESSES table and no OS processes running), but here the Wizard updated the statuses of the manager processes as if they completed normally so the ICM couldn’t identify them as being “dead”.

This actually means the “Concurrent Manager Recovery” wizard can cause serious issues too - it doesn’t clear up the concurrent_request statuses and it prevents ICM from doing it too, so once we start up the system the terminated requests appear as if they were running. And because of this, the Conflict Resolution Manager might prevent execution of some other programs with the incompatibility rules against the terminated requests. You will need to stop the managers and run cmclean.sql to fix the statuses (and loose the schedules) to get out of this situation.

So what should we do to clean up the concurrent processing tables after crashes or cloning? (Added on Jul 21, 2013)

It appears to me there is no reliable way to clean up the tables properly. The cmclean.sql can remove some schedules without warning. The “Concurrent Manager Recovery” wizard may leave some requests in the running state even if they were terminated.

I’m going to open a SR for Oracle to request a proper solution, but meanwhile I’d suggest to use the cmclean.sql, but make sure to check it’s outputs carefully and reschedule any requests which got cleaned up (as described above).

P.S. The description of the behavior of ICM in this blog post is a result of investigation performed on R12.1.3. I believe it behaves the same way in R12.1 and probably even in R12.0 and 11i, but I didn’t check. MOS Article ID 134007.1 which contains the cmclean.sql script is valid for Applications versions 10.7 to 12.1.3 – be careful with it independently of the version of your e-Business Suite installation.

Performance Settings of Concurrent Managers

2013-04-30T07:52:45-05:00

This blog post was originally released on Pythian blog

This is the second article in a series about internals and performance of concurrent managers. In this post, we’ll take a look at three important settings that affect the performance of the concurrent managers: number of processes, “sleep seconds”, and “cache size”. This article might be a bit on the theoretical side, but it should provide a good understanding of how these settings actually affect the behavior and performance of concurrent managers. Most of the statements in this article build off of information from my previous post: The Internal Workflow of e-Business Suite Concurrent Manager Process. It may be helpful to take a look at it before continuing with this one.

Life cycle of a Concurrent Request

The interesting thing about tuning concurrent managers is the fact that we don’t tune a particular query or a running process, but we actually tune the pending time of concurrent requests. The goal of the tuning is to make sure concurrent requests start executing soon enough after the time they have been scheduled for. Let’s take a look at the life cycle of a concurrent request:

The Life cycle of a Concurrent Request (CR)

Based on the diagram above, the pending time of the request is the interval between the time the request was scheduled to start and the time it actually started. This time can be split in two parts:

Pending for Conflict Resolution Manager (CRM) - Here the CRM checks the incompatibility rules effective for the pending concurrent request against other running requests. The CRM allows the request to execute only when all incompatible requests have completed.
Pending for Concurrent Manager (CM) - This is the time spent waiting for an available concurrent manager process. It also includes the time the CM process takes to fetch the request from FND_CONCURRENT_REQUESTS table and to start executing it. “Pending for CM” is the interval that can be tuned by altering the number of manager processes, “sleep seconds” and the “cache size” settings.

Requirements

Understanding the requirements is a mandatory step for any tuning attempt. Otherwise, it’s hard to know when to stop tuning as can become unclear if the performance is sufficient or not. When you’re tuning concurrent managers the requirements can be defined by answering a simple question: How long is the request allowed to stay pending after the scheduled start? You should also keep in mind the following items while thinking about the answer:

Be realistic - “Requests should start immediately” is not a valid answer. It’s simply not possible because of how concurrent managers work. If there is anything you have to run immediately - concurrent programs are not the correct way of doing it.
Think of groups of concurrent programs - If you have a similar requirements for a group of requests (i.e. a program printing invoices while the customer is waiting on-site should start executing in 10 seconds after it’s submitted), a dedicated concurrent manager should be implemented for them.
Unsure of requirements? - If the requirements are not known, ask the end users if they have experienced situations when requests stay in the queue for too long. Was it because the managers couldn’t cope with the amount of incoming requests? If so, the settings might be too low.
Work Shifts - If the requirements differ at different times of the day, concurrent manager Work Shifts can be used to define different settings depending on the time of the day.

Settings

The settings related to concurrent managers are explained in the documentation: Oracle E-Business Suite System Administrator’s Guide - Configuration. But I find the explanations are too often unclear to effectively tune the concurrent managers. In this chapter I’ll reveal the basic principles behind each of the three main settings and will describe how I utilize them to control the performance of concurrent managers.

Number of Processes

From the Documentation: “The number of operating system processes you want your work shift to run simultaneously. Each process can run a concurrent request.”

It’s important to add that there is no coordination between processes of a concurrent manager. If all concurrent processes fetched from the FND_CONCURRENT_REQUESTS table at exactly the same time - they all would read exactly the same information about pending requests. A simple mechanism of row locking is utilized later to allow execution of a concurrent request on a single manager process.
Choosing the correct number of concurrent processes is not easy as the workloads are not constant and the number of pending requests can vary. Here are some clues you should consider (I’m also planning a future blog post about measuring the actual utilization levels of concurrent managers):
- Don’t configure too many processes! Your hardware is limited. If you configure too many processes you’ll exhaust the server resources. For example, if you run lots of big Oracle Reports (CPU-intensive workload) and you have 4 CPUs on your only web/forms/concurrent node, configuring 8 processes for the concurrent manager is risky. If 8 reports where to be generated at the same time, the users could experience web/forms slowness. The same applies to the database tier. Additionally, if too many processes are configured they can use a significant amount of resources even when they are idle. This is especially important for RAC configurations. (This too will be explained in one of my upcoming blog posts)
- Don’t configure too few processes! If the number of processes is insufficient, the requests will start queuing. If that becomes an issue, increase the number of processes slightly. Consider defining a Work Shift with a different number of processes at different times of the day to accommodate your requirements.
- Don’t be afraid of queuing! Queuing is normal, especially if all requests still manage to start as expected based on the requirements. If you see recurring queuing at particular times, check with the users if that is going to be a problem.
- Start low! If you’re unsure of what setting should be used, start with a low number of processes. Check if any queueing occurs and if the users start complaining.

Sleep Seconds

From the Documentation: “The sleep time for your manager during this work shift. Sleep time is the number of seconds your manager waits between checking the list of pending concurrent requests (concurrent requests waiting to be started)”

The documentation is inaccurate - It should clearly state that it’s effective for each manager process - i.e. if you have 5 concurrent manager processes for the Standard Manager and the “Sleep Seconds” setting is set to 30, then the average time between checks for pending requests (if all managers are idle) is 30 / 5 = 6 seconds.
A manager process sleeps only when there are no pending requests - The manager process checks the requests queue immediately after it has processed the last request it fetched. A common misconception is that if the rate of incoming requests is very high, the “Sleep Seconds” should be low to process all of them quickly. Not true! If the rate of incoming requests is high, there is a good chance some requests will be executing at any given time. So, when they complete, the requests queue will be checked immediately and the new requests will be started.
What value to use? - Calculate it! Three parameters are important to estimate the “Sleep Seconds” (S) setting: the number of manager processes (N), the average utilization level (U) of concurrent managers (this setting will be explained later), and the average time of how long the request is allowed to be pending (T). As “Sleep Seconds” are effective only for the idle processes, it can be calculated using the following: S = N * (1 - U) * T.
- Example 1: if N = 5 processes, U= 20%, T=20 seconds - let’s calculate the “Sleep Seconds” setting: S = 5 * (1 - 0.2) * 20 = 5 * 0.8 * 20 = 80 seconds. It seems high, but think about it - if the average utilization of 5 processes is 20%, then there are 4 idle processes at any given time. Each of these will have a sleep interval of 80 seconds, so on average the requests queue will be checked every 20 seconds.
- Example 2: if N = 3 processes, U= 90%, T=20 seconds: S = 3 * (1 - 0.9) * 20 = 3 * 0.1 * 20 = 6 seconds. This example reveals a problem as the calculated “Sleep Seconds” are lower than the requirement we have set - this means the requirement can’t be reached with the number of running processes. Think about it - we have 3 processes each utilized 90% of time, it’s impossible to meet the 20 seconds goal because most of the time all managers are busy. There simply aren’t enough processes to execute the incoming requests. The defined requirements can be reached only if at least one manager process is idle - This scenario also describes a “perfect world”, where all but one manager is busy, so all new requests are picked up in time and the processing overhead of the idle manager processes is minimal.

Cache Size

From the Documentation: “The number of requests your manager remembers each time it reads which requests to run”

Almost useless setting - Unless you have a manager with only one running process. If multiple manager processes are running, there is a good chance that most of the cached requests will be processed (remember, the processes don’t coordinate the work - they compete the work) by other manager processes while the first request is running.
- Example, there are 10 manager processes and 10 requests are submitted. One of the manager processes starts executing the 1st request, the other managers start running the remaining requests. So, by the time the request completes all the cached requests will be obsolete, but the manager process will try to lock the corresponding rows in FND_CONCURRENT_REQUESTS table anyway, and will fail for all 9 requests. It will then immediately query the queue to check if more requests are pending.
- I think it’s best to set “Cache Size” setting to 1 so the hardware resources aren’t spent on trying to lock the processed requests, but rather on checking the requests queue.
Request priorities are cached too - If you have a cache size greater than 1, keep in mind the request priorities are cached too. If the priority is changed for a cached request, the manager process will not notice it.

Summary

I have to admit this article turned out to be much more complicated to write than I expected, even if there are just a few settings to describe. The problem is that the concurrent processing environment is changing all the time. At one moment it’s completely idle, and then suddenly there is a spike of incoming requests. It makes it impossible to tune for all situations. There will always be instances when any configuration works better and others when it makes things worse, but I hope I was able to outline the significance of each configurable parameter so the overall picture is clearer.

As I promised, two other articles are lined up for the future - one to describe the actual utilization levels of concurrent managers and the other to look into overhead of an idle concurrent processing environment. Stay tuned!

Which Partitions Does The SQL Statement Access?

2013-04-05T08:03:50-05:00

This blog post was originally released on Pythian blog.

This is a short blog post on how one can prove that a particular partition of the table or index is accessed by a specific query.

The solution to this question seemed interesting enough to share with you all. Here’s a short description of what I had:

A SQL with sql_id = 9kuvj1g38hja2 is the TOP 1 statement in “User I/O” section of AWR report.
The statement queries data from multiple tables. One of them is a fairly large table named “ALERT”. It is range partitioned by date, and each year has it’s own partition (Y2013, Y2012, …).
The SQL takes 5 – 10 seconds to complete and is executed 500 – 5000 times each hour (sometimes even ~10 of them are running simultaneously).
~95% of elapsed time is spent on db file sequential reads.
Diagnostic pack is licensed (it is required for this method as ASH is used).

We also had a statement so that none of the queries should be reading data older then 1 year, but looking at the statistics I suspected there was something wrong. So here’s how I found out older data from the ALERT table was accessed:

As the query is executed very often, it has many appearances in ASH.
As the statement does lots of db file sequential reads, there are many p1 (file_id) and p2 (block_id) combinations available from ASH.
file_id and block_id can be joined to DBA_EXTENTS to find out the segment which the block belongs to. That is where the partition names being accessed by the SQL where found out.

The challenge with this approach was the fact that I had more then 100K samples for 9kuvj1g38hja2 waiting on db file sequential reads in DBA_HIST_ACTIVE_SESS_HISTORY, and there were millions of extents in the database. How do I join them efficiently? I can fetch only the segments we’re interested in from DBA_EXTENTS before joining it all to DBA_HIST_ACTIVE_SESS_HISTORY (Thanks Yury for this great hint)! So this is the query that solved the mystery:

with
exts as (select /*+ materialize */ partition_name, file_id, block_id, blocks
         from dba_extents where owner='SECRET' and segment_name='ALERT'),
stats as (select /*+ materialize */ p1, p2, count(*) cnt
          from DBA_HIST_ACTIVE_SESS_HISTORY where sql_id='9kuvj1g38hja2'
           and event='db file sequential read' group by p1, p2)
select partition_name, sum(s.cnt) read_count
  from exts e, stats s
 where e.file_id=s.p1
   and s.p2 between e.block_id and e.block_id+e.blocks-1
 group by partition_name
 order by 2;

It’s short and simple, and it also answered the question I asked. Is old data accessed by the SQL statement? Yes.

PARTITION_NAME                 READ_COUNT
------------------------------ ----------
Y2006                                 111
Y2007                                5645
Y2008                                5698
Y2013                               10388
Y2009                               13473
Y2010                               21600
Y2011                               37186
Y2012                               56184

 8 rows selected

Be careful when interpreting the results! If the SQL does not report a partition as being accessed, it doesn’t mean it was not accessed. ASH is based on sampling – maybe we got “lucky” and samples didn’t cover reads from that partition? Maybe all blocks belonging to the partition were in buffer cache and no reads were needed? But, if the SQL reported a partition as being accessed, we can be 100% sure it really was, and for me that was enough.

The Internal Workflow of e-Business Suite Concurrent Manager Process

2013-03-22T06:45:06-05:00

This blog post was originally released on Pythian blog

Concurrent processing is one of the key elements of any e-Business Suite system. It provides scheduling and queueing functionality for background jobs and it’s used by most of the applications modules. As many things depend on concurrent processing it’s important to make sure the configuration is tuned for your requirements and hardware specification.

This is the first article in a series about performance of concurrent processing. We’ll take a closer look at the internals of concurrent managers, the settings that affect their performance and the ways of diagnosing performance and configuration issues. Today we’ll start with an overview of the internal workflow of a concurrent manager process. Enjoy the reading.

There are few things that need to be clear before we start:

This is only about how the concurrent processing framework (the concurrent managers) works and it’s not about the concurrent requests executed in the framework.
There are multiple types of concurrent managers in eBS - internal manager, conflict resolution manager, workflow agent listener service, standard manager, etc. Their roles are different and in this post I’ll discuss only managers that pick up scheduled concurrent requests from the “queue” and execute them - or specifically all managers that have type of “Concurrent Manager” set in Concurrent managers’ definition form. Typical examples of these managers are “Standard Manager”, “Inventory Manager” and probably your own custom concurrent managers created for processing specific types of concurrent requests.

Concurrent Managers Form

It is important to understand the internal workflow of a concurrent manager because otherwise it’s hard to realize how a configuration change actually affects the system. Several years ago I had to implement an online change of a specialization rule and it triggered a bounce of all Standard Manager processes - that’s when I realized I had to understand how it worked and since then I have spent lots of hours looking into internals of concurrent managers. I’m not saying everything is 100% clear for me now - there are too many little things that matter in certain situations. This series of posts will be more about concepts and I hope you’ll find it useful.

So how does a concurrent manager process work? Here is a diagram I created to explain it:

Internal workflow of a concurrent manager process

I’ve numbered each step of the diagram to provide a bit more details about them:

This is where the story begins. There is no EXIT state in the diagram as the managers normally process requests in an infinite loop. Obviously, there is a way for a concurrent manager process to receive the command to quit when the managers need to be shut down, but that’s not included here for simplicity.
Internal Concurrent Manager (ICM) requests the Service Manager (FNDSM) to start up the Concurrent Manager process. For the Standard Manager processes the binary executable FNDLIBR is started, for the Inventory Manager it’s INVLIBR, and there are others too.
The manager process connects to the database and reads the settings (e.g profile options, sleep seconds, cache size).
The process saves information about itself in FND_CONCURRENT_PROCESSES table (os process id, database name, instance name, DB session identifiers, logfile path and name and others). It also updates FND_CONCURRENT_QUEUES by increasing the value of RUNNING_PROCESSES.
The concurrent manager process collects information from the database to build the SQL for querying the FND_CONCURRENT_REQUESTS table. The query will be used every time the manager process will look for scheduled concurrent requests. This is the only time the manager process reads the Specialization Rules (which programs it is allowed to execute and which not) from the database. Keep in mind, if the specialization rules are changed while the managers are running, they are bounced without warning as that is the only way to update the specialization rules cached by the manager process.
The SQL (from step 4) is executed to collect information about pending concurrent requests from FND_CONCURRENT_REQUESTS table.
The results are checked to verify if any requests are pending for execution.
If no requests are pending for execution the manager process sleeps and then goes to step 5. The “Sleep Seconds” parameter of the “Work Shifts” settings of the concurrent manager determines how long the process sleeps before FND_CONCURRENT_REQUESTS table is queried again. This is the only time the “sleep seconds” setting is used.
If there is at least one concurrent request pending for execution the concurrent manager process caches rowids for the FND_CONCURRENT_REQUESTS rows of pending concurrent requests. The “Cache Size” setting of the concurrent manager specifies how many rowids to cache.
The cached list of rowids is checked to verify if there are any unprocessed concurrent requests (rows in FND_CONCURRENT_REQUESTS table) left .If none are left - the processing returns to step 5 and the FND_CONCURRENT_REQUESTS table is queried again.
The next unprocessed rowid is picked from the process cache and the processing starts.
Concurrent manager process executes a SELECT-for-UPDATE statement to lock the STATUS_CODE in FND_CONCURRENT_PROCESSES for the request it’s about to process. This is the mechanism to ensure that each concurrent request is executed only once and only by one manager process even if many processes are running simultaneously. The SELECT-for-UPDATE statement can complete with “ORA-00054: resource busy and acquire with NOWAIT specified” or “0 rows updated” if another manager process has started processing the request already.
If the STATUS_CODE of the request was locked successfully the concurrent manager executes the concurrent request. The processing moves to step 9 where the cached list of concurrent requests (rowids) is being checked again.

The workflow is not very complex, but it’s important to remember there are normally multiple concurrent manager processes running at the same time and they are competing for the requests to run. This competition introduces another dimension of tuning for settings, like number of concurrent manager processes, sleep seconds or cache size. Stay tuned for the next post in the series to find out more!

The Easy Way of Finding Similar SQL Statements

2013-01-24T18:48:28-06:00

This blog post was originally released on Pythian blog

As we all know proper use of bind variables in SQL statements is a must to make transaction processing applications scalable. So how do we find the queries that don’t use bind variables and have to be parsed each time they are executed? There is number of ways, but this article is all about the most effective way I know. If you have a better one - let me know please!

I still remember the days when I used the method given us by Tom Kyte on asktom.oracle.com - and it worked perfectly! The basic idea was to capture all SQL statements from v$sqlarea, remove constants from the SQL text and then count the occurrences of the same SQL. Would I use the same method now? Unlikely. The post is antique - it was written back in year 2000, the database software has evolved since then and we’ve been given more effective means to achieve similar results.

Starting with 10gR2, two interesting columns where introduced in number of views and tables - EXACT_MATCHING_SIGNATURE and FORCE_MATCHING_SIGNATURE. I could find them in at least V$SQLAREA, V$SQL, STATS$SQL_SUMMARY, DBA_HIST_ACTIVE_SESS_HISTORY, DBA_HIST_SQLSTAT and I bet there are more. As you see, they are present all over the place - Shared Pool, ASH, AWR, Statspack (This means we have a good choice of sources to look for problematic SQLs)

Documentation says EXACT_MATCHING_SIGNATURE is “Signature calculated on the normalized SQL text. The normalization includes the removal of white space and the uppercasing of all non-literal strings” and FORCE_MATCHING_SIGNATURE is “Signature used when the CURSOR_SHARING parameter is set to FORCE”. It also appears the signature is just another hash value calculated from SQL statement, but this time it’s a hash of normalized (removed spaces, etc.) SQL statement. Lets’s have a look!

TEST@TEST> select 1 from dual where DUMMY='B';

no rows selected

TEST@TEST> select 1 from dual where DUMMY='A';

no rows selected

TEST@TEST> select 1 from dual where DUMMY= 'A';

no rows selected

TEST@TEST> select 1 from dual where dummy=  'A';

no rows selected

TEST@TEST> col exact_matching_signature for 99999999999999999999999999
TEST@TEST> col sql_text for a50
TEST@TEST> select sql_id, exact_matching_signature, SQL_TEXT from v$sqlarea where UPPER(sql_text) like '%DUMMY%' order by UPPER(sql_text);

SQL_ID           EXACT_MATCHING_SIGNATURE SQL_TEXT
------------- --------------------------- --------------------------------------------------
6vum4z2c1rpua        13015969835749972382 select 1 from dual where dummy=  'A'
b8fj5dkrqzkrq        13015969835749972382 select 1 from dual where DUMMY= 'A'
gfrsz0vuczzag        13015969835749972382 select 1 from dual where DUMMY='A'
18k1ys5nhrrbk         1525540498770831959 select 1 from dual where DUMMY='B'
ckzhurpxb9utu         5788880286169998087 select sql_id, exact_matching_signature, SQL_TEXT
                                          from v$sqlarea where UPPER(sql_text) like '%DUMMY%
                                          ' order by UPPER(sql_text)

It’s easy to notice all SQL_IDs are different, but EXACT_MATCHING_SIGNATURE is the same for 3 of statements because of normalization.

As noted before, FORCE_MATCHING_SIGNATURE is calculated from SQL text as if CURSOR_SHARING would be set to TRUE (you don’t have to set it to TRUE to get the signature values). CURSOR_SHARING=FORCE forces SQL statements to share cursors by replacing constants with bind variables, so all statements that differ only by constants share the same cursor. Let’s have a look at FORCE_MATCHING_SIGNATURE values for the same SQLs:

TEST@TEST> col force_matching_signature for 99999999999999999999999999
TEST@TEST> select sql_id, force_matching_signature, SQL_TEXT from v$sqlarea where UPPER(sql_text) like '%DUMMY%' order by UPPER(sql_text);

SQL_ID           FORCE_MATCHING_SIGNATURE SQL_TEXT
------------- --------------------------- --------------------------------------------------
6vum4z2c1rpua        13154199455204052618 select 1 from dual where dummy=  'A'
b8fj5dkrqzkrq        13154199455204052618 select 1 from dual where DUMMY= 'A'
gfrsz0vuczzag        13154199455204052618 select 1 from dual where DUMMY='A'
18k1ys5nhrrbk        13154199455204052618 select 1 from dual where DUMMY='B'
ckzhurpxb9utu         8230152823949618578 select sql_id, exact_matching_signature, SQL_TEXT
                                          from v$sqlarea where UPPER(sql_text) like '%DUMMY%
                                          ' order by UPPER(sql_text)

2fxmcn8hvv59p         8805530522791470645 select sql_id, force_matching_signature, SQL_TEXT
                                          from v$sqlarea where UPPER(sql_text) like '%DUMMY%
                                          ' order by UPPER(sql_text)

I think you know what happens next - finding similar statements becomes as easy as querying the chosen data source (shared pool, AWR, ASH, Statspack) and grouping statements by FORCE_MATCHING_SIGNATURE. Here’s an example for finding one of the top statements not using bind variables properly:

TEST@TEST> col force_matching_signature for 99999999999999999999999999
TEST@TEST> select * from (select force_matching_signature, count(*) &quot;Count&quot; from v$sqlarea group by force_matching_signature order by 2 desc) where rownum<=3;

 FORCE_MATCHING_SIGNATURE        Count
--------------------------- ----------
 3832233612528870918             13251
 7415896326081021278              1772
 12487066559404946962             1642

TEST@TEST> set long 99999
TEST@TEST> select sql_fulltext from v$sql where force_matching_signature=7415896326081021278 and rownum=1;
SQL_FULLTEXT
--------------------------------------------------------------------------------
select account_id from accountequip where accountequip.equipment_id = 1279768275
448 and accountequip.uninstalltime is null

R12.2 online patching – What are the hidden costs?

2012-01-09T11:32:35-06:00

This blog post was originally released on Pythian blog

One of the hot topics at the UKOUG 2011 Technology and E-Business Suite Conference last December was the upcoming release of Oracle e-Business Suite R12.2. The new release will bring us lots of new features, usability improvements and new versions of technology stack components (Oracle Database 11g R2 and Oracle Fusion Middleware 11g R1 as the application server), but the most important and impressive new feature of course will be online patching. Online patching is supposed to change the game completely. All owners of E-Business Suite environments know that patching requires downtime. Although it can be reduced with various techniques (e.g. staged APPL_TOP), some downtime is still required to apply a number of changes. Online patching will not eliminate downtime completely, but will reduce it significantly by using “Edition Based Redefinition” (EBR) at the database level and using a secondary applications file system for online patching. In fact, all patching activity will be an online operation; downtime will be required only to switch from one version to another.

I attended a great presentation at UKOUG 2011, “Oracle E-business Suite’s use of edition-based redefinition for online patching,” delivered by Bryn Llewellyn from Oracle. The presentation provided thorough insight on how EBR works and how E-Business Suite will be able to use it for online patching. It answered many questions, but raised a few more questions and concerns. I’ll try to summarize in this post the main concerns I’ve been thinking about after attending this presentation. At this point I have to stop and add a disclaimer. Actually, it’s a double disclaimer, because there was a disclaimer on the 1st slide of the presentation saying something like “don’t take it for granted” as we were discussing features of a product that is not yet released. Since I’m giving you my thoughts on a product which I haven’t seen, here it is: “don’t take it for granted”^2!

Limitations of EBR

Before we move on, I have to give a short background on a few important specifics of EBR, which are taken from the EBR documentation and were also discussed during the presentation (You can read full documentation in Oracle® Database Advanced Application Developer’s Guide Chapter 19):

All schema objects can be divided in 2 groups, editionable and non-editionable:
- Editionable objects are synonyms (except public synonyms), views, and all PL/SQL object types (functions, libraries, packages and package bodies, procedures, triggers, types and type bodies).
- All other objects are non-editionable (e.g. tables, indexes, materialized views, database links, etc.)
Editionable objects can be editioned or potentially editioned
- Editioned editionable objects reside in an editions-enabled schemas;
- Editionable objects residing in schemas that are not editions-enabled are potentially editioned (I really don’t like this term, as these objects are in fact non-editioned)
All non-editionable objects are always non-editioned (no matter what schema they reside in)
IMPORTANT RULE! A non-editioned (and based on my experiments also potentially editioned) object cannot depend on an editioned object. For example:
- A function-based index cannot depend on an editioned function.
- A materialized view cannot depend on an editioned view.
- A table cannot have a column of a user-defined data type (collection or ADT) whose owner is editions-enabled.
- A materialized view cannot depend on editioned function

Basically, the most important message here is that there are new restrictions introduced on dependencies between different objects.

EBR, R12.2, online patching and our customizations.

As described before, “a non-editioned object cannot depend on an editioned object”. Think about it! I’ve seen lots of materialized views depending on functions, packages or views, and tables with user-defined data types.

Oracle will need to remove as much of these dependencies as possible (as they need as many editioned objects as possible to have a way to change them online); otherwise the online patching would be too limited (which I really hope it won’t be). The presenter also mentioned that a new schema APPS_NE (APPS Non-Editioned) would be introduced in R12.2 to hold potentially editioned objects for which these dependencies would not be removed. Oracle will take care of these dependencies by separating the editionable objects into 2 schemas instead of one– they will make it work for their objects, don’t worry.

Problem 1: What I’m worried about are customizations – I’ve seen quite a few environments where custom materialized views are created depending on views and functions. As the views and functions will likely become editioned in R12.2, these custom materialized views will become invalid. What options do we have?

We can get lucky and it might turn out that the dependent objects will become non-editioned in schema APPS_NE. In this case, we’d only need to change the schema for the referenced object (remember, APPS synonyms will be editioned);
We might need to create duplicate functions in an editions-disabled schema to reference in our custom materialized views If the function we are duplicating is seeded, we will likely miss the moment it changes, so there is an increased risk to get wrong data from the MV. (By the way, creating a non-editioned wrapped function for editioned function is not allowed);
We might need to rewrite our materialized views to depend on tables directly (not synonyms or views), but EBS also brings editioning views that allow access to “different versions of data” from the same table. MVs cannot depend on editioning views – so our MVs might require changes whenever the table is modified and a new version of an editioning view is created.

Similar problems could also be introduced with other customized non-editioned objects.

Problem 2: Online patching might not be as effective as advertised. I’m basing this statement on the information that there would be another schema – APPS_NE – introduced for non-editioned (or potentially editioned) objects. Online patching would not be able to patch these objects – some downtime would be required here as I understand, but probably Oracle has some tricks in mind for this. Also, RDBMS patches would still need downtime.

Problem 3: Based on the information available, there would be quite a lot of reorganization going on in seeded schemas, and I still haven’t seen a good document explaining what is going to be changed, and how we should prepare for the changes. This will definitely affect customizations, but there’s still no guideline released on how to make customizations ready for R12.2. Additionally, I think the upgrade to R12.2 might require much downtime to reorganize all the schemas.

Despite all the concerns and unknown variables, I’m very excited to see Oracle trying to do something that has never been done before! Did you know the main reason they started creating the Edition Based Redefinition feature was the requirement from the ERP team to implement the online patching feature? As the EBR was specifically designed to allow online patching for Oracle e-Business Suite, I hope they will have gotten all the gears right to implement it efficiently enough. R12.2 should be out in 11 months, and I really can’t wait to get my hands dirty with it.

UKOUG2011 E-Business Suite Concurrent Processing Performance Analysis

2011-12-05T07:03:50-06:00

This blog post was originally released on Pythian blog

Another December has started and you know what it means right? It’s the Christmas time? NO! It’s time for UKOUG Technology and e-Business Suite 2011 conference! There are not many things that can make a passionate Apps DBA more excited than that. UKOUG Tech & eBS conferences are the only ones in Europe that have such a rich content for Application DBAs and this year it will not be different.

For me, the UKOUG Conference 2011 is going to be a very special one. I have some really good reasons for that: For the 1st time I’m representing Pythian at an international conference, I’m proud of that, but it also adds some additional pressure on my shoulders. It’s the 5th time I’m presenting at UKOUG conference (a little anniversary for me). And, I’m meeting few of my colleagues in person, which is really important as I’m working remotely from home and therefore miss seeing people and this is a good chance for me to catch up a bit on that.

How did I get here? Even though there are many eBS sessions during these conferences I think there’s not enough content specifically for Apps DBAs, there are a lot of presentation usually surrounding upgrading 11i to R12, R12 to Fusion Apps, etc., there’s always some presentations on how to integrate eBS with this and that, there’s almost always some presentations on how to configure something and presentations about what’s on the roadmap for us. In my opinion there are not enough presentations for Apps DBAs that would give you something that would be useful every day, something you can use when you go back to work after the conference. Therefore I thought I needed to improve the situation and submitted a presentation “Concurrent Processing Performance Analysis for Apps DBAs” and fortunately enough it got accepted.

If you are an apps DBA, please come and see the presentation on Wednesday, December 7th in Hall 10B at 14:25. I’ll give you the tools and knowledge to analyze the concurrent processing performance on your own. You will know how to find answers for questions like these after the presentation:

Are there any periods of time when the pending times of the concurrent requests are longer than expected?
Which concurrent programs are affected the most by increased pending times
How to find out which requests exactly were pending to be executed by the manager?
What was the root cause the request was pending (incompatibility with other requests or all of the managers were occupied)
Which were the conflicting requests?
Which managers could have run the requests?
Which requests were processed on these concurrent managers at that time?
What was the utilization level of particular concurrent managers?
Are there too many, too few concurrent processes configured?

I hope to see you on Wednesday if you interested!

Back to School: Elementary Physics for DBAs

2011-10-07T06:31:59-05:00

This blog post was originally released on Pythian blog

“Hello World!” I guess that’s the most appropriate way to start my 1st blog post under pythian.com domain. I’m going to start slow, but hopefully will pick up speed and have at least couple of posts each month to share with you. I’ve been blogging at http://appsdbalife.wordpress.com/ until now and I haven’t decided yet what the future will be for my previous blog, I wouldn’t like it to become some kind of a zombie page that’s been long dead but still wandering around the internet world.

Enough intros, let’s get to business! I hope this blog post doesn’t get lost in the huge amount of posts related to OOW 2011.

A few days ago I was asked to estimate how much space needed to be added to the ASM diskgroup to handle the database growth for one year without additional need of adding disks. Obviously, to estimate the disk space that needed to be added I had to know what the DB size will be in one year from now. It looked like an easy task as I knew we were recording the size of the database every day.

I checked the size of the database – it was 887.3Gb and a month ago the size of the database was 856.9Gb, so it had increased by 30.4Gb in one month. 30.4Gb multiplied by 12 months equals 364.8Gb, so the database will be ~1252Gb after one year. That was easy right?

But wait a second! Wouldn’t it be logical that as the DB becomes bigger (there is more activity; there are more users and more transactions, more everything) then it grows quicker too? So I checked how much the DB increased during one month exactly one year ago – 21.2Gb (the DB size was 740.3Gb at that time). Looking at this number I instantly knew the DB size I calculated earlier was not correct.

This was the moment where physics kicked in. Velocity, distance and time could easily be substituted for DB growth in one month, size of the DB and time. Also, as the “velocity” has increased during one year we deal with some acceleration too. I graduated high school 10 years ago, so I had to look up the formulas. At first I needed to calculate acceleration and assuming the acceleration stays the same I would be able to calculate the “distance” or (how much the DB would grow in one year). I used the following formulas (where s = distance, u = initial velocity, v = final velocity, a = uniform acceleration, t = time):

v^2 = u^2 + 2as
s = ut + (at^2)/2

I applied the 1st formula on the variables I had (v=30.4 Gb/month, u=21.2 Gb/month, t=12 months, s=887.3-740.3=147 Gb), so the results were:

v^2 = u^2 + 2as
30.4^2 = 21.2^2 + 2a147
924.16 = 449.44 + 294*a
474.72 = 294*a
a = ~1.6147 (Gb/month^2)

As I had the acceleration of the DB growth I could calculate the estimated growth of the database during one year (u=30.4 Gb/month, t=12 months, a = ~1.6147 Gb/month^2), and the result was:

s = ut + (at^2)/2
s = 30.412 + (1.614712^2)/2
s = 364.8 + 1.6147*144/2
s = 364.8 + 116.2584
s = ~481.1 Gb

Based on these calculations the size of the DB would be 887.3 + 481.1 = 1368.4Gb, which is quite different from 1252Gb that I calculated before. I used the estimated DB size of 1368.4GB to calculate how much disk space needed to be added to the ASM disk group in the end.

Of course, the acceleration of the DB growth might not be uniform in reality but I think it’s still better to use this, as it’s likely to be more accurate compared to result based on the assumption that the DB growth is constant. You can do the same calculations for different types of data – it is not bound to the database, use it to estimate the total size of apache logs or the total size of concurrent request output files if you’re running e-Business Suite. Keep in mind the estimates are only estimates don’t rely on them 100% and they are going to be more accurate if the input data are accurate.

I hope you enjoyed reading this. See you next time!