I wrote a developerWorks article on finding problem SQL in your package cache. But I refine and play with my SQL over time, so I thought I’d share the version I’ve been using recently.

Purpose

To find problem SQL in the Package Cache across several importance performance categories. Computes how bad a particular statement is in comparison to other statements in the package cache by computing what percentage of a certain resource is used by a particular query.

Version

This SQL has been tested on DB2 9.7 fixpack 9a. It should work on any version of DB2 from 9.7 up. See below for syntax that works on DB2 9.1

Statement #1 – DB2 9.7 and up

WITH SUM_TAB (SUM_RR, SUM_CPU, SUM_EXEC, SUM_SORT, SUM_NUM_EXEC) AS (
        SELECT  FLOAT(SUM(ROWS_READ)),
                FLOAT(SUM(TOTAL_CPU_TIME)),
                FLOAT(SUM(STMT_EXEC_TIME)),
                FLOAT(SUM(TOTAL_SECTION_SORT_TIME)),
                FLOAT(SUM(NUM_EXECUTIONS))
            FROM TABLE(MON_GET_PKG_CACHE_STMT ( 'D', NULL, NULL, -2)) AS T
        )
SELECT
        SUBSTR(STMT_TEXT,1,10) as STATEMENT,
        ROWS_READ,
        DECIMAL(100*(FLOAT(ROWS_READ)/SUM_TAB.SUM_RR),5,2) AS PCT_TOT_RR,
        TOTAL_CPU_TIME,
        DECIMAL(100*(FLOAT(TOTAL_CPU_TIME)/SUM_TAB.SUM_CPU),5,2) AS PCT_TOT_CPU,
        STMT_EXEC_TIME,
        DECIMAL(100*(FLOAT(STMT_EXEC_TIME)/SUM_TAB.SUM_EXEC),5,2) AS PCT_TOT_EXEC,
        TOTAL_SECTION_SORT_TIME,
        DECIMAL(100*(FLOAT(TOTAL_SECTION_SORT_TIME)/SUM_TAB.SUM_SORT),5,2) AS PCT_TOT_SRT,
        NUM_EXECUTIONS,
        DECIMAL(100*(FLOAT(NUM_EXECUTIONS)/SUM_TAB.SUM_NUM_EXEC),5,2) AS PCT_TOT_EXEC,
        DECIMAL(FLOAT(STMT_EXEC_TIME)/FLOAT(NUM_EXECUTIONS),10,2) AS AVG_EXEC_TIME
    FROM TABLE(MON_GET_PKG_CACHE_STMT ( 'D', NULL, NULL, -2)) AS T, SUM_TAB
    WHERE DECIMAL(100*(FLOAT(ROWS_READ)/SUM_TAB.SUM_RR),5,2) > 10
        OR DECIMAL(100*(FLOAT(TOTAL_CPU_TIME)/SUM_TAB.SUM_CPU),5,2) >10
        OR DECIMAL(100*(FLOAT(STMT_EXEC_TIME)/SUM_TAB.SUM_EXEC),5,2) >10
        OR DECIMAL(100*(FLOAT(TOTAL_SECTION_SORT_TIME)/SUM_TAB.SUM_SORT),5,2) >10
        OR DECIMAL(100*(FLOAT(NUM_EXECUTIONS)/SUM_TAB.SUM_NUM_EXEC),5,2) >10
    ORDER BY ROWS_READ DESC FETCH FIRST 20 ROWS ONLY WITH UR;

Statement #2 – for DB2 9.1 and 9.5

WITH SUM_TAB (SUM_RR, SUM_CPU, SUM_EXEC, SUM_SORT, SUM_NUM_EXEC) AS (
        SELECT  FLOAT(SUM(ROWS_READ)),
                FLOAT(SUM(TOTAL_USR_CPU_TIME)) + FLOAT(SUM(TOTAL_SYS_CPU_TIME)),
                FLOAT(SUM(TOTAL_EXEC_TIME)),
                FLOAT(SUM(TOTAL_SORT_TIME)),
                FLOAT(SUM(NUM_EXECUTIONS))
            FROM SYSIBMADM.SNAPDYN_SQL AS T
        )
SELECT
        SUBSTR(STMT_TEXT,1,10) as STATEMENT,
        ROWS_READ,
        DECIMAL(100*(FLOAT(ROWS_READ)/SUM_TAB.SUM_RR),5,2) AS PCT_TOT_RR,
        TOTAL_USR_CPU_TIME + TOTAL_SYS_CPU_TIME as TOTAL_CPU_TIME,
        DECIMAL(100*((FLOAT(TOTAL_USR_CPU_TIME)+FLOAT(TOTAL_SYS_CPU_TIME))/SUM_TAB.SUM_CPU),5,2) AS PCT_TOT_CPU,
        TOTAL_EXEC_TIME,
        DECIMAL(100*(FLOAT(TOTAL_EXEC_TIME)/SUM_TAB.SUM_EXEC),5,2) AS PCT_TOT_EXEC,
        TOTAL_SORT_TIME,
        DECIMAL(100*(FLOAT(TOTAL_SORT_TIME)/SUM_TAB.SUM_SORT),5,2) AS PCT_TOT_SRT,
        NUM_EXECUTIONS,
        DECIMAL(100*(FLOAT(NUM_EXECUTIONS)/SUM_TAB.SUM_NUM_EXEC),5,2) AS PCT_TOT_EXEC,
        DECIMAL(FLOAT(TOTAL_EXEC_TIME)/FLOAT(NUM_EXECUTIONS),10,2) AS AVG_EXEC_TIME
    FROM SYSIBMADM.SNAPDYN_SQL AS T, SUM_TAB
    WHERE DECIMAL(100*(FLOAT(ROWS_READ)/SUM_TAB.SUM_RR),5,2) > 10
        OR DECIMAL(100*((FLOAT(TOTAL_USR_CPU_TIME)+FLOAT(TOTAL_SYS_CPU_TIME))/SUM_TAB.SUM_CPU),5,2)  >10
        OR DECIMAL(100*(FLOAT(TOTAL_EXEC_TIME)/SUM_TAB.SUM_EXEC),5,2) >10
        OR DECIMAL(100*(FLOAT(TOTAL_SORT_TIME)/SUM_TAB.SUM_SORT),5,2) >10
        OR DECIMAL(100*(FLOAT(NUM_EXECUTIONS)/SUM_TAB.SUM_NUM_EXEC),5,2) >10
    ORDER BY ROWS_READ DESC FETCH FIRST 20 ROWS ONLY WITH UR;

Sample Output

STATEMENT  ROWS_READ            PCT_TOT_RR TOTAL_CPU_TIME       PCT_TOT_CPU STMT_EXEC_TIME       PCT_TOT_EXEC TOTAL_SECTION_SORT_TIME PCT_TOT_SRT NUM_EXECUTIONS       PCT_TOT_EXEC AVG_EXEC_TIME
---------- -------------------- ---------- -------------------- ----------- -------------------- ------------ ----------------------- ----------- -------------------- ------------ -------------
SELECT ACA              5671140      81.71               657030        1.13                 1422         0.76                       0        0.00                 1860         6.92          0.76
WITH SYSIB               865658      12.47              1813570        3.12                 2903         1.56                       4        4.04                  214         0.79         13.56
select sna                62808       0.90                17975        0.03                   29         0.01                      13       13.13                   16         0.05          1.81
CALL SYSIB                 9418       0.13             17493719       30.18                45506        24.50                       0        0.00                 2121         7.89         21.45
SELECT T.t                 6600       0.09               383749        0.66                  617         0.33                      62       62.62                 2200         8.18          0.28
select str                 4353       0.06                52841        0.09                24344        13.10                       2        2.02                    3         0.01       8114.66
CALL SYSIB                   41       0.00             34754421       59.97                61356        33.03                       0        0.00                 2949        10.97         20.80

  7 record(s) selected.

Caveats and Modifications

• The statement text returned is limited to a small size to make the formatting work better here. You’ll want to either expand it so you can see the whole statement, or instead select the EXECUTABLE_ID and then later query mon_get_pkg_cache_stmt to get the STMT_TEXT based on that EXECUTABLE_ID.
• You can change the sort order to match your preference. I like rows_read because it’s most likely to help me identify where I can create an index to help a query.
• This statement only returns rows that consume 10% or more of one of the identified critical resources. You can adjust that percent in the where clause to match a different value or remove items from the where clause if you’re not interested in problem statements in a particular category.
• Output is also limited to 20 rows – this is arbitrary, and you may want to alter it. I usually find my biggest problems within 5 or fewer statements.
• The ‘D’ in the call of the mon_get_pkg_cache_stmt table function limits my output to only dynamic SQL. If you would like to look at both dynamic and static sql, you can use NULL instead

I have worked with DB2 on Windows on and off over the years and have largely not enjoyed it all that much. Most likely because the vast majority of my time is spent on UNIX and Linux systems, so when I end up at a windows command line, my fingers type things like “ls” and “grep” before I can even stop them. I think this is a common condition for DB2 consultants and DBAs – most of us spend the majority of our time on Linux or UNIX or even both and then have to jump into a Windows system and still be proficient.

After a week of spending more time with a Windows database server, I was ready to try just about anything. My husband happens to work with VMWare, and is always talking about PowerShell, so I thought I’d give it a try. I didn’t even realize until I looked into it that I could use a PowerShell command line. This post focuses on the use of PowerShell at the command line with DB2. I’m sure I’ll have follow-on posts about actually scripting in PowerShell

About PowerShell

I am by no means an expert. In my job as a consultant, I have to go into a large number of environments, so it is important that I can work in the “lowest common denominator” of scripting languages. I may love Perl when I have the choice, but Perl requires actual installation for too many systems to be something I can rely upon having. My choices are generally then ksh for Linux/UNIX and in the past, batch for Windows. Starting with Windows2008, PowerShell is available by default. You can also bolt it on to Windows 2003. This makes it qualify for my “lowest common denominator” for new scripts moving forward.

Using PowerShell as Your Command Line

One of the frustrations on Windows for someone used to a command line has always been that you cannot easily just execute DB2 commands from any command line. You generally have to pull up a command window and use db2cmd when executing batch scripts or DB2 commands from batch scripts. It is really easy to pull up and set up PowerShell to work with DB2.

All actions taken here are as the DB2 instance owner.

First to find PowerShell on a Windows server, you can use the search box on the start menu. This is what it looks like to do that on Windows 2008:

Clicking on that top entry will get you into PowerShell. You will then have to configure PowerShell to run DB2 commands using this command:

set-item -path env:DB2CLP -value "**$$**"

Better yet, set it up so that PowerShell automatically does this whenever you bring up PowerShell as this user (the DB2 instance owner). To do this, open up PowerShell. At the PowerShell prompt, do this:

notepad $profile

In the file opened, enter the same set-item command:

set-item -path env:DB2CLP -value "**$$**"

Save and close the file. If you try opening a PowerShell window at this point, you will get this error:

To get around this error, execute the following in your PowerShell window:

Set-ExecutionPolicy remoteSigned

That will ask you a question and looks like this:

Obviously, you should understand the implications of changing this security-related parameter before changing it, particularly on a production system.

At this point, if you create an icon on the desktop for PowerShell by dragging it from your start menu search, and then double click it, you will get a PowerShell prompt that you can execute DB2 commands at.

Useful Basics

List Files

One thing that has really driven me nuts in the past is that to list files, my fingers type “ls -latr” before my brain is even in the loop. In a traditional DB2 command window, this leads to a minor error and a minor swear from me as I engage my brain in the process to remember what the command is on Windows.

I cannot tell you how happy I was the first time I typed just “ls” at the power shell prompt and got approximately what I was expecting. It really made my whole evening.

But the fact remains that I still type “ls -latr” which still gets me an error in PowerShell. I haven’t yet figured out how/if I can alias that out, but aliases do exist in PowerShell. To sort the files based on their date in ascending order, you can use:

ls | sort -property LastWriteTime

In PowerShell, ls is really just an alias for the Get-ChildItem PowerShell function. You can use man in PowerShell too, and it’s also helpful for using to look at aliases like ls to see what they actually point to, like so:

Grep

One of the things I’ve always missed at a Windows command line was grep. I use grep a lot, usually running some command and piping it to grep to get only the information I want. Well, grep doesn’t appear in the default PowerShell aliases, but I can at least generally emulate its behavior with the select-string commandlet:

PS E:\DB2\NODE0000> db2 get dbm cfg | select-string -pattern "DIAGPATH"

 Diagnostic data directory path               (DIAGPATH) =

Remember also, that Windows is not generally case sensitive, so using the syntax above is equivalent to the following command on Linux or UNIX:

db2 get dbm cfg | grep -i diagpath

I’m sure there are many more cool things that can be done with select-string.

Head and Tail

Fairly frequently, I want to see the first few or the last few lines of a file. I can do that fairly easily in PowerShell with the select comandlet:

PS D:\xtivia> db2 get dbm cfg | select -first 15


          Database Manager Configuration



     Node type = Enterprise Server Edition with local and remote clients



 Database manager configuration release level            = 0x0b00



 Maximum total of files open               (MAXTOTFILOP) = 16000

Counting Lines

Counting lines of output can be quite useful in some scenarios, though I do more of this through SQL than I used to. Here’s one useful example:

PS> db2 list applications | Measure-Object -Line

                        Lines Words                         Characters                    Property
                        ----- -----                         ----------                    --------
                          473

You can obviously use Measure-Object for other purposes as well.

VI

I miss vi at the command line. The first command I issue after logging in to many Linux/UNIX servers is set -o vi. I love to be able to easily search through command history and am so used to vi editing commands that I use them automatically – it’s rather frustrating to do this at any other command line, because you end up typing ‘hhhhhhhhhhh’ when what you really wanted to do was scroll left.

I spent a bit of time searching and couldn’t find a way to get this functionality at the PowerShell command line. Please let me know if anyone knows a way.

On the other hand, I can use aliases in that same profile that I used to set up the automatic use of DB2 commands to make it so that instead of getting error messages every time my fingers type vim filename, I instead get notepad opening the file.

First, at a PowerShell prompt, I issue:

powershell_ise $profile

Then a scripting window pops up that helps me with syntax. With these aliases added, mine now looks like this:

Save and close, and the next time you open PowerShell, the aliases are available for use.

Other Useful Aliases

The default aliases include those for cp, mv, rm and others. You can list the full list of aliases in your environment, including any custom ones, by issuing:

Get-Alias

Redirection

The standard redirection characters – |, >, < work in PowerShell, as does a favorite of mine – tee.

Summary

With all this good stuff at the command line, I’m much more looking forward to scripting than I was with batch. I feel like it’s possible I won’t feel so lost coming from my Perl and ksh roots.

I mentioned this SQL in my article on LISTAGG, but since I made use of it myself recently, I thought it would be worth listing in my DB2 Administrative SQL Cookbook series.

Purpose

To list the columns of a table in a comma separated list. This is particularly useful if you have a large table, and need to list all columns explicitly or need to change only one or two columns in an export statement. This statement works for a large number of columns (186 in the case of my example), and orders the columns by their column number in the table. The table used in this example has essentially the same structure as mon_get_bufferpool, with a couple of extra columns added.

Version

This SQL only works on DB2 9.7 and up. It is harder before then due to the lack of the LISTAGG function.

Statement

select listagg(cast(colname as varchar(10000)), ', ') within group (order by colno) as columns 
from syscat.columns 
where   tabschema='VDBA' 
        and tabname='DB2_BUFFERPOOL' 
with ur;

Sample Output

BP_NAME, MEMBER, DBNAME, AUTOMATIC, DIRECT_READS, DIRECT_READ_REQS, DIRECT_WRITES, DIRECT_WRITE_REQS, POOL_DATA_L_READS, POOL_TEMP_DATA_L_READS, POOL_XDA_L_READS, POOL_TEMP_XDA_L_READS, POOL_INDEX_L_READS, POOL_TEMP_INDEX_L_READS, POOL_DATA_P_READS, POOL_TEMP_DATA_P_READS, POOL_XDA_P_READS, POOL_TEMP_XDA_P_READS, POOL_INDEX_P_READS, POOL_TEMP_INDEX_P_READS, POOL_DATA_WRITES, POOL_XDA_WRITES, POOL_INDEX_WRITES, DIRECT_READ_TIME, DIRECT_WRITE_TIME, POOL_READ_TIME, POOL_WRITE_TIME, POOL_ASYNC_DATA_READS, POOL_ASYNC_DATA_READ_REQS, POOL_ASYNC_DATA_WRITES, POOL_ASYNC_INDEX_READS, POOL_ASYNC_INDEX_READ_REQS, POOL_ASYNC_INDEX_WRITES, POOL_ASYNC_XDA_READS, POOL_ASYNC_XDA_READ_REQS, POOL_ASYNC_XDA_WRITES, POOL_NO_VICTIM_BUFFER, POOL_LSN_GAP_CLNS, POOL_DRTY_PG_STEAL_CLNS, POOL_DRTY_PG_THRSH_CLNS, VECTORED_IOS, PAGES_FROM_VECTORED_IOS, BLOCK_IOS, PAGES_FROM_BLOCK_IOS, UNREAD_PREFETCH_PAGES, FILES_CLOSED, POOL_DATA_GBP_L_READS, POOL_DATA_GBP_P_READS, POOL_DATA_LBP_PAGES_FOUND, POOL_DATA_GBP_INVALID_PAGES, POOL_INDEX_GBP_L_READS, POOL_INDEX_GBP_P_READS, POOL_INDEX_LBP_PAGES_FOUND, POOL_INDEX_GBP_INVALID_PAGES, POOL_ASYNC_DATA_GBP_L_READS, POOL_ASYNC_DATA_GBP_P_READS, POOL_ASYNC_DATA_LBP_PAGES_FOUND, POOL_ASYNC_DATA_GBP_INVALID_PAGES, POOL_ASYNC_INDEX_GBP_L_READS, POOL_ASYNC_INDEX_GBP_P_READS, POOL_ASYNC_INDEX_LBP_PAGES_FOUND, POOL_ASYNC_INDEX_GBP_INVALID_PAGES, POOL_XDA_GBP_L_READS, POOL_XDA_GBP_P_READS, POOL_XDA_LBP_PAGES_FOUND, POOL_XDA_GBP_INVALID_PAGES, POOL_ASYNC_XDA_GBP_L_READS, POOL_ASYNC_XDA_GBP_P_READS, POOL_ASYNC_XDA_LBP_PAGES_FOUND, POOL_ASYNC_XDA_GBP_INVALID_PAGES, POOL_ASYNC_READ_TIME, POOL_ASYNC_WRITE_TIME, BP_CUR_BUFFSZ, POOL_QUEUED_ASYNC_DATA_REQS, POOL_QUEUED_ASYNC_INDEX_REQS, POOL_QUEUED_ASYNC_XDA_REQS, POOL_QUEUED_ASYNC_TEMP_DATA_REQS, POOL_QUEUED_ASYNC_TEMP_INDEX_REQS, POOL_QUEUED_ASYNC_TEMP_XDA_REQS, POOL_QUEUED_ASYNC_OTHER_REQS, POOL_QUEUED_ASYNC_DATA_PAGES, POOL_QUEUED_ASYNC_INDEX_PAGES, POOL_QUEUED_ASYNC_XDA_PAGES, POOL_QUEUED_ASYNC_TEMP_DATA_PAGES, POOL_QUEUED_ASYNC_TEMP_INDEX_PAGES, POOL_QUEUED_ASYNC_TEMP_XDA_PAGES, POOL_FAILED_ASYNC_DATA_REQS, POOL_FAILED_ASYNC_INDEX_REQS, POOL_FAILED_ASYNC_XDA_REQS, POOL_FAILED_ASYNC_TEMP_DATA_REQS, POOL_FAILED_ASYNC_TEMP_INDEX_REQS, POOL_FAILED_ASYNC_TEMP_XDA_REQS, POOL_FAILED_ASYNC_OTHER_REQS, SKIPPED_PREFETCH_DATA_P_READS, SKIPPED_PREFETCH_INDEX_P_READS, SKIPPED_PREFETCH_XDA_P_READS, SKIPPED_PREFETCH_TEMP_DATA_P_READS, SKIPPED_PREFETCH_TEMP_INDEX_P_READS, SKIPPED_PREFETCH_TEMP_XDA_P_READS, SKIPPED_PREFETCH_UOW_DATA_P_READS, SKIPPED_PREFETCH_UOW_INDEX_P_READS, SKIPPED_PREFETCH_UOW_XDA_P_READS, SKIPPED_PREFETCH_UOW_TEMP_DATA_P_READS, SKIPPED_PREFETCH_UOW_TEMP_INDEX_P_READS, SKIPPED_PREFETCH_UOW_TEMP_XDA_P_READS, PREFETCH_WAIT_TIME, PREFETCH_WAITS, POOL_DATA_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_INDEX_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_XDA_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_ASYNC_DATA_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_ASYNC_INDEX_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_ASYNC_XDA_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_COL_L_READS, POOL_TEMP_COL_L_READS, POOL_COL_P_READS, POOL_TEMP_COL_P_READS, POOL_COL_LBP_PAGES_FOUND, POOL_COL_WRITES, POOL_ASYNC_COL_READS, POOL_ASYNC_COL_READ_REQS, POOL_ASYNC_COL_WRITES, POOL_ASYNC_COL_LBP_PAGES_FOUND, POOL_COL_GBP_L_READS, POOL_COL_GBP_P_READS, POOL_COL_GBP_INVALID_PAGES, POOL_COL_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_ASYNC_COL_GBP_L_READS, POOL_ASYNC_COL_GBP_P_READS, POOL_ASYNC_COL_GBP_INVALID_PAGES, POOL_ASYNC_COL_GBP_INDEP_PAGES_FOUND_IN_LBP, POOL_QUEUED_ASYNC_COL_REQS, POOL_QUEUED_ASYNC_TEMP_COL_REQS, POOL_QUEUED_ASYNC_COL_PAGES, POOL_QUEUED_ASYNC_TEMP_COL_PAGES, POOL_FAILED_ASYNC_COL_REQS, POOL_FAILED_ASYNC_TEMP_COL_REQS, SKIPPED_PREFETCH_COL_P_READS, SKIPPED_PREFETCH_TEMP_COL_P_READS, SKIPPED_PREFETCH_UOW_COL_P_READS, SKIPPED_PREFETCH_UOW_TEMP_COL_P_READS, BP_PAGES_LEFT_TO_REMOVE, BP_TBSP_USE_COUNT, POOL_DATA_CACHING_TIER_L_READS, POOL_INDEX_CACHING_TIER_L_READS, POOL_XDA_CACHING_TIER_L_READS, POOL_COL_CACHING_TIER_L_READS, POOL_DATA_CACHING_TIER_PAGE_WRITES, POOL_INDEX_CACHING_TIER_PAGE_WRITES, POOL_XDA_CACHING_TIER_PAGE_WRITES, POOL_COL_CACHING_TIER_PAGE_WRITES, POOL_DATA_CACHING_TIER_PAGE_UPDATES, POOL_INDEX_CACHING_TIER_PAGE_UPDATES, POOL_XDA_CACHING_TIER_PAGE_UPDATES, POOL_COL_CACHING_TIER_PAGE_UPDATES, POOL_CACHING_TIER_PAGE_READ_TIME, POOL_CACHING_TIER_PAGE_WRITE_TIME, POOL_DATA_CACHING_TIER_PAGES_FOUND, POOL_INDEX_CACHING_TIER_PAGES_FOUND, POOL_XDA_CACHING_TIER_PAGES_FOUND, POOL_COL_CACHING_TIER_PAGES_FOUND, POOL_DATA_CACHING_TIER_GBP_INVALID_PAGES, POOL_INDEX_CACHING_TIER_GBP_INVALID_PAGES, POOL_XDA_CACHING_TIER_GBP_INVALID_PAGES, POOL_COL_CACHING_TIER_GBP_INVALID_PAGES, POOL_DATA_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_INDEX_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_XDA_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_COL_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_ASYNC_DATA_CACHING_TIER_READS, POOL_ASYNC_INDEX_CACHING_TIER_READS, POOL_ASYNC_XDA_CACHING_TIER_READS, POOL_ASYNC_COL_CACHING_TIER_READS, POOL_ASYNC_DATA_CACHING_TIER_PAGE_WRITES, POOL_ASYNC_INDEX_CACHING_TIER_PAGE_WRITES, POOL_ASYNC_XDA_CACHING_TIER_PAGE_WRITES, POOL_ASYNC_COL_CACHING_TIER_PAGE_WRITES, POOL_ASYNC_DATA_CACHING_TIER_PAGE_UPDATES, POOL_ASYNC_INDEX_CACHING_TIER_PAGE_UPDATES, POOL_ASYNC_XDA_CACHING_TIER_PAGE_UPDATES, POOL_ASYNC_COL_CACHING_TIER_PAGE_UPDATES, POOL_ASYNC_DATA_CACHING_TIER_PAGES_FOUND, POOL_ASYNC_INDEX_CACHING_TIER_PAGES_FOUND, POOL_ASYNC_XDA_CACHING_TIER_PAGES_FOUND, POOL_ASYNC_COL_CACHING_TIER_PAGES_FOUND, POOL_ASYNC_DATA_CACHING_TIER_GBP_INVALID_PAGES, POOL_ASYNC_INDEX_CACHING_TIER_GBP_INVALID_PAGES, POOL_ASYNC_XDA_CACHING_TIER_GBP_INVALID_PAGES, POOL_ASYNC_COL_CACHING_TIER_GBP_INVALID_PAGES, POOL_ASYNC_DATA_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_ASYNC_INDEX_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_ASYNC_XDA_CACHING_TIER_GBP_INDEP_PAGES_FOUND, POOL_ASYNC_COL_CACHING_TIER_GBP_INDEP_PAGES_FOUND, BASELINE_TIMESTAMP, TIMESTAMP, MAIL_ID 

Current statistics are vital to performance. If I’m encountering a database I haven’t regularly been supporting, and don’t know the maintenance plans and schedules, I frequently query to get an idea of how current they are.

Purpose

To report the dates of statistics collection and the number of tables statistics were collected for on each date. Usually a small list that should give an idea of how current statistics are.

Version

Multiple versions required due to types of tables and views that should or should not have statistics on them, added in recent releases.

Statement for 10.5

select  date(stats_time) as date,
        count(*) as count
from syscat.tables
where   (type = 'T' or (type = 'V' and substr(property,13,1) = 'Y'))
        and volatile !='C'
        and substr(property,21,1) != 'Y'
group by date(stats_time)
with ur;

Statement for 9.1, 9.5, 9.7 and 10.1

select  date(stats_time) as date,
        count(*) as count
from syscat.tables
where   (type = 'T' or (type = 'V' and substr(property,13,1) = 'Y'))
        and volatile !='C'
group by date(stats_time)
with ur;

Statement for 8.2 and earlier

select  date(stats_time) as date,
        count(*) as count
from syscat.tables
where   type = 'T'
        and volatile !='C'
group by date(stats_time)
with ur;

Sample Output

DATE       COUNT
---------- -----------
02/24/2015         869
-                  332

  2 record(s) selected.

Notes and Details

This SQL looks for tables (type = 'T') or statistical views (type = 'V' and substr(property,13,1) = 'Y')) that are not volatile (volatile !='C'), but not synopsis tables (substr(property,21,1) != 'Y'). Runstats cannot be done on synopsis tables. Synopsis tables are only used for column-organized tables. Statistical views are not used very extensively from what I’ve seen, but if they are used, it’s critical to make sure that statistics are collected on them.

This week, I’ll be giving two presentations for two users groups – one in St. Louis and one in Wisconsin.

Presentations

The two presentations I’ll be giving are:

DB2 Basics: Writing SQL with Examples on DB2 10.5 System Views and Table Functions

Some DBAs write SQL every day and others not as much. Come and learn the basics of writing SQL along with a few tips on more advanced features of SQL. Examples will use system views and table functions available in DB2 10.5. This session is geared towards DBAs who don’t write SQL on a regular basis and anyone who wants to take a basic look at writing SQL. It includes useful examples against system tables and also covers a bit of more advanced SQL toward the end.

HADR & TSAMP Advanced Topics

Beyond the simple how-to for HADR and TSAMP, come and learn about how the various HADR SYNCMODEs function, learn how to use the HADR toolkit provided on developerWorks in the real world, details of TSAMP that go beyond db2haicu, and the new TSAMP Configuration Checker tool available from DB2 support. A grab-bag of advanced HADR and TSAMP topics, with real-world experience and tips.

Locations and Details

St Louis

In St Louis, the users group meeting takes place at the IBM Facility at:
IBM Hazelwood Complex
325 McDonnell Blvd
St. Louis, MO 63042

Pre-registration is required for all attendees at: https://ibm.biz/STLDUG201503_Registration

Wisconsin

In Wisconsin, the users group meeting is at:
Aurora Medical Center – Summit, WI
36500 Aurora Dr.
Summit, WI 53066

The cutoff date for pre-registration has passed, so if you’d like to sign up, email wdug @ wdug.com for details.

GENERATED ALWAYS can be a blessing or a curse. I’m not talking about identity columns here, but about creating a column that is actually a duplicate of some part of the data to boost performance. Sure, in 10.5 we can do indexes on expressions, but for some clients I have trouble just getting them to go to a supported level, much less the latest and greatest. There are still some cases where I use this trick, though I analyze the situation thoroughly before using it.

Why

In this case, my client needs to take just one character out of an integer column and use it in multiple where clauses. When they do this in the where clause, it inevitably results in a table scan, since the use of functions eliminates the use of indexes. The table is kind of central in their application design, and not very large. Also, it is a custom application, so the are able to change what SQL the application uses. In this case, duplicating some of the data to improve performance is no worse than creating indexes – that is, it must be done cautiously, but the right data can improve performance and in this case also concurrency. The driving factor here is a locking problem that is still being investigated. Some application comes in and locks up a couple of rows in the table for 5 or 10 minutes. Meanwhile, the query that uses the functions comes in and tries to get a share lock on the whole table in order to do the table scan. My client happens to be a bit sharp in this area, and discovered that when this issue occurs, if they run the query without the substring, it completes just fine, but with the function it gets a lock timeout. I’m fairly sure that this is because it does not have to scan the whole table, but instead hits an index and only gets the share locks on a few individual rows that meet its other criteria.

How

To alleviate the problem and allow us to index the data being queried, I’m adding a column that contains only the character added by the substr function being applied. The query we’re tuning for looks something like this:

SELECT * 
FROM schema.table 
WHERE   processed = 'FALSE'
        and (substr(cast(comm_id as char(9)), 9, 1) = '4')
ORDER BY insert_dt;

If I run an explain on this, as expected, I get a full table scan:

Access Plan:
-----------
    Total Cost:         7642.65
    Query Degree:       1

          Rows 
         RETURN
         (   1)
          Cost 
           I/O 
           |
         289.002 
         TBSCAN
         (   2)
         7642.65 
          3542 
           |
         289.002 
         SORT  
         (   3)
         7642.65 
          3542 
           |
         289.002 
         TBSCAN
         (   4)
         7642.58 
          3542 
           |
         180626 
     TABLE: ECROOKS 
         TABLE:
           Q1

By adding a column and having the apps altered to query it instead, I can not only add and make use of an index, but I can also eliminate a type conversion.

Note that I cannot only issue the alter table statement to add the column. If I do so, I get this:

DB21034E  The command was processed as an SQL statement because it was not a
valid Command Line Processor command.  During SQL processing it returned:
SQL20054N  The table "QUALCOMM.SEND_WORKFLOW_INFO" is in an invalid state for
the operation.  Reason code="22".  SQLSTATE=55019

Looking up the error and specifically the sections related to RC 22, I see:

PS D:\xtivia> db2 ? SQL20054N


SQL20054N The table "" is in an invalid state for
          the operation.  Reason code = "".

Explanation:

 The table is in a state that does not allow the operation.  The
reason code indicates the state of the table that prevents the
operation.

…

 22 The generated column expression cannot be added or altered
because the table is not in check pending mode.

…

 22 Use SET INTEGRITY FOR  OFF before altering the
table. Then alter the table and use SET INTEGRITY FOR
 IMMEDIATE CHECKED FORCE GENERATED to generate    the
values for the new or altered column.

Therefore the syntax that I actually have to use to accomplish this is:

set integrity for ecrooks.table off;
alter table ecrooks.table add column comm_id_substr char(1) generated always as (substr(cast(comm_id as char(9)),9,1));
set integrity for ecrooks.table immediate checked force generated;

After adding the column, I also have to add an index on it – the column itself still gets me a table scan. The index in this case is:

create index ecrooks.TEMP_IDX_2 on table ecrooks.table (comm_id_substr, processed) allow reverse scans collect detailed statistics;

Since I know I have to go back to the table for the data, I only include the columns needed in the where clause. Given the table structure in this case, I could achieve index-only access by putting every column in the table in the index, but that is overkill in this (and almost every) case.

Results

The revised query to make use of this new column (and index) is:

SELECT * 
FROM schema.table 
WHERE   processed = 'FALSE'
        and comm_id_substr = '4'
ORDER BY insert_dt;

After adding the column and the index, the changed query uses it, and sees a vast performance improvement

Access Plan:
-----------
    Total Cost:         26.0669
    Query Degree:       1

              Rows 
             RETURN
             (   1)
              Cost 
               I/O 
               |
            0.999994 
             TBSCAN
             (   2)
             26.0669 
             2.03771 
               |
            0.999994 
             SORT  
             (   3)
             26.0667 
             2.03771 
               |
            0.999994 
             FETCH 
             (   4)
             26.0661 
             2.03771 
         /-----+------\
    0.999994          179918 
     IXSCAN       TABLE: ECROOKS 
     (   5)           TABLE
     12.8014            Q1
        1 
       |
     179918 
 INDEX: ECROOKS 
   TEMP_IDX_2
       Q1

I’ve played with the clpplus at least once before, but have generally thought of it as a tool created to satisfy those coming from Oracle and looking for Oracle-like features. One of the features I actually liked about Oracle in the class and certification tests that I took for it was the ability to specify values for an SQL statement stored in a file on execution. When I lamented the fact that DB2 doesn’t have this feature on twitter, @idbjorh was quick to remind me that such functionality is indeed available using CLPPLUS.

Starting CLPPLUS

I actually found this part annoying. Almost as annoying as connecting to an Oracle database. First off, I had not enabled TCPIP for my toy SAMPLE database on the Ubuntu VM where I do a lot of my playing. No need to, as I just connect locally for anything I’ve been testing thus far. CLPPLUS uses JDBC type 4 drivers and thus TCP/IP is required. So I set up TCP/IP like this:

$ db2set DB2COMM=TCPIP
$ db2 update dbm cfg using SVCENAME db2c_db2inst1

And finally by adding this line to /etc/services:

db2c_db2inst1   50000/tcp

That’s all pretty standard stuff, and it’s likely that for any real server implementation you’ll already have set it up. Port numbers can vary and should vary for production implementations for security reasons.

I also find it annoying to have to specify the full jdbc url to connect, when I’m used to just issuing db2 connect to SAMPLE. Well, there’s a workaround for that too. You can find details on that in Ian’s guest blog post A Tale of Two Connections, or in his IDUG article on the same topic.

Creating a File With SQL I Can Pass Parameters To

I went through several iterations to get what I wanted just passing the parameters in. This is what I ended up with:

select 
    i.lastused
    , substr(indname,1,20) as indname
    , substr(colnames,1,50) as colnames
    , fullkeycard as fullkeycard
    , card as table_card
    , decimal(clusterfactor,10,5) as clusterfactor
    , indextype
from syscat.indexes i 
    join syscat.tables t 
        on i.tabname=t.tabname 
        and i.tabschema=t.tabschema
where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur;
EXIT

And I have to execute it like this (output shown too):

$ clpplus -nw db2inst1/db2inst1@localhost:50000/sample @desc_indexes.clpplus DB2INST1 SALES
CLPPlus: Version 1.6
Copyright (c) 2009, 2011, IBM CORPORATION.  All rights reserved.


Database Connection Information :
---------------------------------
Hostname = localhost 
Database server = DB2/LINUXX8664  SQL10054 
SQL authorization ID = db2inst1 
Local database alias = SAMPLE 
Port = 50000 

 
Original statement:where t.tabschema=upper('&1') and t.tabname=upper('&2') with
ur
New statement with substitutions:where t.tabschema=upper('DB2INST1') and
 t.tabname=upper('SALES') with ur

LASTUSED   INDNAME             
---------- --------------------
COLNAMES                                                    FULLKEYCARD
-------------------------------------------------- --------------------
          TABLE_CARD CLUSTERFACTOR INDEXTYPE
-------------------- ------------- ---------
0001-01-01 IX_SALES01          
+REGION                                                               4
                  41      -1.00000 REG      

I again find this annoying because running the query at the regular command line, I get exactly what I want for output – a fairly wide output that shows me everything in a nice format. So I have to add a line to get reasonable output:

SET LINESIZE 160
select 
    i.lastused
    , substr(indname,1,20) as indname
    , substr(colnames,1,50) as colnames
    , fullkeycard as fullkeycard
    , card as table_card
    , decimal(clusterfactor,10,5) as clusterfactor
    , indextype
from syscat.indexes i 
    join syscat.tables t 
        on i.tabname=t.tabname 
        and i.tabschema=t.tabschema
where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur;
EXIT

Now my output looks like this:

$ clpplus -nw db2inst1/db2inst1@localhost:50000/sample @desc_indexes.clpplus DB2INST1 SALES
CLPPlus: Version 1.6
Copyright (c) 2009, 2011, IBM CORPORATION.  All rights reserved.


Database Connection Information :
---------------------------------
Hostname = localhost 
Database server = DB2/LINUXX8664  SQL10054 
SQL authorization ID = db2inst1 
Local database alias = SAMPLE 
Port = 50000 

 
Original statement:where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur
New statement with substitutions:where t.tabschema=upper('DB2INST1') and t.tabname=upper('SALES') with ur

LASTUSED   INDNAME              COLNAMES                                                    FULLKEYCARD           TABLE_CARD CLUSTERFACTOR INDEXTYPE
---------- -------------------- -------------------------------------------------- -------------------- -------------------- ------------- ---------
0001-01-01 IX_SALES01           +REGION                                                               4                   41      -1.00000 REG   

That’s better. Now let’s have some fun with the formatting options:

SET LINESIZE 160;
set headsep '!';
ttitle center 'Indexes for &1 . &2';
column lastused             heading 'Last!Used'
column indname      format a20  heading 'Index Name'
column colnames     format a40  heading 'Column Names'
column fullkeycard          heading 'Index!Cardinality'
column card             heading 'Table!Cardinality'
column clusterfactor    format 99.9999  heading 'Cluster!Factor' 
column indextype            heading 'Index!Type'
select 
    i.lastused
    , indname
    , colnames
    , fullkeycard
    , card
    , clusterfactor
    , indextype
from syscat.indexes i 
    join syscat.tables t 
        on i.tabname=t.tabname 
        and i.tabschema=t.tabschema
where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur;
EXIT

Which gives me:

CLPPlus: Version 1.6
Copyright (c) 2009, 2011, IBM CORPORATION.  All rights reserved.


Database Connection Information :
---------------------------------
Hostname = localhost 
Database server = DB2/LINUXX8664  SQL10054 
SQL authorization ID = db2inst1 
Local database alias = SAMPLE 
Port = 50000 

 
Original statement:where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur
New statement with substitutions:where t.tabschema=upper('DB2INST1') and t.tabname=upper('SALES') with ur

                                                                   Indexes for DB2INST1 . SALES                                                                 

Last                                                                                    Index                Table                Cluster Index    
Used       Index Name           Column Names                                      Cardinality          Cardinality                 Factor Type     
---------- -------------------- ---------------------------------------- -------------------- -------------------- ---------------------- ---------
0001-01-01 IX_SALES01           +REGION                                                     4                   41                -1.0000 REG      

Notice I was able to get the substr functions out of my SQL, and do the formatting though pure CLPPLUS commands. It did make my Cluster Factor column rather wider, though. My column headings are more natural English, with breaks where I want them.

I’ve added another index in my SAMPLE database to show off the next feature. This may be one that actually makes me want to use clpplus.

SET LINESIZE 160;
set headsep '!';
set wrap on;
ttitle center 'Indexes for &1 . &2';
column lastused             heading 'Last!Used'
column indname      format a20  heading 'Index Name'
column colnames     format a20  heading 'Column Names'
column fullkeycard          heading 'Indexr!Cardinality'
column card             heading 'Table!Cardinality'
column clusterfactor    format 99.9999  heading 'Cluster!Factor' 
column indextype            heading 'Index!Type'
select 
    i.lastused
    , indname
    , colnames
    , fullkeycard
    , card
    , clusterfactor
    , indextype
from syscat.indexes i 
    join syscat.tables t 
        on i.tabname=t.tabname 
        and i.tabschema=t.tabschema
where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur;
EXIT

And the output now looks like:

$ clpplus -nw db2inst1/db2inst1@localhost:50000/sample @desc_indexes.clpplus DB2INST1 SALES
CLPPlus: Version 1.6
Copyright (c) 2009, 2011, IBM CORPORATION.  All rights reserved.


Database Connection Information :
---------------------------------
Hostname = localhost 
Database server = DB2/LINUXX8664  SQL10054 
SQL authorization ID = db2inst1 
Local database alias = SAMPLE 
Port = 50000 

 
Original statement:where t.tabschema=upper('&1') and t.tabname=upper('&2') with ur
New statement with substitutions:where t.tabschema=upper('DB2INST1') and t.tabname=upper('SALES') with ur

                                                                   Indexes for DB2INST1 . SALES                                                                 

Last                                                               Indexr                Table                Cluster Index    
Used       Index Name           Column Names                  Cardinality          Cardinality                 Factor Type     
---------- -------------------- -------------------- -------------------- -------------------- ---------------------- ---------
0001-01-01 IX_SALES01           +REGION                                 4                   41                -1.0000 REG      
0001-01-01 IX_SALES02           +SALES_DATE+SALES_PE                   41                   41                -1.0000 REG      
                                RSON+REGION+SALES+SE                                                                           
                                C_LABLE                                                                                        

I very much like the wrap option. I just wish I could tell it a charachter to wrap on. If I could wrap on each + or -, that would be some nifty output. I also like the wrap idea for when I’m querying mon_get_pkg_cache and want to see data in a tabular format, while still getting an SQL statement I can read. I just wish it would wrap on spaces there – I tried and it does not.

Here’s another neat thing you can do – prompt the user for input. Here’s the file I use:

SET LINESIZE 160;
set headsep '!';
set wrap on;
column lastused             heading 'Last!Used'
column indname      format a20  heading 'Index Name'
column colnames     format a20  heading 'Column Names'
column fullkeycard          heading 'Indexr!Cardinality'
column card             heading 'Table!Cardinality'
column clusterfactor    format 99.9999  heading 'Cluster!Factor' 
column indextype            heading 'Index!Type'

accept TABLE_SCHEMA prompt 'Table Schema:'
accept TABLE_NAME   prompt 'Table Name:'

define TABLE_SCHEMA
define TABLE_NAME

ttitle center 'Indexes for &TABLE_SCHEMA . &TABLE_NAME';

select 
    i.lastused
    , indname
    , colnames
    , fullkeycard
    , card
    , clusterfactor
    , indextype
from syscat.indexes i 
    join syscat.tables t 
        on i.tabname=t.tabname 
        and i.tabschema=t.tabschema
where t.tabschema=upper('&TABLE_SCHEMA') and t.tabname=upper('&TABLE_NAME') with ur;
EXIT

And the output I get:

$ clpplus -nw db2inst1/db2inst1@localhost:50000/sample @desc_indexes.clpplus 
CLPPlus: Version 1.6
Copyright (c) 2009, 2011, IBM CORPORATION.  All rights reserved.


Database Connection Information :
---------------------------------
Hostname = localhost 
Database server = DB2/LINUXX8664  SQL10054 
SQL authorization ID = db2inst1 
Local database alias = SAMPLE 
Port = 50000 

Table Schema: DB2INST1
Table Name: SALES
DEFINE TABLE_SCHEMA = DB2INST1
DEFINE TABLE_NAME = SALES
 
Original statement:where t.tabschema=upper('&TABLE_SCHEMA') and t.tabname=upper('&TABLE_NAME') with ur
New statement with substitutions:where t.tabschema=upper('DB2INST1') and t.tabname=upper('SALES') with ur

                                                                   Indexes for DB2INST1 . SALES                                                                 

Last                                                               Indexr                Table                Cluster Index    
Used       Index Name           Column Names                  Cardinality          Cardinality                 Factor Type     
---------- -------------------- -------------------- -------------------- -------------------- ---------------------- ---------
0001-01-01 IX_SALES01           +REGION                                 4                   41                -1.0000 REG      
0001-01-01 IX_SALES02           +SALES_DATE+SALES_PE                   41                   41                -1.0000 REG      
                                RSON+REGION+SALES+SE                                                                           
                                C_LABLE                                                                                        

The lines in red were where it actually prompted me for the input. And the neat thing is that I can STILL specify the table schema and the table name when I execute the file – it will only prompt me if I do not specify them

Summary

I started out annoyed and I ended up having fun. I could have done most of this faster in ksh or perl, but the wrapping, I don’t know how to do off the top of my head. Will I keep using CLPPLUS? I doubt it – a combination of momentum and some odd issues that Ian reports in his post. It is tempting, though. I may try to use it more to see how it goes.

I’d love to hear your thoughts on clpplus if you’ve tried it, below in the comments.

While using SQL to look at statistics overall is often useful, sometimes we need to look at or report on only a specific subset of tables.

Purpose

To report the time of last statistics collection (runstats) on a given list of tables. Using a pair of values is less common, but can be very useful when querying for things like tables in different schemas

Version

Should work on most versions of DB2.

Statement

select  substr(tabschema,1,18) as tabschema, 
        substr(tabname,1,30) as tabname, 
        stats_time
    from syscat.tables as t
    where (tabschema, tabname) in (VALUES('DB2','CUSEXT'),
                                        ('DB2','CUSMAS'),
                                        ('DB2','CUSTHIER'),
                                        ('DB2','SHIPPING_LOCATIONS'),
                                        ('DB2','SSOCSSWHS'),
                                        ('DB2','SSOWHCL'),
                                        ('DB2ONLINE','DEALERNUMBERS'),
                                        ('DB2ONLINE','USERTYPE')
                                    ) 
    with ur

Sample Output

TABSCHEMA          TABNAME                        STATS_TIME
------------------ ------------------------------ --------------------------
DB2                SSOCSSWHS                      2014-12-06-02.27.31.861000
DB2                SSOWHCL                        2014-12-06-02.27.31.954000
DB2                SHIPPING_LOCATIONS             2014-12-06-02.07.08.345000
DB2ONLINE          USERTYPE                       2014-12-06-02.55.00.228000
DB2                CUSTHIER                       2014-12-06-02.27.34.497000
DB2                CUSEXT                         -
DB2                CUSMAS                         2014-12-06-02.15.50.063000
DB2ONLINE          DEALERNUMBERS                  2014-12-06-02.52.51.577000

  8 record(s) selected.

One of the awesome things about running scripts locally on a DB2 server is that if they’re run as a privileged user, you do not have to specify the password. This makes for easier scripting without storing or encrypting passwords. When I first connected to a database with PowerShell, it took me a bit to figure out how to do the password-less local connection, so I thought I would share.

Please note that I started with the connection information and script available here: http://myblog4fun.com/archive/2012/01/14/using-powershell-to-access-db2.aspx

To connect to the SAMPLE database and run a simple query, this syntax works:

$cn = new-object system.data.OleDb.OleDbConnection("Provider=IBMDADB2;DSN=SAMPLE;User Id=;Password=;");
$ds = new-object "System.Data.DataSet" "dsEmployee"

$q = "SELECT EMPNO"
$q = $q + ",FIRSTNME"
$q = $q + ",LASTNAME"
$q = $q + " FROM ECROOKS.EMPLOYEE"

$da = new-object "System.Data.OleDb.OleDbDataAdapter" ($q, $cn)

$da.Fill($ds)

$dtPerson = new-object "System.Data.DataTable" "dtEmployee"
$dtPerson = $ds.Tables[0]
$dtPerson | FOREACH-OBJECT { " " + $_.EMPNO + ": " + $_.FIRSTNME + ", " + $_.LASTNAME }

I found the connection string syntax that would work here: http://129.33.205.81/support/knowledgecenter/SSEPGG_10.1.0/com.ibm.db2.luw.apdv.sample.doc/doc/vb/s-ReadMe-txt.html?lang=en

While that sample is directed at Visual Basic, the connection strings are the same, so I was able to add the syntax in to the PowerShell script from the other blog.

I’m a control freak. I think that control freaks tend to make good DBAs as long as they don’t take it too far. My position for years has been that I would rather control my runstats, reorgs, and backups directly than trust DB2’s automatic facilities. But I also try to keep an open mind. That means that every so often I have to give the new stuff a chance. This blog entry is about me giving automatic maintenance a try. I am NOT recommending it yet, but here’s how I approached it and what I saw.

Environment

The environment I’m working with here is a brand new 10.5 (fixpack 5) database. It uses column-organization for most tables, but has a small subset of tables that must be row-organized. I still refuse to trust my backups to automation – I want them to run at a standard time each day. But for runstats and reorgs, I’m using this article to both look into what to do with the row-organized and the column-organized tables and how to set up controls around what times things happen. The environment I’m working on happens to use HADR.

Database Configuration Parameter Settings

Here’s the configuration I’m starting with for the Automatic maintenance parameters:

 Automatic maintenance                      (AUTO_MAINT) = ON
   Automatic database backup            (AUTO_DB_BACKUP) = OFF
   Automatic table maintenance          (AUTO_TBL_MAINT) = ON
     Automatic runstats                  (AUTO_RUNSTATS) = ON
       Real-time statistics            (AUTO_STMT_STATS) = ON
       Statistical views              (AUTO_STATS_VIEWS) = OFF
       Automatic sampling                (AUTO_SAMPLING) = OFF
     Automatic reorganization               (AUTO_REORG) = ON

I think those are the defaults for a BLU database, though I may have already set AUTO_DB_BACKUP to OFF manually myself. I’m also going to turn auto_stats_views on, in case I create one of those. This is the syntax I use for that:

-bash-4.1$ db2 update db cfg for SAMPLE using AUTO_STATS_VIEWS ON
DB20000I  The UPDATE DATABASE CONFIGURATION command completed successfully.

And now these configuration parameters look like this:

 Automatic maintenance                      (AUTO_MAINT) = ON
   Automatic database backup            (AUTO_DB_BACKUP) = OFF
   Automatic table maintenance          (AUTO_TBL_MAINT) = ON
     Automatic runstats                  (AUTO_RUNSTATS) = ON
       Real-time statistics            (AUTO_STMT_STATS) = ON
       Statistical views              (AUTO_STATS_VIEWS) = ON
       Automatic sampling                (AUTO_SAMPLING) = OFF
     Automatic reorganization               (AUTO_REORG) = ON

Runstats on Row-Organized Tables

The first thing that I know I want to do is to set up profiles for runstats on my row-organized tables.

Setting Profiles

I want to use this syntax for all of my row organized tables in this database:

runstats on table <SCHEMA>.<TABLE> with distribution and detailed indexes all

If I had larger row-organized tables, I might consider using sampling, but my row-organized tables in this database happen to be on the smaller side.

There doesn’t seem to be a way to set a default syntax other than through creating profiles on individual tables. Would love to know in a comment below if I’m missing something there.

To set my profiles, I’m going to use a little scripting trick I like to use when I have to do the same thing for many objects. In this case, I have 106 tables where I want to set the profile identically, so first I create a list of those tables using:

db2 -x "select  substr(tabschema,1,18) as tabschema
        , substr(tabname,1,40) as tabname
from syscat.tables
where   tableorg='R'
        and tabschema not like ('SYS%')
with ur" > tab.list

This creates a file called tab.list that has only the names of my tables – the -x on the command ensures that column headings and the summary row I have telling me how many rows are not returned as a part of the query.

Next, I loop through that list with a one-line shell script:

cat tab.list |while read s t; do db2 connect to bcudb; db2 -v "runstats on table $s.$t with distribution and detailed indexes all set profile"; db2 connect reset; done |tee stats.profile.out

Note that I could have used “set profile only” if I didn’t also want to actually do runstats on these tables, but in my case, I wanted to both do the runstats and set the profile. I then checked stats.profile.out for any failures with this quick grep:

 cat stats.profile.out |grep SQL |grep -v DB20000I |grep -v "SQL authorization ID   = DB2INST1"

Everything was successful.

Setting up a schedule

I don’t want my runstats to kick off any old time they feel like it. I want to restrict them to run between 1 am and 6 am daily. To do this, I need to set up an automatic maintenance policy. There are samples that I can start with in $HOME/sqllib/samples/automaintcfg.

I first made a copy of DB2MaintenanceWindowPolicySample.xml, renaming it and moving it to a working directory. I ensured my new file contained these lines:

<DB2MaintenanceWindows
xmlns="http://www.ibm.com/xmlns/prod/db2/autonomic/config">
 <OnlineWindow Occurrence="During" startTime="01:00:00" duration="5">
   <DaysOfWeek>All</DaysOfWeek>
   <DaysOfMonth>All</DaysOfMonth>
   <MonthsOfYear>All</MonthsOfYear>
 </OnlineWindow>
</DB2MaintenanceWindows>

I don’t want to set an offline window at this time, because I don’t have one. The sample file has great information on how to configure different scenarios. While there is no option in the xml file to specify a database or different databases, I’m setting the policy with a command against a database, so I named the file with the database name in it and keep it in a place I can easily find it later so that I can change it if need be.

By default, the online window is 24/7.

Now that I have an XML file that will do what I want, I can set that as the policy using the AUTOMAINT_SET_POLICYFILE procedure, like this:

-bash-4.1$ db2 "call sysproc.automaint_set_policyfile( 'MAINTENANCE_WINDOW', 'DB2MaintenanceWindowPolicyBCUDB.xml' )"
SQL1436N  Automated maintenance policy configuration file named

Well, oops, that didn’t work so well. I learned that the xml file you want to use must be in $HOME/sqllib/tmp. ALSO, it must be readable by the fenced user ID. With the way I have it set up (with the fenced user id in the primary group of my instance id), this is what I had to do to make that work:

-bash-4.1$ cp DB2MaintenanceWindowPolicyBCUDB.xml $HOME/sqllib/tmp
-bash-4.1$ chmod 740 $HOME/sqllib/tmp/DB2MaintenanceWindowPolicyBCUDB.xml

I was then able to successfully call the stored procedure:

-bash-4.1$ db2 "call sysproc.automaint_set_policyfile( 'MAINTENANCE_WINDOW', 'DB2MaintenanceWindowPolicyBCUDB.xml' )"

  Return Status = 0

When DB2 reads the file in, it is not depending on that file to exist forever. It is storing the information from the file in the database. You can use the AUTOMAINT_GET_POLICYFILE and AUTOMAINT_GET_POLICY stored procedures to pull that information back out. Remember that there is only one policy for each of the automatic maintenance categories, so it is best to first get the policy, change it, and then set it, so you do not accidentally overwrite what is already there.

Phew. Ok, that’s what I had to do to set things up for my row-organized tables. My column organized tables will also get runstats by default, and for the sake of trying it, I’m going to go with the defaults there and see what happens. And by see what happens, I mean I’ll be querying up a storm to see what DB2 is doing.

Reorgs

This database does not have an offline maintenance window. So I need to configure online reorgs to occur. Much like with runstats, I’m going to let the BLU tables go for a while and see if the the hype from IBM about just letting DB2 take care of it is really all that. The only reorgs there are for space reclaimation anyway. But for my row-organized tables, I want to make sure they’re taken care of.

Man, was I disappointed to discover that inplace/notruncate reorgs are STILL not supported as a part of automatic maintenance. This means that I cannot do table reorgs through DB2’s automation facilities … off to re-write my reorg script for yet another employer, I guess.

I’m trying to see if DB2 can at least manage my index reorgs for me online, though, with this syntax in my file:

<DB2AutoReorgPolicy
xmlns="http://www.ibm.com/xmlns/prod/db2/autonomic/config">
 <ReorgOptions dictionaryOption="Keep" indexReorgMode="Online"  useSystemTempTableSpace="false" />
 <ReorgTableScope maxOfflineReorgTableSize="52">
  <FilterClause />
 </ReorgTableScope>
</DB2AutoReorgPolicy>

And, of course, implementing that file with:

-bash-4.1$ cp DB2AutoReorgPolicyBCUDB.xml $HOME/sqllib/tmp
-bash-4.1$ chmod 740 $HOME/sqllib/tmp/DB2AutoReorgPolicyBCUDB.xml
-bash-4.1$ db2 "call sysproc.automaint_set_policyfile( 'AUTO_REORG', 'DB2AutoReorgPolicyBCUDB.xml' )"

  Return Status = 0

I think the combination of that and no offline reorg window defined will get me what I want on the index side anyway.

Not long ago, I posted some of the details of using PowerShell as your command line for DB2 on Windows. I am definitely addicted to PowerShell as my command line when I have to work on Windows servers.

I also have to write some scripts for some Windows systems. My first is a dynamic reorg/runstats/rbind script. I’m not done writing it yet, but I’m learning a lot and want to share what I’m learning with my readers.

Goals for this script

This is my fourth time writing this script. For one employer, I wrote this script first in KSH and then in Perl. Then I wrote it again for another employer in Perl. I seem to have always been writing scripts in situations where my employers end up owning them, and I’m a bit of a square about respecting that. Now I need a version that works on Windows, and I’m not allowed to use Perl. My choices are Batch, PowerShell, or VB. VB doesn’t appeal to me because it is so different from the scripting languages I am used to. Using batch and having to call db2cmd.exe is just annoying. And my husband happens to know PowerShell and has helped me with some of the basics, so that has been a plus.

I was a bit disappointed to learn that O’Reilly doesn’t have a “Learning PowerShell” book – their “Learning Perl” book is the single best technical book I have ever read. So I decided to just dig in and see what I could work out.

This script minimally needs to:

1. Connect to a local database without specifying a password
2. Run runstats on all tables, using a dynamically generated list of tables
3. Generate/read reorgchk information in some format
4. Determine what kind of table reorgs to do on which tables based on the above output
5. Determine what kind of index reorgs to do on which tables base on the above output
6. Perform the reorgs
7. Not reorg the same table more than once for the table and once for the indexes
8. Not die because one table action failed
9. Perform runstats on reorged tables
10. Perform reorg
11. Write a log file of everything done

The following features might not be in the initial version but are high on my priority list to add

• Take a list of tables to handle or except from the actions of the script
• Manage concurrency – limiting the number of online table reorgs
• Manage concurrency – Run multiple index reorgs at one time
• Handle some error situations – including retrying reorgs in some situations
• Potentially and optionally force off blocking connections?

I’m testing my script and playing with it on a local Windows VM with the SAMPLE database. I’ll have to add/remove data to get some more interesting results for the Reorg formulas in the near future.

I am a novice with PowerShell, so please use any code snippets here with extreme caution and only after thorough testing.

Connecting Locally Without Specifying a Password

This was an early challenge. I have decided to attempt to use a database interface rather than simply executing command line commands. This is different than the approach that I’ve taken in the past. It was relatively easy to find help on how to connect and to execute a query, but to connect to a local database without specifying a password actually took looking up blog and forum entries for the OLEDB provider that were not specific to PowerShell.

I detailed my efforts in this area in this quick blog entry: http://db2commerce.com/2015/03/19/how-to-connect-to-a-local-db2-database-without-specifying-a-password-in-powershell/

Calling Stored Procedures

The next thing I needed to be able to do is to run a stored procedure. Specifically the reorgchk_tb_stats and reorgchk_ix_stats stored procedures. This actually turned out to be fairly easy, especially since the output they send to screen is just not something I’m going to use. I just treat the call statement like it’s a query. This still involves:

1. Defining a connection object, if that has not already been done
2. Defining a data set for the data returned to go into. Not that I’m actually going to do anything with that data in this case.
3. Defining the query string for the call statement
4. Defining a data object by passing it the connection object and the query string
5. Executing the statement by using the Fill property of the data object to populate the data set

Note that not all that terminology may be 100% correct. I definitely feel like this is a blog entry I will look back on in the future and wonder that I managed to do anything at all with so little understanding. But it’s working so far.

Here’s the code I used to accomplish those steps.

#Define connection string for the database
$cn = new-object system.data.OleDb.OleDbConnection("Provider=IBMDADB2;DSN=$db_name;User Id=;Password=;");
#Define data set for first query
$ds = new-object "System.Data.DataSet" "dsReorgCHK"
#Define query to populate reorgchk information
$q = "call reorgchk_tb_stats('T', 'ALL')"
# Define data object given the specific query and connection string
$da = new-object "System.Data.OleDb.OleDbDataAdapter" ($q, $cn)
# Fill the data set - essentially run the query. 
  $da.Fill($ds)

Error Checking

Being thourough, I also wanted to make sure I was catching any SQL error messages with two goals in mind. First, for many of the statements I run in this script, I know I won’t want the whole script to die just because a statement fails – which SQL errors seem to cause in some situations. Second, I know that particularly once I get to the reorg statements (and maybe the runstats), there are some errors that I want to look for and either take specific action based on (maybe re-issue the reorg if it fails due to deadlock?), or write notes out to the output based on the error.

I had initially hoped I could grab the SQLCA somehow and pull from it. I’m still not convinced that’s not possible – I just have had a bear of a time figuring out how to do it. If anyone knows, please comment and help me out.

Barring that, I know I can parse the output for an SQL code like I always have in Perl. My first challenge in this area was to capture the error at all. My initial attempts at an error just led to the script stopping executing at that point. After some time away and some research, I found try/catch/finally. This is a way of executing a line of code, and if it produces an error to do something with the error information before proceeding on.

The syntax when I wrap that around my Fill statement above looks like this:

try {
  $da.Fill($ds)
}
catch {
  $err=$_
  $err >> $output_file
  $q >> $output_file
  $SQL_err = ParseSQLError $err
}
finally
{
}

Note that ParseSQLError – that’s my own function to parse through the output. It is pretty rudimentary at this point, and will likely evolve as I test various failure scenarios. Right now it looks like this:

# Function to parse out an SQL error code
function ParseSQLError ($err_text) {
  $err_code = [regex]::matches($err_text, "SQL\d{4}\D{1}") | %{$_.value}
  if (!$err_code) { $err_code = [regex]::matches($err_text, "SQL\d{5}\D{1}") | %{$_.value} }
  if ($debug -eq 1) {Write-Host "Error code: $err_code"}
}

I fall back to beloved regexes that Perl made so easy. The ones above capture the matching text and return it. I used different ones for 4 and 5 digit SQL codes, it would be easy to merge those together. I don’t currently capture return codes or SQLSTATEs or anything.

Basics of a Function

This parsing function was my first function in PowerShell. It’s a piece of code I expect to use over and over again within a script, and I don’t want to have to maintain it in more than one location. The function format seems easy and non difficult. I find here and a lot of places where $_ is a similar concept as it was in Perl. I’m not yet to the point of worrying about the scope of a lot of my variables – it doesn’t seem as strict as it was in Perl. I’ve seen a few references to a strict mode that I will probably enable at some point and tighten up my code. I’ve been so focused so far on just getting through the basics. Things like, when you call a function, the parameters are not in parentheses and they are separated by spaces and not commas.

Working wiht Query Output

Queries are run the same way I executed my call statement above, but after you’ve done the Fill, you continue to work with the result sets much as if they were tables. This is actually fantastic and easy. After the Fill for my query of session.tb_stats, I used this code (with $q set to the SQL):

  # Parse the output into a table format where we can refer to each column name and iterate through
  $reorgchk_table = new-object "System.Data.DataTable" "reorgchk_table"
  $reorgchk_table = $dq.Tables[0]
  # for each row of our data table, perform these actions
  $reorgchk_table | FOREACH-OBJECT { 
  if ( $_.CLUSTERING_INDEX -ne "-" ) { 
    ReorgTable $_.TABSCHEMA $_.TABNAME $_.CLUSTERING_INDEX
  } else {
    ReorgTable $_.TABSCHEMA $_.TABNAME
  }
} #End FOREACH-OBJECT

Now this is a preliminary version of the code where I’m generating reorg statements for every table reguardless of whether they are flagged for reorg or my own thresholds exceeded for reorg. I code one baby step at a time. The ReorgTable is a function that currently just writes out the reorg syntax so I can look at it. later it will actually handle the reorging of things. Notice that within the FOREACH-OBJECT on $reorgck_table, I refer to each value with $_.COLUMN_NAME. These are mapped for me by the Tables action on $dq (my data set). The COLUMN_NAMEs are read in from the query output, and I believe are case sensitive (not much is with PowerShell). Ask me how I discovered they’re case sensitive. Go on, I dare you. Let’s just say I was banging my head against that brick wall for at least 20 minutes.

What I really like here is that I don’t have to parse the output into variables and do splits to define what is in which column. I would guess that I would have also gotten something like this with the Perl DBI that I never used. These references to the values can be used in comparisons, can be written out to file or whatever you might want to do with them.

Just for a bit of balance, my currently very simple ReorgTable function:

function ReorgTable ($tabschema,$tabname,$qual_index) {
  $reorg_stmt = "reorg table " + $tabschema + "." + $tabname
  if ( $qual_index ) { 
    $reorg_stmt += " index " + $qual_index  
  }
  $reorg_stmt += " inplace allow write access"
  $reorg_stmt >> $output_file
  $reorg_stmt
}

I’ll add to this using ADMIN_CMD to execute the reorgs. I also haven’t yet coded in runstats functionality either on the front end or the back. rbind, and basic concurrency – there’s a lot left to do here and a lot left to share in a future blog entry. I also need to make sure I’m closing connections and clearing variables once I’m done with them. If I end up needing to run any queries multiple times, I’d like to learn how to do a separate prepare/execute. I’m remembering how fun scripting is!

My favorite week of the year is the IDUG North American Technical Conference. It’s better than Christmas if you ask me. I get to see my DB2 family and learn all kinds of new stuff. I come away refreshed and inspired.

I always like to share my opinions and some tips for first-timers before the conference, along with some of my can’t-miss sessions.

Changes in 2015

There are some interesting changes this year. Instead of full-day seminars the day before the conference at an extra charge(of over $400), these seminars are spread across the conference and only involve a $50 materials fee. On the one hand, I like this approach, but on the other, they have to be good to justify missing an entire day or half-day worth of sessions.

The conference this year is four and a half days instead of the normal three and a half.

Certifications are again being offered at free and reduced rates this year, but are only being offered the middle three days of the conference. The hours on these three days are extended though, and it is nice to see early morning and evening options for timing – now if IBM would just make up some new LUW certifications for me – there’s so little I haven’t already taken! How about updates to the application developer and stored procedure developer exams? I’m looking forward to the next version release just so I can get my hands on some new certificaiton exams!

Several hands-on-labs are being offered this year – I’m interested to see the format and details of these, and hope I can fit one into my schedule to try it out.

Conference Information on the IDUG Website

Some conference information is available on the IDUG Website. The main page for the conference is:
http://www.idug.org/na20159
There are a number of tabs and links there that can give good information there. If you want an overview, click on the Attendee tab. This gives you a overview of the conference week. Some good general information is also available in [a PDF Brochure] (http://www.idug.org/d/do/5499).

Certification

IDUG offers one free certification exam, and if you pass that one, you get one more for free. So if you’re planning on taking more than one, take the one you’re more sure of first. This is a great opportunity to take tests even if you’re not fully prepared – even the practice exams online are not free. Additional certification exams are just $25, a real bargain. Certification hours are a bit different this year:
Tuesday, May 5: 9:00 – 18:00
Wednesday, May 6: 7:00 – 18:00
Thursday, May 7: 7:00 – 20:00

The eligible exams are listed here: http://www.idug.org/p/cm/ld/fid=739

Choosing Sessions

I am consistently frustrated with conferences – both hosted by IDUG and by IBM because they do not make it easy to identify or search on the session speaker. I like to start with my favorite speakers – to see if they have anything new for me or in some cases things that I may have seen before, but I’m a different person, so I might get something new out of a presentation. After looking at my favorite speakers, I then like to fill in with new speakers, interesting topics, and so on.

This year, I’m looking forward to sessions from Michael Kwok, Steve Rees, John Hornibrook, Paul Turpin, Dan Luksetich, Adam Storm, Matt Huras, and Michael Krafick. I see Chris Eaton on the schedule and am intrigued – his topic seems highly technical when most of what I’ve seen from him lately has been looking at Oracle compatability and competitive comparisons. And I’m also looking forward to some sessions from newer speakers – Pavan Kristipati, Ken Shaffer, and Saurabh Agrawal.

Laying Out Your Schedule Ahead of Time

I hear there will be a mobile app, but as of the time I’m writing this, it is not yet available. IDUG will send out an email to all registered attendees when the mobile app is available. You can still lay out your schedule ahead of time, though I have no idea if it will translate into the mobile app or not. It’s a bit hard to get to this year, but start by clicking on the “Registration” Tab.
From that screen, click on one of the blue “REGISTER” buttons. They look like this:

On that screen, click on “View Your Registration Details”:

Then click on “My Agenda”:

Once you’re in that screen, you have options to edit your agenda. For each time slot, you can see the available sessions and select one or more that you are interested in. I tend to start by adding every session for the 4 tracks I’m interested in to my agenda, then look at the descriptions and speakers one by one and remove the ones that are less interesting. I end up with a lot of double scheduled slots this way, but this also allows me to talk to others at the conference about the sessions I’m interested in and make last-minute decisions. I don’t know how it ends up this way, but there are some times where there are really three can’t miss sessions, and others where there doesn’t seem to be much going on.

From this screen, there are also options to print your agenda and search for sessions. I didn’t find the search to be working by keyword or speaker name, but it can help you search by session type or time/day.

Who Won’t Be There This Year

One of my favorite people on the planet, Melanie Stopfer, will not be at the conference this year. IDUG just won’t be the same without her. I’ve never been to a conference that Melanie wasn’t at.

Susan Visser, the queen of publishing and networking won’t be there either. One of my early years attending IDUG Susan took me around and introduced me to everyone. She heads up IBM publishing for Information Management, and is usually the one running the book table or booth wherever it is. She’s turned into a social media goddess. I always drop by to see her when I want to meet some new folks and get a dose of girl-power. I will miss her.

Dale McInnis presents awesome technical details on High Availability, and I have learned so much from him over the years. I love how his presentations change over time to give me more technical detail. His abstracts were not accepted this year. I’m sad to not be able to hear him speak at this conference! However, I will feel less bad if I don’t get a session approved one of these years – if Dale can be rejected, then I’m sure to be at some point.

My Top Picks

Dang, as usual, I want to go to two sessions in almost every time slot. I have to start by mentioning the two sessions I’m presenting:

1. C13 – DB2 Basics: Writing SQL with Examples on DB2 10.5 System Views and Table Functions – Wed, May 06, 2015 (02:15 PM – 03:15 PM) – In the spirit of my DB2 Basics series on the blog, this starts at a very easy level. If you frequent Dan L presentations, you will be bored. But I also include several things that even experienced DBAs might not use much like Common Table Expressions. This session is designed for the physical DBAs who can do a redirected restore in their sleep, and even an explain, but haven’t had to write much SQL. The MON_GET monitoring interfaces require the ability to write SQL, and I share a bunch of SQL I find useful in administering DB2.
2. D17 – Supporting WebSphere Commerce DB2 Databases in the Real World Including WCS 7 and DB2 10.5 – Thu, May 07, 2015 (01:00 PM – 02:00 PM) – This session is heavily focused on WebSphere Commerce and some of the details that DBAs supporting WebSphere Commerce databases need to know that are in the DBA realm or cross the line slightly to the application side of the line.

Other sessions I’m excited about, in date/time order:

• C01 – Going BLU: How to Convert Existing Databases & Tables to BLU Acceleration – Matthias Nicola and Naresh Chainani
• C02 – DB2 DPF: What a single partitioned DBA needs to know in 10 points – Pavan Kristipati
• C03 – DB2 Cancun: A Technical Overview – Matt Huras
• D04 – More Sage Advice: Invaluable DB2 LUW Performance Insights from Around the Globe – Scott Hayes
• D05 – Performance “Worst Practices” for DB2 LUW – Steve Rees
• F05 – Do You See What I See? The Secret Powers in the DB2 Administrative Routines and Views – Steve Mazer
• D06 – Data Purge Algorithm – Efficiently Purge Terabytes of Data from the Database – Saurabh Agrawal
• E06 – Hidden Gems: The Top 10 Most Under-utilized Functions in DB2 – Matthias Nicola and Jim Seeger (saw this at Insight – it’s excellent!)
• D07 – Top Ten DB2 Explain Tips – John Hornibrook
• I’m conflicted about which SIG to go to – SIG 1A- Fun with SQL or SIG 1C-DB2 BLU Acceleration
• C12 – DB2 LUW Performance FAQs, volume 5 – Solving More Puzzles! – Steve Rees
• D12 – Doing the Long Jump : How To Upgrade By 4 DB2 for LUW Versions In One Step – Philip Nelson
• E15 – Configuration (Mis) Management – Deploying DDL changes – Ken Shaffer
• C15 – Why the University of Toronto is feeling BLU – Faster, Smaller, and Simpler in Only a Few Hours! – Brenda Boshoff and Adam Storm. Brenda was my thread chair at the very first conference I ever spoke at, and Adam is just awesome.
• E16 – BLU Insert, Update and Delete – An Insider’s Perspective – Adam Storm (wow, two sessions in a row!)
• C16 – db2batch Tips and Tricks – Paul Turpin
• SIG2C – LUW Mini Panel
• D19 – Monitoring BLU Acceleration In Depth – David Kalmuk
• E20 – DB2 10.5 New Features – A Query Optimizer Perspective – John Hornibrook
• D21 – Getting Top Performance from DB2 pureScale – a Migration Guide – Steve Rees

Educational Seminars and Hands On Labs

The educational seminars are spread throughout the conference this year. the ones I find most interesting are:

1. EDS4 – DB2 for LUW Top Gun Performance Highlights
2. EDS6 – Advanced SQL Coding and Performance

I’ve also heard the certification workshop is good if you’re working on the exam it is targeted at. I’ve heard mixed reviews of the PD and Troubleshooting one – people who loved it and people who really didn’t.

I suspect some of the Hands-On labs will be similar to the ones at Insight – if so, HOL5 – OLTAP – The New Frontier of Database Workload Powered by DB2 with BLU Acceleration may be good. Too bad they didn’t take the HADR/TSA one from Insight – that one was awesome.

Mini-Panels

The LUW mini panel is usually one of my favorite sessions of the conference. Even if I don’t have questions, I love hearing the details the experts get into or avoid answering others’ questions. It’s a can’t-miss.

Evening Events

If you just go to sessions and keep to yourself, you’re missing half of the conference. I’m an introvert, and even I go to something every evening. The friends you make and the contacts are so valuable. There is usually a party thrown by IBM and one by CA. And there’s the Dine-Around

Dine Around

The Dine-around is a group dinner at some restaurant. You pay for your own meal. Each dine-around is “anchored” by two well-known speakers, and generally has about 20 attendees. This gives you an opportunity to speak in a more intimate environment with speakers and gives some structure for those new to the conference.

Dine around sign up will be put up on a bulletin board somewhere – IDUG should announce it in the general session.

I’ll be one of the hosts at one of the dine-arounds this year – I don’t yet know who I’ll be with or where we’ll be going.

Zero Weight Gain

It is easy to let go of good habits and revisit bad ones during IDUG. Do you have a fitness tracker? Join me on matchup.io to compete with others to still meet your activity goals during the conference!

Summary

I am so looking forward to the IDUG conference! I also love to meet readers, so if you see me, come up and introduce yourself and we can chat. I’ll be in the Expo at the IDUG DB2 hub from 11:30 to 11:45 on Thursday, so come say hello then. Can I start packing yet?

When connecting between different versions of DB2, you generally need to bind packages from the different versions against the DB2 database. Just because you’ve bound db2ubind and db2cli on the server does not mean that all possible clients connecting in are covered. Sometimes binds happen implicitly, but sometimes they don’t. And you don’t necessarily want to grant a user rights to bind packages just because they’re going to need to do it once every 6 months when you apply a Fixpack or upgrade. It used to be that we didn’t have much choice in the matter – we had to get a client of the right version to bind from that client. But, luckily, in recent years, IBM has provided us with the files we need to perform binds for any version and Fixpack we may need.

Scenario

After a client’s recent upgrade (from 9.5 to 10.1), they reported getting this error when running SQL from the Control Center:

SQL0572N  Package "NULLID.SQLC2H20" is inoperative

I had, of course, run binds for db2ubind, db2cli, db2schema, and an overall db2rbind just after the upgrade. But this error clearly tells me that there’s something not bound – likely db2ubind.

I proceeded to ask the client their version, and they sent me this screen shot:

That translates to 9.7 Fixpack 0. I suggested the client be upgraded (preferably to 10.1, which is the server version), but also proceeded to get the bind files and bind them.

Resolution

First I had to get the bind files. I had no desire to install 9.7 Fixpack 0 on any of my test systems, so I got the files from the IBM site: http://www-01.ibm.com/support/docview.wss?uid=swg21501302
Each of the files there contains directories for each fixpack.

After uploading the file to my system and untarring/uncompressing it, I have these directories:

fixpak00a_s090610  fixpak02_s100514   fixpak04_s110330  fixpak07_s121002
fixpak00_s090521   fixpak03a_s101006  fixpak05_s111017  fixpak08_s130316
fixpak01_s091114   fixpak03_s100901   fixpak06_s120629  fixpak09_s131204

Note that in each directory name is the Build level that matches up to the value in the screen shot above – I can use that to understand which directory I need in this case. I think it is possible that if you have a special build, these numbers might not match up.

Changing directories into fixpak00_s090521, I see the normal list of bind files like I would in $HOME/sqllib/bnd. It is then an easy matter to connect to the database and issue the bind command. I do get some fairly normal warnings.

$ db2 connect to redacted

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.1.4
 SQL authorization ID   = DB2INST1
 Local database alias   = REDACTED

$ db2 "bind @db2ubind.lst blocking all grant public"

LINE    MESSAGES FOR db2ubind.lst
------  --------------------------------------------------------------------
        SQL0061W  The binder is in progress.

LINE    MESSAGES FOR db2ueiwi.bnd
------  --------------------------------------------------------------------
 2239   SQL0204N  "SYSTEM.SYSUSERAUTH" is an undefined name.
                  SQLSTATE=42704
 2243   SQL0204N  "SYSTEM.SYSUSERAUTH" is an undefined name.
                  SQLSTATE=42704

LINE    MESSAGES FOR db2clpnc.bnd
------  --------------------------------------------------------------------
        SQL0595W  Isolation level "NC" has been escalated to "UR".
                  SQLSTATE=01526

LINE    MESSAGES FOR db2arxnc.bnd
------  --------------------------------------------------------------------
        SQL0595W  Isolation level "NC" has been escalated to "UR".
                  SQLSTATE=01526

LINE    MESSAGES FOR db2ats_sps.bnd
------  --------------------------------------------------------------------
 1173   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1203   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1234   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1482   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1499   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1517   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1555   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1679   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1696   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1715   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1732   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1895   SQL0204N  "SYSTOOLS.ADMINTASKSTATUS" is an undefined name.
                  SQLSTATE=01532
 1950   SQL0204N  "SYSTOOLS.ADMINTASKSTATUS" is an undefined name.
                  SQLSTATE=01532
 1962   SQL0204N  "SYSTOOLS.ADMINTASKS" is an undefined name.
                  SQLSTATE=01532
 1979   SQL0204N  "SYSTOOLS.ADMINTASKSTATUS" is an undefined name.
                  SQLSTATE=01532

LINE    MESSAGES FOR db2adminotm.bnd
------  --------------------------------------------------------------------
  342   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  371   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  455   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  536   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  570   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  590   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  671   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  767   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532
  903   SQL0204N  "SYSTOOLS.ADMIN_MOVE_TABLE" is an undefined name.
                  SQLSTATE=01532

LINE    MESSAGES FOR db2ubind.lst
------  --------------------------------------------------------------------
        SQL0091N  Binding was ended with "0" errors and "28"
                  warnings.

When you’re considering dropping and re-creating a table, view, or MQT, it is critical to ensure that you consider any dependencies. When dropping a table, any MQTs that rely on it will be dropped, and views marked inoperative. With multiple levels, it can be difficult to Identify everything. Recursive SQL to the rescue!

Purpose

To list all MQTs and views dependent on a single table, MQT, or view. This SQL does NOT look at foreign keys to determine child tables.

Version

Should work on most versions of DB2.

Statement

Note: replace SCHEMA_NAME and TABLE_NAME with the values for the table you are looking for dependencies for

with d1 (tabschema, tabname, dtype, bschema, bname) as (select tabschema
    , tabname
    , dtype 
    , bschema
    , bname
from syscat.tabdep 
where bschema='SCHEMA_NAME' and bname='TABLE_NAME'
UNION ALL
Select d2.tabschema
    , d2.tabname
    , d2.dtype 
    , d2.bschema
    , d2.bname
from d1, syscat.tabdep d2 where d2.bschema=d1.tabschema and d2.bname=d1.tabname 
)
select * from d1
 with ur

Sample Output

TABSCHEMA TABNAME                           DTYPE BSCHEMA    BNAME
--------- --------------------------------- ----- ---------- ----------------------
SQL0347W  The recursive common table expression "DB2ADMIN.D1" may contain an
infinite loop.  SQLSTATE=01605

DB2       DP_WEEKLY_SNAPSHOT                V     DB2         ALL_ORDERS
DB2       DIGGER_PORTAL_PERFORMANCE         S     DB2         ALL_ORDERS
DB2       DIGGER_PORTAL_WEEKLY_SNAPSHOT     S     DB2         ALL_ORDERS
THESYS    DIGGER_PROFILE                    V     DB2         ALL_ORDERS
THESYS    DIGGER_STATS_COUNT                S     DB2         ALL_ORDERS
THESYS    TALLIER_EXPORT_TO_TPS             V     DB2         ALL_ORDERS
ONNTRAK   ORDER_SUMMARY                     V     DB2         ALL_ORDERS
THESYS    DIGGER_STATS                      V     THESYS      DIGGER_STATS_COUNT

  8 record(s) selected with 1 warning messages printed.

The SQL0347W is returned with any recursive SQL and can be ignored as long as you’re careful not to actually incur an infinite loop.

IDUG North America is over for another year. I enjoyed every minute of the conference this year, and was basically going solid from 7 AM to midnight every single day. I look forward to catching up on my sleep at home. I am typically someone who needs a solid 8-10 hours and I don’t drink and don’t party, so this is an unusual pattern for me.

Overall Conference

I am no longer surprised to meet people at conferences who have read my blog. The thing that surprised me this time was the fact that every single LUW person I talked to knew of the blog and recognized me. I loved speaking to people about technical problems and helping people with directions to investigate with issues they described to me. There was one dinner where I introduced myself to a room of maybe 50 people, and when I mentioned the blog, I got a round of applause.

This was much needed motivation for me. The blog is a lot of work and I was in a phase over the last few weeks where I was wondering if I really should be continuing to write it – if I might do better spending the time on it in really pushing a users group in Denver, volunteering for IDUG, or in actually having a personal hobby.

It’s hard just looking at the numbers on a computer for me to remember that each of those numbers is a person, and some of them are having trouble finding what they need until they find this blog. The conference helped me realize that, for me, it is a moral imperative to keep blogging. It really does help people and is one of my ways of contributing to the world. If that sounds cheesy or conceited, I’m sorry, but I don’t take it back.

If you read this blog and it helps you, I want you to dedicate some time to paying it forward – mentor another dba or person in your life, start a blog or podcast, record an educational video, volunteer with IDUG, run or contribute to a users’ group, participate in this awesome DB2 community that we have. I really believe that we have something special in the cooperation and knowledge sharing that we participate in. If you’re worried about the future of DB2, this kind of participation is one of the big things that can make a software product successful.

My Favorite Sessions and Speakers

I have to call out a few things that were just the most awesome about this conference.

One session that stood out in my mind was E21 – DB2 10.5 New Query Optimizer Features – a Query Optimizer Perspective. It was a session that had such good technical material that several things were over my head – a feeling I really value these days as it becomes harder for me to find advanced technical material that both applies directly to what I do and is also hard to understand. John Hornibrook will be on my priority list for speakers to hear at future conferences.

On the speaker side, I learned so much and got so much from Steve Rees’ presentations. Steve is a very nice person and I’ve interacted with him before – he helped to review one of my developer works articles. I got more ideas for blog articles and good pieces of technical information in his sessions than anywhere else. His sessions included:
D05 – Performance “Worst Practices” for DB2 LUW
Wow, the thresholds that it sometimes seem so hard to find for some performance metrics were all over this presentation – I have EIGHT blog entry ideas from this session alone, and have a couple of areas I want to do further research on.
C12 – DB2 LUW Performance FAQs, volume 5 – solving more puzzles!
I got 4 blog post ideas out of this session, and I got a direction for investigating a very specific performance problem that I’ve been fighting for a client. There was an excellent description of how to analyze if your package cache is too large.
D19 – Getting Top Performance from DB2 pureScale – a Migration Guide
This session had technical details on pureScale performance that were just excellent, and gave me an excellent direction for a pureScale sandbox that I would like to set up. No blog entry ideas, but I took a lot of notes to go back to later if I get to a place where I get to blog about pureScale.
In Melanie Stopfer’s absence Steve should absolutely get the title of best IBM speaker at this conference to me.

A speaker I hadn’t noticed at previous conferences is David Kalmuk. I loved how his slides included detailed SQL that could be used to investigate and solve problems. While I don’t get my hands on WLM much outside of a couple of BLU implementations, I still got a few great ideas for technical things to try out of C09 – Rounding up Outlaw Queries in DB2. From E18 – The Latest in Advanced Performance Diagnostics for SQL, I got two great blog entry ideas, and wrote a note about “finally details on how to query the freaking activity event monitor tables” – which is something I’ve struggled with. Also, I loved the insight that mon_get_pkg_cache_stmt only shows completed SQL while mon_get_activity detail shows details for currently executing statements. And finally (D21/19) – Monitoring BLU Acceleration in Depth netted me another blog entry idea and really provided some details I had been missing on how to look at BLU performance.

Newer Speakers

I’m proud to spend time mentoring and supporting newer speakers – people who are speaking for the first or second time. Now granted, this is only my third conference speaking. It is amazing to me how quick it is to become a mentor.

A few newer speakers:

Saurabh Agrawal:

Abhik Roy:

db2commerce.com bloggers

My own presentation:

Michael Krafick, a regular guest-blogger:

Pavan Kristipati – his second IDUG and his second IDUG speaking (and a guest-blogger):

Sessions I wish I Hadn’t Missed

There is so much going on at a conference, and there are just too many good things to be able to go to everything. Here are a few sessions I wished I could have attended:

• I hear Dale McInnis had a second replacement session that he did. I’m not sure when, but I missed that it was happening, and was not there
• D02 Sizing DB2 BLU Solutions with Jessica Escott
• C04 DB2 Business Continuity with Chris Eaton
• D09 DB2 Data Movement Utilities – A Comparison – Jeyabarathi Chakrapani (JB)
• D12 Doing the Long Jump: How to Upgrade by 4 DB2 for LUW Versions in One Step with Philip Nelson
• C15 Why the University of Toronto is Feeling BLU – Faster, Smaller, and Simpler in Only a Few Hours – Brenda Boshoff and Adam Storm
• D15/D16 – Query Performance and Optimixer Specific Type of Issues with Anthony Reina
• E21 – the ABCs of Filter Factors (aka Selectivity) with Joe Geller

Panels

My favorite sessions of any conference are the Panels, and both the DB2 LUW pannel and the BLU SIG did not dissapoint. If you’ve ever been in a session that I attended, you probably realize that I’m a big question-asker myself. But that’s not why I like these sessions. I like these sessions to hear the questions that others are asking. I learn so much from what others ask and how these panels seem to go in interesting directions.

One interesting thing that was emphasized at the Panel on Thursday was the DB2 RFE Community. This allows people to request enhancements to DB2 and vote on which other public requests are most important to clients.

In the session, IBM said that you have to mark your request public to have it voted on. However, when I went in to submit one, I could not change it to public. If I look up the top requests for DB2 LUW, they ALL have only one vote. So I think there’s a problem with this feature. Anyone else experience this or have a different experience?

Networking and Friends

I am a hermit and an introvert. I work from home, and enjoy that for the lack of human interaction it requires from me. But at IDUG I completely flip modes and somehow just feed on the energy of social interaction. When I go to IBM Insight, it tends to trigger anxiety and while I still socialize and have a fun time with some people, I get time up in my room alone to recover from that. I have a group of IDUG friends that I would rather hang out with than be alone, and that’s saying something for me.

I also make contacts with people in the DB2 community who are just as much DB2 Nerds as I am, and who I can ask technical questions of when I can’t figure something out or when I need additional details for my blog or for a particular technical problem.

Some fun pictures:

Dine-around at Legal Sea Foods:

The IBM had a party at a place with a mechanical Bull – Here are a couple of pics of that. It was hilarious to see my DB2 heroes riding the bull.

Step challenge results

Well, it’s no surprise to me that Ian beat the rest of us by 20,000 in the IDUG Step Challenge:

I’m proud to have worked hard enough to come in second. You should have seen me wandering around the Philly and Denver airports to make that work rather than sitting and relaxing.

Let’s keep the spirit of competition up – anyone with a fitness tracker can join a new challenge starting May 17 here: https://matchup.io/group/5469/db2-after-idug-challenge. We need at least one more person to join – matchup.io works with a wide range of fitness trackers – join us today. This challenge is different than the one done at IDUG. The one at IDUG was an overall-count challenge. This one will pit each person against another person each day, and the one with the most wins at the end is the winner.

And the Award Goes To

While I’m limited on what I see and cannot make overall statements about what was best when it was impossible for me to see more than maybe 25% of the sessions, there are a few categories I’d like to recognize speakers or sessions in. The “award” is simply bragging rights.

Most Blog Entry Ideas

Winning both for the most blog entry inspirations in a single session AND the most blog entry ideas across the whole conference is Steve Rees. I came away with no fewer than 8 blog ideas from one of his sessions, and a whopping total of 12 blog ideas across the 3 sessions of his that I attended.

Most Challenging Useful Information

Winning in the category of useful information that really challenged Ember’s brain to understand is John Hornibrook’s session – E21 – DB2 10.5 New Query Optimizer Features – a Query Optimizer Perspective

Most Entertaining Speaker

Adam Storm wins for using an awesome restaurant example to both entertain and walk us through BLU inserts, updates, and deletes in E16 – BLU Insert, Update, and Delete – An Insider’s Perspective. He manages to be both entertaining and incredibly informative at the same time. I aspire to work on the entertainment portion of my speaking, and he and Michal Krafick are my role models for this.

First Download

The award for the first presentation I’m downloading when IDUG makes the full audio available Goes to Paul Turpin for C16 – db2batch Tips and Tricks – I was sad to have to miss this session.

Presentation I Sure Hope They Recorded

This award goes to D07 – Top Ten DB2 Explain Tips by John Hornibrook. I’ll be downloading the slides to check this one out, but sure hope they recorded it as well.

Best Hair

Scott Hayes. Need I say more? I also love Scott’s contributions to the DB2 community and his work in mentoring DBAs, even when they don’t buy his software.

Next Year

The IDUG North American technical conference next year will be in Austin, TX.

For the 5th blogiversary of db2commerce.com, I’m taking a European trip. It has been more than decade since I’ve been overseas, so I’m very much looking forward to it.

I’ll be speaking at the DB2 Symposium in Einhoven on 11 June! I’ll be giving a day-long intensive on Lessons Learned Running DB2 for E-Commerce. An outline of what I’ll be talking about:

• High Availability – The Database has to be Online Before Anyone Cares About Performance
  • Overview of high-availability options and considerations
  • How does HADR work – knowing the details can help you make configuration decisions
  • How to set up HADR – a step-by-step guide
  • Using the HADR tools – HADR Simulator, DB2 Log Scanner, and the HADR Calculator
  • How to set up TSAMP to automate HADR failover – another step-by-step guide
  • TSAMP Advanced Topics – beyond db2haicu
• Monitoring E-Commerce and Transaction-Processing Databases
  • Looking for database problems and helping others look for problems at other layers
  • What to watch during peak retail and transactional periods
  • Using db2top for real time monitoring and lock investigation
  • Using db2pd for system investigation
  • Using the MON_GET* table functions
• Indexing Tips From the Real World
  • Why low-cardinality indexes are bad for performance
  • Finding problem SQL that could benefit from indexing
  • Mining your package cache for problem SQL
• Data Pruning – Smaller is Better!
  • Separating workloads for best performance
  • Tips and tricks for deleting large amounts of data
• Useful SQL for Administering DB2 Databases

The DB2 Symposium is a slightly different format than other conferences and educational opportunities. It provide three days, each with a different speaker with tracks for DB2 for LUW and for Z/OS. It is a fairly small group setting that allows attendees to really connect with the speaker.

You can still sign up if you’d like to attend. There are a couple of great LUW options on 10 June as well – you can sign up for just one day or for multiple days.

I was a bit shocked when I searched my own blog for entries on db2look and came up with nothing. While it’s not a complicated tool, db2look is an essential tool.

What db2look Can Do

db2look is a tool provided with DB2. db2look’s primary purposes are:

• Mimic Statistics – To generate the SQL to update statistics with the idea that they will be run in one database to match statistics in another database.
• Generate SQL – To generate all or part of the SQL required to re-create a database.

The tool is generally referred to as a mimicking tool – generating the SQL to mimic something from one database in another database. It can be used for a single table, a schema, or the whole database. There are a wide range of options to specify exactly how deep you want to go with the mimicking.

IBM DB2 Info Center page on db2look

Authorization Required

For much of what db2look does, you’ll just need select on the system catalog tables, which in many cases may be granted to PUBLIC by default. For some options, you may need one of the following:

• SYSADM
• SYSCTRL
• SYSMAINT
• SYSMON
• DBADM
• EXECUTE privilege on the ADMIN_GET_STORAGE_PATHS table function

You’ll also need connect authorization on the database, as db2look establishes its own database connection. If no one has ever run db2look on that databse before, you may have to also bind the db2look packages (which db2look may try to do automatically), and therefore may need bindadd permission.

db2look Syntax

The db2look syntax is not difficult. You can run db2look either for a local database or over a remote connection – though it doesn’t play nice across versions sometimes if the local and remote versions are significantly different. You always have to provide db2look with a database name using the -d option. If connecting to a remote database, you will also have to provide a user name and password.

In DB2 10.1 and DB2 10.5 a couple of new options have been added – the -createdb option will also generate the create database statement needed to create a new, identical database. The -printdbcfg option will print out the database configuration statements as well. These were added to 10.1 in a fixpack, so are only available if you either created the database on a more recent fixpack, or if you issued the db2updv10 command after a fixpack or version upgrade (and you should have).

The -e option is what you want if you’re extracting SQL to re-create objects or the database. You can optionally specify tables or schemas to limit the output. You may want to use the -x or -xd options to also generate grant statements for the objects you’re extracting the SQL for. If your intention is to mimick the whole database, the -l option is useful to also generate the SQL to create things like tablespaces and bufferpools. If part of what you’re extracting includes triggers or SQL stored procedures, you may want to use the -td <delimiter> option to specify a non-standard statement delimiter (something other than ;).

The -m option is what you want to generate the update statements to mimic statistics.

The -o option is used to specify an output file name.

There are certainly other options you may want to consider using in specific situations.

When to use db2look

There are a number of scenarios where db2look is useful.

• Generating the syntax to create a single table in one environment that already exists in another environment.
• Generating the syntax to compare an object in one environment to an object in another environment.
• As part of a disaster recovery plan. If you regularly generate a full db2look, you’re prepared to re-create objects that may have been incorrectly dropped or changed, or in a worst-case scenario to create an empty version of your database.
• Generating the syntax from a production environment that can be used to create a non-production or development environment.
• Generating the syntax to move all objects in a schema from one database to another (object definition only)
• Generating the syntax to re-create objects as a backout plan for object changes
• Along with db2move, db2look can be used to move a DB2 database from one environment to another when a backup/restore is not possible (such as between DB2 versions that are more than 2 versions apart or between differing OS platforms)
• Generating the syntax to recreate everything in the database, but then altering the output file before it is executed to make significant structure changes (though admin_move_table is now often used for these within the same database)

db2look Caveats and Warnings

db2look does not extract the data within the database in any way. You cannot use it as a replacement for backup/restore. Also be aware that db2look output generated on some newer versions may not directly work on older versions – clauses like organize by row work on DB2 10.5, but not on any older versions.

In some older versions, the options to specify a table or schema were not very reliable. This may have been as far back as DB2 version 7, but it is easy to make a mistake and not get the SQL you thought you were getting. Always review the output file before relying on it – especially in situations where you are relying on having the data.

db2look does not have a significant performance impact, especially considering the normally infrequent basis when it is used. It does not affect database availability to generate the file, but if you run the generated file somewhere, that can have drastic impacts on database structure – with the right set of options, the file output will drop objects.

This week marks the 5th blogiversary of db2commerce.com! Wow, 5 years!

A few statistics:

Readers: thanks for your support, your attention and your involvement. I’m happy to have written so much material and to have such a great community to share it with!

I recently ran into a couple of db2look problems for different clients, and thought I’d share them in case someone else runs into them.

db2look is a tool used to mimic databases, objects, or statistics. If you are not familiar with it, check out my post on DB2 Basics: db2look.

db2look Problem with Packages Bound

This problem occurred when I was generating the syntax for a database to look at the syntax of a specific SQL stored procedure. I tend to generate full database syntax even when I’m looking only for a specific object because I kind of like having db2look files laying around – they tend to be handy. Also in some now-antiquated version, the options for pulling the syntax for only a single object were very unreliable, so I got into the habit of generating the file for the whole database.

Anyway, I went to run db2look and got this error (note, this was on a windows server – the same error could occur on other platforms):

PS D:\> db2look -d sample -e -a -x -o db2look.ddl
-- Generate statistics for all creators
-- Creating DDL for table(s)
-- Output is sent to file: db2look.ddl


SQL0805N  Package "NULLID.DB2L2K1M 0X4141414141644864" was not found.
SQLSTATE=51002

This error clearly tells me that there’s some issue with the binding of packages relating to db2look. This isn’t normal, as db2look will normally take care of the binds for me. To figure out which specific package I need to bind, I went to this useful DB2 reference page, and searched on the package name returned (DB2L2K1M). I find that the bind I need to perform is for the file db2lkfun.bnd:

Once I have that information, it’s a simple matter to bind the package. I happen to be using an ID that has dbadm. I connect to the database, and change directories to the bnd subdirectory of the SQLLIB directory (this is done from a powershell prompt – syntax for the cd may be different in a DB2 command window):

PS D:\> cd C:\"Program Files"\IBM\SQLLIB\bnd
PS C:\Program Files\IBM\SQLLIB\bnd> db2 bind db2lkfun.bnd blocking all grant public sqlerror continue

LINE    MESSAGES FOR db2lkfun.bnd
------  ----------------------------------------------         ----------------------
        SQL0061W  The binder is in progress.
        SQL0091N  Binding was ended with "0" errors and "0" warnings.

After this is complete, db2look ran as expected with no further problems.

db2look Problem when Run as Lower-Privilged ID

Now most of the time when I’m doing anything with a DB2 database, I have SYSADM and DBADM. So I don’t personally have many issues with permissions – but users come to me when they do. This issue was a user on a fairly newly-created database, so db2look wasn’t something they had run before as this user in this environment. They were using an application-specific ID to extract the syntax for a single table.

I was able to log in as the user in this case to exactly re-create the problem – which is nice when your security setup allows it.

The error they were getting was (this one is a Linux box):

app_user@server1:~$ db2look -d sample -e -z schema1 -t table1 -o table1.ddl
-- No userid was specified, db2look tries to use Environment variable USER
-- USER is: APP_USER
-- Specified SCHEMA is: SCHEMA1
-- The db2look utility will consider only the specified tables 
-- Creating DDL for table(s)

-- Schema name is ignored for the Federated Section
-- Output is sent to file: table1.ddl
--An error has occured during Binding

Error Message = 
SQL0551N  The statement failed because the authorization ID does not have 
the required authorization or privilege to perform the operation.  
Authorization ID: "APP_USER".  Operation: "EXECUTE". Object: 
"NULLID.DB2L2K1S".  SQLSTATE=42501


SQLCA   
Size    = 136
SQLCODE = -551
Tokens  = APP_USER � EXECUTE � NULLID.DB2L2K1S
RDS fn  = SQLRA13D
RC      = 0x801A006D = -2145779603
Reason  = 0x0000 = 0
Reason2 = 0x0000 = 0
Line #  = -20
Warning flags =         

In this case, DB2 is attempting to bind the db2look packages for us, but is unable to because this lower-privileged user does not have permissions needed to perform the binds or use the bound packages.

In this case, the easiest solution is to explicitly bind the packages, granting access to PUBLIC. To do that, you must know the names of the three packages related to db2look:

• db2look.bnd
• db2lkfun.bnd
• db2lksp.bnd

Once you know the package names, it is just a matter of changing directories to the bnd sub-directory of the SQLLIB directory and connecting to the database as a privileged user and performing the binds with the grant to PUBLIC. Note that you want to be careful of what permissions you give to PUBLIC – I tend to like to make sure PUBLIC does not have connect authority, which makes it a group for any permission you want to give to anyone who can connect to the database. Some restrict PUBLIC’s use even more than that, so depending on your security set up, you may be doing the grant differently.

db2inst1@server1:~/sqllib/bnd$ db2 connect to sample

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.5.5
 SQL authorization ID   = DB2INST1
 Local database alias   = SAMPLE

db2inst1@server1:~/sqllib/bnd$ db2 "bind db2look.bnd blocking all grant public sqlerror continue"

LINE    MESSAGES FOR db2look.bnd
------  ----------------------------------------------         ----------------------
        SQL0061W  The binder is in progress.
        SQL0091N  Binding was ended with "0" errors and "0" warnings.
db2inst1@server1:~/sqllib/bnd$ db2 "bind db2lkfun.bnd blocking all grant public sqlerror continue"

LINE    MESSAGES FOR db2lkfun.bnd
------  ----------------------------------------------         ----------------------
        SQL0061W  The binder is in progress.
        SQL0091N  Binding was ended with "0" errors and "0" warnings.
db2inst1@server1:~/sqllib/bnd$ db2 "bind db2lksp.bnd  blocking all grant public sqlerror continue"

LINE    MESSAGES FOR db2lksp.bnd
------  ----------------------------------------------         ----------------------
        SQL0061W  The binder is in progress.
        SQL0091N  Binding was ended with "0" errors and "0" warnings.

Once these three binds were complete, I logged back in as the application-specific id and tried the db2look again:

app_id@server1:~$ db2look -d sample -e -z schema1 -t table1 -o table1.ddl
-- No userid was specified, db2look tries to use Environment variable USER
-- USER is: APP_ID
-- Specified SCHEMA is: SCHEMA1
-- The db2look utility will consider only the specified tables 
-- Creating DDL for table(s)

-- Schema name is ignored for the Federated Section
-- Output is sent to file: table1.ddl

I actually had a blog entry started on this topic before IDUG. I knew this was possible, but not exactly how to do it. Then I sat in Scott Hayes’ session – D04 – More Sage Advice: Invaluable DB2 LUW Performance Insights from Around the Globe, and he had all the details I needed to really get rolling on trying this out – saving me valuable time in research.

Indexing is relatively easy with a simple data model, or even a complicated one that doesn’t join a lot of tables, once you get used to it. But get to a point where you’ve got views and tables and dozens of objects involved, where the time is not primarily spent on reading data from the tables and indexes, but in the joining and filtering, and it starts to get difficult.

I’m calling this multiple index regression analysis because we’re looking for which recommended indexes most impact our query performance. The DB2 design advisor can recommend many indexes to improve performance, and some of them are more impactful than others. In some analysis I was doing last week, one of the indexes turned out to only reduce query execution time by 0.0001 timerons – not worth the overhead and space for that particular index. This does not use actual statistical formulas like those used in Multiple Linear Regression, but I could see a whole-workload approach where that would be interesting.

This is a highly advanced DB2 topic. It is assumed that the reader understands terms like Timeron and gets the concepts and can read an explain and db2advis output.

Scenario

In this example, I’m working with this query:

select * from order_trace  where  c_state = ? and dl_ap_date2 >= ? and  reference not in (select reference from edi_204  where edi_204.reference = reference and edi_204.order_id = order_id  and CUST_s_location_id = ? ) and (status in (? ,? ) OR ORDER_TYPE = ? )  order by dl_ap_date2, dl_ap_time2, act_state, act_city, actual, order_id;

Doesn’t look so bad, right? except order_trace is a view that is quite complicated. So my explain plan looks like this:

Access Plan:
-----------
    Total Cost:         35208.6
    Query Degree:       1

                                                                                                                                                                                       Rows 
                                                                                                                                                                                      RETURN
                                                                                                                                                                                      (   1)
                                                                                                                                                                                       Cost 
                                                                                                                                                                                        I/O 
                                                                                                                                                                                        |
                                                                                                                                                                                      2.92646 
                                                                                                                                                                                     >^NLJOIN
                                                                                                                                                                                     (   2)
                                                                                                                                                                                      35208.6 
                                                                                                                                                                                      11324.4 
                                                                                                                                                                          /-------------+--------------\
                                                                                                                                                                      2.92646                             1 
                                                                                                                                                                     >^NLJOIN                          IXSCAN
                                                                                                                                                                     (   3)                            (  66)
                                                                                                                                                                      35062.3                          50.0091 
                                                                                                                                                                      11318.5                             2 
                                                                                                                                                      /-----------------+-----------------\              |
                                                                                                                                                  2.92646                                    1         595580 
                                                                                                                                                 >^NLJOIN                                 IXSCAN   INDEX: DB2     
                                                                                                                                                 (   4)                                   (  65)      ORDER_ID
                                                                                                                                                  34989.1                                 25.0083        Q1
                                                                                                                                                  11315.6                                    1 
                                                                                                                              /---------------------+----------------------\                |
                                                                                                                          2.92646                                             1            13398 
                                                                                                                         >^NLJOIN                                          IXSCAN     INDEX: SYSIBM  
                                                                                                                         (   5)                                            (  64)   SQL991011111520810
                                                                                                                          34915.9                                          25.0074          Q2
                                                                                                                          11312.7                                             1 
                                                                                             /------------------------------+-------------------------------\                |
                                                                                         2.92646                                                               1            2976 
                                                                                         NLJOIN                                                             IXSCAN     INDEX: SYSIBM  
                                                                                         (   6)                                                             (  63)   SQL991011113441210
                                                                                         34842.7                                                            25.0112          Q3
                                                                                         11309.8                                                               1 
                                        /--------------------------------------------------+---------------------------------------------------\              |
                                    2.92646                                                                                                       1          13398 
                                    TBSCAN                                                                                                     NLJOIN   INDEX: DB2     
                                    (   7)                                                                                                     (  32)      DRIVER_ID
                                    33816.6                                                                                                    268.586        Q4
                                    11237.2                                                                                                      19 
                                      |                                                                                             /------------+-------------\
                                    2.92646                                                                                        1                              1 
                                    SORT                                                                                        NLJOIN                         TBSCAN
                                    (   8)                                                                                      (  33)                         (  59)
                                    33816.6                                                                                     243.01                         12.7956 
                                    11237.2                                                                                       17                              1 
                                      |                                                                                /----------+----------\                   |
                                    2.92646                                                                           1                         1                 1 
                                    HSJOIN                                                                         NLJOIN                    TBSCAN            TEMP  
                                    (   9)                                                                         (  34)                    (  56)            (  60)
                                    33816.6                                                                        191.871                   38.3582           12.7911 
                                    11237.2                                                                          13                         3                 1 
                  /-------------------+--------------------\                                                /--------+--------\                |                 |
              139659                                       2.92646                                         1                     1              1                 1 
              ^HSJOIN                                      HSJOIN                                       ^NLJOIN               TBSCAN         TEMP              GRPBY 
              (  10)                                       (  13)                                       (  35)                (  38)         (  57)            (  61)
              9380.69                                      24432.7                                      37.7969               140.072        38.3536           12.7892 
              3516.17                                       7721                                           2                    10              3                 1 
         /------+-------\                        /-----------+-----------\                         /------+------\              |              |                 |
     139659              97515               2685.75                     152.176                  1                 1            1              1             0.364844 
     IXSCAN             IXSCAN               NLJOIN                      ^NLJOIN               IXSCAN            IXSCAN       TEMP           IXSCAN            IXSCAN
     (  11)             (  12)               (  14)                      (  19)                (  36)            (  37)       (  39)         (  58)            (  62)
     5805.14            3567.73              6632.09                     17800.5               25.0075           25.5699      140.067        38.3518           12.7892 
     2657.04            859.125              2970.8                      4750.2                   1                 2           10              3                 1 
       |                  |               /----+-----\                 /---+----\                |                 |            |              |                 |
     139659              97515        1875.29        1.43218       152.176         1            2976             139659          1           440024             19946 
 INDEX: DB2         INDEX: DB2        TBSCAN         IXSCAN        TBSCAN       IXSCAN    INDEX: BFAIRCHILD  INDEX: DB2       NLJOIN     INDEX: DB2        INDEX: DB2     
      WIZ29       IDX201171957580000  (  15)         (  18)        (  20)       (  31)   IDX201162148270000      WIZ1171      (  40)   IDX201161942390000    IDX_STOP_1
       Q23                Q25         209.649        38.3519       15258.7      25.5699          Q10               Q11        140.066          Q8                Q5
                                      17.3389           3          3605.98         2                                            10 
                                        |              |             |            |                                /------------+-------------\
                                      1875.29        139659        152.176      139659                            1                              1 
                                      SORT       INDEX: DB2        SORT     INDEX: DB2                         TBSCAN                         TBSCAN
                                      (  16)   IDX201171959320000  (  21)       WIZ1171                        (  41)                         (  50)
                                      209.649          Q24         15258.7        Q22                          76.1435                        51.1416 
                                      17.3389                      3605.98                                     5.00001                           4 
                                        |                            |                                           |                              |
                                      1875.29                      152.176                                   1.3729e-005                         1 
                                      IXSCAN                       NLJOIN                                      SORT                           TEMP  
                                      (  17)                       (  22)                                      (  42)                         (  51)
                                      209.323                      15258.6                                     76.1433                        51.137 
                                      17.3389                      3605.98                                     5.00001                           4 
                                        |                     /------+-------\                                   |                              |
                                       97515                56               2.71743                         1.3729e-005                         1 
                                  INDEX: DB2              TBSCAN             IXSCAN                            NLJOIN                         TBSCAN
                                IDX201171957580000        (  23)             (  26)                            (  43)                         (  52)
                                        Q26               25.0247            332.82                            76.1429                        51.1352 
                                                             1               35.2989                           5.00001                           4 
                                                            |              /---+----\                    /-------+-------\                      |
                                                            56         803.566       54670              1              1.3729e-005               1 
                                                          SORT         TBSCAN   INDEX: DB2           TBSCAN              FETCH                SORT  
                                                          (  24)       (  27)    IDX_ORDER_01        (  44)              (  48)               (  53)
                                                          25.0246      78.3923        Q21            51.1352             25.0077              51.135 
                                                             1         6.08317                          4                1.00001                 4 
                                                            |            |                             |              /----+-----\              |
                                                            56         803.566                          1       1.3729e-005        519        1.00058 
                                                          IXSCAN       TEMP                          SORT         IXSCAN     TABLE: DB2       FETCH 
                                                          (  25)       (  28)                        (  45)       (  49)          CODE        (  54)
                                                          25.0201      78.1369                       51.135       25.0072          Q15        51.1347 
                                                             1         6.08317                          4            1                           4 
                                                            |            |                             |            |                    /------+------\
                                                            519        803.566                       1.00058        519              1.00058         1.9199e+007 
                                                      INDEX: DB2       UNIQUE                        FETCH    INDEX: DB2             IXSCAN        TABLE: EDI     
                                                        IDX_CODE_01    (  29)                        (  46)     IDX_CODE_01          (  55)   EDI_204_ADDITIONAL_FIELDS
                                                            Q27        78.0063                       51.1347        Q15              38.3527             Q18
                                                                       6.08317                          4                               3 
                                                                         |                      /------+------\                        |
                                                                       803.566              1.00058         1.9199e+007            1.9199e+007 
                                                                       IXSCAN               IXSCAN        TABLE: EDI             INDEX: EDI     
                                                                       (  30)               (  47)   EDI_204_ADDITIONAL_FIELDS   EDI_204_A_F_NDX
                                                                       77.9929              38.3527             Q13                    Q18
                                                                       6.08317                 3 
                                                                         |                    |
                                                                    2.23029e+007          1.9199e+007 
                                                                   INDEX: DB2           INDEX: EDI     
                                                                 IDX201171956050000     EDI_204_A_F_NDX
                                                                         Q28                  Q13

A bit more horrendous, that, and looking at it in detail, there’s no obvious gigantic table scan or even fetch from a table after reading an index that provides a significant amount of the time.

Running db2advis on that, we get:

  19  indexes in current solution
 [34706.0000] timerons  (without recommendations)
 [19035.0000] timerons  (with current solution)
 [45.15%] improvement


--
--
-- LIST OF RECOMMENDED INDEXES
-- ===========================
-- index[1],    0.548MB
   CREATE UNIQUE INDEX "ECROOKS "."IDX1505210120080"
   ON "DB2     "."ORDER" ("TRAILER_SIZE" ASC, "DL_AP_DATE2"
   ASC, "CUSTOMER_RATE" ASC, "NOTIFIED_DATE" ASC, "DATE_ENTERED"
   ASC, "COMMODITY" ASC, "NEEDS_PAPERWORK" ASC, "LINE_OF_BUSINESS"
   ASC, "TOTAL_CHARGE" ASC, "MILES" ASC, "TERMINAL" ASC,
   "SCAC" ASC, "SEAL" ASC, "PO_NUM" ASC, "PU_NUM" ASC,
   "BL_NUM" ASC, "REFERENCE" ASC, "WHO_ENTERED" ASC,
   "ORDER_TYPE" ASC, "TRACTOR_ID" ASC, "DRIVER_ID" ASC,
   "DL_AT_TIME2" ASC, "DL_AT_DATE2" ASC, "DL_AT_TIME1"
   ASC, "DL_AT_DATE1" ASC, "DL_AP_TIME2" ASC, "DL_AP_TIME1"
   ASC, "DL_AP_DATE1" ASC, "PU_AT_TIME2" ASC, "PU_AT_DATE2"
   ASC, "PU_AT_TIME1" ASC, "PU_AT_DATE1" ASC, "PU_AP_TIME2"
   ASC, "PU_AP_DATE2" ASC, "PU_AP_TIME1" ASC, "PU_AP_DATE1"
   ASC, "S_LOCATION_ID" ASC, "CHASSIS" ASC, "CHECK_DIGIT"
   ASC, "TRAILER" ASC, "STATUS" ASC, "ORDER_ID" ASC)
   INCLUDE ("C_LOCATION_ID", "ACT_LOCATION_ID", "BTO_LOCATION_ID")
   ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[2],   20.087MB
   CREATE INDEX "ECROOKS "."IDX1505210116380" ON "DB2     "."EDI_204"
   ("CUST_S_LOCATION_ID" ASC, "ORDER_ID" ASC, "REFERENCE"
   ASC) ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[3],    0.048MB
   CREATE UNIQUE INDEX "ECROOKS "."IDX1505210115300"
   ON "DB2     "."VENDOR_UNIT" ("VENDOR_UNIT_ID" ASC)
   INCLUDE ("VENDOR_ID") ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[4],    5.778MB
   CREATE UNIQUE INDEX "ECROOKS "."IDX1505210115350"
   ON "DB2     "."LOCATION" ("LOCATION_ID" ASC) INCLUDE
   ("NAME") ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[5], 1673.419MB
   CREATE INDEX "ECROOKS "."IDX1505210115360" ON "EDI     "."EDI_204_ADDITIONAL_FIELDS"
   ("REFERENCE" ASC, "FIELD_NAME" ASC, "INSERT_DT" ASC,
   "FIELD_VALUE" ASC) ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[6],    0.013MB
   CREATE INDEX "ECROOKS "."IDX1505210116350" ON "DB2     "."CODE"
   ("CODE_TEXT" ASC, "TYPE" ASC, "CODE_ID" ASC, "CODE"
   ASC) ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[7],    9.739MB
   CREATE UNIQUE INDEX "ECROOKS "."IDX1505210121040"
   ON "DB2     "."RAIL_ETA" ("ORDER_ID" ASC) INCLUDE
   ("STATE") ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;

Some of those indexes I’m not likely to try. index[1] for example is awfully wide. index[2] is nearly identical to an existing index, with the difference being in the order of the last two columns. And 19 total indexes that DB2 wants to use for this one query? Wow, that’s a lot.

Also, I don’t have a full-size test or QA environment – the data size in my lower environments is significantly smaller, so it is hard for me to scale down to see what happens with different indexes in those environments. One option here is to mimic statistics in my lower environments. I’ve done that before, but I want to try mimicking the new indexes instead.

Setup steps

First, my explain tables are all currently shared tables. With multiple DBAs potentially working in this environment, I have two real choices to isolate my data – #1 is to create my own set of explain tables and make use of them. #2 is to note the time before I collect data and then the time after I collect explain and db2advis data and look only within that period to verify that only the data I want is included. I’m chosing the second option. With only 3 potential people running explains and none of us dedicated to that full time, this seems like a good option – we’re not likely to conflict.

Running the Explain and db2advis

I need to capture the explain and db2advis data before I work on anything. Note that I happen to be working in a Windows environment from a powershell prompt for this work – everything here would be the same on Linux/UNIX.

To capture my initial data, I’ll use:

PS D:\xtivia\queries> db2 "select current timestamp from sysibm.sysdummy1"

1
--------------------------
2015-05-21-10.00.07.793000

  1 record(s) selected.

PS D:\xtivia\queries> db2 set current explain mode explain
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 -tvf query13.sql
select * from db2.order_trace  where  c_state = ? and dl_ap_date2 >= ? and  reference not in (select reference from db2.edi_204 edi_204 where edi_204.reference = reference and edi_204.order_id = order
_id  and CUST_s_location_id = ? ) and (status in (? ,? ) OR ORDER_TYPE = ? )  order by dl_ap_date2, dl_ap_time2, act_state, act_city, actual, order_id
SQL0217W  The statement was not executed as only Explain information requests
are being processed.  SQLSTATE=01604

PS D:\xtivia\queries> db2exfmt -d comtrak2 -e db2admin -1 -o query13_exfmt.txt
DB2 Universal Database Version 9.7, 5622-044 (c) Copyright IBM Corp. 1991, 2009
Licensed Material - Program Property of IBM
IBM DATABASE 2 Explain Table Format Tool

Connecting to the Database.
Connect to Database Successful.
Output is in query13_exfmt.txt.
Executing Connect Reset -- Connect Reset was Successful.
PS D:\xtivia\queries> db2advis -d comtrak2 -i query13.sql -p |tee query13_advis.txt

Using user id as default schema name. Use -n option to specify schema
execution started at timestamp 2015-05-21-10.01.12.315000
found [1] SQL statements from the input file
Recommending indexes...
total disk space needed for initial set [1715.827] MB
total disk space constrained to         [35280.399] MB
Trying variations of the solution set.
Optimization finished.
  19  indexes in current solution
 [38772.0000] timerons  (without recommendations)
 [18742.0000] timerons  (with current solution)
 [51.66%] improvement


--
--
-- LIST OF RECOMMENDED INDEXES
-- ===========================
-- index[1],    0.532MB
   CREATE UNIQUE INDEX "DB2ADMIN"."IDX1505231651490"
   ON "DB2     "."ORDER" ("TRAILER_SIZE" ASC, "DL_AP_DATE2"
   ASC, "CUSTOMER_RATE" ASC, "NOTIFIED_DATE" ASC, "DATE_ENTERED"
   ASC, "COMMODITY" ASC, "NEEDS_PAPERWORK" ASC, "LINE_OF_BUSINESS"
   ASC, "TOTAL_CHARGE" ASC, "MILES" ASC, "TERMINAL" ASC,
   "SCAC" ASC, "SEAL" ASC, "PO_NUM" ASC, "PU_NUM" ASC,
   "BL_NUM" ASC, "REFERENCE" ASC, "WHO_ENTERED" ASC,
   "ORDER_TYPE" ASC, "TRACTOR_ID" ASC, "DRIVER_ID" ASC,
   "DL_AT_TIME2" ASC, "DL_AT_DATE2" ASC, "DL_AT_TIME1"
   ASC, "DL_AT_DATE1" ASC, "DL_AP_TIME2" ASC, "DL_AP_TIME1"
   ASC, "DL_AP_DATE1" ASC, "PU_AT_TIME2" ASC, "PU_AT_DATE2"
   ASC, "PU_AT_TIME1" ASC, "PU_AT_DATE1" ASC, "PU_AP_TIME2"
   ASC, "PU_AP_DATE2" ASC, "PU_AP_TIME1" ASC, "PU_AP_DATE1"
   ASC, "S_LOCATION_ID" ASC, "CHASSIS" ASC, "CHECK_DIGIT"
   ASC, "TRAILER" ASC, "STATUS" ASC, "ORDER_ID" ASC)
   INCLUDE ("C_LOCATION_ID", "ACT_LOCATION_ID", "BTO_LOCATION_ID")
   ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[2],   21.126MB
   CREATE INDEX "DB2ADMIN"."IDX1505231648190" ON "DB2     "."EDI_204"
   ("CUST_S_LOCATION_ID" ASC, "ORDER_ID" ASC, "REFERENCE"
   ASC) ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[3],    0.048MB
   CREATE UNIQUE INDEX "DB2ADMIN"."IDX1505231647110"
   ON "DB2     "."VENDOR_UNIT" ("VENDOR_UNIT_ID" ASC)
   INCLUDE ("VENDOR_ID") ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[4],    5.782MB
   CREATE UNIQUE INDEX "DB2ADMIN"."IDX1505231647160"
   ON "DB2     "."LOCATION" ("LOCATION_ID" ASC) INCLUDE
   ("NAME") ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[5], 1678.521MB
   CREATE INDEX "DB2ADMIN"."IDX1505231647170" ON "EDI     "."EDI_204_ADDITIONAL_FIELDS"
   ("REFERENCE" ASC, "FIELD_NAME" ASC, "INSERT_DT" ASC,
   "FIELD_VALUE" ASC) ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[6],    0.013MB
   CREATE INDEX "DB2ADMIN"."IDX1505231648160" ON "DB2     "."CODE"
   ("CODE_TEXT" ASC, "TYPE" ASC, "CODE_ID" ASC, "CODE"
   ASC) ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;
-- index[7],    9.806MB
   CREATE UNIQUE INDEX "DB2ADMIN"."IDX1505231652450"
   ON "DB2     "."RAIL_ETA" ("ORDER_ID" ASC) INCLUDE
   ("STATE") ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;
   COMMIT WORK ;


snip

1449 solutions were evaluated by the advisor
DB2 Workload Performance Advisor tool is finished.
PS D:\xtivia\queries> db2 set current explain mode no
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 "select current timestamp from sysibm.sysdummy1"

1
--------------------------
2015-05-21-10.03.06.369000

  1 record(s) selected.

What this gets me is all of the details about this query without any changes, and the recommended indexes – in this case 7 new indexes. I don’t really like all of those indexes, but I need to figure out which ones will give me the most impact and analyze other characteristics about them. Note the use of the -p option on the db2advis. This causes DB2 to retain information about the explain plan and recommended indexes.

We also need to note from this process the starting time and the ending time for the analysis – in this case:
Start: 2015-05-21-10.00.07
End: 2015-05-21-10.03.06

Note Costs

The next step is to gather the costs of this query before and after the indexes recommended by the db2 design advisor. Note that we have to fill in the timestamps from above to limit the data returned.

select 
    dec(total_cost,20,4) as before_total_cost
    , dec(io_cost,20,4) as before_io_cost
    , dec(CPU_cost,20,4) as before_cpu_cost
    , dec(Comm_cost,20,4) as before_comm_cost
from Explain_Operator
    ,(select min(explain_time) as mintime
        from Explain_Operator 
        where operator_type = 'RETURN' 
          and explain_time between timestamp('2015-05-21-10.00.07') and timestamp('2015-05-21-10.03.06')) as b
where 
    explain_time = b.mintime
    and operator_type = 'RETURN' 
with UR ;
BEFORE_TOTAL_COST      BEFORE_IO_COST         BEFORE_CPU_COST        BEFORE_COMM_COST
---------------------- ---------------------- ---------------------- ----------------------
            38772.4492             11552.6152         705355200.0000                 0.0000

  1 record(s) selected.

And:

select 
    dec(total_cost,20,4) as after_total_cost
    , dec(io_cost,20,4) as after_io_cost
    , dec(CPU_cost,20,4) as after_cpu_cost
    , dec(Comm_cost,20,4) as after_comm_cost
from Explain_Operator
    ,(select max(explain_time) as maxtime
        from Explain_Operator 
        where operator_type = 'RETURN' 
          and explain_time between timestamp('2015-05-21-10.00.07') and timestamp('2015-05-21-10.03.06')) as b
where 
    explain_time = b.maxtime
    and operator_type = 'RETURN' 
with UR ;
AFTER_TOTAL_COST       AFTER_IO_COST          AFTER_CPU_COST         AFTER_COMM_COST
---------------------- ---------------------- ---------------------- ----------------------
            18741.7597              5339.1113         131031752.0000                 0.0000

  1 record(s) selected.

What we have in the explain tables at this point is really two sets of data – the data from the original explain, and the data from an explain as if all of the recommended indexes existed.

Now here’s a query I prefer to work with of the existing and recommended indexes for this query. It uses both data on existing indexes from syscat.indexes and syscat.tables, along with data on the proposed new indexes from advise_index. It’s also one I run over and over again to help me keep track of what indexes I’m trying things with:

with ts as ( select max(explain_time) as maxtime
        from Explain_Operator 
        where operator_type = 'RETURN' 
          and explain_time between timestamp('2015-05-21-10.00.07') and timestamp('2015-05-21-10.03.06'))
select substr(name,1,25) as indname
    , substr(tbcreator,1,13) as tabschema
    , substr(tbname,1,18) as tabname
    , coalesce(si.fullkeycard, ai.fullkeycard) as fullkeycard
    , st.card
    , coalesce(si.uniquerule, ai.uniquerule) as uniquerule
    , use_index
    , exists
    , substr(coalesce(si.colnames, ai.colnames),1,60) as colnames 
from advise_index ai 
    left outer join syscat.indexes si on ai.tbcreator=si.tabschema and ai.tbname=si.tabname and ai.name=si.indname 
    left outer join syscat.tables st on st.tabschema=ai.tbcreator and st.tabname=ai.tbname 
    join ts on explain_time = ts.maxtime
order by exists, use_index, uniquerule desc, indname with ur;

I’m actually masaging that a bit with clpplus to get the output I want because substr isn’t doing so well at the powershell prompt for me at the moment:

                                                                              Index                Table                                                                                         
Index Name           Table Schema    Table Name                         Cardinality          Cardinality UNIQUERULE USE_INDEX EXISTS Column Names                                                
-------------------- --------------- ------------------------- -------------------- -------------------- ---------- --------- ------ ------------------------------------------------------------
IDX1505231647110     DB2             VENDOR_UNIT                               2976                 2976 U          Y         N      +VENDOR_UNIT_ID-VENDOR_ID                                   
IDX1505231647160     DB2             LOCATION                                140445               140445 U          Y         N      +LOCATION_ID-NAME                                           
IDX1505231651490     DB2             ORDER                                      642                52497 U          Y         N      +TRAILER_SIZE+DL_AP_DATE2+CUSTOMER_RATE+NOTIFIED_DATE+DATE_E
                                                                                                                                     NTERED+COMMODITY+NEEDS_PAPERWORK+LINE_OF_BUSINESS+TOTAL_CHAR
                                                                                                                                     GE+MILES+TERMINAL+SCAC+SEAL+PO_NUM+PU_NUM+BL_NUM+REFERE...  

IDX1505231652450     DB2             RAIL_ETA                                649793               649793 U          Y         N      +ORDER_ID-STATE                                             
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          Y         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          Y         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231648160     DB2             CODE                                         5                  520 D          Y         N      +CODE_TEXT+TYPE+CODE_ID+CODE                                
IDX1505231648190     DB2             EDI_204                                 295885              1014841 D          Y         N      +CUST_S_LOCATION_ID+ORDER_ID+REFERENCE                      
DRIVER_ID            DB2             DRIVER                                   13519                13519 U          Y         Y      +DRIVER_ID+FIRST_NAME+LAST_NAME+PREVIOUS_NUM+DISPATCHER     
IDX201162151160000   DB2             CITY                                     97516                97516 U          Y         Y      +CITY_ID+STATE+CITY+TIME_ZONE                               
WIZ1166              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+ZIP+LONGITUDE+LATITUDE+NAME+CITY_ID            
WIZ1171              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+NAME+NUMBER                                    
SQL991011111520810   DB2             DRIVER                                   13519                13519 P          Y         Y      +DRIVER_ID                                                  
SQL991011113441210   DB2             VENDOR_UNIT                               2976                 2976 P          Y         Y      +VENDOR_UNIT_ID                                             
IDX_CODE_01          DB2             CODE                                       518                  520 D          Y         Y      +TYPE+CODE_TEXT+CODE                                        
IDX_STOP_1           DB2             STOP                                     15883                21568 D          Y         Y      +ORDER_ID                                                   
IDX201161942390000   EDI             EDI_204_HUB_CUSTOMER                    451173               451173 D          Y         Y      +REFERENCE+DESCRIPTION                                      
IDX201171957580000   DB2             CITY                                     97516                97516 D          Y         Y      +STATE+CITY+CITY_ID                                         
IDX201171959320000   DB2             LOCATION                                140445               140445 D          Y         Y      +CITY_ID+DROP+ZIP+LONGITUDE+LATITUDE+NAME+LOCATION_ID       

I pull that output into Excel so I can note details as I work through each index and easily play with the numbers. It looks something like this:

I hide and add some columns, but I’m running this same query between every change I make so I keep track of what I’ve done and not done.

Analysis Techniques

At this point, there are two different methods that Scott laid out in his presentation on this topic – Index Addition and Index Subtraction.

Index Addition

The point of index addition is to run an explain on the query with each proposed index individually, to see the impact of that index alone. This works particuarly well when a great impact is seen from a few indexes and the combination of indexes does not make much difference. To use this method, we update ADVISE_INDEX to have USE_INDEX set to N for all indexes that do not already exist:

PS > db2 "update advise_index set use_index = 'N' where exists= 'N'"
DB20000I  The SQL command completed successfully.
                                                                              Index                Table                                                                                         
Index Name           Table Schema    Table Name                         Cardinality          Cardinality UNIQUERULE USE_INDEX EXISTS Column Names                                                
-------------------- --------------- ------------------------- -------------------- -------------------- ---------- --------- ------ ------------------------------------------------------------
IDX1505231647110     DB2             VENDOR_UNIT                               2976                 2976 U          N         N      +VENDOR_UNIT_ID-VENDOR_ID                                   
IDX1505231647160     DB2             LOCATION                                140445               140445 U          N         N      +LOCATION_ID-NAME                                           
IDX1505231651490     DB2             ORDER                                      642                52497 U          N         N      +TRAILER_SIZE+DL_AP_DATE2+CUSTOMER_RATE+NOTIFIED_DATE+DATE_E
                                                                                                                                     NTERED+COMMODITY+NEEDS_PAPERWORK+LINE_OF_BUSINESS+TOTAL_CHAR
                                                                                                                                     GE+MILES+TERMINAL+SCAC+SEAL+PO_NUM+PU_NUM+BL_NUM+REFERE...  

IDX1505231652450     DB2             RAIL_ETA                                649793               649793 U          N         N      +ORDER_ID-STATE                                             
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          N         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          N         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231648160     DB2             CODE                                         5                  520 D          N         N      +CODE_TEXT+TYPE+CODE_ID+CODE                                
IDX1505231648190     DB2             EDI_204                                 295885              1014841 D          N         N      +CUST_S_LOCATION_ID+ORDER_ID+REFERENCE                      
DRIVER_ID            DB2             DRIVER                                   13519                13519 U          Y         Y      +DRIVER_ID+FIRST_NAME+LAST_NAME+PREVIOUS_NUM+DISPATCHER     
IDX201162151160000   DB2             CITY                                     97516                97516 U          Y         Y      +CITY_ID+STATE+CITY+TIME_ZONE                               
WIZ1166              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+ZIP+LONGITUDE+LATITUDE+NAME+CITY_ID            
WIZ1171              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+NAME+NUMBER                                    
SQL991011111520810   DB2             DRIVER                                   13519                13519 P          Y         Y      +DRIVER_ID                                                  
SQL991011113441210   DB2             VENDOR_UNIT                               2976                 2976 P          Y         Y      +VENDOR_UNIT_ID                                             
IDX_CODE_01          DB2             CODE                                       518                  520 D          Y         Y      +TYPE+CODE_TEXT+CODE                                        
IDX_STOP_1           DB2             STOP                                     15883                21568 D          Y         Y      +ORDER_ID                                                   
IDX201161942390000   EDI             EDI_204_HUB_CUSTOMER                    451173               451173 D          Y         Y      +REFERENCE+DESCRIPTION                                      
IDX201171957580000   DB2             CITY                                     97516                97516 D          Y         Y      +STATE+CITY+CITY_ID                                         
IDX201171959320000   DB2             LOCATION                                140445               140445 D          Y         Y      +CITY_ID+DROP+ZIP+LONGITUDE+LATITUDE+NAME+LOCATION_ID       

Now one by one we set use_index equal to ‘Y’ for each index and run an explain using the ‘evaluate indexes’ option.

PS D:\xtivia\queries> db2 "update advise_index set use_index = 'Y' where name = 'IDX1505231647110'"
DB20000I  The SQL command completed successfully.

                                                                              Index                Table                                                                                         
Index Name           Table Schema    Table Name                         Cardinality          Cardinality UNIQUERULE USE_INDEX EXISTS Column Names                                                
-------------------- --------------- ------------------------- -------------------- -------------------- ---------- --------- ------ ------------------------------------------------------------
IDX1505231647160     DB2             LOCATION                                140445               140445 U          N         N      +LOCATION_ID-NAME                                           
IDX1505231651490     DB2             ORDER                                      642                52497 U          N         N      +TRAILER_SIZE+DL_AP_DATE2+CUSTOMER_RATE+NOTIFIED_DATE+DATE_E
                                                                                                                                     NTERED+COMMODITY+NEEDS_PAPERWORK+LINE_OF_BUSINESS+TOTAL_CHAR
                                                                                                                                     GE+MILES+TERMINAL+SCAC+SEAL+PO_NUM+PU_NUM+BL_NUM+REFERE...  

IDX1505231652450     DB2             RAIL_ETA                                649793               649793 U          N         N      +ORDER_ID-STATE                                             
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          N         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          N         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231648160     DB2             CODE                                         5                  520 D          N         N      +CODE_TEXT+TYPE+CODE_ID+CODE                                
IDX1505231648190     DB2             EDI_204                                 295885              1014841 D          N         N      +CUST_S_LOCATION_ID+ORDER_ID+REFERENCE                      
IDX1505231647110     DB2             VENDOR_UNIT                               2976                 2976 U          Y         N      +VENDOR_UNIT_ID-VENDOR_ID                                   
DRIVER_ID            DB2             DRIVER                                   13519                13519 U          Y         Y      +DRIVER_ID+FIRST_NAME+LAST_NAME+PREVIOUS_NUM+DISPATCHER     
IDX201162151160000   DB2             CITY                                     97516                97516 U          Y         Y      +CITY_ID+STATE+CITY+TIME_ZONE                               
WIZ1166              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+ZIP+LONGITUDE+LATITUDE+NAME+CITY_ID            
WIZ1171              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+NAME+NUMBER                                    
SQL991011111520810   DB2             DRIVER                                   13519                13519 P          Y         Y      +DRIVER_ID                                                  
SQL991011113441210   DB2             VENDOR_UNIT                               2976                 2976 P          Y         Y      +VENDOR_UNIT_ID                                             
IDX_CODE_01          DB2             CODE                                       518                  520 D          Y         Y      +TYPE+CODE_TEXT+CODE                                        
IDX_STOP_1           DB2             STOP                                     15883                21568 D          Y         Y      +ORDER_ID                                                   
IDX201161942390000   EDI             EDI_204_HUB_CUSTOMER                    451173               451173 D          Y         Y      +REFERENCE+DESCRIPTION                                      
IDX201171957580000   DB2             CITY                                     97516                97516 D          Y         Y      +STATE+CITY+CITY_ID                                         
IDX201171959320000   DB2             LOCATION                                140445               140445 D          Y         Y      +CITY_ID+DROP+ZIP+LONGITUDE+LATITUDE+NAME+LOCATION_ID

With that set, we set the current explain mode, and explain the query:

PS D:\xtivia\queries> db2 set current explain mode evaluate indexes
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 -tvf query13.sql
select * from db2.order_trace  where  c_state = ? and dl_ap_date2 >= ? and  reference not in (select reference from db2.edi_204 edi_204 where edi_204.reference = reference and edi_204.order_id = order
_id  and CUST_s_location_id = ? ) and (status in (? ,? ) OR ORDER_TYPE = ? )  order by dl_ap_date2, dl_ap_time2, act_state, act_city, actual, order_id
SQL0217W  The statement was not executed as only Explain information requests
are being processed.  SQLSTATE=01604

PS D:\xtivia\queries> db2 set current explain mode no
DB20000I  The SQL command completed successfully.

Note that the explain mode used is not yes or explain, but evaluate indexes.

Next, we must query the cost to record in our spreadsheet:

select 
    dec(total_cost,20,4) as this_index_total_cost
    , dec(io_cost,20,4) as this_index_io_cost
    , dec(CPU_cost,20,4) as this_index_cpu_cost
    , dec(Comm_cost,20,4) as this_index_comm_cost
from Explain_Operator
    ,(select max(explain_time) as maxtime
        from Explain_Operator 
        where operator_type = 'RETURN') as b
where 
    explain_time = b.maxtime
    and operator_type = 'RETURN' 
with UR ;

select 
THIS_INDEX_TOTAL_COST  THIS_INDEX_IO_COST     THIS_INDEX_CPU_COST    THIS_INDEX_COMM_COST
---------------------- ---------------------- ---------------------- ----------------------
            38772.4492             11552.6152         705353856.0000                 0.0000

  1 record(s) selected.

If you’re keeping track, that’s ZERO impact on timerons for that particular index. Now we have to move through each index in turn. I’m only showing the second one so you understand that each time we mark the other indexes as not being used, and mark only one index as being used:

PS D:\xtivia\queries> db2 "update advise_index set use_index = 'N' where exists= 'N'"
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 "update advise_index set use_index = 'Y' where name = 'IDX1505231647160'"
DB20000I  The SQL command completed successfully.

                                                                              Index                Table                                                                                         
Index Name           Table Schema    Table Name                         Cardinality          Cardinality UNIQUERULE USE_INDEX EXISTS Column Names                                                
-------------------- --------------- ------------------------- -------------------- -------------------- ---------- --------- ------ ------------------------------------------------------------
IDX1505231647110     DB2             VENDOR_UNIT                               2976                 2976 U          N         N      +VENDOR_UNIT_ID-VENDOR_ID                                   
IDX1505231651490     DB2             ORDER                                      642                52497 U          N         N      +TRAILER_SIZE+DL_AP_DATE2+CUSTOMER_RATE+NOTIFIED_DATE+DATE_E
                                                                                                                                     NTERED+COMMODITY+NEEDS_PAPERWORK+LINE_OF_BUSINESS+TOTAL_CHAR
                                                                                                                                     GE+MILES+TERMINAL+SCAC+SEAL+PO_NUM+PU_NUM+BL_NUM+REFERE...  

IDX1505231652450     DB2             RAIL_ETA                                649793               649793 U          N         N      +ORDER_ID-STATE                                             
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          N         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          N         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231648160     DB2             CODE                                         5                  520 D          N         N      +CODE_TEXT+TYPE+CODE_ID+CODE                                
IDX1505231648190     DB2             EDI_204                                 295885              1014841 D          N         N      +CUST_S_LOCATION_ID+ORDER_ID+REFERENCE                      
IDX1505231647160     DB2             LOCATION                                140445               140445 U          Y         N      +LOCATION_ID-NAME                                           
DRIVER_ID            DB2             DRIVER                                   13519                13519 U          Y         Y      +DRIVER_ID+FIRST_NAME+LAST_NAME+PREVIOUS_NUM+DISPATCHER     
IDX201162151160000   DB2             CITY                                     97516                97516 U          Y         Y      +CITY_ID+STATE+CITY+TIME_ZONE                               
WIZ1166              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+ZIP+LONGITUDE+LATITUDE+NAME+CITY_ID            
WIZ1171              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+NAME+NUMBER                                    
SQL991011111520810   DB2             DRIVER                                   13519                13519 P          Y         Y      +DRIVER_ID                                                  
SQL991011113441210   DB2             VENDOR_UNIT                               2976                 2976 P          Y         Y      +VENDOR_UNIT_ID                                             
IDX_CODE_01          DB2             CODE                                       518                  520 D          Y         Y      +TYPE+CODE_TEXT+CODE                                        
IDX_STOP_1           DB2             STOP                                     15883                21568 D          Y         Y      +ORDER_ID                                                   
IDX201161942390000   EDI             EDI_204_HUB_CUSTOMER                    451173               451173 D          Y         Y      +REFERENCE+DESCRIPTION                                      
IDX201171957580000   DB2             CITY                                     97516                97516 D          Y         Y      +STATE+CITY+CITY_ID                                         
IDX201171959320000   DB2             LOCATION                                140445               140445 D          Y         Y      +CITY_ID+DROP+ZIP+LONGITUDE+LATITUDE+NAME+LOCATION_ID       

PS D:\xtivia\queries> db2 set current explain mode evaluate indexes
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 -tvf query13.sql
select * from db2.order_trace  where  c_state = ? and dl_ap_date2 >= ? and  reference not in (select reference from db2.edi_204 edi_204 where edi_204.reference = reference and edi_204.order_id = order
_id  and CUST_s_location_id = ? ) and (status in (? ,? ) OR ORDER_TYPE = ? )  order by dl_ap_date2, dl_ap_time2, act_state, act_city, actual, order_id
SQL0217W  The statement was not executed as only Explain information requests
are being processed.  SQLSTATE=01604

PS D:\xtivia\queries> db2exfmt -d comtrak2 -1 -o query13_ind2_exfmt.txt
DB2 Universal Database Version 9.7, 5622-044 (c) Copyright IBM Corp. 1991, 2009
Licensed Material - Program Property of IBM
IBM DATABASE 2 Explain Table Format Tool

Connecting to the Database.
Connect to Database Successful.
Base table information incomplete
Output is in query13_ind2_exfmt.txt.
Executing Connect Reset -- Connect Reset was Successful.
PS D:\xtivia\queries> db2 set current explain mode no
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 -tvf this_index_cost_query.sql
select dec(total_cost,20,4) as this_index_total_cost , dec(io_cost,20,4) as this_index_io_cost , dec(CPU_cost,20,4) as this_index_cpu_cost , dec(Comm_cost,20,4) as this_index_comm_cost from Explain_Op
erator ,(select max(explain_time) as maxtime from Explain_Operator where operator_type = 'RETURN') as b where explain_time = b.maxtime and operator_type = 'RETURN' with UR

THIS_INDEX_TOTAL_COST  THIS_INDEX_IO_COST     THIS_INDEX_CPU_COST    THIS_INDEX_COMM_COST
---------------------- ---------------------- ---------------------- ----------------------
            38772.4492             11552.6152         705354496.0000                 0.0000

  1 record(s) selected.

Note that I do tend to run a db2exfmt each time. I’m not technically required to, but I find them useful to have to go back and look at. Again, this index appears to have ZERO impact by itself.

When I’ve gone through each one, this is what my spreadsheet looks like:

Note that by far, the biggest single impact is that wide index that I really don’t like the idea of. Only one other of the 7 recommended new indexes shows any impact by itself. This leads me to wonder why DB2 is even recommending some of the others. I’m hoping that it is because a combination of them makes a real difference. Let’s try Index Subtraction to see if that is true.

Index Subtraction

For Index Subtraction, we tell DB2 to give us the numbers as if all of the proposed indexes exist, and we take away only one at a time to see the impact of not having that particular index. The steps are very similar, just with slightly different updates to the USE_INDEX column.

To start with, we want all of the recommended indexes for this query to be marked with USE_INDEX of Y.

PS D:\xtivia\queries> db2 "update advise_index set use_index='Y' where name in ('IDX1505231647110','IDX1505231647160','IDX1505231652450','IDX1505231647170','IDX1505231648160','IDX1505231648190','IDX15
05231651490')"
DB20000I  The SQL command completed successfully.
                                                                              Index                Table                                                                                         
Index Name           Table Schema    Table Name                         Cardinality          Cardinality UNIQUERULE USE_INDEX EXISTS Column Names                                                
-------------------- --------------- ------------------------- -------------------- -------------------- ---------- --------- ------ ------------------------------------------------------------
IDX1505231647110     DB2             VENDOR_UNIT                               2976                 2976 U          Y         N      +VENDOR_UNIT_ID-VENDOR_ID                                   
IDX1505231647160     DB2             LOCATION                                140445               140445 U          Y         N      +LOCATION_ID-NAME                                           
IDX1505231651490     DB2             ORDER                                      642                52497 U          Y         N      +TRAILER_SIZE+DL_AP_DATE2+CUSTOMER_RATE+NOTIFIED_DATE+DATE_E
                                                                                                                                     NTERED+COMMODITY+NEEDS_PAPERWORK+LINE_OF_BUSINESS+TOTAL_CHAR
                                                                                                                                     GE+MILES+TERMINAL+SCAC+SEAL+PO_NUM+PU_NUM+BL_NUM+REFERE...  

IDX1505231652450     DB2             RAIL_ETA                                649793               649793 U          Y         N      +ORDER_ID-STATE                                             
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          Y         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231647170     EDI             EDI_204_ADDITIONAL_FIELDS             19897638             19897638 D          Y         N      +REFERENCE+FIELD_NAME+INSERT_DT+FIELD_VALUE                 
IDX1505231648160     DB2             CODE                                         5                  520 D          Y         N      +CODE_TEXT+TYPE+CODE_ID+CODE                                
IDX1505231648190     DB2             EDI_204                                 295885              1014841 D          Y         N      +CUST_S_LOCATION_ID+ORDER_ID+REFERENCE                      
DRIVER_ID            DB2             DRIVER                                   13519                13519 U          Y         Y      +DRIVER_ID+FIRST_NAME+LAST_NAME+PREVIOUS_NUM+DISPATCHER     
IDX201162151160000   DB2             CITY                                     97516                97516 U          Y         Y      +CITY_ID+STATE+CITY+TIME_ZONE                               
WIZ1166              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+ZIP+LONGITUDE+LATITUDE+NAME+CITY_ID            
WIZ1171              DB2             LOCATION                                140445               140445 U          Y         Y      +LOCATION_ID+NAME+NUMBER                                    
SQL991011111520810   DB2             DRIVER                                   13519                13519 P          Y         Y      +DRIVER_ID                                                  
SQL991011113441210   DB2             VENDOR_UNIT                               2976                 2976 P          Y         Y      +VENDOR_UNIT_ID                                             
IDX_CODE_01          DB2             CODE                                       518                  520 D          Y         Y      +TYPE+CODE_TEXT+CODE                                        
IDX_STOP_1           DB2             STOP                                     15883                21568 D          Y         Y      +ORDER_ID                                                   
IDX201161942390000   EDI             EDI_204_HUB_CUSTOMER                    451173               451173 D          Y         Y      +REFERENCE+DESCRIPTION                                      
IDX201171957580000   DB2             CITY                                     97516                97516 D          Y         Y      +STATE+CITY+CITY_ID                                         
IDX201171959320000   DB2             LOCATION                                140445               140445 D          Y         Y      +CITY_ID+DROP+ZIP+LONGITUDE+LATITUDE+NAME+LOCATION_ID       

Now one at a time, we will mark the indexes to not be used, and run the explain using evaluate indexes.

PS D:\xtivia\queries> db2 "update advise_index set use_index='N' where name = 'IDX1505231647110'"
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 set current explain mode evaluate indexes
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 -tvf query13.sql
select * from db2.order_trace  where  c_state = ? and dl_ap_date2 >= ? and  reference not in (select reference from db2.edi_204 edi_204 where edi_204.reference = reference and edi_204.order_id = order
_id  and CUST_s_location_id = ? ) and (status in (? ,? ) OR ORDER_TYPE = ? )  order by dl_ap_date2, dl_ap_time2, act_state, act_city, actual, order_id
SQL0217W  The statement was not executed as only Explain information requests
are being processed.  SQLSTATE=01604

PS D:\xtivia\queries> db2exfmt -d comtrak2 -1 -o query13_excl_ind1_exfmt.txt
DB2 Universal Database Version 9.7, 5622-044 (c) Copyright IBM Corp. 1991, 2009
Licensed Material - Program Property of IBM
IBM DATABASE 2 Explain Table Format Tool

Connecting to the Database.
Connect to Database Successful.
Base table information incomplete
Base table information incomplete
Base table information incomplete
Base table information incomplete
Base table information incomplete
Base table information incomplete
Output is in query13_excl_ind1_exfmt.txt.
Executing Connect Reset -- Connect Reset was Successful.
PS D:\xtivia\queries> db2 set current explain mode no
DB20000I  The SQL command completed successfully.
PS D:\xtivia\queries> db2 -tvf this_index_cost_query.sql
select dec(total_cost,20,4) as this_index_total_cost , dec(io_cost,20,4) as this_index_io_cost , dec(CPU_cost,20,4) as this_index_cpu_cost , dec(Comm_cost,20,4) as this_index_comm_cost from Explain_Op
erator ,(select max(explain_time) as maxtime from Explain_Operator where operator_type = 'RETURN') as b where explain_time = b.maxtime and operator_type = 'RETURN' with UR

THIS_INDEX_TOTAL_COST  THIS_INDEX_IO_COST     THIS_INDEX_CPU_COST    THIS_INDEX_COMM_COST
---------------------- ---------------------- ---------------------- ----------------------
            18741.7597              5339.1113         131033120.0000                 0.0000

  1 record(s) selected.

When I cycle through each index that way, ensuring that in each round I eliminate only one, this is what my spreadsheet looks like:

Results

With this information, I can immediately eliminate indexes: IDX1505231647110, IDX1505231647160, IDX1505231652450, IDX1505231648160, and IDX1505231647170. This leaves me with two indexes to consider. Given the fact that EDI_204 is one of the largest and most active tables in my database, I won’t be adding that index for the small additional impact that it gives for this query. I’m left with just one major index:

   CREATE UNIQUE INDEX "DB2ADMIN"."IDX1505231651490"
   ON "DB2     "."ORDER" ("TRAILER_SIZE" ASC, "DL_AP_DATE2"
   ASC, "CUSTOMER_RATE" ASC, "NOTIFIED_DATE" ASC, "DATE_ENTERED"
   ASC, "COMMODITY" ASC, "NEEDS_PAPERWORK" ASC, "LINE_OF_BUSINESS"
   ASC, "TOTAL_CHARGE" ASC, "MILES" ASC, "TERMINAL" ASC,
   "SCAC" ASC, "SEAL" ASC, "PO_NUM" ASC, "PU_NUM" ASC,
   "BL_NUM" ASC, "REFERENCE" ASC, "WHO_ENTERED" ASC,
   "ORDER_TYPE" ASC, "TRACTOR_ID" ASC, "DRIVER_ID" ASC,
   "DL_AT_TIME2" ASC, "DL_AT_DATE2" ASC, "DL_AT_TIME1"
   ASC, "DL_AT_DATE1" ASC, "DL_AP_TIME2" ASC, "DL_AP_TIME1"
   ASC, "DL_AP_DATE1" ASC, "PU_AT_TIME2" ASC, "PU_AT_DATE2"
   ASC, "PU_AT_TIME1" ASC, "PU_AT_DATE1" ASC, "PU_AP_TIME2"
   ASC, "PU_AP_DATE2" ASC, "PU_AP_TIME1" ASC, "PU_AP_DATE1"
   ASC, "S_LOCATION_ID" ASC, "CHASSIS" ASC, "CHECK_DIGIT"
   ASC, "TRAILER" ASC, "STATUS" ASC, "ORDER_ID" ASC)
   INCLUDE ("C_LOCATION_ID", "ACT_LOCATION_ID", "BTO_LOCATION_ID")
   ALLOW REVERSE SCANS COLLECT SAMPLED DETAILED STATISTICS;

Given how wide this index is, I’m really concerned about adding it to a table that is already a bit over-indexed, so I haven’t decided yet if I’m going to add it. But at least I now understand that it’s the only index really worth considering for this query.

I find that for now, I like to use both the index addition and subtraction methods to really see what difference there might be in the results.

A big thanks to Scott Hayes for covering this topic in depth at his IDUG North America presentation this year. This process has become a regular tool in my arsenal.