Purpose

These statements calculates the buffer pool hit ratio for both BLU and non-BLU activity. This post includes three SQLs – for calculating at the database, bufferpool, and tablespace level. The separate BLU calculations are not yet included in sysibadm.bp_hitratio as of the writing of this article.

Source

While I’ve modified this statement, it comes from these two sources, both influenced by David Kalmuk:
http://www.ibm.com/developerworks/data/library/techarticle/dm-1407monitor-bluaccel/index.html
http://www.dbisoftware.com/blog/db2nightshow.php?id=619

Statement for Database Level

WITH POOL_DATA( ROW_DATA_PAGES_FOUND,
                COL_DATA_PAGES_FOUND,
                DATA_PAGES_FOUND,
                ROW_DATA_L_READS,
                COL_DATA_L_READS,
                DATA_L_READS )
AS (SELECT SUM(POOL_DATA_LBP_PAGES_FOUND -
               POOL_ASYNC_DATA_LBP_PAGES_FOUND) AS ROW_DATA_PAGES_FOUND,
           SUM(POOL_COL_LBP_PAGES_FOUND -
               POOL_ASYNC_COL_LBP_PAGES_FOUND) AS COL_DATA_PAGES_FOUND,
           SUM(POOL_DATA_LBP_PAGES_FOUND +
               POOL_COL_LBP_PAGES_FOUND -
               POOL_ASYNC_DATA_LBP_PAGES_FOUND -
               POOL_ASYNC_COL_LBP_PAGES_FOUND) AS DATA_PAGES_FOUND, 
           SUM(POOL_DATA_L_READS +
               POOL_TEMP_DATA_L_READS) AS ROW_DATA_L_READS,
           SUM(POOL_COL_L_READS +
               POOL_TEMP_COL_L_READS) AS COL_DATA_L_READS,
           SUM(POOL_DATA_L_READS +
               POOL_COL_L_READS +
               POOL_TEMP_DATA_L_READS +
               POOL_TEMP_COL_L_READS) AS DATA_L_READS
     FROM TABLE(MON_GET_DATABASE(-2)) AS T)
SELECT CASE WHEN DATA_L_READS > 0
            THEN DEC((FLOAT(DATA_PAGES_FOUND) / 
                      FLOAT(DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS DATA_PAGE_HIT_RATIO,
       CASE WHEN ROW_DATA_L_READS > 0
            THEN DEC((FLOAT(ROW_DATA_PAGES_FOUND) /
                      FLOAT(ROW_DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS ROW_DATA_PAGE_HIT_RATIO,
       CASE WHEN COL_DATA_L_READS > 0
            THEN DEC((FLOAT(COL_DATA_PAGES_FOUND) /
                      FLOAT(COL_DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS COL_DATA_PAGE_HIT_RATIO 
FROM POOL_DATA
with ur;

Sample Output for BPHR at the Database Level

DATA_PAGE_HIT_RATIO ROW_DATA_PAGE_HIT_RATIO COL_DATA_PAGE_HIT_RATIO
------------------- ----------------------- -----------------------
              98.61                   99.68                   94.98

  1 record(s) selected.

Statement for Buffer Pool Level

WITH POOL_DATA( BP_NAME,
        ROW_DATA_PAGES_FOUND,
                COL_DATA_PAGES_FOUND,
                DATA_PAGES_FOUND,
                ROW_DATA_L_READS,
                COL_DATA_L_READS,
                DATA_L_READS )
AS (SELECT substr(BP_NAME,1,18),
       (POOL_DATA_LBP_PAGES_FOUND -
               POOL_ASYNC_DATA_LBP_PAGES_FOUND) AS ROW_DATA_PAGES_FOUND,
           (POOL_COL_LBP_PAGES_FOUND -
               POOL_ASYNC_COL_LBP_PAGES_FOUND) AS COL_DATA_PAGES_FOUND,
           (POOL_DATA_LBP_PAGES_FOUND +
               POOL_COL_LBP_PAGES_FOUND -
               POOL_ASYNC_DATA_LBP_PAGES_FOUND -
               POOL_ASYNC_COL_LBP_PAGES_FOUND) AS DATA_PAGES_FOUND, 
           (POOL_DATA_L_READS +
               POOL_TEMP_DATA_L_READS) AS ROW_DATA_L_READS,
           (POOL_COL_L_READS +
               POOL_TEMP_COL_L_READS) AS COL_DATA_L_READS,
           (POOL_DATA_L_READS +
               POOL_COL_L_READS +
               POOL_TEMP_DATA_L_READS +
               POOL_TEMP_COL_L_READS) AS DATA_L_READS
     FROM TABLE(MON_GET_BUFFERPOOL('',-2)) AS T)
SELECT  BP_NAME,
    CASE WHEN DATA_L_READS > 0
            THEN DEC((FLOAT(DATA_PAGES_FOUND) / 
                      FLOAT(DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS DATA_PAGE_HIT_RATIO,
       CASE WHEN ROW_DATA_L_READS > 0
            THEN DEC((FLOAT(ROW_DATA_PAGES_FOUND) /
                      FLOAT(ROW_DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS ROW_DATA_PAGE_HIT_RATIO,
       CASE WHEN COL_DATA_L_READS > 0
            THEN DEC((FLOAT(COL_DATA_PAGES_FOUND) /
                      FLOAT(COL_DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS COL_DATA_PAGE_HIT_RATIO 
FROM POOL_DATA
with ur;

Sample Output for BPHR at the Database Level

BP_NAME            DATA_PAGE_HIT_RATIO ROW_DATA_PAGE_HIT_RATIO COL_DATA_PAGE_HIT_RATIO
------------------ ------------------- ----------------------- -----------------------
IBMDEFAULTBP                     99.97                   99.97                       -
BUFF32K                          94.95                   98.74                   94.70
BUFF16K                          99.98                   99.98                   99.98
BUFF8K                           99.24                   99.24                       -
BUFF4K                           99.13                   99.13                       -
IBMSYSTEMBP4K                        -                       -                       -
IBMSYSTEMBP8K                        -                       -                       -
IBMSYSTEMBP16K                       -                       -                       -
IBMSYSTEMBP32K                       -                       -                       -

  9 record(s) selected.

Statement for Tablespace Level

WITH POOL_DATA( TBSP_NAME,
        ROW_DATA_PAGES_FOUND,
                COL_DATA_PAGES_FOUND,
                DATA_PAGES_FOUND,
                ROW_DATA_L_READS,
                COL_DATA_L_READS,
                DATA_L_READS )
AS (SELECT substr(TBSP_NAME,1,30),
       (POOL_DATA_LBP_PAGES_FOUND -
               POOL_ASYNC_DATA_LBP_PAGES_FOUND) AS ROW_DATA_PAGES_FOUND,
           (POOL_COL_LBP_PAGES_FOUND -
               POOL_ASYNC_COL_LBP_PAGES_FOUND) AS COL_DATA_PAGES_FOUND,
           (POOL_DATA_LBP_PAGES_FOUND +
               POOL_COL_LBP_PAGES_FOUND -
               POOL_ASYNC_DATA_LBP_PAGES_FOUND -
               POOL_ASYNC_COL_LBP_PAGES_FOUND) AS DATA_PAGES_FOUND, 
           (POOL_DATA_L_READS +
               POOL_TEMP_DATA_L_READS) AS ROW_DATA_L_READS,
           (POOL_COL_L_READS +
               POOL_TEMP_COL_L_READS) AS COL_DATA_L_READS,
           (POOL_DATA_L_READS +
               POOL_COL_L_READS +
               POOL_TEMP_DATA_L_READS +
               POOL_TEMP_COL_L_READS) AS DATA_L_READS
     FROM TABLE(MON_GET_TABLESPACE('',-2)) AS T)
SELECT  TBSP_NAME,
    ROW_DATA_L_READS,
    COL_DATA_L_READS,
    CASE WHEN DATA_L_READS > 0
            THEN DEC((FLOAT(DATA_PAGES_FOUND) / 
                      FLOAT(DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS DATA_PAGE_HIT_RATIO,
       CASE WHEN ROW_DATA_L_READS > 0
            THEN DEC((FLOAT(ROW_DATA_PAGES_FOUND) /
                      FLOAT(ROW_DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS ROW_DATA_PAGE_HIT_RATIO,
       CASE WHEN COL_DATA_L_READS > 0
            THEN DEC((FLOAT(COL_DATA_PAGES_FOUND) /
                      FLOAT(COL_DATA_L_READS)) * 100, 5, 2)
            ELSE NULL
       END AS COL_DATA_PAGE_HIT_RATIO 
FROM POOL_DATA
with ur;

Sample Output for BPHR at the Tablespace Level

TBSP_NAME                      ROW_DATA_L_READS     COL_DATA_L_READS     DATA_PAGE_HIT_RATIO ROW_DATA_PAGE_HIT_RATIO COL_DATA_PAGE_HIT_RATIO
------------------------------ -------------------- -------------------- ------------------- ----------------------- -----------------------
SYSCATSPACE                               112771783                    0               99.92                   99.92                       -
TEMPSPACE1                                273494060                    0               99.99                   99.99                       -
USERSPACE1                                      265                    0               83.01                   83.01                       -
LOBSPACE1                                     13381                    0               80.24                   80.24                       -
CMIPS_TS16K                                83283779             11317952               99.98                   99.98                   99.98
AG_IE_RSLT_TS                                512863              4686219               82.48                   99.84                   80.58
AG_IE_RSN_TS                                    411                  280               79.59                   84.42                   72.50
TAB16K                                          265                    0               99.24                   99.24                       -
SSIRS_LG_TBL_TS                               38312              8598834               72.95                   71.75                   72.96
CIS_ODS_LG_TBL_TS                            226824              1900955               82.21                   99.63                   80.13
WF_ODS_LG_TBL_TS                              13168             12020654               67.19                   97.71                   67.16
CIS_ODS_TS32K                               6848238             11398098               96.43                   99.71                   94.45
SSIRS_TS32K                                 4104862             76690676               99.44                   98.72                   99.48
EBT_ODS_TS32K                                  9933              5329227               74.42                   90.58                   74.39
WEBFILES_ODS_TS32K                             1212               177281               92.36                   85.23                   92.41
SMART_ODS_TS32K                                3311                 9262               76.31                   94.23                   69.90
CWS_ODS32K                                  1471139             75392240               99.75                   97.88                   99.78
GIS_UTIL_TS32K                                89161              1769118               85.44                   56.74                   86.88
TAB4K                                     224206209                    0               99.13                   99.13                       -
TAB8K                                           265                    0               99.24                   99.24                       -
TAB32K                                        70847              1170441               84.40                   87.02                   84.24
SSIRS_DMART_STG_TS                              265                    0               77.73                   77.73                       -
SYSTOOLSPACE                                3061926                    0               99.82                   99.82                       -
SYSTOOLSTMPSPACE                                  0                    0                   -                       -                       -

  24 record(s) selected.

I have been paying a bit of attention to cross-platform SQL optimization lately, and read this interesting post:
https://www.periscopedata.com/blog/use-subqueries-to-count-distinct-50x-faster.html?utm_content=bufferaf11f&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer

Being a bit of an experimenter, the first thing that I wondered is how DB2 would handle this scenario. Would the DB2 optimizer be smarter than others, or would the same hold true for DB2 that held true for PostgreSQL?

Environment

The environment I’m testing on is my little Linux VM (Ubuntu) running DB2 Express-C 10.5, fixpack 5 on my laptop. DB2 Express-C is free to use on a small machine. This is my default experimentation environment for anything that doesn’t require features I cannot get in Express-C. Express-C is actually shockingly complete – I’m working on some experimentation with Native Encryption, and I can also do that in this environment. I don’t have the same data set as the people who wrote the article. I am using the db2 sample database, created using the db2sampl command. To generate a reasonable volume of data, I took the EMPPROJACT table and inserted it into itself until I had over 10 million rows.

$ db2 "select count(*) from empprojact"

1          
-----------
   10099008

  1 record(s) selected.

I also completed runstats on the table:

$ db2 runstats on table empprojact with distribution and detailed indexes all
DB20000I  The RUNSTATS command completed successfully.

I created the explain tables, and the explain view as described in my post about a more platform-independent way of looking at explain.

The Experiment

I plan to perform three tests. First a test as-is with the default sample database schema. Next, a test with an added index (which is low-cardinality). Both of those will be with SQL similar to the original SQL in the article that is my inspiration. For the final test, I’ll re-write the SQL in a similar way as the article.

Base SQL

Here is the starting SQL. I’ve written it to be as similar to the PostgreSQL SQL as possible:

SELECT actkwd, count(distinct(empprojact.empno)) as count
FROM act join empprojact on act.actno=empprojact.actno
group by actkwd
order by count desc;

Test #1: Baseline

How long does this SQL run as a baseline?

$ db2batch -d sample -f distinct_query.sql
* Timestamp: Thu Nov 12 2015 13:53:49 MST
---------------------------------------------

* SQL Statement Number 1:

SELECT actkwd, count(distinct(empprojact.empno)) as count
FROM act join empprojact on act.actno=empprojact.actno
group by actkwd
order by count desc;

ACTKWD COUNT      
------ -----------
CODE             8
TEST             8
LOGIC            7
MANAGE           6
DOC              5
OPERAT           4
MAINT            3
ADMQS            2
TEACH            2
ADMDB            1
ADMDC            1
ADMSYS           1
COURSE           1
DEFINE           1
ECOST            1
LEADPR           1
SPECS            1

* 17 row(s) fetched, 17 row(s) output.

* Elapsed Time is:       5.544549 seconds

* Summary Table:

Type      Number      Repetitions Total Time (s) Min Time (s)   Max Time (s)   Arithmetic Mean Geometric Mean Row(s) Fetched Row(s) Output
--------- ----------- ----------- -------------- -------------- -------------- --------------- -------------- -------------- -------------
Statement           1           1       5.544549       5.544549       5.544549        5.544549       5.544549             17            17

* Total Entries:              1
* Total Time:                 5.544549 seconds
* Minimum Time:               5.544549 seconds
* Maximum Time:               5.544549 seconds
* Arithmetic Mean Time:       5.544549 seconds
* Geometric Mean Time:        5.544549 seconds
---------------------------------------------
* Timestamp: Thu Nov 12 2015 13:53:56 MST

The timing is pretty similar when my bufferpools are hot and when they’re cold, but I’m checking both each test to be sure.

The explain looks like this:

Explain Plan                                                                                        
----------------------------------------------------------------------------------------------------
ID | Operation                |                           Rows |  Cost                              
 1 | RETURN                   |                                | 49569                              
 2 |  TBSCAN                  |             18 of 18 (100.00%) | 49569                              
 3 |   SORT                   |             18 of 18 (100.00%) | 49569                              
 4 |    GRPBY (COMPLETE)      |            18 of 540 (  3.33%) | 49569                              
 5 |     TBSCAN               |           540 of 540 (100.00%) | 49569                              
 6 |      SORT (UNIQUE)       |      540 of 10099008 (   .01%) | 49569                              
 7 |       HSJOIN             |                 10099008 of 18 | 46443                              
 8 |        TBSCAN EMPPROJACT | 10099008 of 10099008 (100.00%) | 46172                              
 9 |        IXSCAN XACT2      |             18 of 18 (100.00%) |     0                              
                                                    

As expected, the empprojact table is getting a full table scan, and that is taking up most of the time.

Test #2: Base SQL with (Low Cardinality) Index

Next, I added an index to eliminate that table scan:

$ db2 "create index ecrooks.I_empprojact_01 on empprojact (actno,empno) collect detailed statistics"
DB20000I  The SQL command completed successfully.

Note that this index only has a cardinality of 53 on a table of over 10 million rows – if this were the real world, I’d be looking to see if MDC could better help me with the performance of this query.

Here’s what the timing looks like with the index:

$ db2batch -d sample -f distinct_query.sql
* Timestamp: Thu Nov 12 2015 14:33:24 MST
---------------------------------------------

* SQL Statement Number 1:

SELECT actkwd, count(distinct(empprojact.empno)) as count
FROM act join empprojact on act.actno=empprojact.actno
group by actkwd
order by count desc;

ACTKWD COUNT      
------ -----------
CODE             8
TEST             8
LOGIC            7
MANAGE           6
DOC              5
OPERAT           4
MAINT            3
ADMQS            2
TEACH            2
ADMDB            1
ADMDC            1
ADMSYS           1
COURSE           1
DEFINE           1
ECOST            1
LEADPR           1
SPECS            1

* 17 row(s) fetched, 17 row(s) output.

* Elapsed Time is:       2.105718 seconds

* Summary Table:

Type      Number      Repetitions Total Time (s) Min Time (s)   Max Time (s)   Arithmetic Mean Geometric Mean Row(s) Fetched Row(s) Output
--------- ----------- ----------- -------------- -------------- -------------- --------------- -------------- -------------- -------------
Statement           1           1       2.105718       2.105718       2.105718        2.105718       2.105718             17            17

* Total Entries:              1
* Total Time:                 2.105718 seconds
* Minimum Time:               2.105718 seconds
* Maximum Time:               2.105718 seconds
* Arithmetic Mean Time:       2.105718 seconds
* Geometric Mean Time:        2.105718 seconds
---------------------------------------------
* Timestamp: Thu Nov 12 2015 14:33:26 MST

We cut the time in half. Here’s what the explain looks like:

Explain Plan                                                                                        
----------------------------------------------------------------------------------------------------
ID | Operation                     |                         Rows |  Cost                           
 1 | RETURN                        |                              | 15657                           
 2 |  TBSCAN                       |           18 of 18 (100.00%) | 15657                           
 3 |   SORT                        |           18 of 18 (100.00%) | 15657                           
 4 |    GRPBY (COMPLETE)           |          18 of 540 (  3.33%) | 15657                           
 5 |     TBSCAN                    |         540 of 540 (100.00%) | 15657                           
 6 |      SORT (UNIQUE)            |    540 of 10099008 (   .01%) | 15657                           
 7 |       NLJOIN                  |               10099008 of 18 | 12531                           
 8 |        IXSCAN XACT2           |           18 of 18 (100.00%) |     0                           
 9 |        IXSCAN I_EMPPROJACT_01 | 561056 of 10099008 (  5.56%) |   750                           
                                                                                                    
Predicate Information                                                                               
 7 - JOIN (Q2.ACTNO = Q1.ACTNO)                                                                     
 9 - START (Q2.ACTNO = Q1.ACTNO)                                                                    
      STOP (Q2.ACTNO = Q1.ACTNO)               

That is somewhat better. Note that the unique is applied in step 6, after the indexes for both tables have been scanned.

Test #3: Rewrite

Finally, I rewrote the SQL to be similar to the article:

select
  actkwd,
  log_counts.ct
from act
join (
  select distinct_logs.actno,
  count(1) as ct
  from (
    select distinct actno, empno
    from empprojact
  ) as distinct_logs
  group by distinct_logs.actno
) as log_counts 
on log_counts.actno = act.actno
order by log_counts.ct desc;

And here’s what the timing looks like:

 
$ db2batch -d sample -f distinct_query_rewrite.sql
* Timestamp: Thu Nov 12 2015 14:52:01 MST
---------------------------------------------

* SQL Statement Number 1:

select
  actkwd,
  log_counts.ct
from act
join (
  select distinct_logs.actno,
  count(1) as ct
  from (
    select distinct actno, empno
    from empprojact
  ) as distinct_logs
  group by distinct_logs.actno
) as log_counts
on log_counts.actno = act.actno
order by log_counts.ct desc;

ACTKWD CT         
------ -----------
CODE             8
TEST             8
LOGIC            7
MANAGE           6
DOC              5
OPERAT           4
MAINT            3
ADMQS            2
TEACH            2
ECOST            1
DEFINE           1
LEADPR           1
SPECS            1
COURSE           1
ADMSYS           1
ADMDB            1
ADMDC            1

* 17 row(s) fetched, 17 row(s) output.

* Elapsed Time is:       1.136768 seconds

* Summary Table:

Type      Number      Repetitions Total Time (s) Min Time (s)   Max Time (s)   Arithmetic Mean Geometric Mean Row(s) Fetched Row(s) Output
--------- ----------- ----------- -------------- -------------- -------------- --------------- -------------- -------------- -------------
Statement           1           1       1.136768       1.136768       1.136768        1.136768       1.136768             17            17

* Total Entries:              1
* Total Time:                 1.136768 seconds
* Minimum Time:               1.136768 seconds
* Maximum Time:               1.136768 seconds
* Arithmetic Mean Time:       1.136768 seconds
* Geometric Mean Time:        1.136768 seconds
---------------------------------------------
* Timestamp: Thu Nov 12 2015 14:52:03 MST

Wow, that’s nearly half the time as the last test!

Here’s what the explain plan looks like for it:

Explain Plan                                                                                        
----------------------------------------------------------------------------------------------------
ID | Operation                    |                           Rows |  Cost                          
 1 | RETURN                       |                                | 12706                          
 2 |  TBSCAN                      |             17 of 17 (100.00%) | 12706                          
 3 |   SORT                       |             17 of 17 (100.00%) | 12706                          
 4 |    HSJOIN                    |             17 of 17 (100.00%) | 12706                          
 5 |     IXSCAN XACT2             |             18 of 18 (100.00%) |     0                          
 6 |     GRPBY (COMPLETE)         |             17 of 53 ( 32.08%) | 12706                          
 7 |      UNIQUE                  |       53 of 10099008 (   .00%) | 12706                          
 8 |       IXSCAN I_EMPPROJACT_01 | 10099008 of 10099008 (100.00%) | 12527                          
                                                                                                    
Predicate Information                                                                               
 4 - JOIN (Q3.ACTNO = Q4.ACTNO)  

Notice here that the “UNIQUE” is applied as the second step (#7), and the total cost is slightly better, with the actual performance being half the time of the last run.

Conclusions

I came into this experiment hoping the DB2 optimizer would manage to do things the most efficient way, even with the starting syntax. This was not the case. Even with DB2’s very strong optimizer, how you write SQL still has a significant impact on performance.

It is a busy time of year for DBAs. Many of us will put in a large number of hours between Friday and Monday keeping ecommerce databases running and our company’s profits rolling. But it’s also a time of year in the USA where we take a moment to be thankful for the things around us and make sure that the people we are thankful to and for are aware of it. I am thankful for a lot of things in the world, but here are some of the things from the DB2 community that I am thankful for.

• First on my list is my DB2 buddy Mike Krafick (@MKrafick). I first encountered Mike a few months after I started writing blog entries at least once a week. He was running the IDUG twitter feed at the time and asked me if it was OK if he promoted some of my posts (my answer was, of course, heck yeah!). I was seeing maybe 100 page views a day at that point (less than a tenth of where I am now). A mutual friend made sure we met at IDUG in Denver that year, and Mike has become the first person I ask DB2 questions to and the first person I run anything questionable by. He checks my ego when I need it and is excellent when I need to vent. We’ve continued to push each other to accomplish more and contribute more to the DB2 community. For all of you who have a friend like this in the DB2 community, I hope you tell them you appreciate them this week.
• I’m thankful for Ian Bjorhovde (@idbjorh) – whose name I can say and usually spell. He is probably the smartest person I know. He is the first person I go to on the really tough questions. He is the person I go to when I’m trying to reason something out or get to a contact I need for the difficult DB2 stuff. He is also one of the few people in the DB2 community with political views nearly as liberal as mine. He and Fred Sobotka (whose last name I cannot say and cannot spell without looking it up) do the “bi-monthly” podcast The Whole Package Cache – which I still love, even if they haven’t managed to put out that many this year.
• I’m thankful for my mentor – Melanie Stopfer (@mstopfer1). I don’t know why I’m still amazed at how I come to her with a question on presenting or explaining a topic or a pure technical question and she can so simply and easily set me on the right path. I’m immensely grateful to her for all I have learned from her and for a couple of fantastic opportunities she threw my way this year. When I first met Melanie many years ago by taking one of her classes at IBM in Boulder I knew I wanted to be like her in my work. I wanted to know the answers or know who to ask, and to share my knowledge with all who would listen. I would surely not be where I am today without her example and help.
• I’m thankful for other DB2 buddies I hang out with at conferences – Ken Shafer (@aerodata), Roland Schock (@ARSDB2), Pavan Kristipati (@pkristipati), Iqbal Goralwalla (@iqbalgoralwalla), Fitz, and others. These people are like a second family.
• I’m thankful for IBM. Love them or hate them (or sometimes both at once), they’re the basis for my career, and better than the other guys.
• I’m thankful for Scott Hayes and the work he does with the DB2 Night Show (@db2nightshow) in the DB2 community. Scott gives me the opportunity to present on the DB2 Night Show every year. I learned a lot from his presentation at IDUG NA this year on comparing the impact of the indexes recommended by the DB2 design advisor. Also, I cannot wait to get my hands on the new version of Brother Panther to play with the comparative index analysis. When he asks for contestants for DB2’s Got Talent, go sign up – you have no idea how far it can take you.
• I’m thankful for the opportunity to mentor. There are several DBAs that I make a point to spend time with and work with them on writing or presenting, and I learn so much and get so much from that relationship. Abhik, Saurabh, Luke Numrych, and others.
• I’m thankful for anyone who works hard to make user group meetings or IDUG conferences happen and run smoothly. Your work is appreciated, and you are the true builders of the DB2 community.
• I’m thankful for the IDUG Content Committee – great stuff this year: http://www.idug.org/p/bl/et/blogid=278
• I’m thankful for all the individual IBMers who have helped me on specific issues this year – Paul Bird, Pavel Sustr, Dale McInnis, Markus Mueller, Jessica Escott/Rockwood, Mike Cornish, Roger Sanders, Scott Maberry, Naresh Chainani and so many others.
• I’m thankful for Klaas Brant – Who gave me the opportunity to speak for an entire day at the DB2 Symposium, thereby getting me off the North American continent for the first time in nearly a decade. I really enjoyed the DB2 Symposium, and wish the format would take off more in the USA.
• I’m thankful for one of my new favorite blogs, Use the Index, Luke and author Markus Winand (@MarkusWinand). Reading cross-platform views on tuning developers for optimal SQL performance has expanded my horizons this year. This content has also led to two of my favorite blog entries to write this year.
• I am thankful for developerWorks articles on deep technical topics.
• I am thankful to every single person I talked to at a conference who told me they read my blog. I am simply awful with names and faces, but you are the ones who make me feel like a rockstar and make me realize that people are really helped with what I write. I’m sure I handle it horribly when you come up and introduce yourself, but I’m just not used to it yet. I picture each of these encounters in my head throughout the year and just smile because they really do mean so much to me and push me to keep writing.
• I’m so thankful to my husband for so many reasons, but in relation to the DB2 community, it’s his support that makes it so I have the time to write. He gives me at least one night a week to just go and focus on blogging. Also, he’s a talented VMWare guy and has helped me navigate issues both technical and work-related. I love that we can talk tech with each other and we both really get it.

There have to be so many more people and resources I’m missing here. I very much enjoy this DB2 community that we continue to build every day.

Performance tuning of BLU databases is an evolving topic at this time. For BLU databases, sort memory cannot be tuned by STMM, and so must be manually tuned.

BLU also makes extensive use of sort memory. It is a bit of a misnomer at this point. It is more of a working memory area that is used for hashes, grouping, aggregations and more. Each query may easily allocate more than one sortheap. Each operator in the explain plan can have its own sort heap if needed.

Sort Memory Parameters

With BLU at this time, sort memory areas cannot be automatically tuned. However, some parameters, like SHEAPTHRES_SHR, specify a maximum. It is a good idea to see how close you are getting to that maximum.

To see your setting for sort heap parameters, you can use this syntax:

$ db2 get db cfg for SAMPLE |grep -i sort
 Sort heap thres for shared sorts (4KB) (SHEAPTHRES_SHR) = 3400000
 Sort list heap (4KB)                         (SORTHEAP) = 200000

Yes, that’s over 3 million pages.

Shared Sort Memory

To see the top amount of memory used for all SORTHEAPs at one time, I can use:

$ db2 "select sort_shrheap_top from table(mon_get_database(-2))"

SORT_SHRHEAP_TOP    
--------------------
             2711847

  1 record(s) selected.

This tells me that given my current settings, I have quite a bit of overhead in SHEAPTHRES_SHR, especially since this database has been continually up for over 70 days.

The general guideline for SHEAPTHRES_SHR in BLU databases is that we want to aim for 1/2 to 1/3 of the memory available to DB2. The BLU implementation I am working with here is a small one – with just 8 CPUs and 64 GB of memory. I also have two instances on this server, meaning that after memory for the OS (figure 20% for small systems), and memory for the other smaller instance, I’m allocating about 42.5 GB or 11,161,836 4K pages for this instance/database. With those numbers, my SHEAPTHRES_SHR is right about 30% of the available memory. Which is particularly interesting, because I did not intentionally arrive at that amount, but have rather increased it several times as errors occurred during pre-go-live testing on this database.

Individual SORTHEAP

While my overall maximum for sort memory is probably about right, I also need to see if sorts are overflowing and needing more than the 200,000 pages in a single sortheap. I can start by looking at the high water mark for sortheap. I can do that with this query:

$ db2 "select SORT_CONSUMER_SHRHEAP_TOP from table(mon_get_database(-2))"

SORT_CONSUMER_SHRHEAP_TOP
-------------------------
                   200000

  1 record(s) selected.

In this case, we can see that the high water mark matches my SORTHEAP value, indicating that in at least one case, something is using the entire amount of memory available to it. With that in mind, I need to understand how often my SORTHEAP is being overflowed. This SQL works for that:

WITH SORT_CONSUMERS(TOTAL_SORT_CONSUMERS,
        TOTAL_SORT_CONSUMER_OVERFLOWS) 
                AS (SELECT (TOTAL_SORTS + TOTAL_HASH_JOINS +
        TOTAL_HASH_GRPBYS + TOTAL_OLAP_FUNCS + TOTAL_COL_VECTOR_CONSUMERS) 
                AS TOTAL_SORT_CONSUMERS, 
        (SORT_OVERFLOWS + HASH_JOIN_OVERFLOWS +
                HASH_GRPBY_OVERFLOWS + OLAP_FUNC_OVERFLOWS) 
                AS TOTAL_SORT_CONSUMER_OVERFLOWS 
        FROM TABLE(MON_GET_DATABASE(-2)) AS T)
        SELECT TOTAL_SORT_CONSUMER_OVERFLOWS,
                TOTAL_SORT_CONSUMERS, 
                CASE WHEN TOTAL_SORT_CONSUMERS > 0 THEN
                DEC((FLOAT(TOTAL_SORT_CONSUMER_OVERFLOWS)/
                        FLOAT(TOTAL_SORT_CONSUMERS)) * 100, 5, 2)
                ELSE 
                        NULL 
                END AS PCT_SORT_CONSUMER_OVERFLOWS 
        FROM SORT_CONSUMERS
WITH UR;

TOTAL_SORT_CONSUMER_OVERFLOWS TOTAL_SORT_CONSUMERS PCT_SORT_CONSUMER_OVERFLOWS
----------------------------- -------------------- ---------------------------
                         3945              3897812                        0.10

  1 record(s) selected.

Only 0.10% of my sorts are overflowing – generally not bad. However, since I know that this system has extremely low concurrency, and it’s not coming close to using my shared sort memory, and I’m not yet experiencing memory pressure, I’m going to increase my SORTHEAP.

Summary

These same analyses can be done for non-BLU databases, but if STMM is managing sort for you in those situations, you may not have to tune these areas. In BLU databases it is critical to do these kinds of analyses to properly tune sort memory.

Note: I increased my SORTHEAP to 300000 pages, and still saw the same kinds of numbers a week later – overflows about 0.10% of the time, and the same exact HWM for shared sort. I’ll be increasing SORTHEAP again and seeing how things go.

A big thanks to Scott Maberry – much of the SQL in this blog entry is derived from SQL he shared with me.

I have worked with three different vendors for backup solutions for various clients in the last few months, and several others over the course of my career. When either making archive log decisions or evaluating vendors, it is important to know the DB2 details behind what is going on

Why Archive Transaction Logs at All?

First and foremost, archive logging must be enabled to take online backups. Even in some of the smaller data warehouses I support, it is unusual to make use of circular logging, even in non-production environments, much less production. I’ve seen a number of different recovery schemes and also a number of real-world failures, both easily handled and catastrophic. The firm I work for now is often called upon to help clients in what I like to think of as Extreme Recovery Scenarios – when people who are not trained DB2 DBAs have been supporting a database, and through questionable decisions have gotten themselves into a very painful recovery situation.

Archiving transaction logs is by far the norm over circular logging. The ONLY advantage circular logging has over archive logging is the ease in administering the transaction log files themselves, which is not that big of an advantage. Archiving transaction logs also enables the use of very handy tools like HADR.

The biggest advantage of archiving transaction logs is the ability to roll a database forward to any point in time for which you have appropriate backup images and transaction logs. For many of my clients this means that I can easily restore their database to any given second in the last two weeks. This type of recovery is less commonly needed when HADR is used, but I have still needed it in some of those extreme recovery scenarios.

If transaction logs are archived, they should be archived only to a separate directory, filesystem, or vendor device. They should not be allowed to accumulate in the active log directory. My policy on this comes from many times when a client has come to me who has accidentally deleted an active log file for a database, often because they don’t differentiate or know the difference between active and archived transaction logs. That is a painful recovery scenario most of the time

Why Externalize Archived Transaction Logs

To achieve your recovery goals, you must have the correct backup files and transaction logs. To protect against failures on the database server, backup files AND transaction logs should both be moved to an external solution as soon as possible after they are generated. In the cheapest solution, this can be simply copying the files to another server after they are generated. This ensures that if you lose the database server entirely and permanently, you still have your most precious resource – your data.

Where Externalized Archived Transaction Logs Can Go

One solution for externalizing archived transaction logs can be to choose a local disk as the location for archived transaction logs and then use OS or other processes to copy those files elsewhere on a regular basis. Without a vended enterprise backup solution, I find that clients are rarely willing to copy or back up the contents of this directory more often than daily, which can be inadequate for some failure scenarios and recovery goals.

TSM

I’ve worked with archiving logs to TSM since it was called ADSM and the only option was a compiled userexit. Today, the options on the DB2 side for this are easy, but as with all vendor solutions, you really need a talented engineer/admin for TSM or the vendor solution for it to be the least bit practical. When using TSM, you set LOGARCHMETH1 to TSM:mgmtclass_name, and have your TSM admin do any other required setup for the DB2 instance owning ID. Both backup images and transaction logs can easily be extracted using the db2adutl command. If you need to access the files for your primary database server on some other server, you have two choices. You can either work with your TSM admin to be able to directly access the files on TSM from the other server, or you can extract the files to disk on the primary server and then scp or use other means to move them to the other server.

After enabling TSM, always do basic restore and retrieval tests to make sure you can get things back from TSM when you need to.

Other Vendors

Using other vendors is similar in some respects. You can set LOGARCHMETH1 to VENDOR:/path/to/vendorLib/<library name>. However, you want to make sure this path is not too long, as that can cause problems. You also may need to specify options using the VENDOROPT DB CFG parameter. The values here should be supplied by the vendor in their documentation for DB2.

When you use another vendor, in addition to the basic restore/retrieve tests, you also will need to document the syntax and ability to do each of the following:

• List transaction logs
• List database backup images
• Retrieve backup image to disk
• Retreive transaction log files to disk

Surprisingly, not all vendors can handle the last two – at all. They may only support when DB2 retrieves the files via the restore or rollforward commands. To me, these are critical to have, as restoring is not always the only thing I want to do with these files – sometimes I need to move them to other locations or supply them to IBM support in extreme cases. If I had a say in choosing the backup solution, not allowing this would be grounds for elimination.

USEREXIT as a DB CFG Parameter was Deprecated; Use of a Userexit was Not

It is still possible to use userexit today. This is WRONG and it should not be used.

Userexit is a compiled executable written in C++ that defines what an archive of a transaction log does. When I started working with DB2 many years ago, it was the only way to get archived transaction logs to any vendor device, including TSM. It was a real pain, because you had to find the right C compiler to compile it after defining the paths or details, and you also had to save somewhere a copy of the un-compiled code so you could go back and figure out what it was doing. If you didn’t have that plain text file, then you had no way of knowing where archived transaction logs were really going. If you wanted to change where they were going, you had to recompile it.

This year, I ran into a client actually using a vendor-supplied userexit, and it had all those same problems. It took a couple of weeks for us to figure the whole thing out. The whole reason all of the LOGARCHMETH1 and related parameters were introduced was to avoid these issues. Don’t use userexit.

FAILARCHPATH

Whenever using TSM or other vendor solution, always set FAILARCHPATH. This gives you a safety margin if you connection to the vendor device fails. I like to set it to a separate monitored filesystem, and then if I get a space alert on that filesystem, I know there is a problem with my vendor archive logging. This is easier to catch than parsing the diag log (though I do that too).

Loading data into a DB2 database using the LOAD utility has a lot of ins and outs. If a DBA has spent a lot of time working in a database without referential integrity or check constraints, then they may forget to check for tables in a check-pending state after loading data. To illustrate the points here, I’ll be using the SAMPLE database and examples you can work through to learn the concepts in detail.

The Problem

This problem is fairly immediately obvious, as a table in a check-pending state cannot be accessed, and will return this error message:

SQL0668N  Operation not allowed for reason code "1" on table 
"DB2INST1.PROJACT".  SQLSTATE=57016

Where ‘”DB2INST1.PROJACT”‘ is replaced with the table name in question. Looking up that error code to get the details of that specific reason code, we find:

$ db2 ? SQL0668N


SQL0668N  Operation not allowed for reason code "" on table
      "".

Explanation: 

Access to table "" is restricted. The cause is based on the
following reason codes "":

1        

         Reason code 1 can be returned in multiple scenarios, including
         the following examples:

          
         *  The table is in the Set Integrity Pending No Access state,
            which means the integrity of the table is not enforced and
            the content of the table might be invalid.
         *  An operation on a parent table or an underlying table that
            is not in the Set Integrity Pending No Access state may also
            receive this error if a dependent table is in the Set
            Integrity Pending No Access state.
         *  An attempt is made to issue the SET INTEGRITY statement
            against a user-maintained materialized query table without
            specifying the IMMEDIATE UNCHECKED clause.



User response: 

Respond to this error according to the reason code:

1        

         Bring the table named "" out of the Set Integrity
         Pending No Access state according to the type of table:

          
         *  For a user-maintained materialized query table, execute the
            statement with the IMMEDIATE UNCHECKED option.
         *  For materialized query tables that are not user-maintained,
            execute the SET INTEGRITY statement with the IMMEDIATE
            CHECKED option.

The suggested solution is a bit confusing here if you’re not dealing with a materialized query table. The suggested user response is not listed for plain, everyday tables.

How Tables Get Into a Check-Pending State

The most common way for a table to get into a check-pending state is for a LOAD … REPLACE to be performed on a table that has foreign key constraints defined.

There are also some cases where you manually put a table into a check pending state when adding generated columns or performing other actions – though in those cases, you’re likely to be aware of the check-pending state and the need to clear it.

Here are the official cases when a table may be placed into check pending by a LOAD operation from the IBM DB2 Knowledge Center:

• The table has table check constraints or referential integrity constraints defined on it.
• The table has generated columns and a V7 or earlier client was used to initiate the load operation.
• The table has descendent immediate materialized query tables or descendent immediate staging tables referencing it.
• The table is a staging table or a materialized query table.

To understand when tables are placed into a check-pending state, I’m using the DB2 LUW SAMPLE database.

The SALES table plays no role in any foreign key relationships. So if we load data into it, we do not get a check-pending state:

$ db2 "export to sales.ixf of ixf select * from sales"
SQL3104N  The Export utility is beginning to export data to file "sales.ixf".

SQL3105N  The Export utility has finished exporting "41" rows.


Number of rows exported: 41

$ db2 "load from sales.ixf of ixf replace into sales nonrecoverable"
SQL3501W  The table space(s) in which the table resides will not be placed in 
backup pending state since forward recovery is disabled for the database.

SQL3109N  The utility is beginning to load data from file 
"/db2home/db2inst1/sales.ixf".

SQL3500W  The utility is beginning the "LOAD" phase at time "12/06/2015 
15:08:09.283448".

SQL3150N  The H record in the PC/IXF file has product "DB2    02.00", date 
"20151206", and time "150729".

SQL3153N  The T record in the PC/IXF file has name "sales.ixf", qualifier "", 
and source "            ".

SQL3519W  Begin Load Consistency Point. Input record count = "0".

SQL3520W  Load Consistency Point was successful.

SQL3110N  The utility has completed processing.  "41" rows were read from the 
input file.

SQL3519W  Begin Load Consistency Point. Input record count = "41".

SQL3520W  Load Consistency Point was successful.

SQL3515W  The utility has finished the "LOAD" phase at time "12/06/2015 
15:08:09.596726".


Number of rows read         = 41
Number of rows skipped      = 0
Number of rows loaded       = 41
Number of rows rejected     = 0
Number of rows deleted      = 0
Number of rows committed    = 41

select  rtrim(tabschema) || '.' || rtrim(tabname) as qual_tab
    , const_checked 
from syscat.tables 
where CONST_CHECKED like '%N%' 
    or status != 'N'
    or access_mode != 'F'
with ur;
QUAL_TAB                                                                                                                                                                                                                                                          CONST_CHECKED                   
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------------------

  0 record(s) selected.

You’ll notice that I get a backup pending warning because my sample database is using circular logging. But the table is not placed in a check-pending state because it does not meet the conditions for that.

However, if I do the same thing for a table with refferential integrity, I do get a check pending state. Note that the data I am loading into the table is exactly the data that is already there, so this state is completely independent of the data being used:

$ db2 "export to projact.ixf of ixf select * from projact"
SQL3104N  The Export utility is beginning to export data to file 
"projact.ixf".

SQL3105N  The Export utility has finished exporting "65" rows.


Number of rows exported: 65

$ db2 "load from projact.ixf of ixf replace into projact nonrecoverable"
SQL3501W  The table space(s) in which the table resides will not be placed in 
backup pending state since forward recovery is disabled for the database.

SQL3109N  The utility is beginning to load data from file 
"/db2home/db2inst1/projact.ixf".

SQL3500W  The utility is beginning the "LOAD" phase at time "12/06/2015 
15:25:20.727176".

SQL3150N  The H record in the PC/IXF file has product "DB2    02.00", date 
"20151206", and time "152503".

SQL3153N  The T record in the PC/IXF file has name "projact.ixf", qualifier "",
and source "            ".

SQL3519W  Begin Load Consistency Point. Input record count = "0".

SQL3520W  Load Consistency Point was successful.

SQL3110N  The utility has completed processing.  "65" rows were read from the 
input file.

SQL3519W  Begin Load Consistency Point. Input record count = "65".

SQL3520W  Load Consistency Point was successful.

SQL3515W  The utility has finished the "LOAD" phase at time "12/06/2015 
15:25:20.936021".

SQL3500W  The utility is beginning the "BUILD" phase at time "12/06/2015 
15:25:20.941460".

SQL3213I  The indexing mode is "REBUILD".

SQL3515W  The utility has finished the "BUILD" phase at time "12/06/2015 
15:25:21.021448".


Number of rows read         = 65
Number of rows skipped      = 0
Number of rows loaded       = 65
Number of rows rejected     = 0
Number of rows deleted      = 0
Number of rows committed    = 65

$ db2 -tvf unchecked_query1.sql
select  rtrim(tabschema) || '.' || rtrim(tabname) as qual_tab
    , const_checked 
from syscat.tables 
where CONST_CHECKED like '%N%' 
    or status != 'N'
    or access_mode != 'F'
with ur;

QUAL_TAB                                                                                                                                                                                                                                                          CONST_CHECKED                   
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------------------
DB2INST1.PROJACT                                                                                                                                                                                                                                                  NYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

  1 record(s) selected.

In this case, the PROJACT table was placed into a check-pending state.

Identifying Tables in Check-Pending

There are several places to look for tables in a check-pending state. My favorite is the CONST_CHECKED column of SYSCAT.TABLES. If a table is in check-pending due to foriegn key constraints, then the first letter will be an N. If it is due to check constraints, then the second character will be N. The IBM DB2 Knowledge Center page on SYSCAT.TABLES explains the other possible values.

If the table is inaccessible due to an integrity state, the STATUS column will have a value other than N, and the ACCESS_MODE column will have a value other than F. So this query is really over-checking a bit, but it looks at all 3 columns that could indicate a problem:

select  rtrim(tabschema) || '.' || rtrim(tabname) as qual_tab
    , const_checked 
from syscat.tables 
where CONST_CHECKED like '%N%' 
    or status != 'N'
    or access_mode != 'F'
with ur;

Getting Tables Out of Check-Pending

To remove a table from a check-pending state, we need to first create an exception table, and then run a SET INTEGRITY statement to cause DB2 to apply all foreign key and check constraints and move exceptions to our exception table. Since LOADing data often affects more than one table, AND setting integrity for a table may put other tables in a check pending state, it is critical not to just address the one table you’re working with, but to examine the database for tables that need these actions. Because of this, I use SQL to write the SQL that I need to take these actions. It may seem like overkill in this scenario, but this is just the sample database, without much complexity.

Here is the SQL statement I use to generate the CREATE TABLE statements for exception tables:

select 'create table dba.except_' || rtrim(tabname) || ' like ' 
    || rtrim(tabschema) || '.' || rtrim(tabname) || ';' 
from syscat.tables 
where CONST_CHECKED like '%N%' 
    or status != 'N'
    or access_mode != 'F'
with ur;

I have that in a file and execute it with db2 -txf to get this in the example I’m using here:

create table dba.except_PROJACT like DB2INST1.PROJACT;

Or if it’s just one, I can also copy and paste at the command line. Execution of that looks like this:

$ db2 "create table dba.except_PROJACT like DB2INST1.PROJACT"
DB20000I  The SQL command completed successfully.

Obviously you can alter the naming here to match what works for you. I like to make it really obvious that a table is an exception table and like to put it in a separate schema to make it obvious that it’s a temporary table that I can go drop later.

After the exception table is created, you then need to generate the set integrity statement. It might be easier to create a set integrity statement for each table, but I like to do them all in one statement. The main reason here is that I’ve seen (in a WebSphere Commerce database) a case where there are circular foreign key relationships between two or three tables, and when that happens, all of the tables must be specified in the same set integrity statement. This means that if we specify all tables in set integrity pending in one statement, that kind of issue won’t trip us up.

Here’s the SQL I use to write the set integrity statement. Note that this has to be a bit complex to handle multiple tables. This SQL will only work on DB2 9.7 and higher due to the use of the LISTAGG function.

select 'set integrity for ' || listagg(rtrim(tabschema) || '.' || rtrim(tabname),', ') 
    || ' immediate checked for exception ' || listagg( 'in ' 
    || rtrim(tabschema) || '.' || rtrim(tabname) || ' use dba.except_' 
    || rtrim(tabname), ', ') || ';' 
from syscat.tables 
where CONST_CHECKED like '%N%' 
    or status != 'N'
    or access_mode != 'F'
with ur;

Again, I like to put that in a file and execute with db2 -txf and pipe it to a file. In the example we’re following here, the generated statement looks like this:

set integrity for DB2INST1.PROJACT immediate checked for exception in DB2INST1.PROJACT use dba.except_PROJACT;

If you changed the exception table naming in the previous statement, you’ll need to change it here, too. Executing that statement looks like this:

$ db2 "set integrity for DB2INST1.PROJACT immediate checked for exception in DB2INST1.PROJACT use dba.except_PROJACT"
SQL3601W  The statement caused one or more tables to automatically be placed 
in the Set Integrity Pending state.  SQLSTATE=01586

The SQL3601W is a fairly common warning message here. It tells us that setting integrity on this table caused other tables to be placed into a check-pending state. We now have to go through this process again to identify the tables and remove the check-pending state:

$ db2 -tvf unchecked_query1.sql
select  rtrim(tabschema) || '.' || rtrim(tabname) as qual_tab 
, const_checked 
from syscat.tables 
where CONST_CHECKED like '%N%' 
or status != 'N' 
or access_mode != 'F' 
with ur

QUAL_TAB                                                                                                                                                                                                                                                          CONST_CHECKED                   
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------------------
DB2INST1.EMPPROJACT                                                                                                                                                                                                                                               NYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

  1 record(s) selected.

$ db2 -txf unchecked_query.sql |tee rem_check_pending.sql
create table dba.except_EMPPROJACT like DB2INST1.EMPPROJACT;                                                                                                                                                                                                                                                                                                                                                                    

set integrity for DB2INST1.EMPPROJACT immediate checked for exception in DB2INST1.EMPPROJACT use dba.except_EMPPROJACT;   

$ db2 -tvf rem_check_pending.sql |tee rem_check_pending.out
create table dba.except_EMPPROJACT like DB2INST1.EMPPROJACT
DB20000I  The SQL command completed successfully.

set integrity for DB2INST1.EMPPROJACT immediate checked for exception in DB2INST1.EMPPROJACT use dba.except_EMPPROJACT
DB20000I  The SQL command completed successfully.

When you’re done, you’ll want to review the exception tables and see if there are any rows you need to address in them:

$ db2 list tables for all |grep EXCEPT_
EXCEPT_EMPPROJACT               DBA             T     2015-12-06-15.56.43.014510
EXCEPT_PROJACT                  DBA             T     2015-12-06-15.42.45.996397
$ db2 "select count(*) from DBA.EXCEPT_PROJACT"

1          
-----------
          0

  1 record(s) selected.

$ db2 "select count(*) from DBA.EXCEPT_EMPPROJACT"

1          
-----------
          0

  1 record(s) selected.

In this example, I did the LOAD … REPLACE with only the data that was already in the table, so there are no exceptions to worry about.

Please be aware that depending on the data size and the constraints to be checked, that the set integrity command can take quite a long time – longer even than the LOAD command itself took. When testing the duration of a LOAD be sure to always include testing of the time to run the SET INTEGRITY commands.

Script It

If you’re going to do this very often at all, it makes sense to write a script – in KSH, PERL, POWERSHELL or your language of choice – that will loop through this process for you. Especially if you’re working with a database with a web of referential integrity, you want a script that will look for any tables in check pending and repetitively work through this until there are no tables left in check-pending status.

I find the DB2 sample database useful for trying new things and proving to myself how things work.

Why Use the Sample Database?

The DB2 sample database is useful for a variety of reasons.

Testing Connectivity

Sometimes when building a new database server, we need to test connectivity from an application server or some random client into the new server. Having the sample database available makes this easy because we can test the full range of what connecting does without having to have our real database created yet.

Testing Ability to Create Database

Part of verification on a newly built database server may include testing whether a specific user can create databases or whether a database can be created on a specific path. The sample database is perfect for this.

Testing Various Functionality

There is always some new feature or some aspect of DB2 that I am investigating to understand exactly how it works. While much of this may be for blogging purposes, some of it is just to build my own knowledge and understanding for a particular process. No DB2 expert knows everything about DB2 and every detail of how it works, and for me, one of the best ways to understand something is to do it. The sample database on a Linux VM is a safe place for me to do this without risking any development environments.

How to Create the Sample Database

The sample database is very easy to create. You can simply run:

$ db2sampl

  Creating database "SAMPLE"...
  Connecting to database "SAMPLE"...
  Creating tables and data in schema "DB2INST1"...
  Creating tables with XML columns and XML data in schema "DB2INST1"...

  'db2sampl' processing complete.

Especially on undersized machines like my local Linux VM, this command can take minutes to run.

This creates a UTF-8 database called SAMPLE on the path defined by the DBM configuration parameter DFTDBPATH. It creates tables and other objects in a schema with the same name as the user executing the command (really the CURRENT_SCHEMA special register). It also imports both standard relational data and XML data into the tables.

You can change the name of the database created by using the -name option on the db2sampl command, or the path using the -dbpath option.

Changing the database name can allow you to create more than one sample database. If you do not change the name, but use the -force option on the db2sampl command, DB2 will drop the existing sample database and re-do the process. I probably end up dropping and re-creating my sample database on my local VM every couple of months, because as I’m trying stuff I end up changing things, and need to reset them to work through something.

Some of the details about the sample database are available in the IBM DB2 Knowledge Center.

Dropping the Sample Database

If you don’t need the sample database any more, you can drop it using the drop database command. This command always strikes fear in my heart, and I hate running it, so whenever you type “DROP” think twice to make sure it’s what you want to do.

$ db2 drop db SAMPLE
DB20000I  The DROP DATABASE command completed successfully.

Site Stats

2015 was again a record year for db2commerce.com. This year saw over 380,000 page views, which is more than 100,000 more than last year! I expect next year to not be such a drastic increase because my traffic has been fairly even through this year instead of the rapid series of upward steps I’ve seen in previous years. Despite the drastic increase in views, this was the first year I didn’t double the page views from the previous year. By contrast, in 2012 – the first year when I published an average of at least one post a week, I saw 65,000 page views the whole year. I do believe I may be saturating the DB2 audience.

In the life of db2commerce.com, I’ve had over 820,000 page views, which means I should pass a million in 2016!

I still get about half my traffic from the US while about half comes from other countries. India, Germany, and Canada are the top foreign countries I see page views from.

This year, I published about 63 posts, an average of more than one per week. I personally wrote 52 of those for a total of more than 62,000 words. Guest bloggers added another 17,513 words. I had SEVEN different guest bloggers this year, which beats my previous high of two. A big thanks to all of my guest bloggers:

• Ian Bjorhovde – 3 posts
• Abhik Roy – 3 posts
• Pavan Kristipati – 2 posts
• Jim Reutener – 1 post
• Luke Numrych – 1 post
• Mike Krafick – 1 post

Though I had more guest bloggers this year, I had fewer posts than in past years – last year, I had 14 guest blogger posts from my two guest bloggers.

Since I started the blog, I’ve written more than 420,000 words.

My best day this year was March 3, when I had 1,813 page views in one day!

Ember’s Year

The highlight of my professional year this year was presenting at the DB2 Symposium in Veldhoven, Netherlands. It was the first time I have been paid to speak, and the trip I took to explore Cologne and surrounding area afterwards was awesome.

I also spoke at IDUG in Philadelphia in May, and at Midwest Users groups in Milwaukee and St. Louis in March. I attended IBM Insight in November. I presented on the DB2 Night Show on October 16th.

At the end of 2014, I switched employers, moving from a consulting firm focused on WebSphere Commerce to a consulting firm focused on database consulting (XTIVIA). The move was absolutely the right choice for me. While I still miss some of the people I worked with before, I’ve had such exposure to different environments that it has been just awesome for me.

This year alone, I’ve had the opportunity to work on Native Encryption and two different production BLU databases. I’ve worked with DB2 running on a Pure Application server. I’ve done extensive performance tuning and upgrade work with DB2 on Windows. In fact, I’ve worked with DB2 on RH Linux, AIX, Windows, and HP-UX all in one year. I’ve done fixpacks or upgrades on most of those platforms. I’ve learned Power Shell. I’ve written more SQL (mostly on MON_GET table functions) than ever before, and have continued to build my skills in architecting and implementing various high availability and disaster recovery options for DB2. I’ve done health checks for at least 4 production WebSphere Commerce databases. I’ve spoken to a number of companies about their plans and ideas and steered them in the right direction and helped them ask the right questions.

More than ever, I’m sure that an expert does not know everything about a topic. They know what questions to ask and how to find the answers to those questions through experimentation, testing, reading, researching, or reaching out to others. In the spirit of Douglas Adams, it is not about the answers, it is about the questions.

Next Year

I plan to spend more time on BLU performance next year. As has been my hope for several years now, I still hope to find a client that will let me get my hands on pureScale. I also get to speak again at the DB2 Symposium in The Netherlands again in April!

Several IBMers hinted on the DB2 Night show Eggnog Party on December 18th that a new version or release of DB2

may

be coming next year – I’m looking forward to playing with that!

Happy New Year!

Happy New Year to all of my readers! I hope this year was as great for you as it was for me, and may your new year be even better!

Thanks to a late cancellation on the DB2Night Show, I’ll be presenting on Friday, April 18. I’ll be talking about why low-cardinality indexes negatively impact performance. It is the same topic as my developerWorks article, but I think it comes across better in a presentation format. It’s a really detailed topic, and I really enjoy it for that reason.

Sign up and come listen on Friday, April 18 at 9 AM Mountain Time: http://www.dbisoftware.com/blog/DB2NightShowNews.php?id=503

I’m so excited! The IDUG North American Technical Conference is less than a month away. Much like Melanie Stopfer mentioned in her blog entry on IDUG, this is one of MY favorite weeks of the year. There’s something about the IDUG conference. I have been to IBM’s DB2-related conference (which they’re now calling IBM Insight), and it was huge and interesting, but there is something more intimate about IDUG. It’s fun to see the same small group of friends each year and it’s a great place to make new DBA friends. I say friends rather than “contacts” because for me that’s more what this conference is about. Finding and catching up with people that I consider friends and talk to throughout the year. I’m a bit of an introvert, but for this one week, I become a social person, and it’s fun.

Conference for Newbies

The IDUG Conference is basically a week of DB2 education. There are such a variety of topics that each day there’s at least one, and possibly many, time slots where I have trouble figuring out which session I want to go to.

General Format

Like many conferences, IDUG consists of approximately hour-long presentations. The meat of the conference runs from Tuesday morning (May 13) through Friday mid-day (May 16). Monday consists of a day of extra-fee seminars. If you can get the funding for them. I particularly recommend Scott Hayes and Martin Hubel’s Top Gun Performance workshop. I have also heard good things about Roger Sanders’ Certification Preparation workshop. Personally, this year, I’m going for Dan Luksetitch’s Advanced SQL Coding and Performance workshop. I’m sure I’ll be blogging about it afterwards.

Usually the attendance peters out on Friday.

Registration

The check in desk is open every day starting on Sunday evening. The hours are:

 
Sunday:      4:30pm – 6:00pm
Monday:      7:00am – 5:00pm
Tuesday:     7:00am – 6:00pm
Wednesday:   7:00am – 5:00pm
Thursday:    7:00am – 6:15pm
Friday:      7:00am – 10:00am

Registration location:  Encanto Foyer, Second Level

Evenings

There are some interesting evening things. Usually IBM throws a party, and I believe it is on Wednesday night this year. It’s nothing like the big name bands at IBM Insight and other conferences, but it’s a good social chance to meet people and chat if they don’t play the music too loud. The Thursday night dine around is your chance to find one or two recognizable names in the DB2 world and just sit at a small table with them and chat. Each person pays their own bill, but they usually choose interesting restaurants, and I always loved getting some small group time with people I admired. I am proud to be one of the hosts of a dine around this year(squeeeee – am I a big DB2 name now?). I also lucked out in that I’m co-hosting with my DB2 hero, Melanie Stopfer, and we’re at Pizzeria Bianco, which I hear was named the top pizza restaurant in the United States a few years ago. Phew, this vegetarian did not pull a BBQ joint!

Building your schedule

If you haven’t been to IDUG before, or you’re not really familiar with the DB2 community, then navigating the sessions can be a bit difficult. The IDUG website has an agenda builder that lets you figure out for each time slot which session you want to go to. There is no requirement to sign up ahead of time, and no benefit for signing up through the agenda builder. Once you have registered for the conference, you should be able to go to the Attendee tab for the conference, and scroll down a bit to click on the large yellow “My Conference” button. It looks like this:

Once in the tool, click on “My Agenda” and then “Edit Agenda”. You can print out a compact sheet that will fit in your conference badge before heading out to IDUG, which is useful.

A tip in the tool – you can add a session to your agenda, then click on it to view the details to decide if you really want it. It is easy to remove the session if you discover it is not what you want.

There are fewer sessions that are outright sales pitches for IBM or for vendor tools, but you may still run into those. I most like to find speakers that I like and find the sessions that they are doing. But the IDUG agenda builder does not allow you to search on speaker name! The search feature doesn’t even let you search on a string. The way to look at things for IDUG is to look at sessions for the track that interests you. There are usually 2 tracks each for LUW and z/OS, a developer track, a track for big data, and a miscellaneous track (known as a la carte). You can even add every session for a track or two to your agenda, and then go through and filter out the things you don’t want to see.

Because using this tool doesn’t change the how the conference is run, I like to add all sessions I am interested in, including conflicting ones. It’s nice to have a backup plan and to talk to friends about which session you’re going to next, though I tend to avoid having more than 2 in a slot so I’m not making too many decisions on the fly. While at the conference, I tend to tweet where I’m going next or what I think are good bets for the day. There may be others tweeting this kind of information, so it’s nice to watch the #IDUG hashtag.

When looking at sessions, be aware of:
* The session’s track – this can give you an idea if the session is in an area you are interested in.
* The session’s audience experience level – it can be frustrating to end up in a session that is way too advanced for your knowledge in a subject area, or in one that is way too basic. Stretch yourself, just be aware of when you are doing so.
* The session’s presentation platform – many things that are true for LUW are not true for z/OS, and vice versa. If a presentation is cross platform, it will either list cross-platform or it will list both LUW and z/OS.

Vendor Expo

There’s a vendor Expo. It can be an interesting place to find tools for DB2 and talk details with the people who know about them. It’s also a good place to pick up swag for the kids. There’s usually a “passport to prizes” where you have certain vendors stamp/sign a card and can then be entered to win a host of prizes. You have to be present at the drawing to win. The Expo opens Tuesday night, and is open Wednesday around lunch and Wednesday evening, and then usually closes on Thursday after lunch. The reception on Tuesday night usually has drinks and light food – you can make a dinner out of it if you’re on a budget, or leave room to find dinner with friends later in the evening.

Since there are often food and drinks in or near this room (lunch is often next door or in the same room), it tends to be a place people hang out when there are no sessions.

There are two times – Wednesday after lunch, and Thursday before lunch, when there are Vendor Solution Presentations as the only options for a time slot. These can be interesting for exploring tools you might be interested in. They can be pretty sales-y, but they can also show you a tool in more detail.

When looking at the tools and also at sessions, remember to make sure you’re looking at things for LUW if you work with LUW or z/OS if you work with z/OS. It can sometimes be interesting to cross over and explore things on the dark side, but it’s frustrating if you don’t pay attention and end up with too much that doesn’t relate to what you want to learn about.

Presentation Downloads

All presentations will be available for download from the IDUG site. Usually before the conference even starts. We speakers are required to submit our presentations a month before the conference starts. If you’re really torn about which session to go to, you can look at the presentation to see which one interests you more. Some presentations also include great notes from the speaker. I’ve also seen people either print or make notes on PDF copies of the presentations. You’ll always see me with an iPad or laptop in hand taking my own notes, though historically I haven’t done it on PDFs. Might try that this year, we’ll see. Gotta love Evernote.

Certification

If you are ready for certification, they’ll be offering one free certification test on IBM Information Management products. If you pass the first one you take, then you may take another for free. Additional exams are only $25, a great discount for the ~$200 they’re charging for exams these days. This is usually a cost justification for going to the conference, and I haven’t taken a DB2 certification test outside of a conference in years. Especially if you’re going for a certification upgrade, just absorbing the information during the week may help you be ready for the exams. Certification hours are:

Tuesday, May 13:     10:00 - 17:00
Wednesday, May 14:   9:00 - 16:00
Thursday, May 15:    8:00 - 17:00
Friday, May 16       8:00 - 12:00

You can find the list of available exams here: http://www.idug.org/p/cm/ld/fid=502

Food

Generally IDUG provides lunch. There is usually no breakfast, so eat before starting the day. There usually is not even coffee available until the morning break time. Usually there is coffee and tea at the morning break. The afternoon break tends to include sodas and sometimes cookies or other light munchies. The only food provided in the evenings is what is specifically listed – usually light food in the Expo hall some nights. You pay your own way for the dine-arounds.

On Monday/Tuesday, I am usually searching for my Diet Coke dealer for the week. I need something like the “Kennedy plan” that Lily on “How I Met Your Mother” had for the last season, but with Diet Cokes. Every time I turn around, I want a cold Diet Coke in my hand.

Wi-Fi

IDUG tells me that there will be wifi throughout the conference this year, including in the sessions. Please be kind to the speakers and give them your full attention.

Often finding power outlets can be an issue, too. If I’ve got my laptop on me, then I’ll also have a mini-power-squid with an outlet or two available to share.

We all have day jobs and many of us may have to take breaks from sessions in order to take care of work tasks. One conference I promised my boss that I’d put in a 40-hour work week, while attending the conference. Let me tell you, that was no fun, and boy was I fried at the end of that week. I may have to play an on-call role for half of this conference – we’ll see. I also keep meaning to blog throughout the conference, but tend to run out of steam on that. Usually my blog entries take an hour or two to write, and that on top of a full day of sessions and evening fun and keeping up with work just doesn’t usually happen. I do tweet heavily throughout the conference, so follow me on twitter – @ember_crooks.

Wardrobe

I tend to go more business than casual, but many people will be in jeans. My main reason is because I only travel on business about 4 or 5 weeks a year, so I have to actually use those business clothes. The one thing I change from business wear, though is usually to wear my Birks with everything. No pantyhose and comfortable shoes are my concession to the casual. Another reason I tend towards business is because I freeze my rear off at these conferences, and at least all my business stuff tends to have a jacket. I may actually add in some sweater tights to improve my thermal factor. Hmm, what do I have that works with long underwear?

Zero Weight Gain

It is hard to keep up with healthy eating habits when on trips like this. And I usually don’t do great, but I do try some, and that includes bringing workout gear in the delusion that I’ll actually work out. I do plan to do some walking, and I think Ian Bjhovde may actually set up a competition for those with FitBits or Jawbone Ups. IDUG is usually at smaller venues than conferences such as IOD/IBM Insight. That means less built-in walking. In Phoenix, I’m pretty sure it will in the 90’s or higher every day, so walking outdoors is mostly out. Maybe we’ll get a nice cool day on Sunday, and someone will have a car and we can go out hiking or something. At least it’s unlikely to be the near 24/7 rain we saw last year in Orlando. Maybe I can get a room on a higher floor and take the stairs most of the time

Come and find Ember at IDUG

I will certainly be at IDUG NA in Phoenix. I have the following scheduled events:

• Tuesday, May 13th at 3:15 PM in Ahwatukee – Session E03 Why Low-Cardinality Indexes Negatively Impact Performance
• Thursday, May 15th at 9:15 AM in Camelback – Session D10 High Availability Two-Step: HADR and Power-HA. I’m co-presenting this one with DB2’s Got Talent winner Michael Krafick
• Thurday night dine around. Paper sheets for signup will go up on Tuesday after either the keynote or the spotlight sessions. These usually fill up really quickly, so sign up as soon as the sheets go up. Melanie Stopfer and I will be hosting a group at Pizzeria Bianco.

Additionally, I promise you that I want to talk to each and every blog reader and love meeting folks. So if you see me, don’t hesitate to introduce yourself. And take a picture! I’d love to have pictures to post on the blog and on Twitter.

Some of Ember’s Top Picks

I always enjoy the Special Interest Groups (SIGs) on Wednesday at 3:30. I am always torn between two or three of them. There is also a panel on Thursday afternoon at 4:45 PM in Paradise Valley that I enjoy – it’s an open-format question and answer with the top experts in DB2 up on the podium – so fun to be able to ask anything and see the questions others are asking.

One of my favorite sessions that I go to every chance I get is Matt Huras’ DB2 Internals for Administrators sessions. This is a two-part session on Tuesday, and every time I go, I take something different away from it. There is simply no other place you can go to learn what DB2’s doing in this kind of depth. I generally come out bleary eyed from the technical details, and I remember being vastly overwhelmed when I was a DBA of about 3 years. But truly it’s such awesome details and will make just about anyone think about something they haven’t before.

Another favorite for me is anything Melanie Stopfer presents. She will cram more information into your brain in one hour than many other presenters would in a day. Her presentations also have awesome student notes. I even go to her sessions when it is something I don’t expect to apply to me, because I will invariably learn something that will help me in my job anyway. Her upgrade presentations are simply a giant bomb of useful information from the real world – it’s not just about how the IBM Toronto Lab expects things to work. Her presentations this year are:

• Wed May 14, 8:00 AM, Paradise Valley – C05 Best Practices Upgrade to DB2 10.5 with BLU Acceleration
• Thu May 15, 1:00 PM, Camelback – D11 Using Row and Column Access Controls (RCAC) with DB2 LUW

There are many other good speakers. I like Michael Kwok (his presentation conflicts with mine! Darn it!), Michael Krafick (Wed 8 AM, Attack of the Blob! (Managing BLOBs within a DB2 database)), Scott Hayes (Wed 9:15 AM, Sage Advice: 20 Invaluable DB2 LUW Performance Insights from Around the Globe), Kohli (Wed 10:30 AM, Nifty Performance Tricks: Essentials for Success), and Dale McInnis (Thu 2:15 PM, Continuous Availability with DB2 AND Fri 9:15 AM, High availability disaster recovery and performance – a marriage made in heaven).

I’m so looking forward to the conference, and hope to see you there!

This technote caught my eye when @db2_support tweeted it the other day ago. It was titled as “DB2 might choose a sub-optimal query execution plan due to distribution statistics”. The title alone intrigued me. As I read through, I thought that I needed to do a bit of investigation to better understand the issue. It also seemed like something I could write a query or two to investigate and see if it was occurring in my databases.

Description of the Issue

This particular problem appears to be based on the fact that DB2’s runstats utility only looks at the first 32 characters of a longer text field when collecting distribution statistics. I always recommend distribution statistics because the more information the DB2 optimizer has, the more likely it is to generate a truly great access plan.

Distribution statistics means the quantiles and frequently occurring values are stored in SYSSTAT.COLDIST. By default, distribution statistics will store the 10 most frequently occurring values and 20 quantile values. Quantile values means that DB2 stores the value found 1/20th of the way through the table. If the table had just 20 rows, this means that every value would be stored. If the table had 40 values for this column, every other would be stored. With the default settings, 20 quantile values are stored for each and every column in each and every table, no matter what the size of the table.

The problem appears to be that the runstats utility considers only the first 32 characters of a string, no matter what the length of that string is. This presents a problem because DB2 relies on these values to estimate how many rows a query will return. If DB2 estimates incorrectly, it might choose the wrong access path to retrieve the data, possibly resulting in a query that performs worse than it might.

I have a fair number of mid-sized (50-1000 character) VARCHAR fields in the WebSphere Commerce databases that I support. I could also immediately think of two tables where it would be likely that I would have fields where the first 32 characters were the same, but the string was different later on.

Looking to see if the Conditions for the Issue Exist

The first thing I did was to develop a query to help me find tables/columns where the potential for this problem existed. With thousands of tables each with many columns, it’s just too much for me to know immediately where I might face this issue. So I developed this query to pull potential areas out of sysstat.coldist.

SELECT     substr(tabname,1,20) AS tabname, 
    substr(colname,1,20) AS colname, 
    length(colvalue) AS col_length, 
    valcount, 
    substr(colvalue,1,40) AS colvalue, 
    seqno 
FROM sysstat.coldist 
WHERE   tabschema='WSCOMUSR' 
    AND tabname NOT LIKE 'TI_%'
    AND colvalue is not null 
    AND length(colvalue) > 30 
    AND type ='F' 
    AND valcount>100 
ORDER BY    3 DESC, 
        tabname 
WITH UR;

What this query does is look only at the frequently occurring values, and look for ones with higher numbers of occurrences, where the length of the value is over 30 characters. I found it interesting that there were no values in my database longer than 37 characters, which confirms for me that there is some internal limit. Oddly enough, the COLVALUE column in SYSSTAT.COLDIST is defined as VARCHAR(254). When I ran this query, I came up with output like the following:

TABNAME              COLNAME              COL_LENGTH  VALCOUNT             COLVALUE                                 SEQNO 
-------------------- -------------------- ----------- -------------------- ---------------------------------------- ------
ATCHAST              ATCHASTPATH                   35                  438 'images/catalog/apparel/apparel_55'           2
ATCHAST              ATCHASTPATH                   35                  402 'images/catalog/apparel/apparel_40'           3
ATCHAST              ATCHASTPATH                   35                  355 'images/catalog/kitchenware/kitche'           4
...
SRCHSTAT             SUGGESTION                    32                19001 'fallen, face, fast, fade, else'              4
CTXDATA              SERVALUE                      31               178211 'null&null&null&null&null&null'               4
PX_ELEMENT           TYPE                          31                 8486 'IncludeCatalogEntryIdentifier'               1
PX_ELEMENT           TYPE                          31                 1909 'ExcludeCatalogEntryIdentifier'               3

  310 record(s) selected. 

310 or fewer tables is something I can deal with looking at, at least.

From this point, it requires a bit more knowledge of your application or your data model to dig down into what might actually be a problem area. Just because a value is more than 30 characters does not mean it is a problem. But if you look at what the field is used for and put that in context of the table, you may be able to find areas where the value changes in the later characters.

In the case of one of the tables here, ATCHAST, the table is used to store information about ATtaCHment ASseTs. The data stored includes the paths to files. It is assumed that many of the files are stored in similar locations, and therefore their paths may start similarly, but change at the end. This is exactly the kind of column I need to be looking into.

Once you have used a query like the one above, and have applied your own knowledge about the data model to determine which tables/columns might be a problem, you’ll want to examine if the problem actually exists

In the case of the ATCHAST table above, I would run something like this:

SELECT substr(atchastpath,1,70) AS atchastpath FROM atchast WHERE atchastpath LIKE 'images/catalog/apparel/apparel_40%' WITH UR

ATCHASTPATH                                                                     
----------------------------------------------------------------------
images/catalog/apparel/apparel_40x40/IMG_0034_q.jpg                   
images/catalog/apparel/apparel_40x40/IMG_0055_q.jpg                   
images/catalog/apparel/apparel_40x40/IMG_0077_q.jpg                   
images/catalog/apparel/apparel_40x40/IMG_0062_q.jpg                   
images/catalog/apparel/apparel_40x40/IMG_0054_q.jpg                   
images/catalog/apparel/apparel_40x40/IMG_0058_q.jpg                   
images/catalog/apparel/apparel_40x40/IMG_0020_q.jpg    `
...

I can see in this case, that indeed the values are the same until the last few characters. This tells me clearly that this problem has a real potential for happening in this case.

Now the next piece is to consider if this column is used in a way that matters. Would I ever have a where clause that would look for other information in the table by specifying the value in this column? Or a query that would look for values that were greater than or less than this value? There may be some cases where the answer is no. In 10.1 and higher, it may be useful to create a usage list to capture the SQL on the table in question.

When reviewing SQL in this context, remember that SQL with parameter markers (?) cannot ever make use of distribution statistics, so remember to discount any SQL you see along those lines.

Fix for the Issue

Here’s an interesting part. This is a technote and not an APAR. To me, this means that they have no intention of fixing it. C’mon, IBM, can runstats be improved to at least handle 254 characters for distribution statistics?

The stated fix is not to use distribution statistics at all, for the specific columns. That would probably lower the scope of the problem when it is causing access plans to be significantly off, but all that does is make it so DB2 assumes a normal distribution – which is just another way of estimating incorrectly, just hopefully not as far off.

Intriguing thought – SYSSTAT tables are update-able. I wonder if I could manually calculate the values to replace problem values like these, and then update SYSSTAT.COLDIST? If I did, would the optimizer be able to handle the longer values?

This is the first in a series of blog entries on reorgs. I was talking with a friend recently, and he pointed out that I only have a few articles on reorgs, and they’re very specific to complicated parts of reorg. I have scripted my own reorgs and made a point of educating myself on reorg, so I’m starting with several basic articles and may add in some more advanced articles.

Right now I plan to start with this article – What is a Reorg? – and do additional entries on how to determine if a reorg is needed and the different options when running a reorg. This series will also heavily cover reorgchk, and I may add another article in on the SQL that can be used to calculate reorgchk formulas.

Why Reorgs are Important

Reorgs are one of the keys to maintaining performance on a DB2 LUW database. One of the items on my list for the most basic regular tasks that DBAs need to perform is to run reorgs on the tables where reorgchk says they are needed on a regular basis. My databases have appropriate reorgs done on them weekly, though monthly may be frequently enough.

Over time, the regular activity in a database can cause minor issues that reorg can solve. If reorgs are not done in a timely manner, those minor issues can cause performance-killing and space-eating issues.

General Facts About Reorgs

A reorg is done at the table level, but may also specify an index. Some kinds of reorgs affect only tables or only indexes. A reorgchk can be run either at the table or the database level to help determine which kinds of reorgs are needed for what tables. Reorgchk requires current runstats to correctly make these determinations. Some kinds of reorgs make a table completely offline, while others keep the table online for a portion of the reorg.

Purposes of Reorgs

Data Clustering

Clustering Index

A clustering index in DB2 means an index that is defined with the CLUSTER keyword. Clustering indexes means that DB2 attempts to maintain the table in the same order as the index. It is not guaranteed to be in the same order, though. If the data will not fit on the page that the clustering order would prefer, then DB2 will try nearby pages, but will not move data around to make it work. There is no clustering by default like there is for some other RDBMSes, like MS SQL Server. If you reorg a table with a clustering index, DB2 will reorder the table data to exactly match the order of the clustering index. If you add a clustering index, you will have to reorganize the table before the data will be reordered.

Clustering a Table Over an Index that is Not Defined as a Clustering Index

If you do not have a clustering index on a table, you can still use reorg to order the data in the order of any index, simply by specifying it in the reorg command. If you do this, DB2 makes no attempt to maintain the table data in the order of the index – it is only on reorg that this is done. If there is a clustering index on a table, you cannot reorg the data on a different index.

Grouping Empty Pages and Physical Continuity of Table Data

Over time, as rows are deleted from a table, the space that those rows took up is not freed. It may be reused by rows inserted into the tables, if they fit and APEND mode is not used for the table. But especially if a large amount of data has been deleted, the data on the table pages may actually be sparse. One of the main things that reorg does is to completely fill up the tables (respecting PCTFREE), and push all of the empty space to the end of the table. Depending on the reorg options you choose, it may also free that space up to be used by other tables in the tablespace. Other than freeing space on disk, the other advantage here is that data that may be frequently accessed together (by table scans or by prefetching) is physically located together on disk.

Compression Dictionary Generation

There are a number of times that compression dictionaries are generated for DB2 row level compression, and it may depend on your version. But there are situations where you may want to explicitly reset/recreate the dictionary, and there is an option on the reorg command to reset the compression dictionary

Reorg-requiring operations

There are some operations that require a table reorganization after they are done. These include:

• Data type changes that increase the size of a varchar or vargraphic column
• Data type changes that decrease the size of a varchar or vargraphic column
• Altering a column to include NOT NULL
• Altering a column to inline LOBS
• Altering a column to compress the system default or turn off compression for the system default
• Altering a table to enable value compression
• Altering a table to drop a column
• Changing the PCTFREE for a table
• Altering a table to turn APEND mode off
• Altering a table or index to turn compression on

Note that some of the above are “reorg recommended” operations, where you can do up to 3 before a reorg is required, and you are limited in some of the operations that can be done against the table until a reorg is completed. Others require immediate reorganization before any actions may be performed against the table. Other things in the list above simply won’t get the benefit of the change until the reorg is completed.

Index maintenance

In general, one of the purposes of reorgs may be to rebuild indexes. How, when, and even if this happens depends on the type of reorg you are running.

When rows are deleted from indexes, the RIDs are essentially marked as deleted, and not actually deleted. When they’re marked this way, they’re called pseudo deleted RIDs. There are two ways that these index entries are finally deleted. If MINPCTUSED is not set for the index (which is the default), then reorg can clean them up. When invoked for indexes, reorg can rebuild all of the indexes or it can go through a cleanup only mode to simply remove the pseudo-deleted rows and pages. This reorg process can solve problems with leaf page fragmentation, low-density indexes, indexes with too many levels, and also will respect PCTFREE – to ensure there is free space on index pages for additional inserts.

When reorging indexes for a table, you can only reorg all indexes for a table, and not an individual index. The exception is when you’re reorging non-partitioned indexes on a partitioned table.

Summary

This blog entry covered what reorgs do at a high level. Future entries about reorgs will cover reorgchk and how to determine what tables to reorg and the syntax and specifics on the various types of reorgs.

What a Commit is

Database management systems have the concept of a commit. This concept has to do with grouping sets of actions together such that they all succeed or fail as a group. This represents the A in the famous ACID properties of a transaction system. The A stands for Atomicity – meaning that a transaction may consist of multiple small parts, but that if one of those parts fails, then the transaction as a whole must also fail to have made any changes to the data. This concept is especially critical in relational databases which may be normalized into many smaller tables, which leads to a transaction consisting of more than just a single update in a single location.

Assuming that an application does not have autocommit turned on, the following represents and example of a transaction and the use of a commit.

BEGIN TRANSACTION
select columns from table a where id < NNNN;
update table_a set col1=1234;
update table_b set col2=5678;
insert into table_c .... ;
commit work;
END TRANSACTION

In the above example, if the update statement for table_b fails, then the application will detect that and the commit work statement will never be executed or a rollback statement may be executed. DB2 will rollback the update to table_a and the insert into table_c.

A commit also has to do with the D in ACID - Durability. The durability property means that once a commit happens, the data persists (traditionally on disk), even if power is lost or other likely failures occur. This is a major reason that databases have transaction logs.

When Commits Occur

The DB2 command line usually has autocommit turned on by default. This means that if you are simply issuing db2 commands from a command line, then you do not generally have to commit. Many applications manage commits in other ways, including some GUI database administration or access applications.

When designing an application, designers and developers must be certain that commits are happening when they are supposed to.

DBAs also advocate for frequent commits during long-running actions like data loads or large deletes. Largely, this is so the transaction log files do not fill up.
Frequent commits support the highest level of concurrent data access.

What Commits Do

DB2 uses a method referred to as write-ahead logging. This means that when a commit occurs, the data is written directly to the transaction logs. Data gets to the tables and such on disk asynchronously, through the buffer pool. A commit does not write data out to the tables itself. This saves time - the end user is not sitting and waiting on as much I/O.

A side note - DB2 writes both redo and undo data to the transaction logs - it is not like Oracle where redo and undo logs are separate things.

DB2 has a memory buffer called the log buffer. As data is changed, it is written to the log buffer, committed or not. The log buffer is then written out to disk either when it becomes full or when any connection does a commit. The commit is not successful until it has been externalized from the log buffer to the log files on disk.

Uncommitted data can thus be written from the log buffer into the log files when this occurs. But DB2 tracks and knows which transactions are committed, and which are not.

In the image above, an agent writes a commit record to the log buffer (1). The logger process then writes the log buffer to disk (2) - either immediately or shortly based on MINCOMMIT and other factors. The agent waits for the acknowledgement from the logger process that the log records for the agent's commit have been externalized to disk (3), up to and including that commit record. More than one agent may be waiting at a time. Since this is where end users are waiting on physical I/O, it makes sense to apply your fastest disk to the transaction log files.

Note that the commit does not touch bufferpools or tablespace data. As statements have come through, they have been writing all the information for both undo and redo to the log buffer, so it's already there or in the transaction log files waiting for the commit.

If a database crashes, then when it comes back up, it goes through a process called crash recovery, which looks at the transaction log files and rolls forward through any transactions needed. After it completes that "forward phase" of crash recovery, it goes and rolls back any transactions that were in the log files and were not committed.

Overall

Wow, overall, the conference ran so smoothly this year. I saw very few issues. The sessions were well clustered by track and all located in a small area. It was nearly always clear where to go and what was going on. My hat is off to the IDUG Conference Planning Committee – I cannot imagine how much work it took to put on such a conference.

I am going to mention a few things that I had issues with, but I want to state out front, that the solution to any issues you saw or that I saw with the conference is to volunteer and give your time to help IDUG make things better. You can apply to volunteer on IDUG’s website. I spend so much time on this blog that I don’t feel able to volunteer right now. Maybe I will in another couple of years. So any criticisms I have must be taken with the grain of salt that I am not willing to put my own shoulder to the wheel to help fix them at this time.

The thing that really struck me this year was how many people had read this blog and appreciated it. It really touched me to see that I am having an impact. To see that the stuff I write really does help. It is easy sitting in my basement to forget that my blog represents a significant resource and a voice outside of IBM. I may not know everything about DB2 for LUW, but what I do know, I like to share. Significant information for most blog entries comes from the IBM Knowledge Center, but even then, consolidating and re-organizing the data can be useful.

Location

I liked the location of Phoenix. For me it was close to home, and I have family in the area, so it was nice to see family. The hotel itself seemed the perfect size – plenty of meeting places, but not overly huge. I was lucky enough to get a great nightly rate, so it was affordable. The restaurants and such across the street were just fine. I did not really get more than a few blocks from the hotel except for on Sunday. As a vegetarian I had pretty much the same thing for Lunch each day, but the vegetarian option was always available, and it was not bad.

Social Media

I do feel like IDUG is struggling with social media a bit. There were very few tweets from the official IDUG account – something they did much better with last year in Orlando. I also wish some information was more easily available and more consistent. It took several emails for me to get some very basic information pre-conference. As a blogger, I wish they would send me a packet with more information that I could possibly use. I was particularly disappointed that the dine-arounds were not consistently announced and did not seem to be published anywhere. They were mentioned in a couple of pre-conference webinars, but seemed to be a moving target. I was the co-host of one of them, and was informed less than two days ahead of time that the location had changed.

I would really like to see IDUG with a larger Twitter presence.

I do like how the content committee seems to be getting some good things out there, and I am excited to see what they come up with.

Networking

Wow, what a great place to network with others. I met so many new people and made new contacts. I think I did a decent job of making sure that a few new attendees were introduced to people and found people to hang out with. It was an interesting place for me to be. It was the first year that I spent so much time supporting colleagues and less time than before going to sessions for my own education.

I was hardly able to sit or stand alone for more than a few minutes without meeting someone who reads the blog or is aware of who I am. I loved that and felt like a rockstar. I am, by nature, an introvert, and about the scariest thing to me is walking into a room and making small talk with people I have never met before. It makes it so much easier and funner when people walk up to me.

I very much enjoyed the breakfasts. Thanks to Sponsors DBI, DanL, and Gunning Technologies. I sure hope IDUG figures out how to continue that.

Problems

I wish there would have been a speaker practice room. My co-presenter and I had trouble finding another room to steal for a bit to work on our presentation. I am glad we did grab an unused room during the VSPs because we found that the text on a couple of our slides was difficult to read. It looked just fine on a laptop screen or monitor, but would have been unreadable with the projector if we had not noticed that and changed it. There are small things like that for which speaker practice rooms are very valuable.

Technical Education

I call the sessions technical education, because that has always been the biggest value of the IDUG conference to me – the educational sessions. I cannot get such in-depth technical information on varied topics at 10 times the cost anywhere else.

I also somehow managed not to land in a single session where IBM was trying to sell me something as the majority of the session. I don’t know if that’s IDUG getting better or me getting better with choosing sessions.

In nearly all of the time slots there was a clear choice for me as to which session I wanted. I appreciated that there were very few times where two of my favorite speakers were up against each other. That said, it was the very first conference I have ever been to where I have not managed to go to one of Melanie Stopfer’s presentations. For her upgrade presentation, I was moderating for and supporting a friend – and had attended the same awesome presentation at IOD in November. Melanie’s other presentation was on RCAC which is not something I see myself as likely to use in the next year, and she was up against Steeve Rees. I was glad I had picked Steve’s presentation – I think it was the best presentation of the entire conference for me.

Monday

I started out my week with one of the educational seminars. I went to Dan L’s Advanced SQL Coding and Performance workshop. It was very much worth the time and money. Dan is a wizard with SQL. I can hardly wait to apply some of the techniques he presented to a query I have been working on. It really took my SQL knowledge to the next level. I do not write a lot of SQL, but I do help our developers sometimes, and feel like I can now be more effective with that.

Tuesday

Starting out the regular conference week with Matt Huras’ two-part internals session was the right way to start for me. I have probably been to about 4 or 5 of them since the first time he blew my mind with them back when I had been a DBA only about 3 years. But I always pick up new details, and the real value add for me this time was a wealth of blog ideas.

Overall, I have come out of the week with over a dozen very specific blog ideas – the conference is such a wealth of inspiration for me.

My own session on Low-Cardinality indexes went OK. I got a lot of good comments on it, though I feel like I was talking too fast, and already have some thoughts on how to tweak it to be even better before IDUG in Prague if my presentation is accepted there. I had over 30 people show up, which was awesome.

I enjoyed Leo Pedron’s presentation. He had a bit of an inappropriate question asker. There was good information there. Some of his methodology was similar to my favorite developerWorks article on emulating monitor reset on DB2 9.7 and above. I am still not sure about the “here’s a bunch of stuff I use” approach to presentation creating. I think I prefer a presentation that sticks more to a tight theme.

Wednesday

Wednesday morning, I though that Mike Krafick’s LOB presentation went extremely well, and he had a good turnout considering he was up against the very popular and good Melanie Stopfer

Scott Hayes’ presentations always have some interesting elements, and his backpack trick in Wednesday morning’s session did not disappoint. That said, man, Scott, we get the soup slide already. I would be happy to see that slide gone.

I must admit to skipping the Vendor presentations in an attempt to find a place to practice a presentation with a co-presenter.

Pavan is a promising new speaker I saw on Wednesday afternoon, though I thought that the “DPF” in his presentation title was not needed – he had a lot of good tips that would work just as well in non-dpf environments.

I went to the SQL dojo, and continue to think that is a fun format. Though my team never got the keyboard.

Thursday

Thursday was probably my favorite day at the conference. I was missing a bit of that “mind=blown” experience that I am used to getting from IDUG conferences. There was great information up to this point, and great inspiration, but man, my favorite session of the whole conference was Steve Rees’ Performance FAQs. I took pages of notes.

My own presentation with Mike Krafick went well on Thursday morning.

We had a great turnout.

I also very much enjoyed Adam Storm’s comical and informative presentation. I am not great with humor, though I can cover information. Adam has both down pat.

The Thursday keynote was interesting – certainly it spoke to the choir of DBAs on the topic of big data.

The final session on Thursday was one of my favorites – the LUW Panel. I love the panels because I get to see what questions others are asking and see answers that I never would have thought of asking about. I always learn something.

Thursday evening, I co-hosted a dine-around with Melanie Stopfer. Fun as a co-host for the first time.

Friday

Friday had my second favorite session of the conference – Dale McInnis’ detailed look at HADR and Performance. I have so many things to play with and try coming out of that session. Again, I took volumes of notes. I also enjoyed Paul Turpin’s session on analyzing what is causing wait time in a database. How could I not love a speaker calling out my blog as a resource that he used extensively?

Summary

Overall, the conference was great, and I hope to get a chance to go to IDUG in Prague later in the year. If I do, it will be my first time at IDUG in Europe, and actually my first time out of the country in about 10 years. I’ve spent a fair amount of time overseas over the years – it has just been a while. Anyone have recommendations for learning a few basic phrases in Czech? I also hope to make it to IBM Insight in October.

Reorgs are one critical piece of both performance maintenance and maintaining how much disk space is used. In my first blog entry in this series, DB2 Basics: What is a Reorg, I talked about some purposes of reorgs. In this entry, I’m going to talk about how to determine what kind of reorg is needed on what tables. I will have a future blog entry to discuss index reorgs in detail. The advice I’m giving here is directed at single-partition databases with non-partitioned tables.

Reorg’s Place in the Database Maintenance Cycle

I run a standard maintenance cycle weekly. It consists of:

• RUNSTATS on all tables (yes, including system tables)
• REORGCHK on all tables
• REORGS of tables that indicate a need
• RUNSTATS on tables that have been REORGed
• rbind with the \all option
• flush package cache

REORGCHK

Reorgchk is a DB2 utility that calculates a number of formulas to determine what kinds of reorgs are needed on what tables. You can run reorgchk on a single table, on a schema, or on all tables in a database. While you can optionally have reorgchk update statistics, I do not recommend it. Reorgchk is supposed to detect the type of statistics that were last gathered and base what kind of statistics on that, but do you really want to take the chance with something as critical as RUNSTATS? Since I have scripts already from older versions, I collect runstats first, then run reorgchk.

The Reorgchk page in the Knowledge Center actually has a lot of good information on it. It tells you things like the recommended types of reorgs based on the output of reorgchk.

The simplest form of the reorgchk command is this:
db2 "reorgchk current statistics on table all" >reorgchk.out

Running that statement requires a database connection. The output may be extensive, especially if you have a significant number of tables.

When you run reorgchk, there are two sections – one for tables and one for the indexes on tables. To make things easier, those are covered separately below.

Tables

The table section of reorgchk looks something like this:

Table statistics:

F1: 100 * OVERFLOW / CARD < 5
F2: 100 * (Effective Space Utilization of Data Pages) > 70
F3: 100 * (Required Pages / Total Pages) > 80

SCHEMA.NAME                     CARD     OV     NP     FP ACTBLK    TSIZE  F1  F2  F3 REORG
----------------------------------------------------------------------------------------
Table: WSCOMUSR.ATTRIBUTE
                                   0      0      0     68      -        0   0   0   0 -**
Table: WSCOMUSR.ATTRTYPE
                                   7      0      1      1      -      448   0   - 100 ---
Table: WSCOMUSR.ATTRVAL
                               29097    653    623    624      -  2473245   2  99 100 ---
Table: WSCOMUSR.ATTRVALDESC
                               29013      6    874    874      -  3452547   0  99 100 ---
Table: WSCOMUSR.ATTRVALUE
                                   0      0      0   1386      -        0   0   0   0 -**
Table: WSCOMUSR.DMEXPLOG
                               34165      0    847   1056      -  3348170   0  79  80 --*
Table: DBAMON097.TABLE_HIST_CRON
                              256149      0   1652   2284      - 53535140   0  71  72 --*

The above output gives a lot of information including information on the size of the tables. There are three formulas that reorgchk calculates – F1, F2, and F3. They are described at the top of the reorgchk output, and calculated for each table. The column at the far right is what tells us that a table reorg is actually needed. Note that there are three values – either dashes - or asterisks *. Each of these represents one of the three formulas, in order. An asterisk indicates that a table reorg is needed for that table, based on the results of that specific query.

F1

F1 is described in the reorgchk header as:
F1: 100 * OVERFLOW / CARD < 5
What this formula is looking for is overflow records. In DB2 a row is stored contiguously on a single page. When actions like updates to VARCHAR columns cause a row to no longer fit on a page, DB2 will replace the original row on the page with a pointer to the new location for the row. If more than one relocation of the row is done, the original location is updated with the true location, so DB2 never has to do more than one hop to get from the pointer to the data. However, even that one hop means that the I/O (whether from bufferpool or from disk) is doubled, which can be detrimental to performance. A reorg will get rid of these pointers and ensure the data is in the location within the table where it is expected to be. This process involves updating the row’s rid, and thus indexes must be rebuilt or changed to match the new location.

Formula F1 looks to ensure that 5% or less of the total rows in the table are overflows. I’m going to share SQL I’ve used to calculate these values manually back in version 8.2 and 9.7. Use it at your own risk, and I have not verified it on 10.1 or 10.5.

For F1, this is a handy query to find tables that need reorgs:

select     substr(tabschema,1,18) as tabschema, 
        substr(tabname,1,30) as tabname, 
        card, 
        overflow, 
        case when card > 0 
            then decimal(100*(float(overflow)/float(card)),10,2) 
            else -1 
        end as F1 
    from syscat.tables 
    where overflow >0 with ur

I’m not suggesting you make use of the SQL above, but just providing it for information. It gives output that looks like this:

TABSCHEMA          TABNAME                        CARD                 OVERFLOW             F1
------------------ ------------------------------ -------------------- -------------------- ---------------
SYSIBM             SYSTABLES                                     11298                  141            1.24
SYSIBM             SYSCOLUMNS                                    36135                   87            0.24
SYSIBM             SYSINDEXES                                     6404                   22            0.34
SYSIBM             SYSPLANDEP                                     1666                    1            0.06
WSCOMUSR           CLEANCONF                                       163                    1            0.61
SYSTOOLS           HMON_ATM_INFO                                  1841                 1138           61.81
SYSTOOLS           ADMIN_MOVE_TABLE                              18947                   26            0.13

F2

F2 is described in the reorgchk header as:
100 * (Effective Space Utilization of Data Pages) > 70

While that’s a good human description of what reorgchk is looking for, it’s not very technical. This formula tells us how effectively DB2 is making use of space on pages. As a result, it is likely to be flagged for extremely small tables. I eliminate tables with a cardinality of 0 from reorgs, because they are sometimes flagged on this formula, and it wastes time.

I don’t have SQL that works for this formula. I played with it some, and don’t have it. Basically what DB2 does for this one is to calculate how many bytes would be used based on the cardinality and the average row size, and then calculates the size of the table based on the current number of pages, and looks for how close the value based on pages is to the value based on cardinality. Some table geometries will always cause this one to be flagged.

F3

F3 is described in the reogchk header as:
100 * (Required Pages / Total Pages) > 80

Basically DB2 is checking to make sure that the total pages in the table is not more than 20% higher than the required number of pages.

The sql calculation is much easier here, again use this SQL at your own risk:

select  substr(tabschema,1,18) as tabschema, 
        substr(tabname,1,30) as tabname, 
        npages, 
        fpages, 
        case 
            when fpages >0 then decimal(100 * (float(npages)/float(fpages)),5,2) 
            else -1 
        end as F3 
from syscat.tables 
where type='T' 
        and npages >1 
        and fpages >1 
with ur

The output looks like this:

TABSCHEMA          TABNAME                        NPAGES               FPAGES               F3
------------------ ------------------------------ -------------------- -------------------- ---------------
SYSIBM             SYSTABLES                                      1363                 1363          100.00
SYSIBM             SYSCOLUMNS                                     2112                 2114           99.90
SYSIBM             SYSINDEXES                                      658                  660           99.69
SYSIBM             SYSVIEWS                                         57                   57          100.00
SYSIBM             SYSVIEWDEP                                       53                   53          100.00
SYSIBM             SYSPLAN                                          68                   68          100.00
SYSIBM             SYSPLANDEP                                       51                   51          100.00
SYSIBM             SYSSECTION                                       85                   87           97.70

Table Reorg Decisions

One approach for scripters of reorgs is to parse out the table name for any row with a *. That actually works in this table reorg section, but does not in the index section I’ll talk about in a future post. There’s a boon in recent versions for people who want to script this. The REORGCHK_TB_STATS procedure can be used to return this information in a more friendly format for scripting. It is executed like this:

db2 "call REORGCHK_TB_STATS('T','SYSIBM.SYSCOLUMNS')"


  Result set 1
  --------------

  TABLE_SCHEMA                                                                                                                     TABLE_NAME                                                                                                                       DATAPARTITIONNAME                                                                                                                CARD                 OVERFLOW             NPAGES               FPAGES               ACTIVE_BLOCKS        TSIZE                F1          F2          F3          REORG
  -------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- ----------- ----------- ----------- -----
  SYSIBM                                                                                                                           SYSCOLUMNS                                                                                                                                                                                                                                                                       36135                   87                 2112                 2114                   -1              8021970           0          94         100 ---

  1 record(s) selected.

But that’s not all. That alone is just a different format of data to parse through for the values. Running that populates a table called SESSION.TB_STATS that is session specific that you can now query to get particular values like this:

select  substr(table_schema,1,18) as tabschema, 
        substr(table_name,1,30) as tabname, 
        f1, 
        f2, 
        f3, 
        reorg 
    from SESSION.TB_STATS

TABSCHEMA          TABNAME                        F1          F2          F3          REORG
------------------ ------------------------------ ----------- ----------- ----------- -----
SYSIBM             SYSCOLUMNS                               0          94         100 ---

  1 record(s) selected.

I learned that nifty trick over at Phillip Carrington’s blog. I haven’t tried out the rest of his methodology, but thought that tip was good.

Calling REORGCHK_TB_STATS this way should collect data for all tables:
call REORGCHK_TB_STATS('T','ALL')

Selecting Table Reorg Syntax

Any syntax choices at this point are not determined by the reorgchk output but by other properties of the reorg. There are two basic options you have – classic or inplace. The former is also referred to as offline. Inplace can be referred to as online, but it is not fully online (see When is a Reorg Really Online?)

The Knowledge Center has a good page on reorg syntax.

Classic Table Reorgs

The most obvious disadvantage of classic reorgs is that the table is unavailable for significant portions of them. Because of this and the availability requirements of the e-commerce databases I work with, I don’t often do this kind of reorg. You can specify ALLOW READ ACCESS to allow some access to the table while the reorg is occurring. One of the “advantages” of a classic reorg is that you can specify a temporary tablespace for the reorg to use. Be careful, though – the reorg may require a good three times the table size if you specify a temporary tablespace, and should only require around two times the table size if you do not specify one. Classic reorgs can also be used to reorganize LOB data – which you should not be doing on a regular basis, but may have to do to free up space from deleted LOBs or to inline LOBs.

Inplace Table Reorgs

I run inplace table reorgs on a regular basis. They are largely online. They also have the advantage of only moving small amounts of data at time so you don’t have to have two to three times the size of your table in available space. The big disadvantage here is in transaction log space. Inplace reorgs eat transaction log space like nothing else, so be prepared. Usually, the ALLOW WRITE ACCESS keywords are specified for inplace reorgs. While it involves not releasing any empty pages at the end of the table at the end of the reorg, specifying NOTRUNCATE TABLE does make for the most online reorg possible.

Keep in mind that an inplace reorg is an asynchronous operation – the command will return right away, but the reorg will continue to run in the background. This means you can easily tank your server if you improperly script these and reorg every table in your database at the same time. It is critical to control for how many online reorgs are running at once. There are options on the reorg command that will allow you to stop or to pause running inplace reorgs.

Running online reorgs write data to the history file. You can also use SNAPTAB_REORG to query reorg statuses for running and recently completed reorgs. db2pd also provides data. I think that monitoring and querying reorg statuses will have to be a separate blog entry.

My earlier post on table reorgs covered Reorg’s place in a database maintenance cycle and running reorgchk. If you haven’t already read it, check it out: How to Tell When a Table Reorg is Needed

INDEXES

After the section on tables, REORGCHK output has a section with details on each index for each table that REORGCHK looked at.

The index section of reorgchk looks something like this:

Index statistics:

F4: CLUSTERRATIO or normalized CLUSTERFACTOR > 80
F5: 100 * (Space used on leaf pages / Space available on non-empty leaf pages) > MIN(50, (100 - PCTFREE))
F6: (100 - PCTFREE) * (Amount of space available in an index with one less level / Amount of space required for all keys) < 100
F7: 100 * (Number of pseudo-deleted RIDs / Total number of RIDs) < 20
F8: 100 * (Number of pseudo-empty leaf pages / Total number of leaf pages) < 20

SCHEMA.NAME                 INDCARD  LEAF ELEAF LVLS  NDEL    KEYS LEAF_RECSIZE NLEAF_RECSIZE LEAF_PAGE_OVERHEAD NLEAF_PAGE_OVERHEAD  PCT_PAGES_SAVED  F4  F5  F6  F7  F8 REORG
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Table: WSCOMUSR.ATTR
Index: SYSIBM.SQL120212212515070
                                170     1     0    1     0     170            8             8                566                 566                0  98   -   -   0   0 -----
Index: WSCOMUSR.I0001203
                                170     2     0    2     0     170           22            22                516                 516                0  62 147   -   0   0 *----
Index: WSCOMUSR.I0001204
                                170     1     0    1     0       6            4             4                710                 710                0  98   -   -   0   0 -----
Index: WSCOMUSR.I0001464
                                170     1     0    1     0       2            2             2                822                 822                0  96   -   -   0   0 -----
Index: WSCOMUSR.X_ATTR_IX1
                                170     4     0    2     0     170           66            66                236                 236                0  97 110   -   0   0 -----
Index: WSCOMUSR.X_ATTR_IX2
                                170     2     0    2     0     170           30            30                392                 392                0  95 178   -   0   0 -----
Index: WSCOMUSR.X_ATTR_IX3
                                170     2     0    2     0     170           25            25                442                 442                0  62 158   -   0   0 *----
Table: WSCOMUSR.ATTRIBUTE
Index: SYSIBM.SQL120212212515260
                                  0    16    16    2     0       0           12            12                476                 476                0 100 105   -   0 100 ----*
Index: WSCOMUSR.I0000019
                                  0    26    26    2     0       0           16            16                442                 442                0 100 105   -   0 100 ----*
Index: WSCOMUSR.I0000298
                                  0     5     5    2     0       0           11            11                516                 516                0 100 105   -   0 100 ----*
Table: WSCOMUSR.BUSEVENT
Index: SYSIBM.SQL120212212516210
                            1.1e+07 48067  1414    4 8e+05 1.1e+07           12            12                948                 948                0  68  62 266   7   2 *-*--
Index: WSCOMUSR.BUSEVENT_IX1
                            1.1e+07 25184     0    3     0 9048618           10            10               1048                1048                0  89  91   1   0   0 -----
Index: WSCOMUSR.I0001104
                            1.1e+07 10123     0    3    39       1            3             3               1894                1894                0 100  86   4   0   0 -----
Table: WSCOMUSR.DMEXPLOG
Index: SYSIBM.SQL120212212521010
                              34214  4717  4439    4  1406   34214            8             8                566                 566                0  93  59 +++   3  94 ----*
Index: WSCOMUSR.DMEXPLOG_IX1
                              34358  3252  3036    4    25   33629           10            10                516                 516                0 100  83 +++   0  93 ----*
Index: WSCOMUSR.I0001198
                              34292  9708  8957    5  4285   34267           36            36                416                 416                0  76  55 +++  11  92 *---*
Index: WSCOMUSR.I0001199
                              34332  8933  8377    4   163   33960           22            22                354                 354                0  90  50 +++   0  93 -*--*

It's nice to use a wide screen when viewing this output on a server - a 72-character wide window may drive you insane.

Note that the format is very similar to tables. There are numerical details about each index. There are the results of several forumulas - this time F4 through F8, and there is a final column on the right with flags when a reorg MAY be needed.

The output is a bit different in that there are sections for each table that the indexes are grouped in. This always made the output harder for parsing for scripting, though there are better options in more recent versions for scripting. It is not just the line formatting, but in some ways, the decisions to be made are a bit more complicated.

Let's work through the formulas one by one.

F4

Formula F4 is described in the index section header as:
F4: CLUSTERRATIO or normalized CLUSTERFACTOR > 80

This formula or flag is looking at how well clustered the table is over a particular index. The values of CLUSTERRATIO or CLUSTERFACTOR are pulled directly from the SYSCAT.INDEXES system catalog view. IF we want the table clustered over this particular index, the way to get rid of this flag is to reorganize the table ON the index. Think about this one. In many cases it is impossible to have a table clustered over all of the indexes at once. We cannot blindly reorg based on this flag. If we go that direction, we keep reorganizing the table over and over again on one index after another.

The important thing with F4 is to decide, on a table by table basis, whether we want the data clustered over an index. If we do, and if there is some reason you cannot or do not want to define that index as a clustering index, then and only then you should reorg the table on that index if the F4 formula is flagged. Reorging a table on an index can increase your reorg time.

F4 is the only flag in the index section that has the possibility to prompt a table reorg. The other formulas will indicate various kinds of index reorgs.

Index Reorgs

Unless you are using table partitioning, you may only reorg all indexes for a table. You may not reorg a specific index for a table without reorging the remaining indexes. There are multiple types of index reorgs that can be done. Index reorgs are kind of online (see the end of this article for how online), and do use resources. If you are lucky enough to do offline, classic table reorgs, indexes are rebuilt as a part of them. They are not rebuilt as a part of online table reorgs. If you are only doing online table reorgs, you will want to make sure you're also doing the appropriate index reorgs. I will talk about each of the remaining formulas and then revisit what kinds of reorgs are suggested by each flag.

F5

Formula F5 is described in the index section header as:
F5: 100 * (Space used on leaf pages / Space available on non-empty leaf pages) > MIN(50, (100 - PCTFREE))

This formula is trying to calculate how much of the space allocated to non-empty leaf pages is used, and checking to see if that is greater than either 50% or the space we expect to be used on pages given the value of PCTFREE for this index. If you need a reminder on what leaf pages are, see DB2 Basics: What is an index?

PCTFREE is a value that is set for each index and is respected in certain situations like reorg and load. It tells DB2 to leave a certain portion of each index leaf page open so that index keys can be inserted in proper place in the index without having to allocate new pages. It is set to 10% by default.

If we have a high percentage of space open on our non-empty leaf pages, it may indicate that our indexes can benefit from some consolidation and cleanup.

The following should work to calculate F5, if nleaf is greater than one and ((nleaf-num_empty_leafs-1) * (PAGESIZE-96)) does not return zero. Remember that any calculations provided are for reference and understanding only, and they are not guaranteed to be correct. The IBM Knowledge Center page on REORGCHK describes the calculations in detail, but does not provide SQL for performing them.

select (100 * ((fullkeycard * ((select SUM(CASE WHEN A.typename = 'VARCHAR' THEN A.AVGCOLLEN - 2 ELSE A.AVGCOLLEN END + CASE WHEN NULLS = 'Y' THEN 0 ELSE 0 END ) + 0 FROM SYSCAT.COLUMNS A, syscat.indexcoluse B WHERE A.colname=B.colname and B.indschema=i.indschema and B.indname=i.indname and A.tabschema=i.tabschema and A.tabname=i.tabname)+9)) + (card-fullkeycard) * 5) / ((nleaf-num_empty_leafs-1) * (PAGESIZE-96)))
from syscat.indexes i
        join syscat.tables t    on i.tabschema=t.tabschema
                                and i.tabname=t.tabname
        join syscat.tablespaces ts      on i.tbspaceid = ts.TBSPACEID
where i.indschema = 'WSCOMUSR' and i.indname='IPF00008' with ur

F6

Formula F6 is described in the index section header as:
F6: (100 - PCTFREE) * (Amount of space available in an index with one less level / Amount of space required for all keys) < 100

What this formula does is checks to see if an index with one less level would have enough space for all data, respecting PCTFREE. Usually that's a significant change, so this formula I don't find flagged as often as some of the others. It's another that is hard to get a query working to calculate, so I don't have that available. Check out the IBM Knowledge Center page on REORGCHK, and if you have SQL that works for it, post in the comments below so that others can enjoy your SQL prowess!

F7

Formula F7 is described in the index section header as:
100 * (Number of pseudo-deleted RIDs / Total number of RIDs) < 20

With type 2 indexes, index keys are essentially marked as deleted rather than actually deleted. These are called pseudo-deleted RIDs. F7 checks to make sure that less than 20% of the RIDS in the table are marked for deletion.

Here is the query for calculating this one - it's pretty easy:

select  substr(indname,1,30) as indname, 
        decimal(100*(FLOAT(NUMRIDS_DELETED)/FLOAT(NUMRIDS_DELETED+INDCARD)),10,5) as F7 
    from syscat.indexes 
    where tabschema='WSCOMUSR' 
        and tabname='ATTRVALDESC' 
    with ur

INDNAME                        F7
------------------------------ ------------
SQL120212212515340                  0.06914
I_ATTRVALDESC01                     0.06255
X_ATTRVALDESC_IX1                   0.18416
I0001468                            0.03293
I0001470                            0.03293

  5 record(s) selected.

This one is likely to be flagged if a lot of data is deleted from the table.

F8

Formula F8 is described in the index section header as:
F8: 100 * (Number of pseudo-empty leaf pages / Total number of leaf pages) < 20

This one is looking for entire pages where all of the RIDs have been marked as deleted. It checks to make sure that no more than 20% of the pages in the index are fully marked as deleted. Again, writing the SQL to calculate this yourself is not hard:

select  substr(indname,1,30) as indname, 
        decimal(100*(FLOAT(NUM_EMPTY_LEAFS)/FLOAT(NLEAF)),10,5) 
    from syscat.indexes 
    where tabschema='WSCOMUSR' 
        and tabname='CATENTDESC' 
    with ur

INDNAME                        F8
------------------------------ ------------
SQL120212212517170                  0.00000
I0000304                            0.00000
I0001196                            7.88177
I0001248                            0.00000
IPF00002                            9.30232

  5 record(s) selected.

Index Reorg Decisions

The cheat of just looking for lines in reorgchk with a * simply does not work for indexes. There are too many complicated decisions to be made. Luckily for scripting, we have similar constructs as we did for table reorgs.

Start with REORGCHK_IX_STATS like this:

$ db2 "call REORGCHK_IX_STATS('T','SYSIBM.SYSCOLUMNS')"


  Result set 1
  --------------

  TABLE_SCHEMA                                                                                                                     TABLE_NAME                                                                                                                       INDEX_SCHEMA                                                                                                                     INDEX_NAME                                                                                                                       DATAPARTITIONNAME                                                                                                                INDCARD              NLEAF                NUM_EMPTY_LEAFS      NLEVELS     NUMRIDS_DELETED      FULLKEYCARD          LEAF_RECSIZE         NONLEAF_RECSIZE      LEAF_PAGE_OVERHEAD   NONLEAF_PAGE_OVERHEAD PCT_PAGES_SAVED F4          F5          F6          F7          F8          REORG
  -------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------- -------------------- -------------------- -------------------- ----------- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------- --------------- ----------- ----------- ----------- ----------- ----------- -----
  SYSIBM                                                                                                                           SYSCOLUMNS                                                                                                                       SYSIBM                                                                                                                           INDCOLUMNS01                                                                                                                                                                                                                                                                     18841                  289                    0           3                    0                18841                   40                   40                  710                   710               0          95          90          19           0           0 -----
  SYSIBM                                                                                                                           SYSCOLUMNS                                                                                                                       SYSIBM                                                                                                                           INDCOLUMNS02                                                                                                                                                                                                                                                                     18841                   32                    0           2                    0                   14                   18                   18                  822                   822               0          79          93          -1           0           0 *----
  SYSIBM                                                                                                                           SYSCOLUMNS                                                                                                                       SYSIBM                                                                                                                           INDCOLUMNS03                                                                                                                                                                                                                                                                     18841                   32                    0           2                    0                    1                    4                    4                  822                   822               0         100          92          -1           0           0 -----

  3 record(s) selected.

  Return Status = 0

Again this first part returns good data, but not in an easily scriptable format. But the real power is in the population of the SESSION.IX_STATS table, which can be queried for each element like this:

select  substr(table_schema,1,18) as tabschema,
        substr(table_name,1,30) as tabname,
    substr(index_schema,1,18) as indschema,
    substr(index_name,1,30) as indname,
        f4,
        f5,
        f6,
    f7,
    f8,
        reorg
    from SESSION.IX_STATS;

TABSCHEMA          TABNAME                        INDSCHEMA          INDNAME                        F4          F5          F6          F7          F8          REORG
------------------ ------------------------------ ------------------ ------------------------------ ----------- ----------- ----------- ----------- ----------- -----
SYSIBM             SYSCOLUMNS                     SYSIBM             INDCOLUMNS01                            95          90          19           0           0 -----
SYSIBM             SYSCOLUMNS                     SYSIBM             INDCOLUMNS02                            79          93          -1           0           0 *----
SYSIBM             SYSCOLUMNS                     SYSIBM             INDCOLUMNS03                           100          92          -1           0           0 -----

  3 record(s) selected.

Now you should have the data you need in the format you need it to make decisions. These recommendations come straight from the IBM Knowledge Center page on REORGCHK - a very useful page.

• If the results of the calculations for Formula 1, 2 and 3 do not exceed the bounds set by the formula and the results of the calculations for Formula 4, 5 or 6 do exceed the bounds set, then index reorganization is recommended.
• If only the results of the calculations Formula 7 exceed the bounds set, but the results of Formula 1, 2, 3, 4, 5 and 6 are within the set bounds, then cleanup of the indexes using the CLEANUP option of index reorganization is recommended.
• If the only calculation result to exceed the set bounds is the that of Formula 8, then a cleanup of the pseudo empty pages of the indexes using the CLEANUP PAGES option of index reorganization is recommended.

Your reorg command for indexes will always start with REORG INDEXES ALL FOR TABLE and include the table name and one of the syntax options outlined above. There aren't online or offline choices or other options for indexes. Unless you're using the CLEANUP ONLY or CLEANUP ONLY PAGES options, what DB2 is doing is building new indexes side-by-side with the old it indexes. It will then acquire a Z lock on the whole table to make the switch. The switch may be fairly quick. But that's also a super-exclusive lock, which blocks all table access, including shared and read-only locks. I believe this even includes application using the UR or Uncommitted Read isolation level.

Share your own thoughts and experiences with index reorgs!

It has been four years since I started this blog, and two and a half years since I committed to blogging very regularly.

Wow, what a ride it has been. Blogging has skyrocketed my career. I’m now not just an IBM Champion, but also an IBM Gold Consultant. I have spoken at two conferences, written two developerWorks articles, and am planning for more. I have some idea of how much there is that I don’t know, and how many people know more than I do, so I hesitate to say that I’ve reached my career goal of becoming an internationally recognized DB2 Guru. But what I do know, I do my best to share. I’m starting to get to the point where I have opportunities to help other up and coming speakers and writers, and what fun that is!

I don’t blog for these benefits, though. They’re great, but the real reason I blog is because when I search for things online, I don’t always find the answers. I have some answers to share, and can prevent that from happening to others, at least on some topics. I also search my own blog for something at least once a week. I don’t remember all the details of things I’ve written or done, so it’s great for remembering the details and syntax things that I haven’t done in a while. I get my own blog quite frequently when searching for answers online, and most of the time I’m annoyed because it’s the stuff I already know. Sometimes it puts a smile on my face because I know if Google shows it to me, it’s showing it to others. Finally, a couple of times a year, I’m surprised to find that I’ve already solved the problem and forgotten so completely about it and am surprised to find the article.

I get a warm glow with every comment, every email, every person I meet who tells me they read. It’s not as surprising as it used to be, but it is so good to know that I’m really helping people. Even the comments that disagree with me or correct me – a good disagreement is great for learning and for others to see how many perspectives there are in the DB2 community.

I’ve been blogging for 4 years, 2.5 of those with weekly posts, and just in the past two weeks have had 3 days finally above 1,000 page views per day. I recently saw a sustained 20% jump in page views, and suspect that it’s due to a change in Google’s algorithm, since well over 60% of my traffic comes from search engines and very heavily from Google.

Writing a blog is hard work. It’s especially hard to keep up with the consistency that will really build significant readers. Finding Guest Bloggers is helping me in this area. Mike Krafick is a regular guest blogger who is great at helping me with content in a pinch, and is just full of ideas for social media and online presence.

This post is not all navel gazing. I want to encourage the budding bloggers out there to find a way to consistently blog. It helps you advance in your career, for sure. It helps you write more clearly and correctly. It helps you retain technical knowledge. It helps you gain technical knowledge as you read and research and fact-check yourself and try experiments. It helps other people. Even if you write about topics that others have written on, it still adds another perspective and good details.

Here’s to another 4 years!

A story of fail and recover.

Problem Discovery and Description

Sometimes, I think that I subconsciously knew that something was wrong. I woke up before 5 AM and couldn’t get back to sleep for no real reason that I could figure out. I gave up on sleep around 5:15 and went to take a shower. On the way to the shower, I read work email and found this:

At the same time, I received a note from the project manager indicating that the hosting provider had apparently rebooted all of our “PROD” and “UAT” database servers in the night in an attempt to uninstall HACMP.

This is a newer hosting provider for us, chosen by our client without much input from us. The databases in question are not yet supporting a “live” site – only being configured and developed on by developers from several companies. With tight timelines and global companies, they’re used most of the time.

We had discovered two weeks earlier that the hosting provider was not taking any system level backups because the systems were not yet considered live. This was a surprise to us, as we are used to hosting providers providing basic services like backup from the moment servers are provided. We had brought it up as a risk and a problem with the hosting provider.

Anyway, I went straight to my home office, and logged on to the server in question. The first issue I saw was that the many-line /etc/services file had been replaced by a simple two-line file that indeed did not list any database lines at all. That’s an easy fix on it’s own, and the SA and I quickly worked to get the required lines back in place.

Did I mention that this happened on Friday the 13th?

Once /etc/services was corrected, db2 still would not start, this time with:

db2start
06/16/2014 16:25:13     0   0   SQL1042C  An unexpected system error occurred.
SQL1032N  No start database manager command was issued.  SQLSTATE=57019

An error message that strikes fear into the heart of any DB2 DBA.

I found that I could get the db2 instance up on the primary server in each TSA/HADR cluster by doing the following. I don’t pretend to understand why, but doing parts of it in various combinations did not work.

1. db2iupdt
2. installSAM – which failed because it found TSA was already installed
3. uninstallSAM – which failed because it thought TSA was still active
4. installSAM – which failed because it found TSA was already installed
5. db2iupdt

Once I had the db2 instance up, I could not use db2haicu -delete. My lssam on the primary at this point looked like this:

This was ugly. But it actually wasn’t as scary as what I got on the standby. I couldn’t get the standby DB2 instance to start with any combination of commands, and it similarly would not let me install or uninstall TSAMP. This is what I got from an lssam there:

On the standby, any attempt at db2haicu failed because the DB2 instance was down.

Clearly, the hosting provider uninstalling HACMP had uninstalled some files that TSAMP uses, and had also altered /etc/services and wiped out most of the entries there. Because the hosting provider had not been taking system backups, there was no way to restore the system and apparently no rollback plan. Reportedly, the hosting provider was following a series of steps provided by IBM. To exacerbate the problem, the hosting provider performed these steps on 12 database servers in 6 HADR/TSAMP clusters at the same time.

Resolving the problem

After opening a PMR with DB2 support (and getting DB2 support to consult someone with TSA expertise), our system admin was able to get TSAMP to a point where I could uninstall it successfully. I uninstalled it and re-installed it. I was then still unable to start the DB2 instances. I got the same error:

db2start
06/16/2014 16:25:13     0   0   SQL1042C  An unexpected system error occurred.
SQL1032N  No start database manager command was issued.  SQLSTATE=57019

And in the db2diag.log, I found this:

2014-06-16-16.25.13.072800+000 E1823213A907         LEVEL: Error
PID     : 40960086             TID : 1              PROC : db2star2
INSTANCE: db2inst2             NODE : 000
HOSTNAME: redacted
EDUID   : 1
FUNCTION: DB2 UDB, high avail services, sqlhaGetObjectState2, probe:400
MESSAGE : ECF=0x90000552=-1879046830=ECF_SQLHA_OBJECT_DOES_NOT_EXIST
          Cluster object does not exist
DATA #1 : String, 35 bytes
Error during vendor call invocation
DATA #2 : unsigned integer, 4 bytes
29
DATA #3 : String, 28 bytes
db2_db2inst2_redacted02s_0-rs
DATA #4 : signed integer, 4 bytes
4
DATA #5 : unsigned integer, 4 bytes
1
DATA #6 : String, 0 bytes
Object not dumped: Address: 0x000000011018C324 Size: 0 Reason: Zero-length data
DATA #7 : unsigned integer, 8 bytes
1
DATA #8 : signed integer, 4 bytes
0
DATA #9 : String, 0 bytes
Object not dumped: Address: 0x000000011018B11C Size: 0 Reason: Zero-length data

2014-06-16-16.25.13.073839+000 E1824121A586         LEVEL: Error
PID     : 40960086             TID : 1              PROC : db2star2
INSTANCE: db2inst2             NODE : 000
HOSTNAME: redacted_02s
EDUID   : 1
FUNCTION: <0>, <0>, <0>, probe:1164
RETCODE : ECF=0x90000552=-1879046830=ECF_SQLHA_OBJECT_DOES_NOT_EXIST
          Cluster object does not exist
DATA #1 : String, 63 bytes
libsqlha: sqlhaGetObjectState() call error from wrapper library
DATA #2 : String, 0 bytes
Object not dumped: Address: 0x000000011018B11C Size: 0 Reason: Zero-length data
DATA #3 : signed integer, 4 bytes
0

2014-06-16-16.25.13.097492+000 E1824708A387         LEVEL: Error
PID     : 40960086             TID : 1              PROC : db2star2
INSTANCE: db2inst2             NODE : 000
HOSTNAME: redacted02s
EDUID   : 1
FUNCTION: DB2 UDB, high avail services, sqlhaSetStartPreconditions, probe:18246
RETCODE : ECF=0x90000557=-1879046825=ECF_SQLHA_CLUSTER_ERROR
          Error reported from Cluster

2014-06-16-16.25.13.097733+000 E1825096A465         LEVEL: Severe
PID     : 40960086             TID : 1              PROC : db2star2
INSTANCE: db2inst2             NODE : 000
HOSTNAME: redacted02s
EDUID   : 1
FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:5104
MESSAGE : ZRC=0x827300D4=-2106392364=HA_ZRC_CLUSTER_ERROR
          "Error reported from Cluster"
DATA #1 : String, 66 bytes
An error was encountered when interacting with the cluster manager

I could not get db2haicu -delete to work at any point. Apparently in 10.1/10.5 the CLUSTER_MGR DBM cfg parameter became informational and can only be set through db2haicu. This meant that in order to un-set it so I could get back to a point where I could reconfigure TSAMP, I had to drop and re-create every DB2 instance. There were 14 of them.

I made sure I had database backups before undertaking this. I then did the following:

1. db2cfexp backup.db2cfexp backup
2. db2 get dbm cfg |tee dbmcfg.out
3. db2 list db directory |tee dbdir.out
4. db2set -all |tee db2set.out
5. db2 list node directory
6. switched to root and did a db2idrop
7. re-created the instance with a db2icrt
8. db2cfimp backup.db2cfexp
9. Set the parameters not covered by cfexp. In my case, this included:
   1. all DFT_MON parameters
   2. SVCENAME
   3. SYSMON group
10. I then compared the dbm cfg and db2set from before and after to make sure everything was fine

The db2cfimp re-cataloged the database for me, meaning I did not have to restore it.

After I had all of the DB2 instances re-created and started, I was then able to fully re-do the TSAMP configuration on all 6 HADR/TSAMP clusters. This is where I was so grateful I had taken complete documentation when I originally set up the clusters. I had Word documents with all the info I needed to re-do each and every cluster.

At some point, DB2 support referred to the approach I was taking as a Sledgehammer approach. It may well have been, but having software partially uninstalled is a scary thing to me. How do I know I’m not missing some critical files somewhere? Also when I asked, DB2 support confirmed that CLUSTER_MGR is not configurable other than through db2haicu, and that dropping and recreating the db2 instances was my only option at that point. I call on IBM to make the CLUSTER_MGR parameter configurable again! I could have saved several hours of work by not having to do that part, at least.

I thought I would share this issue in the hopes that it helps someone else. I don’t claim that the actions above are the best – if you’re in a similar scenario, please consult IBM support to get what you need to recover. With luck, others are taking system level backups or have hosting providers that have a rollback plan for every system change, so no one will ever encounter this but me.

I don’t know about you, but I’m pretty addicted to db2top. I started using it way back when it was a download from Alphaworks. It is simply unparalleled for free real-time monitoring for DB2. No, it’s not full enough of a feature set to be your only monitoring tool, but I can barely troubleshoot a locking issue any more without it. I tried last night, and it took me four times as long, at least.

Why did I even try to troubleshoot an occurring issue without db2top? Well, I have these nifty new AIX servers that were allocated by someone in another country. And every time I try db2top on them, I got something that looked like this:

I can see a few parts that look like db2top there, but nothing readable, and it doesn’t get better if I change the screen size or if I hit a key to get to a different type of screen.

Searching the web brought me to this APAR:
http://www-01.ibm.com/support/docview.wss?uid=swg1IC73092

That is for 9.5. I am most certainly not using 9.5. These servers are 10.5, Fixpack 3. I had the same exact problem on 14 servers and 16 instances. My LANG variable was already correct. The ultimate resolution was to add these lines to my instance owner’s .profile:

LC_COLLATE="C"
LC_CTYPE="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=

export LC_COLLATE
export LC_CTYPE
export LC_MONETARY
export LC_NUMERIC
export LC_TIME
export LC_MESSAGES
export LC_ALL

This was really a revelation for me when I took DanL‘s SQL workshop before the IDUG conference in Phoenix.

I started out as a physical or systems DBA at IBM, and until studying for certification, I hardly wrote a statement more complicated than select * from syscat.bufferpools. After being a DBA for probably 5 years or more I started to understand a bit more about SQL, but was very strictly limited by my box at IBM. I was not supposed to be writing SQL – that was someone else’s job. But I could look for indexes to help the SQL that ran against my databases. In the last 6 years, my role has involved more development support – helping developers write and tune SQL.

Sometimes the DB2 optimizer makes re-writing SQL irrelevant. But other times, rewriting SQL can get you performance improvements. This article focuses tightly on one kind of re-write you can try. According to DanL, you can rewrite a join as a non-correlated subquery. You can also rewrite it as a correlated subquery. All three methods generally work for the same results. I had never thought of it as quite that fluid – I thought each had their places. Let’s work through each of these to understand what it means, using a real-world example. I’ll use DB2 LUW system catalog tables on a 10.5 system.

The question this query is trying to answer is “What tablespaces do I have that have tables in them with LOB columns?”

Join

A join is simply a query with an explicit or implicit join in the query. The from clause lists two or more tables. One or more join predicates are specified either explicitly in join clauses within the from clause, or implicitly within the where clause. Explicit join syntax is considered best practice for clarity. There is no subquery.

This is probably the way I would have naturally written the query for this particular item. Here’s what this query looks like using join methodology:

select distinct( substr(tbspace,1,30) ) as tbspace
from syscat.tables tab join syscat.columns col on tab.tabschema=col.tabschema and tab.tabname=col.tabname
where col.typename in ('BLOB','CLOB')
        and tab.type='T'
with ur;

TBSPACE
------------------------------
DBA32K
LOB_32K
REF_TAB16K
REF_TAB4K
SYSCATSPACE
SYSTOOLSPACE
TAB16K
TAB4K
TAB8K
USERSPACE1

  10 record(s) selected.

Non-Correlated Subquery

A subquery is a query within parentheses that is within another select statement. It can be in the select clause, the from clause, or the where clause. A non-correlated subquery means that the inner query does not have in its where clause any references to the outer query.

This one came the least naturally to me of the three methods, and took me a bit to work out.

select  distinct( substr(tbspace,1,30) ) as tbspace
from syscat.tables tab
where (tab.tabschema, tab.tabname) in (select tabschema, tabname from syscat.columns where typename in ('BLOB','CLOB'))
        and tab.type='T'
with ur;

TBSPACE
------------------------------
DBA32K
LOB_32K
REF_TAB16K
REF_TAB4K
SYSCATSPACE
SYSTOOLSPACE
TAB16K
TAB4K
TAB8K
USERSPACE1

  10 record(s) selected.

Correlated Subquery

A correlated subquery is a subquery that includes in its where clause references to one or more items from the outer query.

This one made a medium amount of sense to me – I could see myself writing the query this way.

select distinct( substr(tbspace,1,30) ) as tbspace
from syscat.tables tab
where exists (select 1 from syscat.columns col where tab.tabschema=col.tabschema and tab.tabname = col.tabname and typename in ('BLOB','CLOB'))
        and tab.type='T'
with ur;

TBSPACE
------------------------------
DBA32K
LOB_32K
REF_TAB16K
REF_TAB4K
SYSCATSPACE
SYSTOOLSPACE
TAB16K
TAB4K
TAB8K
USERSPACE1

  10 record(s) selected.

Which Performs Better?

There is no blanket answer to this. In the case of this specific query, I explained all three versions, and DB2 did the access plans all exactly identical, so there was no difference. This is a very simple example, however, and it could easily make a difference in some situations. It really is best to test a query all three ways to see if one works better. Testing involves minimally writing the query three ways and then running db2exfmt and db2advis on it each way and comparing. It may also involve db2caem to get actuals for each method and even db2batch to test the actual timing of the query each way.