Exploring Business Intelligence

Tuning: Min and Max in same query

Please read this article.

Karen explains why having min and max in the same query impacts performance and how to tackle the same.

Decfloat datatype not supported in BODI / BODS

Recently I was working with a few DB2 UDB 9.5 tables and discovered that decfloat datatype columns were not recognized even though I had checked the option to recognize unsupported datatypes as varchar.

Read the rest of this entry »

Why surrogate key?

Why should we use surrogate keys in a data warehouse?

Here is why.

1. To separate the DWH from the operational environment
Think of this. You have a dimension loading from transactional tables. All of a sudden, the OLTP people decide to re-use all closed / inactive account id’s. For them, it wont matter a bit. For the DWH team, it will.
Read the rest of this entry »

Get BODI job schedule info from repository

Here is a query that you can run on the BODI repository to fetch the job schedule details.

Select upper (al_lang.NAME) as jobname,
upper (al_sched_info.sched_name) as schedule_name,
al_sched_info.start_time as start_time,
al_sched_info.host_name host_server
from di_edw.al_lang al_lang full outer join di_edw.al_sched_info al_sched_info on al_lang.guid = al_sched_info.job_guid
where active = 1
and al_lang.object_type = 0
and TYPE = 0
and al_lang.object_key =
(SELECT MAX (object_key)
FROM di_edw.al_lang l
WHERE l.NAME = al_lang.NAME AND l.object_type = 0 AND l.TYPE = 0)
ORDER BY 1, 2

Handling many to many relationships (corrections made)

(There were issues with the design in my original post. Those have been corrected. I have let the errors stay and have highlighted where I have made corrections so that you can understand better)

Consider this situation.

2 new guys Ravi and Raj go to our bank and open new savings accounts 101 and 102. And Ravi deposits 100 and Raj deposits 144. This can be represented by the following.

CLICK ON THE IMAGES BELOW TO SEE A LARGER PICTURE

Now suddenly, Ramu and Ramya come to the bank and open a joint account 103. And they deposit 1000. How will you represent this? What will you enter for CUST_KEY ? Will you enter 3 or 4? It was the same transaction of Rs 1000. But which customer to indicate? What will you fill in the red ? position below?

Read the rest of this entry »

Migration of a DW solution from Oracle to DB2

Recently we migrated an entire DW module from Oracle to DB2. And our ETL was done using Business Objects Data Integrator.

Below is a sequence of steps that we followed to migrate the entire solution (including data structures, ETL code and universe)

Assumption:
–>All estimates done for 4 DB2 databases, 3 federated systems, 62 tables, 33 indexes, 6 DI jobs, 110 data flows, 151 transformations, 16 SQL transforms, 34 SQL() scripts, 220 objects in 23 classes and 2 reports.
–>Add 20% time as buffer for unexpected issues.
–>Some tasks can be executed in parallel, while some have strong dependency on completion of previous tasks.

Work breakdown of migration and rough estimation:

1. Initiation & planning : 40 Hrs
Establish objectives and goals
Requirements gathering
Establish scope
Plan resources
Identify development environment

2. Assessment : 40 Hrs
Assess tool and technology needs
Assess and understand technical requirements
Establish technical guidelines
Finalize tools and development environment

Read the rest of this entry »

NULL and NOT IN (Oracle)

Here is a very interesting scenario.

I have an employee table called EMP.

And I have a DEPT table.

Now, I write a query to find out all the employees who are part of any department. (DEPT_ID is properly updated and found in DEPT table)

Now, I want to know if any employee is not part of any department.

What??? No results? I know that employee with EMP_ID 9 doesn’t belong to any department. Where did he go?

Is Oracle drunk? Is it giving me wrong results? Wait let me check with another query.

Read the rest of this entry »

BO Data Services XI 3.0

BO Data Services is a new offering from Business Objects.

Well, it looks like a new bundling and not a new product by itself.

BO Data quality + BO Data Integrator = BO Data Services.

I tried to gather some information regarding this. Here is what I have found so far.

Read the rest of this entry »

Times.. they are changing.

Looks like traditional data warehousing will soon be taken over by a sea of changes.

(Click on underlined words to open the links)

Integration of OLTP with DW? Read about it here.

Here is a discussion on new approaches in DW.

If you want to know Werner’s opinion on the future of Business Objects Data Integrator (BODI), see here.

Appears like BO Data quality + BO Data Integrator = BO Data Services. A discussion on it here.

And is Informatica the next acquisition target? Some say maybe… some say no.

Yahoo pipes vs ETL

If you are an ETL developer, you would be handling sources like Oracle, DB2, flat files, etc.

What if your source was a website? Or a Yahoo search result? And what if you could play around with multiple websites, blogs, flickr, custom searches… mash it up together, edit and transform it to create your own version of what you want to read.

Welcome to Yahoo Pipes!

Read the rest of this entry »