• Introduction to SAP CRM - part 1
• Introduction to SAP CRM - part 2
• SAP CRM details

What do you think is the future of BODI / BODS?

I have almost stopped working on BODS. Well, almost… except for a fix here or there… once in a while. Most of my focus is now on SAP BI.  

But I have a soft side towards BODI/BODS. It is a tool that I know very well. It is also a tool that has a lot of potential, but is underestimated a lot. It has undergone a sea of changes since the ACTA days. And it equates well with the Informatica’s and IBM Information Server’s of the world.

Personally, I feel that this tool has a good future. It has shaped up pretty well, has added more functionality, and integrates well with SAP R/3, but more importantly is non-SAP in focus. It should do well in the next few years.

Okay, here is an interesting discussion on the same. And Werner’s comments are promising. Need to keep a tab on the developments.

Typically, it holds the name of the Datasource, its type (attribute, text or hierarchy), the extract method, extract structure, etc.

Instead of me writing about something that I am not good at (yet :-)..), let me point you to a link that explains ROOSOURCE. Please see the explanation here.

I am not sure if these notes will help you any. This is mainly for my own use. In the process, if it helps you, I will be happy.

And also, I would be happy to receive your comments on it too.

Morning Day 1

1. BW history ( 2.1 -> 3.0 -> 3.1 -> 3.5 -> BI 7.0 and 7.1 now)
2. Mainly deployed where R/3 is used
3. In R/3 database tables are of 2 types : Master and transaction
4. Infoobjects - characteristic IO is for master data
5. Infoobjects - Key figure IO is for trans data
6. RSA 1 - code for Data warehousing workbench
7. RSA6 - to check where DS is created
8. RSA3 - extract checker

Noon Day 1

1. Architecture overview
2. PSA is a temp storage area
3. Loading to PSA is mandatory in 7.0
4. BEx is for excel based reporting
5. Web Application Designer - WAD is for web based reporting
6. Mobile Intelligence is also used but not much
7. IF R/3 is customized, SAP BI 7.0 also would need customization
8. SAP provided content is called business content
9. Activate BI content, to create definitions in the database
10. Transaction data is stored in infocubes and each field is known as infoobject
11. Key figure - facts
12. Characterstics - dimensional attributes
13. InfoArea is a folder where infoobject or infocube is stored
14. Infocatalog - stores characteristics or key figures
15. Steps
    1. Create info area
    2. Create info catalog (contains chars and key figures IOs)
    3. Create IO - chars and key figures
16. Time and unit are mandatory dimensions

Morning Day 2

1. To track time, data is stored in T tables. Else in P tables
2. Master data is NOT in the dimension table
3. SID links characteristic to the dimension table
4. Modelling techniques
   1. ERM
   2. MDM - multi dim model
5. By default 3 dimensions are created in SAP BI - Time, unit and data package
6. Maximum 16 dimensions including 3 above. (net it is 13)
7. Max 248 chars in one dimension

Noon Day 2

1. P is only attributes except name
2. H table is for hierarchy
3. T table is for text
4. Mapping is done in a transformation
5. Infopackage - pulls data from a file and into PSA
6. DTP - extract from PSA, transform and then load to Infoobjects and Infoproviders
7. Flat file support is only for CSV & ASCII
8. Steps in loading a cube
   1. Create info area
   2. Create info catalog
   3. Create info object - characters
   4. Create info object - key figures
   5. Activate them all
   6. Create the Datasource
   7. Specify source and edit / copy the fields (Ctrl Y for copy)
   8. Create infopackage to load data into PSA
   9. Go to info provider and select “Insert Characteristic as infoprovider”
   10. Choose the characteristic given in the Infoobject
   11. Create transformation
   12. Create DTP
   13. Execute Infopackage and DTP job requests

   

The flow of data into Infocube

Morning Day 3

1. R/3 datasource has 2 structures - extract and transfer
2. Extract structure - need to specify the source. It will show all the fields in the source
3. Transfer structure -
4. Length of R/3 sources - 18 minimum
5. RSA6 - to check where DS is created
6. RSA3 - extract checker
7. Datasources of 3.X are different from 7.0
8. Steps loading from R/3
   1. Create infoobject
   2. In Info area, insert it as char
   3. Use RSO2 to create DS
   4. Select trans, master or text option and give it a name (like DS_MAT_ATR)
   5. Select the application component (R/3 -> MM -> MMIO)
   6. Enter short, medium and long description
   7. Enter table name (like MARA)
   8. Click on save
   9. From the window that pops up, select local object
   10. Select the needed fields and hide unwanted and click on save
   11. Datasource is created
   12. Go to main window and hit RSA1
   13. Go to Datasources tab and under right infoarea (like Material Management MM-IO), right click and select replicate metadata
   14. Select as Datasource (and not 3.x option)
   15. Select background
   16. You will get the DS here
   17. Make changes to the DS if needed and activate
   18. Create infopackage and make necessary changes
   19. Start job

Noon Day 3

1. DTP is available in 7.0 onwards only.
2. Update rules is the name for transformations in 3.5
3. RSA5- to see SAP predefined datasources from BUsiness content
4. In standard installation, first go to RSA1 - > BI Content, select cube and say In data flow before and then hit RSA5 and activate all required datasources
5. DSO - data store object
6. ODS in 3.X is now known as DSO

Morning Day 4

1. ODS types - standard and transactional
2. DSO types - standard, write optimized and direct update
3. DSO supports delta. Cube has no delta support.
4. DSO stores transactional whereas cube stores summarized
5. DSO is for consolidation & formalization of data
6. You can report using the DSO too
7. DSO has overwrite funtionality. Cube has only summarized.
8. Full upload is taken from Active data
9. Delta upload is taken from change log
10. Active data, change log, activation queue - see manual page 268
11. X value in the change log means before image. Blank means after image

Noon day 4

1. Loading data to Infocube using DSO
   1. Create datasource
   2. Import the structure from flat file and copy the template and activate
   3. Create the infopackage to load data to PSA
   4. Go to infoprovider
   5. Assuming that the infoarea and infocatalog for char and key are already done
   6. Create DSO
   7. Drag and drop key and data fields from the infoobject catalogs and activate
   8. Create the transformation with the target as DSO and source as the Datasource (of flat file)
   9. Create DTP
   10. Create infocube
   11. Create dimensions
   12. Drag and drop characteristics from infoobject catalog
   13. Drag and drop key figures from IO catalog
   14. Activate
   15. Create transformation, this time with target as Infocube and source as DSO
   16. Create DTP
   17. Execute infopackage job and check if flat file data is loaded to PSA
   18. Execute DTP to load DSO
   19. Right click on DSO and select manage
       
       
   20. You will see the DTP job there
   21. Click on contents and see if new data is there
   22. Go back to requests, select the request and activate it
   23. A new window will appear. Select the request and start the job
   24. Now check the data in the content tab in active data
   25. Execute DTP for infocube
   26. Right click on infocube and select Display data.
   27. Check columns needed and if needed, click on Use DB aggregation
   28. See if data is loaded.
2. Now for the delta load to Infocube
   1. Repeat all steps above
   2. Now go and change key figure data in the flat file
   3. Execute Infopackage for loading PSA
   4. Execute DTP to load DSO
   5. Right click on DSO and select manage
   6. Activate the new request
   7. Check the changed log
   8. Execute DTP to load infocube
   9. Check the data in infocube and see if incremental is done
      
      

Day 4 noon

1. Virtualprovider overview
2. Virtualprovider creation steps
   1. Right click on Infoarea and select create Virtualprovider
   2. Give the name and if you want to copy details from a cube, specify it. Activate it
   3. Go to datasources and create a datasource
   4. In the extraction tab, say Direct access allowed
   5. Specify the flat file and copy template and activate it
   6. Go to Virtual provider and create transformation and DTP
   7. Activate but DO NOT EXECUTE as it is not needed
   8. Infopackage creation for datasource is not needed
   9. Right click on virtual provider and select Activate direct access
   10. Right click on Virtualprovider and select display data
3. RDA

Day 5

1. Aggregates - used to improve query performance
2. Steps to create an aggregate
   1. Right click on infocube and choose Maintain aggregate
   2. Select Create by yourself
   3. Create new aggregate or hit F5
   4. Drag and drop the characteristics for aggregation
   5. If you want to choose Fixed values, select it by right clicking on Characteristic
   6. Click on Aggregate and activate it
   7. Start the job (immediate or later)
   8. Click on the Spectacle symbol to view data or hit SHIFT+F9
      
      

   

3. Process chains - used to automate loading
4. Process chains work only with files present on Application server and not with local workstation files
   1. Go to transaction window and hit RSPC
   2. Click on create, enter name
   3. Create a start process by clicking on variant and hit OK
   4. Click on Change selection, choose immediate and save
   5. Save again and go back
   6. Click on Continue
   7. Go to process types
   8. Insert the Infopackage for master data datasource
   9. Insert the DTP for masterdata
   10. If all masterdata is done, create an AND
   11. Next, insert the infopackage for transaction datasource
   12. Insert DTP for DSO
   13. Insert Activate datastore object from data target administration
   14. Insert DTP for infocube
   15. Click on schedule and execute
   16. Click on logs and see execution
   17. Refresh to see status
5. 
6. Creation of master data, cube and loading from scratch
   1. Create Infoarea
   2. Create Infocatalog for char
   3. Create Infocatalog for Key fig
   4. Create char Infoobjects, mention data type and length
   5. Assign attributes to master object
   6. Create key fig infoobjects, assign unit and currency (0Currency, 0Unit, etc are default)
   7. Go to datasources
   8. Create datasources, specify source, (for Process chains, the source has to be on application server), copy template in fields tab (use CTRL Y) and activate
   9. Preview data
   10. Create infopackage and activate
   11. Repeat for all master data and transaction data
   12. Go to infoprovider
   13. For masterdata, right click on infoarea and select insert char as infoprovider
   14. Create transformation and DTP
   15. For transaction data, create DSO
   16. Create transformation and DTP
   17. Create infocube
   18. Create transformation and DTP
   19. See Process chain picture above for execution flow
7. Infoset creation
   1. Right click on Infoarea and select create infoset
   2. Choose a start with object
   3. From the screen that appears, choose a cube from the left
   4. Join the objects using the common object
   5. Right click on link and choose required join

We know that SCD 2 is about preserving all the changes in the dimension records. Let us see the logic behind how we can implement it.

NOTE: The steps below assume SCD 2 having a Begin_date and End_date and a Current_Flag column

1. Check if the incoming row is already present in the target table (dimension) using the source primary key
2. If it doesn’t exist in the target dimension
   1. Generate a surrogate key
   2. Enter source record’s date as the Begin_date
   3. Enter the default end date (which could be 31/12/2099) as the End_date
   4. If you have a Current_flag column, set it as ‘Y’ or ‘1′ (or whatever you want)
   5. Insert into the dimension
3. If the row exists in the target
   1. Check if the incoming and target current record are different (at least for one chosen attribute)
   2. If they are same, do nothing
   3. If they are different, do the following
      1. For the record in the target table, change (update) the End_date to source record’s date and set the Current_flag to ‘N’ or ‘0′ or whatever
      2. Take the incoming record, generate a surrogate key, enter source record’s date as the Begin_date and the default date as the End_date. Also set the Current_flag to ‘Y’ or ‘1′. Insert into the dimension

NOTE: The End_date of the previous record and the Begin_date of the current record are assumed to be the same. But some people prefer them to be different dates ie) End_date is 1 day lesser than the next records Current_date.

Ok, what is a stage area?

It is that part of a data warehouse where data is stored physically (in database or in files), but as an intermediate step before loading the target data warehouse / data marts. It is where activities like cleansing, de-duplication, etc take place. It is like a pit stop for a racing car before reaching the destination.

Some characteristics of the staging area are

1. accessible to and owned by ETL / DW team
2. OLAP / reporting teams do not have access to it
3. indexed very little
4. ETL developers are usually free to create / drop tables, controlled though (by the architect or modeling team)

Types of staging areas:

1. Persistent staging - stage data is not deleted, if you want to maintain history.
2. Transient staging - stage data is deleted after each ETL load

Most data warehouses have one or more staging areas, the types being either persistent or transient or both.

But should you really have a stage area? Can’t you do without it? After all these days, ETL tools are more capable of handling more data in memory fully.

Is staging necessary or is it an overhead?

I think it is very critical to have a staging area, especially if you are moving a lot of data from multiple sources. Here are a few reasons why :

1. Intermediate processing: Often you need to perform transformations, cleansing and other processing on huge chunks of data from multiple sources. It is not feasible to do all of this in memory purely (extraction from source, processing and direct load to target). We need intermediate steps where data can be stored, the ETL tool can take a breather and start again. This also helps in several other ways discussed below.
2. Auditing: Having a stage area provides a means of knowing the condition of data at certain points during the ETL cycle. If your ETL job fails mid-way, you can look at the stage tables and figure out the condition of data as it passes through the various steps. Thus it provides for easy auditing.
3. Recovery: Suppose if your ETL job fails midway, you would need to fetch the data once again from the source and repeat the entire process all over again, if you do not maintain a stage area. Having stage areas could mean that the job can be restarted from the point of failure.
4. Backup: If your source data gets overwritten, the original data is lost.. probably forever. And there is no way in which you can perform a reconciliation with the DW data against the source. But if you had a persistent stage, your data would be backed up in these intermittent databases.
5. Cleansing, synchronizing, de-duping data, etc: If you have multiple sources, stage areas are a neat method to synchronize your data, at the same time, to cleanse and conform them. (For eg, if your customer data is from 3 different sources, it makes sense to pull them all into a stage area and have then synched. This is especially true if the different sources are not available at the same time)
6. Helps reduce contention on busy source (OLTP) systems: It helps bring down the contention on the source systems. Once data is pulled, you can work on it in the stage area and get it transformed to your needs. If you were to apply these transformations on source data directly without a stage, you would be engaging the connections to the source for much longer. Thsi would impact OLTP systems adversely.
7. Increases availability of data: If one of your source systems is temporarily down, you still will have access to the data which is staged.
8. Joins are easier and faster: It is much easier and faster if the tables are in the same database schema as against different disparate databases.

Source: Several, but notably, Ralph Kimball, The data warehouse ETL toolkit.

There are several ways to estimate the effort needed to complete an ETL job. Work Breakdown Structure (WBS) is popular. And so is Function Point Analysis (FPA).

But the most widely used is the one that factors in complexity based on the understanding of things like source, target, resources on project, etc.

Though I haven’t really seen anyone use this method to perfection, it is a good place to start with. Some people argue against this method, but I see this as a complementary option to whatever method you have.

So, here is a list of points that I think would be useful when you do any ETL effort estimation. I have grouped it under 5 heads: Source, target, transformations, resources, other.

Source based:

1. No of different sources & types
2. Incremental extraction needs
3. Profiling of data sources
4. Cleansing / de-duplication dirty data sources
5. Availability of documentation / transition of knowledge of source data
6. Access control & management, if needed
7. Data volumes for unit testing

Stage / Target based:

1. Proper database design available (primary keys defined, right columns indexed, etc)
2. Familiarity of the team with the various source, staging and target systems
3. No of different target types
4. Truncate / append / merge options
5. Fact / dimension / stage load

Transformation / features / mechanisms based:

1. No of transformations and their complexity
2. Usage of reusability feature
3. Usage of features like CDC, SCD, etc
4. Need for error / failure handling and recovery mechanisms
5. Auditing and validation needs of ETL
6. Complex functions / calculations for measures
7. Parallelism / DOP / etc

Resource based :

1. Which is the tool used? What is the skill level of the resources (fresh, medium skilled, expert)?
2. Availability of the resources (part time, full time, etc)

Other :

1. Are there any people, task or decision dependencies?
2. Unit / system & integration testing needs
3. Migration / deployment needs
4. Performance and tuning needs
5. Project management / status reporting needs
6. How clear is the requirement? Is scope properly defined?
7. How well is the project being managed?
8. Have you factored in time for rework?

Few interesting links on this topic :
1. Vincent McBurney’s article on estimating ETL development time
2. FPA
3. FPA based estimation for a Data warehouse

What do you think? Is open source database a viable solution?

The link says (as quoted by Forrester) that the market share for

• Open source database is at $850 million
• Commercial databases is $ 16 billion

Key take aways from this article:

1. Many vendors are now open to the idea of a open source database. I personally feel small and medium sized enterprises are the ones who would be keen on it.
2. Forrester says the open source databases available today are capable of handling 80% of applications today. That to me is a surprisingly high number.

In my opinion, this is an interesting area for the future. Today we have a few reliable open source databases like

1. Postgre SQL
2. MySQL

But open source has its own advantages and disadvantages. Below are a few that I can think of.

Advantages:

1. Cheap / free to use and deploy.
2. Is almost capable of handling all small enterprise applications.

Disadvantages:

1. Very difficult to migrate from a commercial database to open source as of today. Can change soon.
2. Though database itself is cheap, sometimes the cost of maintaining it is high. And finding resources capable of working on an open source database is a challenge.
3. Post implementation support can be a challenge. But it is improving.
4. Lack of proper case studies and published implementations make enterprises skeptical of open source.
5. Awareness and confidence is poor.

As a parting thought, the article also rightly says that companies should try the open source way for small to medium sized applications that are NON-MISSION-CRITICAL and evaluate the database first. If found satisfactory, they should migrate to an open source database for their other applications.