Evolution of the BO XI platform – from XI R2 to XI 3.1 SP2

With BO XI 3.1 SP2 out in July this year, it is probably time to make a trip down the years to find out how the XI platform has evolved and matured.

The timeline:

• XI R2 SP2 – service pack release in March 2007 with productivity pack – QaaWS and LiveOffice connectors
• XI 3.0 – new major release in February 2008 – the first release after SAP acquired BOBJ in October 2007
• XI 3.1 – upgrade release in September 2008
• XI 3.1 SP2 – service pack release on 24 July 2009 – with enhanced SAP integration

Where were we with XI R2:

• Change to Crystal service-oriented platform (Crystal 10 architecture)
• Ability to plug Crystal Reports, Web Intelligence, Desktop Intelligence, OLAP Intelligence, Dashboard Manager, Performance Manager directly into the framework
• Single repository, security, system management, publishing, portal
• Infoview (Replaced old BO Infoview and Crystal ePortfolio)
• Central Management Console (CMC)
• Import Wizard (upgrades from BO 5, 6, XI, Crystal 8.5, 9, 10)
• Desktop Intelligence (new name for BO full client + ability to query and display Unicode data)
• Publishing, Encyclopedia, Discussions, OLAP Intelligence, Performance Management
• Changes to Data Integrator, Composer, Metadata Manager

XI 3.0 (Titan)

• All administration moved to the Central Management Console – CMC – with new GUI
• Bulk action support in CMC
• Central Configuration Manager – CCM is still there (to manage multiple nodes) with 2 entries : Tomcat & SIA
• Server Intelligence Agent (SIA) – handles service dependencies
• Server Intelligence in CMC – clone server deployments
• Repository Federation – replicate repository on other BO cluster
• Repository Diagnostic Tool (Infostore vs FileStore – repair inconsistencies between CMS database entries and files in FRS)
• Improved Import Wizard
• Web Intelligence Rich Client (offline viewing of WebI reports, no session timeout)
• Data change tracking in Web Intelligence
• Designer – “Database delegated” projection on measures
• Universe based on stored procedures
• Prompt syntax extension (persistent/primary_key undocumented features, finally!)
• Personal data provider – combine data from Excel, text, csv and get into a single report
• Smart cubes – support for non-additive measures (percentages, ratios) and RDBMS analytical functions
• Multi language support – dimensions, measures, prompts automatically localized to report viewer’s language
• Native Web Intelligence printing (without PDF)
• Enbed image in Web Intelligence report
• Hyperlinks dialog box makes links easy to create – syntax generated by WebIntelligence (remember opendocument()?)

What’s new in XI 3.1

• Support for multi-forest Active Directory authentication
• IP v6 support
• Lifecycle Management Tool (LCMBIAR files, replace Import Wizard)
• Saving Web Intelligence documents as CSV (data-only files) – new sheets for every 65K rows of data
• Web Intelligence Autosave
• “Begin_SQL” SQL prefix variable
• Prompt syntax extension (support for key-value pairs!)
• Business Objects Voyager enhancements
• Live Office enhancements
• WebIntelligence – Automatic loading of cached LOVs, interactive drag-drop, report filter bar, cancel refresh-on-open

What’s new in XI 3.1 SP2

• WebIntelligence Input controls
• OLAP universe based access to SAP BW using MDX
• BI services – expose WebIntelligence document components as web services
• Query on Query
• Fold-unfold UI improvements
• New SDKs – WebI Calculation Extension Points, Custom Data Provider plugin, Interactive viewing extension and integration points
• New universe SQL parameter SMART_AGGREGATE allows using most detailed aggregate tables
• @Prompt editor in universe Designer
• Backward compatibility is broken. Web Intelligence documents created using BO XI 3.1 SP2 cannot be opened in BO XI 3.1

In one of my next posts, I’ll cover selected new features in detail.

-Maloy

Developing a Business Objects security model – BO XI 3.1

While developing a Business Objects security model, you need to focus on the different types of security:

Functional Security – this would govern access to specific application features, e.g. editing reports, drilling down, ability to schedule reports etc.

Data Security – this governs access to specific data – rows or columns or cells as per authorization

Infrastructure Security – governs physical and electronic access to systems

The infrastructure security is the first to be designed. This typically happens when the architecture is being drawn up. It is important to get as much early visibility into the various ways the system is likely to be used, not only in the present but also in the foreseeable future, so that adjustments and capacity for future planning can be done to the extent possible. This also helps in deciding on the type of data security that would be required initially, though this can change over time.

The various security considerations for access control include:

Identification - whether it is a valid user? Usually taken care of by password management

Authentication - whether the user is allowed to use the system? This can be done by BO or externally with a third party tool, including but not limited to LDAP / Active Directory etc.

Authorization - governs fine grained entitlements or access – which parts of the application and data can the user access?

Let us look at the security approaches to authorization. (I will cover the various approaches to authentication and single-sign-on in a separate post).

Security policies can be held in the BO repository (functional + data security)

• Authentication can be performed by BO or externally
• Incorporates security policies in the BO repository
• Supports row-level and column-level security
• Data security can be controlled at application, connection, universe and report level

Custom security utilizing security tables, and joins forced in Universe Designer  (functional + data security)

• Includes custom-built security tables to store users, groups, privileges etc. The joins to these are forced in report queries.
• BO users are mapped to data in these tables – the data can be maintained with ETL processes
• The @BOUSER variable can be used to get the user logins and can be used for implementing row/column level security
• Allows both user-centric and object-centric views by querying the security tables

Table mapping or virtual private views – can be implemented with Oracle VPD and label security

• Allows fine grained access control with airtight cell-level security if required
• Policies setup in Oracle VPD, labels control column access, multiple views for multiple users
• Works for ad-hoc queries also
• Requires thorough testing to prevent sql-injection attacks; can lead to performance problems due to additional predicates
• Can easily become overly complex; however a must-have where airtight security is required

Third party authorization using SiteMinder or LDAP or Active Directory

• Authorization is based on directory entries in LDAP or Active Directory (people/group/role/IP address or rule)
• Fine grained access control still requires some form of usage of BO or the database for auxiliary authorization.

What should be the preferred approach? The answer is “Well, it depends!” The approach depends on what is actually required and is feasible at your particular organization. In all cases however (except for VPD), there are a few best practices to be followed, if BO is used and CMC is used to configure security:

• Grant rights to groups on folders, rather than individual objects to minimize complexity
• Use pre-defined rights wherever possible, and Custom Access Levels instead of Advanced Rights
• Avoid breaking inheritance to minimize complexity and simplify maintenance
• Add multiple users to the Administrators group, rather than sharing the administrator account, for better traceability
• Set up an audit policy and periodically review your deployment
• Document and maintain the security structure outside the CMC - a spreadsheet can be a good choice.
• Use Permissions Explorer, Check Relationships and Security Query to diagnose and correct security issues. These are also useful to verify tasks are completed without issues, while adding/deleting/modifying principals/objects/rights.
• Allocate time and document the process for the administrators and support staff and prepare for their training on new workflows in CMC in BO XI 3.1

- Maloy

Change the location Google Desktop Search indexes your data

Desktop search has become an important component of our everyday work. With the amount of information explosion, it is only natural that users and enterprises move towards enabling desktop (and enterprise) search for users – subject of course to appropriate security and access controls. BI vendors have moved into this new business space that has opened up and seems to be one of the most promising. While Business Objects had announced support for the Google Search appliance and Google Desktop back in 2006, their most important announcement lately has been the launch of the Business Objects Explorer (formerly known as Polestar) product. More about that in a later post…

Google Desktop Search is one of the most widely used desktop search appliances.  One would expect it to have an intelligent installer as well. Unfortunately, it doesn’t allow you to either choose the installation directory or the location for the search index. It installs in your system drive without providing any means to modify it from the Options setting. This can be quite annoying and frustrating if your system drive is not set up with a huge amount of space, as the Google Desktop search index will expand soon and hog a lot of space (up to 2 GB) on the system drive. I will show a tip here on how you can get around this issue by modifying the location of the Google Desktop search index to change it from the default system drive and without having to rebuild the index.

1. Right click and exit Google Desktop.

2. Open Windows Explorer and navigate to C:\Documents and Settings\<username>Local Settings\Application Data\Google\<google desktop search>

Note: If you’re unable to see “Local Settings” – (it’s a hidden folder) – change your folder options from Tools – View – Show hidden files and folders.

3. Move the <google desktop search> folder to a different drive, e.g. D:\ Google Desktop\<google desktop search>

4. Open the Windows registry editor from Start – Run – typing regedit – Hit Enter.

5. Navigate to HKEY_CURRENT_USER\Software\Google\Google Desktop.

6. Select the “data_dir” key in right pane, double-click to change its value to the new location of the <google desktop search index>

7. Exit the registry editor.

8. Restart Google Desktop Search.

BusinessObjects universe design best practices

Having relocated from the Silicon Valley to Bangalore a year back, I’m now working in an MIS – strategic reporting role. In my role to evangelize the use of BI best practices and tools, one of the foremost is that of universe design.  As a matter of fact, I’m currently being involved in formalizing a BI policy around the tools we use most – Oracle, Informatica and SAP Business Objects (along with migration from our legacy BO to the XI platform!) – so a lot of my current work is related to best practices, design guidelines and preparing unit test checklists for my team of developers.

So here goes my list of universe design best practices. Being the cornerstone of the Business Objects semantic layer, the universe design becomes one of the most important (next only to the data warehouse design if there is one, and foremost if there is none) aspects of getting the right data out there in time for analysis and decision making.

The best practices are grouped by the reporting area they belong to.

Universe design: object creation

• Object and class naming should be in business terms – so that it makes sense to the end-user. This also reduces development overhead since reports can use descriptions out-of-the-universe, instead of editing headers or creating report level variables.
• All objects should have help text or usage information – corollary from above.
• Object formatting should preferably be done at the universe level.
• Pre-build condition objects in the universe rather than forcing users to build conditions for reports.
• Build logic into objects – translate code, common calculations etc rather than forcing users to do it in report variables.
• Avoid using WHERE clauses in the object definitions; use CASE statement instead. In most cases, using WHERE clause will return incorrect results when similar objects are included in the result set, due to combined restrictions imposed by the multiple WHERE clauses.
• Use aggregation in all measure objects – to push the aggregation to the database wherever the performance bottleneck is likely to be BO server and the database performance is optimal. Generally the database is much more powerful at doing aggregation calculations, and this also reduces the volume of data to be transported over the network.
• All measure objects should include aggregation functions for projection. When this is not included, BO will not automatically roll-up the data in the report, which could result in incorrect data and analysis.  Note that in the 3.0 version of Designer, a new feature – Database Delegated projection function is available to take care of these anomalies while doing “averages” for instance.
• Use Custom LOVs or cascading prompts to display LOVs where hierarchies and numerous values are involved.
• Use relative date objects for scheduling e.g. Today, Yesterday, Previous Month etc. Create a separate class to contain these reporting objects – this helps in improving maintainability.
• Use dynamic HTML in objects where required to avoid users having to build it in report variables – end users wouldn’t like to code hyperlinks themselves, but would love to have an object which when clicked can lead them to Google Maps for example.
• Use contexts in universes having multiple fact tables – this helps in getting your measures (built from multiple fact tables) right.
• Use derived tables to define measures dependent on multiple fact tables.
• Use derived tables to reduce complexity of queries to be written by users or in place of views or procedures. A note of caution here: Use derived tables sparingly. If you have access to the database or DBA and can get views or tables created for the same purpose, go with it rather than using derived tables. This is not only to push the logic and work closer to the database, but also to take care of the performance and maintainability aspects. Exceptions to this include cases where your derived table may include a prompt which would restrict the number of rows returned and thus improve performance over a conventional view.
• Reuse code with @Variable. Reuse interactive objects with @Where (if you use them at all).
• Use @Prompt syntax for conditions and interactive objects where input values are likely to change or absence of prompt would lead to inaccurate values or unacceptable query response times. Also make sure regularly used conditions e.g. current year / latest date should not have prompts to avoid annoying users.
• “To limit the number of objects created to avoid user confusion, build interactive objects with @Prompt syntax followed by additional OR clause to include “”All”" condition.

E.g. ‘ALL’ IN @Prompt(‘Enter Value or ALL’,'A’, ‘Class\Object’,multi,)

OR

Table.Column IN @Prompt(‘Enter Value or ALL’,'A’, ‘Class\Object’,multi,)”

Universe design: resolving join and performance problems

• To resolve a chasm trap, define a context for each table at the “many” end of the joins.
• To resolve a fan trap, create an alias table for the table producing the multiplied aggregation. Create a 1:1 join between the original and the alias tables. Modify the select statement to use the columns from the alias table instead of the original table.
• Use of contexts should be evaluated w.r.t. use of aliases for resolving join issues, to take care of maintainability of code.
• Integrity checks on the universe structure, parsing of objects, joins, contexts, detecting loops etc is mandatory. If you wish to use Business Objects to help you detect fan traps or chasm traps – you must set the cardinality on the joins. Do not rely on BO to suggest the cardinality – this is often erroneous, based on the records sample that BO fetches for each table.
• Uncheck the “Multiple sql statements for each measure” option in universe parameters, if this is not required for resolving any join problems. This option should be checked if the measures being retrieved in the same query involve different tables. “Prevent Cartesian product” should be checked, as should there be limits placed on the number of records returned and the time for the sql connection – to prevent runaway queries which can bring the database down to its knees and cause an outage for all users.

Universe design: optimization / miscellaneous

• Use shortcut joins wherever possible to reduce number of tables used in a query
• Use aggregate tables /materialized views with aggregate awareness set up to improve query performance
• Use keys instead of labels where possible to take care of index awareness benefits of performance and uniqueness
• Use the JOIN_BY_SQL parameter to shift process from BO server to database wherever the bottleneck for performance is the BO server and the database performance is optimal.
• Update the .prm files to enable access to custom SQL functions and improve help text
• Do not use derived tables instead of aggregate tables.
• Turn off LOVs for all dimension and detail objects that are redundant or not required. This prevents performance problems when users inadvertently click on the “Values” and the query sets to return all the IDs or other irrelevant data.
• Consider using linked universes with a master kernel universe to ensure consistent dimensions across multiple universes

This list is certainly not an exhaustive one – but a work-in-progress. I’d update it as and when I compile more; meanwhile if you feel anything has been left out, drop in a line.

SPC – Using statistics to get insight from BI

There is a well known adage that if you keep doing the same thing and expect different results, that is a sure sign of idiocy.  In the BI world too, we come across several instances where people take it for granted that the ‘BI tool’ will magically generate insight and spur ‘intelligence’ rather than ‘idiocy’. Yet the very practices of reporting the same measures, or of creating reports for metrics just because they are now made available by the tool, without sparing any ‘intelligence’ into what will generate insight is a major cause  of failures of BI.  Most of the leading commercial BI products are expensive and cost a lot of money in maintenance and support, so it is rather important to understand how to design the proper metrics and KPIs (key process indicators) which would generate insight. Even more important is to have a process focus and a general idea of the basics of statistical process control, in order to make sure that the right decisions are made and resources are spent on processes and strategies where they would have the most impact.

Statistical Process Control (SPC) is quite well known in the manufacturing industry and also in software engineering. In effect, it applies rules of statistics to the processes that are followed to predict whether a process is stable (and therefore in control) and its output is predictable or not and how to identify out-of-control processes and take corrective measures. Quality aids like causal analysis done using brainstorming/ nominal group techniques/ Ishikawa diagrams or fish-bone analysis are helpful in analyzing outliers and reasons of deviation from control limits. A substantive discussion of SPC and quality process areas is not possible in this post so I’ll just touch upon some concepts concisely.

PDCA – Plan-Do-Check-Act cycle, proposed by economist William Shewhart and later by quality guru Dr. Edward Deming. This is the foundation of the management and feedback cycle underlying any software engineering process.

Control limits – Any process which follows the Gaussian normal distribution would have a normal bell-shaped curve and be subject to control limits. The stability of the process can be gauged by the outliers (number and pattern of data points falling outside the control limits).

Causes of deviation: Outliers indicate deviation from a stable and predictable process. Causes of deviation could be due to special causes or common causes. Common causes are like background noise and may be present in stable processes. Special causes must be removed and steps taken to prevent their occurrence to bring a process under control. Common causes may be reduced to have a sharper curve with a narrower band of control limits and have greater control on the process.

 

 

Control Chart (Image courtesy: Wikipedia)

Users of BI tools haven’t tapped into the power of SPC to gain insight and control operational processes to the extent possible. There is even danger of damaging with a stable and in-control process due to tinkering with the process based on common-cause variation observed in operational reports. Part of the reason for SPC not gaining sufficient currency is that business analysts are not trained in the basics of SPC or quality processes like DAR (defect analysis and resolution) but mostly it is due to there not being any BI product in the market so far which allows easy use of SPC analysis. It is only of late that vendors like SAP-Business Objects have come out with specific SPC modules and predictive analytics in the BI product marketplace.

BI is a specialized discipline which involves a lot of investment on the part of customers in terms of pre-sale-evaluation (proof-of-concepts / comparisons), implementations, maintenance and support. However the returns from BI implementations are not easy to quantify and ROI (return on investment) figure calculations could be vague and incorrect. Using SPC along with the right quality process framework allows in maximizing the value of BI implementations, as well as provides a ready-reckoner for calculating ROI based on projected process improvements based on statistical control limits.