There are quite a few schools of thought and available literature on the lifecycle of BI implementation and maturity in organizations, defining the models. Most are proprietary models provided by consultancies, which are primarily based on technical point of view or applies the knowledge management function to BI, following the Ladder of Business Intelligence (LOBI) model.

LOBI includes 6 levels of maturity moving up the knowledge management value chain from Facts > Data > Information > Knowledge > Understanding > Enabled Intuition.

There are several other models in the public domain e.g.

• Business Information maturity model
• AMR research’s BI/Performance management maturity model
• Business Intelligence development model
• Business Intelligence maturity hierarchy
• Infrastructure optimization maturity model

I’ll not go into the details of the models above but discuss the three of the more popular and well documented models available.

1. The TDWI BI Maturity Model

The Data Warehousing Institute (TDWI) is a premier body in the field of BI and eponymous Data warehousing and proposes a six stage BI maturity model. The underlying assumption being that BI implementation in organizations typically evolves from a low-value cost centre operation to a high value strategic utility to provide competitive advantage.

Stage 1: Prenatal – Executive perception is that of a cost-center, which primarily churns out static reports for management operational reporting. It is also the stage which costs the most.

Stage 2: Infant – The BI function’s role is to inform executives, with several reports leading to “spreadmarts”

A ‘Gulf‘ separates Stage 2 and Stage 3.

Stage 3: Child – The BI function’s role is perceived to empower workers, and this is the first evolution into an analytical system where OLAP and ad-hoc reports are used off data marts.

Stage 4: Teenager – The BI function has evolved into a performance monitoring system by now, using Dashboards and Scorecards, supported by data warehouses.

A ‘Chasm‘ separates Stage 4 and Stage 5.

Stage 5: Adult – This is where the ROI from the BI function shoots up, with predictive analytics answering what-if questions making the BI a strategic utility. The TDWI thinks that organizations’ BI architecture has evolved to have enterprise DW by now, with BI becoming a ‘Drive the Business’ function.

Stage 6: Sage – The BI function at this stage has the highest ROI and decreasing costs based off Analytic Services (SOA) with pervasive BI (e.g. embedded BI) making it ‘Drive the market’

2. The HP Business Intelligence Maturity Model

It has 5 stages based on the evolution of Business enablement, Information technology and program management.

Stage 1 – Operation (Running the business) – involves ad-hoc solutions focused at project activities alone

Stage 2 – Improvement (Measuring and monitoring the business)– involved localized solutions with project management

Stage 3 – Alignment – includes shared resources with program management and governance integrating performance management and BI programs

Stage 4 – Empowerment – includes enterprise operationalization with portfolio management focusing on organization innovation and people productivity through knowledge management

Stage 5 – Transformation (Change the business) – involves enterprise services tracked by service management creating strategic agility and differentiation

3. Gartner BI Maturity Model

Gartner, the IT research and advisory group’s BI maturity model is based on 3 key areas of assessment – people, processes and metrics. It has 5 maturity levels:

Level 1 – Unaware – Spreadsheet and information anarchy, one-off report requests

Level 2 – Tactical – Usage limited to few executives with data inconsistency and stovepipe systems

Level 3- Focused – Specific ser if users realize value, with focus on specific business need and BI competency centre (BICC) in place

Level 4 – Strategic – Business objectives drive the BI and performance management systems with well defined and enforced governance policies and standards

Level 5 – Pervasive – Use of BI is extended to suppliers and customers, information is trusted (holy grail of single version of truth) with analytics embedded in business processes

Barriers to BI adoption and maturity

The 3 models discussed above do a good job of explaining the continuum of maturity levels, which makes it difficult to identify explicit stages, however the common theme across these are:

1. Each model has at least 5 stages of maturity – this is more than a simple 1-2-3. This indicates the path of BI evolution is longer and more complex than most think while jumping onto the BI bandwagon
2. Each model starts with operational / one-off reporting and culminates in pervasive BI where BI is embedded in business processes and provides actionable insight for strategic advantage
3. The models do not focus on technology alone and hinge on the involvement of people and process as well. Moving from one-off reporting to driving the enterprise involves big changes in organization culture and business processes and not just implementing the latest BI tool off the market.

The main barriers to BI adoption and demonstrating its worth as a strategic tool lies in its complexity. BI is a broad area encompassing both technical and non-technical aspects like people and process; therefore the models can only provide a prescriptive framework which needs to be adapted by each organization. It is important to understand that various departments of an organization can be at varying levels of maturity and not every organization follows the same trajectory of evolution or has to go through each stage.

It is however noteworthy that for organizations trying to move from basic levels (e.g. TDWI Level 2 –Infant stage) to higher levels (e.g. TDWI Level 5 – Adult stage) may find it very difficult to leapfrog levels. In fact regressing stages is also possible due to changes like mergers and acquisitions of organizations at different levels of maturity where differences across people, processes and technology may be difficult to reconcile or could be delayed. The TDWI model recognizes these difficulties as:

• The Gulf – between level 2 (infant) and level 3(child) – mainly due to differences in executive perception, data quality issues and spreadmart anarchy
• The Chasm – between level 4(teenager) and level 5(adult) – mainly due to differences in executive perception, spreadmarts, architectural inflexibility or lock-ins

It is important to take lessons from the BI maturity models and develop a BI strategy while planning to implement BI, rather than as a bolt-on which can provide instant ROI. The strategy needs to focus on quick wins at inception to build buy-ins and get executive sponsorship which is critical to the funding of the BI program and would help overcome organizational barriers in people and processes, and then should build on its success with incremental gains and asking the right questions. We’ll look at developing a BI strategy in a subsequent post.

In the following section I’ll try to cover the major changes that have been effected in the following products:

Semantic Layer - A new tool, Information Design Tool enhances the Universe Designer. The universes created by this tool are identified by the .UNX file extension and allow connections to multiple data sources.

The universe designer is still there. Renamed as universe design tool, it allows creating single data source universes (.UNV file extension) as before.

Conversion of previous universe .unv versions is supported only for relational universes created in previous universe designer versions and not possible for OLAP universes or universes based on stored procedures or Data Federator data source.

No authentication is required to start the information design tool. Users can create and edit unsecured resources (data foundations, business layers, connections) in local projects and publish them to the repository to make them secure.

Connections to relational data sources, OLAP data sources as well as SAP NetWeaver BEx query can be created, be local (saved locally as .cnx files) or secured (stored in the repository).

The newly named “Data foundations” are analogous to the schema browsers in Universe Designer. They contain the schema of relevant tables and joins from one or more relational databases that are used as a basis for one or more business layers.

The business layer is the universe metadata. Depending on the type of data source for the business layer, several types of objects e.g. folders, dimensions, analysis dimensions, measures, attributes, filters, hierarchies (OLAP only) can be created and edited in the business layer.

Search - enhancements include a new enhanced search engine allowing search by document attributes as well as content. Search results can be filtered and refined easily and the search GUI is integrated in the BI launch pad

There are also enhanced options through the OpenSearch API which enables integration with other search systems like Google Search Appliance, Microsoft SharePoint portal and NetWeaver Enterprise Search.

BI Portal - includes a new look re-designed web portal (InfoView) now called the BI LaunchPad providing a rich new user experience. It provides quick and easy access to BI applications and search, a handy list of recently used reports, scheduled documents, alerts etc., multiple tabs and pinning options, and a reduction in the manual steps for common tasks like:

• Ability to create new folder while Saving
• Schedule and Send To actions in Document viewers
• Auto-refresh in History page

Building on the rapid-marts offering that the then BOBJ used to have and leveraging SAP’s industry and line of business expertise, these new applications are based on the SAP Business Objects XI platform – WebIntelligence, Crystal Reports and Dashboards (formerly Xcelsius). Bill McDermott, the joint CEO of SAP, described it as “complete and ready-to-go” and claimed the applications can be deployed in as less as eight weeks.

You may remember the brouhaha created by SAS last year , when it kicked off the controversy on Business Analytics being the future, rather than Business Intelligence. Going back even further, Oracle already had this in its Siebel Analytics pre-built analytic applications for various industries. Therefore, it would seem that SAP is already late in the game, but considering that neither Microsoft nor IBM have similar offerings, it may not be too bad for SAP. Better late than never…

Under the hood:

The pre-packaged analytic applications are based on the BusinessObjects XI platform – with the universe as the semantic layer or metadata model. It can be based on both SAP and non-SAP data, OLTP and data warehouse, relational and unstructured.  SAP would work with its partners HP and Teradata to optimize the analytic solutions on their hosting and data warehousing solutions.

Business Analytics dashboards are Xcelsius flash files which can be used with web services/QAWS to deliver real-time analytics. It may also be possible to use these with SAP Business Objects Explorer (formerly Polestar) and/or SAP BW Accelerator or the SAP high-performance analytic appliance (HANA).

Business Analytics vs. Business Intelligence – Revisiting the controversy:

When SAS created this controversy last year, an important point noted by many was the SAS home page titled:

SAS | Business Intelligence Software and Predictive Analytics

It’s important to see how the rebranding has reflected in a change to the SAS home page a year hence. It now reads:

SAS | Business Analytics and Business Intelligence Software

SAS Institute was always viewed as a niche vendor, operating in the pure-play statistical and predictive analytics space and this marketing was to re-brand SAS’ offerings to move it mainstream.  In effect, it signaled the market assessment by these major vendors, that in tough times, customers were seeking shorter lead times and demanding better tools which are quick and easy to introduce and provide quicker return on investment.  As we come around the downturn, with SAP still focusing on this segment, it is clear that traditional BI is clearly seen as complex, costly and difficult to implement.

Open questions:

There are several questions open at the moment, given that this is an initial launch. SAP plans to offer more applications over the next year-18 months in collaboration with customers and its partners.  The partners include Aster Group, Blueprint, Capgemini, Column5, CSC, Fusion Consulting, The Glenture Group, LSI Consulting and syskoplan and surely it would take quite a while for the ecosystem to develop.  It remains to be seen whether the prepackaged analytics catches on like Xcelsius dashboards did for BOBJ.

It is not clear whether the prepackaged analytics would be positioned at the bigger enterprises or the SME segment only, as its success could cannibalize revenues from the flagship Enterprise XI suite.

There are also questions around the scalability of the framework the analytic applications are built on. The extensibility APIs and reference architectures for partners to build their own add-ons and plugins / applications  of their own is not yet out (planned in 2011), so it’s not quite like the iPhone/iPad app store yet. It is also not clear how customizations to the applications would be supported or to what extent these could be customized.  The long awaited universe rewrite including data federation might be a part of plans if the analytic applications turn out to be truly backend-agnostic and do support future in-memory data structures (SAP’s acquisition of Sybase would indicate likely support for the Sybase ASE in-memory database). If this happens, it would be in line with earlier plans to roll-out in-memory EPM and OLTP solutions.

Overall impression: The BT Summit was heavily focused on cloud computing with half of second day having a deep dive into Amazon’s EC2 cloud offering, and several keynotes. SOA and web services, REST and similar architectural sessions were interspersed but definitely not a first-class citizen. BI came a poor third with a poor choice of sessions, and more of a rehash of what is out there for everyone, rather than something on the cutting-edge including use of appliances and columnar databases, as also in-memory databases and use of Flash and AJAX for interactive BI front-ends.

Session-wise review: (Speaker profiles available here). I was able to speak to and ask questions of Vinod Kumar, Vijay Doddavaram, Abhinav Agarwal and Dr. Bob Marcus.

Keynotes:

• Smart services, bright future by Howard Charney

Probably the highlight of the keynotes, this was a pep-talk about the inevitable interconnected future with smart products and services and for good measure Charney threw out some statistics on broadband growth and bandwidth usage and India’s readiness and potential in the scheme of things.

• Give cloud a chance by Ramkumar Kothandaraman

The worst of the lot – this started by comparing the spectrum of offerings in the cloud from Amazon’s DIY EC2 and AWS, Google appengine and apps to Microsoft’s Azure and ended up as a promo touting Azure as the best buy among all.

• Cloud computing: State of the union address by Shouvick Mukherjee

A very good keynote, focusing on what makes sense to migrate to the cloud and what doesn’t, what are the hidden costs, the myth of unlimited elasticity in the cloud and what Yahoo is doing to use open source software like Hadoop and Hive for cloud computing. In the short time span, Shouvick also tried to address some of the other considerations – including re-architecting existing applications, availability, data storage and movement considerations.

• Managing IT in turbulent times by Som Sharma

This post-lunch keynote by Sharma was a rambling talk on how technology keeps redefining our lives, and why it is important to think outside-the-box. He used the example of the iPhone to illustrate how such thinking has the potential to alter the established rules of the industry and redefine it as we know it.

• A New world to protect by Nils Puhlmann

Puhlmann provided the security perspective on how easy it to break/hack enterprise systems and how anti-virus and anti-spyware are always playing catch-up, the entire economy that is spawned by the “bad-guys” in technology and why our systems need to be smart and be built from the ground-up for security rather than as an afterthought. He provided valuable insights into what questions we should ask ourselves as we embrace cloud computing, the changing technology landscape making it easy for consuming information but easier still for the security breachers. Puhlmann concluded by suggesting it may be worthwhile including a level of risk assessment and mitigation, and collaboration with ethical hackers, rather than trying to do the impossible of removing all security threats.

• Moving towards a virtual enterprise by Srinivas Varadarajan

Barely managed to sleep through it – this one talked about moving towards a virtual enterprise – with a focus on virtualized architecture, including cloud computing. As boring as they can get.

Other sessions:

• SOA, Composite Applications, and Cloud Computing: Three pillars of a modern technology solution by Robert Schneider

Robert  Schneider presented the different facets of SOA, Composite applications (superset of mash-ups) and Cloud computing and contrasted them regarding the time to yield benefits, the maturity of the vision, involvement and buy-in from business and where they lie in the tactical-strategic plane. There wasn’t anything regarding why we are stuck with these three for a modern technology solution, or what other paradigms are out there beyond the old-world enterprise computing framework, possibly due to time constraints.

• Self-service analysis and the future of Business Intelligence by Vinod Kumar

A lot of the BI folks were waiting for this, as Vinod performed the Project Gemini (Office 2010 Excel and PowerPivot) demo live for the first time in India, with several folks, including yours truly, sitting on the stairs. [We have had to rely on Youtube videos and MS Office 2010 preview videos earlier]. The demo was impressive fetching over 13 million records into Excel using a standard DDR laptop, using compression and in-memory technologies. The bigger question around unleashing another round of Excel hell went unanswered due to time constraints, however the presentation probably hinted at Microsoft’s vision of “self-service BI” or so-called “underground-BI” as the power-users of Excel (estimated at 2M worldwide, at 4% of the Excel user base) have been doing. Microsoft’s strategy around pushing SharePoint adoption in the Enterprise was made clear tacitly with SharePoint being the only “portal” to publish and share BI analysis (typical size of these Excel spreadsheets is upwards of 200MB) with other users in the enterprise.

• Designing and Implementing RESTful web services by Eben Hewitt

Eben Hewitt started off with a very brief comparison between SOAP (Simple Object Access Protocol) modeled more on the lines of RPC (Remote Procedure Call) and REST (Representational State Transfer) and clarified that REST is more an architectural style rather than specifications. The remainder of the talk delved into details of implementation of REST – usage of simple ‘verbs’ and complexity in ‘nouns’, uniform interface, using named resources, java REST frameworks like Jersey, MIME types – JSON, XML, YAML and HTTP operations supported – POST, GET, PUT and DELETE.

• Business Intelligence project execution on a shoestring by Rajesh Ramaswamy

I attended with some expectations on how a BI project can be executed possibly with open-source or free software like MySQL/Postgres, Pentaho/Talend, Jaspersoft/MicroStrategy reporting suite etc., but was highly disappointed by the presentation. Ramaswamy spoke on BI usage, barriers to BI adoption, costs of BI implementation and spewed statistics like m&m’s with cursory references to Forrester, Gartner and “research studies”, but there wasn’t anything tangible on how to go about a project execution except for some common-sense talk on “evaluating options” between open-source and licensing costs, offshoring and outsourcing, RDBMS vs. analytica databases and appliances etc.

• Business Intelligence – Leveraging and Navigating during current challenging times by Vijay Doddavaram

Vijay spoke of the current global economic downturn and how it had taken everyone unawares during the downturn as well as when the current quarter the tide seems to have returned. With the example of a fictitious company in China, he illustrated the importance of trade-off between tactical and strategic decision making and whether and how business intelligence can make a difference in either a downturn and the upswing (whether it is a U, V, or a W curve). Thought-provoking, one couldn’t help feel that BI software has not yet eliminated the “intelligence” that people bring to the table, and made a distinct point about the “human analysis/intelligence” against the out-of-the-box actionable-intelligence marketed by the BI vendors. It would have been interesting to prolong the discussion, with a focus on the “predictive-analytics” offerings in the market (from SAP, WPC, SPSS and the open-source R etc.), we had once again run out of time, and it was the last session of the day as well.

• Towards a unified Business Intelligence and Enterprise Performance Management Strategy by Abhinav Agarwal

Abhinav is from Oracle and he used this session to basically present the BI and EPM strategy of Oracle. Refreshing when contrasted with the usual Oracle marketing hype, Abhinav made it a point to stress the difficulty of delivering best-in-breed products due to numerous acquisitions and the inevitable integrations compared to the disruptive start-ups which could be one-trick ponies but nevertheless manage to push the technology envelope. Most of the session focused on Oracle BI server offering and the roadmap of integrating with the Fusion middleware, and brief touchpoints on the capabilities of the Oracle BI server: federated queries (acquired from nQuire, which Siebel systems had acquired, prior to being bought by Oracle), and real-time updates, including Oracle RTD (Real-time Decisions) and the segregation of the BI and EPM software offerings.

• 10 Things software architects should know by Eben Hewitt

I was able to attend part of it, but for the most part- the bottomline of this talk was the trade-offs architects need to make and understanding there may not be a “solution” to a problem, it may just be “moving the problem” – the idea that each “solution” brings its own issues and tradeoffs into the picture. Being more focused on java APIs and cloud computing frameworks, it could have done better with something related to networks and database architecture in general for audience to relate better (for most of my time, I couldn’t relate to a BI applications and data-warehousing infrastructure).

• Cloud initiatives and standards roadmaps by Dr. Bob Marcus

Being late from an overcrowded dining hall, I was able to attend part of this. Bob spoke of the various public and private initiatives including those from the federal government, NASA Nebula and made the distinction early on between the types of offerings on the cloud: SaaS (Software as a service), IaaS (Infrastructure as a service) and PaaS (Platform as a Service). He mentioned in passing the data.gov and apps.gov initiatives of the Obama administration as also about RACE (Rapid Access Computing Environment) from the Dept. of Defense – Defense Information Systems Agency.

• Building Enterprise Dashboards by Vivek Khurana

Vivek Khurana did a very short presentation to an overflowing hall on clichéd but nevertheless important aspects of information visualization while designing dashboards: clutter vs. simplicity, proper designing of KPIs, importance of delivery to mobile devices, and learning from news aggregation sites and portals on presentation.

• Implementing Enterprise 2.0 using Open Source products by Udayan Banerjee

Banerjee did a great job of presenting what his vision of implementing Enterprise 2.0 in NIIT was – implementing SLATES (coined by Andrew McAfee) – Search, Links, Authoring, Tags, Extensions and Signals. Within half-an-hour he navigated us through using open-source products for collaboration using blogs and wiki (MediaWiki), using single-sign-on with enterprise databases, using links and tag clouds and integrating Search as well as implementing a text-based instant messenger.

• ECM-CMIS and the emergence of standards by Alan Pelz-Sharpe

I had missed the earlier session of Alan on lessons learnt using SharePoint, so I made it a point to attend the last of this at the summit – even though it meant I had no clue sometimes of what was being talked about! Alan spoke of the emergence of the multi-vendor CMIS standard for Enterprise Content Management – the various facets of ECM – from digital and media assets, email archiving, Internet content, web analytics, document types, rich media and the problems with the earlier Java standards like JSR 170 – most notably the absence of support from Microsoft. He also spoke about the vendor landscape and a 9-block rating similar to Gartner’s magic quadrant, plus various other important standards, including XAM – eXtensible Access Method – a storage standard developed by SNIA (Storage and Networking Industry Association)

Presentation files: Most presentation files are available here. You’ll need to register though to download.

- Maloy

The timeline:

• XI R2 SP2 – service pack release in March 2007 with productivity pack – QaaWS and LiveOffice connectors
• XI 3.0 – new major release in February 2008 – the first release after SAP acquired BOBJ in October 2007
• XI 3.1 – upgrade release in September 2008
• XI 3.1 SP2 – service pack release on 24 July 2009 – with enhanced SAP integration

Where were we with XI R2:

• Change to Crystal service-oriented platform (Crystal 10 architecture)
• Ability to plug Crystal Reports, Web Intelligence, Desktop Intelligence, OLAP Intelligence, Dashboard Manager, Performance Manager directly into the framework
• Single repository, security, system management, publishing, portal
• Infoview (Replaced old BO Infoview and Crystal ePortfolio)
• Central Management Console (CMC)
• Import Wizard (upgrades from BO 5, 6, XI, Crystal 8.5, 9, 10)
• Desktop Intelligence (new name for BO full client + ability to query and display Unicode data)
• Publishing, Encyclopedia, Discussions, OLAP Intelligence, Performance Management
• Changes to Data Integrator, Composer, Metadata Manager

XI 3.0 (Titan)

• All administration moved to the Central Management Console – CMC – with new GUI
• Bulk action support in CMC
• Central Configuration Manager – CCM is still there (to manage multiple nodes) with 2 entries : Tomcat & SIA
• Server Intelligence Agent (SIA) – handles service dependencies
• Server Intelligence in CMC – clone server deployments
• Repository Federation – replicate repository on other BO cluster
• Repository Diagnostic Tool (Infostore vs FileStore – repair inconsistencies between CMS database entries and files in FRS)
• Improved Import Wizard
• Web Intelligence Rich Client (offline viewing of WebI reports, no session timeout)
• Data change tracking in Web Intelligence
• Designer – “Database delegated” projection on measures
• Universe based on stored procedures
• Prompt syntax extension (persistent/primary_key undocumented features, finally!)
• Personal data provider – combine data from Excel, text, csv and get into a single report
• Smart cubes – support for non-additive measures (percentages, ratios) and RDBMS analytical functions
• Multi language support – dimensions, measures, prompts automatically localized to report viewer’s language
• Native Web Intelligence printing (without PDF)
• Enbed image in Web Intelligence report
• Hyperlinks dialog box makes links easy to create – syntax generated by WebIntelligence (remember opendocument()?)

What’s new in XI 3.1

• Support for multi-forest Active Directory authentication
• IP v6 support
• Lifecycle Management Tool (LCMBIAR files, replace Import Wizard)
• Saving Web Intelligence documents as CSV (data-only files) – new sheets for every 65K rows of data
• Web Intelligence Autosave
• “Begin_SQL” SQL prefix variable
• Prompt syntax extension (support for key-value pairs!)
• Business Objects Voyager enhancements
• Live Office enhancements
• WebIntelligence – Automatic loading of cached LOVs, interactive drag-drop, report filter bar, cancel refresh-on-open

What’s new in XI 3.1 SP2

• WebIntelligence Input controls
• OLAP universe based access to SAP BW using MDX
• BI services – expose WebIntelligence document components as web services
• Query on Query
• Fold-unfold UI improvements
• New SDKs – WebI Calculation Extension Points, Custom Data Provider plugin, Interactive viewing extension and integration points
• New universe SQL parameter SMART_AGGREGATE allows using most detailed aggregate tables
• @Prompt editor in universe Designer
• Backward compatibility is broken. Web Intelligence documents created using BO XI 3.1 SP2 cannot be opened in BO XI 3.1

In one of my next posts, I’ll cover selected new features in detail.

-Maloy

Functional Security – this would govern access to specific application features, e.g. editing reports, drilling down, ability to schedule reports etc.

Data Security – this governs access to specific data – rows or columns or cells as per authorization

Infrastructure Security – governs physical and electronic access to systems

The infrastructure security is the first to be designed. This typically happens when the architecture is being drawn up. It is important to get as much early visibility into the various ways the system is likely to be used, not only in the present but also in the foreseeable future, so that adjustments and capacity for future planning can be done to the extent possible. This also helps in deciding on the type of data security that would be required initially, though this can change over time.

The various security considerations for access control include:

Identification - whether it is a valid user? Usually taken care of by password management

Authentication - whether the user is allowed to use the system? This can be done by BO or externally with a third party tool, including but not limited to LDAP / Active Directory etc.

Authorization - governs fine grained entitlements or access – which parts of the application and data can the user access?

Let us look at the security approaches to authorization. (I will cover the various approaches to authentication and single-sign-on in a separate post).

Security policies can be held in the BO repository (functional + data security)

• Authentication can be performed by BO or externally
• Incorporates security policies in the BO repository
• Supports row-level and column-level security
• Data security can be controlled at application, connection, universe and report level

Custom security utilizing security tables, and joins forced in Universe Designer  (functional + data security)

• Includes custom-built security tables to store users, groups, privileges etc. The joins to these are forced in report queries.
• BO users are mapped to data in these tables – the data can be maintained with ETL processes
• The @BOUSER variable can be used to get the user logins and can be used for implementing row/column level security
• Allows both user-centric and object-centric views by querying the security tables

Table mapping or virtual private views – can be implemented with Oracle VPD and label security

• Allows fine grained access control with airtight cell-level security if required
• Policies setup in Oracle VPD, labels control column access, multiple views for multiple users
• Works for ad-hoc queries also
• Requires thorough testing to prevent sql-injection attacks; can lead to performance problems due to additional predicates
• Can easily become overly complex; however a must-have where airtight security is required

Third party authorization using SiteMinder or LDAP or Active Directory

• Authorization is based on directory entries in LDAP or Active Directory (people/group/role/IP address or rule)
• Fine grained access control still requires some form of usage of BO or the database for auxiliary authorization.

What should be the preferred approach? The answer is “Well, it depends!” The approach depends on what is actually required and is feasible at your particular organization. In all cases however (except for VPD), there are a few best practices to be followed, if BO is used and CMC is used to configure security:

• Grant rights to groups on folders, rather than individual objects to minimize complexity
• Use pre-defined rights wherever possible, and Custom Access Levels instead of Advanced Rights
• Avoid breaking inheritance to minimize complexity and simplify maintenance
• Add multiple users to the Administrators group, rather than sharing the administrator account, for better traceability
• Set up an audit policy and periodically review your deployment
• Document and maintain the security structure outside the CMC - a spreadsheet can be a good choice.
• Use Permissions Explorer, Check Relationships and Security Query to diagnose and correct security issues. These are also useful to verify tasks are completed without issues, while adding/deleting/modifying principals/objects/rights.
• Allocate time and document the process for the administrators and support staff and prepare for their training on new workflows in CMC in BO XI 3.1

- Maloy

Google Desktop Search is one of the most widely used desktop search appliances.  One would expect it to have an intelligent installer as well. Unfortunately, it doesn’t allow you to either choose the installation directory or the location for the search index. It installs in your system drive without providing any means to modify it from the Options setting. This can be quite annoying and frustrating if your system drive is not set up with a huge amount of space, as the Google Desktop search index will expand soon and hog a lot of space (up to 2 GB) on the system drive. I will show a tip here on how you can get around this issue by modifying the location of the Google Desktop search index to change it from the default system drive and without having to rebuild the index.

1. Right click and exit Google Desktop.

2. Open Windows Explorer and navigate to C:\Documents and Settings\<username>Local Settings\Application Data\Google\<google desktop search>

Note: If you’re unable to see “Local Settings” – (it’s a hidden folder) – change your folder options from Tools – View – Show hidden files and folders.

3. Move the <google desktop search> folder to a different drive, e.g. D:\ Google Desktop\<google desktop search>

4. Open the Windows registry editor from Start – Run – typing regedit – Hit Enter.

5. Navigate to HKEY_CURRENT_USER\Software\Google\Google Desktop.

6. Select the “data_dir” key in right pane, double-click to change its value to the new location of the <google desktop search index>

7. Exit the registry editor.

8. Restart Google Desktop Search.

So here goes my list of universe design best practices. Being the cornerstone of the Business Objects semantic layer, the universe design becomes one of the most important (next only to the data warehouse design if there is one, and foremost if there is none) aspects of getting the right data out there in time for analysis and decision making.

The best practices are grouped by the reporting area they belong to.

Universe design: object creation

• Object and class naming should be in business terms – so that it makes sense to the end-user. This also reduces development overhead since reports can use descriptions out-of-the-universe, instead of editing headers or creating report level variables.
• All objects should have help text or usage information – corollary from above.
• Object formatting should preferably be done at the universe level.
• Pre-build condition objects in the universe rather than forcing users to build conditions for reports.
• Build logic into objects – translate code, common calculations etc rather than forcing users to do it in report variables.
• Avoid using WHERE clauses in the object definitions; use CASE statement instead. In most cases, using WHERE clause will return incorrect results when similar objects are included in the result set, due to combined restrictions imposed by the multiple WHERE clauses.
• Use aggregation in all measure objects – to push the aggregation to the database wherever the performance bottleneck is likely to be BO server and the database performance is optimal. Generally the database is much more powerful at doing aggregation calculations, and this also reduces the volume of data to be transported over the network.
• All measure objects should include aggregation functions for projection. When this is not included, BO will not automatically roll-up the data in the report, which could result in incorrect data and analysis.  Note that in the 3.0 version of Designer, a new feature – Database Delegated projection function is available to take care of these anomalies while doing “averages” for instance.
• Use Custom LOVs or cascading prompts to display LOVs where hierarchies and numerous values are involved.
• Use relative date objects for scheduling e.g. Today, Yesterday, Previous Month etc. Create a separate class to contain these reporting objects – this helps in improving maintainability.
• Use dynamic HTML in objects where required to avoid users having to build it in report variables – end users wouldn’t like to code hyperlinks themselves, but would love to have an object which when clicked can lead them to Google Maps for example.
• Use contexts in universes having multiple fact tables – this helps in getting your measures (built from multiple fact tables) right.
• Use derived tables to define measures dependent on multiple fact tables.
• Use derived tables to reduce complexity of queries to be written by users or in place of views or procedures. A note of caution here: Use derived tables sparingly. If you have access to the database or DBA and can get views or tables created for the same purpose, go with it rather than using derived tables. This is not only to push the logic and work closer to the database, but also to take care of the performance and maintainability aspects. Exceptions to this include cases where your derived table may include a prompt which would restrict the number of rows returned and thus improve performance over a conventional view.
• Reuse code with @Variable. Reuse interactive objects with @Where (if you use them at all).
• Use @Prompt syntax for conditions and interactive objects where input values are likely to change or absence of prompt would lead to inaccurate values or unacceptable query response times. Also make sure regularly used conditions e.g. current year / latest date should not have prompts to avoid annoying users.
• “To limit the number of objects created to avoid user confusion, build interactive objects with @Prompt syntax followed by additional OR clause to include “”All”" condition.

E.g. ‘ALL’ IN @Prompt(‘Enter Value or ALL’,'A’, ‘Class\Object’,multi,)

OR

Table.Column IN @Prompt(‘Enter Value or ALL’,'A’, ‘Class\Object’,multi,)”

Universe design: resolving join and performance problems

• To resolve a chasm trap, define a context for each table at the “many” end of the joins.
• To resolve a fan trap, create an alias table for the table producing the multiplied aggregation. Create a 1:1 join between the original and the alias tables. Modify the select statement to use the columns from the alias table instead of the original table.
• Use of contexts should be evaluated w.r.t. use of aliases for resolving join issues, to take care of maintainability of code.
• Integrity checks on the universe structure, parsing of objects, joins, contexts, detecting loops etc is mandatory. If you wish to use Business Objects to help you detect fan traps or chasm traps – you must set the cardinality on the joins. Do not rely on BO to suggest the cardinality – this is often erroneous, based on the records sample that BO fetches for each table.
• Uncheck the “Multiple sql statements for each measure” option in universe parameters, if this is not required for resolving any join problems. This option should be checked if the measures being retrieved in the same query involve different tables. “Prevent Cartesian product” should be checked, as should there be limits placed on the number of records returned and the time for the sql connection – to prevent runaway queries which can bring the database down to its knees and cause an outage for all users.

Universe design: optimization / miscellaneous

• Use shortcut joins wherever possible to reduce number of tables used in a query
• Use aggregate tables /materialized views with aggregate awareness set up to improve query performance
• Use keys instead of labels where possible to take care of index awareness benefits of performance and uniqueness
• Use the JOIN_BY_SQL parameter to shift process from BO server to database wherever the bottleneck for performance is the BO server and the database performance is optimal.
• Update the .prm files to enable access to custom SQL functions and improve help text
• Do not use derived tables instead of aggregate tables.
• Turn off LOVs for all dimension and detail objects that are redundant or not required. This prevents performance problems when users inadvertently click on the “Values” and the query sets to return all the IDs or other irrelevant data.
• Consider using linked universes with a master kernel universe to ensure consistent dimensions across multiple universes

This list is certainly not an exhaustive one – but a work-in-progress. I’d update it as and when I compile more; meanwhile if you feel anything has been left out, drop in a line.

There is a well known adage that if you keep doing the same thing and expect different results, that is a sure sign of idiocy.  In the BI world too, we come across several instances where people take it for granted that the ‘BI tool’ will magically generate insight and spur ‘intelligence’ rather than ‘idiocy’. Yet the very practices of reporting the same measures, or of creating reports for metrics just because they are now made available by the tool, without sparing any ‘intelligence’ into what will generate insight is a major cause  of failures of BI.  Most of the leading commercial BI products are expensive and cost a lot of money in maintenance and support, so it is rather important to understand how to design the proper metrics and KPIs (key process indicators) which would generate insight. Even more important is to have a process focus and a general idea of the basics of statistical process control, in order to make sure that the right decisions are made and resources are spent on processes and strategies where they would have the most impact.

Statistical Process Control (SPC) is quite well known in the manufacturing industry and also in software engineering. In effect, it applies rules of statistics to the processes that are followed to predict whether a process is stable (and therefore in control) and its output is predictable or not and how to identify out-of-control processes and take corrective measures. Quality aids like causal analysis done using brainstorming/ nominal group techniques/ Ishikawa diagrams or fish-bone analysis are helpful in analyzing outliers and reasons of deviation from control limits. A substantive discussion of SPC and quality process areas is not possible in this post so I’ll just touch upon some concepts concisely.

PDCA – Plan-Do-Check-Act cycle, proposed by economist William Shewhart and later by quality guru Dr. Edward Deming. This is the foundation of the management and feedback cycle underlying any software engineering process.

Control limits – Any process which follows the Gaussian normal distribution would have a normal bell-shaped curve and be subject to control limits. The stability of the process can be gauged by the outliers (number and pattern of data points falling outside the control limits).

Causes of deviation: Outliers indicate deviation from a stable and predictable process. Causes of deviation could be due to special causes or common causes. Common causes are like background noise and may be present in stable processes. Special causes must be removed and steps taken to prevent their occurrence to bring a process under control. Common causes may be reduced to have a sharper curve with a narrower band of control limits and have greater control on the process.

Control Chart (Image courtesy: Wikipedia)

Users of BI tools haven’t tapped into the power of SPC to gain insight and control operational processes to the extent possible. There is even danger of damaging with a stable and in-control process due to tinkering with the process based on common-cause variation observed in operational reports. Part of the reason for SPC not gaining sufficient currency is that business analysts are not trained in the basics of SPC or quality processes like DAR (defect analysis and resolution) but mostly it is due to there not being any BI product in the market so far which allows easy use of SPC analysis. It is only of late that vendors like SAP-Business Objects have come out with specific SPC modules and predictive analytics in the BI product marketplace.

And so the time has never been better for start-ups with innovative technologies. From the self-tuning column databases of Vertica to the data compression technologies of Infobright and the lock-free database from Ants Software , there is renewed interest in reinventing the RDBMS to optimize performance.

The competition has already hotted up in the high-end data warehousing segment with the introduction of appliances. With both Netezza and DATAllegro gaining traction rapidly, incumbent Teradata is feeling the heat.

Another entrant to the Web 2.0 database race is Database as a service (DBaaS). With Google opening up its cloud computing platform and making available Bigtable , and Microsoft offering a beta release of SQL Server Data Services (SSDS), the incumbent Amazon’s S3 and SimpleDB is getting some competition.

The incumbent conventional RDBMS vendors like Oracle are not resting on their laurels.
With its aggresive acquisition strategy, Oracle has acquired Sleepycat’s open-source embedded database BerkeleyDB in early 2006 and now licenses it commercially. With its other acquisition (2005) – TimesTen, now integrated as an in-memory database, Oracle has been targeting SaaS ISVs to sell its Oracle SaaS platform.
IBM has already moved into the BI infrastructure segment since 2005 with its pSeries Data Warehousing Balanced Configuration Units and evolved on to the Infosphere Balanced Warehouse applicances. Apart from SSDS, Microsoft has designed SQL Server 2008 with data warehouse features like the star join query, SSIS persistent lookups or the MERGE SQL statement. Most of the biggies (e.g. BOBJ acquiring Inxight) have acquired small companies with technologies to search and analyze unstructured data.

Yet with all this “new and improved”, the newer and innovative technologies are gaining a lot of traction at least in the data warehousing/ETL space. It remains to be seen if the trend catches on in the OLTP segment as well. But for now, the staid and bland database segment is on fire.