What’s new in SAP BusinessObjects Enterprise BI 4.0 platform or BOXI 4.0 aka Aurora?

SAP BI 4.0 release (codenamed Aurora) has been the first major release of the BI platform since SAP acquired BusinessObjects. In this release, the semantic layer (universe layer for the uninitiated) has been re-worked completely to expose all business data under a single umbrella. The self-service BI portal (aka Infoview) has been revamped with a new AJAX based design and providing quicker and easier access to content. Publishing and distribution of BI content to mass audience has been made easier. There are also improvements to the lifecycle management (LCM tool) and platform administration (CMC, CCM) from a single console. This is in a nutshell are the changes that Aurora or SAP BO 4.0 bring, allowing BI content to be delivered across different channels ranging from the browser (BI Launch Pad, SharePoint, SAP NetWeaver Portal, Java Portal) to desktop (widgets), MS-Office and mobile.

In the following section I’ll try to cover the major changes that have been effected in the following products:

Semantic Layer - A new tool, Information Design Tool enhances the Universe Designer. The universes created by this tool are identified by the .UNX file extension and allow connections to multiple data sources.

The universe designer is still there. Renamed as universe design tool, it allows creating single data source universes (.UNV file extension) as before.

Conversion of previous universe .unv versions is supported only for relational universes created in previous universe designer versions and not possible for OLAP universes or universes based on stored procedures or Data Federator data source.

No authentication is required to start the information design tool. Users can create and edit unsecured resources (data foundations, business layers, connections) in local projects and publish them to the repository to make them secure.

Connections to relational data sources, OLAP data sources as well as SAP NetWeaver BEx query can be created, be local (saved locally as .cnx files) or secured (stored in the repository).

The newly named “Data foundations” are analogous to the schema browsers in Universe Designer. They contain the schema of relevant tables and joins from one or more relational databases that are used as a basis for one or more business layers.

The business layer is the universe metadata. Depending on the type of data source for the business layer, several types of objects e.g. folders, dimensions, analysis dimensions, measures, attributes, filters, hierarchies (OLAP only) can be created and edited in the business layer.

Search - enhancements include a new enhanced search engine allowing search by document attributes as well as content. Search results can be filtered and refined easily and the search GUI is integrated in the BI launch pad

There are also enhanced options through the OpenSearch API which enables integration with other search systems like Google Search Appliance, Microsoft SharePoint portal and NetWeaver Enterprise Search.

BI Portal - includes a new look re-designed web portal (InfoView) now called the BI LaunchPad providing a rich new user experience. It provides quick and easy access to BI applications and search, a handy list of recently used reports, scheduled documents, alerts etc., multiple tabs and pinning options, and a reduction in the manual steps for common tasks like:

• Ability to create new folder while Saving
• Schedule and Send To actions in Document viewers
• Auto-refresh in History page

Alerting, Monitoring & Auditing - The alerting framework allows triggering of alerts based on events (schedule completion, ETL completion, system monitoring etc.) or data conditions as also reactions to those events e.g. scheduling report to run or send notification message. Subscription to alerts is made easier with a consistent workflow, allowing notification emails or messages in the BI Launch Pad.

New monitoring applications are available to keep tabs on system health and performance (server metrics, custom probes, user-defined watch conditions, visualization dashboard in CMC) and integrate with infrastructure monitoring tools like Tivoli and SAP Solution Manager.

Auditing enhancements include simplified system wide configuration, auto-purging of old data and an enhanced audit store schema which simplifies reporting and application development.

Lifecycle Management - The LCM console replaces the import wizard. It allows connection override in bulk mode automatically, supports version control and rollback, is audit-able and provides scripting facility.

Upgrades and deployments - A new optimized upgrade management tool is provided, combining the best of Import Wizard and Database Migration tool in XI 3.x. This caters to one-click full upgrade or selective incremental upgrades, allowing direct upgrade from XI R2 SP2 or later. There’s enhanced scalability in deployment with virtualization and 64-bit support.

Review of the BT Summit – Cloud computing, SOA and BI tracks

I attended the Business Technology Summit in Bangalore last week – 3^rd and 4^th November. There were 3 tracks on cloud computing, Service Oriented Architecture and Business Intelligence, and I chose a mix of sessions across each.

Overall impression: The BT Summit was heavily focused on cloud computing with half of second day having a deep dive into Amazon’s EC2 cloud offering, and several keynotes. SOA and web services, REST and similar architectural sessions were interspersed but definitely not a first-class citizen. BI came a poor third with a poor choice of sessions, and more of a rehash of what is out there for everyone, rather than something on the cutting-edge including use of appliances and columnar databases, as also in-memory databases and use of Flash and AJAX for interactive BI front-ends.

Session-wise review: (Speaker profiles available here). I was able to speak to and ask questions of Vinod Kumar, Vijay Doddavaram, Abhinav Agarwal and Dr. Bob Marcus.

Keynotes:

• Smart services, bright future by Howard Charney

Probably the highlight of the keynotes, this was a pep-talk about the inevitable interconnected future with smart products and services and for good measure Charney threw out some statistics on broadband growth and bandwidth usage and India’s readiness and potential in the scheme of things.

• Give cloud a chance by Ramkumar Kothandaraman

The worst of the lot – this started by comparing the spectrum of offerings in the cloud from Amazon’s DIY EC2 and AWS, Google appengine and apps to Microsoft’s Azure and ended up as a promo touting Azure as the best buy among all.

• Cloud computing: State of the union address by Shouvick Mukherjee

A very good keynote, focusing on what makes sense to migrate to the cloud and what doesn’t, what are the hidden costs, the myth of unlimited elasticity in the cloud and what Yahoo is doing to use open source software like Hadoop and Hive for cloud computing. In the short time span, Shouvick also tried to address some of the other considerations – including re-architecting existing applications, availability, data storage and movement considerations.

• Managing IT in turbulent times by Som Sharma

This post-lunch keynote by Sharma was a rambling talk on how technology keeps redefining our lives, and why it is important to think outside-the-box. He used the example of the iPhone to illustrate how such thinking has the potential to alter the established rules of the industry and redefine it as we know it.

• A New world to protect by Nils Puhlmann

Puhlmann provided the security perspective on how easy it to break/hack enterprise systems and how anti-virus and anti-spyware are always playing catch-up, the entire economy that is spawned by the “bad-guys” in technology and why our systems need to be smart and be built from the ground-up for security rather than as an afterthought. He provided valuable insights into what questions we should ask ourselves as we embrace cloud computing, the changing technology landscape making it easy for consuming information but easier still for the security breachers. Puhlmann concluded by suggesting it may be worthwhile including a level of risk assessment and mitigation, and collaboration with ethical hackers, rather than trying to do the impossible of removing all security threats.

• Moving towards a virtual enterprise by Srinivas Varadarajan

Barely managed to sleep through it – this one talked about moving towards a virtual enterprise – with a focus on virtualized architecture, including cloud computing. As boring as they can get.

Other sessions:

• SOA, Composite Applications, and Cloud Computing: Three pillars of a modern technology solution by Robert Schneider

Robert  Schneider presented the different facets of SOA, Composite applications (superset of mash-ups) and Cloud computing and contrasted them regarding the time to yield benefits, the maturity of the vision, involvement and buy-in from business and where they lie in the tactical-strategic plane. There wasn’t anything regarding why we are stuck with these three for a modern technology solution, or what other paradigms are out there beyond the old-world enterprise computing framework, possibly due to time constraints.

• Self-service analysis and the future of Business Intelligence by Vinod Kumar

A lot of the BI folks were waiting for this, as Vinod performed the Project Gemini (Office 2010 Excel and PowerPivot) demo live for the first time in India, with several folks, including yours truly, sitting on the stairs. [We have had to rely on Youtube videos and MS Office 2010 preview videos earlier]. The demo was impressive fetching over 13 million records into Excel using a standard DDR laptop, using compression and in-memory technologies. The bigger question around unleashing another round of Excel hell went unanswered due to time constraints, however the presentation probably hinted at Microsoft’s vision of “self-service BI” or so-called “underground-BI” as the power-users of Excel (estimated at 2M worldwide, at 4% of the Excel user base) have been doing. Microsoft’s strategy around pushing SharePoint adoption in the Enterprise was made clear tacitly with SharePoint being the only “portal” to publish and share BI analysis (typical size of these Excel spreadsheets is upwards of 200MB) with other users in the enterprise.

• Designing and Implementing RESTful web services by Eben Hewitt

Eben Hewitt started off with a very brief comparison between SOAP (Simple Object Access Protocol) modeled more on the lines of RPC (Remote Procedure Call) and REST (Representational State Transfer) and clarified that REST is more an architectural style rather than specifications. The remainder of the talk delved into details of implementation of REST – usage of simple ‘verbs’ and complexity in ‘nouns’, uniform interface, using named resources, java REST frameworks like Jersey, MIME types – JSON, XML, YAML and HTTP operations supported – POST, GET, PUT and DELETE.

• Business Intelligence project execution on a shoestring by Rajesh Ramaswamy

I attended with some expectations on how a BI project can be executed possibly with open-source or free software like MySQL/Postgres, Pentaho/Talend, Jaspersoft/MicroStrategy reporting suite etc., but was highly disappointed by the presentation. Ramaswamy spoke on BI usage, barriers to BI adoption, costs of BI implementation and spewed statistics like m&m’s with cursory references to Forrester, Gartner and “research studies”, but there wasn’t anything tangible on how to go about a project execution except for some common-sense talk on “evaluating options” between open-source and licensing costs, offshoring and outsourcing, RDBMS vs. analytica databases and appliances etc.

• Business Intelligence – Leveraging and Navigating during current challenging times by Vijay Doddavaram

Vijay spoke of the current global economic downturn and how it had taken everyone unawares during the downturn as well as when the current quarter the tide seems to have returned. With the example of a fictitious company in China, he illustrated the importance of trade-off between tactical and strategic decision making and whether and how business intelligence can make a difference in either a downturn and the upswing (whether it is a U, V, or a W curve). Thought-provoking, one couldn’t help feel that BI software has not yet eliminated the “intelligence” that people bring to the table, and made a distinct point about the “human analysis/intelligence” against the out-of-the-box actionable-intelligence marketed by the BI vendors. It would have been interesting to prolong the discussion, with a focus on the “predictive-analytics” offerings in the market (from SAP, WPC, SPSS and the open-source R etc.), we had once again run out of time, and it was the last session of the day as well.

• Towards a unified Business Intelligence and Enterprise Performance Management Strategy by Abhinav Agarwal

Abhinav is from Oracle and he used this session to basically present the BI and EPM strategy of Oracle. Refreshing when contrasted with the usual Oracle marketing hype, Abhinav made it a point to stress the difficulty of delivering best-in-breed products due to numerous acquisitions and the inevitable integrations compared to the disruptive start-ups which could be one-trick ponies but nevertheless manage to push the technology envelope. Most of the session focused on Oracle BI server offering and the roadmap of integrating with the Fusion middleware, and brief touchpoints on the capabilities of the Oracle BI server: federated queries (acquired from nQuire, which Siebel systems had acquired, prior to being bought by Oracle), and real-time updates, including Oracle RTD (Real-time Decisions) and the segregation of the BI and EPM software offerings.

• 10 Things software architects should know by Eben Hewitt

I was able to attend part of it, but for the most part- the bottomline of this talk was the trade-offs architects need to make and understanding there may not be a “solution” to a problem, it may just be “moving the problem” – the idea that each “solution” brings its own issues and tradeoffs into the picture. Being more focused on java APIs and cloud computing frameworks, it could have done better with something related to networks and database architecture in general for audience to relate better (for most of my time, I couldn’t relate to a BI applications and data-warehousing infrastructure).

• Cloud initiatives and standards roadmaps by Dr. Bob Marcus

Being late from an overcrowded dining hall, I was able to attend part of this. Bob spoke of the various public and private initiatives including those from the federal government, NASA Nebula and made the distinction early on between the types of offerings on the cloud: SaaS (Software as a service), IaaS (Infrastructure as a service) and PaaS (Platform as a Service). He mentioned in passing the data.gov and apps.gov initiatives of the Obama administration as also about RACE (Rapid Access Computing Environment) from the Dept. of Defense – Defense Information Systems Agency.

• Building Enterprise Dashboards by Vivek Khurana

Vivek Khurana did a very short presentation to an overflowing hall on clichéd but nevertheless important aspects of information visualization while designing dashboards: clutter vs. simplicity, proper designing of KPIs, importance of delivery to mobile devices, and learning from news aggregation sites and portals on presentation.

• Implementing Enterprise 2.0 using Open Source products by Udayan Banerjee

Banerjee did a great job of presenting what his vision of implementing Enterprise 2.0 in NIIT was – implementing SLATES (coined by Andrew McAfee) – Search, Links, Authoring, Tags, Extensions and Signals. Within half-an-hour he navigated us through using open-source products for collaboration using blogs and wiki (MediaWiki), using single-sign-on with enterprise databases, using links and tag clouds and integrating Search as well as implementing a text-based instant messenger.

• ECM-CMIS and the emergence of standards by Alan Pelz-Sharpe

I had missed the earlier session of Alan on lessons learnt using SharePoint, so I made it a point to attend the last of this at the summit – even though it meant I had no clue sometimes of what was being talked about! Alan spoke of the emergence of the multi-vendor CMIS standard for Enterprise Content Management – the various facets of ECM – from digital and media assets, email archiving, Internet content, web analytics, document types, rich media and the problems with the earlier Java standards like JSR 170 – most notably the absence of support from Microsoft. He also spoke about the vendor landscape and a 9-block rating similar to Gartner’s magic quadrant, plus various other important standards, including XAM – eXtensible Access Method – a storage standard developed by SNIA (Storage and Networking Industry Association)

Presentation files: Most presentation files are available here. You’ll need to register though to download.

- Maloy

Agile Development for BI

How can you reduce development costs and improve software reliability and accuracy at the same time? How can you make IT work together with Business while architect-ing your BI applications? If these goals sound contradictory and difficult to achieve, then Agile development may well fit the bill. Indeed in numerous BI projects, one or the other flavor of Agile is used to attain these very goals.

Defining Agile
There are several Agile development methodologies available:
• eXtreme Programming (XP)
• SCRUM
• Feature-Driven Development (FDD)
• Crystal Clear
• Dynamic Systems Development Method (DSDM)
• Adaptive Software Development (ASD) and more…

At the core of any flavor of Agile development methodology is the iteration, which may last from 1 to 4 weeks (one unit of time) to develop a piece of the software. Each iteration is treated as an entire software project with its associated planning, design, coding, testing and documentation tasks.

What is it about Agile development which makes it particularly suitable for data warehousing and business intelligence projects?

* Agile emphasizes on communication be it through meetings (be it through the phone, VOIP, web or IM) over written documents. The idea is to get the user involved much early in the development process and incorporate their feedback, so as to minimize the risk of developing faulty software. For organizations adopting BI, very often users are clueless about the systems to build, the technology to use or even the range of analysis they require. Products are often bought after effective sales pitches from vendors and left to IT to deploy and architect. In such cases, IT can use Agile methodologies like DSDM, SCRUM or ASD to flesh out the requirements and deliver BI which actually provides insight rather than building a monolithic and unreliable data warehouse difficult to query and administer.

* Agile gels well with the evolutionary approach required for a data warehousing / BI lifecycle. Requirements change over time, and the iterations of the Agile methodology (with database re-factoring and evolutionary data modeling ) is more efficient in capturing these changes than the classical waterfall approach.

* Proof of the concept, technology and architecture is crucial to justify continued investment in DW/BI projects, especially on the enterprise scale. This is simpler and easier to do with Agile.

* Agile imbibes every member of the project team with extra responsibilities, making them owners of discrete functions and helps the project manager overcome the ‘taskmaster‘ stereotype and concentrate on being a leader or a visionary.

BI is essentially gaining competitive edge by insight into your business through lagging (measures) and leading (predictive model-based) metrics, which allows feedback cycles and restructuring of processes (Plan-Do-Check-Act Deming cycle). This essentially involves cooperation and teamwork across functions to model and understand the multi-faceted perspectives. Teamwork being the foundation of Agile, it is a natural fit for projects in BI and data warehousing.

~biguru

SaaS BI – Software as a service model in Business Intelligence

SaaS has taken off in a big way in the past few years. And BI has not been lagging behind. For leading vendors like BOBJ (an SAP company) or Cognos (an IBM company), it’s going to be close to 2 years now since they started their SaaS BI.

What exactly are the advantages of SaaS over the traditional approach?

Benefits are aplenty, from zero costs of purchasing hardware, hiring of key IT personnel like system administrators and DBAs, to minimal implementation and maintenance costs. With CRM being made affordable to SME (small and medium-sized enterprises) by salesforce.com (note the huge success of its AppExchange), and the fact that large enterprise adoption of BI was complete, meant that the only scope for future growth in BI would come from upgrades of platforms (analogous to Microsoft’s Windows operating system – think of BOBJ upgrading from 6.5 to XI) or from expansion in the SME space with better offerings.
In fact, with Google distributing its Apps, Serena jumping on to the bandwagon with its project and portfolio management (PPM) software, and even Tata Consultancy Services launching its own ITaaS (IT as a service), SaaS is almost confirmed to be the software delivery model of the future.

The marginal cost of acquiring SME customers is reduced due to the nature of the SaaS service, and this has encouraged more and more BI companies to venture into exploring this delivery model for BI. There are niche players well entrenched in this SaaS BI space: LucidEra, Oco, Information Builders and PivotLink – and a few of them provide the entire range of BI – including ETL and not limited only to reporting. But now the bigger fish have also ventured into this space. Going by the success of BOBJ’s crystalreports.com or its Information OnDemand (a project which biguru was involved in developing) and IBM Cognos Now!, there seems no doubt over the future direction that BI delivery is headed. Did I mention almost all of these can work together with Salesforce?

As much as SaaS BI is redefining delivery mechanisms from the perspective of the vendors, it also involves a paradigm shift for the customers. Finally, it is less a question of technology, than it is about the benefits to the business, the ROI and focusing on the core competency of the enterprise. In a sense, SaaS ‘outsources’ all of the IT to the vendor, leaving business to deal with issues like implementation and managing requirements. It is also a pointer to the maturing of the industry – from discovering requirements to providing relevant information and insight.

The technology implications for the vendors are daunting. For once, they need to expand in domains outside their core areas, from maintaining data centers to ensuring bug-free roll-outs, even though they’ll also have the luxury of not having to maintain multiple version stacks. On the web, users become far more demanding, and are far less willing to break their heads while figuring out why something won’t work due to a missing dll or patch etc.

Improvements (both aesthetics, ease of use and navigation) in the user interface, as well as performance requirements have already seen a sort of standardization on Adobe Flash as the medium of delivery for dashboards. Even though AJAX/Javascript is also in the picture, the complexity of coding and longer cycle times seems to be tilting the scales in favor of Flash. As user experience gets paramount, BOBJ has launched its latest visualization offering Xcelsius 2008 (version 5) which also includes an SDK for interoperability with Adobe Flex, to build sites like Information OnDemand.
With BI getting more business-process centric, SAP is now focused on embedding BI capabilities in its range of offerings. Business Objects is currently tinkering with TAWS (Take Action as a Web Service) in its Labs, which would allow its dashboards to use web services to plug into the service-oriented-application-architecture. With respect to hot-plugging into the SOA architecture of BI, however, Oracle has made great strides and even IBM Cognos is better placed.

The future of SaaS BI and enterprise BI SOA seems interesting indeed.

Trends in Business Intelligence

By definition Business Intelligence (BI) has been about making decisions based on information obtained from meaningful data. At one point of time we had terms like decision support systems (DSS) to define technology which has finally evolved into BI as we know it now. Or has it? Somewhere along the line, we became entangled in the technology aspect of it and with all the buzzwords of data warehousing, data mining, dimensional modeling, data marts, CRM and SCM it is not difficult to see why. It is only of late that the focus has moved back to the “decision making” aspect of BI rather than focusing on the technology per se.

In the Gartner BI Summit which was held in the first quarter of the 2007 calendar year, we got to see the focus shifting from traditional data warehousing and OLAP applications to the technology which ties together all the OLAP, reporting and query tools along with performance management applications and dashboard-ing tools. The net effect is the coming together of information, analysis and performance management to assist decision making, the ultimate goal of BI.

There has been a lot of M&A activity going on in the data warehousing-BI-performance management space throughout this year. However this wave of consolidation (the last time we saw such a thing was in 2003) hasn’t yet pointed to a specific direction in which technology will move and decide the future of BI. Rather what can be made of this M&A boom is the rush to acquire “hot” technologies which can be bet on to power the future. Three distinct categories of applications can be found in this trend:

1. Traditional reporting and OLAP applications. These include all flavors of OLAP including the Business Objects popularized relational OLAP as well as the various multi-dimensional models,notably Microsoft Analysis Services. This set of applications has become the “hygiene factor” in the BI industry
2. Strategy or Performance driven BI applications. These have variously been called strategy management applications or performance management applications and over the last few years this has been a major growth driver for BI as the traditional analyst-reporting-tools market has matured and scope of growth has decreased. Performance management is the new USP to get that cutting edge over competition.
3. Workflow BI applications. An offshoot of dashboard tools as well as the recent Web 2.0 buzz of using AJAX, web services, Flash and similar technologies, this set of applications appears to position themselves midway between traditional enterprise applications and BI reporting. Embedded within workflows or processes, these applications add the ability to “take action” based on a decision made on the basis of the information provided by BI.

With the latest acquisitions of Business Objects by SAP and Cognos by IBM, it remains to be seen how these technology trends shape the industry. The next couple of years promise to be interesting on a scale we haven’t witnessed before in this space.