Thrive or Survive – the changing rules for databases

Not since the late seventies, when Larry Ellison’s Relational Software Inc. (RSI) turned out the first commerically available RDBMS – Oracle, has there been such rapid changing of the rules (read disruption) in the database industry.
With Web 2.0 pushing enterprise adoption, and the ensuing information explosion in the maze of audio, video, data and ever-growing data warehouses, it seems that the conventional relational database systems are growing tired. With estimates of unstructured data being anywhere between 80% to 95% of all business data, and the ever changing requirements imposed by Web 2.0 – storage of pictures, audio and video, the demands being made on conventional RDBMS technology are monstrous. With the load window available being fixed due to availability and uptime requirements, the ever increasing data to be loaded into data warehouses, the bulking-up of the data due to usage of XML based formats, conflicting requirements of SQL and XQuery, the database is also being challenged by the demands of business intelligence.

And so the time has never been better for start-ups with innovative technologies. From the self-tuning column databases of Vertica to the data compression technologies of Infobright and the lock-free database from Ants Software , there is renewed interest in reinventing the RDBMS to optimize performance.

The competition has already hotted up in the high-end data warehousing segment with the introduction of appliances. With both Netezza and DATAllegro gaining traction rapidly, incumbent Teradata is feeling the heat.

Another entrant to the Web 2.0 database race is Database as a service (DBaaS). With Google opening up its cloud computing platform and making available Bigtable , and Microsoft offering a beta release of SQL Server Data Services (SSDS), the incumbent Amazon’s S3 and SimpleDB is getting some competition.

The incumbent conventional RDBMS vendors like Oracle are not resting on their laurels.
With its aggresive acquisition strategy, Oracle has acquired Sleepycat’s open-source embedded database BerkeleyDB in early 2006 and now licenses it commercially. With its other acquisition (2005) – TimesTen, now integrated as an in-memory database, Oracle has been targeting SaaS ISVs to sell its Oracle SaaS platform.
IBM has already moved into the BI infrastructure segment since 2005 with its pSeries Data Warehousing Balanced Configuration Units and evolved on to the Infosphere Balanced Warehouse applicances. Apart from SSDS, Microsoft has designed SQL Server 2008 with data warehouse features like the star join query, SSIS persistent lookups or the MERGE SQL statement. Most of the biggies (e.g. BOBJ acquiring Inxight) have acquired small companies with technologies to search and analyze unstructured data.

Yet with all this “new and improved”, the newer and innovative technologies are gaining a lot of traction at least in the data warehousing/ETL space. It remains to be seen if the trend catches on in the OLTP segment as well. But for now, the staid and bland database segment is on fire.

SaaS BI – Software as a service model in Business Intelligence

SaaS has taken off in a big way in the past few years. And BI has not been lagging behind. For leading vendors like BOBJ (an SAP company) or Cognos (an IBM company), it’s going to be close to 2 years now since they started their SaaS BI.

What exactly are the advantages of SaaS over the traditional approach?

Benefits are aplenty, from zero costs of purchasing hardware, hiring of key IT personnel like system administrators and DBAs, to minimal implementation and maintenance costs. With CRM being made affordable to SME (small and medium-sized enterprises) by salesforce.com (note the huge success of its AppExchange), and the fact that large enterprise adoption of BI was complete, meant that the only scope for future growth in BI would come from upgrades of platforms (analogous to Microsoft’s Windows operating system – think of BOBJ upgrading from 6.5 to XI) or from expansion in the SME space with better offerings.
In fact, with Google distributing its Apps, Serena jumping on to the bandwagon with its project and portfolio management (PPM) software, and even Tata Consultancy Services launching its own ITaaS (IT as a service), SaaS is almost confirmed to be the software delivery model of the future.

The marginal cost of acquiring SME customers is reduced due to the nature of the SaaS service, and this has encouraged more and more BI companies to venture into exploring this delivery model for BI. There are niche players well entrenched in this SaaS BI space: LucidEra, Oco, Information Builders and PivotLink – and a few of them provide the entire range of BI – including ETL and not limited only to reporting. But now the bigger fish have also ventured into this space. Going by the success of BOBJ’s crystalreports.com or its Information OnDemand (a project which biguru was involved in developing) and IBM Cognos Now!, there seems no doubt over the future direction that BI delivery is headed. Did I mention almost all of these can work together with Salesforce?

As much as SaaS BI is redefining delivery mechanisms from the perspective of the vendors, it also involves a paradigm shift for the customers. Finally, it is less a question of technology, than it is about the benefits to the business, the ROI and focusing on the core competency of the enterprise. In a sense, SaaS ‘outsources’ all of the IT to the vendor, leaving business to deal with issues like implementation and managing requirements. It is also a pointer to the maturing of the industry – from discovering requirements to providing relevant information and insight.

The technology implications for the vendors are daunting. For once, they need to expand in domains outside their core areas, from maintaining data centers to ensuring bug-free roll-outs, even though they’ll also have the luxury of not having to maintain multiple version stacks. On the web, users become far more demanding, and are far less willing to break their heads while figuring out why something won’t work due to a missing dll or patch etc.

Improvements (both aesthetics, ease of use and navigation) in the user interface, as well as performance requirements have already seen a sort of standardization on Adobe Flash as the medium of delivery for dashboards. Even though AJAX/Javascript is also in the picture, the complexity of coding and longer cycle times seems to be tilting the scales in favor of Flash. As user experience gets paramount, BOBJ has launched its latest visualization offering Xcelsius 2008 (version 5) which also includes an SDK for interoperability with Adobe Flex, to build sites like Information OnDemand.
With BI getting more business-process centric, SAP is now focused on embedding BI capabilities in its range of offerings. Business Objects is currently tinkering with TAWS (Take Action as a Web Service) in its Labs, which would allow its dashboards to use web services to plug into the service-oriented-application-architecture. With respect to hot-plugging into the SOA architecture of BI, however, Oracle has made great strides and even IBM Cognos is better placed.

The future of SaaS BI and enterprise BI SOA seems interesting indeed.

Acquisitions in BI – End of the best of breed?

Last time I talked about the trends in the BI space. Let’s look into one of those trends – M&As and how it is shaping BI as we know it.
A recent BusinessWeek article claimed the end of the best-of-breed approach to BI with the demise of the major pure-play BI vendors like BOBJ now owned by SAP thanks to a friendly takeover or Hyperion acquired by Oracle or the latest – IBM‘s acquisition of Cognos. While it is true that some of the major players have been acquired by the bigger businesses, there is no reason as to why the mid-size companies should go out of business.
For one, with the recent convergence of BI into the SOA (service oriented architecture), most pure-play vendors have hastened to upgrade their offerings, which means that there isn’t any single vendor which goes the entire distance in the BI-SOA convergence. Business Objects which introduced its web services add-on QAAWS for its dashboarding product Crystal Xcelsius, has now added support for embedding Xcelsius in its latest release of reporting products – Crystal Reports 2008. Yet to be acquired pure-play BI vendors like Actuate and MicroStrategy are adding SOA interoperability in product upgrades.
As is common in the technology industry, something disruptive always keeps happening. The latest trend riding on this disruption is the use of data warehousing appliances and the use of in-memory databases and in-memory calculations. From Netezza to QlikTech to Tableau, there is plenty happening in the BI innovations scenario which run counter to the forces set in motion by the big four (IBM, SAP, Oracle and Microsoft).
The most convincing force against the consolidation spree is the fact that different vendors are better at different aspects of the entire BI space, so to build competency across the entire BI spectrum would require multiple acquisitions in a string-of-pearls approach along with subsequent costly integration and the resulting delays. It remains to be seen how IBM integrates Cognos’ offerings with its existing product lines, even as SAP has given BOBJ an independent run.

Best of breed works because there are investments with licensing of pure-play technologies which work with all the databases including IBM’s DB2 and Oracle’s eponymous database, as well as the middle-ware from both SAP and Oracle, both of which have Java as the underlying open standard. With Oracle itself licensing Informatica in its data warehouse offerings, it is easy to see that it is not the end of the road for the smaller players as long as they stick to innovation, the technology credo.
Add to this mix the open source products from JasperSoft, Pentaho and Talend, and, best of breed does start sounding like the best value for money.

Trends in Business Intelligence

By definition Business Intelligence (BI) has been about making decisions based on information obtained from meaningful data. At one point of time we had terms like decision support systems (DSS) to define technology which has finally evolved into BI as we know it now. Or has it? Somewhere along the line, we became entangled in the technology aspect of it and with all the buzzwords of data warehousing, data mining, dimensional modeling, data marts, CRM and SCM it is not difficult to see why. It is only of late that the focus has moved back to the “decision making” aspect of BI rather than focusing on the technology per se.

In the Gartner BI Summit which was held in the first quarter of the 2007 calendar year, we got to see the focus shifting from traditional data warehousing and OLAP applications to the technology which ties together all the OLAP, reporting and query tools along with performance management applications and dashboard-ing tools. The net effect is the coming together of information, analysis and performance management to assist decision making, the ultimate goal of BI.

There has been a lot of M&A activity going on in the data warehousing-BI-performance management space throughout this year. However this wave of consolidation (the last time we saw such a thing was in 2003) hasn’t yet pointed to a specific direction in which technology will move and decide the future of BI. Rather what can be made of this M&A boom is the rush to acquire “hot” technologies which can be bet on to power the future. Three distinct categories of applications can be found in this trend:

1. Traditional reporting and OLAP applications. These include all flavors of OLAP including the Business Objects popularized relational OLAP as well as the various multi-dimensional models,notably Microsoft Analysis Services. This set of applications has become the “hygiene factor” in the BI industry
2. Strategy or Performance driven BI applications. These have variously been called strategy management applications or performance management applications and over the last few years this has been a major growth driver for BI as the traditional analyst-reporting-tools market has matured and scope of growth has decreased. Performance management is the new USP to get that cutting edge over competition.
3. Workflow BI applications. An offshoot of dashboard tools as well as the recent Web 2.0 buzz of using AJAX, web services, Flash and similar technologies, this set of applications appears to position themselves midway between traditional enterprise applications and BI reporting. Embedded within workflows or processes, these applications add the ability to “take action” based on a decision made on the basis of the information provided by BI.

With the latest acquisitions of Business Objects by SAP and Cognos by IBM, it remains to be seen how these technology trends shape the industry. The next couple of years promise to be interesting on a scale we haven’t witnessed before in this space.