Thrive or Survive – the changing rules for databases

Not since the late seventies, when Larry Ellison’s Relational Software Inc. (RSI) turned out the first commerically available RDBMS – Oracle, has there been such rapid changing of the rules (read disruption) in the database industry.
With Web 2.0 pushing enterprise adoption, and the ensuing information explosion in the maze of audio, video, data and ever-growing data warehouses, it seems that the conventional relational database systems are growing tired. With estimates of unstructured data being anywhere between 80% to 95% of all business data, and the ever changing requirements imposed by Web 2.0 – storage of pictures, audio and video, the demands being made on conventional RDBMS technology are monstrous. With the load window available being fixed due to availability and uptime requirements, the ever increasing data to be loaded into data warehouses, the bulking-up of the data due to usage of XML based formats, conflicting requirements of SQL and XQuery, the database is also being challenged by the demands of business intelligence.

And so the time has never been better for start-ups with innovative technologies. From the self-tuning column databases of Vertica to the data compression technologies of Infobright and the lock-free database from Ants Software , there is renewed interest in reinventing the RDBMS to optimize performance.

The competition has already hotted up in the high-end data warehousing segment with the introduction of appliances. With both Netezza and DATAllegro gaining traction rapidly, incumbent Teradata is feeling the heat.

Another entrant to the Web 2.0 database race is Database as a service (DBaaS). With Google opening up its cloud computing platform and making available Bigtable , and Microsoft offering a beta release of SQL Server Data Services (SSDS), the incumbent Amazon’s S3 and SimpleDB is getting some competition.

The incumbent conventional RDBMS vendors like Oracle are not resting on their laurels.
With its aggresive acquisition strategy, Oracle has acquired Sleepycat’s open-source embedded database BerkeleyDB in early 2006 and now licenses it commercially. With its other acquisition (2005) – TimesTen, now integrated as an in-memory database, Oracle has been targeting SaaS ISVs to sell its Oracle SaaS platform.
IBM has already moved into the BI infrastructure segment since 2005 with its pSeries Data Warehousing Balanced Configuration Units and evolved on to the Infosphere Balanced Warehouse applicances. Apart from SSDS, Microsoft has designed SQL Server 2008 with data warehouse features like the star join query, SSIS persistent lookups or the MERGE SQL statement. Most of the biggies (e.g. BOBJ acquiring Inxight) have acquired small companies with technologies to search and analyze unstructured data.

Yet with all this “new and improved”, the newer and innovative technologies are gaining a lot of traction at least in the data warehousing/ETL space. It remains to be seen if the trend catches on in the OLTP segment as well. But for now, the staid and bland database segment is on fire.

Acquisitions in BI – End of the best of breed?

Last time I talked about the trends in the BI space. Let’s look into one of those trends – M&As and how it is shaping BI as we know it.
A recent BusinessWeek article claimed the end of the best-of-breed approach to BI with the demise of the major pure-play BI vendors like BOBJ now owned by SAP thanks to a friendly takeover or Hyperion acquired by Oracle or the latest – IBM’s acquisition of Cognos. While it is true that some of the major players have been acquired by the bigger businesses, there is no reason as to why the mid-size companies should go out of business.
For one, with the recent convergence of BI into the SOA (service oriented architecture), most pure-play vendors have hastened to upgrade their offerings, which means that there isn’t any single vendor which goes the entire distance in the BI-SOA convergence. Business Objects which introduced its web services add-on QAAWS for its dashboarding product Crystal Xcelsius, has now added support for embedding Xcelsius in its latest release of reporting products – Crystal Reports 2008. Yet to be acquired pure-play BI vendors like Actuate and MicroStrategy are adding SOA interoperability in product upgrades.
As is common in the technology industry, something disruptive always keeps happening. The latest trend riding on this disruption is the use of data warehousing appliances and the use of in-memory databases and in-memory calculations. From Netezza to QlikTech to Tableau, there is plenty happening in the BI innovations scenario which run counter to the forces set in motion by the big four (IBM, SAP, Oracle and Microsoft).
The most convincing force against the consolidation spree is the fact that different vendors are better at different aspects of the entire BI space, so to build competency across the entire BI spectrum would require multiple acquisitions in a string-of-pearls approach along with subsequent costly integration and the resulting delays. It remains to be seen how IBM integrates Cognos’ offerings with its existing product lines, even as SAP has given BOBJ an independent run.

Best of breed works because there are investments with licensing of pure-play technologies which work with all the databases including IBM’s DB2 and Oracle’s eponymous database, as well as the middle-ware from both SAP and Oracle, both of which have Java as the underlying open standard. With Oracle itself licensing Informatica in its data warehouse offerings, it is easy to see that it is not the end of the road for the smaller players as long as they stick to innovation, the technology credo.
Add to this mix the open source products from JasperSoft, Pentaho and Talend, and, best of breed does start sounding like the best value for money.