Turf War

As Microsoft targets the higher levels of the data warehouse pyramid and Teradata reaches down to smaller enterprises, we’ll soon be able to choose between two great BI solutions. Let the battle commence…

October was a great month for BI freaks, with Microsoft’s BI conference in Seattle and Teradata’s Partners conference in Las Vegas the following week. This juxtaposition is all the more fascinating because comparing and contrasting the messages from the two companies gave an interesting insight into the future of BI.

Microsoft
Last month I mentioned Microsoft’s announcement of Madison. In a nutshell, this is the result of crossing SQL Server with DATAllegro (the MPP data warehouse appliances provider). One of DATAllegro’s attractions for Microsoft was its modular technology. This makes it possible for the part of DATAllegro’s technology that offers impressive scalability to be run successfully on any relational database engine. Since the acquisition, Microsoft has grafted this technology onto SQL Server and the result will be launched in the BI-focused upgrade to SQL Server 2008, due in the first half of 2010.

This means that for our SQL Server warehouses, now topping out (in practical terms) at around 20TB, the volume of data they can handle comfortably will be increased for online analytical processing (OLAP) work by at least an order of magnitude – the figure of 200TB was cited. (Note the qualification there – it is not yet clear what impact Madison will have on transactional performance.) But there’s more to it than that.

Since DATAllegro technology allowed queries to run against relational databases very, very rapidly, Madison might tempt some SQL Server users to move from the Kimball warehouse model (using dimensional structures in the core data warehouse) to the Inmon model (relational). In other words, they’d be using a relational structure to support OLAP, which is exactly the model that Teradata supports – a remark that brings us effortlessly to Teradata’s conference.

Teradata
Teradata is one of the earliest data warehousing companies, and has become incredibly successful. The only reason it’s not a household name is that it has always focused on the largest companies in the world for its customer base – typical customers are in the Fortune 500. Unlike some, Teradata has never forgotten that it is a technology-driven company, so it has retained a good mix of technical as well as business guys at the top. This sharp technological focus is hugely to the company’s advantage in the IT world and it certainly encourages loyalty: I couldn’t find a single delegate who was anything other than very enthusiastic about the company and what it is doing. Try to find another technology company for which that is true.

Teradata announced an extension to its product range, the Extreme Data Appliance 1550. The range now extends from the Active Enterprise Data Warehouse 5550 at the top, able to support the world’s biggest warehouses, to the software-only edition, capable of running on a Wintel box. Smack in the middle is the new product and the price per terabyte of user data is a mere $16.5k – an amazing entry-level price for a world-class solution, as you’ll realise if you compare it with the cost of other data warehouse appliances.

Reading the signs
Microsoft’s mantra has always been “BI for the masses”, in other words making BI available to as many people as possible. Microsoft is often vilified and rarely praised in the press, but more than any other company it has transformed BI from the incredibly expensive preserve of the few to a cost-effective solution embraced by many an SME.

Teradata is at the opposite end of the scale: it has been doing data warehousing for longer than anyone else and in my opinion offers the most powerful warehousing technology there is. It provides BI solutions to the biggest companies on the planet, including Wal-Mart, British Airways and FedEx.

The Big M and Mr T approaches to data warehousing are very different. Microsoft advocates moving data into a warehouse and then moving subsets out to data marts, where the data is organised into OLAP cubes and queries run against it. Teradata starts with the same step – moving data to a warehouse – but then keeps the data in relational form (the Inmon ideal) in its proprietary, mega-fast relational database engine, and runs queries against it there.

The announcements made by both companies, with Teradata extending its product range and Microsoft improving its scalability, indicate that a turf war may be beginning. In the past Teradata has stated that its market is the top 1,000 companies worldwide, but that’s changing. With its current range of products Teradata is aiming to expand its market further down the pyramid of potential adopters (Figure 1). If you stack organisations on the basis of size, as we moves downwards each band contains a dramatically increased number of potential purchasers, but the BI budget of the individual companies decreases substantially.


Figure 1: As Microsoft scales up, Teradata moves down – with inevitable consequences

Teradata has watched the rise and rise of the data warehouse appliance in mid-size companies and has entered that arena by offering its own range of appliances. Mid-size companies have the potential to grow into larger companies, and Teradata’s strategy is to encourage them early into the fold – which is probably why the words “Intelligence for all” made several appearances during the conference.

On the Microsoft front, the arrival of Madison will enable ever-larger volumes of data to be handled and encourage Inmon-type warehouse architectures (dispensing with the need for OLAP data marts). You can see where this is going. As Microsoft gazes further up the pyramid for new customers and Teradata peers down from its current heady heights, the two might some day find themselves glaring eye-to-eye over a potential new account. Maybe not today, maybe not tomorrow, but soon – and for the rest of their lives.

For us, the customers, it’s all good news. Two extremely good solutions are coming our way. Five years ago I wouldn’t have thought for one second of advising a really large company to consider Microsoft’s BI solution or a mid-sized one to look at Teradata. Now, depending on requirements, both pieces of advice could make sense. We live in interesting times.

Real time is golden…
Realtime BI is, of course, impossible. There will always be a delay in transferring data from source systems into the warehouse because the actual transfer inevitably takes some period of time. However, that doesn’t stop people from trying and “Very Close To Real Time Really” is perfectly possible. However, VCTRTR is a shocking abbreviation, so let’s simply call it real time.

There are two approaches to loading data in real time: you can either load batches of transactions from the sources (every 10 minutes, every minute, whatever) or you can load each transaction as it completes in the source system. And therein lies a problem – it’s that word “transaction”. If all our source systems were modern RDBMSs running beautifully designed databases and capable of identifying individual transactions, each transaction could be sent off to the warehouse (or to a batch) as soon as it had completed and everything would be hunky dory. Sadly this is not the case. Many source systems are quite incapable of actively sending transactions to an external system (such as a data warehouse) and if this is the case in your organisation, you’ll be thwarted in your attempts to achieve realtime BI.

But such obstacles are made to be overcome and specialist companies have devised systems that allow crumbly source systems to actively deliver transactions to a warehouse. One of these is GoldenGate Software and I spoke to Alok Pareek (VP, Technology) and Sami Akbay (VP, Marketing and Product Management) at Las Vegas. Depending on the system, GoldenGate can provide a software platform that “provides log-based change data capture”. Its approach is essentially to read the log generated by an intractable source system and derive from it the transactions that have taken place. Messages are then built that neatly relay the source system activity to the warehouse in a form the warehouse can understand.

So, if you’re struggling in your quest for realtime BI, why not check out www.goldengate.com ?