Richard Hackathorn: June 2010 Archives

At the wonderful new offices of ISI, Balanced Insight presented their story about collaborative BI as offered in their Consensus product. Tom Hammergren, Founder/CEO, and Mike Venerable, Board Member, lead the discussion. Mike and Tom are both long time BI/DW industry folks. Mike wrote a book with Chris Adamson and was the leader of Talus, who was acquired by Sagent, who was acquired by Pitney Bose, etc. Tom was also on the founding team at Cognos when they created PowerPlay and Impromptu and the developer of the Sybase Warehouse Studio. Tom has three DW books out there, including the latest update of Data Warehousing for Dummies.

Balanced Insights was founded in 2003 by several BI veterans (like Tom) with seed funding. They have been profitable with cash flow positive. They have a 20-25 notable customers, such as Fidelity Investments, Nationwide Insurance, T-Moble, AT&T, Nike, and Subway.

Tom noted that their target is the leader of BI within the IT group. However, the motivation comes mainly from the business users who have clear business problems and are aware of the potential of BI/DW to solve those problems. Their vision is combining social software with BI delivery software to increase our customer’s fact based decision making capabilities by providing a collaborative BI ecosystem.

Tom argues... Current BI delivery is challenged by being inefficient and labor intensive, with business users disconnecting when dealing with the technical issues. As the business demands bigger information, the need for higher project velocity requires Agile BI Development. Tom summarized, "How our Consensus product engages people is wildly different than other Agile BI tools." Claudia asked, "What is Agile BI?" Tom replied, "Consensus is the way of doing things. Our product facilitate agile delivery methods through a model-driven architecture, which permits faster progress through the design stages, not just cut/paste layout of the user interface." The Consensus tool interfaces (builds upon) BI suites (Business Objects, Cognos, etc.) with OLAP tools (Cognos DMR, Microsoft SSAS, etc.) and data warehouses (IBM DB2, MySQL, Oracle, Netezza, etc.).

As far as engaging users, Tom went over an example of Consensus’ collaborative approval workflow process. This capability is engineered into the agile development activities so that each BI deliverable can be reviewed by a broad set of distributed users with all feedback recorded for future reference within the Consensus Repository.

A Forrester report reviewed several related vendors according to a variety of criteria. Tom argued that many of these vendors do not overlap on their core functions. Going through customer examples, Tom summarized their value proposition as: Fast with half the time, Productive with half the cost, and Engaging with continuous feedback.

Cindi Howson asked about integration among the various BI tools through the development steps. Tom gave several examples of how the data structures of one tool in migrated into another stage to be use by another tools.

Tom ended with their future plans (which is confidential).

My Take...

Balanced Insight is focusing on a very important aspect of BI/DW - effective collaborative BI, amid typical infrastructures composed of diverse fragmented tools. This is quite a challenge! Their vision is to do end-to-end collaborative BI; however, at this stage, they are doing collaborative BI delivery. Tom mentions agile development, but they need to clearly link into the framework and terminology of agile development process.

The challenge for Balanced Insight to provide the full BI ecosystem is the definition of the BI function layer that sits over various technical layers, such as user access, target data, data movement, data quality, and source data. What is the business value proposal behind providing this BI layer? Digging into this question should reveal the chaos (and inefficiencies and risk) in the ad hoc activities that characterize most BI projects.  Balanced Insights need to make that messy situation vividly clear in their market positioning and then leverage it.

Teradata presented several areas where they are pushing the limits, such as multi-temperature, temporal extensions, and the like. Todd Walter, CTO, did the honors. He started with hardware stuff, because hardware will change the basic assumptions of BI professionals about what they can do. As data ages, its access frequency declines...in general. There are exceptions with seasonal analysis, fraud cases, etc.

Todd shows real data on data access frequency, as shown in the figure. 43% of IO processing concentrated on only 1% of the data (as measured by storage allocation units).

Several hardware developments will impact DW architectures, such as 2.5" SFF disks, in-drive encryption, 6 Gb/s SAS drives, and enterprise Solid State Drives (SSD).

As shown to the left, there is a giant discontinuity in processor performance versus disk performance. Multi-core CPU are turbo-charging, while disk drives lag. SSD will provide performance that will rebalanced this performance gap.

An interesting aspect is that DW architects have had to buy excess storage to maintain IO performance so that part of each spindle has to be left unused if all the data is hot. NOTE: Todd checked the ratio of SSD-HDD and said, "Believe it or not, it is not a math error. HDDs are at the high end of 10**2 and SSDs are at the low end of 10**5 and the diff is roughly 150x." So, my correction to this figure is wrong. Please disregard! :)

An analogy is like a delivery truck. If you fill it up, the truck goes at only half speed. So, you only fill it half way. However, the implication is that very light stuff can go FREE in back of the truck. As a result, Teradata is moving toward multi-temperature storage hierarchies where the layers are seamless. This is an old story but with some new twists.

In the second half, Todd covered the new enhancements to the new version (due out in September) in compression, geospatial, and temporal. Fascinating discussion. I tried to listen more and type less. Teradata has previously focused on compression having zero processing overhead. That is now changing. Now it is a good tradeoff to expend CPU resources to enhance compression.

Geospatial extensions has lots of potential new applications. A discussion of the privacy issue emerged focusing on applications that locate and track customers.

Todd flew through many temporal examples, showing how the SQL syntax is greatly simplified with these temporal extensions.

My Take...

Once the new storage developments settle, the DBMS landscape will change dramatically! For many vendors, traditional DB data access layer will be obsoleted. Teradata has been talking about multi-temperature issues for years. However, this is the first time I see a strategic (and disruptive) vision. Shawn Rogers put is aptly, "This is an asteroid...a dinosaur killer!"

We should have had geospatial and temporal extensions to DW ten years ago. Todd agreed, "It is way overdue!" A concern of mine is that there are no temporal standards, unlike geospatial. However, Todd remarked that they have followed a comprehensive proposal that builds on prior research (Snodgrass et al). Unfortunately the proposal is inactive in the standards committees at this time.