How To Make Good Decisions in Deploying the Logical Data Warehouse

By Rich Hughes,

A recent article addresses the challenges facing businesses trying to improve their results by analyzing data. As Hadoop’s ability to process large data volumes continues to gain acceptance, Dwaine Snow provides a reasonable method to examine when and under what circumstances to deploy Hadoop alongside your PureData System for Analytics (PDA).   Snow makes the case that traditional data warehouses, like PDA, are not going away because of the continued value they provide. Additionally, Hadoop distributions also are playing a valuable role in meeting some of the challenges in this evolving data ecosystem.

The valuable synergy between Hadoop and PDA are illustrated conceptually as the logical data warehouse in Snow’s December 2014 paper (Link to Snow’s Paper).

The logical data warehouse diagrams the enterprise body of data stores, connective tissue like APIs, and the cognitive features like analytical functions.  The logical data warehouse documents the traditional data warehouse, which began about 1990, and its use of structured data bases.  Pushed by the widespread use of the Internet and its unstructured data exhaust, the Apache Hadoop community was founded as a means to store, evaluate, and make sense of unstructured data.  Hadoop thus imitated the traditional data warehouse in evaluating value from the data available, then retaining the most valuable data sources from that investigation.  As well, the discovery, analytics, and trusted data zone architecture of today’s logical data warehouse resembles the layered architecture of yesterday’s data warehouse.

Since its advent some 10 years ago, Hadoop has branched out to servicing SQL statements against structured data types, which brings us back to the business challenge:  where can we most effectively deploy our data assets and analytic capabilities?  In answering this question, Snow discusses the fit-for-purpose repositories which for success, require inter-operability across the various zones and data stores.  Each data zone is evaluated for cost, value gained, and required performance on service level agreements.

By looking at this problem as a manufacturing sequence, the raw material / data is first acquired, then manipulated into a higher valued product—in this case, the value being assessed by the business consumer based on insights gained and speed of delivery.  Hadoop distributed file environments shows its worth in storing relatively larger data volumes and accessing both structured and unstructured data.  Traditional data warehouses like IBM’s PureData System for Analytics display their value in being the system of record where advanced analytics are delivered in a timely fashion.

In an elegant cost benefit analysis, Snow provides the tools necessary to weigh where best to deploy the different, but complimentary data insight technologies.  A listing of Total Cost of Ownership (TCO) for Hadoop includes four line items:

  1. Initial system cost (hardware and software)
  2. Annual system maintenance cost
  3. Setup costs to get the system ‘up and running’
  4. Costs for humans managing the ongoing system administration

Looking at just the first cost item, which is sometimes reduced to a per Terabyte price like $1,000 per TB, is but part of the story.  The article documents the other unavoidable tasks for deploying and maintaining a Hadoop cluster.  Yes, $200,000 might be the price for the hardware and software for a 200TB system, but over a five year ownership, industry studies are cited in ascribing the other significant budget expenses.  Adding up the total costs, the conclusion is that the final amount could very well be in excess of $2,000,000.

The accurate TCO number is then subtracted from the business benefits of using the system, which determines net value gained.  And business benefits are accrued, Snow notes, from query activity.  Only 1% of the queries in today’s data analytic systems require all of the data, which makes that activity perfect for the lower cost and performance Hadoop model.  Conversely, 90% of current queries require only 20% of the data, which matches well with the characteristics of the PureData System for Analytics:  reliability with faster analytic performance.  What Snow has shown is the best-of-breed nature of the Logical Data Warehouse, and as the ancient slogan suggests, how to get more “bang for the buck”.

About Rich Hughes,

Rich Hughes is an IBM Marketing Program Manager for Data Warehousing.  Hughes has worked in a variety of Information Technology, Data Warehousing, and Big Data jobs, and has been with IBM since 2004.  Hughes earned a Bachelor’s degree from Kansas University, and a Master’s degree in Computer Science from Kansas State University.  Writing about the original Dream Team, Hughes authored a book on the 1936 US Olympic basketball team, a squad composed of oil refinery laborers and film industry stage hands. You can follow him on @rhughes134

Leveraging In-Memory Computing For Fast Insights

By Louis T Cherian,

It is common knowledge that an in-memory database is fast, but what if you had an even faster solution?
Think of a next generation in-memory database, which is

  • Faster, with speed of thought analytics to get insights
  • Simpler, with reduced complexity and improved performance
  • Agile, with multiple deployment options and low risk for migration
  • Competitive, by delivering products to market much faster

We are talking about combination of innovations that make IBM BLU Acceleration, the next generation in-memory solution.

So, what really goes into making IBM BLU Acceleration, the next generation in-memory solution?

  • The in-chip analytics allows the data to flow through the CPU very quickly, making it faster than “conventional” in-memory solutions
  • With actionable compression, one can perform a broad range of operations on data, while it is still compressed
  • With data skipping, any data that is not needed to be touched to answer a query is skipped over and that results in dramatic performance improvements
  • The ability to run all operational reports on transactional data as it is captured with the help of shadow tables ,  arguably  the most notable feature in the  DB2 10.5 “Cancun Release”

To know more about leveraging in-memory computing for fast insights with IBM BLU Acceleration, watch this video: http://bit.ly/1BZq1lo

For more information, visit : http://www-01.ibm.com/software/data/data-management-and-data-warehousing/dw.html

About Louis T. Cherian,

Louis T. Cherian is currently a member of the worldwide product marketing team at IBM that focuses on data warehouse and database technology. Prior to this role, Louis has held a variety of product marketing roles within IBM, and in Tata Consultancy Services, prior to joining IBM.  Louis has done his PGDBM from Xavier Institute of Management and Entrepreneurship, and also has an engineering degree in computer science from VTU Bangalore.

IBM’s Point of View on Data Warehouse Modernization  

By Louis T. Cherian,

The world of Data Warehousing continues to evolve, with an unimaginable amount of data being produced each moment and advancement of technologies that allow us to consume this data.  This provides new capabilities for organizations to make better informed business decisions, faster.
To take advantage of this opportunity in today’s era of Big Data and the Internet of things, our customers really need to have a solid Data Warehouse modernization strategy. Organizations should look to optimize with new technology and capabilities like:

  • in-memory databases,  to speed analytics,
  • Hadoop to analyze unstructured data to enhance existing analytics,
  • Data warehouse appliances with improved capabilities and performance

To understand more about the importance of Data Warehouse Modernization and to get answers to questions like:

  • What is changing in the world of Data Warehousing?
  • Why should customers act now and what should they do?
  • What is the need for companies to modernize their Data Warehouse?
  • How are IBM Data Warehousing Solutions able to address the need of Data Warehouse Modernization?

Watch this video by the IBM Data Warehousing team to know more about the breadth and depth of IBM Data Warehouse solutions. For more information, you can visit our website .

About Louis T. Cherian,

Louis T. Cherian is currently a member of the worldwide product marketing team at IBM that focuses on data warehouse and database technology. Prior to this role, Louis has held a variety of product marketing roles within IBM, and in Tata Consultancy Services, prior to joining IBM.  Louis has done his PGDBM from Xavier Institute of Management and Entrepreneurship, and also has an engineering degree in computer science from VTU Bangalore.

 

Safety Insurance Company Gains a Better View of its Data And its Customers with IBM PureData System for Analytics and Cognos

By Louis T.Cherian,

The success of a firm in the highly competitive insurance industry depends not only on its ability to get new customers, but also retaining the most valuable customers. One way to do this is to offer these customers the most suitable policies at the best rates. So how can a company the size of Safety Insurance, which has been in existence since 1979 identify their most valuable customers?

And how could they maintain consistency when offering multi-policy incentives, given that they  deal in dozens of types of policies to millions of policyholders, when  the customer data is fragmented across numerous policy systems? Moreover, with its customer data fragmented across numerous policy systems, the actuaries were spending all their time building new databases instead of analyzing them, and eventually ending up in getting multiple versions of the truth, making it difficult for the business to make informed decisions.

This is where the combination of IBM PureData System for Analytics and Cognos opened up a whole new world of analytics possibilities that enables Safety Insurance to run their business more wisely and more efficiently.

How did they do it?

  • Switching to a powerful analytics solution

Safety Insurance teamed up with New England Systems, (IBM Business Partner) and decided to deploy the IBM PureData System for Analytics to provide a high-performance data warehouse platform that would unite data from all of its policy and claims systems. They also implemented IBM Cognos Business Intelligence to provide sophisticated automated analysis and reporting tools.

  • Accelerating delivery of mission-critical information

Harnessing the immense computing power of IBM PureData System for Analytics enables Safety Insurance to generate reports in a fraction of the time previously needed. Moreover, automating report generation enables actuaries to focus on their actual job which is analyzing figures rather than building and compiling and them. Automation also standardizes the reporting process, which improves consistency and reduces the company’s reliance on a particular analyst’s individual knowledge

  • Identifying and retaining high-value customers

By providing a “single view” of the customer across all types of insurance gives a new level of insight into customer relationships and total customer value. By revealing how valuable a particular policyholder is to the overall business, the company will be able to provide more comprehensive service, better combinations of products, and consistent application of multi-policy discounts.

To know more about this success story, watch this video where Christopher Morkunas (Data Governance Manager) from Safety Insurance Company talks about how they were able to gain a better view of their existing data and its products with the combination of IBM PureData System for Analytics and Cognos.

About Louis T. Cherian,

Louis T. Cherian is currently a member of the worldwide product marketing team at IBM that focuses on data warehouse and database technology. Prior to this role, Louis has held a variety of product marketing roles within IBM, and in Tata Consultancy Services, prior to joining IBM.  Louis has done his PGDBM from Xavier Institute of Management and Entrepreneurship, and also has an engineering degree in computer science from VTU Bangalore.