Why Are Customers Architecting Hybrid Data Warehouses?

By Mona Patel

As a leader in IT, you may be  incented or mandated to explore cloud and big data solutions to transform rigid data warehousing environments into agile ones to match how the business really wants to operate.  The following questions must come to mind:

  • How do I integrate new analytic capabilities and data sets to my current on-premises data warehouse environment?
  • How do I deliver self service solutions to accelerate the analytic process?
  • How do I leverage commodity hardware to lower costs?

For these questions, and more, organizations are architecting hybrid data warehouses.  In fact, these organizations moving towards hybrid are referred to as ‘Best In Class’ according to The Aberdeen Group’s latest research: “Best In Class focus on hybridity, both in their data infrastructure and with their analytical tools as well.  Given the substantial investments companies have made in their IT environment, a hybrid approach allows them to utilize these investments to the best of their ability while explore more flexible and scalable cloud-based solutions as well.”  To hear more about these ‘Best In Class’ organizations, watch the 45 minute webcast.

How do you get to this hybrid data warehouse architecture with the least risk and most reward?  IBM dashDB delivers the most flexible, cloud database services to extend and integrate with your current analytics and data warehouse environment, addressing all the challenges related to leveraging new sources of customer, product, and operational insights to build new applications, products, and business models.

To help our clients evaluate hybrid data warehouse solutions, Harvard Research Group (HRG) provides an assessment of IBM dashDB.  In this paper, HRG highlights product functionality, as well as 3 uses cases in Healthcare, Oil and Gas, and Financial Services.   Security, Performance, High Availability, In-Database Analytics, and more are covered in the paper to ensure future architecture enhancements optimize IT rather than adding new skills, complexities, and integration costs. After reading this paper, you will find that dashDB enables IT to respond rapidly to the needs of the business, keep systems running smoothly, and achieve faster ROI.

To know more on dashDB check out the video below:

 

About Mona,

mona_headshotMona Patel is currently the Portfolio Marketing Manager for IBM dashDB, the future of data warehousing.  With over 20 years of analyzing data at The Department of Water and Power, Air Touch Communications, Oracle, and MicroStrategy, Mona decided to grow her career at IBM, a leader in data warehousing and analytics.  Mona received her Bachelor of Science degree in Electrical Engineering from UCLA.

Start Small and Move Fast: The Hybrid Data Warehouse

by Mona Patel

In the world of cutting edge big data analytics, the same obstacles in gaining meaningful insight still exists – ease of getting data in and getting data out.  To address these long standing issues, the utmost flexibility is needed, especially when layered with the agile needs of the business.

Why spend millions of dollars replacing your data and analytics environment with the latest technology promise to address these issues, when can you to leverage existing investments, resources, and skills to achieve the same, and sometimes better, insight?

Consider a hybrid data warehouse.  This approach allows you to start small and move fast. It provides the best of both worlds – flexibility and agility without breaking the bank.  You can RAPIDLY serve up quality data managed by your data warehouse, blended with newer data sources and data types in the cloud, and apply integrated analytics such as Spark or R – all without additional IT resources and expertise.  How is this possible?  IBM dashDB.

Read Aberdeen’s latest report on The Hybrid Data Warehouse.

mona's blog

 

Watch Aberdeen Group’s Webcast on The Hybrid Data Warehouse.

Let me give you an example.  We live in a digital world, with organizations now very interested in improving customer data capture across mobile, web, IoT, social media, and more for newer insights.  A telecommunications client was facing heavy competition and wanted to quickly deliver unique mobile services for an upcoming event in order to acquire new customers by collecting and analyzing mobile and social media data.  Taking a hybrid data warehouse approach, the client was able to start small and move fast, uncovering new mobile service options.

Customer information generated from these newer data sources were blended together with existing customer data managed in the data warehouse to deliver newer insights.  IBM dashDB provided a high performing, public cloud data warehouse service that was up and running in minutes.  Automatic transformation of unstructured geospatial data into structured data, in-memory columnar processing, in-database geospatial analytics, integration with Tableau, and pricing were some of the key reasons IBM dashDB was chosen.

This brings me back to my first point – you don’t have to spend millions of dollars to capitalize on getting data in and getting data out.  For example, clients like the one described above took advantage of Cloudant JSON document store integration, enabling them to rapidly get data into IBM dashDB with ease– no ETL processing required.  Automatic schema discovery loads and replicates unstructured JSON documents that capture IoT, Web and mobile-based data into a structured format.  Getting data or information out was simple, as IBM dashDB provides in-database analytics and the use of familiar, integrated SQL based tools such as Cognos, Watson Analytics, Tableau, and Microstrategy.  I can only conclude that IBM dashDB is a great example of how a highly compatible cloud database can extend or modernize your on-premises data warehouse into a hybrid one to meet time-sensitive business initiatives.

What exactly is a hybrid data warehouse?  A hybrid data warehouse introduces technologies that extend the traditional data warehouse to provide key functionality required to meet new combinations of data, analytics and location, while addressing the following IT challenges:

  • Deliver new analytic services and data sets to meet time-sensitive business initiatives
  • Manage escalating costs due to massive growth in new data sources, analytic capabilities, and users
  • Achieve data warehouse elasticity and agility for ALL business data

mona_dashDB

Still not convinced on the power of a hybrid data warehouse?  Hear what Aberdeen Group’s expert Michael Lock has to say in this 30 min webcast.

About Mona,

mona_headshot

Mona Patel is currently the Portfolio Marketing Manager for IBM dashDB, the future of data warehousing.  With over 20 years of analyzing data at The Department of Water and Power, Air Touch Communications, Oracle, and MicroStrategy, Mona decided to grow her career at IBM, a leader in data warehousing and analytics.  Mona received her Bachelor of Science degree in Electrical Engineering from UCLA.

How dashDB Helps Media Channels Boost Revenues And Viewership

By Harsimran Singh Labana

Did you ever wonder how a media channel decides which ad comes at what time? Well, there is an analytics science behind this.

Cable and broadcast networks pay studios large sums of money for the right to broadcast a specific show or movie at specific times on specific channels. To achieve a return on that investment, networks must design TV schedules and promotional campaigns to maximize viewership and boost advertising revenues.

RSG Media is an IBM dashDB managed service client that partners with cable and broadcast, entertainment, games and publishing firms to provide insights that help maximize revenue from content, advertising and marketing inventories. Shiv Sehgal, Solutions Architect, RSG Media says, “We had the rights data, the scheduling data and the advertising revenues data. If we could combine this with viewership and social media data, we could give our clients a true 360-degree view of their operations and profitability, down to the level of individual broadcasts. The missing piece of the puzzle was to build a data and analytics capability that could bring all the data together and turn it into business insight – and that’s where IBM came in.”

RSG Media chose IBM because of its complete vision for cloud analytics. This includes an integrated set of solutions for building advanced analytics applications and coordinating them with all the relevant data services in the cloud.

RSG Media’s Big Knowledge Platform is built on the IBM® Cloudant® NoSQL document store and the IBM dashDB™ data warehouse service, orchestrated through the IBM Bluemix® cloud application development platform. Cloudant’s Schema Discovery Process (SDP) is used to ingest and translate semi-structured data from more than 50 sources, and structure that data into a schema that the dashDB relational data warehouse understands.

RSG Media is not stopping here and they are excited about Watson Analytics and how it predicts customer behavior.  Learn more about RSG Media success using dashDB and Cloudant solutions on Bluemix.

About Harsimran,
HarryHarsimran Singh Labana is the Portfolio Marketing Manager for IBM’s Data Warehousing team. Working in a worldwide role he ensures marketing support for IBM’s solutions. He has been with IBM for close to five years working in diverse roles like sales and social media marketing. He stays in Bangalore, India with his wife and son.

What Should You Look For In Your Cloud Data Warehouse?

By Rahul Agarwal,

The business benefits of cloud computing are well documented; according to an IBM study, organizations using cloud computing gain a competitive advantage over their peers and can generate two times more revenue and profit.[1]

But is the cloud the right place for data warehousing which has traditionally been deployed on-premise (requiring a significant investment in hardware infrastructure)? A study by the Aberdeen group finds that organizations are increasingly using cloud-based analytics to gain advantages such as four times faster business intelligence deployment times and have 50% more users actively engaged with analytics.[2]

So what parameters should you look for in your cloud data warehouse?

Simplicity

Your data warehouse on the cloud should let you focus on your data and your business problems, not the business of data warehousing (including tuning, planning and integration). It should be simple to set up; ideally providing ‘load-and-go’ simplicity. In addition, it should provide the ability to easily ingest data from a myriad of sources including structured, semi-structured (think JSON) and unstructured.

Speed

Speed-driven data and analytics practices are quickly emerging as a key source of competitive advantage for companies across the world.[3] Hence, it is extremely important for you to try to minimize the time it takes to convert raw data that exists in your enterprise into actionable insight. Today a number of high performance technologies like in-memory computing and in-database analytic capabilities provide the ability to analyze data with high speed and precision. By running the analytics in the database where the data resides you will gain huge efficiencies. When you couple in-memory technology with analytics, you are able to get answers to your business questions as fast as you can think of the next question to ask – no waiting for analytic results to run.

Interoperability with business intelligence tools

Your cloud data warehouse should provide you the ability to write and execute your own analytic queries, or leverage other analytic and BI capabilities provided by tools such as Cognos, Looker, Aginity Workbench, Tableau, and others.  Integration with such tools will help you better visualize and interact with your data, enabling a richer business intelligence experience.

Security

Your cloud data warehouse should be designed to keep your data secure with the same rigor that has come to be expected from an on-premise data warehouse. Any security/data breach can put your business operations at risk and has the potential not only to damage your company’s reputation but also its top and bottom line.

Introducing dashDB

dashDB is a fully managed data warehouse in the cloud meets all of these criteria for simplicity, security and analytics in an instant. dashDB is simple to get up and running and helps you to build the capability to deliver answers to your business questions easily and as fast as you can think.

To learn more and get started with the freemium, check out www.dashdb.com.

About Rahul Agarwal

Rahul Agarwal is a member of the worldwide product marketing team at IBM that focuses on data warehouse and database technology. Rahul has held a variety of business management, product marketing, and other roles in other companies including HCL Technologies and HP before joining IBM.  Rahul studied at the Indian Institute of Management, Kozhikode and holds a bachelor of engineering (electronics) degree from the University of Pune, India. Rahul’s Twitter handle :  @rahulag80

[1] http://www-03.ibm.com/press/us/en/pressrelease/42304.wss

[2] https://www14.software.ibm.com/webapp/iwm/web/signup.do?source=sw-infomgt&S_PKG=ov26256

[3] http://www-935.ibm.com/services/us/gbs/thoughtleadership/2014analytics/

Data Warehouse Modernization: Vetting Forrester’s Return-on-Investment Calculations

By James Kobielus,

It’s well known that, in a prior life, I was an industry analyst focusing on the data warehousing (DW) market. So I think I have a good mental radar for identifying high-quality, data-driven DW research when I see it.

When you’re researching the potential return on investment (ROI) for a DW, you have to be rigorously quantitative, precise, and comprehensive in your approach. Enterprises often place the DW at the very heart of their big data and analytics strategies. Solid ROI metrics must support DW projects of any scope, and the range of competing alternatives demands a decision-support framework that facilitates apples-to-apples comparisons. Many DW projects involve modernizations at various levels, so ROI calculations must be adept at characterizing the potential bottom-line impact of new technologies, platforms, tools, and practices.

Sure, anybody can pull ROI estimates out of thin air, but finding metrics that can help you make DW investments with confidence can prove tricky. In that regard, I’ve long felt that Forrester Consulting’s Total Economic Impact (TEI) methodology is the best ROI calculation framework for information-technology (IT) investments of any sort. Grounded in Forrester’s extensive survey- and interview-based research, its TEI studies incorporate fine-grained benefit, cost, risk, and flexibility variables into an underlying spreadsheet-based model. The drivers, use cases, assumptions, formulas, and data intrinsic to Forrester’s ROI calculations are totally transparent, so you can vet them for yourself. On any specific TEI use case under scrutiny, Forrester projects the resulting ROI analysis over a risk-adjusted 5-year horizon from the point of view a typical “composite organization” that uses the technology in question.

Back in my Forrester days, I sweated these details when constructing a now-outdated TEI study of the DW appliance market. So naturally I was very curious when, over the holiday season, IBM made available a new Forrester TEI covering our entire Information Management (IM) solution portfolio, but with a core focus on DW.

On my first pass through the report, I noticed the sorts of high-level rollup numbers that usually figure into most marketing collateral or blogs on these kinds of studies. Specifically, Figure 1 states a 5-year risk-adjusted return of 148% and total benefits (present value) of $31.2 million for the typical composite organization. Still being an analyst at heart, I drilled more deeply into the study itself to determine what exactly it refers to.

The first thing you see, from Figure 2, is that, among the three use cases in this Forrester TEI, “DW modernization” accounts for around $5m of the benefits, with “security intelligence extension” a little over $3m and a whopping ~$23m from “enhanced 360-degree view of the customer.” Clearly, all of those are essentially DW-related returns.

When vetting a TEI, it’s best to single out the specific use case of interest. In my case, I focused on Forrester’s DW modernization use case, which estimates the quantitative bottom line from cost reductions and value enhancements due to more efficient storage and processing, speedier performance, and agile analytics. These are in line with the chief DW modernization drivers cited in Figure 6, which were derived from Forrester’s in-depth decision-maker interviews.

In terms of concrete decision support for DW professionals evaluating modernization initiatives, the real payoff from this study is on pages 27-30. These spell out the full assumptions for the use case, including scope of solutions included, size of the composite organization’s IT budget, percentage of that budget allocated to data and storage, number and growth of terabytes of DW storage, percent reductions in storage cost, number of staff using big data analytics, and so on.

Pay close attention to the solution scope under DW modernization. Forrester took the right approach by not limiting their analysis to DWs in the older, much more limited sense of premises-based analytic databases specializing only in structured, at-rest data for operational business intelligence. As they state on page 27, they included the broader sweep of big-data analytics, information integration, and governance solutions in IBM’s IM solution portfolio.

If they’d gone with a traditional DW scope, such as the one this former analyst included in his 2010 study, Forrester would have ignored the substantial evolution that this marketplace has experienced in this decade. If Forrester had stuck with that scope in this latest study, it would probably have limited its TEI to IBM PureData for Analytics, IBM DB2 with BLU Acceleration, and IBM Digital Analytics Accelerator for System Z. But it did the right thing this time around (reflecting what our customers are doing) by including our Hadoop, streaming, discovery, and InfoSphere IIG offerings in the scope of a hybridized, cloud-focused DW infrastructure.

To see how far mainstream DW solutions have advanced into cloud-centric hybrid architectures, check out this blog I published a few months ago on the new IBM dashDB. I’m assuming that Forrester’s exclusion of dashDB, as well as Watson Analytics and DataWorks, from this recent study was due principally to their need to lock down their project’s scope many months ago before these specific solutions were launched.

For enterprise analytics and IT professionals, the DW modernization ROI that you calculate for your own situation depends on the assumptions you make and how you adjust the Forrester TEI model’s parameters to align with those. The beauty of the Forrester TEI methodology is that its model can be easily customized and use cases easily extended to do justice to the complex range of technologies in DW modernization initiatives. Depending on the project and your requirements, DW modernization may include various blends of new technologies (e.g., Hadoop, in-memory), new topologies (e.g., hybrid, distributed, and zone architectures), new sources (e.g., machine, social, & mobile data), new form factors (e.g., cloud, appliance), new tooling (e.g., governance, curation, archiving), new development frameworks (e.g., MapReduce), and new scaling and performance approaches (e.g., consolidation, compression, scale-out).

If I have any quibble with the latest Forrester TEI, it’s with their apparent exclusion of traditional DW use cases, such as operational BI (the focus of our Cognos portfolio), from their scope. Also, Forrester doesn’t give the newer DW use cases, such as in-database analytics for statistical modeling and data science (the focus of our SPSS portfolio), as much emphasis as I’d wish.

But those are just scoping issues that can be easily addressed if Forrester ever chooses to take this TEI analysis in those directions in coming years.

About James, 

James Kobielus is IBM Senior Program Director, Product Marketing, Big Data Analytics solutions. He is an industry veteran, a popular speaker and social media participant, and a thought leader in big data, Hadoop, enterprise data warehousing, advanced analytics, business intelligence, data management, and next best action technologies. Follow James on Twitter : @jameskobielus

Hybrid Data Warehousing – The Best of All Worlds

By Wendy Lucas, 

When it comes to data warehousing, organizations are progressing along the maturity curve at their own individual pace.  Today, most organizations have some form of warehouse and business intelligence in place, or recognize the need for it and the benefits it can drive.  But we all know that technology doesn’t stand still.  And so, you are now faced with a new step in your progression towards data warehouse maturity – the move to cloud.

Building Momentum

Cloud applications started with a fairly narrow focus.  A few years ago, you may have viewed the cloud as a viable platform for mobile applications or just a way to keep your contacts synchronized between your devices (by the way, that is still my favorite cloud use case).  IT organizations have begun looking to the cloud as a way to cut costs, but the strong momentum behind cloud adoption indicates there is more to it than that!

According to a recent IBM Tech Trend study, cloud adoption is up 92% since 2012.  The same study shows that organizations identified as pacesetters are 10x more likely to increase workforce efficiency with the cloud, 5x more likely to enhance communication and collaboration and report 4X better customer experience.   Pick your research outlet and you will find similar statistics.

One of Forrester’s top cloud computing predictions for 2015 is that “hybrid cloud management gets real” in terms of having the tools to allow you to manage across multiple on-premise and cloud platforms.

I believe that cloud use cases are the driving factor behind the growth and momentum of cloud technologies. Data warehousing on the cloud is no exception, where the general need is to deliver analytics to the organization faster.  Let’s explore specific data warehouse use cases.

Use case 1: development, testing, prototyping and sandboxing

A safe place to start might be establishing a cloud environment for warehouse development and testing.   Do you need the ability to test key functions like ETL processes or analytic applications without the need to setup more costly infrastructure on-premise?  Why not consider testing in the cloud? Perhaps you need an environment in which to do quick prototyping or sandboxing?   Whether it’s an environment that is a temporary or persistent, a cloud data warehouse instance can be quickly stood up and used for prototyping and sandboxing with very minimal cost.

Use case 2: Do more with less when when you are at capacity

Organizations are also considering cloud as a way to expand capacity of their existing data warehouse.  In the context of the logical data warehouse, data assets can reside on the cloud to serve up specific types of data to specific applications.

Use case 3: self-service analytics

Organizations can use the cloud as a data layer for self-service business intelligence and analytic capability, especially for applications that need data that’s already in the cloud, for example if your marketing organization need to analyze unstructured social media data.

Both IT and the line of business can benefits from these and other use cases.  IT organizations are able to reduce infrastructure costs and simplify budgets by shifting capital expense to an operational expense model.   Perhaps most importantly, the flexibility and agility of a cloud option provides faster time to insight for end users who need insight immediately.

What should I move to the cloud?

If cloud is so great, why not move everything to the cloud?  The reality is there are some applications that will remain on-premise for some time to come (or forever).  Systems that require large amounts of on-premise or sensitive data or that are generating large volumes of data may not be easily moved to the cloud.  It may make sense to leave these in the on-premise data warehouse systems that have matured over decades and are fulfilling the needs of those applications quite well.   But like discussed above, you may not want to incur capital expenditure or longer deployment times for things like data marts, development and test environments or analytics for data already in the cloud and these represent ideal opportunities to use a cloud data warehouse

There isn’t a one-size fits all answer, which is why hybrid environments make the most sense.  A hybrid environment can provides the best of all worlds – the ability to keep your large, on-premise warehouses in place, allow compliance with security and regulatory reporting, and fulfill the needs of traditional reporting and analysis.  , All of this is done while continuing to reduce costs and increase flexibility and speed of deployment for new applications in the cloud.  Just like most things, its best to pick the right tool for the job.

What tools can help me get there?

IBM data warehouse solutions offer the breadth and depth of capabilities required to effectively support a hybrid environment.   On cloud, IBM dashDB is our exciting new data warehouse and analytics as a service that concluded the beta program for Cloudant and enterprise plan on December 18th and is now generally available.  It pulls together the lightning fast performance of DB2 with BLU Acceleration with market leading in-database analytic capabilities from Netezza.  Think of it as the combination of the fastest data warehouse and analytic platform, combined with the flexibility and agility of the cloud.  dashDB will continue to evolve in a way that preserves analytic and application portability between the cloud and on-premise systems.  Most importantly, as you modernize with a hybrid cloud approach, the enterprise plan is available to support you at scale.

And of course on premises, IBM offers DB2 with BLU Acceleration as a software-only solution or the IBM PureData System for Analytics as a ready to go data warehouse appliance. Putting these pieces together, you can support your hybrid data warehousing needs with proven technologies that offer the best of all worlds.

For more information, please visit dashDB.com.

About Wendy,

Wendy Lucas is a Program Director for IBM Data Warehouse Marketing. Wendy has over 20 years of experience in data warehousing and business intelligence solutions, including 12 years at IBM. She has helped clients in a variety of roles, including application development, management consulting, project management, technical sales management and marketing. Wendy holds a Bachelor of Science in Computer Science from Capital University and you can follow her on Twitter at @wlucas001

dashDB grows and improves in a flash!

By Dennis Duckworth, 

IBM® dashDB™ continues to grow and improve with some announcements made on December 18, 2014.  Additional plans are now available to everyone and new features have been included in dashDB.

New Deployment Options

  • Enterprise Plan available to all – The Enterprise Plan for dashDB is a dedicated cloud infrastructure with tera-scale capacity. This offering is now available to anyone. Contact your IBM Information Management Sales representative to get started!
  • Cloudant deployments of dashDB now offer higher capacity, paid usage plans – We are adding an Entry plan that is fully integrated with Cloudant supporting up to 50 GB of uncompressed data, for $50/month. The freemium offering for data usage below 1 GB will remain.
  • Expanded Geographic Presence – dashDB can now be deployed in our UK availability region in addition to our existing North American region.

New Cool and Useful Features

As a result of input from beta program participants, we have added new features to dashDB to make it even more useful:

  • Improved SQL Editor – The SQL query and editor capabilities have been expanded to allow a full range of SQL to be submitted via the web browser, including the ability to load and save SQL scripts. SQL validation and error checking is also included.

DD 1

  • Better Workload Monitoring – Get a much better idea of what’s running in your dashDB instance, including specific statements and connections. Set it to monitor in near real-time, drill into the details of a session, or terminate a session if needed.

 DD2

  • Command Line Support – Sometimes, you need a command line interface for scripting and automation. dashDB now includes CLPPlus support – so you now get a command line user interface that lets you connect to databases and define, edit, and run statements, scripts, and commands.
  • UDX Support – Some applications or algorithms require user-defined functions (UDFs) and user-defined aggregates (UDAs). dashDB now supports these out of the box so you can implement and run your own algorithms right inside the database.

New Security Features and Capabilities

Data security is always a consideration, and dashDB now includes new security features:

  • SSL Support for all Connections – It’s not enough to automatically encrypt data at rest, we need to encrypt it in motion too. dashDB now supports SSL for all connections to the database.

DD3

  • Select Guardium Reports for all PlansdashDB now has bundled Guardium reports for all plans including the Enterprise and Cloudant integrated plans. This allows for automatic discovery of sensitive data, as well as access reports and details of SQL statements that were run against that data.

DD4

We will continue to add new features and capabilities to dashDB over the coming months, so watch this space!

If you have not started analyzing your data in dashDB, what are you waiting for? Get started with dashDB on Bluemix or Cloudant at dashDB.com.

About Dennis Duckworth

Dennis Duckworth, Program Director of Product Marketing for Data Management & Data Warehousing has been in the data game for quite a while, doing everything from Lisp programming in artificial intelligence to managing a sales territory for a database company. He has a passion for helping companies and people get real value out of cool technology. Dennis came to IBM through its acquisition of Netezza, where he was Director of Competitive and Market Intelligence. He holds a degree in Electrical Engineering from Stanford University but has spent most of his life on the East Coast. When not working, Dennis enjoys sailing off his backyard on Buzzards Bay and he is relentless in his pursuit of wine enlightenment. You can follow Dennis on Twitter