Turn Up The Power For Software-Defined Data Warehousing

by Mona Patel

Interview with Mukta Singh

As big data analytics technologies such as Spark and Hadoop continue their move into the mainstream, you might think that the traditional data warehouse is becoming less important.

Actually, nothing could be further from the truth.

To enable data of all types to be ingested, transformed, processed and analyzed efficiently, many companies are choosing to build hybrid analytics architectures that plug cloud and open source technologies such as Spark and Hadoop into on-premises environments. At the heart of these hybrid architectures lies the data warehouse – a highly reliable resource that provides a single source of truth for enterprise reporting and analytics.

This raises an important question: since the data warehouse is so central to the hybrid analytics architecture, how can we make sure it performs well and cost-effectively?

Traditional wisdom is that the infrastructure doesn’t matter – that running these vital systems of record on commodity hardware is perfectly adequate. But when you look at the numbers, you may begin to question that view.

To understand why the right hardware – in this case, IBM Power Systems – can make a real difference, I spoke with Mukta Singh, Director of Data Warehousing at IBM. In my conversation with Mukta, we take a deeper dive into why IBM’s software-defined data warehouse – IBM dashDB Local – on IBM Power Systems offers a better price/performance ratio compared to commodity hardware.

Mona_Blog

Mona Patel: Can you tell our readers a little bit about the Power Architecture? What is so unique about it?

Mukta Singh: IBM Power Systems is the dominant server platform in today’s Unix market, with over 50 percent market share. It has also become a leading platform for Linux systems, and we have seen tremendous growth in that area in recent years.

Unlike commodity servers, which typically use x86 processors, Power servers use IBM’s Power Architecture, a unique processor architecture that has been designed specifically for big data and analytics workloads.

Mona Patel: How does IBM dashDB Local integrate with Power Systems?

Mukta Singh: dashDB Local is a software-defined data warehouse offering that has been optimized for rapid deployment and ease of management. Essentially, the system runs in a Docker container, which means it can be flexibly deployed on different types of hardware either on-premises or in a private or public cloud environment.

One of the options today is to deploy your dashDB Local container on IBM Power Systems – it runs completely transparently, and it’s optimized to allow the dashDB engine to take advantage of the unique features of the Power Architecture.

If you want to move an existing dashDB Local environment from x86 to Power Systems, that’s easy too. The latest-generation POWER8 processors can operate in little-endian (LE) mode, which is the same byte order that x86 processors use. That means that you can move a dashDB container from one platform to the other without making any changes to your applications or data.

At a higher level, we have also ensured that running dashDB on Power Systems offers the same user experience as it does on x86, so the database and OS management, monitoring and integration aspects are exactly the same. The skills are completely transferable from one platform to another, so it’s a free choice and users don’t have to worry about being locked in.

Mona Patel: Can you tell us about the benefits that the Power Architecture provides for dashDB Local?

Mukta Singh: Well, for example, dashDB’s analytics engine is built on IBM BLU Acceleration – a columnar, in-memory technology that cuts query run-times from hours or minutes to just seconds.

BLU Acceleration is designed to take advantage of multi-threaded cores, and Power processors have more threads per core than most current x86 processors. In fact, if you compare an IBM POWER8 processor to an Intel Broadwell EX, it has four times as many threads per core. That means if you have a query that BLU can parallelize, you will get much better performance from Power Systems.

Similarly, because dashDB’s BLU Acceleration does all the processing in-memory, the bandwidth between the processor and the memory is very important. Again, Power Systems has a huge advantage here, with four times as much memory bandwidth as the x86 equivalent.

Finally, the processor’s cache size is important. BLU is engineered to do the majority of its processing in the CPU cache. That means it doesn’t need to repeatedly access data from RAM, which is usually a much slower process. Power processors offer four times as much cache than x86, which means they offer lower latency and reduce the need to access RAM even further. So they play to the strengths of dashDB’s query engine.

Mona Patel: So how do those numbers translate in terms of performance and cost-efficiency?

Mukta Singh: We’ve done a benchmark with dashDB Local of a 24-core POWER8 server versus a 44-core x86 server.

The Power server was 1.2 times faster in terms of throughput, despite having 45 percent fewer cores. Or to look at it another way, each POWER8 core offered 2.2 times more throughput than the x86 equivalent. Leadership performance and competitive pricing for Power scale-out servers deliver a very compelling price-performance-optimized solution with dashDB Local.

Mona Patel: How do you see the market for dashDB Local on Power Systems? Is this something that customers have been asking for?

Mukta Singh: Even when we started bringing dashDB Local to market last year, there were Power clients who were interested. As I mentioned earlier, Power has a dominant share of the Unix market, and there are thousands of companies whose businesses are built on DB2 or Oracle databases running on Power Systems. For companies that rely on Power Systems already, the idea of running dashDB Local on their existing infrastructure is very attractive.

But the results of our benchmark suggests that this isn’t just a good idea for existing Power clients – it’s also an opportunity for new clients to start out running dashDB on a hardware platform that is tailor-made for high-performance analytics.

And for any client who currently runs dashDB on x86 servers, the message we’d like to get across is that it’s easy to move to Power Systems. It’s faster, it’s more cost-effective, and you still get all the ease of use and ease of management that you’re used to with your existing dashDB environment.

Mona Patel: OK, last question: where can our readers go to learn more about dashDB Local on Power? Can they try out dashDB Local on Power Systems before they buy?

Mukta Singh: Yes, we offer a free trial with a Docker ID – please visit dashDB.com to learn more and access the trial.

About Mona,

mona_headshotMona Patel is currently the Portfolio Marketing Manager for IBM dashDB, the future of data warehousing.  With over 20 years of analyzing data at The Department of Water and Power, Air Touch Communications, Oracle, and MicroStrategy, Mona decided to grow her career at IBM, a leader in data warehousing and analytics.  Mona received her Bachelor of Science degree in Electrical Engineering from UCLA.

One Cloud Data Warehouse, Three Ways

by Mona Patel

There’s something very satisfying about using a single, cloud database solution to solve many business problems.  This is exactly what BPM Northwest experiences with IBM dashDB when delivering Data and Analytics solutions to clients worldwide.

The exciting success with dashDB compelled BPM Northwest to share implementations and best practices with IDC.

In the webcast they team up to discuss the value and realities of moving analytical workloads to the cloud.   Challenges around governance, data integration, and skills are also discussed as organizations are very interested and driven to seize the opportunities of a cloud data warehouse.

In the webcast, you will hear three ways that you can utilize IBM dashDB:

  • New applications, with some integration with on-premises systems
  • Self-service, business-driven sandbox
  • Migrating existing data warehouse workloads

After watching the webcast, please think about how IBM dashDB use cases discussed can apply to your challenges and if a hybrid data warehouse is the right solution for you.

Want to give IBM dashDB on Bluemix a try?  Before you sign up for a free trial, take a tutorial tour on the IBM dashDB YouTube channel to learn how to load data from your desktop, enterprise, and internet data sources, and then see how to run simple to complex SQL queries with your favorite BI tool, or integrated R/R Studio. In fact, watch how IBM dashDB integrates with other value added Bluemix services such as Dataworks Lift and Watson Analytics so that you can bring together all relevant data sources for newer insights.

mona_blog

About Mona,

mona_headshotMona Patel is currently the Portfolio Marketing Manager for IBM dashDB, the future of data warehousing.  With over 20 years of experince analyzing data at The Department of Water and Power, Air Touch Communications, Oracle, and MicroStrategy, Mona decided to grow her career at IBM, a leader in data warehousing and analytics.  Mona received her Bachelor of Science degree in Electrical Engineering from UCLA.

Why Are Customers Architecting Hybrid Data Warehouses?

By Mona Patel

As a leader in IT, you may be  incented or mandated to explore cloud and big data solutions to transform rigid data warehousing environments into agile ones to match how the business really wants to operate.  The following questions must come to mind:

  • How do I integrate new analytic capabilities and data sets to my current on-premises data warehouse environment?
  • How do I deliver self service solutions to accelerate the analytic process?
  • How do I leverage commodity hardware to lower costs?

For these questions, and more, organizations are architecting hybrid data warehouses.  In fact, these organizations moving towards hybrid are referred to as ‘Best In Class’ according to The Aberdeen Group’s latest research: “Best In Class focus on hybridity, both in their data infrastructure and with their analytical tools as well.  Given the substantial investments companies have made in their IT environment, a hybrid approach allows them to utilize these investments to the best of their ability while explore more flexible and scalable cloud-based solutions as well.”  To hear more about these ‘Best In Class’ organizations, watch the 45 minute webcast.

How do you get to this hybrid data warehouse architecture with the least risk and most reward?  IBM dashDB delivers the most flexible, cloud database services to extend and integrate with your current analytics and data warehouse environment, addressing all the challenges related to leveraging new sources of customer, product, and operational insights to build new applications, products, and business models.

To help our clients evaluate hybrid data warehouse solutions, Harvard Research Group (HRG) provides an assessment of IBM dashDB.  In this paper, HRG highlights product functionality, as well as 3 uses cases in Healthcare, Oil and Gas, and Financial Services.   Security, Performance, High Availability, In-Database Analytics, and more are covered in the paper to ensure future architecture enhancements optimize IT rather than adding new skills, complexities, and integration costs. After reading this paper, you will find that dashDB enables IT to respond rapidly to the needs of the business, keep systems running smoothly, and achieve faster ROI.

To know more on dashDB check out the video below:

 

About Mona,

mona_headshotMona Patel is currently the Portfolio Marketing Manager for IBM dashDB, the future of data warehousing.  With over 20 years of analyzing data at The Department of Water and Power, Air Touch Communications, Oracle, and MicroStrategy, Mona decided to grow her career at IBM, a leader in data warehousing and analytics.  Mona received her Bachelor of Science degree in Electrical Engineering from UCLA.

Start Small and Move Fast: The Hybrid Data Warehouse

by Mona Patel

In the world of cutting edge big data analytics, the same obstacles in gaining meaningful insight still exists – ease of getting data in and getting data out.  To address these long standing issues, the utmost flexibility is needed, especially when layered with the agile needs of the business.

Why spend millions of dollars replacing your data and analytics environment with the latest technology promise to address these issues, when can you to leverage existing investments, resources, and skills to achieve the same, and sometimes better, insight?

Consider a hybrid data warehouse.  This approach allows you to start small and move fast. It provides the best of both worlds – flexibility and agility without breaking the bank.  You can RAPIDLY serve up quality data managed by your data warehouse, blended with newer data sources and data types in the cloud, and apply integrated analytics such as Spark or R – all without additional IT resources and expertise.  How is this possible?  IBM dashDB.

Read Aberdeen’s latest report on The Hybrid Data Warehouse.

mona's blog

 

Watch Aberdeen Group’s Webcast on The Hybrid Data Warehouse.

Let me give you an example.  We live in a digital world, with organizations now very interested in improving customer data capture across mobile, web, IoT, social media, and more for newer insights.  A telecommunications client was facing heavy competition and wanted to quickly deliver unique mobile services for an upcoming event in order to acquire new customers by collecting and analyzing mobile and social media data.  Taking a hybrid data warehouse approach, the client was able to start small and move fast, uncovering new mobile service options.

Customer information generated from these newer data sources were blended together with existing customer data managed in the data warehouse to deliver newer insights.  IBM dashDB provided a high performing, public cloud data warehouse service that was up and running in minutes.  Automatic transformation of unstructured geospatial data into structured data, in-memory columnar processing, in-database geospatial analytics, integration with Tableau, and pricing were some of the key reasons IBM dashDB was chosen.

This brings me back to my first point – you don’t have to spend millions of dollars to capitalize on getting data in and getting data out.  For example, clients like the one described above took advantage of Cloudant JSON document store integration, enabling them to rapidly get data into IBM dashDB with ease– no ETL processing required.  Automatic schema discovery loads and replicates unstructured JSON documents that capture IoT, Web and mobile-based data into a structured format.  Getting data or information out was simple, as IBM dashDB provides in-database analytics and the use of familiar, integrated SQL based tools such as Cognos, Watson Analytics, Tableau, and Microstrategy.  I can only conclude that IBM dashDB is a great example of how a highly compatible cloud database can extend or modernize your on-premises data warehouse into a hybrid one to meet time-sensitive business initiatives.

What exactly is a hybrid data warehouse?  A hybrid data warehouse introduces technologies that extend the traditional data warehouse to provide key functionality required to meet new combinations of data, analytics and location, while addressing the following IT challenges:

  • Deliver new analytic services and data sets to meet time-sensitive business initiatives
  • Manage escalating costs due to massive growth in new data sources, analytic capabilities, and users
  • Achieve data warehouse elasticity and agility for ALL business data

mona_dashDB

Still not convinced on the power of a hybrid data warehouse?  Hear what Aberdeen Group’s expert Michael Lock has to say in this 30 min webcast.

About Mona,

mona_headshot

Mona Patel is currently the Portfolio Marketing Manager for IBM dashDB, the future of data warehousing.  With over 20 years of analyzing data at The Department of Water and Power, Air Touch Communications, Oracle, and MicroStrategy, Mona decided to grow her career at IBM, a leader in data warehousing and analytics.  Mona received her Bachelor of Science degree in Electrical Engineering from UCLA.

IBM dashDB Local preview to be featured at Cloud Expo East 2016 in New York

by Cindy Russell, IBM Data Warehouse Marketing

With an expected attendance of over 6,000 from June 7th through 9th, Cloud Expo New York is the one and only original event in which technology vendors can meet to experience and discuss the entire world of the Cloud.

On June 7, 2016, we are thrilled to be rolling out IBM dashDB Local™ as an open preview.  This exciting new IBM hybrid data warehouse offering is part of the dashDB family of data management solutions. dashDB Local delivers dashDB technology via container technology for implementations such as private and virtual private clouds. It is an ideal solution when you want cloud-like simplicity, yet need more control over applications and data. Participants in this preview program have been very enthusiastic about this technology, and you can read more in Mitesh Shah’s blog.

dashDB Local is:

·        Open: Leverage the power of an open warehouse platform

·        Flexible & Hybrid: Easily deploy the right workload to the right platform

·        Fast: Quickly realize business outcomes with advanced processing technologies

·        Simple: Automated management features help to lower the analytics cost model

 

dashDB Local will also be featured in this speaking session at Cloud Expo!

Building Your Hybrid Data Warehouse Solution with dashDB

Matthias Funke, Hybrid Data Warehouse Offering and Strategy Lead at IBM

June 7, 2016 from 11:40 AM – 12:15 PM

 

See the dashDB family in action at IBM Booth #201, the SoftLayer Booth!

cloud expo booth

Register to attend

IBM Booth #201

June 7, 2016

Cloud Expo East 2016

Javits Center, New York, NY

Note: First time attendees will need to create a new account.

As you respond to increasing requests for new analytics, you need fast and flexible technology. Learn how dash DB’s cloud-based managed service and new container-based edition gives you the speed and flexibility that you need.

Want to get started now with dashDB Local? Learn about and download our preview here: ibm.biz/dashDBLocal

How dashDB Helps Media Channels Boost Revenues And Viewership

By Harsimran Singh Labana

Did you ever wonder how a media channel decides which ad comes at what time? Well, there is an analytics science behind this.

Cable and broadcast networks pay studios large sums of money for the right to broadcast a specific show or movie at specific times on specific channels. To achieve a return on that investment, networks must design TV schedules and promotional campaigns to maximize viewership and boost advertising revenues.

RSG Media is an IBM dashDB managed service client that partners with cable and broadcast, entertainment, games and publishing firms to provide insights that help maximize revenue from content, advertising and marketing inventories. Shiv Sehgal, Solutions Architect, RSG Media says, “We had the rights data, the scheduling data and the advertising revenues data. If we could combine this with viewership and social media data, we could give our clients a true 360-degree view of their operations and profitability, down to the level of individual broadcasts. The missing piece of the puzzle was to build a data and analytics capability that could bring all the data together and turn it into business insight – and that’s where IBM came in.”

RSG Media chose IBM because of its complete vision for cloud analytics. This includes an integrated set of solutions for building advanced analytics applications and coordinating them with all the relevant data services in the cloud.

RSG Media’s Big Knowledge Platform is built on the IBM® Cloudant® NoSQL document store and the IBM dashDB™ data warehouse service, orchestrated through the IBM Bluemix® cloud application development platform. Cloudant’s Schema Discovery Process (SDP) is used to ingest and translate semi-structured data from more than 50 sources, and structure that data into a schema that the dashDB relational data warehouse understands.

RSG Media is not stopping here and they are excited about Watson Analytics and how it predicts customer behavior.  Learn more about RSG Media success using dashDB and Cloudant solutions on Bluemix.

About Harsimran,
HarryHarsimran Singh Labana is the Portfolio Marketing Manager for IBM’s Data Warehousing team. Working in a worldwide role he ensures marketing support for IBM’s solutions. He has been with IBM for close to five years working in diverse roles like sales and social media marketing. He stays in Bangalore, India with his wife and son.

What you need to know: Software-Defined Environments (SDE) for data warehousing and more

by James Cho and Maria Attarian

There’s a new kid on the block and it’s called SDE!  This is a new term that stands for Software Defined Environment (SDE), and it is here to change the way we think about the world of application, integration and middleware – as well as data warehouses. But first things first.  Let’s talk about the SDE and how it can help you as you deliver more end-user services more easily.

What is an SDE and why use one when you have traditional environment approaches?

Put simply, a Software Defined Environment (SDE) optimizes the entire computing infrastructure — compute, storage and network resources. An SDE can automatically tailor itself to meet the needs of the workload that must be executed.

In comparison to traditional environment approaches, compute, storage, and network resources are allocated and assigned to workloads manually and this is the problem from which the need for the SDE technology emerged. In order to remove manual steps, SDEs takes into account application characteristics, best-available resources and service level policies when dynamically allocating resources to workloads. An SDE also strives to deliver continuous, “on the fly” optimization and reconfiguration to address infrastructure issues.

So what are the fundamental ingredients to doing this? Policy-based compliance checks and updates are essential and make an SDE easy to manage. Delivering in the public or private cloud requires high-speed analytic processing capabilities, as well as rapid integration, automation and optimization. When factors such as these are in place, it becomes clear that SDE technology helps accelerate business success and brings value to the customer because the solution is responsive and adaptive.

So what does IBM have to offer in the SDE space?

dashDB Local (currently in preview) is the IBM data warehouse offering for SDEs such as private clouds, virtual private clouds and other infrastructures that support the Docker container technology. It is designed to provision a full data warehouse stack in minutes and helps you manage the service in your own public or private cloud, while maintaining existing operational and security processes.

There are three design principles that dashDB Local tackles head-on based on feedback from our customers.

  1. So simple that anyone can deploy it

By packaging our software stack into a Docker container, provisioning dashDB Local can be as simple as one docker run command on Linux servers that have the Docker engine installed. It can be as easy as a Docker hub search for “dashDB” followed by a single click on “CREATE” using Docker kitematic on Windows or Mac machines. Software stack updates are as simple as your mobile app using the same docker run command against a new version of the container on your existing installation.

  1. Flexible enough to deploy anywhere

dashDB Local can be deployed on any supported Docker installations on Linux, Cloud, and on OSX and Windows platforms with minimal prerequisites. Entry level hardware requirements start at 8GB RAM and 20GB of storage, which is suitable for a development / test environment or QA work on your laptop. For larger servers like 48 core 3 TB RAM servers, the dashDB container will auto-configure to the host it is installed on. Persistent durable storage of your choice must be mounted in /mnt/clusterfs to hold your data. To summarize, it is flexible enough to empower you to use the hardware what you already have in your data center or in the public cloud of your choice.

  1. Independent of your infrastructure capabilities

Existing monitoring and security overlays can remain on your Host OS while the dashDB stack is isolated inside its container. You can fully utilize existing infrastructure capabilities like copy and replication services of your storage. Existing monitoring tools such as systems management, network monitoring, even popular cloud management tools such as openstack, kubernetes, or public cloud monitoring tools like AWS Cloudwatch can continue to be used. The isolation of a dashDB Local container allows you to embrace your own data center standards. Thus, it is independent and empowers you to do what you already know how to do.

For more information on dashDB Local, please visit the public Docker repository. An early access preview of dashDB Local is now available. Test it out and help shape the solution. Test it for yourself.  Request dashDB Local preview access here.

About James and Maria,

James ChoJames Cho is a Senior Technical Staff Member and  Chief Architect for IBM dashDB Local. He has been a technical leader of integrated warehouse solutions and appliances at IBM for over 15 years. He currently focuses on data warehouse solutions delivered in public and private cloud data centers. His previous experience includes Data Warehouse DBA, publication of Industry standard TPC performance benchmarks, and BI architecture and deployment responsibilities. James is a 1996 graduate of the University of Texas.  He holds a bachelor of science in computer science.

Follow James on LinkedIN

 maria attarianMaria Attarian has worked for IBM for the past four years on data warehousing technologies such as PureData System for Analytics and dashDB. In her focus as a software engineer, Maria is charged with new and innovative ways to deliver better products for clients and takes on a variety of challenges in this role. Maria is also active with the IEEE Young Professionals Toronto Chapter.  She served as the chairperson of this group for more than a year and remains an active member of the group.  Maria hold a master’s degree from the University of Waterloo and a bachelor of engineering degree from the National Technical University of Athens in Greece.