CIO Bulletin

WinterCorp: The Leading Experts in Scalable Data Management

Cloud data warehousing is booming as never before. No hardware commitments required, no data center space, no data center staff, no capital approvals. A simple proof-of-concept can often be done on a free trial. At first, everything looks good … and, soon you can sail straight off toward the horizon with your new cloud data warehouse.

But, about 80% of larger scale cloud data warehouse projects soon find themselves in deep water and stormy weather for which they are ill-equipped.

The problems arise when the data warehouse involves enterprise levels of scale and complexity. The cloud can always provide more resources, but the data warehouse engine may or may not be up to the job. If the engine – or the database design – cannot scale effectively, the project will still fail. Some common problems with the cloud data warehouse in full scale operation are:

it costs too much
it takes too long to load data or respond to queries
it bogs down when there are too many concurrent users or jobs
it is unreliable or must be taken down too often
it is fragile and not easily modified or extended
it is immature and lacks critical functions

These problems are usually not because the data warehouse is in the cloud, but because it is on the wrong cloud data warehouse engine or uses an incorrect database or solution design.

The sinister part of this is that the platform looks fine during the proof-of-concept. The performance and cost problems only emerge as the system approaches its full operating capability. An insufficient or incomplete proof-of-concept is genuinely dangerous and risky for the customer, often leading to catastrophic project failure, loss of investment and unacceptable delay and disruption of critical business objectives.

Customers are also misled by references and case studies where reference account is not solving a problem of comparable scale and complexity. Often the cloud vendors point to large data warehouses in production that sound impressive. But, frequently these shining examples of success are just overgrown, but structurally simple, data marts. An integrated data warehouse at enterprise scale is a whole different story.

The scalability challenge is about much more than database size. Other factors that matter are database complexity, query complexity, update volume, data latency, data variety, user concurrency, workload mix, service levels needed, and uptime needed. All of these factors interact with database size to create challenges in performance, scale, and cost – even when the data warehouse is running in a cloud that itself is highly scalable.

WinterCorp is an expert consultant in strategic data management at scale founded by Richard Winter in 1992. Since then, the company has been deeply involved in new developments in data warehousing, analytics and related areas as well as helping customers create the data foundation and data strategies they need to benefit from advances in data science, machine learning and artificial intelligence. WinterCorp has frequently assisted leading companies on their choice of cloud data warehouse platform conducting evaluations, proofs-of-concept and scalability assessments for medium-size and larger enterprises.

In one engagement, a pharma client retained WinterCorp because they had a block buster drug in jeopardy. They could not react to adverse events, often called “real world evidence”, quickly enough. Regulators might have withdrawn approval, putting a billion dollar investment at risk. WinterCorp created a data architecture and recommended a data platform. Now the pharma company can identify new drug safety issues in real time to protect their patients, greatly reducing their risk.

In another case, a large regional bank retained WinterCorp because they feared that regulators would impose unreasonable capital requirements. They could not make timely, accurate reports against upcoming regulatory deadlines. In an independent review, WinterCorp recommended a data warehouse platform that could eliminate data silos and scale to meet their objectives. Now the bank is getting timely, integrated data to support better business decisions and they are able to report on schedule with accurate, consistent data.

Data Platform Assessment/Selection

Richard Winter says that the WinterCorp approach to platform requirements is different, because it begins with a quantified estimate of the customer’s unique requirements. These estimates are based on the customer’s business interests and plans, workloads, database structure and industry and technical trends. WinterCorp takes expected growth and change in the strategic data requirements into account. The resulting evaluation, as well as any needed tests or proofs-of-concept, therefore account for the scale and complexity that the customer actually needs to meet key business objectives.

WinterCorp data platform assessments and evaluations are often enhanced by the WinterCorp Cloud Data Warehouse Lab in which Dr. Norbert Kremer leads the design and performance of live tests and proofs-of-concepts of data warehouse performance, scalability and cost.

Many customers make the mistake of choosing a data warehouse platform based on standard practice or feature lists; by contrast, WinterCorp conducts an engineering evaluation so that the customer’s needs for performance, cost control, scale and data availability are satisfied by means of a conscious strategy.

Scalability Assessment

One of the scariest things that Winter says he sees is a customer migrating a large, integrated data warehouse to the cloud without an affirmative plan to address performance, scalability and cost. Often the target platform is selected without concern for these issues. According to Winter, an executive with a stake in such a project needs to ask, “How do we know that our service levels will be met? How do we know what we will have to spend in production? How do we know that we have a feasible solution with this data warehouse platform?”.

If there is no convincing quantitative answer - if there is no test result, analysis or model to show that the key hurdles will be cleared - then the executive should insist on an independent review to assess the situation. No company should be in the position of relying on hope – or vendor claims – when a large investment or a critical business objective – is at stake.

It is possible to measure, manage and control the engineering risks in a large data warehouse project. The necessity to do so is not diminished by the adoption of the cloud based data warehouse.

WinterCorp can provide an independent assessment of the performance, scalability, cost and other engineering factors for a data warehouse planned, under development or even in production. The findings and recommendations from such a review have helped many customers avoid severe problems and achieve critical business goals.

About the CEO

A specialist in the technology and implementation of analytic data management at scale, Richard Winter advises clients on data strategy and data architecture, focusing on the modern data warehouse and the data lake. He has been retained to make architecture and platform recommendations or perform engineering tests for more than 50 leading enterprises, government agencies, and technology vendors.

In 1992, Mr. Winter founded WinterCorp, one of the few consulting companies in existence that is focused entirely upon data management at a large scale. He currently serves as the CEO and principal architect of WinterCorp. He has been an expert in the technologies of the analytic data platform, commercial and open-source, relational and non-relational, cloud, on-premise, and hybrid, streaming, and batch. Mr. Winter also serves on the faculty of TDWI, The Data Warehouse Institute, where he teaches courses on the architecture and scalability of cloud data warehouse platforms.