Data Growth

The Internet, data tracking systems, and compliance requirements is driving data volume growth at alarming rates. On average, data volumes double every nine months, twice as fast as Moore's Law governing the growth in computer processing power.

This widening gap between computing power and data growth is compounded by the increasing need for in-depth analysis, as organisations move from standard reporting toward interactive ad hoc BI, predictive analysis and the need for near real-time results.

This rapid growth in database size has to date been managed with very expensive and consistent upgrading of hardware and software. However, this trend can no longer keep up with the demands of the business. In addition, the economic impact of this approach is unfavourable.

Todays businesses need a new approach to data warehousing infrastructure that is both specific and flexible. Not only must it meet the demands of the busienss, it must also be easy to deploy and compatible with the customer's existing BI applications and infrastructure.

Demands of BI Infrastructure

Current State of BI

The current BI infrastructure is a collection of hardware, software and storage:

• Database Management System [DBMS] - transaction processing, holding several hundred megabytes worth of records with a few internal users. DBMS has been improved in increasingly complex layers to support terabyte-sized databases and evolving SQL definition.
• Hardware/Operating System - a clustered set of generic boxes optimized for everything from mathematical queries to genome investigation.
• Generic File Systems - manage and serve data for a variety of applications.

Not one of these disparate capabilities has kept pace with database volumes, complexity or performance demands, in spite of attempts such as:

DBMS grid management - adds another layer to already complex DBMS

Increasingly complex SMP boxes and storage area network [SAN] and network attached storage [NAS] designed to improve transactional workloads but have delivered only incremental benefits in the BI space.

Grid and blade architectural improvements - developed for transaction systems, and do not meet the demands of BI

Adding more hardware is NOT the answer to growing BI problems. Doing so requires constant tuning and optimising of user applications. These systems are continually strained by Terabyte-scale databases.

In addition, such patchwork approaches are extremely difficult and time-consuming to manage and maintain.

The current solution to this problem is a data warehouse appliance. In a BI environment, a data warehouse appliance is a machine capable of retrieving decision-support intelligence from terabytes of data in seconds or minutes versus hours or days. They are the only way in which real time or near time decision data can be made available.

Next: Data Warehouse Solutions

Back To Top