Data warehouse appliance

Data Warehouse Appliance
A data warehouse (DW) appliance is an integrated set of servers, storage, OS, DBMS and software specifically pre-installed and pre-optimized for data warehousing. DW appliances provide solutions for the mid-to-large volume data warehouse market, offering low-cost performance on data volumes in the terabyte to petabyte range. Data warehouse appliance vendors provide and service all the parts within the appliance.

As companies recognize the benefits gained from analyzing data and business intelligence, the demand for business analytics explodes. Data warehouses that service analytical queries must reliably support increased workloads as well as rapid data growth. The average data warehouse expands between 50% and 100% each year. The high cost to meet new workload and capacity requirements has allowed DW appliance vendors to enter the market with data warehouse platforms that offer reduced administration and attractive price/performance value.

In a nutshell, DW appliances provide low-cost packaged solutions for parallel data warehouse performance unmatched in many alternate solutions.

Technology Primer
Most DW appliance vendors use massively parallel processing (MPP) architectures to provide high query performance and platform scalability. MPP architectures consist of independent processors or servers executing in parallel. Most MPP architectures implement a “shared nothing architecture” where each server is self-sufficient and controls its own memory and disk. Shared nothing architectures have a proven record for high scalability and little contention. DW appliances distribute data onto dedicated disk storage units connected to each server in the appliance. This distribution allows DW appliances to resolve a relational query by scanning data on each server in parallel. The divide-and-conquer approach delivers high performance and scales linearly as new servers are added into the architecture.

MPP database architectures are not new. Teradata, Tandem, Britton Lee and Sequent designed the first MPP SQL-based architectures. The re-emergence of MPP data warehouses is due to open source and commodity components. Advances in technology have reduced costs and improved performance in storage devices, multi-core CPUs and networking components. Open source RDBMS products, such as Ingres and PostgreSQL, reduce software licenses and allow DW appliance vendors to focus on optimization rather than providing basic database functionality. Open source Linux provides a stable, well-implemented OS for DW appliances.

History
Many consider Teradata’s initial product as the first DW appliance. Since then, Teradata has changed its architecture to a more software-centric multi-platform solution. While Teradata retains some appliance-like benefits, analysts consider Netezza Performance Server as the forerunner in today's data warehouse appliance market, with its fast table scans but limited flexibility.

More recently, a second generation of DW appliances has emerged, marking the move to mainstream vendor integration. The Oracle Information Appliance Initiative combines Oracle Database 10g, with the industry’s leading computer manufacturers NetApp, EMC, HP, IBM, PANTA and Sun Microsystems. Other DW appliance vendors have partnered with major hardware vendors to bring their appliances to market. DATAllegro partners with EMC and Dell and implements open source Ingres on Linux. Greenplum has a partnership with Sun Microsystems and implements Bizgres (a form of PostgreSQL) on Linux. HP Neoview has a wholly owned solution and uses HP NonStop SQL.

Kognitio and ParAccel are “virtual” data warehouse appliances. These solutions provide software-only solutions deployed on clusters of commodity hardware. Kognitio’s homegrown WX2 database runs on several blade configurations. Other players in the DW appliance space include Calpont and column-based Vertica.

The newest entry into the data warehouse appliance market is Dataupia with a product accelerating the performance of Oracle, Microsoft SQL Server, and IBM DB2.

Recently, the market has seen the emergence of data warehouse bundles where vendors combine their hardware and database software together as a data warehouse platform. IBM bundled DB2 Warehousing edition with its i-Series servers to create the BCU (Balance Configuration Unit). Some analysts classify IBM’s BCU as a bundle, while others classify it as a semi-appliance.

Benefits
Reduction in Costs

The total cost of ownership (TCO) of a data warehouse consists of initial entry costs, on-going maintenance costs and the cost of increasing capacity as the data warehouse grows. DW appliances offer low entry and maintenance costs. Initial costs range from $10,000 to $150,000 per terabyte, depending on the size of the DW appliance installed.

The resource cost for monitoring and tuning the data warehouse makes up a large part of the TCO, often as much as 80%. DW appliances reduce administration for day-to-day operations, setup and integration. Many also offer low costs for expanding processing power and capacity.

With the increased focus on controlling costs combined with tight IT Budgets, data warehouse managers need to reduce and manage expenses while leveraging their technology as much as possible making DW appliances a natural solution.

Parallel Performance

DW appliances provide a compelling price/performance ratio. Many support mixed-workloads where a broad range of ad-hoc queries and reports run simultaneously with loading. DW appliance vendors use several distribution and partitioning methods to provide parallel performance. Some DW appliances scan data using partitioning and sequential I/O instead of index usage. Other DW appliances use standard database indexing.

With high performance on highly granular data, DW appliances are able to address analytics that previously could not meet performance requirements.

Reduced Administration

DW appliances provide a single vendor solution and take ownership for optimizing the parts and software within the appliance. This eliminates the customer’s costs for integration and regression testing of the DBMS, storage and OS on a terabyte scale and avoids some of the compatibility issues that arise from multi-vendor solutions. A single support point also provides a single source for problem resolution and a simplified upgrade path for software and hardware.

The care and feeding of DW appliances is less than many alternate data warehouse solutions. DW appliances reduce administration through automated space allocation, reduced index maintenance and in most cases, reduced tuning and performance analysis.

Built-in High Availability

DW appliance vendors provide built-in high availability through redundancy on components within the appliance. Many offer warm-standby servers, dual networks, dual power supplies, disk mirroring with robust failover and solutions for server failure. Scalability

DW appliances scale for both capacity and performance. Many DW appliances implement a modular design that database administrators can add to incrementally, eliminating up-front costs for over-provisioning. In contrast, architectures that do not support incremental expansion result in hours of production downtime, during which database administrators export and re-load terabytes of data. In MPP architectures, adding servers increases performance as well as capacity. This is not always the case with alternate solutions.

Rapid Time-to-Value

Companies increasingly expect to use business analytics to improve the current cycle. DW appliances provide fast implementations without the need for regression and integration testing. Rapid prototyping is possible because of reduced tuning and index creation, fast loading and reduced needs for aggregation in some cases.

Application Uses
DW appliances provide solutions for many analytic application uses, including:


 * Enterprise data warehousing
 * Super-sized sandboxes isolate power users with resource intensive queries
 * Pilot projects or projects requiring rapid prototyping and rapid time-to-value
 * Off-loading projects from the enterprise data warehouse; ie large analytical query projects that affect the overall workload of the enterprise data warehouse
 * Applications with specific performance or loading requirements
 * Data marts that have outgrown their present environment
 * Turnkey data warehouses or data marts
 * Solutions for applications with high data growth and high performance requirements
 * Applications requiring data warehouse encryption

Trends
The DW appliance market is shifting trends in many areas as it evolves:


 * Vendors are moving toward using commodity technologies rather than proprietary assembly of commodity components.
 * Implemented applications show usage expansion from tactical and data mart solutions to strategic and enterprise data warehouse use.
 * Mainstream vendor participation is now apparent.
 * With a lower total cost of ownership, reduced maintenance and high performance to address business analytics on growing data volumes, most analysts believe that DW appliances will gain market share.