Author: Zen and the Art of Computer Programming
Impala is an open source subproject of Apache Hadoop, a distributed data warehouse (DW) product. It was originally supported by Cloudera in 2012. Impala is a SQL query engine based on Apache Hive. Impala can be deployed with other components in the Hadoop technology stack. For example, it can be used with MapReduce, Pig or HDFS.
This article will focus on Impala's high-availability solution design in terms of deployment and management.
High availability is a very important job in the IT industry to ensure that the quality of service is not compromised. With the explosive growth of Internet companies and the surge in business volume, more and more enterprises need to run multiple services on the same platform to improve user experience, improve efficiency and save costs. To meet this challenge, the advantages of cloud computing such as elastic scaling and fast delivery are rapidly changing the traditional IT operation and maintenance model. Therefore, cloud service providers are also developing high-availability solutions for their customers.
A big data cluster running in a production environment typically has the following characteristics:
- The amount of data is huge: the amount of stored data exceeds the capacity of the disk;
- A large number of concurrent queries: handle up to millions of queries at the same time;
- Real-time query requirements: need to respond to real-time query requests;
- Complex data dependencies: there are many different types of applications, databases, and external systems, and these dependencies make maintaining a unified data model a complex task;
- Scalability requirements: Clusters need to be able to dynamically increase or decrease resources to meet real-time changing workloads.
Based on these characteristics, a high availability solution should have the following characteristics:
- Availability: The cluster as a whole must remain in normal operation;
- Failover: When the cluster fails, it must be quickly transferred from the wrong node to another working node;
- Rolling upgrades: must be able to