Shib+Presto ad hoc query

1.1. Objectives of ad hoc queries

The user is a data analyst for product/operations/sales operations;

Data analysts are required to master query SQL query script writing skills, and master the data storage of different businesses in different data marts;

Regardless of whether their computing tasks are submitted to the database or Hadoop, the computing time may be very long, and it is impossible to wait online;

so,

The user submits a computing task (PIG/SQL/Hive SQL), the console informs the task that the task has been queued, and gives friendly prompts such as the approximate computing time. The weight of these tasks is low.

Users and administrators can view the computing tasks in the queue, including the execution time, running time and running results of the executed tasks;

When the calculation task has results, the console interface will have a notification prompt, or send an email prompt, and users can view and download the data online.

1.2. Current technology selection of ad hoc query

Graphical interface: Shib;

Data query engine: Facebook Presto.

1.3.Shib

Development language: Nodejs

Advantages: Shib can directly operate hive and presto through simple configuration. The amount of code is relatively small, and the amount of work to modify is much less.

Disadvantages: Unfamiliar with Nodejs, there is a learning cost.

 

The article is transferred from: http://www.cnblogs.com/zhengyun_ustc/p/55solution7.html