Current data analysis needs of enterprises is growing, and needs more and more self-service and flexibility, we urgently need the flexibility to support multiple data sources quickly
integrated query, reporting and processing of fast visualization, the following is a collection of some of their own understanding and learning, carding a simple data analysis and data
display program
Overall reference architecture design
The layers Description
- data mesh layer
mainly using a variety pg fdw integrate multiple sources of data, data processing may be added while the outer (such as trying materialized, storage column analysis, data conversion) - Data virtualization layer
based pgspider, accelerate a variety of data query, data source that provides the ability to provide screening, health checks and the ability to - Api generic data layer
mainly based cube.js hasura graphql-engine and the ability to provide a variety of data api (rest and graphql) - Api security and routing layer
based api Gatway (or may be openresty apisix and provide routing or security control) - Data presentation layer
can provide flexible data may show this report are based on cube.js data demonstrate the ability and graphql2chartjs
Explanation
The above is a simple integration process, there are a lot of details need to be addressed
Reference material
https://github.com/hasura/graphql2chartjs
https://github.com/cube-js/cube.js
https://github.com/hasura/graphql-engine
https://github.com/rongfengliang/pgspider-docker
https://hub.docker.com/repository/docker/dalongrong/pgspider
https://github.com/pgspider/pgspider
https://www.cnblogs.com/rongfengliang/p/12368353.html