Introduction: This article introduces how to use RDS PG or PolarDB (compatible with PG version or Oracle version) of the database fast display technology provided by the Ganos space-time engine, and only use a hundred lines of code to realize the online fast display and smooth map interaction of hundreds of millions of massive geometric spatial data. And there is no need to pay attention to slice storage and efficiency issues.
Author: Li He
01 Introduction
How to perform online visualization of hundreds of millions of vector spatial data in spatiotemporal databases has always been a problem in the industry. Due to the large amount of data, the traditional method requires the data in the database to be published based on cache slicing in order to be visualized, the operation process is lengthy, and there are a lot of issues to be considered:
- If pre-slicing vector data, how long does the data need to be sliced? How many levels are appropriate? Is there enough hard disk space for storing tiles?
- If real-time tiles are used, can the response time of real-time rendering tiles be guaranteed?
- If you use vector tiles, how big might a small-scale tile be? Will transmission become a bottleneck? How much data can the front-end rendering support?
If you want to quickly browse the large-scale online data in the database, the traditional offline slicing production process used for "basemap services" is almost incomprehensible, which is time-consuming and labor-intensive, and cannot be processed online. Black technology is coming. This article introduces how to use RDS PG or PolarDB (compatible with PG version or Oracle version) of the database fast display technology provided by the Ganos space-time engine, and only use a hundred lines of code to realize the online fast display and smoothness of hundreds of millions of massive geometric spatial data Map interaction, and there is no need to pay attention to slice storage and efficiency issues.
02 Interpretation of technical characteristics
The core of Ganos's online quick display processing is to associate the database with visualization, providing a new visualization indexing technology-Sparse Vector Pyramid (SVP) index. SVP has two key features: fast and economical .
Among them, Kuai refers to two stages of Kuai:
- Pyramid creation is fast : Ganos uses spatial index to divide the data in space and establishes a sparse vector pyramid index based on the density, which reduces the amount of data calculation by 90% compared with the traditional graph cutting process. At the same time, the creation of the pyramid adopts a completely parallel processing mode, and even if the 100 million plot data is generated, it only takes about 10 minutes to generate the pyramid.
- Fast data display : Ganos adopts a visual visibility elimination algorithm to filter out a large amount of data that does not affect the display effect according to the Z-order order, thereby speeding up the efficiency of real-time display. Ganos supports the direct output of raster tiles in PNG format and vector tiles in MVT format, and the response time of real-time rendering and display of 100 million land-like pattern data reaches the second level.
Province also has two dimensions:
- Save disk space : Pyramid index generated from 100 million plot data only occupies extra space of 5% of the original table.
- Save development time : using only simple SQL statements, you can flexibly control the display effect by adjusting the statement parameters.
03 Use steps
Ganos's quick display engine is very simple to use, and SQL functions have been highly encapsulated. It should be noted that before using the quick display engine for the first time, you need to explicitly create the corresponding extension module. The executed statement is as follows:
CREATE EXTENSION ganos_geometry_pyramid CASCADE;
By executing the above statement, the computing components of the pop-up engine will be loaded.
3.1 Build a sparse vector pyramid
Suppose you have created a large vector table and imported data, then you can use Ganos's st_buildpyramid method to create a vector pyramid.
The prototype of the method is as follows . For more detailed parameter descriptions, please refer to the official documentation .
boolean ST_BuildPyramid(cstring table, cstring geom, cstring fid, cstring config)
Note: *Swipe left and right to read
among them
- table: The name of the table where the vector data is located.
- geom: Vector field name.
- fid: The unique identifier of the vector feature record, supports Int4/Int8 type.
- config: The configuration parameter string in json format.
-
- In this example, we specify the name of the vector pyramid and the logical tile size used (this tile size is not a real tile, but only represents a logical division of space)
The actual call is as follows:
ST_BuildPyramid('points', 'geom', 'gid', '{"name":"points_geom","tileSize":512}')
Note: *Swipe left and right to read
We have created a vector pyramid for the geom field of the points table. The name of the pyramid is specified as points_geom , and the logical tile size of the pyramid is set to 512.
3. 2 Get grid tiles
Grid tiles are tiles in the form of pictures, and are the most widely used form of map tiles. Ganos' ST_AsPng method provides the function of dynamically rendering vector data into raster tiles on the database side on demand. This function provides the most basic raster symbolization capabilities, and is more oriented to light-weight scenes that do not require complex symbolization, such as data management systems.
The prototype of the method is as follows . For more detailed parameter description, please refer to the official document :
bytes ST_AsPng( cstring name, cstring tile, cstring style)
among them
- name: The name of the pyramid table.
- tile: tile index row and column number, in the form of Z_X_Y.
- style: rendering style. We can adjust the rendering effect through the following parameters:
-
- point_size: point size, in pixels.
- line_width: line width, which affects the outer borders of line elements and area elements, in pixels.
- line_color: Line rendering color, which affects the outer borders of line and area elements. The first 6 digits are hexadecimal color, and the last 2 digits are hexadecimal transparency.
- fill_color: fill color, which works on surface elements.
- background: background color. Generally set to
FFFFFF00, that is , pure and transparent.
The actual call is as follows:
ST_AsPng('points_geom', '1_2_1','{"point_size": 5,"line_width": 2,"line_color": "#003399FF","fill_color": "#6699CCCC","background": "#FFFFFF00"}')
Note: *Swipe left and right to read
We get the vector tile with index row and column number x=2, y=1, z=1 from the vector pyramid , and render the vector tile as a raster tile according to the style we configured, and return the image in PNG format .
3. 3 Get vector tiles
Vector tiles are an emerging map tile technology, which has the flexibility of configuring styles on the front end. Using WebGL rendering, the effect is more beautiful, and map frameworks such as Mapbox can easily support this format. Using Ganos's ST_Tile method can provide the data in the vector pyramid in the form of vector tiles.
The prototype of the method is as follows . For more detailed parameter descriptions, please refer to the official documentation .
bytea ST_Tile(cstring name, cstring key);
among them
- name: The name of the pyramid. In this example, it is table name_vector field name .
- key: The row and column number of the tile index, in the form of Z_X_Y.
We only need to provide the standard TMS row and column number and the name of the pyramid table to call.
The actual call is as follows:
ST_Tile('points_geom', '1_2_1');
We get the vector tile with index row and column number x=2, y=1, z=1 from the vector pyramid, and the returned data is in standard MVT format.
04 Practical case
4.1 Test data
We prepare two sets of vector data as test examples.
The buildings table is surface data, with a total of 125 million pieces of data, showing the online visualization effect using grid tiles.
gid|geom | ---|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 1|MULTIPOLYGON(((-88.066953 34.916114 0,-88.066704 34.916114 0,-88.066704 34.91602 0,-88.066953 34.91602 0,-88.066953 34.916114 0))) 2|MULTIPOLYGON(((-87.924658 34.994797 0,-87.924791 34.99476 0,-87.924817 34.994824 0,-87.924685 34.994861 0,-87.924658 34.994797 0)))
Note: *Swipe left and right to read
The points table is point data, with a total of 107,000 pieces of data, showing the online visualization effect using vector tiles.
id|geom | --|------------------------------| 1|POINT (113.5350205 22.1851929)| 2|POINT (113.5334245 22.1829781)|
4.2 Full stack architecture
The database-quick display full-stack architecture includes three parts: database server, python server and user side. The full-stack architecture is shown in the figure below.
4.3 Server code
In order to simplify the code and focus more on logical description, we chose Python (compatible with Python 3.6 and above) as the back-end language, the web framework uses the Python-based Flask ( pip install flaskinstallation) framework, and the database connection framework uses Psycopg2 for Python (use pip install psycopg2to install).
It is worth mentioning that we have implemented the most basic functions. When the performance of the Web service itself becomes a bottleneck, it can be optimized according to different platforms and frameworks to obtain better response performance.
We first established a vector pyramid on the backend, and then implemented two interfaces respectively. The vector tile interface uses the data in the points table, and the raster tile interface uses the data in the buildings table, and defines the style for the front end to directly transfer. For the convenience of explanation, the back-end code provides two interfaces of vector raster at the same time, which can be selected as needed in actual use.
# -*- coding: utf-8 -*-
# @File: Vector.py
import json
from psycopg2 import pool
from threading import Semaphore
from flask import Flask, jsonify, Response, send_from_directory
import binascii
# Connection parameter
CONNECTION = "dbname=postgres user=postgres password=postgres host=YOUR_HOST port=5432"
class ReallyThreadedConnectionPool(pool.ThreadedConnectionPool):
"""For
multi-threaded connection pools, improve the response of map tile class high concurrency scenarios.
"""
def __init__(self, minconn, maxconn, *args, **kwargs):
self._semaphore = Semaphore(maxconn)
super().__init__(minconn, maxconn, *args, **kwargs)
def getconn(self, *args, **kwargs):
self._semaphore.acquire()
return super().getconn(*args, **kwargs)
def putconn(self, *args, **kwargs):
super().putconn(*args, **kwargs)
self._semaphore.release()
class VectorViewer:
def __init__(self, connect, table_name, column_name, fid):
self.table_name = table_name
self.column_name = column_name
# 创建一个连接池
self.connect = ReallyThreadedConnectionPool(5, 10, connect)
# 约定金字塔表名
self.pyramid_table = f"{self.table_name}_{self.column_name}"
self.fid = fid
self.tileSize = 512
# self._build_pyramid()
def _build_pyramid(self):
"""创建金字塔"""
config = {
"name": self.pyramid_table,
"tileSize": self.tileSize
}
sql = f"select st_BuildPyramid('{self.table_name}','{self.column_name}','{self.fid}','{json.dumps(config)}')"
self.poll_query(sql)
def poll_query(self, query: str):
pg_connection = self.connect.getconn()
pg_cursor = pg_connection.cursor()
pg_cursor.execute(query)
record = pg_cursor.fetchone()
pg_connection.commit()
pg_cursor.close()
self.connect.putconn(pg_connection)
if record is not None:
return record[0]
class PngViewer(VectorViewer):
def get_png(self, x, y, z):
# Default parameter
config = {
"point_size": 5,
"line_width": 2,
"line_color": "#003399FF",
"fill_color": "#6699CCCC",
"background": "#FFFFFF00"
}
# When using psycpg2, it is more efficient to return the binary data as a hexadecimal string
sql = f"select encode(st_aspng(' {self.pyramid_table}','{z}_{x}_{y}','{json.dumps(config)}'),'hex')"
result = self.poll_query(sql)
# Only when it is returned in the form of a hexadecimal string, it needs to be converted back
result = binascii.a2b_hex(result)
return result
class MvtViewer(VectorViewer):
def get_mvt(self, x, y, z):
# When using psycpg2, it is more efficient to return the binary data as a hexadecimal string
sql = f"select encode(st_tile(' {self.pyramid_table}','{z}_{x}_{y}'),'hex')"
result = self.poll_query(sql)
# Only when returning in the form of a hexadecimal string Need to convert it back
result = binascii.a2b_hex(result)
return result
app = Flask(__name__)
@app.route('/vector')
def vector_demo():
return send_from_directory("./", "Vector.html")
# Define table names, field names, etc.
pngViewer = PngViewer(CONNECTION,'usbf','geom','gid')
@app.route('/vector/png/<int:z>/<int:x>/< int:y>')
def vector_png(z, x, y):
png = pngViewer.get_png(x, y, z)
return Response(
response=png,
mimetype="image/png"
)
mvtViewer = MvtViewer(CONNECTION, 'points', 'geom', 'gid')
@app.route('/vector/mvt/<int:z>/<int:x>/<int:y>')
def vector_mvt(z, x, y):
mvt=mvtViewer.get_mvt(x, y, z)
return Response(
response=mvt,
mimetype="application/vnd.mapbox-vector-tile"
)
if __name__ == "__main__":
app.run(port=5000, threaded=True)
Note: *Swipe left and right to read
Save the above code as a Vector.py file and execute the python Vector.pycommand to start the service.
It is not difficult to infer from the code that no matter what language or framework we use, we only need to encapsulate the vector or raster tile SQL statement as an interface to achieve exactly the same function. Compared to publishing traditional map services, realizing online visualization with Ganos's vector pyramid function is a lighter and more user-friendly choice:
- Regarding the grid tiles, the style can be controlled by changing the code, and the flexibility is greatly enhanced.
- There is no need to introduce other third-party components, nor do you need to perform targeted optimization, there is a satisfactory response performance.
- You can choose any programming language and framework that users are familiar with, and there is no need for complex and professional parameter configuration, which is more friendly to non-geographical practitioners.
4.4 Client code
We choose Mapbox as the front-end map framework to display the vector tile layer and raster tile layer provided by the back-end, and configure the rendering parameters for the vector tile layer.
For the convenience of explanation, the front-end code adds two layers of vector and raster at the same time, which can be selected as needed in actual use.
We create a new file named Vector.html in the same file directory of the back-end code and write the following code. After the back-end service is started, it can be http://localhost:5000/vectoraccessed.
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<title></title>
<link
href="https://cdn.bootcdn.net/ajax/libs/mapbox-gl/1.13.0/mapbox-gl.min.css"
rel="stylesheet"
/>
</head>
<script src="https://cdn.bootcdn.net/ajax/libs/mapbox-gl/1.13.0/mapbox-gl.min.js"></script>
<script src="https://cdn.bootcdn.net/ajax/libs/axios/0.21.0/axios.min.js"></script>
<body>
<div id="map" style="height: 100vh" />
<script>
const sources = {
osm: {
type: "raster",
tiles: ["https://b.tile.openstreetmap.org/{z}/{x}/{y}.png"],
tileSize: 256,
},
};
const layers = [
{
id: "base_map",
type: "raster",
source: "osm",
layout: { visibility: "visible" },
},
];
const map = new mapboxgl.Map({
container: "map",
style: { version: 8, layers, sources },
});
map.on("load", async () => {
map.resize();
// 添加栅格瓦片数据源
map.addSource("png_source", {
type: "raster",
minzoom: 1,
tiles: [`${window.location.href}/png/{z}/{x}/{y}`],
tileSize: 512,
});
// Add raster tile layer
map.addLayer({
id: "png_layer",
type: "raster",
layout: {visibility: "visible" },
source: "png_source",
} );
// Add vector tile data source
map.addSource("mvt_source", {
type: "vector",
minzoom: 1,
tiles: [`${window.location.href}/mvt/{z}/{x }/{y}`],
tileSize: 512,
});
// Add a vector tile layer and add a style to the vector tile
map.addLayer({
id: "mvt_layer",
paint: {
"circle-radius": 4,
"circle-color": "#6699CC",
"circle-stroke-width": 2,
"circle-opacity": 0.8,
"circle-stroke-color": "#ffffff",
"circle-stroke-opacity": 0.9,
},
type: "circle",
source: "mvt_source",
"source-layer": "points_geom",
});
});
</script>
</body>
</html>
Note: *Swipe left and right to read
4.5 Dynamic effects of vector tiles
Different effects can be adjusted at the front end. After adjusting to the new layer parameters, the effect is as follows:
{
"circle-radius": 4,
"circle-color": "#000000",
"circle-stroke-width": 2,
"circle-opacity": 0.3,
"circle-stroke-color": "#003399",
"circle-stroke-opacity": 0.9,
}
4.6 Dynamic effects of grid tiles
05 Integration with PGADmin
The PG database management tool PGAdmin natively supports the visualization of vector data, but due to the lack of quick display technology, it can only display a single object or display a limited result set, and cannot perform a smooth and global view of large-scale vector data. We integrate Ganos's vector quick display function with PGAdmin. When data is stored in the database, you can browse the global situation online, quickly evaluate the data overview, and greatly enhance the experience of data management.
06 Summary
This article starts with the principles and advantages of the sparse vector pyramid, introduces how to use Ganos to implement various functions of online visualization services, and finally realizes a map visualization service that can handle hundreds of millions of data through a hundred lines of code. Readers can further use PG/PolarDB Ganos' server-side fast query and analysis capabilities on the basis of visualization to perform more complex functions such as object attribute query, spatial circle selection, and spatial analysis. This is the transformation brought by the large-scale spatial graphics display acceleration black technology brought by Ganos-the sparse vector pyramid index.
This article is the original content of Alibaba Cloud and may not be reproduced without permission.