My road to open source GIS solutions

It’s been a long time since I’ve updated it, because I’m — holding — big — moving —, yes, it’s today’s article.

Today I will share with you my experience of open source GIS solutions.

--Uh-- Considering the single-chat technical solution, you may fall asleep quickly, so today I will focus on my packaging map API, taking the experience of packaging the map API as a clue, interspersed with some of the things used at the time To the open source GIS architecture.

The article is a bit long. If you just want to learn about the latest open source GIS architecture, you can skip the front and go to the fifth edition and the final summary, but I suggest you start with the first edition because there is no previous 4 There will not be a fifth edition in this version. Just looking at the summary will have the same effect as reading the famous aphorisms. When you read it, you think it makes sense, but you forget it afterwards because you can’t empathize.

origin

I have been working in a company in the traditional IT industry. The company is mainly doing government projects, commonly known as toG business.

In this kind of company, there are usually two applications of GIS. One is GIS combined with the company's business to form a xx geographic information system, or xx a map system, and the other is to combine the company's business to encapsulate a set of map APIs. , Provide map technical support for other business systems of the company, similar to AutoNavi Map API and Baidu Map API. There is usually a set of GIS solutions behind the map API as support

Today we will talk about encapsulating the map API.

From the birth of the first map API version I encapsulated to the present, 7 years have passed, and 3 companies have been through it, and 5 versions have been iterated. Among the five versions, the first version uses ArcGIS, and the next four versions mainly use open source GIS.

First edition 2014

background

The company is engaged in environmental protection business, the business system is mainly developed with C#, and the part of the business system that involves maps is developed with ArcGIS Flex API, and GIS developers are responsible. At that time, maps were mainly developed with Flex, because Flex felt very cool at that time.

With the increase of the project, I found that many business functions are common in the project. Later I discovered that if the business function and the map function are separated, and then the map function written by Flex is encapsulated in a set of JS-oriented interfaces, C# developers can Complete the map-related work by yourself, without being restricted by the Flex language.

So we are going to encapsulate a set of JS map API written in Felx for C# developers.

Technology Architecture

The front desk uses ArcGIS Flex API to develop map functions, Flex supports interaction with JS, and uses this feature to encapsulate the map functions developed by Flex into a JS-oriented interface.
The map background uses ArcGIS Server, and the spatial analysis uses GP service.
The spatial database uses Arc SDE.
A simple help website was developed to provide examples and instructions for calling the front-end JS interface.

Summary of experience

After the release of the Map API, I found an interesting phenomenon in the process of doing technical support. Regarding the use of the Map API, junior C# developers who do not understand GIS at all find it easy to use. The reason is that it can help them solve problems. Find our technical support.

Intermediate C# developers who know a little about GIS find it difficult to use. They will compare our map API with ArcGIS JS API and think the latter is more useful. However, due to the ugly map of ArcGIS JS API, we do not provide technical support. They need to study on their own, and ultimately choose to use our map API.

Senior C# developers who know a little about GIS basically don't use it. Among them, two colleagues' reactions impressed me. One colleague said: "You don't use what you develop yourself." The implication is that things you don't use yourself will not work well. But our idea is that Flex encapsulates a set of JS map interface because Flex has a threshold for entry. Since we GIS developers know Flex, and the geographic information system we develop is all developed with Flex, it must be It is more flexible to use ArcGIS Flex API directly, so it is not necessary.

Another colleague was even better. He went to study Flex directly, and asked us if he didn't. After getting started, he directly encapsulated a set of map interfaces for his own use. We have studied the interface that he encapsulated. Although the function is simpler, the starting point when defining the interface is obviously different from ours. We encapsulate from the perspective of function and try our best to ensure that the reusability of the interface is high, such as adding points, adding Line, buffer and other functions. He encapsulates from the user's point of view, such as finding a bunch of data from the database, throwing this bunch of data directly to the interface, and displaying it on the map. In general, his interface is more encapsulated and closer to his actual usage habits, and our interface is more like a set of JS translated from the ArcGIS Flex interface.

I also noticed a phenomenon. Business developers who use our interface often come over and ask, why is my map not displayed? After more experience, I found that there are usually two reasons. One is that the address of the map service is incorrect. At that time, the map service was published using ArcGIS server. The rest address of the ArcGIS server map service was a web page. After the web page was opened, there were many For the link address of the first level, the business staff do not know which level of address should be copied, and often copy the wrong address, so the map cannot be displayed. The other is that when multiple maps are added, the coordinates between the maps are inconsistent, causing the added map to not be displayed.

Think about it, too, the knowledge of map services and map coordinates is still very difficult for developers who do not understand GIS.

Although the first edition had such problems, it still improved the overall work efficiency of the department at that time.

Second edition 2016

background

I have changed companies. The new company is engaged in network security business and has massive positioning data. The role of GIS is to extract, analyze, and display positioning data to help customers solve business problems.

All the company's systems must be deployed on the customer's intranet. The customer's intranet cannot access the Internet, and the map uses Internet maps. This requires downloading the Internet map tiles and copying them to the customer's intranet for publication.

The company has a GIS application system, and business functions that are strongly related to GIS can be placed here. In addition, when other departments have map functions, they will also find our GIS department. This scene is very similar to the previous company.

So, apart from anything else, the second version of the map API started.

Technology Architecture

This version began to use open source GIS. The GIS front desk chose openlayers. With the experience of the first edition, this edition focused on solving the problem of map resources and map coordinates.
1. Solve the problem of map resources. Unified management of basemap resources, providing a variety of Internet maps, including AutoNavi Maps, Google Maps, Sky Maps, etc. Each map has an id. When the map is initialized, different basemaps are used according to the id.
2. Solve the problem of map coordinates. The wgs84 coordinates are uniformly used externally, and the map API is responsible for converting the wgs84 coordinates into the coordinates that match the base map, including gcj02 coordinates, bd09 coordinates, etc.
In the background of GIS, because the positioning data is stored in the database of the big data framework, only the Internet offline map tiles need to be published in the background, so the tomcat is used to publish the tiles, and then a function of loading local tiles is written in openlayers. Even the GIS background.
Without using the spatial database, the spatial analysis uses ArcGIS Engine to develop the console program, and then uses the Java background to call the console program.
The interface has written detailed description documents and calling examples. Without packaging, it is directly a word document + html sample file.

Summary of experience

Solving the problem of map resources and coordinates can greatly improve user experience.
Regarding the map downloader, although everyone has the ability to write one by themselves, it is really not cost-effective. The best solution is to spend the company's money to buy a license, which is not a few dollars from the company's point of view.
The ArcGIS Engine version, the C# version is relatively stable, and the Java version is super difficult to use and very unstable. It will be shown to you at every turn. Don't try to deploy it to a Linux server. Don't ask how I knew it. In order to deploy on Linux, I first developed a version of Java. After deployment, it crashed several times a day, and the memory overflowed at all times. It was useless at all. There was no way I could only rewrite a version of C# and deploy it on the windows server.

Third edition 2017

background

The company has changed. The new company is engaged in pipe network business. Compared with the previous two, the business and GIS are more relevant. The business needs to edit, store, publish and apply pipe network GIS data. The company's previous maps were developed using ArcGIS, and I was preparing to switch to open source GIS, so after I arrived at the company, I started the development of the third version of the map API.

Before development, I carefully studied AutoNavi and Baidu map API, and asked myself two questions:

1. Why can non-GIS developers use AutoNavi and Baidu map API to solve problems, but can't use ArcGIS, openlayers, leaflet?

2. What problems did non-GIS developers encounter when using the Internet Map API?

The following is my own understanding,

The first problem is because: 1. Non-GIS developers don't need to publish map data by themselves. The maps are all officially provided, so they just need to be useful. 2. The help documents and examples of the Internet Map API are all in Chinese.

The second question, I found after trying to use it myself: 1. The offline use of Internet Map API is a problem. They can only be used online. If you can’t access the Internet, you can’t use it. This problem is in the toG business. It's quite common. 2. The Internet Map API can only use the official map coordinates, and cannot integrate map resources with other coordinates.

From this version, we began to try to learn the advantages of the other party and avoid the other party's problems in the user experience of the Gaode and Baidu map API.

Technology selection

The GIS front desk did not continue to use openlayers, but switched to a lighter leaflet. The considerations at the time were:

The leaflet is small, the core code structure is simple and easy to understand, and it has strong plasticity. It is suitable for transforming into its own API.
Leaflet can be compatible with web and mobile at the same time.
The leaflet plug-in maintained by esri is more compatible with the ArcGIS map service previously released by the company.

Geoserver is selected in the GIS background. Because geoserver has richer data, it can support postGIS and SDE spatial database at the same time.

PostGIS is selected as the GIS spatial database, and the spatial analysis is mainly realized by using the spatial analysis function of postGIS.

Desktop-side development continues to use ArcGIS Engine instead of QGIS. The main reason is that the company has accumulated a lot of technology on ArcGIS Engine before and has formed a mature product, and the conversion cost will be high.

Technology Architecture

A layer of its own interface is encapsulated on the basis of leaflet, and the original leaflet interface is not external. The considerations at the time were:

Like the previous version, it is easy to solve the problem of Internet map offset after packaging. The wgs84 coordinates are used uniformly externally, and the data is converted into corresponding coordinates based on different base maps internally.
It can realize a flat interface similar to JQuery, which is simple and easy to use.
The middle point of leaflet is written as [latitude, longitude], and we are usually more accustomed to using [longitude, latitude], which can be solved by encapsulation by the way.

When complex GIS spatial analysis is required, write extension plug-ins of geoserver, which connect to PostGIS database, and use PostGIS spatial analysis functions to write functions to complete spatial analysis.

Realize the automatic publishing of map services based on the rest interface of geoserver. In the desktop software developed by ArcGIS Engine, the GIS data is first imported into the spatial database, and then the rest interface of geoserver is called to publish the map. The final result is the same as the experience of ArcGIS publishing the map.

Build a portal website, the content includes help documents, map resources, update instructions, etc. The help documents take the interface as a clue, and there are interface instructions and call examples inside the interface. Map resources provide all available maps, including Internet maps and self-published business maps. Each map has an id. According to the id, the map can be easily loaded in the API. You don't need to care about how the map service is published.

Modify all PostGIS, geoserver, and tomcat to the green version to facilitate project deployment.

Summary of experience

You can use SLDEditor software to solve the problem that geoserver is not easy to match pictures. The map of geoserver is not easy to use. I tried to map it on QGIS, and then published it to geoserver. I found that the sld style file format generated after the map on QGIS is not supported by many geoservers, and I don’t know if the version does not correspond to it. There are still other reasons. Finally, I found an open source tool SLDEditor to edit style files, which perfectly solved the problem, but the overall experience is much worse than ArcGIS.
The spatial analysis function of PostGIS is very useful and powerful. Therefore, we mainly use PostGIS to realize the spatial analysis function, such as the graphic buffer function shared before .
The geoserver extension is more suitable for developing and generating map service-related functions. The spatial analysis function of GIS was realized by the geoserver extension plug-in at first. Later, it was discovered that the main function of the extension plug-in's background program was data transmission. The most important analysis link was realized in the spatial database PostGIS. However, the extended development environment of geoserver is more complicated. The java backend written by myself is easy to maintain. The advantage of geoserver extension is that it can directly call the resources and functions of geoserver, which is more suitable for developing and generating map service-related functions.

Fourth edition 2019

background

After the development of the third version, I used it in the department for more than a year, and the overall response was good. Some of my colleagues who understand GIS, used ArcGIS JS API before, and after using the previous version of map API encapsulated by leaflet, they were the first The response is that this is so portable, much smaller than ArcGIS JS API, and it loads quickly.

After the last version was released, we paid attention to the user's usage habits. Everyone's usage habits are basically to look at the examples first. After finding the examples, they will directly copy the code and use them. When the examples do not fully meet the requirements, go to the API. Explain that the best case is that the sample code is well-commented, can be understood at a glance, and can be used by copying it.

Of course, some problems were slowly discovered in the course of using it.

The strongest feedback from colleagues who know GIS is whether they can let go of the native interface. A colleague would find leaflet code on the Internet every time, confirm that it can be implemented, then come to us, let us add this function, and even send the information link, we are very embarrassed. If he can let go of the leaflet native interface, he can solve a lot of work on his own.

Then it was myself. When I needed to study a new feature, my first reaction was not to use my own encapsulated map API, but rather to use the native leaflet to write it, because I felt that the map API encapsulated by myself could be used. Not flexible enough.

Oh my god! ! !

This is not the same as the first edition, when my colleagues said: "You don’t use anything you develop yourself." It’s the same. If there are Flex language factors in the first edition, then this third edition will start from the inside. It's all written in JS outside, there is nothing to explain, just let people say it.

In our usual work, in addition to encapsulating the map API, we still have other work to do. In the previous version, we felt that a large part of our energy was spent on encapsulating basic functions, resulting in no time to study advanced map functions. If we can let go of the original interface, the basic functions can directly use the original interface, and we have more time to study advanced map functions.

Can you let go of the native interface?

There are several problems to let go of the native interface:

The map coordinate offset problem.
Previously, through encapsulation, a coordinate adaptation layer was built between the external interface and the map to solve the problem of coordinate offset. If you let go of the native interface, there is no way to use this method anymore, and you need to think of other methods.
User habits.
Will users who don’t understand GIS get used to the flat interface and feel that the native interface is not easy to understand after letting go? The point in the leaflet is written as [latitude, longitude], and the usual [longitude, latitude] is different. Will anyone be unable to adapt?
Version forward compatibility issue.
In the previous version, in order to pursue the simplicity of the interface, many data format types were simplified. If the native interface is released, compatibility with these formats will result in a lot of work, and compatibility should be considered for every additional function in the future. The problem with this type of data.

solution:

Aiming at the problem of coordinate offset.

There are two solutions. One is to provide users with a coordinate conversion interface, and users can convert the coordinates themselves, but this is not user-friendly. The second is to correct the deviation of the Internet map tiles, so that the map can be unified and no longer offset. This is the most ideal, but there are technical difficulties. However, we finally overcome the technical difficulties and adopted the second solution. For details, see: How to elegantly solve the offset problem of Baidu and Gaode maps in leaflet .
Aiming at user habits.

Why don’t Baidu and AutoNavi’s map APIs use flat interfaces, and people don’t find them difficult to use? After our research, we came to the conclusion that if the interface is not particularly complicated, if the Map API can do: it can solve user problems, there are few bugs, rich examples, and clear documentation, everyone will find it easy to use. Whether the interface is flat is not really important. Moreover, the leaflet's native interface itself is already very concise, and there is no need to encapsulate it from the perspective of simplicity.
For the issue of version forward compatibility.
We decided not to be compatible with the previous version, let the two versions remain at the same time, and slowly transition. We recommend that you use this version for new projects and new products. We will continue to provide technical support for old projects, but no more functional upgrades. In this way, after one or two years, you can slowly switch over. Facts have proved that this decision is correct. Now that more than two years have passed, most of the systems in the department have been switched to new versions.

Technology Architecture

The technical architecture is adjusted as follows on the basis of the previous version:

Let go of the leaflet native interface, no longer encapsulate the interface, and expand the function in the form of a plug-in instead.
Integrate a variety of Internet map resources, solve their coordinate offset problem by correcting the deviation of tiles, and use wgs84 coordinates externally.
The help document is changed from interface as clue to example as clue. The comments in the examples are guaranteed to be complete and clear, and the methods used in the example code are given class reference links.
Translate the leaflet class reference documentation and put it on the platform.
When there are new functional requirements, simple functions are not encapsulated, and examples are given directly, and complex functions are then considered to be encapsulated in plug-ins.
Map analysis functions that are not closely related to geoserver are migrated to the java background, and only functions related to cartography are retained in geoserver.

Summary of experience

The translation of leaflet references was not done well, and a few pages were not translated in total, but the strange thing is that everyone has never complained about this problem. Later, it was observed that users basically did not read the documents, but more of them looked at the examples. If the examples did not, they would directly ask us that the document is actually only us looking at it (covering the face).
When someone asks a question, try to record it as an example. If you can find this question in the example, you will not ask it again.
Try to use tools as much as possible to make the maintenance of the platform simple. If it is too complicated, you don't like to maintain it. In the end, the platform is easy to be scrapped.

Fifth edition 2021

background

This edition has not yet been completed. The technical pre-research work was just completed a year ago. Here is a brief summary of the overall idea to share with you.

In general, the previous version has been very useful, and now there is very little negative feedback from users. The product has also been used as a technical middle station in large projects and provided to other companies for use, and the effect is also very good.

But we still have our own pursuit. Compared with AutoNavi and Baidu map API, we still need to improve the following points:

The beauty of the map.

The base map of the map currently uses Internet map tiles. After overlaying the business data, the annotations in the base map will be obscured. It is not easy to automatically avoid the annotations between the business data. In addition, there is no useful map configuration. The map tool causes the map to appear messy and ugly when displaying diverse data.
Show performance issues.

Maps sometimes need some animation effects, such as the use of animation effects to express the direction and speed of the water in the pipe network. When the amount of data is large, there will be obvious freezes.
Problem with map layout.

The geoserver map is not easy to use. This problem has been mentioned before. Although the color matching file can be generated by using SLDEditor, only one layer can be generated at a time, and there is no way to preview it as a whole. The efficiency is too low and the experience is too poor. The custom map matching tools of AutoNavi and Baidu Maps are very easy to use, and the artists can directly use them, and they have been greedy for a long time.

The goal of this edition is to solve the above three problems.

Technology selection

The front desk uses mapboxgl instead of leaflet for two reasons:

performance. The upper limit of leaflet is about 100,000 (see: How leaflet loads 100,000 data ), and mapboxgl is developed based on webgl technology. The maximum amount of data depends on the performance of the graphics card and the network transmission speed. Ideally, it can easily reach the level of one million.
Aesthetics. mapboxgl has a particularly good support for vector tiles, and combined with maputnik can easily realize the custom map function of Gaode and Baidu maps.

Map layout uses maputnik, and business data uses geoserver to publish vector tiles. Normally, maputnik does not support vector tiles published by geoserver. However, we have solved this problem. For details, please see: How to make vector tile matching artifact maputnik support geoserver

There are two options for basemap data:

Continue to use Internet map grid tiles, which is suitable for situations that require high basemap data accuracy.
Publish OSM vector tile maps locally. Currently, there are no free vector tile resources that can be used directly on the Internet. You can only download the OSM data yourself and publish it locally. The data quality of OSM maps in China is relatively poor. If your business does not require high accuracy of the base map data, and requires relatively high styles, such as a large-screen display system, you can choose this solution. For details, please refer to: How to realize the local release of OSM map and customize the map

The map visualization effect is realized by deck.gl and L7.

Summary of experience

Use maputnik to load the business layer published by geoserver and the OSM base map published locally at the same time, which can realize the deep integration of business data and the base map. For example, put the river in the business data under the road layer of the base map and realize the automatic labeling. The avoidance function, an experience like this is still very cool.
The compliance of the OSM map is in doubt. It is recommended to calibrate China's national borders and use them again.

Future prospects

Solve the problem of small amount of OSM map data.

One imperfect part of the fifth version of the solution is that the amount of data in the OSM map is relatively small in China. This is also the cause of the automatic extraction of building outlines through machine learning in my Flag at the beginning of the year.
Research 3D GIS.

Previous work has always been two-dimensional GIS. Three-dimensional GIS has relatively few services. I have also studied cesium, ArcGIS js API 4.x, three.js, etc., and have used them in the project, but the overall understanding of three-dimensional GIS is still not It is as transparent as a two-dimensional GIS, and cannot be confidently given an open source solution that you are satisfied with like a two-dimensional GIS, so you need to continue to work hard here.

to sum up

About the Map API

In toG companies, if you want to use open source GIS to create a set of APIs that are comparable to Gaode and Baidu Maps in terms of ease of use, you need to pay attention to the following points:

Solve the problem of map resources. Organize the online map resources and organize the business map resources on the project so that users can directly use them externally.
Solve the problem of map coordinate system. To solve the problem of the offset of the Internet map and the conversion between multiple map coordinates, users are allowed to use only one type of coordinate externally.
The basic map function is realized through user education, which is to provide complete calling examples and documentation. Advanced map functions are implemented by encapsulating plug-ins, which can avoid the core API becoming more and more redundant over time.
Calling examples that are rich in scenes and can be used directly are more important than whether the interface is brief or whether the document is detailed.
The image factors that users feel that the Map API is easy to use are sorted from high to low: whether the problem can be solved, whether there are bugs, whether there are rich examples, whether technical support is in place, and whether the documentation is clear.
Do unto others, do not impose on others. Find a real business scenario to use your own API, and continue to find problems, solve problems, and improve functions from it, until you feel it is very useful, so you can force yourself to think about the problem from the user's perspective. If you are unwilling to use it, then it is definitely not easy to use, and it will not go far in the end.
Technical support must be done well. The development of map API requires a long-term and continuous iterative process. In this process, it is inevitable that there will be problems of this kind. If your user support is good, it can help you make up for the bad experience that those problems bring to users.
Learn to improve the efficiency of platform maintenance through tools. Think more about which links in the process of platform maintenance can be completed by automated or semi-automatic methods, such as document generation, update deployment, etc., to save time and to study deeper things.

About open source GIS solutions

The following are the two combination schemes I recommend. In fact, they are all mentioned above. Here is a summary.

Lightweight version: leaflet+geoserver+postGIS

This combination has a lot of information on the Internet, the software is easy to use, and has strong universality, which can meet the needs of most people for two-dimensional GIS.

Vector tile version: mapboxgl+maputnik+geoserver+postGIS+openmaptiles

This combination can build a set of custom maps similar to AutoNavi and Baidu maps, and can also implement white-mode three-dimensional maps. If you are more concerned about the visualization of the map, then you are recommended to use this set.

Okay, let's stop here. If you think it is helpful to you, you can support us through continuous attention and sharing.

Original address: http://gisarmory.xyz/blog/index.html?blog=GISerSolution

Pay attention to " GIS Weapons Library " and get more high-quality GIS articles in the first time.

This article uses Creative Commons Attribution - NonCommercial - ShareAlike 4.0 International License Agreement for licensing. Welcome to reprint, use, and republish, but be sure to keep the article's signature "GIS Weapon Library" (including link: http://gisarmory.xyz/blog/ ), and it must not be used for commercial purposes. The modified works based on this article must be the same License issuance.