For students who have graduated from GIS majors, they must not be too unfamiliar with ArcGIS software, and they also have a certain understanding and experience in using geographic databases. However, regardless of the software operation level, as a WebGIS/GIS developer, how can we complete the automatic parsing and storage operations of geographic databases through the GIS open source framework ? This is the core issue we will discuss in depth next.
Table of contents
Basic system of ArcGIS geographic database
Concepts related to geodatabases
Java backend open source GIS framework
GeoTools Architecture (Architecture)
GDAL（Geospatial Data Abstraction Library）
GDAL: Building Windows development environment
GDAL: Parse GDB based on FileGDB
GDAL parses GDB file geodatabases
GeoTools: GDB data source automatic storage processing
geodatabase
What is a geodatabase?
First introduce what is a geographic database.
For friends who have been fighting in the field of back-end development all year round, they must have a deep understanding of " relational database ". In layman's terms, it is: a computer-based file management system that abstracts entities in the objective world and expresses them in a relational manner. It is stored in the computer in the form of data for software developers to access, update, and even analyze . For example, by abstracting the company's employee groups, we can get a well-known employee information table (tb_employee) , which stores various information such as each employee's employee number, name, entry time, and department. This is also the attribute data that the GIS profession has been talking about all the year round .
We define " geographic database " by analogy. In layman's terms , it is: a computer-based file management system that abstracts geographical entities in the objective world and stores them in the computer in the form of a relational table, which is specially used for GIS. Developers perform operations such as access, update, and analysis . Note: We use the term geographic entity here. Commonly, POI points, a series of parcel areas, administrative division areas, water system ranges, etc., any objective entity that can be expressed by the basic geometric elements of points, lines, and surfaces , we can call it a geographic entity and store it in a geodatabase.
The role of geodatabases
So the question is, what is the significance of geographic databases in the field of GIS development? How to understand the application value of geographic database in practice?
We use the official documentation of ArcGIS products as a benchmark to explain the multi-level meaning of geographic databases in different aspects.
① Data structure : As the original data structure of ArcGIS, geographic database is the main data format used for editing and data management, and undertakes the storage carrier function of geographic data;
②Data storage : As the storage carrier of geographic data, geographic databases are still implemented based on database management systems or file systems. Therefore, in actual development, we will find that various databases such as MySQL, Oracle, PostGreSQL, etc., either in the form of self-contained or in the form of plug-ins, have successively provided storage and query for geometric data such as points, lines, and surfaces. , update and other operations support.
③Information model : The information model of the geographic database is very comprehensive, not limited to geometric elements such as points, lines, and planes—commonly known as vector data; it also naturally includes image data—commonly known as raster data (for example: remote sensing image , DEM digital elevation model, etc.), to realize the representation and management of geographic information. At the same time, geographic databases also support the definition of spatial relationships and spatial integrity rules (for example: topological structural integrity of geometric figures) for geographic data.
④ Versatility . Strictly speaking, geodatabase software supports geographic data in a variety of formats, as well as data exchange operations between different formats, including: shapefiles, computer-aided drafting (CAD) files, triangulated irregular networks (TINs), grids, CAD data, imagery, Geography Markup Language (GML) files, and a host of other GIS data sources .
⑤ Workflow (Workflow) aspect . Geodatabase is not only a tool for data storage, data exchange and integration, but also provides a workflow transaction model based on GIS data, which can naturally realize the abstraction, general description, and integration of business rules between some workflows and their various operation steps. automated processing.
Basic system of ArcGIS geographic database
Geodatabase schema
With reference to relational databases, there is nothing special about the implementation of geodatabases, because geodatabases are still implemented based on the same multi-tier application architecture as in other advanced DBMS applications. As we mentioned before, MySQL, Oracle, etc. provide spatial expansion functions. Therefore, from this level, MySQL can also be used as a geographic database-because it not only supports the storage of GIS data, but also provides for A function module for calculating and outputting results.
Let's take a look at an example of using the PostGreSQL database as a geographic database. The following is the storage form of GIS data in the data table. The following table stores the administrative division data of each province in China (the value of the Geometry field is the corresponding geometric data) ,
In order to more clearly see the display effect of the data we store at the geographical level, the following uses QGIS (an open source GIS desktop software) to connect to the PostGreSQL database, and then visualize the data in the data table.
The figure above is an example of geographic data storage based on the PostGreSQL database.
As for how this mechanism is implemented? We can transfer the implementation to the data type of PostGreSQL. By consulting the official website documents , we will find that PostGreSQL supports the Geometry geometric type field, which is used to store GIS data. In other words, as long as a field is set to the Geometry type, the storage of GIS data can be realized, and a PostGreSQL database can be used as a geographic database . The tables used to store geographic data are stored either as files on disk or in a database in a DBMS such as Oracle, IBM DB2, PostgreSQL, IBM Informix, or Microsoft SQL Server . - This is why we said " there is nothing special about the implementation of the geodatabase " in the above section.
Concepts related to geodatabases
(1) Dataset : Geodatabase contains 3 types of datasets: ① feature class; ② raster dataset; ③ table. Usually we usually store it in the form of a table based on the above three forms to realize the construction of the geographic database.
(2) Geodatabase strategy : An important geodatabase strategy is to make full use of the database management system (DBMS) to expand the GIS dataset and the number of users to a very large scale (for example, from a simple and small database that can only support one or a few users) Databases scale to large databases that can support millions of features and thousands of simultaneous users). Tables are the primary form of storage for geographic datasets. SQL is well suited for querying and setting rows in tables, so a geodatabase strategy is one that takes full advantage of these capabilities. Geodatabases support SQL access to feature geometries in the following DBMSs:
(3) Geodatabase elements : No matter what system GIS users use, they will use the following three basic dataset types.
Geodatabase type
A geodatabase is a "container" that holds a collection of datasets. ArcGIS divides it into three types: ArcSDE geographic database, file geographic database, and personal geographic database.
in:
① File geographic database (gdb) : stored in the form of folders in the file system. Each data set is saved in the form of a file, and a single file can be expanded up to 1TB.
②Personal geographic database (mdb) : All data sets are stored in Microsoft Access data files, the maximum size of which is 2G.
③ ArcSDE geographic database : Also known as multi-user geographic database, this type of database is stored in a relational database using Oracle, Microsoft SQL Server, IBM DB2, IBM Informix or PostgreSQL. This geographic data requires ArcSDE and is unlimited in size and number of users.
The following is the comparison result given by the official ArcGIS document,
PS: Based on the above comparison results, in practical applications, it is recommended to use file geodatabases instead of personal geodatabases.
Java backend open source GIS framework
On the basis of the above content, we further discuss how to parse the GDB file geodatabase based on the open source GIS framework.
The following introduces the open source GIS frameworks that most people are familiar with: geotools, gdal, and both of them provide support for the Java development language.
GeoTools
The official website page of geotools can be viewed here .
Geotools is an open source, OGC-compliant Java development library for processing spatial data (geospatial data) . Its basic structure is as follows,
Based on the Geotools framework, the development of web map services (web services), command line tools (command line tools) and desktop software (desktop applications) can be realized.
core features
(1) Define the spatial concept and data structure interface : mainly based on the Java Topology Suite (JTS), the definition of the Goemetry geometric data entity class is realized, and the filtering support for attribute data and spatial data is realized.
(2) Provides a data access interface : supports feature acquisition, transaction support, and thread lock operations.
Data can be obtained from a variety of files and spatial databases; geographic coordinate system support; map projection operations; filtering and analysis operations based on attribute data and spatial data.
(3) Provides renderer support : a stateless, low-memory-consumption renderer suitable for server-side environments, which can realize complex style map display, text labels and color rendering control.
(4) GeoTools plugin
(5) GeoTools extension : Provides graphics and network support (for finding the shortest path), topology verification, web map server client, xml parsing and encoding binding, and color definition/rendering.
Supported Data Source Formats
(1) Raster format
(2) Database jdbc support
(3) Vector format
(4) XML file
GeoTools Architecture (Architecture)
It is necessary to understand some basic GeoTools Library structures, which will help us quickly find jar packages suitable for business scenarios during the development process.
PS: In actual development, the Maven tool can help us analyze the jar packages that need to be imported.
Based on the above architecture diagram, GeoTools can be divided into the following major modules, and the functions of each module are as follows,
GeoTools Plugins (Plugins)
On the basis of the above core modules, GeoTools has launched various plug-ins to meet actual application requirements.
GeoTools Extensions
On the basis of the top-level architecture, GeoTools also provides a set of extension modules, which can realize functions such as drawing, spatial data topology verification, and map services.
Of course, GeoTools also provides support for XML documents, the details can be viewed here .
GeoTools spin-off project
The following is a list of derivative projects of GeoTools, or some products developed based on GeoTools.
For other content, please refer to the GeoTools official website documentation . Or some articles will be published in the future for detailed discussions.
GDAL（Geospatial Data Abstraction Library）
GDAL is an open source raster spatial data transformation library under the X/MIT license agreement . It utilizes an abstract data model to express the various supported file formats. It also has a collection of command-line tools for data transformation and manipulation . GDAL is also a fairly complex open source GIS framework that provides support for the Java development language.
The reason why GDAL is introduced is because in the previous introduction to the GeoTools framework, we did not see it, and in fact it does not support the parsing operation for the ArcGIS file geographic database (GDB). And GDAL is supported. Through its FileGDB driver module, operations such as data source analysis and table information/element information acquisition of GDB file geodatabase can be realized.
Since GDAL is a very large open source project, the following we only introduce the interesting FileGDB driver part and the environment construction of GDAL.
GDAL: Building Windows development environment
(1) Download the installation package.
The installation package can be downloaded from the GDAL official website , and the current version is release-1900-x64-2.4.4-mapserver-7.4.3 . You can also download other installation packages that match the local JDK environment (zip package is enough).
(2) Decompress the zip package
After the download is complete, decompress the zip package locally (the decompression directory is as shown in the figure below),
Among them: gdal.jar in the bin/gdal/java directory is a dependency package to be used in development.
(3) Configure environment variables
Then configure the environment variables. Configure the path of the subfolder under the bin directory to the system environment variable,
(4) Test the GDAL environment
Use the following command to view the version information of GDAL and test whether GDAL is installed successfully.
# 查看GDAl版本信息
gdalinfo --version
# 获取GDAL支持的驱动列表
gdalinfo --formatsIf information similar to the following is displayed, the configuration is successful.
GDAL: Parse GDB based on FileGDB
To use the Java version of GDAL to parse GDB, you need to merge the dynamic link library (dll file) in the GDAL installation directory into the bin directory in the jdk installation directory , and then import the gdal.jar file into the project normally.
Dynamic link library configuration
Copy all the dll files under the bin path of the gdal installation directory to the bin directory of jdk to configure the dynamic link library.
The Java backend project introduces gdal.jar
In the back-end project built based on Maven, to use GDAL, you only need to import the gdal.jar file. There are two ways to import,
(1) Introduced as a file ,
Create a lib folder in the root directory of the project, and put the gdal.jar file in the gdal installation directory,
For example: A project named gdalApps is created here, that is, copy the gdal.jar file to the gdalApps/lib path,
Then you need to configure the dependencies node of the pom.xml file, the configuration content is as follows,
<!--gdal-->
<dependency>
<groupId>org.gdal</groupId>
<artifactId>gdal</artifactId>
<version>1.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/lib/gdal.jar</systemPath>
</dependency>(2) Introduced in the form of pom.xml file configuration ,
You can also import the gdal.jar development dependency by directly configuring the pom.xml file, but please note: it is strongly recommended that the version of the gdal.jar file introduced in this way be consistent with the version of the locally configured GDAL environment.
<dependency>
<groupId>org.gdal</groupId>
<artifactId>gdal</artifactId>
<version>xxx</version>
</dependency>When gdal.jar is displayed under Maven dependencies, the import is successful.
GDAL parses GDB file geodatabases
GDAL parses the GDB file geodatabase and depends on the FileGDB/OpenFileGDB driver. Therefore, the basic idea of using GDAL to parse GDB is generally as follows,
①Call the ogr.GetDriverByName ( GDBDriver ) method to obtain the FileGDB/OpenFileGDB driver instance driver , and use the driver.Open(filePath) method to obtain a GDB data source to obtain a DataSource instance;
② Use the DataSource instance obtained in ① , call its getLayer(layerIndex) method to obtain each layer Layer object contained in the GDB data source , and then use the GetLayerDefn() method of the object to obtain the meta information of the layer, such as: Spatial reference/coordinate system information, attribute table field information, cursor object to obtain the Feature feature class instance inside the layer, etc. (for details, please refer to the sample code).
③Call the GetNextFeature() method through the Layer layer object to analyze the Feature element information in the layer one by one .
PS: When the Java version of GDAL parses the GDB file geodatabase, the classes and their hierarchical structures used are as follows,
The sample code is as follows,
package com.xwd;
import org.gdal.gdal.gdal;
import org.gdal.ogr.*;
import org.gdal.osr.SpatialReference;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.util.ResourceUtils;
import java.io.FileNotFoundException;
import java.util.*;
/**
* @className ReadGDB_Test
* @description: com.xwd
* @auther: xiwd
* @date: 2023-02-13 - 02 - 13 - 21:04
* @version: 1.0
* @jdk: 1.8
*/
@SpringBootTest(classes = GDALSpringBootApp.class)
@RunWith(value = SpringRunner.class)
public class ReadGDB_Test {
//properties
//static properties
static {
gdal.AllRegister();//设置gdal环境
// 为了支持中文路径，请添加下面这句代码
gdal.SetConfigOption("GDAL_FILENAME_IS_UTF8","YES");
// 为了使属性表字段支持中文，请添加下面这句
gdal.SetConfigOption("SHAPE_ENCODING","");
try {
//防止报错
gdal.SetConfigOption("GDAL_DATA", ResourceUtils.getFile("classpath:gdal-data").getPath());
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
//methods
@Test
public void readGDB(){
String GDBPath = "D:\\GIS_workspace\\gdb_data\\test-data\\中国行政区划数据.gdb\\test.gdb";
String FileDriver = "OpenFileGDB";
Driver driver = null;
DataSource dataSource = null;
driver = ogr.GetDriverByName(FileDriver);
dataSource = driver.Open(GDBPath);
if (Objects.isNull(dataSource)) return;
int layerCount = dataSource.GetLayerCount();
for (int i = 0; i < layerCount; i++) {
Layer layer = dataSource.GetLayer(i);
String layerName = layer.GetName();
double[] extent = layer.GetExtent();
System.out.println("正在解析"+layerName+",范围："+ Arrays.toString(extent));
FeatureDefn featureDefn = layer.GetLayerDefn();
//解析Layer属性字段信息
int fieldCount = featureDefn.GetFieldCount();
Map<String,Object> fieldMap = new HashMap<>();
for (int j = 0; j < fieldCount; j++) {
FieldDefn fieldDefn = featureDefn.GetFieldDefn(j);
String fieldName = fieldDefn.GetName();
int fieldType = fieldDefn.GetFieldType();
String typeName = fieldDefn.GetFieldTypeName(fieldType);
fieldMap.put(fieldName,typeName);
}
System.out.println("图层"+layerName+"范围信息："+fieldMap.toString());
//解析Layer属性表
long featureCount = layer.GetFeatureCount();
for (int j = 0; j < featureCount; j++) {
Feature feature = null;
while ((feature = layer.GetNextFeature()) != null){
Set<Map.Entry<String, Object>> entries = fieldMap.entrySet();
for (Map.Entry<String, Object> next : entries) {
String key = next.getKey();
String value = feature.GetFieldAsString(key);
System.out.print(key + ":" + value);
}
System.out.println();
}
}
System.out.println("\n");
}
}
}
GeoTools: GDB data source automatic storage processing
To automatically load the GDB data source into the library, you need the support of the GeoTools-PostGIS plug-in, and you can introduce the corresponding dependencies in the pom. For spatial reference information, EPSG encoding information is required; for result set filtering through the PostGIS plug-in, CQL module is required.
<!--gt-postgis-jdbc数据库PostGIS操作支持-->
<dependency>
<groupId>org.geotools.jdbc</groupId>
<artifactId>gt-jdbc-postgis</artifactId>
<version>${geotools.version}</version>
</dependency>
<!--gt-cql过滤查询支持-->
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-cql</artifactId>
<version>${geotools.version}</version>
</dependency>
<!-- jts -->
<dependency>
<groupId>org.locationtech.jts</groupId>
<artifactId>jts-core</artifactId>
<version>${jts.version}</version>
</dependency>
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-epsg-hsql</artifactId>
<version>${geotools.version}</version>
</dependency>The complete sample code of GDB data source automatic warehousing processing is as follows,
PS: The following is to serialize the GDB data source into a single small file and store it locally, then deserialize it when storing it, and then perform the storing operation.
package com.xwd;
import org.gdal.gdal.gdal;
import org.gdal.ogr.*;
import org.geotools.data.*;
import org.geotools.data.collection.ListFeatureCollection;
import org.geotools.data.simple.SimpleFeatureCollection;
import org.geotools.data.simple.SimpleFeatureIterator;
import org.geotools.data.simple.SimpleFeatureSource;
import org.geotools.feature.FeatureCollection;
import org.geotools.feature.FeatureIterator;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.feature.simple.SimpleFeatureTypeBuilder;
import org.geotools.filter.text.cql2.CQL;
import org.geotools.filter.text.cql2.CQLException;
import org.geotools.geometry.jts.JTSFactoryFinder;
import org.geotools.geometry.jts.WKBReader;
import org.geotools.referencing.CRS;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.GeometryFactory;
import org.locationtech.jts.io.ParseException;
import org.locationtech.jts.io.WKTReader;
import org.opengis.feature.Feature;
import org.opengis.feature.Property;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.filter.Filter;
import org.opengis.referencing.FactoryException;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.util.ResourceUtils;
import java.io.*;
import java.util.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.function.Function;
import java.util.stream.Collectors;
/**
* @className DataStoreDemo
* @description: com.xwd
* @auther: xiwd
* @date: 2023-02-09 - 02 - 09 - 22:34
* @version: 1.0
* @jdk: 1.8
*/
@SpringBootTest(classes = GDALSpringBootApp.class)
@RunWith(value = SpringRunner.class)
class DataStoreToolClass {
//static properties
private static final Map<String, Object> dbParams = new HashMap<>();
private static final GeometryFactory geometryFactory = JTSFactoryFinder.getGeometryFactory(null);
private static final WKTReader wktReader = new WKTReader(geometryFactory);
//properties
private DataStore dataStore;
static {
/**
* PostGIS连接参数
* dbtype数据库类型(固定值)：Must be the string postgis
* host主机名：Machine name or IP address to connect to
* port端口：Port number to connect to, default 5432
* schema空间：The database schema to access
* database数据库名： The database to connect to
* user用户名：User name
* passwd密码：Password
*/
dbParams.put("dbtype", "postgis");
dbParams.put("host", "localhost");
dbParams.put("port", 5432);
dbParams.put("schema", "public");
dbParams.put("database", "gdb_db");
dbParams.put("user", "postgres");
dbParams.put("passwd", "postgres");
}
/**
* 构造器
*/
DataStoreToolClass() throws IOException {
/**
* FactoryFinder-提供多线程环境的安全、默认同步支持
* DataStoreFinder - 用于获取一个DataStore实例
* * access(访问)：shapefile、database(PostGIS/web service[WFS])
* * create(创建)：shapefile,然后借助FileDataStoreFinder对shp文件进行操作
* PS：DataStore数据源是不可重复的，即以单例的形式存在，但是可以复用
*
*/
this.dataStore = DataStoreFinder.getDataStore(dbParams);
}
/**
* 动态创建表结构
*
* @param tableName 数据表名
* @param fieldsMap 字段->字段类型映射表,强制使用LinkedHashMap,保证字段有序性
* @param crs 空间参考系的EPSG编码
* @return SimpleFeatureType表结构对象
*/
public SimpleFeatureType createSimpleFeatureType(String tableName, LinkedHashMap<String, Class> fieldsMap, String crs) throws FactoryException {
if (Objects.isNull(crs) || "".equals(crs)) {
crs = "4326";
}
SimpleFeatureTypeBuilder ftb = new SimpleFeatureTypeBuilder();
ftb.setName(tableName);
ftb.setCRS(CRS.decode("EPSG:" + crs));
Set<Map.Entry<String, Class>> entries = fieldsMap.entrySet();
for (Map.Entry<String, Class> next : entries) {
if ("geometry".equals(next.getKey())) {
ftb.add(next.getKey(), next.getValue());
} else {
ftb.add(next.getKey(), next.getValue());
}
}
return ftb.buildFeatureType();
}
/**
*
* @param schemaName 数据表名称
* @return 布尔值,true-表示数据表已经存在,无需创建;false-表示数据表不存在,需要创建
*/
public boolean hasSchema(String schemaName){
SimpleFeatureType schema = null;
try {
schema = dataStore.getSchema(schemaName);
return Objects.nonNull(schema);
} catch (Exception e) {
e.printStackTrace();
return false;
}
}
/**
* 删除数据表
* @param schemaName 数据表名
*/
public void dropSchema(String schemaName){
try {
this.dataStore.removeSchema(schemaName);
System.out.println(schemaName+"删除表成功");
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* 动态创建数据表
*
* @param schemaName 数据表名
* @param featureType 数据表结构
* @param dropExistsBefore 否先删除已有同名表-再执行添加操作-true-是;false-否
* @return boolean, 布尔值-true:创建成功;false-创建失败
*/
public boolean createSchema(String schemaName, SimpleFeatureType featureType,boolean dropExistsBefore) {
// create a featureType and write it to PostGIS
try {
boolean hasSchema = this.hasSchema(schemaName);
if (hasSchema) {
if (dropExistsBefore){
this.dropSchema(schemaName);
}else{
return true;
}
}
dataStore.createSchema(featureType);
//尝试获取数据表
SimpleFeatureType schema = dataStore.getSchema(schemaName);
if (Objects.nonNull(schema)) return true;
} catch (Exception e) {
e.printStackTrace();
}
return false;
}
/***
* 为数据表添加数据
* @param schemaName 数据表名
* @param dataMap 包含id值和data数据体的Map映射表
* @return int,影响的行数/添加成功的记录条数
*/
public int inertForSchema(String schemaName,LinkedHashMap<String,Object[]> dataMap) {
if (Objects.isNull(schemaName) || "".equals(schemaName)) return 0;
SimpleFeatureType schema = null;
FeatureIterator iterators = null;
FeatureWriter<SimpleFeatureType, SimpleFeature> featureWriter = null;
int affectRows = 0;
try {
schema = this.dataStore.getSchema(schemaName);//从数据源获取数据表对象
SimpleFeatureBuilder build = new SimpleFeatureBuilder(schema);
List<SimpleFeature> featureList = new ArrayList<>();
Set<Map.Entry<String, Object[]>> entries = dataMap.entrySet();
for (Map.Entry<String, Object[]> next : entries) {
SimpleFeature simpleFeature = build.buildFeature(next.getKey(), next.getValue());
featureList.add(simpleFeature);
}
FeatureCollection collection = new ListFeatureCollection(schema, featureList);
featureWriter = this.dataStore.getFeatureWriterAppend(schemaName, Transaction.AUTO_COMMIT);
iterators = collection.features();
while (iterators.hasNext()){
Feature feature = iterators.next();
SimpleFeature simpleFeature = featureWriter.next();
Collection<Property> properties = feature.getProperties();//获取features属性集合
for (Property property : properties) {
simpleFeature.setAttribute(property.getName(), property.getValue());
}
featureWriter.write();
affectRows++;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (Objects.nonNull(iterators))
iterators.close();
if (Objects.nonNull(featureWriter)) {
try {
featureWriter.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
return affectRows;
}
/**
* 数据表查询-[过滤查询]
* @param schemaName 数据表名称
* @param sqlFilter 过滤条件的SQL语句，例如：模糊查询过滤,name like '%a'
* @return Collection<Property> 查询结果集,查询失败则返回null
*/
public Collection<Property> queryFilterForSchema(String schemaName, String sqlFilter) {
if (Objects.isNull(schemaName) || "".equals(schemaName)) return null;
Filter filter = null;
Query query = null;
FeatureReader<SimpleFeatureType, SimpleFeature> featureReader = null;
Collection<Property> properties = null;
try {
filter = CQL.toFilter("name like '%a'");
query = new Query(schemaName,filter);
featureReader = this.dataStore.getFeatureReader(query, Transaction.AUTO_COMMIT);
while (featureReader.hasNext()) {
SimpleFeature feature = featureReader.next();
properties = feature.getProperties();
System.out.println(Arrays.toString(properties.toArray()));
}
return properties;
} catch (Exception e) {
e.printStackTrace();
}finally {
if (Objects.nonNull(featureReader)) {
try {
featureReader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return properties;
}
/**
* 数据表查询操作-[全量查询]
* @param schemaName 数据表名称
* @return Collection<Property> 查询结果集,查询失败则返回null
*/
public Collection<Property> queryAllForSchema(String schemaName){
if (Objects.isNull(schemaName) || "".equals(schemaName)) return null;
SimpleFeatureSource featureSource = null;
SimpleFeatureIterator iterator = null;
Collection<Property> properties = null;
try {
featureSource = this.dataStore.getFeatureSource(schemaName);
SimpleFeatureCollection features = featureSource.getFeatures();
iterator = features.features();
while (iterator.hasNext()){
SimpleFeature simpleFeature = iterator.next();
properties = simpleFeature.getProperties();
// for (Property property : properties) {
// System.out.print(property.getName() + "(" + property.getType().getBinding().getName() + "):" + property.getValue()+"\t");
// }
}
return properties;
} catch (IOException e) {
e.printStackTrace();
return properties;
} finally {
if (Objects.nonNull(iterator))
iterator.close();
}
}
}
class GDBToolClass{
private static final ExecutorService executorService = Executors.newFixedThreadPool(10);//线程池
private static final String GDBDriver = "OpenFileGDB";
private String GDBPath;
private DataSource dataSource;
static {
gdal.AllRegister();//设置gdal环境
// 为了支持中文路径，请添加下面这句代码
gdal.SetConfigOption("GDAL_FILENAME_IS_UTF8", "YES");
// 为了使属性表字段支持中文，请添加下面这句GDBReadTaskExecutor
gdal.SetConfigOption("SHAPE_ENCODING", "");
// gdal.SetConfigOption("GDAL_DATA", "classpath:gdal-data");
gdal.SetConfigOption("OGR_ORGANIZE_POLYGONS", "ONLY_CCW");//开启此配置-加速读取
try {
//防止报错
gdal.SetConfigOption("GDAL_DATA", ResourceUtils.getFile("classpath:gdal-data").getPath());
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
GDBToolClass(String GDBPath){
this.GDBPath = GDBPath;
Driver driver = ogr.GetDriverByName(GDBDriver);
this.dataSource = driver.Open(this.GDBPath);
}
/**
* 获取所有图层
* @return
*/
public List<Layer> getLayers(){
int count = this.dataSource.GetLayerCount();
List<Layer> layers = new ArrayList<>();
for (int i = 0; i < count; i++) {
layers.add(this.dataSource.GetLayer(i));
}
return layers;
}
/**
* 根据字符串类型描述获取对应的Class运行时类类型
* @param name 类型字符串
* @return Class运行时类类型
*/
private Class getClassByStringName(String name){
if ("".equals(name)||Objects.isNull(name)) return String.class;
switch (name){
case "String":{
return String.class;
}
case "Real":{
return Double.class;
}
default:
return String.class;
}
}
/**
* 获取名称为layerName的图层属性表结构
* @param layerName 图层名称
* @return 以LinkedHashMap<String,Class>描述的属性表结构
*/
public LinkedHashMap<String,Class> getLayerSchemaStructure(String layerName){
Layer layer = this.dataSource.GetLayer(layerName);
System.out.println(layer);
FeatureDefn featureDefn = layer.GetLayerDefn();
LinkedHashMap<String,Class> classMap = new LinkedHashMap<>();
for (int i = 0; i < featureDefn.GetFieldCount(); i++) {
FieldDefn fieldDefn = featureDefn.GetFieldDefn(i);
int fieldType = fieldDefn.GetFieldType();
String typeName = fieldDefn.GetFieldTypeName(fieldType);
String name = fieldDefn.GetName();
System.out.println(typeName+"-"+name);
classMap.put(name,getClassByStringName(typeName));
}
classMap.put("geometry",Geometry.class);
return classMap;
}
}
class FileToolClass{
private static final GeometryFactory factory = JTSFactoryFinder.getGeometryFactory(null);
private static final WKBReader wkbReader = new WKBReader(factory);
FileToolClass(){ }
/**
* 从文件中解析List<Map>结构的数据集
* @param filePath 文件路径
* @return List<Map>结构的数据集
*/
public List<Map<String,Object>> readFile(String filePath){
FileInputStream fis = null;
ObjectInputStream ois = null;
try {
fis = new FileInputStream(filePath);
ois = new ObjectInputStream(fis);
List<Map<String,Object>> object = (List<Map<String, Object>>) ois.readObject();
return object;
} catch (Exception e){
e.printStackTrace();
}
return null;
}
/**
* 从文件中解析List<Map>结构的数据集
* @param file 文件路径
* @return List<Map>结构的数据集
*/
public List<Map<String,Object>> readFileByFile(File file){
FileInputStream fis = null;
ObjectInputStream ois = null;
try {
fis = new FileInputStream(file);
ois = new ObjectInputStream(fis);
List<Map<String,Object>> object = (List<Map<String, Object>>) ois.readObject();
return object;
} catch (Exception e){
e.printStackTrace();
}
return null;
}
/**
*
* @param maps readFile读取单个文件返回的结果集
* @param fieldClassMap 原始GDB中获取到的字段-字段类型的映射Map
* @return 转换之后符合要求的结果集
*/
public LinkedHashMap<String,Object[]> trans2ParamMap(List<Map<String,Object>> maps,LinkedHashMap<String, Class> fieldClassMap){
LinkedHashMap<String,Object[]> mapResult = new LinkedHashMap<>();
for (int i = 0; i < maps.size(); i++) {
Map<String, Object> map = maps.get(i);
int size = fieldClassMap.size();
Object[] objects = new Object[size];
int count = 0;
for (Map.Entry<String, Class> next : fieldClassMap.entrySet()) {
if ("geometry".equals(next.getKey())) {
Geometry geometry = null;
try {
geometry = wkbReader.read((byte[]) map.get(next.getKey()));
objects[count] = geometry;
} catch (ParseException e) {
e.printStackTrace();
objects[count] = null;
}
} else {
objects[count] = map.get(next.getKey());
}
count++;
}
mapResult.put(String.valueOf(i+1),objects);
}
return mapResult;
}
}
public class DataStoreDemo {
//static properties
//测试GDB---GDB数据入库处理关键代码
@Test
public void readAllFilesFromDirectory() throws Exception {
String GDBPath = "D:\\GIS_workspace\\gdb_data\\test-data\\中国行政区划数据.gdb\\test.gdb";
String outputDirectory = "D:\\GIS_workspace\\gdb_data\\test-data\\中国行政区划数据.gdb\\output";
File file = new File(outputDirectory);
if (!file.isDirectory()) throw new Exception("文件夹路径不合法");
boolean dropExistsBefore = true;//是否先删除已有同名表-再执行添加操作-true-是;false-否
GDBToolClass toolClass = new GDBToolClass(GDBPath);//GDB数据源解析类
DataStoreToolClass dataStoreToolClass = new DataStoreToolClass();//数据库操作类
FileToolClass fileToolClass = new FileToolClass();//文件解析操作类
List<Layer> layers = toolClass.getLayers();
for (Layer layer : layers) {
String layerName = layer.GetName();//图层名称/数据表名
LinkedHashMap<String, Class> classMap = toolClass.getLayerSchemaStructure(layerName);//获取图层的数据表结构
SimpleFeatureType simpleFeatureType = dataStoreToolClass.createSimpleFeatureType(layerName, classMap, "4326");
boolean flag = dataStoreToolClass.createSchema(layerName, simpleFeatureType,dropExistsBefore);//创建数据表
//判断数据表是否创建成功
if (flag) {
System.out.println("数据表创建成功");
} else {
System.out.println("数据表创建失败/已存在");
}
File[] files = file.listFiles(new FileFilter() {
@Override
public boolean accept(File pathname) {
return pathname.getName().contains(layerName);
}
});
if (Objects.isNull(files) || files.length == 0) {
System.out.println("数据源-[" + layerName + "]解析失败!");
continue;
}
//解析数据源
for (File currentFile : files) {
List<Map<String, Object>> maps = fileToolClass.readFileByFile(currentFile);
LinkedHashMap<String, Object[]> hashMap = fileToolClass.trans2ParamMap(maps, classMap);
//添加数据到数据库中
int rows = dataStoreToolClass.inertForSchema(layerName, hashMap);
System.out.println("成功添加" + rows + "条记录到[" + layerName + "]表中");
}
}
}
}