In this article, we describe a system for automatically adding doors and windows to a 3D city model of Amsterdam (accessible here). Computer vision is used to extract information about the location of doors and windows from panoramic images of cities. Since this type of street-level imagery is widely available, this method can be used for larger geographic areas.
Recommendation: Use NSDT editor to quickly build programmable 3D scenes.
A 3D city model of Amsterdam under development can be used to more easily communicate spatial plans to the public. In addition, the public can become more involved in the city government's planning decisions and purposes. City models consist of buildings in simplified shapes; in CityGML1 terminology, they are at Level of Detail 2 (LOD2). Adding windows and doors to buildings enables many new use cases, including emergency response planning, urban sustainability, and urban simulation (e.g., the impact of "rights to daylight" on the potential construction of new buildings).
Amsterdam 3D City Model
The system proposed in this paper can be divided into the following three steps:
1. Extraction of facade textures from panoramic images
The first step uses building video data to identify, rectify and extract building texture regions from street panorama images. The figure below is an example illustrating the idea of the first stage of the proposed processing pipeline.
(Left) Panoramic image taken from the Hartenstraat in Amsterdam. (Middle) The location of the panorama image and its relationship to building footprint data. (Right) Corrected facade texture.
2. Detect doors and windows from the extracted texture
The second step is to use Mask R-CNN (a deep convolutional neural network) to detect windows and doors from textured areas of the building. We generated over 980 high-quality segmentation mask images to train the network.
(Left) Corrected facade texture. (Middle left) Manually annotated segmentation mask image, ground truth. (Middle right) Bounding boxes, segmentation masks, and corresponding class labels. (Right) Bounding boxes represent detected windows and doors.
3. Add the detected doors and windows to the 3D model
In the third and final step, the previously detected windows and doors are aligned with the input CityGML LOD2 model to construct the CityGML LOD3 model.
The figure above shows a virtual street scene generated in LOD3 using the proposed system. Visualize with the Azul CityGML viewer.
Here is a demo video of the project, and the code can be downloaded from github .
Original Link: Deep Learning Enhanced 3D City Model—BiMAnt