Researchers from the German Remote Sensing Data Center (DFD) of the German Aerospace Center (DLR), the Department of Geography of the Rheinische Friedrich-Wilhelms-University in Bonn and our Earth Observation Research Cluster (EORC) of the Julius-Maximilians-University in Würzburg teamed up on a study investigating the use of multimodal geo-image data and deep learning for modeling building exposure to natural hazards. The paper titled “Bottom-up building exposure modeling with multimodal earth vision” was just published in ISPRS Journal of Photogrammetry and Remote Sensing by Patrick Aravena Pelizari, Christian Geiß and Hannes Taubenböck.
Here is the abstract of the paper: Effective disaster mitigation and management rely on up-to-date exposure models providing detailed and spatially localized information on vulnerability-relevant characteristics of buildings. This study investigates the potential of heterogeneous multimodal geo-image data—incorporating street-level imagery (SLI), very high-resolution optical remote sensing data, and a normalized digital surface model—for generic large-area building characterization. We introduce a deep multimodal multitask learning methodology for the synergistic fusion of multi-sensor data and efficient multi-criteria building classification. The proposed task-wise modality attention (TMA) fusion optimizes multimodal feature representations for individual inference tasks separately according to their specific requirements. To address the challenge of partially missing SLI data (i.e., the missing modality problem), a transformer-based SLI spatial context encoder leverages spatial correlations between structural building attributes and their visual manifestations to make the semantic information from available SLI widely accessible. With the earthquake-prone metropolis Santiago de Chile as test site, the two scenarios—SLI available and SLI missing—are evaluated through a comprehensive experimental cross-comparison of estimated generalization accuracies for classifying buildings according to five target variables: height, lateral load-resisting system material, seismic building structural type, roof shape, and block position. The results underscore the significant potential of the employed modalities and methods. Across the five addressed attributes, covering a total of 35 thematic classes, the most accurate models achieve mean 𝜅 accuracies of 85.19% and 74.96% for data points with and without SLI coverage, respectively. The presented data and methods allow to generate
an area-wide building exposure model with a unique combination of thematic resolution, spatial detail and coverage.
Here is the link to the full paper: https://www.sciencedirect.com/science/article/pii/S0924271625004241
This research is part of our work in the risk domain with respect to natural hazards. A selection of related papers can be found here:
- https://www.sciencedirect.com/science/article/pii/S0924271623002332
- https://link.springer.com/article/10.1007/s11069-022-05672-6
- https://www.sciencedirect.com/science/article/pii/S0924271621001817
- https://link.springer.com/article/10.1007/s10518-019-00648-7
- https://www.sciencedirect.com/science/article/pii/S0924271614002007
In this general research direction, our team has also edited some Special Issues:
- https://nhess.copernicus.org/articles/special_issue1172.html
- https://link.springer.com/collections/ijccjejcbg
- https://link.springer.com/article/10.1007/s11069-017-2742-5
- https://link.springer.com/journal/11069/volumes-and-issues/68-1
