The EnMAP-Box will be present at the FOSSGIS with a demosession:
Visualisierung und Analyse von Satellitenbildern mit der EnMAP-Box (https://pretalx.com/fossgis2023/talk/9WAMJ9/)
Looking forward meeting you there!
UPDATE - Here
is the recording of the presentation:
The next EnMAP-Box workshop will be held online on 21-23 June, 2021.
The workshop aims to demonstrate the current state of the EnMAP-Box by way of presentations, live demos with Q&A and self-paced tutorials.
Registration for up to 250 participants is now open at the workshop website.
A new EnMAP-Box version has been released. It includes product readers for EnMAP L1B, L2C and L2A, PRISMA L2D
and DESIS L2A images, and a first version of Spectral Math in the QGIS Expression Builder.
A list of all improvements can be found in the changelog.
The EnMAP-Box is a free and open source plugin for QGIS.
It is designed to process imaging spectroscopy data and
particularly developed to handle data from the upcoming EnMAP sensor. The two main goals of
the development are to provide
(i) state-of-the-art applications for the processing of high dimensional spectral and temporal remote sensing data and
(ii) a graphical user interface (GUI) that enhances the GIS oriented visualization capabilities in QGIS by applications
for visualization and exploration of multi-band remote sensing data and spectral libraries.
Therefore, the algorithms provided in the EnMAP-Box will be of high value for many other, especially multi- and hyperspectral EO missions.
The EnMAP-Box plugin bridges and combines efficiently all advantages of QGIS (e.g. for visualization, vector data processing),
packages like GDAL (for data IO or working with virtual raster files) and abundant
libraries for Python (e.g. scikit-learn for EO data classification and PyQtGraph for fast and interactive chart drawing).
The plugin consists of a (i) graphical user interface for hyperspectral data visualization and e.g. spectral library management,
(ii) a set of algorithms, and (iii) a high-level application programming interface (EnMAP API).
The EnMAP-Box is developed at Humboldt-Universität zu Berlin and Universität Greifswald under contract by the Helmholtz Centre Potsdam GFZ and is
part of the EnMAP Core Science Team activities. It is funded by the German Aerospace Centre (DLR) - Project Management
Agency, granted by the Federal Ministry of Economic Affairs and Energy (BMWi; grant no. 50EE1923).
Your feedback is more than welcome! In case you encounter any problems with the EnMAP-Box or have
suggestions of any kind for improving it (or this documentation), please let us know!
Newsletter: Subscribe to the EnMAP mailing list to be informed about new EnMAP-Box releases and other EnMAP related news.
Click here for more information.
Jakimow, B., Janz, A., Thiel, F., Okujeni, A., Hostert, P., van der Linden, S., 2023. EnMAP-Box:
Imaging spectroscopy in QGIS. SoftwareX 23, 101507. https://doi.org/10.1016/j.softx.2023.101507
BibTeX:
@article{JAKIMOW2023101507,
title = {EnMAP-Box: Imaging spectroscopy in QGIS},
journal = {SoftwareX},
volume = {23},
pages = {101507},
year = {2023},
issn = {2352-7110},
doi = {https://doi.org/10.1016/j.softx.2023.101507},
url = {https://www.sciencedirect.com/science/article/pii/S2352711023002030},
author = {Benjamin Jakimow and Andreas Janz and Fabian Thiel and Akpona Okujeni and Patrick Hostert and van der Linden, Sebastian}}
This is a list of Frequently Asked Questions about the EnMAP-Box. Feel free to
suggest new entries!
Installation: no module named pip
In case you run into problems during installation because pip is not available in your python environment
(error message C:/.../python3.exe:Nomodulenamedpip or similar), follow these steps (Windows):
Start the OSGeo4W installer from the OSGeo4W Shell by calling
setup
which will open the OSGeo4W Setup dialog.
Now navigate through the first pages of the dialog, by selecting the following settings:
EnSoMAP requires Numba, a JIT compiler that translates python code into fast machine code.
Unfortunately, Numba is not part of a standard QGIS installation. Installing it afterwards can be tricky,
in particular on the standard Windows and macOS QGIS installations.
If you want to use EnSoMAP and numba, we therefore recommend to use a QGIS that was installed with Conda instead.
Install conda and create a conda environment with QGIS
as described here.
The EnMAP-Box is a freely available, platform-independent software designed to process hyperspectral remote sensing data,
and particularly developed to handle data from the EnMAP sensor.
The development of the EnMAP-Box is funded within the EnMAP scientific preparation program under the
DLR Space Administration (mission lead) and GFZ Potsdam (scientific lead) with resources from the
German Federal Ministry for Economic Affairs and Energy.
Everyone is welcome to contribute to the EnMAP-Box
In case you experience issues with the EnMAP-Box, do not hesitate to submit a
ticket to our Issue Tracker. You are also welcome
to post feature requests or pull requests.
It is recommended to check that your issue complies with the
following rules before submitting:
Verify that your issue is not being currently addressed by other
issues
or pull requests.
If you are submitting a bug report, please:
Report a short description how to reproduce it
If an exception is raised, please provide the full traceback.
If necessary, provide a link to the data that causes the bug
If not feasible to include a reproducible snippet, please be specific about
what part of the EnMAP-Box (functions, widget) are involved and the shape of the data
Include your operating system type and version number of QGIS, Qt and the EnMAP-Box.
Please ensure that code snippets and error messages are formatted in appropriate code blocks.
Forking the EnMAP-box repository allows you to modify the EnMAP-Box code as you like and backup your contributions in a
separated GitHub repository. To create a fork,
The software produced for the EnMAP-Box is licensed according to the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License (SPDX short identifier: GPL-3.0), or (if desired) any later version.
See either https://www.gnu.org/licenses/gpl-3.0.en.html or https://opensource.org/licenses/GPL-3.0 for further details of the license.
A copy of this license is part of the EnMAP-Box repository (LICENSE.txt) and delivered with each release of an EnMAP-Box plugin.
Each source code contribution to the central repository should include a reference to the GPL-3 license terms at the beginning of the file:
"""*************************************************************************** <file name> - <short description> ----------------------------------------------------------------------- begin : <month and year of creation> copyright : (C) <year> <creator> email : <main address>*************************************************************************** This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. * This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this software. If not, see <https://www.gnu.org/licenses/>.***************************************************************************"""
Images and other multimedia content from the EnMAP-Box documentation, i.e files within the EnMAP-Box repositories doc/source folder,
are published under the terms of the CC0 license.
The purpose of CLAs are to clearly define the terms under which intellectual property has been contributed to the
EnMap-Box and thereby allow us to defend the project should there be a legal dispute regarding the software at some
future time.
The EnMap-Box Consortium desires that all maintainers and contributors of ideas, code, or documentation to the
EnMAP-Box project complete, sign, and submit an ICLA.
A signed ICLA is required to be on file before an individual is given commit rights to the EnMap-Box repository.
The ICLA form for filling and signing is available here.
The ICLA is not tied to any employer, so it is recommended to use one’s personal information, e.g. for email address in
the contact details, rather than an email address provided by an employer.
For a corporation that has assigned employees to work on the EnMap-Box, a CCLA is available for contributing
intellectual property via the corporation, that may have been assigned as part of an employment agreement.
Note that a CCLA does not remove the need for every developer to sign their own ICLA as an individual, which
covers both contributions which are owned and those that are not owned by the corporation signing the CCLA.
The CCLA legally binds the corporation, so it must be signed by a person with authority to enter into legal
contracts on behalf of the corporation. The CCLA form for filling and signing is available
here.
Documents may be submitted by email and signed by hand or by electronic signature.
The files should be named icla.pdf and icla.pdf.asc for individual agreements;
ccla.pdf and ccla.pdf.asc for corporate agreements. Zip files, other archives, or links to files are not accepted.
The files must be attached to the mail.
When submitting by email, please fill the form with a pdf viewer, then print, sign, scan all pages into a single
pdf file, and attach the pdf file to an email to enmapbox@enmap.org. If possible, send the attachment from the email address
in the document. Please send only one document per email.
If you prefer to sign electronically, please fill the form, save it locally (e.g. icla.pdf), and sign the file by
preparing a detached PGP signature. For example, gpg –armor –detach-sign icla.pdf
The above will create a file icla.pdf.asc. Send both the file (icla.pdf) and signature (icla.pdf.asc) as attachments
in the same email. Please send only one document (file plus signature) per email. Please do not submit your public key. Instead, please upload your public key to pgpkeys.mit.edu.
Contributors who have not signed an ICLA are in a somewhat fuzzy spot. If they make a large contribution,
or many contributions, then the EnMap-Box maintainers will likely ask to submit an ICLA. However, for small fixes,
infrequent and other minimal or sporadic contributions the terms for licensing and intellectual property still must
be clarified.
For this purpose, barriers for contributing are minimized and contributors pinky swear that they’re
submitting their own work or rather certify that they adhere to the requirements of the DCO defined in
version 1.1 or later at https://developercertificate.org/ by signing-off their pull requests or similar ways of
contributing.
The DCO is very Git-centric, and it only relies on commit metadata.
Indeed, signing-off a commit is just about appending a Signed-off-by. For example a commit like:
Even this approach introduces a low barrier for contributions, it is very easy to use whatever email address you want
for a commit, and the sign-off is just text. Since the issue of trust is important the use of GnuPG signatures
in Git commits is recommended additionally, e.g. with:
$ git commit -s -S (makes GnuPG-signed commits, and)
$ git log --show-signature (shows GnuPG signatures in the log history)
$ git merge --verify-signatures branch (ensures that all commits are signed and valid before performing a merge)
Having to use GnuPG for all commits can be a bit daunting.
Please provide some tests that show if your source code works right.
Unit tests should be located in the repositories enmapboxtesting folder.
To run all tests call:
$ set CI=True
$ python -m nose2 -s enmapboxtesting
Note
The environmental variable CI=True is used to inform test routines to not enter the GUI thread.
If unset, some widgets might pop-up and wait for your input to terminate.
The development of the EnMAP-Box is based on the following schedule.
Note that from EnMAP-Box v3.12+, we optimize releases for the currently supported QGIS Long Term Release (LTR),
while also ensuring compatibility to the currently supported QGIS Latest Release (LR).
This glossary gives an overview of how specific terms are used inside the EnMAP-Box.
All the terms that relate to GIS in general should be consistent with the terms given by the QGIS user manual and GUI.
Because the EnMAP-Box integrates into the QGIS GUI, we try to not (as far as possible) contradict or redefine terminology.
All terms that relate to machine learning should be consistent with the definitions given by Scikit-Learn and the
Scikit-Learn glossary, because we wildly crosslink into the Scikit-Learn docs!
A raster layer styled with a Paletted/Unique values renderer.
The renderer defines the band with category values and a list of named and colored categories.
Styles are usually stored as QML sidecar files.
Category values don’t have to be strictly consecutive.
A vector layer styled with a Categorized renderer.
The renderer defines the field storing the category values
(numbers or strings; expressions not yet supported) and a list of named and colored categories.
Styles are usually stored as QML sidecar files.
Note that in case of numerical category values, the values don’t have to be strictly consecutive.
Note that there is currently no equivalent term for a contiguous vector polygon layer. We may introduce it in the future as needed. For now we expect users to rasterize such a vector layer into a raster layer.
A vector layer styled with a Graduated or a Diagrams renderer.
Styles are usually stored as QML sidecar files.
A Graduated renderer specifies a single field used as continuous-valued variable.
Variable name is given by the field name and color is given by the symbol color.
TODO: screenshot for graduated renderer (see issue #1038)
A Diagrams renderer specifies multiple fields used as continuous-valued variables.
Variable names and colors is given by assigned attribute names and colors.
TODO: screenshot for diagrams renderer (see issue #1038)
Note that in the context of GIS, the epithet “geographic” in “geographic feature” is usually skipped.
In the context of EnMAP-Box, and machine learning in general, the term “feature” is used differently.
A color specified by a 6-digit hex-color string,
where each color component is represented by a two digit hexadecimal number,
e.g. red #FF0000, green #00FF00, blue #0000FF, black #000000, white #FFFFFF and grey #808080.
The style of a layer can be defined in the Layer Styling panel and the Styling tab of the Layer Properties dialog.
Some applications and algorithms take advantage of style information, e.g. for extracting category names and colors.
A raster layer interpreted as a binary mask.
All no data (zero, if missing), inf and nan pixel evaluate to false, all other to true.
Note that only the first band used by the renderer is considered.
A binary file ending on .pkl that contains a pickled Python object, usually a dictionary or list container.
Pickle file content can be browsed via the EnMAP-Box Data Sources panel:
A color specified by a triplet of byte values (values between 0 and 255) representing the red, green and blue color components, e.g. red (255, 0, 0), green (0, 255, 0), blue (0, 0, 255), black (0, 0, 0), white (255, 255, 255) and grey (128, 128, 128).
A vector layer with at least one text, JSON or binary field that is designated to a Spectral Profiles
editor. Such Spectral Profile fields can contain profile data.
Additional metadata can be stored like in any other vector layer, e.g. in text and numeric fields.
Using a vector layer with multiple Spectral Profile fields, it possible to link different profiles to
the same geographic feature, e.g. a white reference profile to a field spectrometer profile relating to
the same spatial position.
A single profile is represented by a dictionary of the following values:
y: list of y values, required (e.g. surface reflectance)
The spectral response describes the sensitivity of a sensor to optical radiation of different wavelengths.
In hyperspectral remote sensing,
the spectral response function is often described by a single full-width-at-half-maximum value.
A category of a classifcation layer that is used as a stratification layer.
Conceptually, a stratum can be seen as a binary mask with all pixels inside the stratum evaluating to True and all other pixels evaluating to False.
Raster metadata management is mainly based on the GDAL PAM (Persistent Auxiliary Metadata) model.
Depending on the type of metadata, managing specific metadata item in the GUI or programmatically can differ.
Details are explained in the specific term descriptions below.
In the GUI, most of the metadata items can be inspected in the Layer Properties dialog,
under GDAL Metadata.
Alternatively, metadata can be managed inside a standard text editor, by opening the GDAL PAM *.aux.xml sidecar file.
If the PAM file not already exists, you can create it manually, but usually, it is also created,
when a raster file is opened inside QGIS. Here is an excerpt of the enmap_berlin.bsq.aux.xml PAM file:
For managing metadata programmatically, you can mostly use the GDAL API classes gdal.Datset and gdal.Band,
or the EnMAP-Box API classes enmapboxprocessing.rasterreader.RasterReader
and enmapboxprocessing.rasterreader.RasterWriter.
Warning
If you want to edit metadata in an editor or programmatically,
be sure to first close the associated raster layer inside QGIS.
Otherwise, QGIS will overwrite your changes again.
To examplify the API usage, we assume the following namespace setup throughout the rest of this section:
fromosgeoimportgdalfromenmapboxprocessing.rasterreaderimportRasterReaderfromenmapboxprocessing.rasterwriterimportRasterWriterfromenmapbox.exampledataimportenmap# use enmap_berlin.bsq raster layer as example datasetdataset:gdal.Dataset=gdal.Open(enmap)raster=RasterReader(enmap)# assume we have a newly created gdal.Dataset object in update modenewDataset:gdal.DatasetnewRaster=RasterWriter(newDataset)# for band-wise interactions, we just use the first bandbandNo=1
The bad band multiplier value is indicating whether a band is usable (1) or not (0).
This information is derived from PAM/Band/Default/bbl.
If that is undefined, it is derived by indexing the ENVI bad bands list from PAM/Dataset/ENVI/bbl.
If that is also undefined, it is assumed, that the band is usable (i.e. value=1):
# get>>>dataset.GetRasterBand(bandNo).GetMetadataItem('bbl')# isn't sufficient in this case>>>dataset.GetMetadataItem('bbl','ENVI')# also not sufficient>>>raster.badBandMultiplier(bandNo)# this will correctly resolve the bad band multiplierNoneNone1# set>>>newDataset.GetRasterBand(bandNo).SetMetadataItem('bbl','1')# set for single band>>>newDataset.SetMetadataItem('bbl','{1, ...., 1}','ENVI')# set for all bands at once>>>newRaster.setBadBandMultiplier(1,bandNo)# set for single band
# get>>>dataset.GetRasterBand(bandNo).GetDescription()>>>raster.bandName(bandNo)band8(0.460000Micrometers)band8(0.460000Micrometers)# set>>>newDataset.GetRasterBand(bandNo).SetDescription('my band name')>>>newRaster.setBandName('my band name',bandNo)
This information is derived from PAM/Band/Default/fwhm.
If that is undefined, it is derived by indexing the ENVI fwhm list from PAM/Dataset/ENVI/fwhm:
# get>>>dataset.GetRasterBand(bandNo).GetMetadataItem('fwhm')# isn't sufficient in this case>>>text=dataset.GetMetadataItem('fwhm','ENVI')# this gives just a string with values for all bands>>>text>>>float(text.strip('{}').split(',')[bandNo-1])# extra processing required to unpack the band FWHM>>>raster.badBandMultiplier(bandNo)# in Nanometers (the default)>>>raster.badBandMultiplier(bandNo,'Micrometers')# in user-defined unitsNone{0.005800,0.005800,0.005800,0.005800,0.005800,0.005800,0.005800,0.005800,0.005800,0.005800,0.005900,0.005900,0.006000,0.006000,0.006100,0.006100,0.006200,0.006200,0.006300,0.006400,0.006400,0.006500,0.006600,0.006600,0.006700,0.006800,0.006900,0.006900,0.007000,0.007100,0.007200,0.007300,0.007300,0.007400,0.007500,0.007600,0.007700,0.007800,0.007900,0.007900,0.008000,0.008100,0.008200,0.008300,0.008400,0.008400,0.008500,0.008600,0.008700,0.008700,0.008800,0.008900,0.008900,0.009000,0.009100,0.009100,0.009200,0.009300,0.009300,0.009400,0.009400,0.009500,0.009500,0.009600,0.009600,0.009600,0.009600,0.009700,0.009700,0.009700,0.011800,0.011900,0.012100,0.012200,0.012400,0.012500,0.012700,0.012800,0.012900,0.013100,0.013200,0.013300,0.013400,0.013500,0.013600,0.013700,0.013800,0.013900,0.014000,0.014000,0.014100,0.014100,0.014200,0.014200,0.014300,0.014300,0.014300,0.014400,0.014400,0.014400,0.014400,0.014400,0.014400,0.014400,0.014400,0.014400,0.014400,0.013700,0.013600,0.013600,0.013500,0.013500,0.013400,0.013400,0.013300,0.013200,0.013200,0.013100,0.013100,0.013000,0.012900,0.012900,0.012800,0.012800,0.012700,0.012700,0.012600,0.012500,0.012500,0.012400,0.012400,0.012300,0.010900,0.010800,0.010800,0.010700,0.010700,0.010600,0.010600,0.010500,0.010500,0.010400,0.010400,0.010400,0.010300,0.010300,0.010200,0.010200,0.010100,0.010100,0.010100,0.010000,0.010000,0.009900,0.009900,0.009900,0.009800,0.009800,0.009700,0.009700,0.009700,0.009600,0.009600,0.009600,0.009500,0.009500,0.009400,0.009400,0.009400,0.009300,0.009300,0.009300,0.009200,0.009200,0.009100,0.009100,0.009100}0.00585.80.0058# set>>>newDataset.GetRasterBand(bandNo).SetMetadataItem('fwhm','0.0058')# set FWHM for single band>>>newDataset.GetRasterBand(bandNo).SetMetadataItem('wavelength_units','Micrometers')# also set the units>>>newDataset.SetMetadataItem('fwhm','{0.0058, ..., 0.0091}','ENVI')# set FWHM for all bands at once>>>newDataset.SetMetadataItem('wavelength_units','Micrometers','ENVI')# also set the units>>>newRaster.setFwhm(5.8,bandNo)# set single band FWHM in Nanometers>>>newRaster.setFwhm(0.0058,bandNo,'Micrometers')# set single band FWHM in user-defined units
# get>>>dataset.GetRasterBand(bandNo).GetNoDataValue()>>>raster.noDataValue(bandNo)>>>raster.noDataValue()# if bandNo is skipped, it defaults to the first band-99.0-99.0-99.0# setnewDataset.GetRasterBand(bandNo).SetNoDataValue(-9999)newRaster.setNoDataValue(-9999,bandNo)newRaster.setNoDataValue(-9999)# if bandNo is skipped, the no data value is applied to all bands
The center wavelength value of a band.
Units should be the same as those used for the fwhm and set in the wavelength units item.
This information is derived from PAM/Band/Default/wavelength.
If that is undefined, it is derived by indexing the ENVI wavelength list from PAM/Dataset/ENVI/wavelength:
# get>>>dataset.GetRasterBand(bandNo).GetMetadataItem('wavelength')# this works, because the GDAL ENVI driver assigns those on-the-fly>>>text=dataset.GetMetadataItem('fwhm','ENVI')# this gives just a string with values for all bands>>>text>>>float(text.strip('{}').split(',')[bandNo-1])# extra processing required to unpack the band wavelength>>>raster.wavelength(bandNo)# in Nanometers (the default)>>>raster.wavelength(bandNo,'Micrometers')# in user-defined units0.460000{0.460000,0.465000,0.470000,0.475000,0.479000,0.484000,0.489000,0.494000,0.499000,0.503000,0.508000,0.513000,0.518000,0.523000,0.528000,0.533000,0.538000,0.543000,0.549000,0.554000,0.559000,0.565000,0.570000,0.575000,0.581000,0.587000,0.592000,0.598000,0.604000,0.610000,0.616000,0.622000,0.628000,0.634000,0.640000,0.646000,0.653000,0.659000,0.665000,0.672000,0.679000,0.685000,0.692000,0.699000,0.706000,0.713000,0.720000,0.727000,0.734000,0.741000,0.749000,0.756000,0.763000,0.771000,0.778000,0.786000,0.793000,0.801000,0.809000,0.817000,0.824000,0.832000,0.840000,0.848000,0.856000,0.864000,0.872000,0.880000,0.888000,0.896000,0.915000,0.924000,0.934000,0.944000,0.955000,0.965000,0.975000,0.986000,0.997000,1.007000,1.018000,1.029000,1.040000,1.051000,1.063000,1.074000,1.086000,1.097000,1.109000,1.120000,1.132000,1.144000,1.155000,1.167000,1.179000,1.191000,1.203000,1.215000,1.227000,1.239000,1.251000,1.263000,1.275000,1.287000,1.299000,1.311000,1.323000,1.522000,1.534000,1.545000,1.557000,1.568000,1.579000,1.590000,1.601000,1.612000,1.624000,1.634000,1.645000,1.656000,1.667000,1.678000,1.689000,1.699000,1.710000,1.721000,1.731000,1.742000,1.752000,1.763000,1.773000,1.783000,2.044000,2.053000,2.062000,2.071000,2.080000,2.089000,2.098000,2.107000,2.115000,2.124000,2.133000,2.141000,2.150000,2.159000,2.167000,2.176000,2.184000,2.193000,2.201000,2.210000,2.218000,2.226000,2.234000,2.243000,2.251000,2.259000,2.267000,2.275000,2.283000,2.292000,2.300000,2.308000,2.315000,2.323000,2.331000,2.339000,2.347000,2.355000,2.363000,2.370000,2.378000,2.386000,2.393000,2.401000,2.409000}0.46460.00.46# set>>>newDataset.GetRasterBand(bandNo).SetMetadataItem('wavelength','0.46')# set wavelength for single band>>>newDataset.GetRasterBand(bandNo).SetMetadataItem('wavelength_units','Micrometers')# also set the units>>>newDataset.SetMetadataItem('fwhm','{0.46, ..., 2.409}','ENVI')# set wavelength for all bands at once >>>newDataset.SetMetadataItem('wavelength_units', 'Micrometers', 'ENVI') # also set the units>>>newRaster.setWavelength(460,bandNo)# set single band wavelength in Nanometers>>>newRaster.setWavelength(0.46,bandNo,'Micrometers')# set single band wavelength in user-defined units
The wavelength units of a band. Valid units are Micrometers, um, Nanometers, nm.
This information is derived from PAM/Band/Default/wavelength_units.
If that is undefined, it is derived from PAM/Dataset/ENVI/wavelength_units:
# get>>>dataset.GetRasterBand(bandNo).GetMetadataItem('wavelength_units')# this works, because the GDAL ENVI driver assigns those on-the-fly>>>dataset.GetMetadataItem('wavelength_units','ENVI')>>>raster.wavelengthUnits(bandNo)MicrometersMicrometersMicrometers# set>>>newDataset.GetRasterBand(bandNo).SetMetadataItem('wavelength_units','Micrometers')# set for single band>>>newDataset.SetMetadataItem('wavelength_units','Micrometers','ENVI')# set for the dataset
Note that when using the RasterWriter for setting wavelength or fwhm information,
the wavelength units are also correctly specified at the same time.
EnMAP-Box provides nearly all of it’s machine learning related functionality by using Scikit-Learn in the background.
So we decided to also adopt related terminology and concepts as far as possible,
while still retaining the connection to GIS and remote sensing in the broader context of being a QGIS plugin.
Most of the following definitions are directly taken from the Scikit-Learn glossary as is, and only expanded if necessary.
The training dataset is split into k smaller sets and the following procedure is followed for each of the k “folds”:
a model is trained using k-1 of the folds as training dataset
the resulting model is used to predict the targets of the remaining part of the dataset
The performance can now be calculated from the predictions for the whole training dataset.
This approach can be computationally expensive,
but does not waste too much data (as is the case when fixing an arbitrary validation set),
which is a major advantage in problems where the number of samples is very small.
A dataset is a complete representation of a learning problem, combining feature data X and target data y.
Datasets are often split into sub-datasets.
One common splitting technique is the train-test split,
where a part of the dataset is held out as a so-called training dataset used for fitting the estimator and
another part is held out as a test dataset used for a final evaluation.
When evaluating different settings (i.e. hyperparameters) for an estimator,
yet another part of the dataset can be held out as a so-called validation dataset.
Training proceeds on the training dataset,
best parameters are found by evaluating against the validation dataset,
and final evaluation can be done on the test dataset.
Holding out a validation datase can be avoided by using cross-validation for hyperparameter tuning.
In QGIS and other GIS, the term feature is well defined as a logical element defined by a point,
polyline or polygon inside a vector layer.
In the context of the EnMAP-Box, we refere to it as geographic feature.
In machine learning, a feature is a component in a so-called feature vector,
which is a list of numeric quantities representing a sample in a dataset.
A set of samples with feature data X and associated target data y or Y form a dataset.
Elsewhere features are known as attributes, predictors, regressors, or independent variables.
Estimators assume that features are numeric, finite and not missing.
n_features indicates the number of features in a dataset.
We usually use this term as a noun to indicate a single feature vector.
Elsewhere a sample is called an instance, data point, or observation.
n_samples indicates the number of samples in a dataset,
being the number of rows in a data array X.
Denotes data that is observed at training and prediction time, used as independent variables in learning.
The notation is uppercase to denote that it is ordinarily a matrix.
Denotes data that may be observed at training time as the dependent variable in learning,
but which is unavailable at prediction time, and is usually the target of prediction.
The notation may be uppercase to denote that it is a matrix, representing multi-output targets, for instance;
but usually we use y and sometimes do so even when multiple outputs are assumed.
Install either the current QGIS Long Term Release (LTR) or the current QGIS Latest Release to run the latest EnMAP-Box.
You can get QGIS here.
Additional information on the installation process is provided in the QGIS Documentation.
In case you already have QGIS installed, you can skip this step.
Extract the qgis-macos-pr.dmg and move it to your app folder
Installing or opening QGIS may raise a verification warning. In that case, open your system preferences, go to
Security & Privacy ‣ General and allow QGIS to be open anyway.
Now you can start QGIS.app from your application folder.
The EnMAP-Box plugin requires several python packages (click
here here for full list).
In the following we demonstrate two variants for installing the required packages (you don’t have to do both!):
The EnMAP-Box includes a Package Installer that offers a more or less one-click solution to install required python packages.
It can be accessed from the menu bar via Project ‣ Package Installer.
If you start the EnMAP-Box for the first time, you may see a warning.
The Package Installer can also be opened by clicking Install Missing.
Use the Yes to All button to install all required python packages. Per default, this will install packages using pip into
the user directory (as this does not require admin rights). It is also possible to install/update packages individually via
the context menu inside the package list.
It might be necessary to restart QGIS and the EnMAP-Box.
In case the Package Installer does not work for you or you do not want to use it,
try installing the packages from the command-line. This variant may be more reliable. See OS specific instructions below:
Run the OSGeo4W Shell as administrator, if possible. A convenient way to access the OSGeo4W Shell is from the start menu.
Depending on the installation method you used to install QGIS, the shortcut will be listed under QGIS 3.xx (standalone installer)
or OSGeo4W (network installer)
Hold Ctrl + Shift and click on the OSGeo4W Shell entry (or right-click and choose Run as administrator)
Activate the Python 3 environment by entering:
py3_env
Note
In more recent versions of QGIS the py3_env call might not be necessary or will simply not work. If so, just continue with the following step (4).
Now all packages will be installed automatically. After completion, the shell should show something like this:
If the package installation was successful, you can close the shell. The required packages are installed now and
you can continue to start/install the EnMAP-Box.
Error
In case you run into problems because pip is not available in your python environment
(error message C:/.../python3.exe:Nomodulenamedpip or similar), follow these steps…
Make sure to execute the pip command in the python environment QGIS is using (usually you can find out where
python is located by running importsys;sys.executable in the QGIS Python Console). You then might have
to change the python3 part in the snippet below with the explicit QGIS python path.
This also applies if you installed QGIS using flatpak, in that case install python modules as described here.
Note
You might also consider to create a virtual environment (python3-mvenv), install the packages as described, set
system-site-packages=true in the pyvenv.cfg, activate this virtual environment and start qgis from there.
Open QGIS and go to Plugins ‣ Manage and Install Plugins ‣ All
In the search bar enter enmap or EnMAP-Box3
Now the EnMAP-Box should be listed in the plugin list:
Select it and click Install plugin (or Upgrade in case you update to a new version)
The following dialog might pop up afterwards:
Depending on whether you want to use the GEE Time Series Explorer check or uncheck the checkbox
and confirm with OK
Start the EnMAP-Box via the icon or from the menubar Raster ‣ EnMAP-Box
(Optional): You can download a demo dataset via Project ‣ Load Example Data
Experimental version
It is also possible to install the most recent develop version of the EnMAP-Box. To do so, make sure that the option
Show also experimental plugins is activated in the plugin manager settings. Once activated, there is an additional button
Install Experimental Plugin in the plugin manager.
Warning
As the experimental tag suggests, this version comes with the newest features and developments, but might also be prone to bugs and crashes.
Click on the Versions tab to see all available version.
Click on the desired version and on the next page click
It is recommended to uninstall previous EnMAP-Box versions (delete folder manually, or in QGIS via
Plugins ‣ Manage and Install Plugins ‣ Installed ‣ EnMAP-Box 3 ‣ Uninstall plugin)
Open Plugins ‣ Manage and Install Plugins ‣ Install from ZIP.
Press … and select the downloaded zip file
(enmapboxplugin.3.x.YYYYMMDDTHHMM.QGIS3.zip) and click Install plugin.
Start the EnMAP-Box via the icon or from the menubar Raster ‣ EnMAP-Box.
Install either the current QGIS Long Term Release (LTR) or the current QGIS Latest Release (LR) to run the latest EnMAP-Box.
You can get the QGIS Standalone Installer here.
For beginners, we recommend using the standalone installers.
More advanced QGIS users can use OSGeo4W installer.
In case you already have the current QGIS LTR or LR version installed, you can skip this step.
In case you have an outdated QGIS version, make sure to install a current version.
Open QGIS and the QGIS Python Console (Ctrl + Alt + P). Type the following and confirm with enter:
importsys;sys.executable
It shows the path of the python executable QGIS is using, usually /usr/bin/python3. It should be the same path as
the command whichpython3 executed in the Terminal returns! Close QGIS.
Start QGIS (from the activated environment, see 6.):
qgis
Hint
You can add a shortcut to your applications menu, so you do not have to open a Terminal and type
the above mentioned commands (6. & 8.) everytime you want to start QGIS with the EnMAP-Box environment:
Create the file ~/.local/share/applications/enmapbox.desktop with the following content (if you used another installation path
in the instructions above change accordingly):
It is recommended to use Micromamba, a minimal installer for conda/mamba.
You can get Micromamba here. You may
of course also use conda, just swap micromamba with conda in the instructions below.
In QGIS go to Plugins ‣ Manage and Install Plugins ‣ All
In the search bar enter enmap
Now the EnMAP-Box should be listed in the plugin list:
Select it and click Install plugin (or Upgrade in case you update to a new version)
The following dialog might pop up afterwards:
Depending on whether you want to use the GEE Time Series Explorer check or uncheck the checkbox
and confirm with OK
Start the EnMAP-Box via the icon or from the menubar Raster ‣ EnMAP-Box
(Optional): You can download a demo dataset via Project ‣ Load Example Data
Experimental version
It is also possible to install the most recent develop version of the EnMAP-Box. To do so, make sure that the option
Show also experimental plugins is activated in the plugin manager settings. Once activated, there is an additional button
Install Experimental Plugin in the plugin manager.
Warning
As the experimental tag suggests, this version comes with the newest features and developments, but might also be prone to bugs and crashes.
Once you successfully installed the EnMAP-Box, you can access the plugin via the icon
in the QGIS toolbar or via Raster ‣ EnMAP-Box from the menubar.
Furthermore, the EnMAP-Box Processing Algorithms provider is available in the Processing Toolbox.
The Graphical User Interface (GUI) of the EnMAP-Box on first open
Tip
Have a look at the User Manual for a detailed description of the GUI.
You can load an example dataset into your project by selecting Project ‣ Add Example Data in the menu bar.
On a fresh installation you will be asked to download the dataset, confirm with OK.
The data will be added automatically into a single map view and will be listed in the Data Sources panel as well.
By default the example data is loaded into a single map view. Let’s rearrange those for better visualisation and in order
to get to know the GUI functionalities:
Click the Open a map view button to add a second map view. This view will appear below the
first map view (Map #1).
We want to arrange the windows so that they are next to each other (horizontally): Click and hold on to the blue area of
Map #2 and drag it to the right of Map #1 (see figure below). The translucent blue rectangle
indicates where the map window will be docked once you stop holding the left mouse button.
In the Map #1 list in the Data Views panel, select hires_berlin.bsq and drag the
layer into Map #2 (you can drag them directly into the map view or the respective menu item under Data Views).
In the next step we link both map views, so that zoom and center are synchronized between both:
Click the button or go to View ‣ Set Map Linking and select
Link map scale and center.
Move the map (using or holding the mouse wheel ) and notice how both map views are synchronized now.
Now we want to change the RGB representation of the enmap_berlin.bsq image:
In the Data Views panel click the Open Raster Layer Styling button, which will open
a new panel. Here you can quickly change the renderer (e.g., singleband gray, RGB) and the band(s) visualized. You can
do so manually using the slider or by selecting the buttons with predefined wavelength regions based on Sentinel-2 (e.g.
G = Green, N = Near infrared).
The raster layer needs to have wavelength information for the latter to work!
In the RGB tab, look for Predefined and click on the dropdown menu . You will find several band
combination presets. Select Colour infrared.
Raster Layer Styling panel with selected Color infrared preset
Try out other renderers and band combinations!
Tip
Once you selected/activated the slider (i.e., clicked on it) you can use the arrow keys ←/→ to
switch back and forth between bands!
In this section we will use a processing algorithm from the EnMAP-Box algorithm provider. The EnMAP-Box adds more than
150 Processing Algorithms to the QGIS processing framework. Their scope ranges from general tasks, e.g. file type
conversions or data import to specific applications like machine learning.
In this example we are converting a polygon dataset with information on different landcover types into a
classification raster, i.e., we are going to rasterize the vector dataset.
First of all, make sure the Processing Toolbox window is opened. If not, activate it via
View ‣ Panels ‣ Processing Toolbox
Open the Rasterize categorized vector layer algorithm under EnMAP-Box ‣ Vector conversion
This section is a collection of ‘recipes’ for common tasks you can perform in the EnMAP-Box. For convenience the recipes might use and refer to the
EnMAP-Box test dataset, but of course it is desired that you use your own data sets as well.
Todo
We will try to continuously update this section. If you miss topics, feel free to suggest new entries here!
Clipping a raster to the bounds of a vector layer is fairly simple in the EnMAP-Box, since vector layers can be
interpreted as masks.
Go to the Processing Algorithms and select Masking ‣ Apply Mask to Raster
Under Raster select the raster layer you want to clip, and under Mask select the vector layer.
Tip
If you want to invert the clip, which means that only pixels are included which are NOT within a polygon,
simply activate the Invert Mask option.
Output example: Input raster (left), vector geometry for clipping (middle) and resulting output (right)
Attention
This method will just mask the raster according to the vector geometries, the extent will not be altered,
which means the raster will not be cropped to the extent of the vector layer. You may use the Translate raster layer
algorithm for this.
Several clustering methods are available in the EnMAP-Box. You can find them in the Processing Toolbox under
EnMAP-Box ‣ Clustering. The usual way to apply these methods is to use a Fit …
algorithm first and then apply it to an image with Predict (unsupervised) classification layer.
This recipe demonstrates the basic workflow of applying clusterers
using K-Means clustering (Fit KMeans) and the test dataset.
See also
You can find all the available clustering algorithms here.
Open the test dataset
In the processing toolbox go to EnMAP-Box ‣ Clustering ‣ Fit KMeans
Specify enmap_berlin.bsq under Raster
Under Output Clusterer specify an output file path and click Run
Now open EnMAP-Box ‣ Clustering ‣ Predict (unsupervised) classification layer
Select enmap_berlin.bsq as input Raster
Under Clusterer click … and select the output .pkl file from the Fit KMeans algorithm
Specify an output filepath for the transformed raster under Clustering and click Run
EnMAP true color image (left) and kmeans cluster result with 8 clusters (right)
Tip
8 clusters is the default of the kmeans algorithm here, if you want to change the number of clusters, run the
Fit Kmeans algorithm with a fewer number, by altering the KMeans() function in the Code window to KMeans(n_clusters=4).
This will reduce the amount of clusters to 4.
The EnMAP Box comes with a variety of filtering algorithms. You can find them in the processing
toolbox under EnMAP-Box ‣ Convolution, Morphology and Filtering. In this section we will demonstrate how to apply them,
by showing a spatial filter and a spectral filter.
In spectral filtering a 1D kernel is applied along the z dimension of the raster (bands). Mind that in some cases you will
reduce the overall number of band values as values in the beginning and at the end will be set to nodata (depending on the kernel type and size).
Open the testdataset. In the processing toolbox go to EnMAP-Box ‣ Convolution, Morphology and Filtering ‣ Spectral Convolution Gaussian1DKernel.
Select enmap_berlin.bsq as Raster
In the code window, change stddev=1 to stddev=2
Specify Output Raster, and click Run
Spectral signature of a pixel, original vs filtered with a Gaussian1DKernel
With the Spatial Generic Filter Processing Algorithm you can use the scipy generic_filter
function to write your own spatial filter functions.
In this example we will write a mode filter (also known as majority filter). This kind of filter is often used to
smooth classification results (reduce salt-and-pepper effects in the image). It calculates the mode of all values in a specifiable
neighborhood around a pixel (e.g. 3x3 pixel window) and assigns this value to the pixel.
In the Processing Toolbox go to and open Convolution, Morphology and Filtering ‣ Spatial Generic Filter
Select an input raster under Raster
In the Code text window you can enter python code. Delete the existing content and enter the following
Majority filter applied to classification image: original classification (left), majority filtered 3x3 window (middle), majority filtered 5x5 window (right)
Tip
Also have a look at the scipy.stats.mode documentation. You can change
the window size by altering the size parameter in the generic_filter function.
You could further improve the function above by putting constraints on the definition of majority (for example, only update the original value if
the frequency of the modal value is higher than 50 percent)
Make sure to open the testdatasets for this example
Specify the input and output parameters according to the screenshot below (you can of course alter the names, but make
sure to also adapt them in the script)
Enter this code in the editor on the right side. You do not need to alter Output Grid and Processing for now.
# select the red bandred=enmap[38]# select the nir bandnir=enmap[64]# calculate ndvindvi=(nir-red)/(nir+red)
Click the run button . The result should be listed in the Data Sources panel.
Make sure to open the testdatasets for this example
Select enmap_berlin.bsq under Inputs and name it enmap. Further select landcover_berlin_polygon.shp and name
it mask.
Under Outputs specify output path and file and name it result
Enter this code in the editor
result=enmap# set all cells not covered by mask to nodataresult[:,mask[0]==0]=noDataValue(enmap)# specify nodata valuesetNoDataValue(result,noDataValue(enmap))# copy metadata to result rastersetMetadata(result,metadata(enmap))
Click the run button . The result should be listed in the Data Sources panel.
If wavelength information is defined for each band of a
raster image, the EnMAP-Box can use it to easily select pre-defined band combinations or to
better compare spectral profiles of sensors with different number of bands.
The wavelength information needs to be defined similar to the ENVI metadata standard:
wavelength units defines the metric unit of the wavelength, e.g. nm, mu, cm,..
wavelength defines the band values in wavelength units, e.g. 450 nm.
fwhm defines the full width at half maximum in wavelength units
Wavelength information can be inspected with the Tools -> Metadata Viewer
In the QGIS python shell, wavelength information can be read as followed:
fromenmapbox.gui.utilsimportparseWavelength,parseFWHMsource='path to raster file / QgsRasterLayer / GDAL DataSource'wavelength,wavelength_unit=parseWavelength(source)fwhm=parseFWHM(source)print(wavelength)print(wavelength_unit)print(fwhm)
Internally, the EnMAP-Box reads wavelength information via the GDAL metadata model:
fromosgeoimportgdalpath=r'path to image'dataset:gdal.DataSet=gdal.Open(path)print(dataset.GetMetadataItem('wavelength_units','ENVI'))print(dataset.GetMetadataItem('wavelengths','ENVI'))
Wavelength information can be set in the same way:
fromosgeoimportgdalpath=r'path to image'dataset:gdal.DataSet=gdal.Open(path)print(dataset.SetMetadataItem('wavelength_units','nm','ENVI'))print(dataset.SetMetadataItem('wavelengths','{490, 560, 665}','ENVI'))dataset.FlushCache()deldataset
Please note that this will only work if the image is not opened in QGIS!.
Several transformation methods are available in the EnMAP-Box. You can find them in the Processing Toolbox under
EnMAP-Box ‣ Transformation. The usual way to apply these methods is to use a Fit …
algorithm first and then apply it to an image with Transform raster layer.
This recipe demonstrates the basic workflow of applying transformations
using a principle component analysis (Fit PCA) and the test dataset.
See also
You can find all the available transformation algorithms here.
Open the test dataset
In the processing toolbox go to EnMAP-Box ‣ Transformation ‣ Fit PCA
Specify enmap_berlin.bsq under Raster
Under Output Transformer specify an output file path and click Run
Now open EnMAP-Box ‣ Transformation ‣ Transform raster layer
Select enmap_berlin.bsq as input Raster
Under Transformer click … and select the output .pkl file from the Fit PCA algorithm
Specify an output filepath for the transformed raster under Transformation and click Run
Results of a PCA transformation: input image on the upper left, RGB representation of the first 3 components on the
upper right and singleband gray visualisation of the same components on the bottom.
The Data Sources panel lists the data in your current project, comparable to the Layers panel in QGIS. The following data types and their
corresponding metadata are available:
Raster Data
File size: Metadata on resolution and extent of the raster
CRS: Shows Coordinate Reference System (CRS) information
Bands: Information on overall number of bands as well as band-wise metadata such as name, class or wavelength (if available)
Note
Depending on the type, raster layers will be listed with different icons:
for default raster layers (continuous value range)
for mask raster layers
for classification raster layers
Also see section on data types for further information.
Vector Data
File size: Shows the file size and extent of the vector layer
CRS: Shows Coordinate Reference System (CRS) information
Features: Information on number of features and geometry types
Fields: Attribute information, number of fields as well as field names and corresponding datatype
Spectral Libraries
File size: Size of the file on hard disk
Profiles: Shows the number of spectra in the library
Models
Buttons of the Data Sources panel:
Button
Description
This button lets you add data from different sources, e.g. raster and vector. Same function as .
Remove layers from the Data Sources panel. First select one or more and then click the remove button.
Collapses the whole menu tree, so that only layer type groups are shown.
Expands menu tree to show all branches.
Synchronizes Data Sources with QGIS.
Tip
If you want to remove all layers at once, right-click in the Data Sources panel and and select Remove all DataSources
The EnMAP-Box also supports Tile-/Web Map Services (e.g. Google Satellite or OpenStreetMap) as a raster layer. Just add them to
your QGIS project as you normally would, and then click the Synchronize Data Sources with QGIS
button. Now they should appear in the data source panel and can be added to a Map View.
The Data Views panel organizes the different windows and their content.
You may change the name of a Window by double-clicking onto the name in the list.
Buttons of the Data Views panel:
Button
Description
Open the Raster Layer Styling panel
Remove layers from the Data Views panel. First select one or more and then click the remove button.
Collapses the whole menu tree, so that only layer type groups are shown.
Expands menu tree to show all branches.
Organization of the Data Views panel:
Example of how different window types and their contents are organized in the Data Views panel. In this case there
are two Map Views and one Spectral Library View in the project.
This menu manages the connection between raster sources and spectral library windows.
When collecting profiles, the Identify tool selects profiles from the top-most raster layer by default. The Profile Source panel allows to change this behaviour
and to control:
the profile source, i.e., the raster layer to collect profiles from,
the style how they appear in the profile plot as profile candidate,
the sampling method, for example to aggregate multiple pixel into a single profile first,
the scaling of profile value.
Overview of the Spectral Profile Sources Window with two labeled spectra and main functionalities
Buttons of the Profile Sources
Button
Description
add a new profile source entry
remove selected entries
Profiles
Define the input data from where to take the spectral information from.
Style
Change style of displayed spectra, i.e. symbol and color
Source
Specify a source raster dataset
Double-clicking in the cell will open up a dropdown menu where you can select from all loaded raster datasets.
Sampling
Select Single Profile or Kernel by double-clicking into the cell.
Scaling
Choose how spectra are sampled.
Define the scaling factors by setting the Offset and Scale value.
Option
Description
SingleProfile
Extracts the spectral signature of the pixel at the selected location
Sample3x3
Extracts spectral signatures of the pixel at the selected location and its adjacent pixels in a 3x3 neighborhood.
Sample5x5
Extracts spectral signatures of the pixel at the selected location and its adjacent pixels in a 5x5 neighborhood.
Sample3x3Mean
Extracts the mean spectral signature of the pixel at the selected location and its adjacent pixels in a 3x3 neighborhood.
Sample5x5Mean
Extracts the mean spectral signature of the pixel at the selected location and its adjacent pixels in a 5x5 neighborhood.
The processing toolbox is basically the same panel as in QGIS. Here you can find all EnMAP-Box processing algorithms
listed under EnMAP-Box. In case it is closed/not visible you can open it by clicking the
button in the menubar or View ‣ Panels ‣ QGIS Processing Toolbox.
This tools lets you inspect the values of a layer or multiple layers at the location where you click in the map view. To select a location (e.g. pixel or feature)
use the Select Cursor Location button together with the Identify cursor location value option activated and click somewhere in the map view.
The Cursor Location Value panel should open automatically and list the information for a selected location. The layers will be listed in the order they appear in the Map View.
In case you do not see the panel, you can open it via View ‣ Panels ‣ Cursor Location Values.
By default, raster layer information will only be shown for the bands which are mapped to RGB. If you want to view all bands, change the Visible setting
to All (right dropdown menu). Also, the first information is always the pixel coordinate (column, row).
You can select whether location information should be gathered for All layers or only the Top layer. You can further
define whether you want to consider Raster and Vector layers, or Vector only and Raster only, respectively.
Coordinates of the selected location are shown in the x and y fields. You may change the coordinate system of the displayed
coordinates via the Select CRS button (e.g. for switching to lat/long coordinates).
You can link multiple Map View with each other, so that the contents are synchronized. The following options are
available:
Link map scale and center
Link map scale
Link map center
In order to link Map View, go to View ‣ Set Map Linking in the menu bar, which will open the following dialog:
Here you can specify the above mentioned link options between the Map Views. You may either specify linkages between pairs
or link all canvases at once (the All Canvases option is only specifiable when the number of Map Views is > 2). Remove
created links by clicking .
The Spectral Library Window offers (almost) the same tools like the standard QGIS attribute table. In addition, it provides views and features specifically to visualize and manage spectral profiles.
It directly interacts with the Map View(s), which means spectra can be directly collected from an image. Furthermore, external libraries (e.g. ENVI Spectral Library) can be imported.
Add a new spectral library view by using the Add Spectral Library Window button in the toolbar or open a new window from the menu View ‣ Add Spectral Library Window.
Overview of the Spectral Library view with several collected and labeled spectra and main tools
Buttons of the Spectral Library Window
Button
Description
Button
Description
Add currently overlaid profiles to the spectral library
Activate to add profiles automatically into the spectral library
Make sure to enable the and button in the menu bar and open a raster from which you want to collect spectra in a new Map View.
Click on a desired pixel position in the opened raster image and a new Spectral Library window opens with the spectral profile of the respective pixel.
Profiles obtained from pixel positions are considered as current or temporary profile candidates. The last profile candidate will be replaced by a new one each time you click on a new pixel position.
Click on Add Profile(s) to keep the candidate profile in the spectral library. Activate Add profiles automatically to collect multiple profiles and display them all in the same spectral library.
As an alternative to the mouse you can also identify and select pixel profiles using the shortcuts to change, select and add pixel profiles to the Spectral Library.
First activate the crosshair for the respective image. Click with the right mouse button in the image. Select Crosshair > Pixel Grid > desired raster image.
Now you should see a red square around your pixel and a red dot indicating the position of the pixel profile.
To identify, select and add a pixel profile, use the following key combinations:
Shortcut
Action
←/↑/↓/→
Move the map
Ctrl + ←/↑/↓/→
Select next pixel in arrow direction
Ctrl + S
Add the selected pixel profile candidate
Add profiles from another raster image
Sometimes, you want to compare spectral profiles from different raster sources. The Spectral Profile Source panel allows you to change the default settings of the
Identify tool so that you can select profiles from different images at the same time.
If the Spectral Profile Source Panel is not already visible, open it via View ‣ Panels ‣ Spectral Profile Sources.
Add another profile source relation with and change the Source to the desired raster images.
If you now collect new spectral information, two profiles will appear in the same Spectral Library Window.
Tip
Change the color of one of the profile by changing the Style in the Spectral Profile Sources.
In a similar way you can compare profiles from the same raster image but using a different sampling methods.
In the second relation set the Source to the same image as the first relation.
Change the Sampling to e.g. a 3x3 Kernel mean profile.
You can also add more information to your spectral library by using the attribute table.
Add additional fields to the table, e.g. in order to add information to every spectrum (id, name, classification label, …).
Activate the Table view and enable the Editing mode.
Now you can use the Add Field dialog to add a new column.
Select a data type of your choice.
A new column is added to the attribute table, which you can edit with a double click.
To delete a column, use the Delete field button.
Tip
When you add a new attribute to the table, you can also choose to use it to store new spectral profiles by checking the Use to store spectral profiles checkbox. String, text and binary format can be used to store spectral profiles.
Add information in the layer properties window
It is also possible to add new information to the attribute table in the Layer Properties of the Spectral Library.
Click on to open the spectral library properties.
Navigate to the Fields tab and add a new field. Note: This view does not allow you to set the option Use to store spectral profiles.
Overview of the Layer Properties / Fields section
In addition, the Layer Properties panel allows you to set a certain widget for a specific column.
Switch to the Attributes Form tab in the Layer Properties, select the desired column and choose a certain widget type, e.g. a default range, color, spectral profiles etc.
Selecting widget types for specific columns
The field calculator
The field calculator allows you to modify or assess spectra and calculate new columns or modify existing ones using an expression.
Overview of the Field Calculator
Selecting spectra
Spectra can be selected in the attribute table and in the plot window itself. Selected spectra will be highlighted (blue background in the table; thicker line in a different color in the plot window).
Hold the Shift key to select multiple spectra.
A selection can be removed by clicking the button.
Selected spectra can be removed by using the button.
Tip
You can inspect an individual value of a spectrum by holding the Alt key and clicking some position along the spectrum
It is also possible to select and filter profiles with the common vector filter and selection tools, e.g. select spectra by expression:
Locations of spectra (if available) can be visualized as a point layer by right-clicking into the map window, and selecting Add Spectral Library > SpectralLibrary #
Create / Modify profiles with the Field Calculator
As already mentioned, the Field Calculator can modify attribute values of all or selected features.
In addition, the field calculator can be used to calculate spectral profiles.
Create a new Spectral Profile field based with Add Field, use string, text or binary format and tick the Ise to store Spectral Profiles box.
Open the field calculator and search for spectralData or spectralMath in the Spectral Libraries tab.
SpectralMath allows you to modify spectral profiles with Python code.
To use the SpectralMath function, select a field from which to take the spectral profiles, define an expression and the format.
spectralMath("<profile field 1>",...,"<profile field n>",'<python code>','<output format>')
Note: The last argument defines the output format. It must correspond to the type you assigned when creating the new column.
Example of calculating new spectral profiles
SpectralData returns spectral profile values.
The following table shows some examples of how spectralMath and spectralData can be used.
Description
Example
Multiply the existing profiles
spectralMath(“profiles”, ‘y *=2’, ‘text’)
Create a new profile with x and y values
spectralMath(‘x,y=[1,2,3],[20,30,25]’)
Return spectral profile values from map with spectral data from spectral profiles in field column “profiles”
spectralData(“profiles”)
Return xUnit string of the spectral profile e.g. ‘nm’ for wavelength unit
The Spectral Processing framework allows you to use raster processing algorithms to create new profiles.
Field values of your spectral library will be converted into artificial one-line raster images. In principally, this can be done with most of the field types:
Field Type
Raster Size (band, height, n)
type
Spectral Profile
nb, 1, n
int/float
integer
1, 1, n
integer
float
1, 1, n
float
text
1, 1, n
int (classification)
These temporary raster images are input to standard QGIS processing algorithms or QGIS processing models.
If they generate raster outputs, these outputs can be converted back into field values of the spectral library:
Raster Output
Spectral library Field Type
(>1, 1, n) int/float
Spectral Profile
(1, 1, n) int
integer
(1, 1, n) float
float
This allows you to use the same algorithms to modify spectral profiles as you may want to use to manipulate raster images.
Furthermore, you can make use the QGIS model builder to create (potentially very large and complex) models and use them for both,
spectral libraries and raster image processing.
To use the Spectral Processing tool open and choose the desired algorithm, e.g. Spectral resampling.
Select the input profiles to be translated to the temporary raster layer and specify the outputs. Select an existing field or enter a name to create a new field.
The Profile Plot displays spectral profiles. Toggling the Profile View icon shows or hides the plot panel.
This can be useful, for example to enlarge the attribute table and focus on attribute modifications.
You can adjust the extent of the visualized data range and units
in the plot context menu
using the mouse cursor while keeping the right mouse button pressed
in the visualization settings view
You can also export the entire plot scene or visible view box by clicking into the plot and select Export.
The visualization settings of the spectral library allow you to customize the view according to your needs.
You can define multiple visualization groups that describe how profiles from a specific field and with specific attributes should be visualized.
Overview of the visualization settings in the Spectral Library window
It is also possible, to change the appearance of the Spectral Library window, i.e., bright or dark.
Moreover, activate or deactivate the crosshair and choose a color.
The Current Profiles section shows you all the spectra that have been collected but do not yet appear in the attribute table. Change the color and symbol, or add a line between the points by double clicking the profile below the Current Profile section and adjust the style settings.
Working with multiple visualization groups
The spectral library visualization settings also allow you to add several profile Groups with different style settings.
Add a second visualization group with .
If you want rename Group “profiles”.
Change the color for both groups in the Color.
Under Field you can specify which spectral profile column of the attribute table you want to use.
If you have more than one column that stores spectral information, you can have different visualization groups using different profiles.
If you have only one column where spectral information is stored, but you have another column storing e.g. class names,
you can use the Filter field to define an expression and select only specific class names, e.g. Impervious and Vegetation and visualize these profiles in different colors.
Loading or Saving a spectral library means to load or save vector files.
Load any vector source in the Data Source Panel into a Spectral Library Viewer.
The vector layer does not need to contain any Spectral Profile fields. You can add or define them afterwards.
If your spectral library uses an in-memory vector layer backend, all data will be lost if the layer is closed.
This is the case if the Spectral Library Viewer was opened from scratch with an empty spectral library.
In this case, don’t forget to export collected profiles before closing the Spectral Library Viewer.
If your spectral library already uses a file backend (e.g. .gpkg, .geojson), Style and other layer specific information
are not saved in the data source file, but the QGIS project or a QGIS specific sidecar .qml file.
Open Layer properties > Symbology > Style > Save Default to create or update the .qml file and ensure that the Spectral Profile fields will be restored when re-opening the data set.
The Export dialog allows you to export all or selected profiles as Geopackage (.gpkg), GeoJSON (.geoson) or ENVI Spectral Library (.sli).
The ENVI Spectral Library does not allow saving profiles with different spectral settings (number of bands, wavelength units, FWHM, …)
in the same file. Therefore, you need to select one (out of multiple) profile fields.
Profiles with different spectral settings will be exported into different ENVI files.
The EnMAP-Box stores the minimum data to plot a single profile in a JSON object. In its most simple way, this JSON object
contains a single array “y” of length n, with n = number of spectral profile values:
{"y":[43,23,45,63,45]}
In this case it can be assumed that the corresponding ‘x’ values are an increasing band index “x”: [0, 1, 2, 3, 4].
The JSON object can describe the “x”, the axis units and a vector of bad band values explicitly:
Member
Content
y
An array with n profile values
x
An array with n profile value locations
yUnit
String that describes the unit of y values
xUnit
String that describes the x value unit
bbl
A bad band list
Other metadata to describe spectra profiles are stored in additional vector layer fields.
As JSON object, a single hyperspectral EnMAP profile may therefore look like:
The Add Product menu allows to import satellite products into the EnMAP-Box.
It is also possible to drag&drop products from the file explorer (e.g. Windows Explorer).
The different import algorithms assign proper metadata like wavelength, band names and data offset and scale, resulting in analysis ready raster data.
Usage
Choose the product type from the Project > Add Product menu.
Class Fraction/Probability Renderer and Statistics
The Class Fraction/Probability Renderer and Statistics tool allows to visualize arbitrary many fraction/probability bands
at the same time, using a weighted average of the original class colors, where the weights are given by the
class fractions/probabilities.
Usage
Start the tool from the Tools > Class Fraction/Probability Renderer and Statistics menu or from the layer context menu inside the Data Views panel.
The Decorrelation Stretch Renderer allows to visualize 3 band. It removes the high correlation commonly found in
optical bands to produce a more colorful color composite image.
Usage
Start the tool from the Tools > Decorrelation Stretch Renderer menu or from the layer context menu inside the Data Views panel.
The HSV Color Raster Renderer allows to visualize 3 bands using the HSV (Hue, Saturation, Value/Black) color model.
Find a mapping example here: https://landmonitoring.earth/portal/ ; select Maps > Global Landcover Dynamics 2016-2020
Usage
Start the tool from the Tools > HSV Color Raster Renderer menu or from the layer context menu inside the Data Views panel.
Select HSV bands.
Interactively explore the map.
GUI
Todo
Find a good dataset, that is comparable to the Global Landcover Dynamics 2016-2020 from GeoVille.
The Image Cube tool visualizes a raster image in an interactive 3D view:
Select the raster image.
Specify the:
Top Plane renderer. It can be any raster renderer known from QIGS, e.g. a Multiband
color renderer that shows the true color bands
Cube & Slice renderer. This must be a render that uses a single band only, e.g. a
Singleband grey or Pseudocolor renderer. It will colorize the band-related pixel values
of the 3D image cube and planes relating to the X, Y or Z slice.
Press Load Data to (re)load and render the raster image values.
The 3D scene contains the following elements:
Top Plane - a raster layer for spatial orientation
Image Cube - a volumetric representation of the raster image, showing the raster bands on the z axis
X Slice - a slice along the raster’s X / column / sample dimension
Y Slice - a slice along the raster’s Y / row / line dimension
Z Slice - a slice along the raster’s Z / band dimension
Box (Image) - a 3D bounding box along the maximum image extent
Box (Subset) - a 3D bounding box to show the extent of the spatial subset that migh be used to focus on specific
image areas
The Reclassify tool is a convenient graphical user interface for reclassifying classification rasters.
Specify the file you want to reclassify under Input File. Either use the dropdown menu to select one of the
layers which are already loaded or use the button to open the file selection dialog.
Under Output Classification you can specify the classification scheme of the output classification which
will be created.
You can import schemes from existing rasters or text files by clicking the button.
Use the button to manually add classes.
To remove entries select the respective rows and click the button.
So save a classification scheme select the desired classes (or use Crtl+A to select all) and click on the
button.
Likewise, you can copy and paste classes by selecting them and clicking the Copy ClassesPaste Classes buttons.
The table is sorted by the Label field in ascending order. The value in Label will become the pixel value
of this class and can not be altered.
Double-click into the Name field in order to edit the class name.
Double-click into the Color field to pick a color.
Under Class Mapping you can reassign the old classes (From) to values of the new classification scheme (To)
Specify the output path for the reclassified image under Output File
The Scatter Plot allows to plot two raster bands, or a raster band and a vector field against each other.
The visualization of both, denstity and scatter is supported.
The Classification Dataset Manager allows to
a) create a new dataset from various sources,
b) presents basic information for each category like value, name, color and number of samples,
c) supports editing of category names and colors, and
d) let’s you easily draw a random sample.
Usage
Start the tool from the Applications > Classification Dataset Manager menu.
Use the tool for different usecases like:
create a dataset
edit a dataset
draw a random subsample
Inspect created datasets inside the Data Sources panel.
The Classification workflow algorithm let’s you easily perform classification analysis and mapping tasks
using remote sensing data.
Usage
Start the algorithm from the Applications > Classification workflow menu.
Select a Training dataset.
Select a Classifier.
Select a Raster layer with features used for mapping.
If cross-validation accuracy assessment is desired,
select the Number of cross-validation folds and a
Output classifier performance report file destination
(this step is skipped by default).
If the classifier supports class probability, you may select an
Output class probability layer file destination
(this step is skipped by default).
In the Quick Mapping section you can very easily define your training dataset, fit a classifier and predict a
classification layer, with only a few clicks.
Live demonstration
For a more elaborated analysis see the Detailed Analysis section.
In the Detailed Analysis section you have more control over individual analysis steps.
When performing a detailed analysis, you can basically go through every subsection from left to right.
But, depending on the usecase, it is also possible to skip individual steps you’re not interested in.
You have various options to create a dataset for subsequent analysis: select a Source option
and click create dataset to create a new dataset`.
In the Editor, category colors and names, and feature names can be changed and saved.
By using the various controls in the Draw samples section, you can easily define a training-test-split setup.
The number of training and test samples to be drawn for each category are listed, and also editable, inside the Editor.
Click split dataset to perform the split, resulting in a training and a test dataset, that can be used in subsequent analysis.
In the Classifier section you can either select a Predifined classifier or provide a user-defined Python
Code snipped. See the https://scikit-learn.org/ documentation for a complete overview.
Click create classifier to create an (unfitted) classifier, that can be used in subsequent analysis.
In the Feature Clustering section you can perform an unsupervised Feature redundancy analysis,
that clusters similar features together: select a Dataset, an Algorithm
and click cluster features to create and an Output report.
After inspecting the report you can perform a Feature subset selection:
select a suitable Number of features and click select features to create a training and a test dataset
with fewer features, that are less correlated and can be used in subsequent analysis.
In the Feature Ranking section you can perform a supervised Feature importance analysis,
that ranks features in terms of their importance for the classification task at hand:
select a Dataset, an Algorithm
and click :guilabel:`rank features to create and an Output report.
After inspecting the report you can perform a Feature subset selection:
select a suitable Number of features and click select features to create a training and a test dataset
with fewer features, that are most important and can be used in subsequent analysis.
In the Model section you can perform Model fitting:
select a Dataset and click fit classifier to create a fitted Output classifier,
that is used in subsequent analysis.
For Model performance analysis select an Algorithm and click assess performance to create an Output report.
In the Classification section you can perform Map prediction:
select a Raster layer with features that matches the features used in Model fitting.
Click predict output products to create an Output classification layer and/or an Output class probability layer.
Note that outputs are opened inside the EnMAP-Box Data Sources panel.
For Map accuracy and area estimation select a Ground truth categorized layer and click assess performance to create an Output report.
In the Settings section you can specify the Output directory (e.g. C:/Users/USERNAME/AppData/Local/Temp/EnMAPBox/ClassificationWorkflow),
that is used as the default file destination path, when creating file outputs.
Note that each output file wigdet (e.g. Output dataset) has a default basename (e.g. dataset.pkl),
that is used to create a default file destination (e.g. C:/Users/USERNAME/AppData/Local/Temp/EnMAPBox/ClassificationWorkflow/dataset.pkl).
If the default file destination already exists, the basename is enumerated (e.g. .dataset_2.pkl) to avoid overwriting existing outputs.
Have a look at the Getting Started for a use case example of the Classification Workflow Application.
Input Parameters:
Training Inputs
Type
Three different types of input data sources are supported and have to be specified beforehand in the dropdown menu.
Depending on the selected input type the user interface shows different options.
Raster/Classification:
Raster: Specify input raster based on which samples will be drawn for training a classifier.
Classification: Specify input raster which holds class information.
Raster/VectorClassification:
Raster: Specify input raster based on which samples will be drawn for training a classifier.
Reference: Specify vector dataset with reference information. Has to have a column in the attribute table with a
unique class identifier (numeric). The class colors and labels are derived from the current Symbology. To set or
change those settings, click the button or go to the Layer Properties (Layer properties ‣ Symbology).
The vector dataset is rasterized/burned on-the-fly onto the grid of the input raster in order to extract the sample.
If the vector source is a polygon dataset, only polygons which cover more than 75% of a pixel in the target grid are rasterized.
labelledLibrary:
Library: Specify input spectral library.
Sampling
Once you specified all inputs in the Training inputs section, you can edit the class colors, names and class sample sizes
in the Sampling submenu.
Note
If set, the class labels and color information is automatically retrieved from the layers current renderer settings
(Layer properties ‣ Symbology).
Sample size Specify the sample size per class, either relative in percent or in absolute pixel counts.
The total sample size is shown below
Save sample: Activate this option and specify an output path to save the sample as a raster.
Training
In the Classifier dropdown menu you can choose different classifiers (e.g. Random Forest, Support Vector Machine)
Model parameters: Specify the parameters of the selected classifier.
Hint
Scikit-learn python syntax is used here, which means you can specify model parameters accordingly. Have a look at
the scikit-learn documentation on the individual parameters, e.g. for the RandomForestClassifier
Save model: Activate this option to save the model file (.pkl) to disk.
Mapping
Raster: Specify the raster you would like to apply the trained classifier to (usually -but not necessarily-
this is the same as used for training)
Mask: Specify a mask layer if you want to exclude certain areas from the prediction.
Outputs:
Classification: Output path where to write the classification image to.
Probability: Output path of the class probability image.
Hint
This outputs the result of a classifiers predict_proba method. Note that depending on the classifier this
option might not be available or has to be activated in the model parameters (e.g. for the Support Vector Machine,
the line svc=SVC(probability=False) has to be altered to svc=SVC(probability=True)
RGB: Generates a RGB visualisation based on the weighted sum of class colors and class probabilities.
Cross-validation Accuracy Assessment
Cross-validation with n-folds : Activate this setting to assess the accuracy of the classification by performing cross
validation. Specify the desired number of folds (default: 3). HTML report will be generated at the specified output path.
Run the classification workflow
Once all parameters are entered, press the button to start the classification workflow.
The Raster math algorithm is a powerful raster calculator inspired by the QGIS Raster calculator,
the GDAL Raster calculator and ENVI Band Math.
In addition to those tools, the EnMAP-Box Raster math calculator supports multi-band arrays, vector layer inputs,
multi-line code fragments and metadata handling.
Usage
Start the algorithm from the Applications > Raster math menu
or from the Processing Toolbox panel.
Specify a single-line expression or a multi-line code fragment to be evaluated inside the Code editor.
Therefor select raster bands or numeric vector fields from the Available data sources tab.
[Optional] Select the destination Grid. If not specified, the grid of the first raster layer is used.
Note that a) all input raster bands are resampled and b) all input vector fields are rasterized
into the destination grid before the calculation.
[Optional] In case you want to perform a spatial operation, be sure to select a proper Block overlap
or select Monolithic processing, to avoid artefacts at the block edges.
[Optional] Note that all inputs are converted to 32-bit floating-point values by default.
[Optional] You can select up to 10 additional raster inputs R1, …, R10 and vector inputs V1, …, V10.
Additionally, a list of raster inputs RS can be selected.
Select an Output raster layer file destination an click Run.
An individual raster band can be accessed using the <layer name>@<band number> syntax, e.g. band number 42:
enmap_berlin@42
In case of a spectral raster, the band nearest to a target wavelength (in nanometers)
can be selected using the <layer name>@<band number>nm syntax, e.g. NIR band at 865 nm:
enmap_berlin@865nm
Note that prominent target wavelength from the Landsat/Sentinel-2 sensors can be selected inside the
Waveband locator tab.
All raster bands can be accessed at once using the <layer name> syntax, e.g.:
enmap_berlin
A band subset can be accessed using the <layer name>@<start>:<stop> syntax, e.g. band numbers 10 to 19:
enmap_berlin@10:20# note that 20 is not included# Note that you can also create a band subset by indexing the the full band array.# This has the slight disadvantage, that all bands are read into memory first.enmap_berlin[9:19]
A raster mask, is a predefined boolean array, which evaluates to False for every pixel containing the no data value,
nan or inf. All other pixel evaluate to True.
Use the <layername>Mask syntax to access the 3d binary mask for all bands,
and the <layername>Mask@<bandnumber> syntax for a 2d single band mask.
2d mask array for a single band: enmap_berlinMask@655nm
A vector mask, is a predefined boolean array, which evaluates to True for every pixel covered by a geometry.
All other pixel evaluate to False.
Use the <layername> syntax to access the 2d binary mask.
2d mask array for a vector layer: landcover_berlin_polygon
Example - mask a raster using a polygon-vector
enmap_berlin*landcover_berlin_polygon
Note that the output raster is correctly masked, but we haven’t set an appropriate no data value,
nor have we taken care of wavelength information or any other metadata.
To properly do this, we need to use multi-line code fragments.
The computation is done block-wise by default to be memory efficient.
The actual block size depends on the system memory.
In rare cases it may be helpful to get some information about the current block, using the special variable block.
get the current block x/y offset: block.xOffset,block.yOffset:
0,0
get the current block x/y size: block.width,block.height:
220,400
If the computation involves a spatial operation, e.g. a spatial convolution filter with a kernel,
be sure to also specify a proper Block overlap.
E.g. for a 5x5 kernel, set at least a block overlap of 2 pixel.
In cases where the spatial operation is not locally limitted to a fixed spatial neightbourhood,
e.g. region growing or segmentation, Monolithic processing can be activated,
where all data is processed in one big block.
To enable more complex computations, multiple outputs and metadata handling, we can use multi-line code fragments.
Example - calculate the NDVI index
In this example we first specify _nir and _red variables to then calculate the _ndvi,
which we pass to the special outputRaster identifier,
that is associated with the Output raster layer:
The underscore postfix _ marks the nir_, red_ and ndvi_ variables as temporary.
Instead of nir_ we can also use _nir, tmp_nir or temp_nir.
The explicite assignment outputRaster=ndvi_ can be avoided,
by selecting an Output raster layer file destination,
where the file basename (without extension) matches the variable name,
e.g. c:/ndvi.tif:
nir_=enmap_berlin@865nmred_=enmap_berlin@655nmndvi=(nir_-red_)/(nir_+red_)# ndvi matches with c:/ndvi.tif
If the file basename isn’t matching correctly, you will get the following error message inside the Log panel:
The following layers were not correctly generated.
• C:/Users/Andreas/AppData/Local/Temp/processing_BVyjbt/3a6c795d9a594937acf441c5a372f448/outputRaster.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
Instead of using temporary variables, you can also just delete unwanted variables as a last step:
To calculate multiple outputs, just define multiple non-temporary variables:
N=enmap_berlin@865nm/1e4# EVI needs data scaled to 0-1 rangeR=enmap_berlin@655nm/1e4B=enmap_berlin@482nm/1e4ndvi=(N-R)/(N+R)evi=2.5*(N-R)/(N+6*R-7.5*B+1)delN,R,B
Note that you can only specify the file destination of one of the outputs,
e.g. by setting Output raster layer to c:/results/ndvi.tif or c:/results/evi.tif.
The other output is written into the same directory as a GeoTiff with the basename
matching the variable name c:/results/<basename>.tif.
You may also keep the default file destination [Save to temporary file] as is,
to write all outputs into a temp folder. In this case, it is fine to just ignore the error message:
The following layers were not correctly generated.
• C:/Users/Andreas/AppData/Local/Temp/processing_BVyjbt/3a6c795d9a594937acf441c5a372f448/outputRaster.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
Also note the shortcuts inside the Available data sources tab context menu
and the Data / Metadata tab.
Example - calculate the NDVI index and set up metadata properly
This example shows how to properly calculate the NDVI index, masking no data pixel and set up output metadata:
# find bandsred=enmap_berlin@655nmnir=enmap_berlin@865nm# calculate NDVIndvi=(nir-red)/(nir+red)# mask no data regionnoDataValue=-9999ndvi[~enmap_berlinMask@655nm]=noDataValuendvi[~enmap_berlinMask@865nm]=noDataValue# set no data value and band namendvi.setNoDataValue(noDataValue)ndvi.setBandName('NDVI',bandNo=1)# clean up temp variablesdelred,nir
Example - copy raster data and metadata
This example shows how to properly copy a raster data and metadata:
Have a look at the Biomass Mapping Tutorial for a use case example of the Regression Workflow Application.
Input Parameters:
Training Inputs
Raster: Specify input raster based on which samples will be drawn for training a regressor.
Reference: Specify vector or raster dataset with reference information (regression target). In case of
vector input, dataset has to have a numeric column in the attribute table with a
target variable of interest. This vector dataset is rasterized/burned on-the-fly onto the grid of
the input raster in order to extract the sample. If the vector source is a polygon dataset, all pixels will be drawn which
intersect the polygon.
Attribute: Attribute field in the reference vector layer which contains the regression target variable.
Sampling
Number of Strata: Specify the desired number of strata sampling. If you don’t want to use
stratified sampling, just enter 1.
Min & Max: Defines the value range in which samples should be drawn.
Sample size: Specify the sample size per stratum, either relative in percent or in absolute pixel counts.
Every stratum is listed below with the value range that is covered by this stratum depicted in square brackets
(e.g., Stratum1:[1.0,4.33]). Furthermore, you can see and alter the number of pixels/samples for each stratum using the spinboxes.
Save sample: Activate this option and specify an output path to save the sample as a raster.
Training
In the Regressor dropdown menu you can choose different regressors (e.g. Random Forest, Support Vector Regression, Kernel Ridge Regression)
Model parameters: Specify the parameters of the selected regressor.
Hint
Scikit-learn python syntax is used here, which means you can specify model parameters accordingly. Have a look at
the scikit-learn documentation on the individual parameters, e.g. for the RandomForestRegressor
Save model: Activate this option to save the model file (.pkl) to disk.
Mapping
Input Raster: Specify the raster you would like to apply the trained regressor to (usually -but not necessarily-
this is the same as used for training)
Cross-validation Accuracy Assessment
Cross-validation with n-folds : Activate this setting to assess the accuracy of the regression by performing cross
validation. Specify the desired number of folds (default: 3). HTML report will be generated at the specified output path.
Run the regression workflow
Once all parameters are entered, press the button to start the regression workflow.
Earth Observation for QGIS (EO4Q) is a collection of EnMAP-Box tools and applications designed to integrate well in both,
EnMAP-Box and QGIS environments.
In both environments, EO4Q applications can be started from the Earth Observation for QGIS (EO4Q) Toolbar:
The GEE Time Series Exlorer integrates Google Earth Engine (GEE) into
QGIS/EnMAP-Box. It allows the interactive exploration of temporal raster data available in
the Earth Engine Data Catalog .
A first version of the GEE Time Series Exlorer was released as a QGIS plugin.
Future versions will be maintained as an EO4Q application that integrates into QGIS and EnMAP-Box GUI.
Live demonstration
Slides from the Living Planet Symposium 2022 talk in Bonn, Germany
The GEE Time Series Exlorer depends on the QGIS Google Earth Engine plugin .
In order to access Earth Engine, you must have an Google account that is authorized for Earth Engine.
If you haven’t used Earth Engine so far, the easiest way to make sure that everything is working on your system,
is to follow the instruction here: https://gee-community.github.io/qgis-earthengine-plugin/
Start the application from the View > Panels > GEE Time Series Exlorer menu or from the
Earth Observation for QGIS (EO4Q) Toolbar.
GEE Time Series Explorer main panel, showing the Data Catalog tab with Landsat collections.
Select a collection.
Load a collection, plot a temporal profile and visualize an image
Select the Landsat 8 Surface Reflectance collection in the Data Catalog tab of the main panel.
Select the NDVI band in the Profile Viewer panel.
Activate the Current Location map tool |icon_location| and select a location on the map.
This will plot the temporal profile for that location in the Profile Viewer panel.
Select a data point in the plot to visualize the associated image.
The image is displayed in it’s default visualization.
Improve image contrast stretch
The default visualization may give a poor image contrast, which you may want to improve.
In the Band Rendering tab of the main panel, you may set suitable min/max values manually,
or specify lower/upper percentile cut off values, e.g. 2% to 98%.
Note that the statistics are calculated for the current map extent.
Visualize derived vegetation indices
Beside visualizing original image bands in Multiband color RGB,
it is possible to visualize derived vegetation indices.
Prepare a cloud-free composite
To create a composite that aggregates all images in a given date range,
we just switch from Image Selection mode to Composite Selection mode in the Profile Viewer panel.
In the plot we can now select a date range and create our first, very cloudy, composite.
By applying a Pixel Quality Filter we can easily exclude all pixel affected by cloud and cloud shadow.
And finally, we can improve the contrast stretch of the visualization.
The Profile Analytics panel allows to visualize various types of spectral, temporal and spatial profiles.
Additionally, profile data can be analysed by user-defined functions (ufuncs).
A ufunc has access to the plot widget and can draw additional plot items.
Usage
Start the tool from the View > Panels > Profile Analytics menu or from the
Earth Observation for QGIS (EO4Q) Toolbar.
Select the Source type, that is providing the profiles.
Note that we’re currently only support raster layer as source,
but we plan to have other sources like profiles from the GEE Time Series Explorer.
Select the Profile type you want to extract from the raster layer:
Select a Raster.
In case of a spatial profile
(i.e. X-Profile, Y-Profile and Profile along a line),
also select a Band. In case of a Z-Profile),
the selected band is ignored.
Style the profile.
Apply data Scaling.
Choose an ufunc to perform Analytics on the profile data and add extra plot annotations.
The ufunc has full access to the plot widget and can add plot items like:
plot line (e.g. fitted data, vegetation regrowth, trend lines, etc.)
plot marker symbols (e.g. forest clear cut events, mowing events, fire events, red-edge inflection point, etc.)
plot text
insert images
Examples can be found under /enmapbox/eo4qapps/profileanalyticsapp/examples/.
Depending on the Profile type and availability of raster metadata,
different X Axis units can be choosen:
Z-Profile values:
can always be plotted against Band Numbers
can be plotted against Wavelength, if band center wavelength is specified
can be plotted against Date Time, if band (aquisition) date time is specified
X-Profile values are always plotted against the Column Number.
Y-Profile values are always plotted against the Row Number.
Profile along a line values are always plotted against the Distance from line start.
In case of X-Profile, Y-Profile and Z-Profile,
use the Cursor Location map tool to select a location that specifies the profile.
In case of Profile along a line,
use the Select Feature map tool to select a line-vector feature that specifies the profile.
GUI
Spectral Z-Profile
Temporal Z-Profiles
Temporal Z-Profile annotated with a Support Vector Regression fit (svr_fitting.py used as ufunc)
The Raster Band Stacking panel allows to stack bands into a new VRT raster layer.
Raster bands can be selected inside the panel or added via drag&drop in various ways.
Usage
Start the tool from the View > Panels > Raster Band Stacking menu or from the
Earth Observation for QGIS (EO4Q) Toolbar.
Add raster sources and select bands:
add a new raster source via the “+” button
select raster(s) inside the Data Sources panel and drag&drop the selection into the table
select band(s) inside the Data Sources panel and drag&drop the selection into the table
select raster layer(s) inside the Data Views panel and drag&drop the selection into the table
select raster files inside the file explorer (e.g. Windows Explorer) and drag&drop the selection into the table
Prepare the final band stack inside the table by:
changing individual band selections
removing rows
moving rows up and down
Choose an output filename and create the band stack.
By default, the output pixel grid (i.e. extent, resolution, crs) is derived automatically (i.e. gdal.BuildVrt defaults).
To use a custom pixel grid, switch to the Raster option and select a raster.
The Sensor Product Import panel allows to import various sensor products via drag&drop.
E.g. a downloaded Landsat product can be imported as is:
Landsat 9 Product Example
All the surface reflectance bands are automatically stacked, band offset and scaling factors are applied,
and proper metadata, like center wavelength and band names, are specified.
Usage
Start the tool from the View > Panels > Sensor Product Import menu or from the
Earth Observation for QGIS (EO4Q) Toolbar.
Drag&drop the product folder, or any file inside the product folder, into the Drop Product Here area.
Run the import algorithm. Note that the result is stored next to the source product.
When opening the product for the next time, this algorithm is not shown.
Hyperspectral EnMAP Level 2A data acquired on the 24th of July 2022 over the City of Potsdam, Germany,
Provider: DLR (german aerospace center)
EnMAP L2A scene ID: ENMAP01-____L2A-DT0000001867_20220724T104526Z_008_V010302_20230628T165614Z (processing v1.3.2)
Ground sampling distance: 30 m
Number of bands: 224 (from 400 to 2450 nm)
Number of bad bands: 6 (filled with no data value)
Data: 330x200x244 (Float32)
aerial_potsdam.tif
Aerial image acquired on the 1st of April 2019 over the City of Potsdam, Germany.
Provider: LGB (Landesvermessung und Geobasisinformation Brandenburg)
Ground sampling distance: 0.2 m resampled to 1 m
Number of bands: 4 (red, green, blue, near-infrared)
Data: 9000x9000x4 (Byte)
Copyrights: GeoBasis-DE/LGB, dl-de/by-2-0 and Geoportal Berlin, dl-de/by-2-0 (data modified). Also see https://geobroker.geobasis-bb.de/gbss.php?MODE=GetProductInformation&PRODUCTID=253b7d3d-6b42-47dc-b127-682de078b7ae
landcover_potsdam_polygon.gpkg
Land cover reference polygons derived from the aerial imagery described above.
Number of polygons: 9677
Level 1 classes: Impervious, Vegetation, Soil, Water
Level 2 classes: Impervious, Low Vegetation, Soil, Tree, Water
Level 3 classes: Roof, Pavement, Low Vegetation, Soil, Tree, Water
landcover_potsdam_point.gpkg
Land cover reference points derived from the aerial imagery described above.
Number of points: 120
Level 1 classes: Impervious, Vegetation, Soil, Water
Level 2 classes: Impervious, Low Vegetation, Soil, Tree, Water
All layers from the Example Dataset opened in one map view
The delta in x direction for creating the isometric perspective. The 3d cube is tilted to the left for negative values, and to the right for positive values.
Default: 1
Delta y (pixel) [number]
The delta in y direction for creating the isometric perspective. The 3d cube is tilted to the downwards for negative values, and upward for positive values.
Create a paletted raster renderer from given categories and set it as the default style of the given raster layer.
This will create/overwrite the QML sidecar file of the given raster layer.
Parameters
Raster layer [raster]
The raster layer for which to create the QML sidecar file.
Comma separated list of tuples with category value, name and color information. E.g.
(1, ‘Urban’, ‘#FF0000’), (2, ‘Forest’, ‘#00FF00’), (3, ‘Water’, ‘#0000FF’)
Class fraction layer from categorized vector layer
Rasterize a categorized vector layer into class fractions. Categories are rasterized at a x10 finer resolution and aggregated to class-wise fractions at destination resolution. This approach leads to fractions that are accurate to the percent.
Classification layer accuracy and area report (for simple random sampling)
Estimates map accuracy and area proportions for (simple) random sampling. We use the formulas for the stratified random sampling described in Stehman (2014): https://doi.org/10.1080/01431161.2014.930207. Note that (simple) random sampling is a special case of stratified random sampling, with exactly one stratum.
Observed and predicted categories are matched by name.
Gaussian process classification (GPC) based on Laplace approximation.
The implementation is based on Algorithm 3.1, 3.2, and 5.1 of Gaussian Processes for Machine Learning (GPML) by Rasmussen and Williams.
Internally, the Laplace approximation is used for approximating the non-Gaussian posterior by a Gaussian. Currently, the implementation is restricted to using the logistic link function. For multi-class classification, several binary one-versus rest classifiers are fitted. Note that this class thus does not implement a true multi-class Laplace approximation.
See Gaussian Processes for further information.
Linear Support Vector Classification.
Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.
This class supports both dense and sparse input and the multiclass support is handled according to a one-vs-the-rest scheme.
Logistic Regression (aka logit, MaxEnt) classifier.
In the multiclass case, the training algorithm uses the one-vs-rest (OvR) scheme if the ‘multi_class’ option is set to ‘ovr’, and uses the cross-entropy loss if the ‘multi_class’ option is set to ‘multinomial’.
Parameters
Classifier [string]
Scikit-learn python code. See LogisticRegression for information on different parameters.
A random forest classifier.
A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled with the max_samples parameter if bootstrap=True (default), otherwise the whole dataset is used to build each tree.
Parameters
Classifier [string]
Scikit-learn python code. See RandomForestClassifier for information on different parameters.
C-Support Vector Classification.
The implementation is based on libsvm. The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples.
The multiclass support is handled according to a one-vs-one scheme.
C-Support Vector Classification.
The implementation is based on libsvm. The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples.
The multiclass support is handled according to a one-vs-one scheme.
A raster layer with bands used as features. Classifier features and raster bands are matched by name to allow for classifiers trained on a subset of the raster bands. If raster bands and classifier features are not matching by name, but overall number of bands and features do match, raster bands are used in original order.
A list of source to targetclass value mappings. E.g. to merge source values 1 and 2 into target value 1, and source values 3 and 4 into target value 2, use {1:1, 2:1, 3:2, 4:2}
Categories [string]
A list of targetcategories in short notation: [(1, ‘Class A’, ‘#e60000’), (2, ‘Class B’, ‘#267300’)]
Implements the BIRCH clustering algorithm.
It is a memory-efficient, online-learning algorithm provided as an alternative to MiniBatchKMeans. It constructs a tree data structure with the cluster centroids being read off the leaf. These can be either the final cluster centroids or can be provided as input to another clustering algorithm such as AgglomerativeClustering.
Parameters
Clusterer [string]
Scikit-learn python code. See Birch for information on different parameters.
Mean shift clustering using a flat kernel.
Mean shift clustering aims to discover “blobs” in a smooth density of samples. It is a centroid-based algorithm, which works by updating candidates for centroids to be the mean of the points within a given region. These candidates are then filtered in a post-processing stage to eliminate near-duplicates to form the final set of centroids.
Seeding is performed using a binning technique for scalability.
Parameters
Clusterer [string]
Scikit-learn python code. See MeanShift for information on different parameters.
A raster layer with bands used as features. Clusterer features and raster bands are matched by name to allow for clusterers trained on a subset of the raster bands. If raster bands and clusterer features are not matching by name, but overall number of bands and features do match, raster bands are used in original order.
2D Box filter.
The Box filter or running mean is a smoothing filter. It is not isotropic and can produce artifact, when applied repeatedly to the same data.
2D Ricker Wavelet filter kernel (sometimes known as a Mexican Hat kernel).
The Ricker Wavelet, or inverted Gaussian-Laplace filter, is a bandpass filter. It smooths the data and removes slowly varying or constant structures (e.g. background). It is useful for peak or multi-scale detection.
2D Ring filter.
The Ring filter kernel is the difference between two Top-Hat kernels of different width. This kernel is useful for, e.g., background estimation.
Python code. See generic_filter for information on different parameters.
Default:
fromscipy.ndimage.filtersimportgeneric_filterimportnumpyasnpdeffilter_function(invalues):# do whatever you want to create the output value, e.g. np.nansumoutvalue=np.nansum(invalues)# replace this line with your code!returnoutvaluefunction=lambdaarray:generic_filter(array,function=filter_function,size=3)
Outputs
Output raster layer [rasterDestination]
Raster file destination.
Command-line usage
>qgis_processhelpenmapbox:SpatialGenericFilter:
----------------Arguments----------------raster:RasterlayerArgumenttype:rasterAcceptablevalues:-Pathtoarasterlayerfunction:FunctionDefaultvalue:fromscipy.ndimage.filtersimportgeneric_filterimportnumpyasnpdeffilter_function(invalues):# do whatever you want to create the output value, e.g. np.nansumoutvalue=np.nansum(invalues)# replace this line with your code!returnoutvaluefunction=lambdaarray:generic_filter(array,function=filter_function,size=3)Argumenttype:stringAcceptablevalues:-StringvalueoutputRaster:OutputrasterlayerArgumenttype:rasterDestinationAcceptablevalues:-Pathfornewrasterlayer----------------Outputs----------------outputRaster:<outputRaster>Outputrasterlayer
1D Box filter.
The Box filter or running mean is a smoothing filter. It is not isotropic and can produce artifacts, when applied repeatedly to the same data.
1D Ricker Wavelet filter kernel (sometimes known as a Mexican Hat kernel).
The Ricker Wavelet, or inverted Gaussian-Laplace filter, is a bandpass filter. It smooths the data and removes slowly varying or constant structures (e.g. background). It is useful for peak or multi-scale detection.
Create classification dataset (from categorized raster layer and feature raster)
Create a classificationdataset by sampling data for pixels that match the given categories and store the result as a pickle file.
If the layer is not categorized, or the band with class values is selected manually, categories are derived from sampled data itself. To be more precise: i) category values are derived from unique raster band values (after excluding no data or zero data pixel), ii) category names are set equal to the category values, and iii) category colors are picked randomly.
Create a classificationdataset from spectral profiles that matches the given categories and store the result as a pickle file.
If the spectral library is not categorized, or the field with class values is selected manually, categories are derived from target data y. To be more precise: i) category values are derived from unique attribute values (after excluding no data or zero data values), ii) category names are set equal to the category values, and iii) category colors are picked randomly.
Field with class values used as target data y. If not selected, the field defined by the renderer is used. If that is also not specified, an error is raised.
Field with spectral profiles used as features [field]
Field with spectral profiles used as feature data X. If not selected, the default field named “profiles” is used. If that is also not available, an error is raised.
Create classification dataset (from categorized vector layer and feature raster)
Create a classificationdataset by sampling data for pixels that match the given categories and store the result as a pickle file.
If the layer is not categorized, or the field with class values is selected manually, categories are derived from the sampled target data y. To be more precise: i) category values are derived from unique attribute values (after excluding no data or zero data values), ii) category names are set equal to the category values, and iii) category colors are picked randomly.
Whether to exclude bands, that are marked as bad bands, or contain no data, inf or nan values in all samples.
Default: True
Field with class values [field]
Field with class values used as target data y. If not selected, the field defined by the renderer is used. If that is also not specified, an error is raised.
Minimum pixel coverage [%] [number]
Exclude all pixel where (polygon) coverage is smaller than given threshold.
Default: 50
Majority voting [boolean]
Whether to use majority voting. Turn off to use simple nearest neighbour resampling, which is much faster, but may result in highly inaccurate class decisions.
Create classification dataset (from categorized vector layer with attribute table)
Create a classificationdataset from attribute table rows that matches the given categories and store the result as a pickle file.
If the layer is not categorized, or the field with class values is selected manually, categories are derived from the target data y. To be more precise: i) category values are derived from unique attribute values (after excluding no data or zero data values), ii) category names are set equal to the category values, and iii) category colors are picked randomly.
Field with class values used as target data y. If not selected, the field defined by the renderer is used. If that is also not specified, an error is raised.
Create a classificationdataset from a JSON file and store the result as a pickle file.
Example file (classifier.pkl.json) can be found in the EnMAP-Box testdata folder).
fromenmapboxprocessing.typingimportNumber,List,Category,ClassifierDump# specify categories and feature namescategories:List[Category]=[Category(value=1,name='class 1',color='#ff0000'),Category(value=2,name='class 2',color='#00ff00')]features:List[str]=['Feature 1','Feature 2','Feature 3']# specify features X as 2d-array with shape (samples, features)X:List[List[Number]]=[[1,2,3],[4,5,6]]# specify targets y as 2d-array with shape (samples, 1)y:List[List[int]]=[[1],[2]]
----------------Arguments----------------code:CodeDefaultvalue:fromenmapboxprocessing.typingimportNumber,List,Category,ClassifierDump# specify categories and feature namescategories:List[Category]=[Category(value=1,name='class 1',color='#ff0000'),Category(value=2,name='class 2',color='#00ff00')]features:List[str]=['Feature 1','Feature 2','Feature 3']# specify features X as 2d-array with shape (samples, features)X:List[List[Number]]=[[1,2,3],[4,5,6]]# specify targets y as 2d-array with shape (samples, 1)y:List[List[int]]=[[1],[2]]Argumenttype:stringAcceptablevalues:-StringvalueoutputClassificationDataset:OutputdatasetArgumenttype:fileDestinationAcceptablevalues:-Pathfornewfile----------------Outputs----------------outputClassificationDataset:<outputFile>Outputdataset
Create classification dataset (from table with categories and feature fields)
Fields with continuous-valued values used as target data y. If not selected, the fields defined by the renderer is used. If those are also not specified, an error is raised.
Field with spectral profiles used as features [field]
Field with spectral profiles used as feature data X. If not selected, the default field named “profiles” is used. If that is also not available, an error is raised.
Fields with values used as used as target data y. If not selected, the fields defined by the renderer are used. If those are also not specified, an error is raised.
Create a regressiondataset from a JSON file and store the result as a pickle file.
Example file (regressor.pkl.json) can be found in the EnMAP-Box testdata folder).
fromenmapboxprocessing.typingimportNumber,List,Target,RegressorDump# specify targets and feature namestargets:List[Target]=[Target(name='variable 1',color='#ff0000'),Target(name='variable 2',color='#00ff00')]features:List[str]=['Feature 1','Feature 2','Feature 3']# specify features X as 2d-array with shape (samples, features)X:List[List[Number]]=[[1,2,3],[4,5,6]]# specify targets y as 2d-array with shape (samples, targets)y:List[List[float]]=[[1.1,1.2],[2.1,2.2]]
----------------Arguments----------------code:CodeDefaultvalue:fromenmapboxprocessing.typingimportNumber,List,Target,RegressorDump# specify targets and feature namestargets:List[Target]=[Target(name='variable 1',color='#ff0000'),Target(name='variable 2',color='#00ff00')]features:List[str]=['Feature 1','Feature 2','Feature 3']# specify features X as 2d-array with shape (samples, features)X:List[List[Number]]=[[1,2,3],[4,5,6]]# specify targets y as 2d-array with shape (samples, targets)y:List[List[float]]=[[1.1,1.2],[2.1,2.2]]Argumenttype:stringAcceptablevalues:-StringvalueoutputRegressionDataset:OutputdatasetArgumenttype:fileDestinationAcceptablevalues:-Pathfornewfile----------------Outputs----------------outputRegressionDataset:<outputFile>Outputdataset
Create regression dataset (from table with target and feature fields)
Whether to include the original library spectra into the dataset.
Default: True
Mixing complexity probabilities [string]
A list of probabilities for using 2, 3, 4, … endmember mixing models. Trailing 0 probabilities can be skipped. The default values of 0.5, 0.5,result in 50% 2-endmember and 50% 3-endmember models.
Default: 0.5, 0.5
Allow within-class mixtures [boolean]
Whether to allow mixtures with profiles belonging to the same class.
Default: True
Class probabilities [string]
A list of probabilities for drawing profiles from each class. If not specified, class probabilities are proportional to the class size.
Create an unsupervised dataset by sampling data from valid pixels and store the result as a pickle file.
A pixel is concidered valid, if the pixel profile is free of no data values, and not excluded by the (optionally) selected mask layer.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Default: 0
Exclude bad bands [boolean]
Whether to exclude bands, that are marked as bad bands, or contain no data, inf or nan values in all samples.
fromenmapboxprocessing.typingimportNumber,List# specify feature namesfeatures:List[str]=['Feature 1','Feature 2','Feature 3']# specify features X as 2d-array with shape (samples, features)X:List[List[Number]]=[[1,2,3],[4,5,6]]
----------------Arguments----------------code:CodeDefaultvalue:fromenmapboxprocessing.typingimportNumber,List# specify feature namesfeatures:List[str]=['Feature 1','Feature 2','Feature 3']# specify features X as 2d-array with shape (samples, features)X:List[List[Number]]=[[1,2,3],[4,5,6]]Argumenttype:stringAcceptablevalues:-StringvalueoutputUnsupervisedDataset:OutputdatasetArgumenttype:fileDestinationAcceptablevalues:-Pathfornewfile----------------Outputs----------------outputUnsupervisedDataset:<outputFile>Outputdataset
Spectral library with feature data X (i.e. spectral profiles). It is assumed, but not enforced, that the spectral characteristics of all spectral profiles match. If not all spectral profiles share the same number of spectral bands, an error is raised.
Field with spectral profiles used as features [field]
Field with spectral profiles used as feature data X. If not selected, the default field named “profiles” is used. If that is also not available, an error is raised.
Create an unsupervised dataset from a tabulated text file and store the result as a pickle file.
The format matches that of the FORCE Higher Level Sampling Submodule.
An example file (force_features.csv) can be found in the EnMAP-Box testdata folder).
Number of samples to draw from each category. Set a single value N to draw N samples for each category. Set a list of values N1, N2, … Ni, … to draw Ni samples for category i.
Number of samples to draw from each bin. Set a single value N to draw N samples for each bin. Set a list of values N1, N2, … Ni, … to draw Ni samples for bin i.
Permutation feature importance is a model inspection technique that is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. This technique benefits from being model agnostic and can be calculated many times with different permutations of the feature.
Evaluate feature multicollinearity by performing hierarchical/agglomerative clustering with Ward linkage using squared Spearman rank-order correlation as distance between features. The result report includes i) pairwise squared Spearman rank-order correlation matrix, ii) clustering dendrogram, iii) inter-cluster correlation distribution, iv) intra-cluster correlation distribution, and v) a clustering hierarchy table detailing selected cluster representatives for each cluster size n.
For further analysis, all relevant results are also stored as a JSON sidecar file next to the report.
Prepare a spectral raster layer from the given product. Wavelength and FWHM information is set and data is scaled according to data gain/offset values.
Parameters
Metadata file [file]
The metadata XML file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata XML file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Prepare a spectral raster layer from the given product. Wavelength and FWHM information is set and data is scaled according to data gain/offset values.
Note that the DESIS L1C spectral data file is band interleaved by pixel and compressed, which is very disadvantageous for visualization in QGIS / EnMAP-Box. For faster exploration concider saving the resulting VRT raster layer as GTiff format via the “Translate raster layer” algorithm.
Parameters
Metadata file [file]
The metadata XML file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata XML file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Prepare a spectral raster layer from the given product. Wavelength and FWHM information is set and data is scaled into the 0 to 10000 range.
Note that the DESIS L2D spectral data file is band interleaved by pixel and compressed, which is very disadvantageous for visualization in QGIS / EnMAP-Box. For faster exploration concider saving the resulting VRT raster layer as GTiff format via the “Translate raster layer” algorithm.
Parameters
Metadata file [file]
The metadata XML file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata XML file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
The EMIT L2A RFL NetCDF product file.
Instead of executing this algorithm, you may drag&drop the NetCDF file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Prepare VNIR and SWIR spectral raster layer from the given product. Wavelength and FWHM information is set and data is scaled according to data gain/offset values.
Parameters
Metadata file [file]
The metadata XML file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata XML file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Prepare a spectral raster layer from the given product. Wavelength and FWHM information is set and data is scaled according to data gain/offset values.
Parameters
Metadata file [file]
The metadata XML file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata XML file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Prepare a spectral raster layer from the given product. Wavelength and FWHM information is set and data is scaled into the 0 to 1 range.
Parameters
Metadata file [file]
The metadata XML file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata XML file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Prepare a spectral raster layer from the given product. Wavelength information is set and data is scaled into the 0 to 1 range.Supports Landsat 4 to 9, collection 2.
Parameters
Metadata file [file]
The *.MTL.txt metadata file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata MTL.txt file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Import PRISMA L1 product from HE5 file to QGIS/GDAL conform GTiff/VRT file format.Note that for the spectral cube and error matrix, the interleave is transposed and stored as GTiff to enable proper visualization in QGIS.All other sub-datasets are stored as light-weight VRT files.
For further details visit the PRISMA Documentation Area.
Parameters
File [file]
The HE5 product file.
The main data contained in the PRS_L1_HRC Swath is the Radiometric Calibrated Coregistersed Hyperspectral Cube. All bands of VNIR Cube and all bands of SWIR Cube are keystone corrected with respect to VNIR Cube band 128 only considering shift Across track.
The PRS_L1_PCO Swath basically contains the Radiometric Calibrated Coregistered Panchromatic Image. The PAN Cube is coregistered with respect to VNIR Cube taking into account of the Along Track coregistration. PAN Cube also takes into account the Across track offset PAN – VNIR. All pixel of PAN Cube are keystone corrected with respect to VNIR Cube band 128 only considering shift Across track.
Instead of executing this algorithm, you may drag&drop the HE5 file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Import PRISMA L2B product from HE5 file to QGIS/GDAL conform GTiff/VRT file format.Note that for the spectral cube and error matrix, the interleave is transposed and stored as GTiff to enable proper visualization in QGIS.All other sub-datasets are stored as light-weight VRT files.
For further details visit the PRISMA Documentation Area.
Parameters
File [file]
The HE5 product file.
The main data contained in the PRS_L2B_HCO Swath is the surface spectral radiance Coregistersed Hyperspectral Cube (in instrument geometric reference).
The main data contained in the PRS_L2B_PCO Swath is the surface panchromatic reflectance image (in instrument geometric reference).
Instead of executing this algorithm, you may drag&drop the HE5 file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Import PRISMA L2C product from HE5 file to QGIS/GDAL conform GTiff/VRT file format.Note that for the spectral cube and error matrix, the interleave is transposed and stored as GTiff to enable proper visualization in QGIS.All other sub-datasets are stored as light-weight VRT files.
For further details visit the PRISMA Documentation Area.
Parameters
File [file]
The HE5 product file.
The main data contained in the PRS_L2C_HRO Swath is the surface spectral reflectance Coregistersed Hyperspectral Cube (in instrument geometric reference).
The main data contained in the PRS_L2C_PCO Swath is the surface panchromatic reflectance image (in instrument geometric reference).
Instead of executing this algorithm, you may drag&drop the HE5 file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Import PRISMA L2D product from HE5 file to QGIS/GDAL conform GTiff/VRT file format with proper coordinate reference system.Note that for the spectral cube and error matrix, the interleave is transposed and stored as GTiff to enable proper visualization in QGIS.All other sub-datasets are stored as light-weight VRT files.
For further details visit the PRISMA Documentation Area.
Parameters
File [file]
The HE5 product file.
The main data contained in the PRS_L2d_HCO Swath is the surface spectral reflectance Coregistersed Hyperspectral Cube (in instrument geometric reference).
The main data contained in the PRS_L2d_PCO Swath is the surface panchromatic reflectance image (in instrument geometric reference).
Instead of executing this algorithm, you may drag&drop the HE5 file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Spectral region [enum]
Spectral region to be imported.
Default: 0
Bad band threshold [number]
If the proportion of erroneous pixels in the VNIR/SWIR Pixel Error Matrix,exceeds the bad band threshold (a value between 0 and 1), the band is marked as a bad band.
If specified, Output VNIR/SWIR Error Matrix raster layer needs to be specified as well.
Prepare a spectral raster layer from the given product. Wavelength information is set and data is scaled into the 0 to 10000 range.
Parameters
Metadata file [file]
The MTD_MSIL2A.xml metadata file associated with the product.
Instead of executing this algorithm, you may drag&drop the metadata file directly from your system file browser a) onto the EnMAP-Box map view area, or b) onto the Sensor Product Import panel.
Band list [enum]
Bands to be stacked together. Defaults to all 10m and 20m bands ordered by center wavelength. Note that the destination pixel size matches the smallest/finest pixel size over all selected bands.
Python code defining the evaluation function. The defined function must return a binary-valued array with same shape as the input array.
Default:
importnumpyasnpdeffunction(array:np.ndarray,noDataValue:float):# if source no data value is not defined, use zero as no data valueifnoDataValueisNone:noDataValue=0# mask no data pixelmarray=np.not_equal(array,noDataValue)# mask inf and nan pixelmarray[np.logical_not(np.isfinite(array))]=0# include further masking criteria herepassreturnmarray
Outputs
Output raster layer [rasterDestination]
Raster file destination.
Command-line usage
>qgis_processhelpenmapbox:CreateMaskRasterLayer:
----------------Arguments----------------raster:RasterlayerArgumenttype:rasterAcceptablevalues:-Pathtoarasterlayerfunction:FunctionDefaultvalue:importnumpyasnpdeffunction(array:np.ndarray,noDataValue:float):# if source no data value is not defined, use zero as no data valueifnoDataValueisNone:noDataValue=0# mask no data pixelmarray=np.not_equal(array,noDataValue)# mask inf and nan pixelmarray[np.logical_not(np.isfinite(array))]=0# include further masking criteria herepassreturnmarrayArgumenttype:stringAcceptablevalues:-StringvalueoutputRaster:OutputrasterlayerArgumenttype:rasterDestinationAcceptablevalues:-Pathfornewrasterlayer----------------Outputs----------------outputRaster:<outputRaster>Outputrasterlayer
Create a stack of Awesome Spectral Indices and/or custom indices.
Credits: the Awesome Spectral Indices project provides a ready-to-use curated list of Spectral Indices for Remote Sensing applications, maintained by David Montero Loaiza.
Note that the list of available indices was last updated on 2021-11-15. Should you need other indices added after this date, please file an issue.
The list of indices to be created. Usage examples:
Create (predefined) NDVI: <code>NDVI</code>
Create stack of NDVI and EVI: <code>NDVI, EVI</code>
Create custom index: <code>MyNDVI = (N - R) / (N + R)</code>
See the full list of predefined Awesome Spectral Indices.
Default: NDVI
Scale factor [number]
Spectral reflectance scale factor. Some indices require data to be scaled into the 0 to 1 range. If your data is scaled differently, specify an appropriate scale factor.E.g. for Int16 data scaled into the 0 to 10000 range, use a value of 10000.
Perform mathematical calculations on raster layer and vector layer data. Use any NumPy-based arithmetic, or even arbitrary Python code.
See the User Manual for detailed usage instructions.
Parameters
Code [string]
The mathematical calculation to be performed on the selected input arrays.
Select inputs in the available data sources section or use the raster layer R1, …, R10 and vector layer V1, …, V10.
In the code snippets section you can find some prepdefined code snippets ready to use.
See the User Manual for detailed usage instructions.
Grid [raster]
The destination grid. If not specified, the grid of the first raster layer is used.
32-bit floating-point inputs [boolean]
Whether to cast inputs to 32-bit floating point.
Default: True
No data value [number]
Specified value is used as no data value for all inputs. If not specified, the original input no data values are used.
Block overlap [number]
The number of columns and rows to read from the neighbouring blocks. Needs to be specified only when performing spatial operations, to avoid artifacts at block borders.
Monolithic processing [boolean]
Whether to read all data for the full extent at once, instead of block-wise processing. This may be useful for some spatially unbound operations, like segmentation or region growing, when calculating global statistics, or if RAM is not an issue at all.
Default: False
Raster layers mapped to RS [multilayer]
Additional list of raster layers mapped to a list variable RS.
Outputs
Output raster layer [rasterDestination]
Raster file destination for writing the default output variable. Additional outputs are written into the same directory. See the User Manual for detailed usage instructions.
Creates a new point layer with the same attributes of the input layer and the raster values corresponding to the pixels covered by polygons or point location.
The resulting point vector contains 1) all input attributes from the Locations vector, 2) attributes SAMPLE_{i}, one for each input raster band, 3) two attributes PIXEL_X, PIXEL_Y for storing the raster pixel locations (zero-based),and 4), in case of polygon locations, an attribute COVER for storing the pixel coverage (%).
Note that we assume non-overlapping feature geometries! In case of overlapping geometries, split the Locations layer into non-overlapping subsets, perform the sampling for each subset individually, and finally concatenate the results.
The mathematical calculation to be performed on the selected input bands in_ar.Result must be copied to out_ar.
For detailed usage information read the VRT Python Pixel Function docs.
The number of columns and rows to read from the neighbouring blocks. Needs to be specified only when performing spatial operations, to avoid artifacts at block borders.
Create RGB image from class probability/fraction layer
Create an RGB image from a class fraction layer or class probability layer.The RGB pixel color of a single pixel is given by the weighted mean of the given categorycolors.The weights are given by class fractions/probabilities (i.e. values between 0 and 1).
For example, pure pixels with cover fractions of 1 appear in its pure category color. A mixed-pixel with a 50% fractions in two categories colored in red and green,appears in a dull yellow ( 0.5 x (255, 0, 0) + 0.5 x (0, 255, 0) = (127, 127, 0) ).
Comma separated list of hex-color strings (e.g. ‘#FF0000’ for red) representing (pure) categorycolors, one color for each band in the given class probability/fraction layer. If not specified, colors have to be specified by a categorized layer (Colors from categorized layer).
Colors from categorized layer [layer]
A categorized layer with (pure) categorycolors, one category for each band in the given class probability/fraction layer. If not specified, colors have to be specified by list (Colors).
Translates categorized raster layer into target grid.
Resampling is done via a two-step majority voting approach. First, the categorized raster layer is resampled at x10 finer resolution, and subsequently aggregated back to the target resolution using majority voting. This approach leads to pixel-wise class decisions that are accurate to the percent.
Whether to use majority voting. Turn off to use simple nearest neighbour resampling, which is much faster, but may result in highly inaccurate decisions.
Convert raster data between different formats, potentially performing some operations like spatial subsetting, spatial resampling, reprojection, band subsetting, band reordering, data scaling, no data value specification, and data type conversion.
Spatial extent for clipping the destination grid, which is given by the source Raster or the selected Grid. In both cases, the extent is aligned with the actual pixel grid to avoid subpixel shifts.
Write/update the ENVI *.hdr header file to enable full compatibility to the ENVI software. The header file stores wavelength, FWHM and bad band multiplier (BBL) information.
Stack raster layers and store the result as a VRT file.This is a slimmed down version of the more powerful/complicated GDAL “Build virtual raster” algorithm.
If you also want to delete or rearrange individual bands, just use the “Subset raster layer bands” algorithm afterwards.
Kernel ridge regression.
Kernel ridge regression (KRR) combines ridge regression (linear least squares with l2-norm regularization) with the kernel trick. It thus learns a linear function in the space induced by the respective kernel and the data. For non-linear kernels, this corresponds to a non-linear function in the original space.
The form of the model learned by KRR is identical to support vector regression (SVR). However, different loss functions are used: KRR uses squared error loss while support vector regression uses epsilon-insensitive loss, both combined with l2 regularization. In contrast to SVR, fitting a KRR model can be done in closed-form and is typically faster for medium-sized datasets. On the other hand, the learned model is non-sparse and thus slower than SVR, which learns a sparse model for epsilon > 0, at prediction-time.
Parameters
Regressor [string]
Scikit-learn python code. See KernelRidge for information on different parameters.
Ordinary least squares Linear Regression.
LinearRegression fits a linear model with coefficients w = (w1, …, wp) to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation.
Parameters
Regressor [string]
Scikit-learn python code. See LinearRegression for information on different parameters.
Linear Support Vector Regression.
Similar to SVR with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.
Parameters
Regressor [string]
Scikit-learn python code. See LinearSVR for information on different parameters.
A random forest regressor.
A random forest is a meta estimator that fits a number of classifying decision trees on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled with the max_samples parameter if bootstrap=True (default), otherwise the whole dataset is used to build each tree.
Parameters
Regressor [string]
Scikit-learn python code. See RandomForestRegressor for information on different parameters.
Epsilon-Support Vector Regression.
The free parameters in the model are C and epsilon.
The implementation is based on libsvm. The fit time complexity is more than quadratic with the number of samples which makes it hard to scale to datasets with more than a couple of 10000 samples.
Epsilon-Support Vector Regression.
The free parameters in the model are C and epsilon.
The implementation is based on libsvm. The fit time complexity is more than quadratic with the number of samples which makes it hard to scale to datasets with more than a couple of 10000 samples.
Receiver operating characteristic (ROC) and detection error tradeoff (DET) curves
Compute receiver operating characteristic (ROC) and detection error tradeoff (DET) curves.
For more details see the Scikit-Learn user guide: Receiver operating characteristic (ROC) and Detection error tradeoff (DET).
Note that observed classes and predicted class probabilities are matched by name.
Estimates map accuracy.We use the formulas as described in Scikit-Learn Regression metrics user guide. Observed and predicted target variables are matched by name.
A raster layer with bands used as features for mapping. Regressor features and raster bands are matched by name. Will be ignored, if map prediction is skipped.
Match regressor features and raster bands by name [boolean]
The number of folds used for assessing cross-validation performance. Will be ignored, if the cross-validation performance assessment is skipped.
Default: 10
Open output cross-validation regressor performance report in webbrowser after running algorithm [boolean]
Whether to open the cross-validation performance report in the web browser. Will be ignored, if the cross-validation performance assessment is skipped.
fromcollectionsimportOrderedDict# Place response function definition here. Should look something like this:responses=OrderedDict()# Option 1: Use band name as dict-key,# together with fully defined list of (<wavelength>, <response>) tuples as dict-value.# Define <response> as values between 0 and 1 for each whole nanometer <wavelength>.responses['Blue']=[(443,0.001),(444,0.002),...,(509,1.0),...,(518,0.002),(519,0.001)]responses['Green']=[(519,0.001),(520,0.002),...,(550,1.0),...,(597,0.003),(598,0.001)]...# Option 2: Use band center wavelength as dict-key,# together with band full width at half maximum (FWHM) as dict-value.# The response function is modelled as an RBF around the center wavelength with a width of FWHM / 2.355.# See https://en.wikipedia.org/wiki/Full_width_at_half_maximum for details.responses[460]=5.8...responses[2409]=9.1
Outputs
Output spectral response function library [fileDestination]
----------------Arguments----------------raster:SpectralrasterlayerArgumenttype:rasterAcceptablevalues:-Pathtoarasterlayerresponse:SpectralresponsefunctionDefaultvalue:fromcollectionsimportOrderedDict# Place response function definition here. Should look something like this:responses=OrderedDict()# Option 1: Use band name as dict-key,# together with fully defined list of (<wavelength>, <response>) tuples as dict-value.# Define <response> as values between 0 and 1 for each whole nanometer <wavelength>.responses['Blue']=[(443,0.001),(444,0.002),...,(509,1.0),...,(518,0.002),(519,0.001)]responses['Green']=[(519,0.001),(520,0.002),...,(550,1.0),...,(597,0.003),(598,0.001)]...# Option 2: Use band center wavelength as dict-key,# together with band full width at half maximum (FWHM) as dict-value.# The response function is modelled as an RBF around the center wavelength with a width of FWHM / 2.355.# See https://en.wikipedia.org/wiki/Full_width_at_half_maximum for details.responses[460]=5.8...responses[2409]=9.1Argumenttype:stringAcceptablevalues:-StringvalueoutputResponseFunctionLibrary:Outputspectralresponsefunctionlibrary(optional)Argumenttype:fileDestinationAcceptablevalues:-PathfornewfileoutputResampledRaster:OutputrasterlayerArgumenttype:rasterDestinationAcceptablevalues:-Pathfornewrasterlayer----------------Outputs----------------outputResponseFunctionLibrary:<outputFile>OutputspectralresponsefunctionlibraryoutputResampledRaster:<outputRaster>Outputrasterlayer
A file with center wavelength information defining the destination sensor. Possible inputs are i) raster files, ii) ENVI Spectral Library files, iii) ENVI Header files, and iv) CSV table files with wavelength column.
A file with center wavelength and FWHM information defining the destination sensor. Possible inputs are i) raster files, ii) ENVI Spectral Library files, iii) ENVI Header files, and iv) CSV table files with wavelength and fwhm columns.
Outputs
Output spectral response function library [fileDestination]
Factor Analysis.
A simple linear generative model with Gaussian latent variables.
The observations are assumed to be caused by a linear transformation of lower dimensional latent factors and added Gaussian noise. Without loss of generality the factors are distributed according to a Gaussian with zero mean and unit covariance. The noise is also zero mean and has an arbitrary diagonal covariance matrix.
If we would restrict the model further, by assuming that the Gaussian noise is even isotropic (all diagonal entries are the same) we would obtain ProbabilisticPCA.
FactorAnalysis performs a maximum likelihood estimate of the so-called loading matrix, the transformation of the latent variables to the observed ones, using SVD based approach.
Parameters
Transformer [string]
Scikit-learn python code. See FactorAnalysis for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Scale each feature by its maximum absolute value.
This estimator scales and translates each feature individually such that the maximal absolute value of each feature in the training set will be 1.0.
Parameters
Transformer [string]
Scikit-learn python code. See MaxAbsScaler for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Transform features by scaling each feature to a given range.
This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one.
Parameters
Transformer [string]
Scikit-learn python code. See MinMaxScaler for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Normalize samples individually to unit norm.
Each sample (i.e. each row of the data matrix) with at least one non zero component is rescaled independently of other samples so that its norm (l1, l2 or inf) equals one.
Parameters
Transformer [string]
Scikit-learn python code. See Normalizer for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Principal component analysis (PCA).
Linear dimensionality reduction using Singular Value Decomposition of the data to project it to a lower dimensional space. The input data is centered but not scaled for each feature before applying the SVD.
Parameters
Transformer [string]
Scikit-learn python code. See PCA for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Transform features using quantiles information.
This method transforms the features to follow a uniform or a normal distribution. Therefore, for a given feature, this transformation tends to spread out the most frequent values. It also reduces the impact of (marginal) outliers: this is therefore a robust preprocessing scheme.
The transformation is applied on each feature independently. First an estimate of the cumulative distribution function of a feature is used to map the original values to a uniform distribution. The obtained values are then mapped to the desired output distribution using the associated quantile function. Features values of new/unseen data that fall below or above the fitted range will be mapped to the bounds of the output distribution. Note that this transform is non-linear. It may distort linear correlations between variables measured at the same scale but renders variables measured at different scales more directly comparable.
Parameters
Transformer [string]
Scikit-learn python code. See QuantileTransformer for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Scale features using statistics that are robust to outliers.
This Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile).
Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Median and interquartile range are then stored to be used on later data using the transform method.
Standardization of a dataset is a common requirement for many machine learning estimators. Typically this is done by removing the mean and scaling to unit variance. However, outliers can often influence the sample mean / variance in a negative way. In such cases, the median and the interquartile range often give better results.
Parameters
Transformer [string]
Scikit-learn python code. See RobustScaler for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Standardize features by removing the mean and scaling to unit variance.
The standard score of a sample x is calculated as:
z = (x - u) / s
where u is the mean of the training samples or zero if with_mean=False, and s is the standard deviation of the training samples or one if with_std=False.
Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Mean and standard deviation are then stored to be used on later data using transform.
Standardization of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual features do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance).
For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.
Parameters
Transformer [string]
Scikit-learn python code. See StandardScaler for information on different parameters.
Approximate number of samples drawn from raster. If 0, whole raster will be used. Note that this is only a hint for limiting the number of rows and columns.
Whether to include the original library spectra into the dataset.
Default: True
Mixing complexity probabilities [string]
A list of probabilities for using 2, 3, 4, … endmember mixing models. Trailing 0 probabilities can be skipped. The default values of 0.5, 0.5,result in 50% 2-endmember and 50% 3-endmember models.
Default: 0.5, 0.5
Allow within-class mixtures [boolean]
Whether to allow mixtures with profiles belonging to the same class.
Default: True
Class probabilities [string]
A list of probabilities for drawing profiles from each class. If not specified, class probabilities are proportional to the class size.
Ensemble size [number]
Number of individual runs/predictions.
Default: 1
Robust decision fusion [boolean]
Whether to use median and IQR (interquartile range) aggregation for ensemble decision fusion. The default is to use mean and standard deviation.
Default: False
Sum-to-one constraint [boolean]
Whether to ensure sum-to-one constraint for predicted fractions.
Default: False
Outputs
Output class fraction layer [rasterDestination]
Raster file destination.
Output classification layer [rasterDestination]
Raster file destination.
Output class fraction variation layer [rasterDestination]
Rasterize a categorized vector layer into a categorized raster layer. Output category names and colors are given by the source layer.
Resampling is done via a two-step majority voting approach. First, the categorized raster layer is resampled at x10 finer resolution, and subsequently aggregated back to the target resolution using majority voting. This approach leads to pixel-wise class decisions that are accurate to the percent.
Exclude all pixel where (polygon) coverage is smaller than given threshold.
Default: 50
Majority voting [boolean]
Whether to use majority voting. Turn off to use simple nearest neighbour resampling, which is much faster, but may result in highly inaccurate class decisions.
Whether to use the feature ID as burn values. Initial value is set to -1. Data type is set to Int32.
Default: False
Add value [boolean]
Whether to add up existing values instead of replacing them.
Default: False
All touched [boolean]
Enables the ALL_TOUCHED rasterization option so that all pixels touched by lines or polygons will be updated, not just those on the line render path, or whose center point is within the polygon.
Number of points to draw from each category. Set a single value N to draw N points for each category. Set a list of values N1, N2, … Ni, … to draw Ni points for category i.
Minimum distance between points (in meters) [number]
A minimum (Euclidean) distance between points can be specified.
Default: 0
Minimum distance between points inside category (in meters) [number]
A minimum (Euclidean) distance between points in a category can be specified.
Default: 0
Random seed [number]
The seed for the random generator can be provided.
Minimum distance between points (in meters) [number]
A minimum distance between points can be specified. A point will not be added if there is an already generated point within this (Euclidean) distance from the generated location.
Default: 0
Random seed [number]
The seed for the random generator can be provided.
The band to draw points from. If not selected, the first renderer band is used.
Number of points per value-range [matrix]
Number of points N to draw from value-range Minimum to Maximum.Value-ranges can be specified by actual values (e.g. 42) or by percentiles (e.g. p0, p50, p100, etc.).
Range boundaries [enum]
The boundary type used for all value-ranges.
Default: 0
Minimum distance between points (in meters) [number]
A minimum (Euclidean) distance between points can be specified.
Default: 0
Minimum distance between points inside category (in meters) [number]
A minimum (Euclidean) distance between points in a value-range can be specified.
Default: 0
Random seed [number]
The seed for the random generator can be provided.
The metadata guide explaines how the EnMAP-Box uses raster metadata.
In general, metadata is used to annotate the actual raster data with additional information to allow tools and
algorithms to make better use of them.
E.g. we need the center wavelength of each band in an optical image to properly plot spectral profiles,
and we need the aquisition datetime of each band in a temporal raster stack to properly plot temporal profiles.
The EnMAP-Box supports multiple levels of raster metadata specifications.
If a metadata item is requested, we look for it inside the following places and sequential order:
Metadata stored inside a STAC *.stac.json sidecar file (e.g. myRaster.bsq.stac.json).
Metadata stored inside a GDAL source file (e.g. myRaster.hdr).
Note that metadata items can be specified in multiple places, with potentially conflicting values.
It is the responsibility of the user, to make sure, that lower-level metadata
(e.g. metadata inside an ENVI *.hdr header file), isn’t shadowed by higher-level specifications (e.g. GDAL PAM or STAC).
The EnMAP-Box supports metadata stored inside a STAC (SpatioTemporal Asset Catalogs) *.stac.json sidecar file.
See the STAC specification for more details: https://stacspec.org/en
For special metadata items, not already covered by an official STAC extensions, we introduce new items with our custom
enmapbox: field name prefix, e.g. enmapbox:bad_band_multiplier
For a hyperspectral raster, the STAC metadata looks like this:
{"stac_extensions":["https://stac-extensions.github.io/eo/v1.0.0/schema.json",# Electro-Optical Extension Specification: https://github.com/stac-extensions/eo"https://stac-extensions.github.io/timestamps/v1.1.0/schema.json"# Timestamps Extension Specification: https://github.com/stac-extensions/timestamps],"properties":{"datetime":"2022-07-24T10:45:26",# Scene acquisition datetime","eo:bands":[# List of band-specific metadata items{# First band"name":"band 1 (418.24 Nanometers)",# Band name (used by tools/algorithms, but not displayed by QGIS!)"center_wavelength":0.41824,# Center wavelength in micrometers"full_width_half_max":0.00699561,# Full width at half maximum in micrometers"enmapbox:bad_band_multiplier":1# Whether a band is a good band (1) or a bad band (0)},{# Second band"name":"band 2 (423.874 Nanometers)","center_wavelength":0.423874,"full_width_half_max":0.006667,"enmapbox:bad_band_multiplier":1},...,{# Last band"name":"band 224 (2445.53 Nanometers)","center_wavelength":2.44553,"full_width_half_max":0.0071581,"enmapbox:bad_band_multiplier":1}]}}
For a 3D temporal band stack (e.g. NDVI bands over time), the STAC metadata looks like this:
{"properties":{"eo:bands":[# List of band-specific metadata items{# First band"name":"NDVI (2022-07-24)","datetime":"2022-07-24T10:45:26"},{# Second band"name":"NDVI (2022-08-05)","datetime":"2022-08-05T10:42:12"},...,]}}
For a 4D temporal band stack with multiple variables over time (e.g. NDVI, EVI bands), the STAC metadata looks like this:
{"properties":{"eo:bands":[# List of band-specific metadata items{# First band (first variable, first date)"name":"NDVI (2022-07-24)","datetime":"2022-07-24T10:45:26"},{# Second band (second variable, first date)"name":"EVI (2022-07-24)","datetime":"2022-07-24T10:45:26"},{# Third band (first variable, second date)"name":"NDVI (2022-08-05)","datetime":"2022-08-05T10:42:12"},{# Fourth band (second variable, second date)"name":"EVI (2022-08-05)","datetime":"2022-08-05T10:42:12"},...,]}}
If the temporal data represents an aggregation of multiple observations inside a temporal range
(e.g. cloud-free composites for individual years), use the start_datetime and end_datetime keys,
instead of datetime:
In case of a 4D spectral-temporal band stack (e.g. yearly Landsat composites over time), use spectral and temporal
metadata together, to fully describe the dataset:
As an alternative to the above described specifications, we also support a more concise ENVI-like specification,
using the well known ENVI Header File
metadata items. This should allow for simple copy&paste approaches,
when moving metadata from an ENVI Header file to a STAC file:
The EnMAP-Box supports metadata stored inside a GDAL PAM (Persistent Auxiliary Metadata) *.aux.xml sidecar file.
For a hyperspectral raster, the GDAL PAM metadata looks like this:
<PAMDataset><PAMRasterBandband="1"># First band<Description>band8(0.460000Micrometers)</Description># Band name<Metadata><MDIkey="wavelength">0.460000</MDI># Center wavelength<MDIkey="fwhm">0.058</MDI># Full width at half maximum<MDIkey="wavelength_units">Micrometers</MDI># Wavelength units<MDIkey="bbl">1</MDI># Whether a band is a good band (1) or a bad band (0)</Metadata></PAMRasterBand><PAMRasterBandband="2"># Second band...</PAMRasterBand>...</PAMDataset>
For a 3D temporal band stack (e.g. NDVI bands over time), the GDAL PAM metadata looks like this:
<PAMDataset><PAMRasterBandband="1"># First band<Description>NDVI(2022-07-24)</Description><Metadata><MDIkey="start_time">2022-07-24T10:45:26</MDI></Metadata></PAMRasterBand><PAMRasterBandband="2"># Second band<Description>NDVI(2022-08-05)</Description><Metadata><MDIkey="start_time">2022-08-05T10:42:12</MDI></Metadata></PAMRasterBand>...</PAMDataset>
For a 4D temporal band stack with multiple variables over time (e.g. NDVI, EVI bands), the GDAL PAM metadata looks like this:
<PAMDataset><PAMRasterBandband="1"># First band (first variable, first date)<Description>NDVI(2022-07-24)</Description><Metadata><MDIkey="start_time">2022-07-24T10:45:26</MDI></Metadata></PAMRasterBand><PAMRasterBandband="2"># Second band (second variable, first date)<Description>EVI(2022-07-24)</Description><Metadata><MDIkey="start_time">2022-07-24T10:45:26</MDI></Metadata></PAMRasterBand><PAMRasterBandband="3"># Third band (first variable, second date)<Description>NDVI(2022-08-05)</Description><Metadata><MDIkey="start_time">2022-08-05T10:42:12</MDI></Metadata></PAMRasterBand><PAMRasterBandband="4"># Fourth band (second variable, second date)<Description>EVI(2022-08-05)</Description><Metadata><MDIkey="start_time">2022-08-05T10:42:12</MDI></Metadata></PAMRasterBand>...</PAMDataset>
If the temporal data represents an aggregation of multiple observations inside a temporal range
(e.g. cloud-free composites for individual years), use the start_datetime and end_datetime keys,
instead of datetime:
In case of a 4D spectral-temporal band stack (e.g. yearly Landsat composites over time), use spectral and temporal
metadata together, to fully describe the dataset:
As an alternative to the above described specifications, we also support a more concise ENVI-like specification,
using the well known ENVI Header File
metadata items. This should allow for simple copy&paste approaches,
when moving metadata from an ENVI Header file to a GDAL PAM file:
An e-mail is send to your address containing a verification code.
After the successful registration, login to the Instrument Planning Portal and enter the User Portal
In the User Portal, the user has to go through the Role Assignment procedure which is used to assign different priorities to observation requests.
For more information about the different User Roles please take a look at the User Manual.
To access the data archive, request Catalogue (Cat1-Distributor)
To request data takes / submit proposals for data takes, request Cat-1
Wait until the respective role is assigned. Note: This will take a few hours.
Once the role has been assigned, a new box will appear in the User Portal, allowing you tp access the EOWEB ® GeoPortal to search the EnMAP Data archive and order images.
The EOWEB ® GeoPortal access link is created for Cat-1 users after the requested role has been approved by the reviewer.
The login button appears on the User Portal page, which automatically directs to the EOWEB ® GeoPortal login page.
Login with the respective User ID associated with the approved role request. The password remains the same as when logging into the Instrument Planning Portal.
Once logged in, click Show advanced map to zoom, pan and draw a rectangle.
Select EnMAP in Filter Collection and confirm with Search.
Note
You may also want to check the EnMAP-HSI (LO), Low Quality box to see additional records that are marked as low quality, but are still good.
Hide Show advanced map to see the results.
Place your order and wait until you get notified.
Step 3: Download data from the FTP Delivery Server
After the order is placed, an e-mail is sent including a link to the FTP server and the zipped data.
Note
The User ID and password to access the FTP server are identical to the EOWEB ® GeoPortal credentials.
To download the data use a FTP client, for example FileZilla. For further help on how to download the data from the FTP server take
a look at Downloading Ordered Data.
This tutorial is part of the HYPERedu online learning platform,
an education initiative within the EnMAP mission hosted
on EO College. HYPERedu provides annotated slide collections and hands-on tutorials using the open-source EnMAP-Box software,
targeting basic principles, methods as well as applications of imaging spectroscopy.
Annotated slide collections for the tutorial Regression-based unmixing of urban land cover and a software description unit for the EnMAP-Box are provided here:
Land cover fraction mapping based on unmixing is well suited to describe the composition of surface
types in heterogeneous environments. Especially when using coarser spatial resolution satellite data
with a high share of mixed pixels, fraction mapping is more useful than a discrete classification. This,
for example, applies to the use of 30 m resolution imagery from the forthcoming spaceborne imaging
spectrometer mission EnMAP for urban mapping.
This tutorial focuses on regression-based unmixing of urban land cover using synthetically mixed
training data from spectral libraries. Hyperspectral images from the airborne HyMap and the
spaceborne EnMAP sensor (here simulated from HyMap), a corresponding spectral library, and
reference land cover information are used for different exercises within this tutorial. These aim at
providing both a theoretical background related to challenges in urban mapping and a hands-on
training for working with the EnMAP-Box.
This tutorial requires at least version 3.6 of the EnMAP-Box 3.
There might be some minor changes for higher versions (e.g., changed menu labels, added parameter options, etc.).
We recommend [1] for a comprehensive overview on imaging spectroscopy of urban environments
and [2] & [3] for detailed introductions into the regression-based unmixing approach using synthetically
mixed training data from spectral libraries.
The tutorial dataset contains an EnMAP-image (simulation) covering an area along the urban gradient of Berlin, Germany,
a second hyperspectral image at higher spatial resolution (HyMap, 3.6 m), a spectral library and detailed land cover
reference information.
Data type
Filename
Description
Raster
hymap_berlin.bsq
Airborne hyperspectral data from the HyMap sensor with a spatial resolution of 3.6m, 111 bands and 346x3200 pixels (ENVI Standard Band Sequential bsq)
Raster
enmap_berlin.bsq
Spaceborne hyperspectral data from the EnMAP sensor (here simulated from HyMAP) with a spatial resolution of 30m, 177 bands and 220x400 pixels (ENVI Standard Band Sequential bsq)
Spectral library
library_berlin.sli
Urban spectral library with 75 pure surface materials categorized in a hierarchical class scheme. The Library was developed from the HyMap image and spectrally resampled to the EnMAP sensor (ENVI spectral library sli with metadata extensions csv & json)
Vector
landcover_berlin.shp
Detailed land cover reference information categorized in a hierarchical class scheme (ESRI Shapefile shp with QGIS layer style file qml and metadata extension json)
The tutorial dataset is a subset extracted from the Berlin-Urban-Gradient dataset [4].
Please cite the dataset as follows:
Airborne imaging spectroscopy data is well suited for urban mapping. The high spectral and spatial resolution
enhances the separability of surface types and preserves the spatial detail of many urban features. This exercise…
provides an insight into how urban areas are depicted by airborne hyperspectral images and introduces a hierarchical
classification scheme commonly adopted for urban mapping
introduces basic functionalities of the EnMAP-Box. You will get to know the graphical user interface, and
you will learn how to load data, visualize raster and vector data, and use the basic navigation tools
Start QGIS and click the icon in the toolbar to open the EnMAP-Box. The GUI of the EnMAP-Box consists of a Menu
and a Toolbar, panels for Data Sources and Data Views, and the QGIS Processing Toolbox including the EnMAP-Box geoalgorithms.
The EnMAP-Box offers simple drag & drop capabilities to load data from an external explorer.
Drag the following datasets from your explorer into the Data Sources panel:
The EnMAP-Box offers Map Windows (Map #) for visualizing raster and vector data. Click the icon
and drag the following datasets from the Data Sources panel into Map #1:
hymap_berlin.bsq
landcover_berlin.shp
Map #1 now appears in the Data Views panel, where the visibility, order and properties of datasets can be modified. Unfold Map #1:
To change the order of stacked layers, drag one layer on top or below another one. Arrange the layer stack so that
landcover_berlin.shp is displayed on top of hymap_berlin.bsq.
To assign a multibandcolor RGB combination to a raster image, right click on the dataset, select Layer Properties and
navigate to Band selection. You can now select predefined composites (RGB, nIR, swIR), or
manually select your bands using the slider or dropdown lists. Display hymap_berlin.bsq as true color composite,
by selecting R-G-B.
You can specify things like contrast enhancement and data stretch under Layer properties ‣ Symbology.
The symbology of landcover_berlin.shp is predefined by a QGIS layer style file (.qml). To change this symbology,
right click on the vector layer, select Layer Properties and navigate to Symbology in the LayerProperties window.
You can now change the symbology in accordance to the QGIS functionality. Use the Column and Classify options to explore the
information content of the attribute table landcover_berlin.shp.
The Toolbar offers common navigation tools for exploring visualized datasets. Make yourself familiar with the following navigation tools:
Note that the mouse wheel can be used alternatively for zooming (roll mouse wheel forward/backward) and panning (press and hold mouse wheel).
Make yourself familiar with the crosshair functionality. To show/hide the crosshair, to change the style, or to display the pixel cell
of a selected layer, right click within MAP #1 and select Crosshair.
Learning activities:
A1: Visually explore the airborne hyperspectral image (hymap_berlin.bsq). What major land cover types do you observe along Berlin’s urban-gradient?
click to expand...
Major land cover types: buildings/roofs, paved areas (e.g., streets, backyards),
trees (e.g., park trees, street trees), grass (e.g., lawns, soccer field), crops (on agricultural sites),
bare soil (e.g., agricultural sites, construction sites), and water (e.g., lakes, swimming pools).
A2: Explore the land cover reference data (landcover_berlin.shp) and draw a flowchart of the hierarchical classification scheme stored in the attribute table.
Forthcoming spaceborne imaging spectroscopy missions create new opportunities for global urban mapping. However,
the step to satellite observations brings with it coarser spatial resolution, resulting in a loss in spatial detail and
an increase in the number of mixed pixels. This exercise…
provides an insight into how urban areas will be depicted by spaceborne hyperspectral images and illustrates
challenges related to spectral mixing when using such data for urban mapping
introduces additional basic functionalities of the EnMAP-Box. You will learn how to work with multiple map views,
and how to visualize image spectra using Spectral Library Windows
Close Map #1 from the previous exercise by using the icon on the blue Map #1 title bar.
The EnMAP-Box enables users to work with multiple Map Windows, which can be flexibly organized and geospatially linked.
Open two new Map Windows. For horizontal arrangement, click and hold on the blue Map #2 title bar and drag it to the
right edge of Map #1. A transparent blue rectangle appears indicating the docking position once you stop holding the mouse button.
Display hymap_berlin.bsq and enmap_berlin.bsq as RGB composite of your choice in Map #1 and Map #2, respectively.
For geospatial linking, click on View in the Menu and select Set Map Linking. In the Map Linking
window, select the Link Map Scale and Center option and close the dialog.
The EnMAP-Box offers Spectral Library Windows (SpectralLibrary #) for visualizing spectra and handling their
metadata. To visualize image spectra, activate the Identify tool together with the Identify raster profiles option. Click on an image pixel. SpectralLibrary #1 opens automatically displaying the respective
pixel spectrum (green line). The plotted spectrum always refers to the top-most raster layer of the respective Map
Window you click on (unless you change this in the Spectral Profile Sources panel).
Make yourself familiar with the following tools in the Spectral Library #1 toolbar:
The icon adds a plotted spectrum to a collection. Each collected spectrum (white line) gets an entry in the attribute table with a predefined name (filename and id).
By clicking next to the icon you can activate the Add Profiles automatically mode, which will automatically add spectra to the collection when clicking on image pixels.
The icon switches on the editing mode. You can now edit the attribute table (e.g. edit the name of a spectrum), add or delete colums using the icons, etc.
You can delete selected spectra in editing mode using the icon (hold Ctrl or Shift to select multiple rows).
The icon saves a spectrum or a collection of spectra as a spectral library. Further information on spectral libraries and library formats will follow in Exercise C.
Learning activities
B1: Visually compare the airborne and spaceborne hyperspectral images (hymap_berlin.bsq, enmap_berlin.bsq). How much of the spatial detail is lost when stepping from airborne to spaceborne scale?
click to expand...
The spatial detail of most urban features (e.g., buildings, streets, trees along streets or in private gardens)
disappears due to spatial aggregation at spaceborne scale. However, large homogenous urban features (e.g., waterbodies, sport grounds, tree stand in parks) remain apparent.
B2: Provide an average estimate on the percentage of pixels covered by 1, 2, 3, and 4 or more land cover classes for both images. Use level 3 of the classification scheme for your estimate. You may use the reference land cover information (landcover_berlin.shp) for orientation.
click to expand...
B3: Compare pairs of spectra from the airborne and spaceborne hyperspectral images (hymap_berlin.bsq, enmap_berlin.bsq). For which urban surface materials is it still possible to collect pure spectra at spaceborne scale, and for which not?
click to expand...
Pure spectra can be collected for homogenous urban surfaces with a patch size of ~100 x 100 m
and larger (e.g., roofing material spectra for large industrial buildings, ground paving material spectra for yards
of industrial complexes, grass spectra on lawns or soccer fields, tree spectra in dense stands, water spectra from water bodies).
Pure spectra cannot be collected for urban surfaces with a patch size below ~100 x 100 m (i.e., for most roofing materials, street asphalt, street trees).
Urban spectral libraries are collections of pure surface spectra (endmembers) representing the spectral diversity
and variability of urban land cover types at high spectral resolution. Library spectra are commonly based on laboratory, field, or image data,
and are well suited for library-based mapping approaches such as unmixing. This exercise…
provides an insight into the design of urban spectral libraries and illustrates challenges related to within-class
variability and between-class similarity during urban mapping
targets the handling of spectral libraries in the EnMAP-Box. You will get to know the spectral library format used in the EnMAP-Box,
and learn how to load and visualize external urban spectral libraries and associated metadata
The EnMAP-Box makes use of labeled spectral libraries, which extend the standard ENVI Spectral Library (SLI) and associated header (HDR) by additional attribute information, e.g., class labels, class colors, description, etc. This information is stored in an ASCII Comma Separated Value (CSV) file and in a JavaScript Object Notation (JSON) file. The CSV file includes an attribute table for each library spectrum (the column ‘spectra names’ links the CSV with the HDR). The JSON file contains the unique attributes of given attribute columns.
Open the library_berlin.hdr, library_berlin.csv, and library_berlin.json files with a text editor and get familiar with the spectral library format used in the EnMAP-Box and the hierarchical classification scheme stored in the attribute information.
Close all Map and Spectral Library Windows from the previous exercise.
To load the urban spectral library, click on the icon to open a new Spectral Library Window and drag library_berlin.sli from the Data Sources panel into SpectralLibrary #1. Get familiar with the representation of the spectral library and the attribute table.
By default, only 64 spectra will be displayed at once in the plot window. To change this number, right-click in the plot
area, go to Others ‣ Max. Profiles and enter a higher number, e.g. 75 (total number of spectra in library_berlin.sli)
To display a subset of spectra in a separate Library Window…
Select the spectra of interest by clicking on their corresponding row numbers (use Ctrl or Shift to select multiple rows). To select spectra with the same attributes, prior sorting of the attribute table by clicking on the corresponding column header is recommended. You can also select the spectra directly in the plot window.
Click on the icon in the toolbar (or Ctrl+C) to copy the selected spectra to clipboard.
Open a second Spectra Library Window. Similar to the work with multiple Map Windows, Spectral Library Windows can be arranged according to the user needs.
Switch on the editing mode in the SpectralLibrary #2 toolbar and use the icon (or Ctrl+V) to paste the copied spectra into SpectralLibrary #2. Switch off the editing mode.
Learning activities
C1: Load the urban spectral library (library_berlin.sli) and display each level 3 class in a separate Spectral Library Window. How diverse is each class with regard to within-class variability?
click to expand...
The roof class shows a very high within-class variability. The classes pavement, low vegetation,
and tree show a high within-class variability. The classes soil and water show a rather low within-class variability.
C2: List classes which show a high between-class similarity and provide an explanation.
click to expand...
The classes roof and pavement are highly similar with regard to the following surface materials:
bitumen vs. asphalt, red clay tiles vs. red sand, grey roofing materials (most likely concrete) vs concrete. The classes
roof and soil are highly similar with regard to the following surface materials: concrete vs. bare soil, red clay tiles vs.
bare soil. The classes low vegetation and tree are highly similar regarding the following vegetation types: darker grass types
(clover, agricultural grassland) vs. brighter trees.
To utilize data from forthcoming spaceborne imaging spectrometer missions for mapping the land cover composition of urban areas,
unmixing is more useful than a discrete classification. This exercise…
introduces a regression-based unmixing approach for land cover fraction mapping. The approach successfully copes with
spectral diversity, variably and mixing, and makes use of synthetic mixtures from spectral libraries for regression model training
demonstrates the work with the ‘Regression-based unmixing (synthMix)’ application of the EnMAP-Box
The training of regression models with synthetically mixed data from spectral libraries for land cover fraction mapping is
implemented as the Regression-based unmixing (synthMix) application in the EnMAP-Box 3. The workflow of the unmixing approach comprises the following steps:
Step 1: An endmember library with associated class labels is used to randomly create a synthetically mixed dataset, i.e., pairs of mixed spectra and mixing fractions, for each class.
Step 2: The synthetically mixed dataset is used to train a regression model for each class.
Step 3: The regression model is applied to an image to derive a fraction map for each class.
The approach can be embedded into an ensemble framework, i.e., steps 1-3 are iterated n-times and the final fraction map for each class is created by combining the intermediate maps. The ensemble modeling allows the inclusion of a multitude of different types of synthetic mixtures into the unmixing process while keeping the training sample size low.
Click on Applications in the Menu and select Regression-based unmixing (synthMix). The graphical widget of the Regression-based unmixing (synthMix)
consists of sections for specifying Inputs, for target Class Selection, for setting Mixing Parameters,
for selecting the Regression Algorithm, and for specifying the Outputs.
The specification of input data is the start of the unmixing process. This includes an Endmember Library with associated class labels
(see Exercise 3), where the Class Attribute drop menu specifies the attribute column associated with the class labels, and the Spectral Image to unmix.
Select / specify the following inputs:
Endmember Library: library_berlin.sli
Class Attribute: level_1
Spectral Image: enmap_berlin.bsq
The selection of the classes of interest, i.e., Target Classes, is the next step of the unmixing process. Synthetically
mixed data, regression models, and fraction maps are only created for target classes. Spectra of excluded classes are still
used as background signatures in the synthetic mixing process.
Select the following Target Classes: impervious, vegetation, soil, water (all level 1 classes)
The mixing parameters steer the process of generating the synthetically mixed data from the endmember library. The Number of Synthetic Mixtures per Class
specifies the total number of mixtures per class to be created. The check option to Include Original Library Endmembers
allows to append the endmember library to the synthetically mixed data, with fractions of either 0% or 100% of a respective target class.
The synthetic mixing process itself is randomized. That is, to generate a synthetic mixture…
… a mixing complexity is randomly assigned. The mixing complexity defines the number of endmembers contributing to a mixture (e.g., 2EM, 3EM).
The random selection is steered by user-defined Mixing Complexity Probabilities (e.g., 2EM=0.6, 3EM=0.4 means that there is a 60%
likelihood that the mixture is made up of two endmembers and a 40% likelihood that the mixture is made up of three endmembers).
The implementation allows the definition of probabilities for 2EM, 3EM and 4EM. Note that probabilities must sum up to 1.
… endmembers are randomly drawn from the library. The number of endmembers is based on the previously assigned mixing complexity.
The first endmember is always drawn from the target class. The following endmembers are drawn based on Class Probabilities,
which are either proportional (class proportions within the library) or equalized (all classes with the same likelihood).
The Allow Within-Class Mixtures check option allows the user to decide whether multiple endmembers of the same class can be drawn to create a mixture.
… random mixing fractions between 0 and 1 (0-100%) are randomly assigned to the previously drawn endmembers. The total sum of fractions is always 1 (100%).
… endmembers are linearly mixed based on the mixing fractions to create the mixture.
Select the following mixing parameters:
Number of Synthetic Mixtures per Class: 1000 (default)
Include Original Library Endmembers: Yes (default)
The selection of the regression algorithm and the setting up of the ensemble are the next steps in the unmixing process. The EnMAP-Box makes
use of the scikit-learn library (see https://scikit-learn.org/stable/index.html) to implement several state-of-the-art algorithms offered in the Regressor
drop menu. Note that the different algorithms lead to varying accuracies and processing times,
particularly when embedding the unmixing process into an ensemble. To do so, activate Use Ensemble and set the Ensemble Size.
Select the following regression settings:
Regressor: RandomForestRegression (default, due to the low processing time)
Use Ensemble: Yes (default), Ensemble Size: 3 (default)
The specification of the outputs is the final step in the unmixing process. By default, the final fraction map is the only output:
Folder: Specifies the output folder where results are saved.
Name: Specifies the base name of the final fraction map. We add the suffix ‘_mean’ to this file as the default ensemble decision fusion is based on averaging.
The advanced options allow user to save additional files created during the unmixing process, and to derive additional maps from the final fraction map.
Decision Fusion (Ensemble): Selection of different statistics to evaluate the ensemble output. The following statistics are implemented: mean (default, suffix ‘_mean’), median (suffix ‘_median’), inter quartile range (suffix ‘_iqr’), and standard deviation (suffix ‘_std’).
Save: Check to save Training Samples, Predictions, and Models. These outputs will be stored for each class and with a suffix ‘_run’ for each ensemble iteration in separate subfolders.
Create Class Fraction RGB: Check to create a RGB class representation of the final fraction map. The RGB color of a specific pixel is the weighted mean value of the original class colors, where the weights are given by the corresponding class fractions.
Derive Classification from Fraction Map: Check to derive a discrete classification map from the final fraction map. The winner class per pixel corresponds to the maximum class fraction.
Specify the following outputs (and skip the advanced options):
Display the newly created fraction_level1_estimation.bsq. The file consists of 4 bands, where each band represents a fraction map of the defined target classes. Display the fraction map in a useful render style and appropriate contrast stretch:
e.g., as multibandcolor RGB composite of three target classes in a single Map Window. For stretching fraction maps to the full range of possible fraction, set Min = 0 and Max = 1.
e.g., as singlegray image per target class in multiple Map Windows. For stretching fraction maps to the full range of possible fraction, set Min = 0 and Max = 1.
Visually explore your fraction map. You may open enmap_berlin.bsq in a separate Map Window for comparison. You may use the Identify tool together with the Identify cursor location values option to display fraction values of pixels.
Learning activities
D1: Visually explore the fraction map (fraction_level1_estimation.bsq). How are level 1 land cover distributed across the urban gradient. Are the fraction values physically plausible?
click to expand...
High impervious fractions can be observed in the city center. A general increase in vegetation
cover and decrease in impervious cover is observed when moving towards suburban areas. Soil is only abundant on single patches,
e.g., along rail tracks or on construction sites. Fractions for each class are in the physically meaningful range between 0 and 1.
The sum of fractions per pixel over all classes is, however, often larger than 1.
D2: Do you observe an over- or underestimation of fractions for specific land cover types indicating errors in map?
click to expand...
Soil fractions are overestimated by around 20%, particularly for areas where red clay tiles /
bitumen / asphalt mixtures are apparent but no soil surfaces. Water fractions are overestimated by around 20%
throughout the city on all impervious surfaces.
Validation of fraction maps is commonly conducted by comparison of estimated and reference fractions using scatterplots and statistical measures (e.g., mean absolute error, root mean squared error, R², slope and intercept of a linear fitted regression model). This exercise……
illustrates the validation procedure for fraction maps
introduces EnMAP-Box geoalgorithms for producing reference fractions from high resolution land cover information and for statistical accuracy assessment of fraction maps.
A reference fraction map is created by rasterizing available reference land cover information to the pixel grid of the estimated fraction map. To obtain reasonable fractions, the reference land cover information needs to be at a significantly higher spatial resolution than the pixel grid. To create reference fractions, use the following processing algorithm Create Raster ‣ Fraction from Vector.
Enter the following data / parameters (use the tool tips for their description):
The Regression Performance algorithm in the Accuracy Assessment tools of the EnMAP-Box geoalgorithms implements the accuracy assessment for quantitative data. Scatterplots and statistical measures are reported in an HTML report. Run the Regression Performance algorithm with the following inputs:
Prediction: fraction_level1_estimation.bsq
Reference: berlin_level1_reference.bsq
Make yourself familiar with the HTML report.
Learning activities
E1: Visually compare your estimated fraction map (fraction_level1_estimation.bsq) with the reference fraction map (berlin_level1_reference.bsq). Do both maps show a good agreement in terms of spatial patterns or are there areas with large differences?
E2: Discuss the accuracy of your fraction map. What are the accuracies for the different classes and which classes show striking errors like underestimation or overestimations of fractions?
AE1: Repeat Exercises D & E using the two other class schemes (level 2, level 3) stored in the spectral library metadata and the land cover reference information. How do the accuracies vary and where are the limitations in mapping the more detailed class levels?
AE2: Explore the effects of changing the mixing parameters on the mapping accuracy of the level 2 classes. For more direct comparison, we recommend to alter only one parameter at a time. We further recommend to use the Random Forest Regression due to the low processing time. For example, …
change the Number of Synthetic Mixtures per Class: e.g. 10 vs. 1000 vs. 2000
do not Include Original Library Endmembers
change the Mixing Complexity Probabilities: e.g. only 2EM vs. only 3EM vs. only 4EM
change the Ensemble Size: e.g. 1 vs. 10 vs. 20
AE3: Compare the performance of the different regression algorithms offered in the EnMAP-Box. Please note that the other regressors have significantly longer processing times.
Regression-based mapping of forest aboveground biomass
Authors: Sam Cooper, Akpona Okujeni, Patrick Hostert, Clemens Jaenicke, Benjamin Jakimow, Andreas Rabe, Fabian Thiel & Sebastian van der Linden
This tutorial is part of the HYPERedu online learning platform,
an education initiative within the EnMAP mission hosted
on EO College. HYPERedu provides annotated slide collections and hands-on tutorials using the open-source EnMAP-Box software,
targeting basic principles, methods as well as applications of imaging spectroscopy.
Annotated slide collections for the tutorial Regression-based mapping of forest aboveground biomass and a software description unit for the EnMAP-Box are provided here:
Forest aboveground biomass (AGB) is a measure of the living and dead plant material in a given area. As such, it is
often used for forest management, assessing fire potential, and is an important metric used in modelling carbon and
nutrient cycles. AGB can be directly measured at a plot level by harvesting and weighing vegetation, but this is both
an expensive and highly invasive process. Through the use of statistical modelling and remotely sensed imagery, AGB can
be mapped across broad, spatially continuous areas using only a small number of directly measured reference plots.
This tutorial focuses on regression-based modeling of forest AGB using the EnMAP-Box. A hyperspectral image mosaic
from the EnMAP sensor (here simulated from AVIRIS imagery) and a corresponding vector dataset containing plot-based AGB
references are used for this tutorial. The aim is to provide an introduction into the functionality of the
EnMAP-Box, as well as hands-on training for implementing regression-based mapping.
This tutorial requires at least version 3.10 of the EnMAP-Box 3. There might be some minor
changes for higher versions (e.g., changed menu labels, added parameter options, etc.).
We recommend [1] and [2] for a comprehensive overview of imaging spectroscopy of terrestrial ecosystems, [3] for
an overview of remote sensing of forest AGB and [4] for a companion study using the same data.
The tutorial data contains a simulated hyperspectral EnMAP image, plot-based AGB references
as well as a land cover map for a small study area located in Sonoma County, California, USA. The simulated EnMAP
image is a subset extracted from the “2013 Simulated EnMAP Mosaics for the San Francisco Bay Area, USA” dataset [5].
AGB reference data was sampled from an existing LiDAR derived AGB map [6]. The land cover map was
taken from the 2011 National Landcover Database (NLCD) [7].
Data type
Filename
Description
Raster
enmap_sonoma.bsq
Simulated spaceborne hyperspectral data from the EnMAP sensor with a spatial resolution of 30m, 195 bands, and 1000x200 pixels (ENVI Standard Band Sequential bsq)
Raster
nlcd_sonoma.bsq
National Land Cover Database 30m classification for the study region (ENVI Standard Band Sequential bsq)
Vector
agb_sonoma.gpkg
343 AGB reference points sampled from the existing LiDAR derived AGB map (GeoPackage gpkg)
This exercise introduces basic functionalities of the EnMAP-Box for this tutorial. You will get to know the graphical
user interface and will learn how to load data, visualize raster and vector data, and use the basic navigation tools.
Additionally, you will learn to work with multiple map views and how to visualize image spectra using Spectral
Library Windows.
Start QGIS and click the icon in the toolbar to open the EnMAP-Box. The GUI of the EnMAP-Box consists of
a Menu and a Toolbar as well as panels for Data Sources and Data Views.
The QGIS Processing Toolbox including the EnMAP-Box algorithms can be optionally added to the GUI by clicking
on View in the Menu and by checking the Processing Toolbox from the Panel list.
To load new datasets into the EnMAP-Box, click the icon and navigate to the file directory
containing your data. Select agb_sonoma.gpkg from the Open data source dialogue and select Open.
Alternatively, the EnMAP-Box offers simple drag & drop capabilities to load data from an external file manager
(e.g. Windows File Explorer). Load enmap_sonoma.bsq by dragging and dropping the file from your file manager
into the Data Sources panel.
All data currently open in the EnMAP-Box will appear in the Data Sources panel.
The EnMAP-Box offers Map Views (Map #) for visualizing raster and vector data. Click the icon to open a
new Map View and drag enmap_sonoma.bsq from the Data Sources panel into Map #1.
In addition to a new Map View opening, a corresponding Data View entry is created in the Data Views panel
which shows all data currently loaded in a given Map View.
The enmap_sonoma.bsq image will be displayed as true color RGB composite. True color rendering is based on predefined
RGB band combinations (R: 658 nm, G: 569 nm, B: 479 nm) stored in the QGIS Style File enmap_sonoma.qml.
To assign a different RGB combination to the RGB channels, right click on the dataset in the Data Views panel, select
Layer Properties and navigate to Symbology. Set Render type to Multiband color and select bands to display in the
red, green and blue color channels. Choose appropriate Min/Max Value Settings (e.g. Cumulative Count Cut: 2-98%).
Common RGB combinations are listed below.
Combination
Red
Green
Blue
TrueColor
658 nm
569 nm
479 nm
nIR
847 nm
658 nm
569 nm
swIR
847 nm
1645 nm
658 nm
Tip
If the raster image has wavelength information associated with it, you may also select an RGB combination from
different custom RGB band combinations (True Color, Colored IR, SWIR-NIR-R or NIR-SWIR-R). Right click on the dataset
in the Data Views panel, select Layer Properties and navigate to Raster Band.
Don’t forget to choose appropriate Min/Max Value Settings.
The Toolbar offers common navigation tools for exploring visualized datasets. Make yourself familiar with the
following navigation tools:
Note that the mouse wheel can be used alternatively for zooming
(roll mouse wheel forward/backward) and panning (press and hold mouse wheel).
For a better orientation when exploring visualized raster images, you may switch on the crosshairs (right click into
Map View and activate Crosshair ‣ Show.
Make yourself familiar with the icon on the toolbar to view pixel values of the displayed raster.
Note: Identify Cursor Info must be activated to access this tool. When activated and used, a new
Cursor Location Values window will open displaying data from the selected pixel. This tool similarly works for
viewing attribute information of displayed vector data.
The EnMAP-Box enables users to work with multiple Map Views, which can be flexibly organized and geospatially linked.
Open a new Map View (Map #2) by clicking the icon.
Note
A new Data view appears corresponding to the newly added Map View.
Display enmap_sonoma.bsq as an RGB composite of your choice in Map #2.
Tip
When loading a raster image to a map view, you may also right click the filename in the Data Sources
panel and select either Open in existing map or Open in new map. If the raster image has wavelength
information associated with it, you may also select a predefined composite from the context menu.
For geospatial linking, click on View in the Menu and select Set Map Linking. In the Map Linking window,
select the Link Map Scale and Center option and close the dialog. You may also
right click a map window and select Link with other maps to initialize the linking process.
Tip
Map Windows can be re-arranged by clicking on the blue Map title bar (Map #) and dragging it to the desired position.
A transparent blue rectangle will appear indicating the docking position once you release the mouse button.
You may also undock map views from the EnMAP-Box window by selecting from the blue Map title bar.
To re-dock a Map View, click and drag the blue Map title bar to an open Map View already docked in the EnMAP-Box window.
To change the order of stacked layers, go to the Data Views panel and drag one layer on top or below
another one. Arrange the layer stack so that agb_sonoma.gpkg is displayed on top of enmap_sonoma.bsq.
By default, vector files are displayed with a single uniform symbol. To change this symbology, right
click on agb_sonoma.gpkg in the Data Views panel, select Layer Properties and navigate to Symbology
in the Layer Properties window. You can now change the symbology in accordance to the QGIS functionality.
Select Graduated from the dropdown menu, and select biomass in Value and Color in Method.
Set the Color ramp to run from white to green.
Press Classify and then OK to display the biomass values associated with each point.
The EnMAP-Box offers Spectral Library Windows (SpectralLibrary #) for visualizing spectra and handling
their metadata.
This tool may also be used to extract and visualize spectra which are spatially associated with vector
data open in the EnMAP-Box, i.e., the AGB reference points. To do this, open a new Spectral Library
window by selecting the icon on the toolbar.
Next, import spectral profiles from other sources by clicking at the icon in the SpectralLibrary #1
menu. Specify the following settings:
Field Value Import: Click on the icon, select biomass and click OK.
Terminate the Import Spectral Profile dialogue with OK. A spectral library is automatically built based on the
geographic location of each point in the vector file. The associated attribute information is displayed in the table on the right.
In Exercise B, you will learn how to create regression models based on the illustrated spectra and related AGB quantities
to predict AGB across the whole image.
Learning Activities
A1: What land cover types are present in the imagery? How are the AGB reference plots distributed throughout the scene?
A2: What different information can you see when switching from a true color composite to a NIR false color composite?
One of the strengths of remote sensing comes from its ability to take high-quality plot measurements of a
variable of interest and building statistical models with which wall to wall maps of this variable can
be created. One of the most common ways of doing this is to create regression models based on the optical
properties of the training data and applying it to large scale imagery. This exercise …
Introduces a regression-based mapping approach for taking plot measurements of AGB and generating
spatial AGB estimates using an input raster of hyperspectral imagery.
Demonstrates the Regression Dataset Manager and the Regression Workflow applications of the EnMAP-Box.
Duration: 20 min
1. Use the Regression Dataset Manager for data preparation
The Regression Dataset Manager offers different options to prepare data for the Regression Workflow
application. In the context of this tutorial, you will create a Regression Dataset from a raster and a vector
layer containing the spectral features (independent variable) and the target variable (dependent variable),
respectively. The regression dataset will be stored as pickle file (.pkl).
Open enmap_sonoma.bsq and agb_sonoma.gpkg in a single Map Window. Close all other
opened Map and Spectral Library Windows.
Click on Applications in the Menu and select Regression Dataset Manager.
To create the Regression Dataset from a raster and a vector layer, click on the
icon and choose Create regression dataset (from continuous-valued vector layer and feature raster).
A new widget will be opened. Run the dialog with the following inputs:
Raster layer with features: select enmap_sonoma.bsq
Fields with targets: select attribute biomass
Output Data: select … ‣ Save to File… and define an output path and file name
(e.g. agb_regression_data.pkl).
After running the dialog, agb_regression_data.pkl will be opened under Models in the Data Sources
panel. Close the Regression Dataset Manager.
Tip
The Regression Dataset Manager offers different random sampling options, e.g. for splitting Regression data
into training and validation data. Once the Regression data is prepared, you can access these options
through the Random Sample button.
2. Use the Regression Workflow for estimating AGB
The Regression Workflow application offers several state-of-the-art regression algorithms from the
scikit-learn library (see https://scikit-learn.org/stable/index.html) for predicting continuous variables.
The application further includes an optional cross-validation for assessing model performances.
Click on Applications in the Menu and select Regression Workflow to open the regression application.
Choose agb_regression_data.pkl as Training dataset.
Select RandomForestRegressor (default, due to the low processing time) as Regressor,
and use the default model parameters. Note that the different algorithms provided lead to varying accuracies
and processing times. Refer to the scikit-learn documentation for more information.
Raster layer with features specifies the raster image to which the regression model will be applied.
Select enmap_sonoma.bsq. Specify output path and file name (e.g. agb_estimation.bsq)
under Output regressor layer to save the result in your working directory.
To make use of a cross-validation, set the Number of cross-validation folds to 10 (default)
and leave the Open output performance report option checked. Specify output path and file name
(e.g. agb_estimation_cv.html) under Output regressor performance report to save the report in
your working directory.
The regression model can be optionally saved, e.g. for applying the model again to a dataset.
Specify output path and file name (e.g. agb_rfmodel.pkl) under Output regressor to save the
result in your working directory.
Click run to start the Regression Workflow.
Tip
All processing options of the Regression Workflow that are labeled as [optional] can be disregarded by
setting the Output to SkipOutput.
3. Assess the model performance for AGB estimation
After running the Regression Workflow, the performance report with scatterplots and statistical measures
will be opened in your default web browser.
Based on the 10-fold cross-validation, you can now access the performance of your model to predict AGB.
After running the Regression Workflow, all outputs will appear in the Data Sources panel.
Close all opened Map/SpectralLibrary Windows. Open enmap_sonoma.bsq as an RGB composite of your choice in Map #1.
Open the agb_estimation.bsq in a new Map View and link to the Map #1. Use the Layer Properties to change the color ramp to white-green (Singleband pseudocolor).
In this exercise, you will learn how to use the ImageMath application to calculate a NDVI map and
generate a forest mask based on the NLCD land cover map. Based on the forest area only, you will then
assess the AGB prediction from Exercise B relative to the NDVI using the Scatter Plot Tool.
The ImageMath tool in the EnMAP-Box allows users to apply a mathematical operation, python function or user defined function to an image.
In the following sections, you will utilize standard numpy array processing protocols
to calculate a NDVI map from two bands of our EnMAP imagery,
to generate a forest mask from the NLCD land cover map,
and to apply a forest mask to both the NDVI and AGB maps.
Close all opened Map/Spectral Library Windows. Display enmap_sonoma.bsq, nlcd_sonoma.bsq
and agb_estimation.bsq in a single or in multiple Map Views.
Open the ImageMath application by going to Applications then selecting ImageMath
ImageMath consists of several panels:
Inputs: defines input variables and variable names.
Outputs: defines output variable names and locations to be saved to.
Code editor: Text editor in which programmatic manipulation of the input datasets can be defined using Python scripting syntax.
Output Grid: Allows users to manually set the output grid.
Processing: Allows users to select block sizes for processing large datasets with limited memory.
Log Window: Displays the status (and error messages) of executed code.
Additionally, a tab for Routines allows users to select a number of common python-based tools for manipulating spatial datasets with linked documentation.
The Normalized Difference Vegetation Index (NDVI) is a commonly used vegetation index that is correlated with
both vegetation cover and AGB. The formula for NDVI is:
\[NDVI = \frac{NIR-Red}{NIR+Red}\]
where NIR is the near-infrared band reflectance (~850nm) and Red is the red band reflectance (~660nm).
We will now calculate NDVI from the EnMAP imagery using ImageMath.
In the Inputs panel, select enmap_sonoma.bsq from the dropdown menu. The variable name is automatically taken
from the filename, but may be changed if desired.
In the Code Editor, define the RED and NIR bands in our EnMAP imagery. These correspond to bands 42 and 73,
respectively, and we can define them using python indexing syntax (i.e. 0 indexed array slicing):
RED=enmap_sonoma[41]NIR=enmap_sonoma[72]
Next, we define the formula we wish to run:
NDVI=(NIR-RED)/(NIR+RED)
In the Outputs panel, define the output variable name as NDVI, and select an output file
path and file name (e.g. ndvi.bsq).
Finally, click on the button to run the script. A new raster dataset ndvi.bsq will appear in the Data Sources panel.
Attention
Input and output variable names in the Code Editor must exactly match corresponding names defined in the Inputs and Outputs panels.
As the model was trained using AGB reference plots from forest areas, only limited inference can be made of the non-forest AGB estimates.
We will therefore apply a forest mask to our AGB map as well as to the NDVI map.
The forest mask will be generated based on the available NLCD land cover map.
Below are the NLCD classes and color legend represented in the raster data. We will consider any pixel to be
forest which is labelled as Deciduous (41), Evergreen (42), or Mixed (43) forest according to the NLCD classification.
Open the ImageMath application and set nlcd_sonoma.bsq as the input file.
Copies the NLCD information to a new object we will manipulate to create the mask
Creates a list of classes which we consider forest
Loops through the x dimension of the raster. For each loop, x will be an integer representing the current location in the x dimension.
Loops through the y dimension of the raster. For each loop, y will be an integer representing the current location in the y dimension. These two loops allow us to look at each element in the array individually. While numpy offers more efficient ways to analyse arrays (See section C4), this is one basic approach.
Check if the element at the current x and y position is in the forest_classes list
If it is, set that value to 0
If it is not
Set that value to 1
Set Forest_Mask as the output and define the path and file name (e.g. forest_mask.bsq) for saving the result.
Run the script by clicking .
A new raster dataset forest_mask.bsq will appear in the Data Sources panel. The resulting mask now has a value of 0
for forested pixels, and 1 for non-forested pixels.
Open the ImageMath application and set agb_estimation.bsq, ndvi.bsq and forest_mask.bsq as the
input files. Note that these datasets need to be opened in a single or in multiple Map Views to make them selectable input files.
Enter the following code into the Code Editor to apply the forest mask to the AGB and NDVI images.
Sets the mask to a Boolean data type (i.e. True/False). The mask file contains binary values where 0
indicates forest (i.e. non-masked pixels) and 1 indicates non-forest (i.e. pixels to be ignored).
In Python, 1 also represents True while 0 represents False, and by setting the datatype to bool,
we explicitly tell Python to treat these values in this manner.
Copies the AGB values to a new array.
Steps through each value in the new array and sets the value to -99 if the mask value is True.
In numpy array speak, this line can therefore read: “For each value in AGB_Masked, if the corresponding
value in Forest_Mask is True (i.e. non-forest), then set that value to -99”. If the mask value is False
(i.e. forested), nothing will happen, and the biomass value will remain in the array.
Sets the no data value for the masked array to -99. This helps the EnMAP-Box to automatically display the
data correctly, and since it is not a realistic value for both AGB and NDVI, we can safely ignore it.
Steps 2-4 are then repeated for NDVI.
Set AGB_Masked and NDVI_Masked as the outputs and define the path and the file names
(e.g. agb_estimation_masked.bsq, ndvi_masked.bsq) for saving results.
Run the script by clicking . The new raster datasets agb_estimation_masked.bsq and ndvi_masked.bsq
will appear in the Data Source panel.
5. Visualize AGB vs. NDVI with the Scatter Plot tool
Close all Map Views.
Open an RGB composite of enmap_sonoma.bsq in Map #1.
Display agb_estimation_masked.bsq and ndvi_masked.bsq in Map#2 and Map#3, respectively and
use the Layer Properties to change the color ramp of both maps to white-green. Link all Map Views.
To investigate the relationship between estimated AGB and NDVI, we will make use of the EnMAP-Box’s Scatter Plot tool.
This is one of several tools integrated into the EnMAP-Box to support data visualization and assessment
Open the Scatter Plot tool by going to Tools then selecting Scatter Plot.
Select agb_estimation_masked.bsq and Band 1 for specifying X values and
ndvi_masked.bsq and Band 1 for specifying Y values.
If a selected raster has multiple bands, you would specify the desired band from the dropdown.
Click on Apply to visualize the Scatter Plot.
The Settings offer different options for visualizing the scatter plot.
You may change the Min and Max values, Coloring or Cumulative count cut
options to improve your visualization.
Under Extent you may choose Currentcanvas or Wholeraster to display data of the current map canvas only or to
display all raster data. Choose Wholeraster.
Under Accuracy you may choose to display Estimated or Actual. Actual will display all available data, while
Estimated will only display a random subset. For large raster extents, Estimated is much faster, and for that reason is the default.
Leave Accuracy as Estimated.
Click on Apply to update the Scatter Plot. If Live update is checked, visualization of the scatter plot will be automatically updated.
The Analytics offers options to assess the relationship between x and y values.
The 1:1 line represents the linear 1:1 relationship between the two variables of the same unit if they were perfectly correlated.
The Fitted line represents the linear regression line fitted between the actual data from the two variables.
The linear regression function, the coefficient of determination (r^2) and the Root Mean Squared Error (rmse) will
be additionally displayed. Activate the Min-max line by checking the box to its left.
AE1: Use the Image Statistics tool in the Tools menu to look at the band statistics for the
biomass predictions both with and without the tree mask applied.
AE2: Because we randomly subsetted the training data prior to model training, the performance of the model
has an element of uncertainty to it. To better understand this, rerun the regression workflow 3-5 times. Then use
the ImageMath tool to calculate the average estimate and variance. How does running the regression in such an
ensemble approach affect the results? What is the spatial pattern of variation in estimates?
AE3: Rerun regression (Exercise B) using NDVI as the input rather than the hyperspectral imagery.
Satellites observe the oceans, coasts and inland waters. Water quality parameters can be estimated from ocean colour
using the in-water algorithm ONNS, which has been developed at the Institute of Coastal Research at the Helmholtz-Zentrum
Geesthacht.
The EnMAP-Box is a free and open source plug-in for QGIS to visualize
and process satellite remote sensing data with special emphasis to the hyperspectral EnMAP sensor.
The following tutorial describes the usability and functionality of the Python-based ONNS processing tool in the EnMAP-Box.
In opposite to most other remote sensing applications it is not the water surface itself that determines the reflectance
(ocean colour), it is the upper layer of the water body beneath. Water is transparent for light in the visible spectral
range. Sunlight is absorbed and scattered along the water column, whereby four constituents are usually differentiated:
pure water itself, phytoplankton particles, sediments, and coloured dissolved organic matter (CDOM). The composition of
the water constituents determines the light penetration depth and portion of the backscattered light. The remote sensing
signal contains the backscattered light from the top layer only, approximately until a depth where only 90 % of the
spectral sunlight is still present (the 90%-depth). This information-containing water depth can be up to 60 m for clear
oceanic waters (spectral maximum around 450 nm), few metres for coastal waters (spectral maximum at 600 nm) or even few
centimetres for very turbid waters. By the way – photosynthesis of phytoplankton is only possible with sufficient
sunlight roughly until the 1%-depth, also referred to as euphotic depth.
The primary aim of ocean colour remote sensing is to quantify the water constituents, i.e. the phytoplankton biomass in
terms of their Chlorophyll-a pigment content, Chl [mg m −3], total (organic + inorganic) suspended matter concentration,
TSM [g m -3], and CDOM absorption at 440 nm, a cdom (440) [m -1]. Beyond
this, additional water mass-characterizing optical parameters, e.g. absorption and scattering properties, can be estimated.
Hyperspectral information, as provided by the EnMAP mission, allow further differentiation of spectral features associated
to algae types (phytoplankton groups) or sediment properties.
The present tutorial is an overview about the usage and functionality of the water algorithm implemented in the EnMAP-Box.
The processor contains three main steps: the atmospheric correction, an optical water type classification, and the
application of suitable in-water algorithms. Ocean, coastal and inland water bodies can be characterized by means of
these algorithms. The core in-water processor including optical water type classification is the ONNS
(OLCI Neural Network Swarm) algorithm [1] version 0.9. It is an EnMAP-precursor algorithm, which is specially developed
for data evaluation of the multi-spectral Ocean and Land Colour Instrument (OLCI) on board
Sentinel-3.
Satellite data from other ocean colour sensors can be analysed as well by means of an intermediate band-shifting procedure [2].
Atmospheric correction must be provided before utilization of the in-water algorithm, different models are available
for this, e.g. Polymer [3] or C2RCC [4] (but may lead to different retrieval results). Alternatively, in situ
remote-sensing reflectance data (in a txt-format) can be processed.
The ONNS algorithm itself consists of several blended neural networks that are specialized for 13 different optical
water classes. These comprise clearest natural waters but also extreme absorbing or scattering waters.
Considered Chlorophyll-a concentrations reach up to 200 mg m −3, non-algae particle concentrations up to 1,500 g m −3,
and CDOM absorption coefficient at 440 nm is up to 20 m −1. The algorithm generates different concentrations of water
constituents, inherent and apparent optical properties, and a color index. In addition, all products are delivered
with an uncertainty estimate.
The present version of the ONNS algorithm is able to process atmospherically corrected data of different ocean
colour sensors, namely OLCI, MERIS, MODIS and VIIRS. The corresponding data format must be NETCDF4. Moreover, ONNS
can handle tabulated remote-sensing reflectances at selected wavebands of OLCI, SeaWiFS, MODIS, MERIS, OCM-2, VIIRS,
SGLI, GOCI-2, EnMAP or PACE/OCI (text format).
One example of (C2RCC) atmospherically corrected Sentinel-3A/OLCI data is provided. The sub-scene shows the coastal
waters off the island Sylt in the German Bight, North Sea (2016-07-20, 09:34 UTC):
S3A_OL_2_WFRC8R_20160720T093421_20160720T093621_20171002T063739_0119_006_307______MR1_R_NT_002_sylt.nc
Note that processing of a satellite scenes may take a long time depending on the number of valid water pixels.
Figure 1: Sentinel-3A/OLCI top-of-atmosphere image with marked area of provided sub-scene (contains modified Copernicus Sentinel data [2016] processed by ESA/EUMETSAT/HZG).
The minimum output of the water algorithm ONNS contains 12 typical ocean colour products each with an estimate of
their associated uncertainties:
Concentration of Chlorophyll-a, Chl [mg m −3],
Concentration of inorganic suspended matter (minerals), ISM [g m -3],
Absorption coefficient of CDOM at 440 nm, a cdom (440) [m -1],
Absorption coefficient of phytoplankton particles at 440 nm, a p (440) [m -1],
Absorption coefficient of minerals at 440 nm, a m (440) [m -1`],
Absorption coefficient of detritus plus gelbstoff at 412 nm, a dg (412) [m -1],
Scattering coefficient of phytoplankton particles at 440 nm, b p (440) [m -1],
Scattering coefficient of minerals at 440 nm, b m (440) [m -1]`,
Total backscattering coefficient of all particles (organic and inorganic) at 510 nm, b bp (510) [m -1],
Downwelling diffuse attenuation coefficient at 490 nm, K d (490) [m -1],
Upwelling diffuse attenuation coefficient at 490 nm, K u (490) [m -1] and
Forel-Ule number, FU [-].
The standard output contains additional derived properties, such as total absorption and scattering coefficients
at 440 nm, optical dominance at 440 nm as well as estimated concentration of dissolved organic carbon (DOC [mg m -3]).
In addition, input remote-sensing reflectances and some information on the optical water types (OWT) are provided.
In a third saving option, excessive information on OWT classification can be stored.
Estimated Chlorophyll-a concentration, CDOM absorption and particulate backscattering from the Sentinel-3 OLCI test scene
Satellite data, e.g. Level-2 products of Sentinel-3/OLCI, are often available in NETCDF4 format, which can be
visualized and processed by means of the QGIS/EnMAP-Box as raster layer. It is possible to process atmospherically
corrected satellite ocean colour data with the ONNS algorithm via the EnMAP-Box processing tools (EnMAP-Box ‣ Water ‣ ONNS).
Overview of available processing tools in the EnMAP-Box. The raster layer data show remote-sensing reflectance at 510 nm.
GUI of the ONNS algorithm upon completion of the processing.
Depending on the size of the satellite scene, processing may take a few minutes up to some hours. The results are
saved in NETCDF4 format and can be viewed and transferred for example as raster layer.
The EnMAP geological mapper (EnGeoMAP), as part of the EnMAP Box, is
an expert-system-based tool for the processing and characterization
of geologic surface cover types from hyperspectral data in general
and EnMAP satellite data in particular. EnGeoMAP 3.1 is
based on the EnGeoMAP 2.0 algorithm developed by Christian Mielke (Mielke et al., 2016).
EnGeoMAP 3.1 is optimized for Windows 10 and 11 and the Windows version of QGIS (LTS).
The software requires a hyperspectral reflectance data cube, a
spectral library with user-defined endmembers, and a user-supplied
color scheme as inputs to run. The software then classifies the
hyperspectral data based on the provided endmembers and represents it
according to the supplied color scheme.
EnGeoMAPhas been primarily designed for EnMAP data, yet it works for
most full-range hyperspectral sensor suites, provided that VNIR and
SWIR data are all available.
The hyperspectral data cubes must be in the Band Sequential (BSQ)
format conforming to L3Harris ENVI header file information (.hdr)
Overlapping bands need to be removed from the dataset i.e., the wavelength succession vector in the header files need to be strictly ascending.
Clipping and interpolation are required in both strong water vapor residuals at 940 nm and 1130 nm to assure an artifact-reduced analysis result.
Nanometer (nm) is the recommended wavelength unit, and the floating-point or unsigned integer (ranging from 0-10000, as is common for many imaging spectrometers) is recommended for the reflectance values.
Adaptive filtering is suggested to further reduce noise and artifacts from reflectance data.
The spectral library files also must be in Band Sequential (BSQ)
format according to the L3Harris ENVI header file information (.hdr).
The necessary spectral library files can be prepared with the EnMAP
Box spectral library wizard or with any other software that is able
to write spectral library files in band sequential (BSQ) format. The
entries of the spectral library are plotted according to the RGB
color code from the CSV file (see the RGB Color Scheme CSV file
section).
Overlapping bands need to be removed from the dataset i.e., the wavelength succession vector in the header files need to be strictly ascending.
For field-measured spectral libraries: Clipping and interpolation are required both strong water vapor residuals at 940 nm and at 1130 nm to assure an artifact-reduced analysis result.
Nanometer (nm) is recommended as wavelength unit, as well as float or an unsigned integer from 0-10000 (standard for many imaging spectrometers) for the reflectance values.
Adaptive filtering is suggested to further reduce noise and artifacts.
The spectral library should be resampled to the spectral resolution of the reflectance data cube for the two datasets to match spectrally.
The RGB (Red, Green, Blue) color scheme needs to be provided by
the user in a CSV file format (UTF-8). The entries have to be in the
right order and matching to the entries of the spectral library.
Through the CSV color file, the user can choose the colors with which
EnGeoMAP classifies the according library entries in the
hyperspectral image. There are several online tools available to
visualize RGB color code for a preview of the chosen colors. CSV
files can either be edited with a standard text editor (Figure 1) or
spreadsheet editor (Figure 2).
Figure 1: Color scheme CSV file example opened in a text editor. Here the legend includes four different endmembers.
Figure 2: Color scheme CSV file example opened in a spreadsheet
editor. Here the legend includes four different endmembers.
The dataset contains a hyperspectral EO-1 Hyperion image data cube
which was acquired over the Gamsberg Mine (South Africa) prior to its
opening, a field-based spectral library, and a CSV color file defining the color codes
of each endmember for visual representation.
Figure 3 shows how you get access to the EnGeoMAP toolbox.
Figure 3: Access to EnGeoMAP.
The EnMAP-Box provides a graphical user interface (GUI) for EnGeoMAP
3.1 (Figure 4). The three main input fields for the location of the
user input files comprises:
The hyperspectral image data
The spectral library file
The CSV color file
Note: For the hyperspectral data and the library file only the path
to the main files are required. The header (.hdr) files need to be in
the same folder as the reflectance data files and are loaded
automatically by the software. Choosing the header files in the data
selection will lead to an error.
The user can choose the minimum reflectance threshold in the VNIR and
SWIR regions. The values should be chosen depending on the noisiness
of the data. They should be as low as possible for data with a good
SNR. - Standard settings are recommended according to Figure 5.
The minimum fit threshold sets the lower cutoff value for the minimum
acceptable valid spectral correlation value between the image
spectrum and the library spectra. Values below this threshold are set
to zero.
For the unmixing data products, the maximum number of endmembers
needs to be set by the user. The default is set to 10 (see Results
section). This number should not exceed the total number of
endmembers embedded in the spectral library file.
Processing is started by pressing the OK button and then pressing OK
in the subsequent notification window.
A status indicator in the lower right corner shows if the software is
still processing the data or is ready to process new data.
Figure 4: The graphical user interface (GUI) of EnGeoMAP 3.1.
The preprocessing module within EnGeoMAP extracts the characteristic
absorption features of the input spectra (Figure 5). These features
are then used in a weighted fitting operation in the spectral module
to calculate the material similarity scores of the unknown pixel
spectrum compared to the reference library. According to those fit
values, a user defined minimum fit threshold may now be applied to
only use those library spectra in a Bounded Value Least
Squares (BVLS) unmixing meeting the user-defined threshold (see
Figure 4 option: Minimum Fit Threshold Weighted Fitting). The
resulting unmixing as well as the correlation data cube are then
sorted in descending order.
The first band of each of the sorted results is then used in the
spatial module together with the CSV file to provide a color-coded
best fit (highest correlation score) material map and highest
abundance (highest BVLS fraction) map.
Figure 5: Basic processing workflow for EnGeoMAP 3.1 with modules,
input and output highlighted.
EnGeoMAP3.1 provides 6 data products which are identifiable by their
suffixes to the original filename (basename). See Table 1 for an
overview of the data products and suffixes.
_correlation_result
Raw correlation scores from the
spectral module for
characteristic absorption
features.
_best_matches_best_fit_coleur
Results relating to the highest
correlation score. The data
product is based on sorted
correlation scores (the first
band contains the highest score
values, the second the second
highest scores and so on)
_best_fit_coleur_class_geotiff
The material / geological
classification thematic map
Standard RGB color maps assigned
from the sorted Band 1 of the
best_matches results using the
.csv colorfile. It can be
directly viewed with the image
viewer of your choice.
Class images,can be used for a
quick legend generation within
the QGIS map composer.
_abundance_result
Raw Bounded Value Least
Squares (BVLS) unmixing scores
from the spectral module for
characteristic absorption
features.
_abundance_unmix_best_unmix_coleur
Highest abundance (highest BVLS
fraction) sorted color map.
Standard RGB color maps assigned
from the sorted Band 1 of the
best_unmix (highest BVLS
fraction) results using the .csv
color file. It can be directly
viewed with the image viewer of
your choice.
Class images,can be used for a
quick legend generation within
the QGIS map composer.
Algorithm, Python code, and GUI was developed by Christian Mielke.
Current version GUI and algorithm product adjustments (EnGeoMAP version 3.1) by Helge L. C. Daempfling.
For theoretical background information on EnGeoMAP algorithm please
consider reading Mielke et al. (2016) EnGeoMAP 2.0 Automated
Hyperspectral Mineral Identification for the German EnMAP Space
Mission.
Mielke, C.; Rogass, C.; Boesche, N.; Segl, K.; Altenberger, U. EnGeoMAP 2.0—Automated Hyperspectral Mineral Identification for the German EnMAP Space Mission. Remote Sens. 2016, 8, 127.
Clark, R.N.; Swayze, G.A.; Livo, K.E.; Kokaly, R.F.; Sutley, S.J.; Dalton, J.B.; McDougal, R.R.; Gent, C.A. Imaging spectroscopy: Earth and planetary remote sensing with the USGS Tetracorder and expert systems. J. Geophys. Res. Planets 2003, 108.
Kokaly, R.F. Spectroscopic remote sensing for material identification, vegetation characterization, and mapping. Proc. SPIE 2012, 8390.
Clark, R.N.; Swayze, G.A.; Wise, R.; Livo, E.; Hoefen, T.M.; Kokaly, R.F.; Sutley, S.J. USGS Digital Spectral Library Splib06a; U.S. Geological Survey: Denver, CO, USA, 2007.
In this tutorial we will show the basic components of the EnMAP-Box.
Participants will learn how to manage, visualize and process raster and vector data inside the EnMAP-Box.
We will detail the multiple-viewer concept of the EnMAP-Box, that allows the exploration of data in different
visualizations at the same time.
Here we will especially highlight, how we capitalize on the spectral characteristics of optical (hyperspectral)
EO data, e.g. by correctly plotting pixel profiles against the given center wavelength,
or by quickly selecting target bands for visualization. Finally, we will perform a Classification Analysis Workflow
using PRISMA or (simulated) EnMAP data to illustrate EO data processing.
Requirements
This tutorial requires version 3.10+ of the EnMAP-Box 3 running in QGIS 3.24+.
Exercise A: Visualize PRISMA data inside the EnMAP-Box
Open the PRISMA raster
In the EnMAP-Box, open the PRS_L2D_STD_20201107101404_20201107101408_0001_SPECTRAL.tif raster and visualize it
in a map view using drag&drop.
Style the PRISMA layer using an RGB visualization
We can improve the visualization using the Raster Layer Styling panel.
As the PRISMA layer contains information about the center wavelength location for each band,
we can directly choose from a list of predefined RGB band combinations,
e.g. Natural color (R-G-B) for showing Red band, Green band and Blue band as an RGB visualization,
or Agricultural 1 (S1-N-B) for showing SWIR 1 band, NIR band Blue band as an RGB visualization.
Use the A, B, …, S2 buttons, to select individual PRISMA bands that best match the predefined
broad bands from the Sentinel-2 sensor, e.g. N is associated with the Sentinel-2 NIR band at 833 nanometers.
Use the widgets in the Min / Max Value Settings to interactively alter and improve the contrast
stretch.
Use a singleband gray visualization to identify bad bands
Due to atmospheric conditions, the PRISMA raster may contain bad bands,
which can be identified easily using a singleband gray visualization.
In the Raster Layer Styling panel select the Gray tab to change the visualization.
Use the band selection slider or the band selection combobox (together with the Arrow-Up and Arrow-Down keys)
to browse the PRISMA bands and identify potential bad bands.
As before, you can also use the A, B, …, S2 buttons.
Plot spectral profiles and show cursor location values
Use the Identify map tool to plot PRISMA spectral profiles and query band values for the selected
location.
In the Spectral Profile Sources panel you can easily alter the plot style.
Use the Classification workflow algorithm to perform a full landcover classification.
This includes training data preparation, fitting a Random Forest Classifier, and predicting landcover classes
and class probabilities.
The original PRISMA raster and the derived classification and class probabilities can be visualized next to each
other using multiple map views.
Finally, visualize the probabilities for the classes impervious, tree and water as RGB multiband color.
Visualize the predicted maps
Use the Classification Statistics app to visualize the derived classification map.
Use the Class Fraction/Probability Statistics app to visualize the derived class probability map.
View the fitted Random Forest Classifier model
The training data and the fitted model can be explored inside the Data Sources panel.
If you like to develop an EnMAP-Box application, or more general, a QGIS and Qt application, we recommend to use
a state-of-the-art Integrated Development Environment (IDE) like PyCharm. It offers run-time debugging,
code completion, spell-checking, syntax highlighting, SCM support, unit-testing and many other helpful things.
Modify the qgis-env.bat config section to fit to your local environment, i.e. set the correct paths to your
local OSGeoW installation and PyCharm executable
@echo off
:: ### CONFIG SECTION ###:: root of local OSGEO4W installationsetOSGEO4W_ROOT=D:\OSGeo4W
:: PyCharm executable, adjust for version updatessetPYCHARM_EXE="C:\Program Files (x86)\JetBrains\PyCharm 2022.1.2\bin\pycharm64.exe":: git binaries and git lfs binariessetBIN_GIT=C:\Program Files\Git\bin
setBIN_LFS=C:\Program Files\Git LFS
Call start_pycharm.bat to open PyCharm within the latest QGIS release.
You can modify the start script to start a different QGIS build. E.g.
Installing QGIS via conda is not officially supported by the QGIS core development team.
The QGIS versions available in conda can therefore differ from the official QGIS versions!
The installation of QGIS within a conda /
anaconda environment is (almost) the same on macOS, windows or linux. Using conda
it is often much easier to install additional python packages while admin rights are not required.`
Depending on the components and applications you like to use, it might be required to install more packages.
If you cloned the EnMAP-Box repository you can also point to the local conda_environment.yml.
Edit the --name or the YAML file itself as you wish. For more information on creating and managing conda environments visit the conda documentation
To easily start applications like PyCharm in this environment, which have not been installed by conda, you might
define an alias during the activation of the environment.
Create an activation script and define an alias for PyCharm:
Mambaforge is a reimplementation of Conda and behaves very similar.
Make sure to read and understand the previous section explaining Conda, before using Mambaforge.
Execute one of the following installation command inside your Mambaforge/Miniforge prompt:
Start PyCharm and add my_repositories/enmap-box as new project via File > Open File or Project
If this is not already the case, tell PyCharm where to find your Git-executable.
Open File > Settings > Version Control > Git to set Path to Git executable. Press Test to check the used Git version.
If not already set in the environment from which you started PyCharm, you can set it explicitly.
Open Run > Debug … > Edit Configurations and add the QGIS_PREFIX_PATH to the User environmental variables.
This way PyCharm runs python files in a environment with QGIS_PREFIX_PATH defined.
Also define the Environment variables for the Python console. Go to File > Settings > Build, Execution, Deployment > Console > Python Console
and add QGIS_PREFIX_PATH to the Environment variables.
You may also modify the shell used in your PyCharm terminal to use the QGIS environment.
Open Tools > Terminal and set the shell path to, for example:
QGIS Installation
Terminal path
OSGeo4W
cmd.exe"/K"qgis_env.bat (see above how to create the qgis_env.bat)
The Qt Assistant allows you to browse fast and offline through Qt help files (*.qch). These files exists for
all Qt classes and the QGIS API. They can be generated event with Sphinx, which allows you to provide your
own source-code documentation as .qch file as well.
Start the Qt Assistant, e.g. from your PyCharm terminal:
The Qt Designer is a powerful tool to create GUI frontends by drawing, drag and drop.
Created GUI form files are saved in a XML file ending with *.ui. These can be called from
python to automatically create the entire GUI backend, e.g. windows and buttons defined with the Qt Designer.
You can start the Qt Designer from your PyCharm terminal by:
(qgis_stable) $>designer
Qt Designer showing the metadataeditor.ui for the Metadata editor.
Qt Creator is the one-in-all IDE to develop Qt C++ applications. It includes the functionality covered by Qt Assistant
(here called Help) and Qt Designer (here called form designer) and helps to browse C++ code. It is the preferred tool to
explore the QGIS C++ source code, for example if you like to better understand what it does behind the QGIS python API.
Qt and the Qt Creator are available at https://www.qt.io/download. Ensure to install the code documentation for the same
Qt version used by QGIS.
If you work on windows and want to test your code based on nightly builds of the upcoming QGIS
version, or like to inspect/debug the QGIS C++ API at runtime, you might use the OSGeo4W
installer to setup your development environment:
Install the nightly build branch qgis-dev and related debug symbols qgis-dev-pdb.
Install other required packages, e.g. pip3 etc. Later on.
In case of missing packages, search and install via OSGeo4W installer first. If not available there, use
the OSGeo4W shell and call pip.
Create a qgis-dev-env.bat to setup your QGIS environment
Terminal Shell Path: cmd.exe “/K” <your path to>qgis-dev.bat
(this is why we created two batch files. qgis-dev.bat setups the environment, but does not start any app)
The qgis-dev.bat will be called when starting the terminal
add <OSGEO4W>appsqgis-devpython and
<OSGEO4W>appsqgis-devpythonplugins as source folders
Adding the QGIS python and pythonplugins folder as content roots.
The scripts directory contains scrips to support developers. Ensure that this folder is part of your python path, e.g add
it as source in your PyCharm project (left-mouse, mark directory as source):
Script
Purpose
scripts/setup_repository.py
Creates resource files etc. required to test the EnMAP-Box by calling other scripts
scripts/create_plugin.py
Create the EnMAP-Box Plugin ZIP file.
scripts/create_plugin.py
Create the two shell-scripts runtests.bat and runtests.sh to run unit tests.
scripts/install_testdata.py
Downloads and installs the EnMAP-Box testdata and qgisresources
scripts/compile_resourcefiles.py
Complies EnMAP-Box resources (*.svg, *.png) into *_rc.py files
scripts/iconselect.py
Opens a browser to show EnMAP-Box and QGIS resources, e.g. icons
Some directories and files are not part of the repository and explicitly ignore in the .gitignore file, but required or
created during the development process. These are:
enmapboxtestdata/
Official EnMAP-Box testdata. Not part of repository code, can be installed calling
scripts/install:_testdata.py.
enmapboxunittestdata/
Unit test testdata. Not part of repository code, can be installed calling
scripts/install:_testdata.py.
qgisresources/
QGIS resources compiled as python modules.
Read here for details.
Add <my_repositories>/enmapbox/ as source location to your PyCharm project
(instead of that in your QGIS active profile!)
(Optional) install the QGIS source code repository.
For the next step, but also if you like to discover the QGIS ++ code, it is recommended to install the
QGIS repository as well. Some EnMAP-Box scripts can use data from the QGIS source code, e.g. to show images that
otherwise are available on runtime in the QGIS Desktop application only.
Run scripts/setup_repository.py to create Qt resource modules and perform a dependency check.
The EnMAP-Box uses the Qt resource system (see https://doc.qt.io/qt-5/resources.html for details) to access icons.
This step creates for each Qt resource file (filename.qrc) a corresponding python module
(filename.py) that contains a binary encrypted description of resources (icons, images, etc.).
During startup, these resources are loaded and can be accessed by resource path strings.
The EnMAP-Box re-uses several icons provided by the QGIS desktop application. For example,
the QGIS icon for raster layers is available at :/images/themes/default/mIconPolygonLayer.svg and can be
visualized in the QGIS python shell as followed:
If we start and develop application from inside PyCharm, we usually don’t have access to QGIS desktop application
resources. However, if you have downloaded the QGIS repository as described above, scripts/setup_repository.py
will look for it, compile the resource files and write them into folder enmap-box/qgisresources.
The most-recent EnMAP-Box testdata is hosted on https://bitbucket.org/hu-geomatics/enmap-box/downloads
If missing or outdatet, it will be downloaded and installed after a user
clicks on Project > Load Example Data. The downloaded data will be extracted into
<root>/enmapboxtestdata, with <root> either being the EnMAP-Box repository folder or the QGIS plugin installation
folder.
Building the EnMAP-Box plugin is done by creating a zip file which the QGIS Plugin Manager
can use to install the EnMAP-Box. This requires that you have setup your development environment
as described in the installation for developers
Calling:
python scripts/create_plugin.py
creates or updates the folder deploy with:
the folder enmapboxplugin. It contains the plugin code and additional files
a file named like enmapboxplugin.3.7.20190214T1125.develop.zip. This is the enmapboxplugin` folder compresses
as zip file, which can be used by the QGIS Plugin Installer to install the EnMAP-Box.
Using the -t keyword adds the EnMAP-Box test data, so that you can use and test the EnMAP-Box in QGIS
with having example data already installed.
Note
The <subsubversion> consists of <date>T<time>.<activebranch> and is generated automatically.
This helps to generate, test and differentiate between EnMAP-Box versions of different development steps.
A successful build ends with a printout like:
### To update/install the EnMAP-Box, run this command on your QGIS Python shell:frompyplugin_installer.installerimportpluginInstallerpluginInstaller.installFromZipFile(r"C:\Users\user\Repositories\QGIS_Plugins\enmap-box\deploy\enmapboxplugin.3.5.20191030T0634.develop.zip")#### Close (and restart manually)QProcess.startDetached(QgsApplication.arguments()[0],[])QgsApplication.quit()## press ENTER
Copy and run it in your QGIS python shell to install the build EnMAP-Box plugin and restart QGIS.
Alternatively, you can install the plugin in QGIS with a few mouse clicks more by:
Open the QGIS Plugin Manager
Install from ZIP with the created ZIP file
Restart QGIS to account for activate changes in python code
To create the HTML based documentation on your local system call:
cd doc
make html
This creates the folder doc/build with the HTML documentation. To view the documentation,
just open doc/build/html/index.html in your webbrowser of choice.
Official EnMAP-Box plugin master versions are named like enmapboxplugin.3.3.20190214T1125.main.zip. They need to be uploaded
to https://plugins.qgis.org/plugins/ using an OSGeo account.
Other versions, e.g. development snapshots are named like enmapboxplugin.3.3.20190214T1125.develop.zip or enmapboxplugin.3.3.20190214T1125.my_feature_branch.zip
and might be distributed using the repositories download section https://bitbucket.org/hu-geomatics/enmap-box/downloads/.
Updates to the EnMAP-Box documentation (folder doc/source) are detected by readthedocs
when pushed to the develop or master branch of the EnMAP-Box repository.
As we like to start our examples from an IDE like PyCharm, we first need to start a QgsApplication.
It provides the main GUI loop that receives user interactions, like mouse clicks and keyboard entries:
If we would’nt call the GUI loop with qgsApp.exec_(), our program will terminate immediately.
The EnMAPBox object is a singleton, which means that there exist only one EnMAPBox instance. You can connect
to a running EnMAP-Box instance like this:
The EnMAP-Box differentiates between Raster, Vector, Spectral Libraries and other data sources. To add new data sources,
you just need to provide a file-path or other unique resource identifier via enmapBox.addSource(r'uri')
enmapBox=EnMAPBox.instance()# add some data sourcesfromenmapboxtestdataimportenmapaspathRasterSourcefromenmapboxtestdataimportlandcover_polygonsaspathVectorSourcefromenmapboxtestdataimportlibraryaspathSpectralLibrary#add a single sourceenmapBox.addSource(pathRasterSource)#add a list of sourcesenmapBox.addSources([pathVectorSource,pathSpectralLibrary])
A the EnMAP-Box uses the QGIS API to visualize spatial data, we can add OpenGIS Web Services (OWS)
like Web Map Services (WMS) and Web Feature Services (WFS) as well:
wmsUri='referer=OpenStreetMap%20contributors,%20under%20ODbL&type=xyz&url=https://tiles.wmflabs.org/hikebike/%7Bz%7D/%7Bx%7D/%7By%7D.png&zmax=17&zmin=1'wfsUri=r'restrictToRequestBBOX=''1'' srsname=''EPSG:25833'' typename=''fis:re_postleit'' url=''https://fbinter.stadt-berlin.de/fb/wfs/geometry/senstadt/re_postleit'' version=''auto'''enmapBox.addSource(wmsUri,name="Open Street Map")enmapBox.addSource(wfsUri,name='Berlin PLZ')..image::../img/gstart_datasources.png:width:100%
EnMAP-Box Data sources are unique; which means that only one data source refers to the same source path.
Adding the same source multiple times does not change the total number of data sources:
fromenmapboximportEnMAPBoxfromenmapboxtestdataimportenmapenmapBox=EnMAPBox(None)enmapBox.addSource(enmap)print('# data sources: {}'.format(len(enmapBox.dataSources())))# add the same source againenmapBox.addSource(enmap)print('# data sources: {}'.format(len(enmapBox.dataSources())))
Note
Data sources are internally described by the enmapbox.gui.datasources.DataSource class.
A DataSource object stores information on the data source uri, name, type and its creation date.
Overwriting a file source and adding it again to the EnMAP-Box will remove the data source first and add it’s again.
This allows to react on changes and to ensure data integrity, e.g. to account for changes in file size and metadata.
The EnMAP-Box Docks to visualize spatial data and spectral libraries. Docks are based on the
pyqtgraphDock which inherits QDockWidget, but can be
arranged much more flexible, e.g. in nested layouts. We use these Docks as containers for specialized widgets,
in particular the enmapbox.gui.mapcanvas.MapCanvas (as MapDock) and the SpectralLibraryWidget (SpectralLibraryDock).
You can create new docks with EnMAPBox.createDock('<dockType>'):
fromenmapbox.gui.enmapboxguiimportEnMAPBox,Dock,MapDock,SpectralLibraryDockenmapBox=EnMAP-Box.instance()enmapBox.createDock('MAP')# a spatial mapenmapBox.createDock('SPECLIB')# a spectral libraryenmapBox.createDock('TEXT')# a text editorenmapBox.createDock('WEBVIEW')# a browserenmapBox.createDock('MIME')# a window to drop mime data (for developers)
Dock titles, visibility and behaviour can be modified:
# modify dock propertiesmapDock1=enmapBox.createDock('MAP')mapDock2=enmapBox.createDock('MAP')mapDock3=enmapBox.createDock('MAP')# set dock titlemapDock1.setTitle('Map 1 (fixed)')mapDock2.setTitle('Map 2 (floated)')mapDock3.setTitle('Map 3 (hidden)')mapDock2.float()mapDock3.setVisible(False)
Three map docks. Map 3 is hidden and therefore visible in the dock manager panel only.
# list map docks onlyfordockinenmapBox.dockManager().docks(dockType='MAP'):assertisinstance(dock,Dock)print(dock)# list all spectral library docksfordockinenmapBox.dockManager().docks(dockType='SPELCIB'):assertisinstance(dock,Dock)print(dock)
Working with spatial data often requires to convert coordinates between different coordinate reference systems (CRS).
The EnMAP-Box classes SpatialPoint and SpatialExtent inherit QgsPointXY and QgsRectangle,
respectively and enhance their parent classes by keeping a reference on the QgsCoordinateReferenceSystem used to specify the
coordinates.
Retrieve a QgsRasterLayer center coordinate and convert it into geographic
lat/lon coordinates:
Be aware that some CRS transformations are not possible and might cause errors. This might happen in particular when
transforming coordinates from regional projected CRS, like an UTM Zone, into a CRS that is defined for another region
of the world.
The EnMAP-Box emits a sigCurrentLocationChanged signal if the Identify map tool is activated and user
left-clicks on a map canvas.
fromenmapbox.guiimportMapTools,SpatialPointdefprintLocation(spatialPoint:SpatialPoint):print('Mouse clicked on {}'.format(spatialPoint))enmapBox.sigCurrentLocationChanged.connect(printLocation)
The overloaded sigCurrentLocationChanged signature can be used to get the map canvas instance where the location was selected from:
defprintLocationAndCanvas(spatialPoint:SpatialPoint,canvas:QgsMapCanvas):print('Mouse clicked on {} in {}'.format(spatialPoint,canvas))enmapBox.sigCurrentLocationChanged[SpatialPoint,QgsMapCanvas].connect(printLocationAndCanvas)
If “Identify raster profile” is activated, the EnMAP-Box extracts SpectralProfiles from the raster layer(s) below the mouse-click location.
These spectra can be received from the sigCurrentSpectraChanged signal:
Applications for the EnMAP-Box define an EnMAPBoxApplication instance that describes basic
information like the Applications name and how a user can start it. The following examples are taken from the
examples/minimumexample package, which you might copy and modify to implement own EnMAPBox Applications.
An EnMAP-Box application inherits from EnMAPBoxApplication and defines basic information like a
name and version. The variable license defaults to GNU GPL-3, but might be changed:
fromqgis.PyQt.QtGuiimportQIconfromqgis.PyQt.QtWidgetsimportQMenu,QAction,QWidget,QHBoxLayout,QLabel,QPushButtonfromenmapbox.gui.applicationsimportEnMAPBoxApplicationfromqgis.coreimport*VERSION='0.0.1'LICENSE='GNU GPL-3'APP_DIR=os.path.dirname(__file__)APP_NAME='My First EnMAPBox App'classExampleEnMAPBoxApp(EnMAPBoxApplication):""" This Class inherits from an EnMAPBoxApplication """def__init__(self,enmapBox,parent=None):super(ExampleEnMAPBoxApp,self).__init__(enmapBox,parent=parent)#specify the name of this appself.name=APP_NAME#specify a version stringself.version=VERSION#specify a licence under which you distribute this applicationself.licence=LICENSE
To become accessible via a menu of the EnMAP-Box, the application needs to implement EnMAPBoxApplication.menu(…) to
return a QAction or (better) a QMenu with multiple QActions. By default, the returned object is added to the EnMAP-Box
“Application” menu from where users can start the QAction that you defined:
defmenu(self,appMenu):""" Returns a QMenu that will be added to the parent `appMenu` :param appMenu: :return: QMenu """assertisinstance(appMenu,QMenu)""" Specify menu, submenus and actions that become accessible from the EnMAP-Box GUI :return: the QMenu or QAction to be added to the "Applications" menu. """# this way you can add your QMenu/QAction to an other menu entry, e.g. 'Tools'# appMenu = self.enmapbox.menu('Tools')menu=appMenu.addMenu('My Example App')menu.setIcon(self.icon())#add a QAction that starts a process of your application.#In this case it will open your GUI.a=menu.addAction('Show ExampleApp GUI')assertisinstance(a,QAction)a.triggered.connect(self.startGUI)appMenu.addMenu(menu)returnmenu
3. Define QgsProcessingAlgorithms for the EnMAPBoxAlgorithm Provider
Your Application might provide one or multiple QgsProcessingAlgorithms for the QGIS Processing Framework. This way your algorithms
become visible in the QGIS Processing Toolbox and can be used in the QGIS Model Builder.
To add your QgsProcessingAlgorithms to the EnMAP-Box Algorithm Provider implement EnMAPBoxApplication.processingAlgorithms(…).
For the sake of simplicity, let’s define a simple function and a QgsProcessingAlgorithm to call it:
defexampleAlgorithm(*args,**kwds)->list:""" An dummy algorithm that prints the provided arguments and keywords and returns its inputs. """print('Start exampleAlgorithm...')text=['Arguments: {}'.format(len(args))]fori,ainenumerate(args):text.append('Argument {} = {}'.format(i+1,str(a)))text.append('Keywords: {}'.format(len(kwds)))forkey,parameterinkwds.items():text.append('{} = {}'.format(key,parameter))print('\n'.join(text))print('exampleAlgorithm finished')returnargs,kwdsclassExampleProcessingAlgorithm(QgsProcessingAlgorithm):""" Exemplary implementation of a QgsProcessingAlgorithm. See https://api.qgis.org/api/classQgsProcessingAlgorithm.html for API documentation """def__init__(self):super(ExampleProcessingAlgorithm,self).__init__()defcreateInstance(self)->QgsProcessingAlgorithm:""" Creates a new instance of the algorithm class. :return: QgsProcessingAlgorithm """returnExampleProcessingAlgorithm()defname(self)->str:return'examplealgorithm'defdisplayName(self):return'Minimal Example Algorithm'defgroupId(self)->str:""" Returns the unique ID of the group this algorithm belongs to. :return: str """returnGROUP_IDdefgroup(self)->str:""" Returns the name of the group this algorithm belongs to. :return: str """returnAPP_NAMEdefinitAlgorithm(self,configuration:dict=None):""" Initializes the algorithm using the specified configuration. :param configuration: dict """self.addParameter(QgsProcessingParameterRasterLayer('pathInput','The Input Dataset'))self.addParameter(QgsProcessingParameterNumber('value','The value',QgsProcessingParameterNumber.Double,1,False,0.00,999999.99))self.addParameter(QgsProcessingParameterRasterDestination('pathOutput','The Output Dataset'))defprocessAlgorithm(self,parameters:dict,context:QgsProcessingContext,feedback:QgsProcessingFeedback):""" Runs the algorithm using the specified parameters. :param parameters: dict :param context: QgsProcessingContext :param feedback: QgsProcessingFeedback :return: dict """assertisinstance(parameters,dict)assertisinstance(context,QgsProcessingContext)assertisinstance(feedback,QgsProcessingFeedback)args,kwds=exampleAlgorithm(parameters)outputs={'args':args,'kwds':kwds}returnoutputs
Now define EnMAPBoxApplication.processingAlgorithms(…) to add the ExampleProcessingAlgorithm to the EnMAPBoxProcessingProvider:
defprocessingAlgorithms(self)->list:""" This function returns the QGIS Processing Framework algorithms specified by your application :return: [list-of-QgsProcessingAlgorithms] """return[ExampleProcessingAlgorithm()]
Calling the ExampleProcessingAlgorithm from the QGIS Processing Toolbox should now create a printout on your python console like:
The startGUI() function is used to open the graphical user interface. A very simple GUI could be:
defonButtonClicked():print('Button was pressed')w=QWidget()w.setLayout(QVBoxLayout())w.layout().addWidget(QLabel('Hello World'))btn=QPushButton()btn.setText('click me')btn.clicked.connect(onButtonClicked)w.layout().addWidget(btn)w.show()
A GUI quickly becomes complex if programmed line-by-line only. We prefer to use the QDesigner. It allows to
draw the GUI frontend which then is saved as *.ui XML file. This file can be translated into the PyQt code where you
just write the backend.
It is proposed that the well known GDAL raster metadata handling is extented to QgsRasterLayer objects via the custom
layer property interface. This approach is refered to as QGIS PAM (in analogy to the existing GDAL PAM).
By default, information is fetched from QGIS PAM first, and, if not available, from GDAL PAM as usual.
Edits to QGIS PAM are stored as map layer custom properties (QgsMapLayer.customProperties),
which are stored in-memory at the layer object, inside the QGIS project, and inside layer QML style file.
API support for QGIS PAM management will be implemented in the enmapboxprocessing.rasterreader.RasterReader class.
GDAL organises raster metadata in a PAM (Persistent Auxiliary Metadata) *.aux.xml file.
Reading and writing GDAL PAM items is done via gdal.Dataset and gdal.Band objects.
Currently we restrict metadata handling to GDAL raster. Metadata IO is implemented using the GDAL API:
This is fully sufficient for raster processing, but can be limiting in GUI applications,
where we usually don’t use gdal.Dataset, but QgsRasterLayer objects.
Problem: QgsRasterLayer class can’t access GDAL PAM directly.
We need to re-open the layer source via gdal.Open, which is usually fine for read-access.
But when writing new items to the GDAL PAM aux.xml file, the approach fails,
because QGIS will overwrite the aux.xml file again, when the QgsRasterLayer object is closed.
Technically, we don’t need any new functions or methods, because we fully rely on the layer custom property interface.
But, the handling of property keys, and QGIS PAM over GDAL PAM priority, can be tedious and should be encapsulated in utils
functions or methods. An example implementation is given by the RasterReader class.
It is proposed to implement functions or methods for reading and writing spectral properties from/to
QGIS PAM. Relevant items are: center wavelength, wavelength units, full width at half maximum (fwhm) and
bad band multiplier values.
Both, QGIS and GDAL, don’t have a concept for spectral property management.
In GDAL we have rudimentary support for ENVI files.
We currently use the ENVI format as a conceptual base for spectral property handling in the EnMAP-Box.
The here proposed approach will integrate spectral property handling into QGIS PAM management,
while honoring the well known ENVI format and naming conventions.
This allows to set/update spectral properties for QgsRasterLayer objects,
which is critical for GUI applications.
API support will be implemented in the enmapboxprocessing.rasterreader.RasterReader class.
GDAL will dump all items into the ENVI dataset domain, including relevant spectral properties like the list of
center wavelength, the wavelength units, the list of full width at half maximum (fwhm) values,
and the list of bad band multipliers (bbl):
Band-wise spectral properties like wavelength, wavelength units, full width at half maximum (fwhm),
and bad band multiplier values, are fetched with the following priorisation:
1. Look at QGIS PAM band default domain.
This is mainly relevant for GUI applications, where we need to set/update spectral properties using QgsRasterLayer objects:
2. Look at GDAL PAM band default domain.
This follows the behaviour of the ENVI driver, that sets wavelength and wavelength units to this location.
We consequently also look for fwhm and bad band multiplier here:
3. Look at GDAL PAM dataset ENVI domain.
This follows the behaviour of the ENVI driver, that sets lists of wavelength, fwhm and bad band multipliers (bbl) and the wavelength units to this location:
4. Look at GDAL PAM dataset default domain::
This follows the behaviour of the ENVI driver, that set the wavelength units to this location.
We consequently also look for lists of wavelength, fwhm and bad band multipliers (bbl) here:
Note that the parseEnviString function is assumed to parse ENVI list string like ‘{1, 2, 3}’ into Python lists like [1, 2, 3].
Guide line 1:
If you need to set band-wise spectral properties in a processing algorithm:
set it to the GDAL PAM band default domain.
This way, i) the information is accessible with the GDAL API,
and ii) consecutive band subsetting via gdal.Translate and gdal.BuildVrt can easily copy the band domains to the destination dataset.
Guide line 2:
If you need to set/update metadata in a GUI application: set it to QGIS PAM.
This is most flexible and secure.
The spectral properties are i) available as custom layer properties,
ii) stored in the QGIS project,
and iii) can be saved to QML layer style files.
Guide line 3:
Do not update GDAL PAM *.aux.xml file,
while the corresponding source is opened as a QgsRasterLayer in QGIS.
QGIS will potentially overwrite any changes, when closing the layer.
Technically, we don’t need any new functions or methods, because we fully rely on QGIS PAM.
But, the handling of property keys, and the assurance of fetching priorities,
can be tedious and should be encapsulated in util functions or methods.
An example implementation is given by the RasterReader class.
To query spectral properties for band 42 (in nanometers), we can use:
It is proposed to implement functions or methods for reading and writing temporal properties from/to QGIS PAM,
while also concidering the existing QgsRasterLayerTemporalProperties layer properties for QgsRasterLayer.
A survey of software (QGIS, GDAL, ENVI, Google Earth Engine) and EO product metadata formats (Landsat, Sentinel-2, EnMAP, DESIS)
shows that there is no clear naming convention for temporal properties. We observe two types of temporal annotation:
a) specification of a scene center time, and b) specification of a temporal range (start time and end time).
The specification of scene center time may be sufficient for individual images (e.g. a single Landsat scene),
but is not suitable for larger area products like daily MODIS composite products,
or other derived cloud-free composites using scenes from a temporal range (e.g. a full year).
Thus, we propose to use the more general temporal range concept.
Usecases where a center time is sufficient, may just specify the start time.
We further propose to support spectral-temporal timeseries stacks
(e.g. a stack of Landsat images, or a stack of cloud-free full-year NDVI composites).
This would allow proper plotting of spectral-temporal timeseries profiles,
which is currently (v3.9) not well supported in the EnMAP-Box.
API support will be implemented in the enmapboxprocessing.rasterreader.RasterReader class.
The GDAL Raster Data Model suggests to specify ACQUISITIONDATETIME as dataset-level IMAGERY-domain metadata:
# see https://gdal.org/user/raster_data_model.html#imagery-domain-remote-sensing<PAMDataset><Metadatadomain="IMAGERY"><MDIkey="ACQUISITIONDATETIME">2021-12-24T12:30:42.123</MDI></Metadata>
ENVI files may specify the acquisition time as dataset-level ENVI-domain metadata:
Since some software use the TIFFTAG_DATETIME to store file creation/manipulation time,
and EO data provider don’t use it to specify acquisition time, we propose to ignore it.
Google Earth Engine
Google Earth Engine images specify system:time_start and system:time_end properties in milliseconds since Unix Epoch (1970-01-01T00:00:00):
The survey shows that there is no clear naming convention for temporal properties. We observe two types of temporal annotation:
a) specification of a scene center time, and b) specification of a temporal range (start time and end time).
All formats only take single scene images into account.
None of the formats is suitable for specifying temporal properties of a timeseries stack
(e.g. a stack of Landsat images, or a stack of cloud-free full-year NDVI composites),
where each band may have an individual temporal range.
The here proposed approach will integrate temporal property handling into QGIS PAM management,
while honoring well known software format and naming conventions, and available QgsRasterLayerTemporalProperties information.
This allows to set/update temporal properties for QgsRasterLayer objects, which is critical for GUI applications.
It also takes care of information stored as GDAL PAM.
A key feature is the support for timeseries stacks,
that is a prerequisite for proper plotting of spectral-temporal timeseries data,
which is not well supported in the EnMAP-Box.
We currently (v3.9) only support single content timeseries stacks (e.g. a stack of NDVI bands).
where time information is specified as decimal years in the ENVI-domain wavelength item.
This quite hacky approach is well known in the ENVI Classic community for creating temporal profile plots.
We propose the following approach for fetching band-specific temporal properties.
Band-wise temporal start time and end time properties are fetched with the following priorisation:
1. Look at QGIS PAM band-level default-domain.
This is mainly relevant for GUI applications, where we need to set/update temporal properties using QgsRasterLayer objects:
3. Look at GDAL PAM dataset-level IMAGERY-domain.
This follows the GDAL Raster Data Model specification, that assumes the ACQUISITIONDATETIME to be set to this location:
Note that the parseDateTimeString function is assumed to parse timestamps into QDateTime objects.
It is proposed to support the following formats:
2021-12-24# date2021-12-24T12:30:42.123...# date time1640349042123# Unix epoche timestamp in milliseconds since 1970-01-01T00:00:00.000
Also note that we don’t support the various sensor product and software naming conventions presented in the survey above.
We assume that the acquisition time is properly set to the GDAL PAM dataset-level IMAGERY-domain during product import.
Guide line 1:
If you need to set band-wise temporal properties in a processing algorithm:
set it to the GDAL PAM band-level default-domain.
This way, i) the information is accessible with the GDAL API,
and ii) consecutive band subsetting via gdal.Translate and gdal.BuildVrt can easily copy the band domains to the destination dataset.
Guide line 2:
If you need to set/update metadata in a GUI application: set it to QGIS PAM.
This is most flexible and secure.
The temporal properties are i) available as custom layer properties,
ii) stored in the QGIS project,
and iii) can be saved to QML layer style files.
Guide line 3:
Do not update GDAL PAM *.aux.xml file,
while the corresponding source is opened as a QgsRasterLayer in QGIS.
QGIS will potentially overwrite any changes, when closing the layer.
Technically, we don’t need any new functions or methods, because we fully rely on QGIS PAM and the QgsRasterLayerTemporalProperties.
But, the handling of property keys, and the assurance of fetching priorities,
can be tedious and should be encapsulated in util functions or methods.
An example implementation is given by the RasterReader class.
To query temporal properties for band 42, we can use:
fromenmapboxprocessing.rasterreaderimportRasterReaderreader=RasterReader(layer)startTime=reader.startTime(42)endTime=reader.endTime(42)centerTime=reader.centerTime(42)# derives temporal range center time
In case of a standart image, where all bands share the same time range, you may skip the band number:
Band property management is a cruitial, but a not well support raster layer preparation step in QGIS and the EnMAP-Box.
We need for example:
band name information to subset or match raster bands by name instead of band numbers, and for setting proper names for reports
wavelength information for plotting spectral profiles
time information for plotting temporal profiles
bad band multipliers to exclude noisy bands from specific plotting/analysis tasks
data offset and scale information for scaling data stored as integer into 0 to 1 reflectance range
Spectral and Temporal Properties is already detailed in RFC 2 and 3.
Note that band no data value handling is fully supported by QGIS API directly via QgsRasterDataProvider methods:
setNoDataValue, setUserNoDataValue, setUseSourceNoDataValue, sourceHasNoDataValue, sourceNoDataValue, userNoDataValues, useSourceNoDataValue.
Problem
Band name, offset and scale information can be accessed via QgsRasterLayer methods,
but can’t be modified. This is especially limiting in GUI applications, where we usually have a read-only handle to
a raster source that we aren’t supposed to modify.
The here proposed approach will integrate band property handling into QGIS PAM management.
This allows to set/update band properties for QgsRasterLayer objects, which is critical for GUI applications.
It also takes care of information stored as GDAL PAM.
We propose the following approach for fetching band-specific properties.
Note that fetching Spectral and Temporal Properties is already detailed in RFC 2 and 3.
Band-wise properties are fetched with the following priorisation.
1. Look at QGIS PAM band-level default-domain.
This is mainly relevant for GUI applications, where we need to set/update band properties using QgsRasterLayer objects:
Note that those information is only concidered by EnMAP-Box applications and algorithms and always ignored by QGIS and GDAL.
To manifest those changes you are required to translate the layer to an intermediate VRT using the EnMAP-Box Translate raster layer algorithm,
which will transfer all QGIS PAM information to GDAL PAM.
2. Use QgsRasterLayer and QgsRasterDataProvider methods for accessing the band name, offset and scale::
This is mainly relevant for processing algorithms:
Note that when reading raster band data, the application or algorithm is responsible for properly applying the band offset and scale.
Notice that GDAL won’t scale data automatically, when calling gdal.Band.ReadAsArray().
On the contrary, QGIS will automatically scale the data using the QgsRasterDataProvider.bandOffset and QgsRasterDataProvider.bandScale information,
but will ignore potential modifications stored in QGIS PAM.
For that reason, we highly encourage the use of the RasterReader.array methode for reading raster data, which will take care of all the data scaling details.
Guide line 1:
If you need to set band-wise properties in a processing algorithm:
set it to the GDAL PAM band-level default-domain or use approriate gdal.Band methods.
This way, i) the information is accessible with the GDAL API,
and ii) consecutive band subsetting via gdal.Translate and gdal.BuildVrt can easily copy the band domains to the destination dataset.
Guide line 2:
If you need to set/update band properties in a GUI application: set it to QGIS PAM.
This is most flexible and secure.
The band properties are i) available as custom layer properties,
ii) stored in the QGIS project,
and iii) can be saved to QML layer style files.
Note that those modifications are only used by EnMAP-Box applications and algorithms, QGIS and GDAL will ignore it!
To manifest those changes in GDAL PAM, translate the layer to an intermediate VRT using the EnMAP-Box Translate raster layer algorithm.
Guide line 3:
Do not update GDAL PAM *.aux.xml file,
while the corresponding source is opened as a QgsRasterLayer in QGIS.
QGIS will potentially overwrite any changes, when closing the layer.
Technically, we don’t need any new functions or methods, because we fully rely on existing QGIS/GDAL API functionality.
But, the handling of property keys, the assurance of fetching priorities, and proper data scaling,
can be tedious and should be encapsulated in util functions or methods.
An example implementation is given by the RasterReader class.
Note that Spectral and Temporal Properties is already detailed in RFC 2 and 3.
This documentation guideline provides an overview of the general style guidelines and on how to contribute to the documentation. It is based on the
QGIS documentation guidelines.
The EnMAP-Box documentation is written in the reStructuredText(reST) format, coupled with some external libraries, e.g. Sphinx.
To keep things consistent, the following chapter contains guidelines on how to style the EnMAP-Box documentation.
Using the reST format for the documentation, allows to create several webpages with chapters, sections and subsections.
The section For Writers in the QGIS documentation guidelines guides you through the most important general rules for contributing to documentation, including the design of headings, inline tags and the insertion of captions and references.
Furthermore, you might want to take a look at the Sphinx Python Documentation Generator Guide that
describes the first steps using Sphinx and reStructuredText in more detail and also gives some examples of how to include e.g. images, source code or hyperlinks.
In addition to the two guidelines mentioned above, we have developed some simple rules for EnMap Box documentation, which are described below.
Use numbering to describe individual steps. This makes it is easier to distinguish
between information and actual steps and to follow them easily:
1. Enable the *Editing mode* by activating |mActionToggleEditing|.
2. Now you can use the |mActionNewAttribute| button to add a new field (mind the type!)
...
Info boxes, tooltips and notes should be aligned with the preceding text, i.e., if the info box follows a bulleted list, the box should also be indented.
Example of placing info boxes within / after bullet points
Images should always be centred, but can vary in size (adjust according to readability).
Also add a caption to the image if it is not placed between two bullet points.
*Placement of images for (left) after body text, (right) within bullet points *
symbol once, which should usually change the Skip setting to installing the most recent
version.
or uncheck 
the checkbox
and confirm with OK
icon or from the menubar 
icon
in the QGIS toolbar or via from the menubar.
Furthermore, the EnMAP-Box Processing Algorithms provider is available in the Processing Toolbox.
Open a map view button to add a second map view. This view will appear below the
first map view (Map #1).
button or go to and select
Link map scale and center.
or holding the mouse wheel 
) and notice how both map views are synchronized now.
Open Raster Layer Styling button, which will open
a new panel. Here you can quickly change the renderer (e.g., singleband gray, RGB) and the band(s) visualized. You can
do so manually using the slider or by selecting the buttons with predefined wavelength regions based on Sentinel-2 (e.g.
G = Green, N = Near infrared).
The raster layer needs to have wavelength information for the latter to work!
. You will find several band
combination presets. Select Colour infrared.
on it) you can use the arrow keys ←/→ to
switch back and forth between bands!
. The result should be listed in the Data Sources panel.
on the toolbar and check or uncheck the desired
toolbar.
Raster Data
for default raster layers (continuous value range)
for mask raster layers
for classification raster layers
Vector Data
Spectral Libraries
Models
Synchronize Data Sources with QGIS
button. Now they should appear in the data source panel and can be added to a Map View.
selects profiles from the top-most raster layer by default. The Profile Source panel allows to change this behaviour
and to control:
Identify cursor location value option activated and click somewhere in the map view.
Select CRS button (e.g. for switching to lat/long coordinates).
Link map scale and center
Link map scale
Link map center
.
button in the toolbar or open a new window from the menu .
and 
to keep the candidate profile in the spectral library. Activate Add profiles automatically 
to collect multiple profiles and display them all in the same spectral library.
and enable the Editing mode 
.
dialog to add a new column.
.
to open the spectral library properties.
button.
button.
and search for spectralData or spectralMath in the Spectral Libraries tab.
and choose the desired algorithm, e.g. Spectral resampling.
button.
.
allows you to export all or selected profiles as Geopackage (.gpkg), GeoJSON (.geoson) or ENVI Spectral Library (.sli).
button.
button to open the file selection dialog.
button to manually add classes.
button.
button.
Copy Classes
Paste Classes buttons.
button or go to the Layer Properties ().
The vector dataset is rasterized/burned on-the-fly onto the grid of the input raster in order to extract the sample.
If the vector source is a polygon dataset, only polygons which cover more than 75% of a pixel in the target grid are rasterized.
Specify the sample size per class, either relative in percent or in absolute pixel counts.
Model parameters: Specify the parameters of the selected classifier.
.
Note that the mouse wheel can be used alternatively for zooming (roll mouse wheel forward/backward) and panning (press and hold mouse wheel).
icon on the blue Map #1 title bar.
option. Click on an image pixel. SpectralLibrary #1 opens automatically displaying the respective
pixel spectrum (green line). The plotted spectrum always refers to the top-most raster layer of the respective Map
Window you click on (unless you change this in the Spectral Profile Sources panel).
of pixels.
icon and navigate to the file directory
containing your data. Select 
from the blue Map title bar.
To re-dock a Map View, click and drag the blue Map title bar to an open Map View already docked in the EnMAP-Box window.
icon, select 
icon and choose Create regression dataset (from continuous-valued vector layer and feature raster).
A new widget will be opened. Run the dialog with the following inputs:
ImageMath
