5. Geo Services Guide¶
This guide covers the details of geo data management and retrieval in rasdaman. The rasdaman Array DBMS is domain agnostic; the specific semantics of space and time is provided through a layer on top of rasdaman, historically known as petascope. It offers spatio-temporal access and analytics through APIs based on the OGC data standard Coverage Implementation Schema (CIS) and the OGC service standards Web Map Service (WMS), Web Coverage Service (WCS), and Web Coverage Processing Service (WCPS).
Note
While the name petascope addresses a specific component we frequently use the name rasdaman to refer to the complete system, including petascope.
5.1. OGC Coverage Standards Overview¶
For operating rasdaman geo services as well as for accessing such geo services through these APIs it is important to understand the mechanics of the relevant standards. In particular, the concept of OGC / ISO coverages is important.
In standardization, coverages are used to represent space/time varying phenomena, concretely: regular and irregular grids, point clouds, and general meshes. The coverage standards offer data and service models for dealing with those. In rasdaman, focus is on multi-dimensional gridded (“raster”) coverages.
In rasdaman, the OGC standards WMS, WCS, and WCPS are supported, being reference implementation for WCS. These APIs serve different purposes:
- WMS delivers a 2D map as a visual image, suitable for consunmption by humans
- WCS delivers n-D data, suitable for further processing and analysis
- WCPS performs flexible server-side processing, filtering, analytics, and fusion on coverages.
These coverage data and service concepts are summarized briefly below; for specific details on coordinate reference system handling see also CRS definition management. Ample material is also available on the Web for familiarization with coverages (best consult in this sequence):
- hands-on demos for multi-dimensional coverage services provided by Jacobs University;
- a series of webinars and tutorial slides provided by EarthServer;
- a range of background information on these standards provided by OGC;
- the official standards documents maintained by OGC:
5.1.1. Coverage Data¶
OGC CIS specifies an interoperable, conformance-testable coverage structure independent from any particular format encoding. Encodings are defined in OGC in GML, JSON, RDF, as well as a series of binary formats including GeoTIFF, netCDF, JPEG2000, and GRIB2).
By separating the data definition (CIS) from the service definition (WCS) it is possible for coverages to be served throuigh a variety of APIs, such as WMS, WPS, and SOS. However, WCS and WCPS have coverage-specific functionality making them particularly suitable for flexible coverage acess, analytics, and fusion.
5.1.2. Coverage Services¶
OGC WMS delivers 2D maps generated from styled layers stacked up. As such, WMS is a visualization service sitting at the end of processing pipelines, geared towards human consumption.
OGC WCS, on the other hand, rpovides data suitable for further processing (including visualization); as such, it is suitable also for machine-to-machine communication as it appears in the middle of longer processing pipelines. WCS is a modular suite of service functionality on coverages. WCS Core defines download of coverages and parts thereof, through subsetting directives, as well as delivery in some output format requested by the client. A set of WCS Extensions adds further functionality facets.
One of those is WCS Processing; it defines the ProcessCoverages
request
which allows sending a coverage analytics request through the WCPS spatio-temporal
analytics language. WCPS supports extraction, analytics, and fusion of
multi-dimensional coverage expressed in a high-level, declarative,
and safe language.
5.2. OGC Web Services Endpoint¶
Once the petascope servlet is deployed (see rasdaman installation guide) coverages can be accessed through service endpoint /rasdaman/ows
.
Note
Endpoint /rasdaman/rasql
, which by default is also available after deploying
rasdaman, does not know about coverages and their services, but only knows domain-agnostic rasql.
For example, assuming that the service’s IP address is 123.456.789.1
and the
service port is 8080
, the following request URLs would deliver the
Capabilities documents for OGC WMS and WCS, respectively:
http://123.456.789.1:8080/rasdaman/ows?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0
http://123.456.789.1:8080/rasdaman/ows?SERVICE=WCS&REQUEST=GetCapabilities&VERSION=2.0.1
5.3. OGC Coverage Implementation Schema (CIS)¶
A coverage consists mainly of:
- domain set: provides information about where data sit in space and time. All coordinates expressed there are relative to the coverage’s Coordinate Reference System or Native CRS. Both CRS and its axes, units of measure, etc. are indiciated in the domain set. Petascope currently supports grid topologies whose axes are aligned with the axes of the CRS. Along these axes, grid lines can be spaced regularly or irregularly.
- range set: the “pixel payload”, ie: the values (which can be atomic, like in a DEM, or records of values, like in hyperspectral imagery).
- range type: the semantics of the range values, given by type information, null values, accuracy, etc.
- metadata: a black bag which can contain any data: the coverage will not understand these, but duly transport them along so that the connection between data and metadata is not lost.
Further components include Envelope
which gives a rough, simplified overview
on the coverage’s location in space and time and CoverageFunction
which is
unused by any implementation known to us.
5.3.1. Coordinate Reference Systems in Coverages¶
Every coverage, as per OGC CIS, must have exactly one Native CRS. Sometimes definitions for such CRSs are readily available, such as with the EPSG registry where 2-D WGS84 is readily available under its code EPSG:4326. In particular spatio-temporal CRSs, however, are not always readily available, at least not in all combinations of spatial and temporal axes. To this end, composition of CRS is supported so that the single Native CRS can be built from “ingredient” CRSs by concatenating these CRSs into a composite one.
For instance, a time-series of WGS84 images would have the following Native CRS:
http://localhost:8080/def/crs-compound?
1=http://localhost:8080/def/crs/EPSG/0/4326
&2=http://localhost:8080/def/crs/OGC/0/AnsiDate
Coordinate tuples in this CRS represent an ordered composition of a geospatial
CRS with Latitude followed by Longitude, as per EPSG:4326, followed by
a temporal coordinate expressed in ISO 8601 syntax,
such as: 2012-01-01T00:01:20Z
.
Several ways exist for determining the Native CRS of coverage domain set:
- in a WCS
GetCapabilities
response, check thewcs:CoverageSummary/ows:BoundingBox@crs
attribute;- in a WCS
DescribeCoverage
response, check the@srsName
attribute in thegml:domainSet
;- in WCPS, use function
crsSet(e)
to determine the CRS of a coverage expression e;
Note
In a coverage also consider the axisLabels
attributes giving the axis
names as used in the coverage, in proper sequence as per CRS;
the uomLabels
attribute contains the units of measure for each axis.
The following graphics illustrates, on the example of an image timeseries,
how dimension, CRS, and axis labels affect the domain set in
a CIS 1.0 RectifiedGridCoverage
.

Note
This handling of coordinates in CIS 1.0 bears some legacy burden from GML;
in the GeneralGridCoverage
introduced with CIS 1.1 coordinate handling is much simplified.
5.3.2. CRS Management¶
the Native CRS of a coverage is given by a URL, as per OGC convention. Resolving this URL should deliver the CRS definition. The OGC CRS resolver is one such service. Its implementation is running SECORE which is part of rasdaman community.
By providing the source code of the OGC resolver it is possible to deploy one’s own resolver under an own URL, such as http://rasdaman.org:8080/def/crs/EPSG/0/27700.
5.3.3. Range Type¶
Range values can be atomic or (possibly nested) records over atomic values, described by the range type. In rasdaman the following atomic data types are supported; all of these can be combined freely in records of values, such as in hyperspectral images or climate variables.
rasdaman type | size | Quantity types |
---|---|---|
boolean |
8 bit | unsignedByte |
octet |
8 bit | signedByte |
char |
8 bit | unsignedByte |
short |
16 bit | signedShort |
unsigned short
= ushort |
16 bit | unsignedShort |
long |
32 bit | signedInt |
unsigned long
= ulong |
32 bit | unsignedInt |
float |
32 bit | float32 |
double |
64 bit | float64 |
complex |
64 bit | cfloat32 |
complexd |
128 bit | cfloat64 |
5.3.4. Nil Values¶
Nil (“null”) values, as per SWE, are supported by rasdaman in an extended way:
- null values can be defined over any data type
- nulls can be single values
- nulls can be intervals
- a null definnition in a coverage can be a list of all of the above alternatives.
Note
It is highly recommended to NOT define null values over floating-point numbers as this causes numerical problems well known in mathematics. This is not related to rasdaman, but intrinsic to the nature and handling of floating-point numbers in computers. If really desired, a floating-point interval should be defined around the desired float null value (this corresponds to interval arithmetics in numerical mathematics).
5.4. OGC Web Coverage Service¶
WCS Core offers request types:
GetCapabilities
for obtaining a list of coverages offered together with an overall service description;DescribeCoverage
for obtaining information about a coverage without downloading it;GetCoverage
for downloading, extracting, and reformatting of coverages; this is the central workhorse of WCS.
WCS Extensions in part enhance GetCoverage
with additional functionality
controlled by further parameters, and in part establish new request types,
such as:
- WCS-T defining
InsertCoverage
,DeleteCoverage
, andUpdateCoverage
requests;- WCS Processing defining
ProcessCoverages
for submitting WCPS analytics code.
You can use http://localhost:8080/rasdaman/ows
as service endpoints to which to
send WCS requests, for example:
http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=GetCapabilities
See example queries in the WCS systemtest which send KVP (key value pairs) GET request and XML POST request to Petascope.
5.4.1. Subsetting behavior¶
In general, subsetting in petascope behaves similarly to subsetting in gdal, with a couple of deviations necessary for n-D. Specifically, subsetting follows the next rules:
Slicing (
geoPoint
): the grid slice with index corresponding to the requested slicing geo point is returned. This is computed as follows:gridIndex = floor((geoPoint - minGeoLowerBound) / axisResolution)
Trimming (
geoLowerBound
:geoUpperBound
): the lower bound of the grid interval is determined as in the case of slicing. The number of returned grid points follows gdal:+ If axis resolution is positive (e.g: ``Long`` axis): gridLowerBound = floor((geoLowerBound - minGeoLowerBound) / axisResolution) numberOfGridPixels = floor(((geoUpperBound - geoLowerBound) / axisResolution) + 0.5) gridUpperBound = gridLowerBound + numberOfGridPixels - 1 + If axis resolution is negative (e.g: ``Lat`` axis): gridLowerBound = floor((geoUpperBound - maxGeoLowerBound) / axisResolution) numberOfGridPixels = floor((geoLowerBound - geoUpperBound) / axisResolution) + 0.5) gridUpperBound = gridLowerBound + numberOfGridPixels - 1
Note
If a trimming subset on an axis with:
(geoUpperBound - geoLowerBound) / axisResolution < 0.5
then, lower grid bound is translated by the slicing formula and upper grid bound is set to lower grid bound.
For example: a 2D coverage has Long
(X) and Lat
(Y) axes with CRS EPSG:4326
.
The resolution for axis Long
is: 10
and the resolution for axis Lat
is: -10
.
The geo bounds of axis Long
are: [0:180]
and the geo bounds
of axis Lat
are [0:90]
.
Calculate slicing on
Long
axis by geo coordinates to grid coordinates:- Long(0): returns [0] - Long(9): returns [0] - Long(10): returns [1] - Long(15): returns [1] - Long(20): returns [2] - Long(40): returns [4] - Long(49.99999): returns [4] - Long(50.0): returns [5]
Calculate trimming on
Long
axis by geo coordinates to grid coordinates:- Long(0:5): returns [0:0] - Long(0:10): returns [0:0] - Long(0:14.999): returns [0:0] - Long(0:15): returns [0:1] - Long(0:24.999): returns [0:1] - Long(0:25.0): returns [0:2] - Long(9,11): returns [0:0]
5.4.2. CIS 1.0 to 1.1 Transformation¶
Under WCS 2.1 - ie: with SERVICE=2.1.0
- both DescribeCoverage``and ``GetCoverage
requests understand the proprietary parameter OUTPUTTYPE=GeneralGridCoverage
which formats the result as CIS 1.1 GeneralGridCoverage
even if it has been
imported into the server as a CIS 1.0 coverage, for example:
http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.1.0
&REQUEST=DescribeCoverage
&COVERAGEID=test_mean_summer_airtemp
&OUTPUTTYPE=GeneralGridCoverage
http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.1.0
&REQUEST=GetCoverage
&COVERAGEID=test_mean_summer_airtemp
&FORMAT=application/gml+xml
&OUTPUTTYPE=GeneralGridCoverage
5.4.3. Polygon/Raster Clipping¶
WCS and WCPS support clipping of polygons expressed in the
WKT format format.
Polygons can be MultiPolygon (2D)
, Polygon (2D)
and LineString (1D+)
.
The result is always a 2D coverage in case of MultiPolygon and Polygon, and
is a 1D coverage in case of LineString
.
Further clipping patterns include curtain
and corridor
on 3D+ coverages
from Polygon (2D)
and Linestring (1D)
.
The result of curtain
clipping has the same dimensionality as the input coverage
whereas the result of corridor
clipping is always a 3D coverage,
with the first axis being the trackline of the corridor by convention.
Below some examples are presented expaining the mimics for WCS.
Syntactically, clipping is expressed by adding a &CLIP=
parameter to the request.
If the SUBSETTINGCRS
parameter is specified then this CRS also applies
to the clipping WKT, otherwise it is assumed that the WKT is in the
Native coverage CRS.
5.4.3.1. Clipping Examples¶
Polygon clipping on coverage with Native CRS
EPSG:4326
, for example:Polygon clipping with coordinates in
EPSG:3857
(fromsubsettingCRS
parameter) on coverage with Native CRSEPSG:4326
, for example:Linestring clipping on a 3D coverage with axes
X
,Y
,ansidate
, for example:Multipolygon clipping on 2D coverage, for example:
Curtain clipping by a Linestring on 3D coverage, for example:
Curtain clipping by a Polygon on 3D coverage, for example:
Corridor clipping by a Linestring on 3D coverage, for example:
Corridor clipping by a Polygon on 3D coverage, for example:
5.4.4. WCS-T¶
Currently, WCS-T supports importing coverages in GML format. The metadata of the coverage is thus explicitly specified, while the raw cell values can be stored either explicitly in the GML body, or in an external file linked in the GML body, as shown in the examples below. The format of the file storing the cell values must be
- 2-D data supported by the GDAL library, such as TIFF / GeoTIFF, JPEG / JPEG2000, PNG, etc;
- n-D data in NetCDF or GRIB format
The coverage metadata must be in XML or JSON format; any other format will lead to an error.
In addition to the WCS-T standard parameters petascope supports additional proprietary parameters.
Note
For coverage management normally WCS-T is not used directly.
Rather, the more convenient wcst_import
Python importing tool
is recommended for Data Import.
5.4.4.1. Inserting coverages¶
Inserting a new coverage into the server’s WCS offerings is done using the
InsertCoverage
request.
Request Parameter | Value | Description | Required |
---|---|---|---|
SERVICE | WCS | service standard | Yes |
VERSION | 2.0.1 or later | WCS version used | Yes |
REQUEST | InsertCoverage | Request type to be performed | Yes |
INPUTCOVERAGEREF | {url} | URl pointing to the coverage to be inserted | One of inputCoverageRef or inputCoverage is required |
INPUTCOVERAGE | {coverage} | A coverage to be inserted | One of inputCoverageRef or inputCoverage is required |
USEID | new | existing | Indicates wheter to use the coverage’s id (“existing”) or to generate a new unique one (“new”) | No (default: existing) |
Request Parameter | Value | Description | Required |
---|---|---|---|
PIXELDATATYPE | GDAL supported base data type (eg: “Float32”) or comma-separated concatenated data types, (eg: “Float32,Int32,Float32”) | In cases where range values are given in the GML body the datatype can be indicated through this parameter. Default: Byte. | No |
TILING | rasdaman tiling clause, see wiki:Tiling | Indicates the array tiling to be applied during insertion | No |
The response of a successful coverage request is the coverage id of the newly inserted coverage. For example: The coverage available at http://schemas.opengis.net/gmlcov/1.0/examples/exampleRectifiedGridCoverage-1.xml can be imported with the following request:
http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1
&REQUEST=InsertCoverage
&COVERAGEREF=http://schemas.opengis.net/gmlcov/1.0/examples/exampleRectifiedGridCoverage-1.xml
The following example shows how to insert a coverage stored on the server on which rasdaman runs. The cell values are stored in a TIFF file (attachment:myCov.gml), the coverage id is generated by the server and aligned tiling is used for the array storing the cell values:
http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1
&REQUEST=InsertCoverage
&COVERAGEREF=file:///etc/data/myCov.gml
&USEID=new
&TILING=aligned[0:500,0:500]
5.4.4.2. Updating Coverages¶
Updating an existing coverage into the server’s WCS offerings is done using the UpdateCoverage
request.
Request Parameter | Value | Description | Required |
---|---|---|---|
SERVICE | WCS | service standard | Yes |
VERSION | 2.0.1 or later | WCS version used | Yes |
REQUEST | UpdateCoverage | Request type to be performed | Yes |
COVERAGEID | {string} | Identifier of the coverage to be updated | Yes |
INPUTCOVERAGEREF | {url} | URl pointing to the coverage to be inserted | One of inputCoverageRef or inputCoverage is required |
INPUTCOVERAGE | {coverage} | A coverage to be updated | One of inputCoverageRef or inputCoverage is required |
SUBSET | AxisLabel(geoLowerBound,geoUpperBound) | Trim or slice expression, one per updated coverage dimension | No |
The following example shows how to update an existing coverage test_mr_metadata
from a generated GML file by wcst_import
tool:
http://localhost:8080/rasdaman/ows?SERVICE=WCS&version=2.0.1
&REQUEST=UpdateCoverage
&COVRAGEID=test_mr_metadata
&SUBSET=i(0,60)
&subset=j(0,40)
&INPUTCOVERAGEREF=file:///tmp/4514863c_55bb_462f_a4d9_5a3143c0e467.gml
5.4.4.3. Deleting Coverages¶
The DeleteCoverage
request type serves to delete a coverage (consisting of
the underlying rasdaman collection, the associated WMS layer (if exists)
and the petascope metadata).
For example: The coverage test_mr
can be deleted as follows:
http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1
&REQUEST=DeleteCoverage
&COVERAGEID=test_mr
5.4.4.4. Renaming coverage¶
The UpdateCoverageId
request allows to update
a coverage’s id and the associated WMS layer if one exists (v10.0+).
For example, the coverage test_mr
can be renamed to test_mr_new
as follows:
http://localhost:8080/rasdaman/admin/UpdateCoverageId?
COVERAGEID=test_mr
&NEWID=test_mr_new
5.4.5. Coverage Metadata Update¶
Coverage metadata can be updated through the interactive rasdaman WSClient
by selecting a text file (MIME type one of: text/xml
, application/json
,
text/plain
) containing new metadata and upload it to petascope.
Then, petascope will read the content of the text file and update corresponding
coverage’s metadata.
Note
This WSClient feature is login protected: OGC WCS > Describe Coverage tab when one is already logged in with petascope admin user in Admin tab.
The service URL for this feature is http://localhost:8080/rasdaman/admin/UpdateCoverageMetadata
which operates through multipart/form-data POST requests. The request should
contain 2 parts: the first part is coverageId to update, the second part is a
path to a local text file to be uploaded to server.
Alternatively, one can use REST API to update a coverage metadata with
petascope admin user’s credentials via basic authentication headers method
by curl tool. For example: Metadata of coverage test_mr_metadata
will be updated from the local XML file at /home/rasdaman/Downloads/test_metadata.xml
:
curl --user petauser:PETASCOPE_ADMIN_PASSWORD
-F "coverageId=test_mr_metadata"
-F "file=@/home/rasdaman/Downloads/test_metadata.xml"
"http://localhost:8080/rasdaman/admin/UpdateCoverageMetadata"
5.5. OGC Web Coverage Processing Service (WCPS)¶
The OGC Web Coverage Processing Service (WCPS) standard defines a protocol-independent language for the extraction, processing, analysis, and fusion of multi-dimensional gridded coverages, often called datacubes.
5.5.1. General¶
WCPS requests can be submitted in both abstract syntax (example) and in XML (example).
For example, using the WCS GET/KVP protocol binding a WCPS request can be sent
through the following ProcessCoverages
request:
http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1
&REQUEST=ProcessCoverage&QUERY=<wcps-query>
Note
In v10+, rasdaman supports the q
parameter besides the query
parameter
to store a WCPS query. A request must contain only one q
or query
parameter.
5.5.2. Polygon/Raster Clipping¶
The proprietary clip()
function with the same effect as clipping is available with WCPS.
The signature is as follows:
clip( coverageExpression, wkt [, subsettingCrs ] )
where
- coverageExpression
is an expression of result type coverage, eg: dem + 10
;
- wkt
is a valid WKT (Well-Known Text) expression, e.g. POLYGON((...))
, LineString(...)
;
- subsettingCrs
is an optional CRS in which the wkt``coordinates are expressed, eg: ``http://opengis.net/def/crs/EPSG/0/4326
.
5.5.2.1. Clipping Examples¶
Polygon clipping with coordinates in
EPSG:4326
on coverage with Native CRSEPSG:3857
:Linestring clipping on 3D coverage with axes
X
,Y
,datetime
.Linestring clipping on 2D coverage with axes
X
,Y
.Multipolygon clipping on 2D coverage.
Curtain clipping by a Linestring on 3D coverage
Curtain clipping by a Polygon on 3D coverage
Corridor clipping by a Linestring on 3D coverage
Corridor clipping by a Polygon on 3D coverage (geo CRS:
EPSG:4326
) with input geo coordinates inEPSG:3857
.
5.5.3. Auto-ratio for scaling X or Y axis in WCPS¶
As aproprietary extension, the scale()
function in WCPS allows to specify
the target extent of only one of the spatial horizontal axes, instead of both.
In this case, the extent of the other axis will be determined automatically
while preserving the original ratio between the two spatial axes.
For example in the request below, petascope will automatically set
the extent of Lat
to a value that preserves the ratio in the output result:
5.5.4. Implicitly add full domains in scale() for unspecified non X/Y axes in WCPS¶
The scale()
function in WCPS will implicitly add the full domains
of unspecified non X/Y axes of a given coverage in the scales’ domain intervals.
In other words, unspecified non X/Y axes will not be scaled.
For example in this query below, if a coverage is 3D and one only
specified X (E)
and Y (N)
axes as scale’s domain intervals,
then the time
axis’s domains will be added implicitly in petascope.
5.5.5. Automatic domain extraction¶
The domain interval can be extracted from a domain, including an
imageCrsDomain
(in modern nomenclature: index domain).
Both the interval - ie: [lowerBound:upperBound]
- and lower as well
as upper bound can be retrieved for each axis.
Syntax:
operator(.lo|.hi)?
with .lo
or .hi
returning the lower bound or upper bound of this interval.
Coverage test_eobstest has 3 dimensions with extent (0:5,0:29,0:39).
Expression imageCrsdomain(c,Long) in this case returns 0:39
whereas imageCrsdomain(c,Long).hi returns 39.
Further, the third argument of the domain()
operator, the CRS URL,
is now optional. If not specified, domain()
will use the CRS of the
selected axis (ie, the second argument) instead.
5.5.6. LET clause in WCPS¶
An optional LET
clause is supported in WCPS queries.
It allows binding alias variables to valid WCPS query sub-expressions,
and subsequently make use of the variables in the return
clause
instead of repeating the aliased sub-expressions.
The syntax is
FOR-CLAUSE
LET $variable := assignment [ , $variable := assignment ]
...
[ WHERE-CLAUSE ]
RETURN-CLAUSE
where
assignment ::= coverageExpression | domainExpression
for $c in (test_mr)
let $a := $c[i(0:50), j(0:40)],
$b := avg($c) * 2
return
encode( scale( $c, { imageCrsDomain( $c ) } ) + $b, "image/png" )
A special shorthand subset expression allows to conveniently specify domains. The variable in the LET clause follows this syntax:
LET $variable := [ dimensionalIntervalList ]
This can readily be used in a subset expression:
coverageVariable[$variable1]
for $c in (test_mr)
let $a := [i(20), j(40)],
$b := 10
return
encode( $c[ $a ] + $b, "itext/json" )
5.5.7. Binary min, max operations in WCPS¶
Since v10+, rasdaman supports binary min()
and max()
operations.
For two compatible coverages A and B they calculate a result coverage
with the minimum / maximum for each pair of corresponding cell values of A and B.
Example:
min(A, B)
where A and B are coverage expressions.
5.5.8. Positional parameter in WCPS¶
Since v10+, rasdaman supports positional parameters in WCPS (non-standard). Positional parameters allow to reference binary or string values in a WCPS query. The binary/string values are specified in a POST request, in addition to the WCPS query.
5.5.8.1. Syntax¶
The syntax for the positional parameter is:
$POSITIVE_INTEGER e.g: $1, $2,..
The endpoint to send WCPS query by POST with extra values is:
localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=ProcessCoverages
with the mandatory parameter query
and optional positional parameters: 1
, 2
,…
The value of a positional parameter can be either binary file content
or string value.
5.5.8.2. Example¶
One can use curl
tool as a client to send a WCPS request with
positional parameters to rasdaman via POST. curl
reads the contents
of the selected files automatically via file paths with @
parameter.
For example, one wants to combine an existing coverage ($c
variable)
with two temporary covarages ($d
and $e
variables) in png
format
from positional parameters $1
, $2
and $3
respectively :
curl -s "http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=ProcessCoverages" \
-F 'query=for $c in (existing_coverage), $d in (decode($1)), $e in (decode($2)) return encode(($c + $d + $e)[Lat(0:90), Long(-180:180)], "$3"))' \
-F "1=@/home/rasdaman/file1.tiff" \
-F "2=@/home/rasdaman/file2.tiff" \
-F "3=png" \
> ~/Downloads/test.png
5.5.9. Decode Operator in WCPS¶
Since v10+, rasdaman supports the non-standard decode()
operator
in WCPS. This feature allows one to combine existing coverages with
temporary coverages created in memory from input files attached
in POST request body.
Since v10+, rasdaman supports the non-standard positional parameters
5.5.9.1. Syntax¶
The syntax for the decode()
operator is:
decode(${positional_parameter})
with ${positional_parameter)
refers to the indicated files in the POST request.
See positional parameters in WCPS.
Note
Currently, WCPS decode()
operator supports only 2D geo-referenced
files readable by GDAL. One way to check if a file is readable by GDAL is
with gdalinfo ${file}
. netCDF/GRIB
format is not supported.
5.5.9.2. Example¶
5.5.10. Case Distinction¶
As another proprietary extension, conditional evaluation is added to WCPS following the overall XQuery-oriented syntax.
5.5.10.1. Syntax¶
SWITCH
CASE condExp RETURN resultExp
[ CASE condExp RETURN resultExp ]*
DEFAULT RETURN resultExpDefault
where condExp
and resultExp
are either scalar-valued or
coverage-valued expressions.
5.5.10.2. Constraints¶
- All condition expressions must return either boolean values or boolean coverages
- All result expressions must return either scalar values, or coverages
- The domain of all condition expressions must be the same
- The domain of all result expressions must be the same (that means same extent, resolution/direct positions, crs)
5.5.10.3. Evaluation Rules¶
If the result expressions return scalar values, the returned scalar value on a
branch is used in places where the condition expression on that branch evaluates
to True
. If the result expressions return coverages, the values of the returned
coverage on a branch are copied in the result coverage in all places where the
condition coverage on that branch contains pixels with value True
.
The conditions of the statement are evaluated in a manner similar to the IF-THEN-ELSE statement in programming languages such as Java or C++. This implies that the conditions must be specified by order of generality, starting with the least general and ending with the default result, which is the most general one. A less general condition specified after a more general condition will be ignored, as the expression meeting the less general expression will have had already met the more general condition.
Furthermore, the following hold:
domainSet(result)
=domainSet(condExp1)
metadata(result)
=metadata(condExp1)
rangeType(result)
=rangeType(resultExp1)
. In case resultExp1 is a scalar, the result range type is the range type describing the coverage containing the single pixel resultExp1.
5.5.10.4. Examples¶
switch
case $c < 10 return {red: 0; green: 0; blue: 255}
case $c < 20 return {red: 0; green: 255; blue: 0}
case $c < 30 return {red: 255; green: 0; blue: 0}
default return {red: 0; green: 0; blue: 0}
The above example assigns blue to all pixels in the $c coverage having a value less than 10, green to the ones having values at least equal to 10, but less than 20, red to the ones having values at least equal to 20 but less than 30 and black to all other pixels.
switch
case $c > 0 return log($c)
default return 0
The above example computes log of all positive values in $c, and assigns 0 to the remaining ones.
switch
case $c < 10 return $c * {red: 0; green: 0; blue: 255}
case $c < 20 return $c * {red: 0; green: 255; blue: 0}
case $c < 30 return $c * {red: 255; green: 0; blue: 0}
default return {red: 0; green: 0; blue: 0}
The above example assigns blue: 255 multiplied by the original pixel value to all pixels in the $c coverage having a value less than 10, green: 255 multiplied by the original pixel value to the ones having values at least equal to 10, but less than 20, red: 255 multiplied by the original pixel value to the ones having values at least equal to 20 but less than 30 and black to all other pixels.
5.5.11. CIS 1.0 to CIS 1.1 Transformation¶
For output format application/gml+xml
WCPS supports delivery
as CIS 1.1 outputType=GeneralGridCoverage
by specifying an additional
proprietary parameter outputType
in the encode()
function.
for c in (test_irr_cube_2)
return encode( c, "application/gml+xml",
"{\"outputType\":\"GeneralGridCoverage\"}" )
5.6. OGC Web Map Service (WMS)¶
The OGC Web Map Service (WMS) standard provides a simple HTTP interface for requesting overlays of geo-registered map images, ready for display.
With petascope, geo data can be served simultaneously via WMS, WCS, and WCPS. Further information:
5.6.1. WMS GetMap: Special Functionality¶
5.6.1.1. Transparency¶
By adding a parameter transparent=true
to WMS requests the returned image
will have NoData Value=0
in the metadata indicating to the client
that all pixels with value 0 value should be considered transparent for PNG
encoding format.
5.6.1.2. Interpolation¶
If in a GetMap
request the output CRS requested is different from
the coverage’s Native CRS petascope will duly reproject the map applying
resampling and interpolation. The algorithm used can be controlled with
the proprietary GetMap
parameter interpolation={method}
;
default is nearest-neighbour interpolation.
See Geographic projection for the methods available and their meaning.
5.6.2. 3D+ Coverages as WMS Layers¶
Petascope allows to import a 3D+ coverage as a WMS layer.
To this end, the ingrdients file used for wcst_import
must contain wms_import": true
. This works for 3D+ coverages
with recipes regular_time_series, irregular_time_series, and general_coverage recipes.
This example
demonstrates how to define an irregular_time_series 3D coverage from 2D GeoTIFF files.
Once the coverage is ingested, a GetMap
request
can use the additional (non-horizontal) axes for subsetting according
to the OGC WMS 1.3.0 standard.
Axis Type | Subset parameter | |
---|---|---|
Time | time=… | |
Elevation | elevation=… | |
Other | dim_AxisName=… (e.g dim_pressure=…) |
According to the WMS 1.3.0 specification, the subset for non-geo-referenced axes can have these formats:
- Specific value (value1): time=‘2012-01-01T00:01:20Z, dim_pressure=20,…
- Range values (min/max): time=‘2012-01-01T00:01:20Z’/‘2013-01-01T00:01:20Z, dim_pressure=20/30,…
- Multiple values (value1,value2,value3,…): time=‘2012-01-01T00:01:20Z, ‘2013-01-01T00:01:20Z, dim_pressure=20,30,60,100,…
- Multiple range values (min1/max1,min2/max2,…): dim_pressure=20/30,40/60,…
Note
AGetMap
request always returns a 2D result. If a non-geo-referenced axis is omitted from the request it will be considered as a slice on the upper bound along this axis. For example, in a time-series the youngest timeslice will be delivered).
Examples:
- Multiple values on time axis of 3D coverage.
- Multiple values on time, dim_pressure axes of 4d coverage.
5.6.3. WMS Layer Management¶
Additional proprietary requests, beyond the WMS standard, allow for service maintenance.
Layers can be easily created from existing coverages in WCS in two ways:
- By specifying WMS setup during import coverage in the respective ingredients file; see wms_import;
- By sending an InsertWCSLayer HTTP request to petascope.
The following proprietary WMS request types serve to manage the WMS offering of rasdaman:
InsertWCSLayer
: create a new WMS layer from an existing coverage.
UpdateWCSLayer
: update an existing WMS layer from an existing coverage which associates with this WMS layer.
- A layer can be removed either directly with a
DeleteLayer
request (since rasdaman v10.0), or indirectly when deleting a coverage (removing the associated WCS coverage). TheDeleteLayer
request is of the form:
5.6.4. WMS Style Management¶
Styles can be created for layers using rasql and WCPS query fragments. This allows users to define several visualization options for the same dataset in a flexible way. Examples of such options would be color classification, NDVI detection etc. The following HTTP request will create a style with the name, abstract and layer provided in the KVP parameters below
Note
For Tomcat version 7+ it requires the query (WCPS/rasql fragment) to be URL-encoded correctly. This site offers such an encoding service.
5.6.4.1. Style Definition Variants¶
WCPS query fragment example (since rasdaman 9.5):
Variable
$c
will be replaced by a layer name when sending aGetMap
request containing this layer’s style.Rasql query fragment examples:
Variable
$Iterator
will be replaced with the actual name of the rasdaman collection and the whole fragment will be integrated inside the regularGetMap
request.Multiple layers can be used in a style definition. Besides the iterators
$c
in WCPS query fragments and$Iterator
in rasql query fragments, which always refer to the current layer, other layers can be referenced by name using an iterator of the form$LAYER_NAME
in the style expression.Example: create a WCPS query fragment style referencing 2 layers (
$c
refers to layer sentinel2_B4 which defines the style):Then, in any
GetMap
request using this style the result will be obtained from the combination of the 2 layers sentinel2_B4 and sentinel2_B8:WMS styling supports a
ColorTable
definition which allows to colorize the result of WMS GetMap request when the style is requested. A style can contain either one or both query fragment and Color Table definitions. TheInsertStyle
request supports two new non-standard extra parameterscolorTableType
(valid values:ColorMap
,GDAL
andSLD
) andcolorTableDefintion
containing corresponding definition, example:Below the supported color table definitions for each color table type are explained:
Rasdaman
ColorMap
: check Coloring Arrays for more details. The color table definition must be a JSON object, for example:GDAL
ColorPalette
: check encode for more details. The color table definition must be a JSON object and contains 256 color arrays incolorTable
array, example:WMS
Styled Layer Descriptor (SLD)
: The color table definition must be valid XML and containsColorMap
element. Check Coloring Arrays for details about the supported types (ramp
(default),values
,intervals
), exampleColorMap
withtype="values"
:
5.6.4.2. WMS Style Removal¶
The proprietary DeleteStyle
WMS request type allows to remove
a particular style of an existing WMS layer.
http://localhost:8080/rasdaman/ows?SERVICE=WMS&VERSION=1.3.0
&REQUEST=DeleteStyle
&LAYER=dessert_area
&NAME=FireMarkup
5.6.5. Testing a WMS Setup¶
A rasdaman WMS service can be tested with any conformant client through
a GetMap
request like the following:
5.6.6. Errors and Workarounds¶
- Cannot load new WMS layer in QGIS
- In this case, the problem is due to QGIS caching the WMS GetCapabilities from the last request so the new layer does not exist (see clear caching solution).
5.6.7. WMS Pyramid Management¶
The following proprietary WMS requests are used to create and delete
downscaled coverages. Internally they are used for efficient zooming in/out
in WMS, and downscaling when using the scale()
function in WCPS
or scaling extension in WCS.
InsertScaleLevel
: create a downscaled collection for a specific coverage and given level; e.g. to create a downscaled coverage of test_world_map_scale_levels that is 4x smaller:DeleteScaleLevel
: delete an existing downscaled coverage at a given level; e.g. to delete the downscaled level 4 of coverage test_world_map_scale_levels:
wcst_import
can send InsertScaleLevel
requests automatically
when importing data with it with scale_levels
option
in the ingredients file, more details here.
5.7. Data Import¶
Raster data in a variety of formats, such as TIFF, netCDF, GRIB, etc.
can be imported in petascope through the wcst_import.sh
utility.
Internally it is based on WCS-T
requests, but hides the complexity and
maintains the geo-related metadata in its so-called petascopedb
while the raster data get ingested into the rasdaman array store.
Building large timeseries/datacubes, mosaics, etc. and keeping them up-to-date as new data arrive available is supported for a large variety of data formats and file/directory organizations.
The systemtest contains many examples for importing different types of data.
5.7.1. Introduction¶
The wcst_import.sh
tool introduces two concepts:
- - Recipe - A recipe is a class implementing the BaseRecipe that based on a set of
- parameters (ingredients) can import a set of files into WCS forming a well defined coverage (mosaic, regular timeseries, irregular timeseries, etc);
- - Ingredients - An ingredients file is a JSON file containing a set of parameters
- that define how the recipe should behave (e.g. the WCS endpoint, the coverage name, etc.)
To execute an ingredients file in order to import some data:
$ wcst_import.sh path/to/my_ingredients.json
Alternatively, wcst_import.sh
tool can be started as a daemon as follows:
$ wcst_import.sh path/to/my_ingredients.json --daemon start
or as a daemon that is “watching” for new data at some interval (in seconds):
$ wcst_import.sh path/to/my_ingredients.json --watch <interval>
For further informations regarding wcst_import.sh
commands and usage:
$ wcst_import.sh --help
The workflow behind is depicted approximately on Figure 5.1.
An ingredients file showing all options possible can be found here; in the same directory there are several examples of different recipes.
5.7.2. Recipes¶
The following recipes are provided in the rasdaman repository:
- Mosaic map
- Regular timeseries
- Irregular timeseries
- General coverage
- Import from external WCS
- Specialized recipes
For each one of these there is an ingredients example under the ingredients/ directory, together with an example for the available parameters Further on each recipe type is described in turn.
5.7.2.1. Common Options¶
Some options are commonly applicable to all recipes.
config section
service_url
- The endpoint of the WCS service with the WCS-T extension enabledmock
- Print WCS-T requests but do not execute anything if set totrue
. Set tofalse
by default.automated
- Set totrue
to avoid any interaction during the ingestion process. Useful in production environments for automated deployment for example. By default it isfalse
, i.e. user confirmation is needed to execute the ingestion.blocking
(since v9.8) - Set tofalse
to analyze and import each file separately (non-blocking mode). By default blocking is set totrue
, i.e. wcst_import will analyze all input files first to create corresponding coverage descriptions, and only then import them. The advantage of non-blocking mode is that the analyzing and importing happens incrementally (in blocking mode the analyzing step can take a long time, e.g. days, before the import can even begin).Note
When importing in non-blocking import mode for coverages with irregular axes, it will only rely on sorted files by filenames and it can fail if these axes’ coefficients are collected from input files’ metadata (e.g: DateTime value in TIFF’s tag or GRIB metadata) as they might not be consecutive. wcst_import will not analyze all files to collect metadata to be sorted by DateTime as in default blocking import mode.
default_null_values
- This parameter adds default null values for bands that do not have a null value provided by the file itself. The value for this parameter should be an array containing the desired null value either as a closed intervallow:high
or single values. E.g. for a coverage with 3 bandsNote
If you specify a null value interval (e.g:
"9.96921e+35:*"
) inwcst_import.sh
, note that during encode it will not be preserved in the encoded result like tiff because null value intervals are not supported by most formats. In this case it is recommended to specify in addition a non-interval null value, which will be used for encode.Note, if set this parameter will override the null/nodata values present in the input files.
tmp_directory
- Temporary directory in which gml and data files are created; should be readable and writable by rasdaman, petascope and current user. By default this is/tmp
.crs_resolver
- The crs resolver to use for generating WCS-T request. By default it is determined from thepetascope.properties
setting.url_root
- In case the files are exposed via a web-server and not locally, you can specify the root file url here; the default value is"file://"
.skip
- Set totrue
to ignore files that failed to import; by default it isfalse
, i.e. the ingestion is terminated when a file fails to import.retry
- Set totrue
to retry a failed request. The number of retries is either 5, or the value of settingretries
if specified. This is set tofalse
by default.retries
- Control how many times to retry a failed WCS-T request; set to 5 by default.retry_sleep
- Set number of seconds to wait before retrying after an error; a floating-point number can also be specified for sub-second precision. Default values is 1.track_files
- Set totrue
to allow files to be tracked in order to avoid reimporting already imported files. This setting is enabled by default.resumer_dir_path
- The directory in which to store the track file. By default it will be stored next to the ingredients file.slice_restriction
- Limit the slices that are imported to the ones that fit in a specified bounding box. Each subset in the bounding box should be of form{ "low": 0, "high": <max> }
, where low/high are given in the axis format. Example:description_max_no_slices
- maximum number of slices (files) to show for preview before starting the actual ingestion.subset_correction
(deprecated since rasdaman v9.6) - In some cases the resolution is small enough to affect the precision of the transformation from domain coordinates to grid coordinates. To allow for corrections that will make the import possible, set this parameter totrue
.
recipes/options section
import_order
- Allow to sort the input files (ascending
(default) ordescending
).Currently, it sorts by datetime which allows to import coverage from the first date or the recent date. Example:tiling
- Specifies the tile structure to be created for the coverage in rasdaman. You can set arbitrary tile sizes for the tiling option only if the tile name isALIGNED
. Example:For more information on tiling please check the Storage Layout Language
wms_import
- If set totrue
, after importing data to coverage, it will also create a WMS layer from the imported coverage and populate metadata for this layer. After that, this layer will be available from WMS GetCapabilties request. Example:
scale_levels
- Enable the WMS pyramids feature. Level must be positive number and greater than 1. Syntax:
hooks section
Since v9.8, wcst_import allows to run bash commands before/after ingestion
by adding optional hooks
configuration in an ingredient file.
There are 2 types of ingestion hooks:
before_ingestion
: run bash commands before analyzing input file(s) (e.g: using gdalwarp to reproject input file(s) from EPSG:3857 CRS to EPSG:4326 CRS and import projected EPSG:4326 input file(s)) to a coverage.after_ingestion
: run bash commands after importing input file(s) to coverage (e.g: clean all projected file(s) from gdalwarp command above).
When importing mode is set to non-blocking ("blocking": false
),
wcst_import will run before/after hook(s) for the file which
is being used to update coverage, while the default blocking importing mode
will run before/after hook(s) for all input files before/after
they are updated to a coverage. Parameters are explained below.
Example: Import GDAL subdatasets
The example ingredients below contains a pre-hook which replaces the collected file path into a GDAL subdataset form; in this particular case, with the GDAL driver for NetCDF a single variable from the collected NetCDF files is imported.
5.7.2.2. Mosaic map¶
Well suited for importing a tiled map, not necessarily continuous; it will place all input files given under a single coverage and deal with their position in space. Parameters are explained below.
5.7.2.3. Regular timeseries¶
Well suited for importing multiple 2-D slices created at regular intervals of time (e.g sensor data, satelite imagery etc) as 3-D cube with the third axis being a temporal one. Parameters are explained below
5.7.2.4. Irregular timeseries¶
Well suited for importing multiple 2-D slices created at irregular intervals of time into a 3-D cube with the third axis being a temporal one. There are two types of time parameters in “options”, one needs to be choosed according to the particular use case:
tag_name
withTIFFTAG_DATETIME
inside image’s metadata (can be checked with gdalinfo filename, not every image has this parameters). Here is an example with the “tag_name” optionfilename
allows an arbitrary pattern to extract the time information from the data file paths. Here is an example with the “filename” option
5.7.2.5. General coverage¶
The general recipe aims to be a highly flexible recipe that can handle any kind of data files (be it 2D, 3D or n-D) and model them in coverages of any dimensionality. It does that by allowing users to define their own coverage models with any number of bands and axes and fill the necesary coverage information through the so called ingredient sentences inside the ingredients.
5.7.2.5.1. Ingredient Sentences¶
An ingredient expression can be of multiple types:
- Numeric - e.g.
2
,4.5
- Strings - e.g.
'Some information'
- Functions - e.g.
datetime('2012-01-01', 'YYYY-mm-dd')
- Expressions - allows a user to collect information from inside the ingested
file using a specific driver. An expression is of form
${driverName:driverOperation}
- e.g.${gdal:minX}
,${netcdf:variable:time:min
. You can find all the possible expressions here. - Any valid python expression - You can combine the types below into a python
expression; this allows you to do mathematical operations, some string parsing
etc. - e.g.
${gdal:minX} + 1/2 * ${gdal:resolutionX}
ordatetime(${netcdf:variable:time:min} * 24 * 3600)
5.7.2.5.2. Parameters¶
Using the ingredient sentences we can define any coverage model directly in the options of the ingredients file. Each coverage model contains a
CRS
- the crs of the coverage to be constructed. Either a CRS url e.g. http://opengis.net/def/crs/EPSG/0/4326 or http://ows.rasdaman.org/def/crs-compound?1=http://ows.rasdaman.org/def/crs/EPSG/0/4326&2=http://ows.rasdaman.org/def/crs/OGC/0/AnsiDate or the shorthand notationsCRS1@CRS2@CRS3
, e.g.EPSG/0/4326@OGC/0/AnsiDate
metadata
- specifies in which format you want the metadata (json or xml). It can only contain characters and in petascope the datatype for this field is CLOB (Character Large Object). For postgresql (the default DBMS for petascopedb) this field is generated by Hibernate as LOB, for which the maximum size is 2GB (source).
global - specifies fields which should be saved (e.g. the licence, the creator etc) once for the whole coverage. Example:
local - specifies fields which are fetched from each input file to be stored in coverage’s metadata. Then, when subsetting output coverage, only associated local metadata will be added to the result. Example:
colorPaletteTable - specifies the path to a Color Palette Table (.cpt) file which can be used internally when encoding coverage to PNG to colorize result. Example:
Since v10, general recipe with slicer
gdal
readscolorPaletteTable
automatically if the first input file (TIFF format with Color Table (RGB with 256 entries)) contains this metadata whencolorPaletteTable
is set toauto
or not specified in the ingredients file. IfcolorPaletteTable
is set to empty, this metadata is ignored when creating coverage’s global metadata.
slicer
- specifies the driver (netcdf, gdal or grib) to use to read from the data files and for each axis from the CRS how to obtain the bounds and resolution corresponding to each file.Note
“type”: “gdal” is used for TIFF, PNG, and other 2D formats.
An example for the netCDF format can be found here and for PNG here. Here’s an example ingredient file for grib data:
5.7.2.5.3. Expressions¶
Each driver allows expressions to extract information from input files.
We will mark with capital letters things that vary in the expression.
E.g. ${gdal:metadata:YOUR_FIELD}
means that you can replace
YOUR_FIELD
with any valid gdal metadata tag (e.g. a TIFFTAG_DATETIME
)
Netcdf
Take a look at this NetCDF example for a general recipe ingredient file that uses many netcdf expressions.
Type | Description | Examples |
---|---|---|
Metadata information | ${netcdf:metadata:YOUR_METADATA_FIELD} |
${netcdf:metadata:title} |
Variable information | ${netcdf:variable:VARIABLE_NAME:MODIFIER}
where VARIABLE_NAME can be any variable in the
file and MODIFIER can be one of:
first|last|max|min; Any extra modifiers will return
the corresponding metadata field on the given
variable |
${netcdf:variable:time:min}
${netcdf:variable:t:units} |
Dimension information | ${netcdf:dimension:DIMENSION_NAME}
where DIMENSION_NAME can be any dimension in the
file. This will return the value on the selected
dimension. |
${netcdf:dimension:time} |
GDAL
For TIFF, PNG, JPEG, and other 2D data formats we use GDAL. Take a look at this GDAL example for a general recipe ingredient file that uses many GDAL expressions.
Type | Description | Examples |
---|---|---|
Metadata information | ${gdal:metadata:METADATA_FIELD} |
${gdal:metadata:TIFFTAG_NAME} |
Geo Bounds | ${gdal:BOUND_NAME} where BOUND_NAME can be
one of the minX|maxX|minY|maxY |
${gdal:minX} |
Geo Resolution | ${gdal:RESOLUTION_NAME} where RESOLUTION_NAME
can be one of the resolutionX|resolutionY |
${gdal:resolutionX} |
Origin | ${gdal:ORIGIN_NAME} where ORIGIN_NAME can be
one of the originX|originY |
${gdal:originY} |
GRIB
Take a look at this GRIB example for a general recipe ingredient file that uses many grib expressions.
Type | Description | Examples |
---|---|---|
GRIB Key | ${grib:KEY} where KEY can be any of the
keys contained in the GRIB file
${grib:messagenumber} is the special value
to get the current processed GRIB message index
(starting from 1) |
${grib:experimentVersionNumber} |
File
Type | Description | Examples |
---|---|---|
File Information | ${file:PROPERTY} where property can be one of
path|name|dir_path|original_path|original_dir_path
original_* allows to get the original input file’s path/directory
(used only when using pre-hook with replace_path
to replace original input file paths with customized file paths). |
${file:path} |
Special Functions
A couple of special functions are available to deal with some more complicated cases:
Function Name | Description | Examples |
---|---|---|
grib_datetime(date,time) |
This function helps to deal with the usual grib date and time format. It returns back a datetime string in ISO format. | grib_datetime(${grib:dataDate},
${grib:dataTime}) |
datetime(date, format) |
This function helps to deal with strange date time formats. It returns back a datetime string in ISO format. | datetime("20120101:1200",
"YYYYMMDD:HHmm") |
regex_extract(input, regex,
group) |
This function extracts information from a string using regex; input is the string you parse, regex is the regular expression, group is the regex group you want to select | datetime(regex_extract('${file:name}',
'(.*)_(.*)_(.*)_(\\d\\d\\d\\d-\\d\\d)
(.*)', 4), 'YYYY-MM') |
replace(input, old, new) |
Replaces all occurrences of a substring with another substring in the input string | replace('${file:path}','.tiff', '.xml') |
Using libraries in expressions
In case, the expression is complex, one can import the python libraries as
statements
in the ingredients file to simplify the calculation.
For example, one needs to calculate the lower bound and upper bound
for the time axis ansi
(starting days from 1978-12-31T12:00:00
).
This can be done by using the useful modules: datetime
and timedelta
from python datatime
library.
Note
The special utility function datetime(date_time_string, format)
to convert
a string of datetime (e.g: "20120101:1200"
) with an input format
(e.g: "YYYYMMDD:HHmm"
) to an ISO date time format will be overridden
when datetime
module is imported in statements
setting.
In this case, one will use the functionalities from python datetime
module instead of parsing and calculating datetime values.
Band’s unit of measurement (uom) code for netCDF and GRIB recipes
- In netCDF recipes you can add uom for each band by referencing the metadata
key of the specific variable. For example, for variable
LAI
:
In GRIB recipes adding uom for bands is same as for netCDF, except that a GRIB expression is used to fetch this information from metadata in the GRIB file. Example:
Local metadata from input files
Beside the global metadata of a coverage, you can add local metadata for each file which is a part of the whole coverage (e.g a 3D time-series coverage mosaiced from 2D GeoTiff files).
In ingredient file of general recipe, under the metadata section add a “local” object with keys and values extracted by using format type expression. Example of extracting an attribute from a netCDF input file:
Afterwards, each file’s envelope (geo domain) and its local metadata
will be added to the coverage metadata under <slice>...</slice>
element
if coverage metadata is imported in XML format. Example of a coverage
containing local metadata in XML from 2 netCDF files:
Since v10.0, local metadata for input files can be fetched from corresponding
external text files using the optional metadata_file
setting. For example:
When subsetting a coverage which contains local metadata section from input files (via WC(P)S requests), if the geo domains of subsetted coverage intersect with some input files’ envelopes, only local metadata of these files will be added to the output coverage metadata.
For example: a GetCoverage
request with a trim such that
crs axis subsets are within netCDF file 1:
The coverage’s metadata result will contain only local metadata from netCDF file 1:
Customized axis labels in coverage
This feature is available since rasdaman version 9.8 for general recipe.
Before, axis labels for a coverage must match axis abbreviations in CRS’s GML
definition when they are configured in the ingredient file under section "slicer"/"axes"
.
With this new feature, one can set an arbitrary name for each axis label by
adding optional configuration "crsOrder"
for each axis accordingly the
position index which starts from 0 of axis in coverage’s CRS.
For example with below configuration, coverage will be created with
3 customized axes MyDateTimeAxis, MyLatAxis and MyLongAxis
based on
coverage’s CRS (AnsiDate (1 DateTime axis) and EPSG:4326 (Lat and Long axes)):
Group several coverage slices into a group
Since v9.8+, wcst_import allows to group input files on irregular axes
(with "dataBound": false
) by optional sliceGroupSize: value (positive integer)
.
E.g:
If each input slice corresponds to index X, and one wants to have slice
groups of size N, then the index would be translated with this option to
X - (X % N)
.
Typical use case is importing 3D coverage from 2D satellite imageries where
time axis is irregular and its values are fetched from input files
by regex expression. Then, all input files which belong to 1 time window
(e.g: "sliceGroupSize"
: 7 (7 days in AnsiDate CRS) will have the same value
which is the first date of this week).
Bands and dimensions metadata in global metadata
Metadata can be individually specified for each band and axis in the ingredient file. Example:
Since v9.7, for this metadata can be automatically derived from the input netCDF files.
band metadata:
For netCDF: If
"bands"
is set to"auto"
or does not exist under"metadata"
in the ingredient file, all user-specified bands will have metadata which is fetched directly from the netCDF file. Metadata for 1 band is collected automatically if: 1) band is not added. 2) band is set to"auto"
.Otherwise, the user could specify metadata explicitly by a dictionary of keys/values. Example:
axis metadata:
For netCDF: If
"axes"
is set to"auto"
or does not exist under"metadata"
in the ingredient file, all user-specified axes will have metadata which is fetched directly from the netCDF file. Metadata for 1 axis is collected automatically if: 1) axis is not added. 2) axis is set to"auto"
. 3) axis is set with${netcdf:variable:DimensionName:metadata}
. The axis label for variable is detected from themin
ormax
value of CRS axis configuration under"slicer/axes"
section. For example:Otherwise, the user could specify metadata explicitly by a dictionary of keys/values.
5.7.2.6. Import from external WCS¶
Allows to import a coverage from a remote petascope endpoint into the local petascope. Parameters are explained below.
5.7.2.7. Import Sentinel-1 Data¶
This is a convenience recipe for importing Sentinel 1 data in particular; currently only GRD/SLC product types are supported, and only geo-referenced tiff files. Below is an example:
The recipe extends general_coverage so
the "recipe"
section has the same structure. However, a lot of information
is automatically filled in by the recipe now, so the ingredients file is much
simpler as the example above shows.
The other obvious difference is that the "coverage_id"
is templated with
several variables enclosed in ${
and }
which are automatically replaced
to generate the actual coverage name during import:
modebeam
- the mode beam of input files, e.g.IW/EW
.polarisation
- single polarisation of input files, e.g:HH/HV/VV/VH
If the files collected by "paths"
are varying in any of these parameters,
the corresponding variables must appear somewhere in the "coverage_id"
(as
for each combination a separate coverage will be constructed). Otherwise, the
ingestion will either fail or result in invalid coverages. E.g. if all data’s mode beam
is IW
, but still different polarisations, the
"coverage_id"
could be "MyCoverage_${polarisation}"
;
In addition, the data to be ingested can be optionall filtered with the
following options in the "input"
section:
modebeams
- specify a subset of mode beams to ingest from the data, e.g. only theIW
mode beam; if not specified, data of all supported mode beams will be ingested.polarisations
- specify a subset of polarisations to ingest, e.g. only theHH
polarisation; if not specified, data of all supported polarisations will be ingested.
Limitations:
- Only GRD/SLC products are supported.
- Data must be geo-referenced.
- Filenames are assumed to be of the format:
s1[ab]-(.*?)-grd(.?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?).tiff
ors1[ab]-(.*?)-slc(.?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?).tiff
.
5.7.2.8. Import Sentinel-2 Data¶
This is a convenience recipe for importing Sentinel 2 data in particular. It relies on support for Sentinel 2 in more recent GDAL versions. Importing zipped Sentinel 2 is also possible and automatically handled.
Below is an example:
The recipe extends general_coverage so
the "recipe"
section has the same structure. However, a lot of information
is automatically filled in by the recipe now, so the ingredients file is much
simpler as the example above shows.
The other obvious difference is that the "coverage_id"
is templated with
several variables enclosed in ${
and }
which are automatically replaced
to generate the actual coverage name during import:
crsCode
- the CRS EPSG code of the imported files, e.g.32757
for WGS 84 / UTM zone 57S.resolution
- Sentinel 2 products bundle several subdatasets of different resolutions:10m
- bands B4, B3, B2, and B8 (base type unsigned short)20m
- bands B5, B6, B7, B8A, B11, and B12 (base type unsigned short)60m
- bands B1, B8, and B10 (base type unsigned short)TCI
- True Color Image (red, green, blue char bands); also 10m as it is derived from the B2, B3, and B4 10m bands.
level
-L1C
orL2A
If the files collected by "paths"
are varying in any of these parameters,
the corresponding variables must appear somewhere in the "coverage_id"
(as
for each combination a separate coverage will be constructed). Otherwise, the
ingestion will either fail or result in invalid coverages. E.g. if all data is
level L1C
with CRS 32757
, but still different resolutions, the
"coverage_id"
could be "MyCoverage_${resolution}"
; the other variables
can still be specified though, so "MyCoverage_${resolution}_${crsCode}"
is
valid as well.
In addition, the data to be ingested can be optionall filtered with the
following options in the "input"
section:
resolutions
- specify a subset of resolutions to ingest from the data, e.g. only the “10m” subdataset; if not specified, data of all supported resolutions will be ingested.levels
- specify a subset of levels to ingest, so that files of other levels will be fully skipped; if not specified, data of all supported levels will be ingested.crss
- specify a list of CRSs (EPSG codes as strings) to ingest; if not specified or empty, data of any CRS will be ingested.
5.7.2.9. Image pyramids¶
This feature (v9.7+) allows to create downscaled versions of a given coverage, eventually achieving something like an image pyramid, in order to enable faster WMS requests when zooming in/out.
By using the scale_levels option of wcst_import
when importing a coverage with WMS enabled, petascope will create downscaled
collections in rasdaman following this pattern: coverageId_<level>
.
If level is a float, then the dot is replaced with an underscore,
as dots are not permitted in a collection name. Some examples:
- MyCoverage, level 2 -> MyCoverage_2
- MyCoverage, level 2.45 -> MyCoverage_2_45
Example ingredients specification to create two downscaled levels which are 8x and 32x smaller than the original coverage:
Two new WCS-T non-standard requests are utilized by wcst_import for this feature, see here for more information.
5.7.2.10. Creating your own recipe¶
The recipes above cover a frequent but limited subset of what is possible to model using a coverage. WCSTImport allows to define your own recipes in order to fill these gaps. In this tutorial we will create a recipe that can construct a 3D coverage from 2D georeferenced files. The 2D files that we want to target have all the same CRS and cover the same geographic area. The time information that we want to retrieve is stored in each file in a GDAL readable tag. The tag name and time format differ from dataset to dataset so we want to take this information as an option to the recipe. We would also want to be flexible with the time crs that we require so we will add this option as well.
Based on this usecase, the following ingredient file seems to fulfill our need:
To create a new recipe start by creating a new folder in the recipes folder.
Let’s call our recipe my_custom_recipe
:
The last command is needed to tell python that this folder is containing python
sources, if you forget to add it, your recipe will not be automatically
detected. Let’s first create an example of our ingredients file so we get a
feeling for what we will be dealing with in the recipe. Our recipe will just
request from the user two parameters Let’s now create our recipe, by creating a
file called recipe.py
Use your favorite editor or IDE to work on the recipe (there are type annotations for most WCSTImport classes so an IDE like PyCharm would give out of the box completion support). First, let’s add the skeleton of the recipe (please note that in this tutorial, we will omit the import section of the files (your IDE will help you auto import them)):
The first thing you need to do is to make sure the get_name()
method returns
the name of your recipe. This name will be used to determine if an ingredient file
should be processed by your recipe. Next, you will need to focus on the
constructor. Let’s examine it. We get a single parameter called session
which
contains all the information collected from the user plus a couple more useful
things. You can check all the available methods of the class in the session.py
file, for now we will just save the options provided by the user that are
available in session.get_recipe()
in a class attribute.
In the validate()
method, you will validate the options for the recipe
provided by the user. It’s generally a good idea to call the super method to
validate some of the general things like the WCST Service availability and so on
although it is not mandatory. We also want to validate our custom recipe options
here. This is how the recipe looks like now:
Now that our recipe can validate the recipe options, let’s move to the describe()
method. This method allows you to let your users know any relevant information
about the ingestion before it actually starts. The irregular_timeseries recipe
prints the timestamp for the first couple of slices for the user to check if
they are correct. Similar behaviour should be done based on what your recipe has
to do.
Next, we should define the ingest behaviour. The framework does not make any
assumptions about how the correct method of ingesting is, however it offers a
lot of utility functionality that help you do it in a more standardized way. We
will continue this tutorial by describing how to take advantage of this
functionality, however, note that this is not required for the recipe to work.
The first thing that you need to do is to define an importer object. This
importer object, takes a coverage object and ingests it using WCST requests. The
object has two public methods, ingest()
, which ingests the coverage into the
WCS-T service (note: ingest can be an insert operation when the coverage was not
defined, or update if the coverage exists. The importer will handle both cases
for you, so you don’t have to worry if the coverage already exists.) and
get_progress()
which returns a tuple containing the number of imported slices and
the total number of slices. After adding the importer, the code should look like
this:
In order to build the importer, we need to create a coverage object. Let’s see how we can do that. The coverage constructor requires a
coverage_id
: the id of the coverageslices
: a list of slices that compose the coverage. Each slice defines the position in the coverage and the data that should be defined at the specified positionrange_fields
: the range fields for the coveragecrs
: the crs of the coveragepixel_data_type
: the type of the pixel in gdal format, e.g. Byte, Float32 etc
The coverage object can be built in many ways, we will present one such method. Let’s start from the crs of the coverage. For our recipe, we want a 3D crs, composed of the CRS of the 2D images and a time CRS as indicated. The following lines of code give us exactly this:
Let’s also get the range fields for this coverage. We can extract them again from the 2D image using a helper class that can use GDAL to get the relevant information:
Let’s also get the pixel base type, again using the gdal helper:
pixel_type = gdal_dataset.get_band_gdal_type()
Let’s see what we have so far:
As you can notice, the only thing left to do is to implement the _get_slices() method. To do so we need to iterate over all the input files and create a slice for each. Here’s an example on how we could do that
And we are done we now have a valid coverage object. The last thing needed is to define the status method. This method need to provide a status update to the framework in order to display it to the user. We need to return the number of finished work items and the number of total work items. In our case we can measure this in terms of slices and the importer can already provide this for us. So all we need to do is the following:
We now have a functional recipe. You can try the ingredients file against it and see how it works.
5.8. Data export¶
WCS formats are requested via the format KVP key (<gml:format>
elements for XML POST requests), and take a valid MIME type as value. Output
encoding is passed on to the the GDAL library, so the limitations on output
formats are devised accordingly by the supported raster formats of GDAL. The valid MIME types which
Petascope may support can be checked from the WCS 2.0.1 GetCapabilities
response:
In case of encode processing expressions, besides MIME types WCPS (and rasql) can also accept GDAL format identifiers or other commonly-used format abbreviations like “CSV” for Comma-Separated-Values for instance.
5.9. rasdaman / petascope Geo Service Administration¶
The petascope conpoment, which geo services contact through its OGC APIs, uses rasdaman for storing the raster arrays; geo-related data parts (such as geo-referencing), as per coverage standard, are maintained by petascope itself.
Petascope is implemented as a war file of Java servlets. Internally, incoming requests requiring coverage evaluation are translated by petascope, with the help of the coverage metadata, into rasql queries executed by rasdaman as the central workhorse. Results returned from rasdaman are forwarded by petascope to the client.
Note
rasdaman can maintain arrays not visible via petascope (such as non-geo objects like human brain images). Data need to be imported via Data Import, not rasql, for being visible as coverages.
For further internal documentation on petascope see Developer introduction to petascope and its metadata database.
5.9.1. Service Startup and Shutdown¶
- For external petascope and SECORE servlets, start/stop external tomcat which deploys these web applications.
- For embedded petascope and secore servlets normally get started and stopped
automatically through the standard scripts,
start_rasdaman.sh``and ``stop_rasdaman.sh
.
5.9.2. petascope Configuration¶
The petascope services are configured in file $RMANHOME/etc/petascope.properties
.
Note
For changes to take effect Tomcat needs to be restarted after editing this file.
5.9.3. petascope Security¶
In $RMANHOME/etc/petascope.properties
, by default only local IP addresses
configured by allow_write_requests_from
setting key are allowed to send
write requests to petascope (e.g: InsertCoverage, DeleteCoverage,…).
Any write requests from a not-listed IP address will be blocked.
However, if one has petascope admin user credentials, configured by
petascope_admin_user
and petascope_admin_pass
setting keys,
then one can send write requests with these credentials via basic authentication
header. Petascope will accept requests as they come from petascope admin user.
This typical case is used by WSClient, when one has logged in as petascope admin user and can send requests to delete a coverage or insert WMS styles for a WMS layer.
5.9.4. Meta Database Connectivity¶
Non-array data of coverages (here loosely called metadata) are stored in
another database, separate from the rasdaman database.
This backend is configured in $RMANHOME/etc/petascope.properties
.
As a first action it is strongly recommended to substitute {db-username} and {db-password} by some safe settings; keeping obvious values constitutes a major security risk.
Note that the choice is exclusive: only one such database can be used at any time. Changing to another database system requires a database migration which is entirely the responsibility of the service operator and involves substantially more effort than just changing these entries; generally, it is strongly discouraged to change the meta database backend.
If necessary, add the path to the JDBC jar driver to petascope.properties
using metadata_jdbc_jar_path
and spring.datasource.jdbc_jar_path
.
Several different systems are supported as metadata backends.
Below is a list of petascope.properties
settings for different systems
that have been tested successfully with rasdaman.
5.9.4.1. Postgresql (default)¶
The following configuration in petascope.properties
enables PostgreSQL
as metadata backend:
5.9.4.2. HSQLDB¶
The following configuration in petascope.properties
enables HSQLDB
as metadata backend:
5.9.5. petascope Standalone Deployment¶
The petascope and secore servlets can be deployed through any suitable
servlet container, or can be operated standalone using its built-in embedded container.
The embedded variant is activated through the directive java_server=embedded
in the respective configuration file.
Below are excerpts from the two configuration files affected showing how to configure this mode.
$RMANHOME/etc/petascope.properties
$RMANHOME/etc/secore.properties
In this standalone mode petascope and secore can be started individually using the central startup/shutdown scripts of rasdaman:
$ start_rasdaman.sh --service [secore | petascope]
$ stop_rasdaman.sh --service [secore | petascope]
Both servlets can beven be started from the command line:
$ java -jar rasdaman.war [ --petascope.confDir={path-to-properties-file} ]
$ java -jar def.war
- The port required by the embedded tomcat will be fetched from the
server.port
setting inpetascope.properties
. Assuming the port is set to 9009, petascope can be accessed via URLhttp://localhost:9009/rasdaman/ows
. - For secore, the port required by the embedded tomcat will be fetched
from the
server.port
setting insecore.properties
. Assuming the port is set to 9010, secore can be accessed via URLhttp://localhost:9010/def
.
Note
Configuration parameter secoredb.path
must be set in
secore.properties
file to a directoy where the effective
SECORE user has write access for creating the XML database files.
5.9.6. petascope Serving Static Content¶
Serving external static content (such as HTML, CSS, and Javascript)
residing outside rasdaman.war
through petascope can be enabled
with the following setting in petascope.properties
:
static_html_dir_path={absolute-path-to-index.html}
with an absolute path to a readable directory containing an index.html
.
This will be served as the root, ie: at URL http://localhost:8080/rasdaman/
.
5.9.7. Logging¶
Configuration file petascope.properties
also defines logging.
The log level can be adjusted in verbosity.
Tomcat restart is required for new settings to become effective.
Note
Make sure that Tomcat has write permissions on the petascope.log
file specified.
5.10. Geo Service Standards Compliance¶
rasdaman community is OGC WCS reference implementation and supports the following conformance classes:
- OGC CIS:
- CIS 1.0:
- Class GridCoverage
- Class RectifiedGridCoverage
- Class ReferenceableGridCoverage
- Class gml-coverage
- Class multipart-coverage
- CIS 1.1:
- Class grid-regular
- Class grid-irregular
- Class gml-coverage
- Class json-coverage
- Class other-format-coverage
- OGC WCS
- WCS 2.0:
- WCS Core
- WCS Range Subsetting
- WCS Processing (supporting WCPS 1.0)
- WCS Transaction
- WCS CRS
- WCS Scaling
- WCS Interpolation
- WMS 1.3.0:
- all raster functionality, including SLD
ColorMap
stylingNote
With WCS 2.1, petascope provides an additional proprietary parameter to request CIS 1.0 coverages to be returned as CIS 1.1 coverages. This is specified by adding parameter
outputType=GeneralGridCoverage
.