# 5. Geo Services Guide¶

This guide covers the details of geo data management and retrieval in rasdaman. The rasdaman Array DBMS is domain agnostic; the specific semantics of space and time is provided through a layer on top of rasdaman, historically known as petascope. It offers spatio-temporal access and analytics through APIs based on the OGC data standard Coverage Implementation Schema (CIS) and the OGC service standards Web Map Service (WMS), Web Coverage Service (WCS), and Web Coverage Processing Service (WCPS).

Note

While the name petascope addresses a specific component we frequently use the name rasdaman to refer to the complete system, including petascope.

## 5.1. OGC Coverage Standards Overview¶

For operating rasdaman geo services as well as for accessing such geo services through these APIs it is important to understand the mechanics of the relevant standards. In particular, the concept of OGC / ISO coverages is important.

In standardization, coverages are used to represent space/time varying phenomena, concretely: regular and irregular grids, point clouds, and general meshes. The coverage standards offer data and service models for dealing with those. In rasdaman, focus is on multi-dimensional gridded (“raster”) coverages.

In rasdaman, the OGC standards WMS, WCS, and WCPS are supported, being reference implementation for WCS. These APIs serve different purposes:

• WMS delivers a 2D map as a visual image, suitable for consunmption by humans

• WCS delivers n-D data, suitable for further processing and analysis

• WCPS performs flexible server-side processing, filtering, analytics, and fusion on coverages.

These coverage data and service concepts are summarized briefly below. Ample material is also available on the Web for familiarization with coverages (best consult in this sequence):

### 5.1.1. Coverage Data¶

OGC CIS specifies an interoperable, conformance-testable coverage structure independent from any particular format encoding. Encodings are defined in OGC in GML, JSON, RDF, as well as a series of binary formats including GeoTIFF, netCDF, JPEG2000, and GRIB2).

By separating the data definition (CIS) from the service definition (WCS) it is possible for coverages to be served throuigh a variety of APIs, such as WMS, WPS, and SOS. However, WCS and WCPS have coverage-specific functionality making them particularly suitable for flexible coverage acess, analytics, and fusion.

### 5.1.2. Coverage Services¶

OGC WMS delivers 2D maps generated from styled layers stacked up. As such, WMS is a visualization service sitting at the end of processing pipelines, geared towards human consumption.

OGC WCS, on the other hand, provides data suitable for further processing (including visualization); as such, it is suitable also for machine-to-machine communication as it appears in the middle of longer processing pipelines. WCS is a modular suite of service functionality on coverages. WCS Core defines download of coverages and parts thereof, through subsetting directives, as well as delivery in some output format requested by the client. A set of WCS Extensions adds further functionality facets.

One of those is WCS Processing; it defines the ProcessCoverages request which allows sending a coverage analytics request through the WCPS spatio-temporal analytics language. WCPS supports extraction, analytics, and fusion of multi-dimensional coverage expressed in a high-level, declarative, and safe language.

## 5.2. OGC Web Services Endpoint¶

Once the petascope geo service is deployed (see rasdaman installation guide) coverages can be accessed through the HTTP service endpoint /rasdaman/ows.

For example, assuming that the service IP address is 123.456.789.1 and the service port is 8080, the following request URLs would deliver the Capabilities documents for OGC WMS and WCS, respectively:

http://123.456.789.1:8080/rasdaman/ows?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.3.0
http://123.456.789.1:8080/rasdaman/ows?SERVICE=WCS&REQUEST=GetCapabilities&VERSION=2.0.1


## 5.3. OGC Coverage Implementation Schema (CIS)¶

A coverage consists mainly of:

• domain set: provides information about where data sit in space and time. All coordinates expressed there are relative to the coverage’s Coordinate Reference System or Native CRS. Both CRS and its axes, units of measure, etc. are indiciated in the domain set. Petascope currently supports grid topologies whose axes are aligned with the axes of the CRS. Along these axes, grid lines can be spaced regularly or irregularly.

• range set: the “pixel payload”, ie: the values (which can be atomic, like in a DEM, or records of values, like in hyperspectral imagery).

• range type: the semantics of the range values, given by type information, null values, accuracy, etc.

• metadata: a black bag which can contain any data: the coverage will not understand these, but duly transport them along so that the connection between data and metadata is not lost.

Further components include Envelope which gives a rough, simplified overview on the coverage’s location in space and time and CoverageFunction which is unused by any implementation known to us.

### 5.3.1. Coverage CRS¶

Every coverage, as per OGC CIS, must have exactly one native Coordinate Reference System (CRS), which is given by a URL. Resolving this URL should deliver the CRS definition. The OGC CRS resolver is an example of a public service for resolving CRS URLs; the same service is also bundled in every rasdaman installation, so that it is avialable locally. More details on this topic can be found in the CRS Management chapter.

Sometimes definitions for CRSs are readily available, such as the 2-D WGS84 with code EPSG:4326 in the EPSG registry. In particular spatio-temporal CRSs, however, are not always readily available, at least not in all combinations of spatial and temporal axes. To this end, composition of CRS is supported so that the single Native CRS can be built from “ingredient” CRSs by concatenating them into a composite one. For instance, a time-series of WGS84 images would have the following Native CRS:

http://localhost:8080/def/crs-compound?
1=http://localhost:8080/def/crs/OGC/0/AnsiDate
&2=http://localhost:8080/def/crs/EPSG/0/4326


Coordinate tuples in this CRS represent an ordered composition of a temporal coordinate expressed in ISO 8601 syntax, such as 2012-01-01T00:01:20Z, followed by latitude and longitude coordinates, as per EPSG:4326.

The native CRS of a coverage domain set can be determined in severay ways:

• in a WCS GetCapabilities response, the wcs:CoverageSummary/ows:BoundingBox@crs attribute;

• in a WCS DescribeCoverage response, the srsName attribute in the gml:domainSet element; Furthermore, the axisLabels attribute contains the CRS axis names according to the CRS sequency, and the uomLabels attribute contains the units of measure for each corresponding axis.

• in WCPS, the function crsSet(e) returns the CRS of a coverage expression e;

The following graphics illustrates, on the example of an image timeseries, how dimension, CRS, and axis labels affect the domain set in a CIS 1.0 RectifiedGridCoverage.

Note

This handling of coordinates in CIS 1.0 bears some legacy burden from GML; in the GeneralGridCoverage introduced with CIS 1.1 coordinate handling is much simplified.

### 5.3.2. Range Type¶

Range values can be atomic or (possibly nested) records over atomic values, described by the range type. In rasdaman the following atomic data types are supported; all of these can be combined freely in records of values, such as in hyperspectral images or climate variables.

Table 5.1 Mapping of rasdaman base types to SWE Quantity types

rasdaman type

size

Quantity types

boolean

8 bit

unsignedByte

octet

8 bit

signedByte

char

8 bit

unsignedByte

short

16 bit

signedShort

unsigned short = ushort

16 bit

unsignedShort

long

32 bit

signedInt

unsigned long = ulong

32 bit

unsignedInt

float

32 bit

float32

double

64 bit

float64

complex

64 bit

cfloat32

complexd

128 bit

cfloat64

### 5.3.3. Nil Values¶

Nil (null) values, as per SWE, are supported by rasdaman in an extended way:

• null values can be defined over any data type

• nulls can be single values

• nulls can be intervals

• a null definnition in a coverage can be a list of all of the above alternatives.

Full details can be found in the null values section.

Note

It is highly recommended to NOT define single null values over floating-point data as this causes numerical problems well known in mathematics. This is not related to rasdaman, but intrinsic to the nature and handling of floating-point numbers in computers. A floating-point interval around the desired float null value should be preferred (this corresponds to interval arithmetics in numerical mathematics).

### 5.3.4. Errors¶

Errors from OGC requests to rasdaman are returned to the client formatted as ows:ExceptionReport (OGC Common Specification). An ExceptionReport can contain multiple Exception elements. For example, when running a WCS GetCoverage or a WCPS query which execute rasql queries in rasdaman, in case of an error the ExceptionReport will contain two Exception elements:

1. One with the error message returned from rasdaman.

2. Another with the rasql query that failed.

For example:

<ows:ExceptionReport>
<ows:Exception exceptionCode="RasdamanRequestFailed">
<ows:ExceptionText>The Encode function is applicable to array arguments only.</ows:ExceptionText>
</ows:Exception>
<ows:Exception exceptionCode="RasdamanRequestFailed">
<ows:ExceptionText>Failed internal rasql query: SELECT encode(1, "png" ) FROM mean_summer_airtemp AS c</ows:ExceptionText>
</ows:Exception>
</ows:ExceptionReport>


## 5.4. OGC Web Coverage Service¶

WCS Core offers the following request types:

• GetCapabilities for obtaining a list of coverages offered together with an overall service description;

• DescribeCoverage for obtaining information about a coverage without downloading it;

• GetCoverage for downloading, extracting, and reformatting of coverages; this is the central workhorse of WCS.

WCS Extensions in part enhance GetCoverage with additional functionality controlled by further parameters, and in part establish new request types, such as:

• WCS-T defining InsertCoverage, DeleteCoverage, and UpdateCoverage requests;

• WCS Processing defining ProcessCoverages for submitting WCPS analytics code.

You can use http://localhost:8080/rasdaman/ows as service endpoints to which to send WCS requests, for example:

http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1&request=GetCapabilities


See example queries in the WCS systemtest which send KVP (key value pairs) GET request and XML POST request to Petascope.

### 5.4.1. Subsetting behavior¶

In general, subsetting in petascope behaves similarly to subsetting in gdal, with a couple of deviations necessary for n-D. Specifically, subsetting follows the next rules:

• Slicing (geoPoint): the grid slice with index corresponding to the requested slicing geo point is returned. This is computed as follows:

gridIndex = floor((geoPoint - minGeoLowerBound) / axisResolution)

• Trimming (geoLowerBound:geoUpperBound): the lower bound of the grid interval is determined as in the case of slicing. The number of returned grid points follows gdal:

• If axis resolution is positive (e.g. Long axis):

gridLowerBound = floor((geoLowerBound - minGeoLowerBound) / axisResolution)
numberOfGridPixels = floor(((geoUpperBound - geoLowerBound) / axisResolution) + 0.5)
gridUpperBound = gridLowerBound + numberOfGridPixels - 1

• If axis resolution is negative (e.g. Lat axis):

gridLowerBound = floor((geoUpperBound - maxGeoLowerBound) / axisResolution)
numberOfGridPixels = floor((geoLowerBound - geoUpperBound) / axisResolution) + 0.5)
gridUpperBound = gridLowerBound + numberOfGridPixels - 1


Note

If a trimming subset is applied on an axis with (geoUpperBound - geoLowerBound) / axisResolution < 0.5, then lower grid bound is translated by the slicing formula and upper grid bound is set to lower grid bound.

For example, a 2D coverage has Long (X) and Lat (Y) axes with CRS EPSG:4326. The resolution for axis Long is 10 and the resolution for axis Lat is -10. The geo bounds of axis Long are [0:180] and the geo bounds of axis Lat are [0:90].

• Calculate slicing on Long axis by geo coordinates to grid coordinates:

- Long(0):          returns [0]
- Long(9):          returns [0]
- Long(10):         returns [1]
- Long(15):         returns [1]
- Long(20):         returns [2]
- Long(40):         returns [4]
- Long(49.99999):   returns [4]
- Long(50.0):       returns [5]

• Calculate trimming on Long axis by geo coordinates to grid coordinates:

- Long(0:5):         returns [0:0]
- Long(0:10):        returns [0:0]
- Long(0:14.999):    returns [0:0]
- Long(0:15):        returns [0:1]
- Long(0:24.999):    returns [0:1]
- Long(0:25.0):      returns [0:2]
- Long(9,11): returns [0:0]


### 5.4.2. CIS 1.0 to 1.1 Transformation¶

Under WCS 2.1 - ie: with SERVICE=2.1.0 - both DescribeCoverage and GetCoverage requests understand the proprietary parameter OUTPUTTYPE=GeneralGridCoverage which formats the result as CIS 1.1 GeneralGridCoverage even if it has been imported into the server as a CIS 1.0 coverage, for example:

http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.1.0
&REQUEST=DescribeCoverage
&COVERAGEID=test_mean_summer_airtemp
&OUTPUTTYPE=GeneralGridCoverage

http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.1.0
&REQUEST=GetCoverage
&COVERAGEID=test_mean_summer_airtemp
&FORMAT=application/gml+xml
&OUTPUTTYPE=GeneralGridCoverage


### 5.4.3. Polygon/Raster Clipping¶

WCS and WCPS support clipping of polygons expressed in the WKT format format. Polygons can be MultiPolygon (2D), Polygon (2D) and LineString (1D+). The result is always a 2D coverage in case of MultiPolygon and Polygon, and is a 1D coverage in case of LineString.

Further clipping patterns include curtain and corridor on 3D+ coverages from Polygon (2D) and Linestring (1D). The result of curtain clipping has the same dimensionality as the input coverage whereas the result of corridor clipping is always a 3D coverage, with the first axis being the trackline of the corridor by convention.

In WCS, clipping is expressed by adding a &CLIP= parameter to the request. If the SUBSETTINGCRS parameter is specified then this CRS also applies to the clipping WKT, otherwise it is assumed that the WKT is in the Native coverage CRS. In WCPS, clipping is done with a clip function, much like in rasql.

Further information can be found in the rasql clipping section. Below we list examples illustrating the functionality in WCS and WCPS.

#### 5.4.3.1. Clipping Examples¶

• Polygon clipping on coverage with Native CRS EPSG:4326, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Polygon clipping with coordinates in EPSG:3857 (from subsettingCRS parameter) on coverage with Native CRS EPSG:4326, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Linestring clipping on a 3D coverage with axes X, Y, ansidate, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Multipolygon clipping on 2D coverage, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Curtain clipping by a Linestring on 3D coverage, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Curtain clipping by a Polygon on 3D coverage, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Corridor clipping by a Linestring on 3D coverage, for example:

• WCS:

▶ show

• WCPS:

▶ show

• Corridor clipping by a Polygon on 3D coverage, for example:

• WCS:

▶ show

• WCPS:

▶ show

Note

Subspace clipping is not supported in WCS or WCPS.

### 5.4.4. WCS-T¶

Currently, WCS-T supports importing coverages in GML format. The metadata of the coverage is thus explicitly specified, while the raw cell values can be stored either explicitly in the GML body, or in an external file linked in the GML body, as shown in the examples below. The format of the file storing the cell values must be

In addition to the WCS-T standard parameters petascope supports additional proprietary parameters, covered in the following sections.

Note

For coverage management normally WCS-T is not used directly. Rather, the more convenient wcst_import Python tool is recommended for Data Import.

#### 5.4.4.1. Inserting coverages¶

Inserting a new coverage into the server’s WCS offerings is done using the InsertCoverage request.

Table 5.2 WCS-T Standard Parameters

Request Parameter

Value

Description

Required

SERVICE

WCS

service standard

Yes

VERSION

2.0.1 or later

WCS version used

Yes

REQUEST

InsertCoverage

Request type to be performed

Yes

INPUTCOVERAGEREF

{url}

URl pointing to the coverage to be inserted

One of inputCoverageRef or inputCoverage is required

INPUTCOVERAGE

{coverage}

A coverage to be inserted

One of inputCoverageRef or inputCoverage is required

USEID

new | existing

Indicates wheter to use the coverage’s id (“existing”) or to generate a new unique one (“new”)

No (default: existing)

Table 5.3 WCS-T Proprietary Enhancements

Request Parameter

Value

Description

Required

PIXELDATATYPE

GDAL supported base data type (eg: “Float32”) or comma-separated concatenated data types, (eg: “Float32,Int32,Float32”)

In cases where range values are given in the GML body the datatype can be indicated through this parameter. Default: Byte.

No

TILING

rasdaman tiling clause, see wiki:Tiling

Indicates the array tiling to be applied during insertion

No

The response of a successful coverage request is the coverage id of the newly inserted coverage. For example: The coverage available at http://schemas.opengis.net/gmlcov/1.0/examples/exampleRectifiedGridCoverage-1.xml can be imported with the following request:

http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1
&REQUEST=InsertCoverage
&COVERAGEREF=http://schemas.opengis.net/gmlcov/1.0/examples/exampleRectifiedGridCoverage-1.xml


The following example shows how to insert a coverage stored on the server on which rasdaman runs. The cell values are stored in a TIFF file (attachment:myCov.gml), the coverage id is generated by the server and aligned tiling is used for the array storing the cell values:

http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1
&REQUEST=InsertCoverage
&COVERAGEREF=file:///etc/data/myCov.gml
&USEID=new
&TILING=aligned[0:500,0:500]


#### 5.4.4.2. Updating Coverages¶

Updating an existing coverage into the server’s WCS offerings is done using the UpdateCoverage request.

Table 5.4 WCS-T Standard Parameters

Request Parameter

Value

Description

Required

SERVICE

WCS

service standard

Yes

VERSION

2.0.1 or later

WCS version used

Yes

REQUEST

UpdateCoverage

Request type to be performed

Yes

COVERAGEID

{string}

Identifier of the coverage to be updated

Yes

INPUTCOVERAGEREF

{url}

URl pointing to the coverage to be inserted

One of inputCoverageRef or inputCoverage is required

INPUTCOVERAGE

{coverage}

A coverage to be updated

One of inputCoverageRef or inputCoverage is required

SUBSET

AxisLabel(geoLowerBound, geoUpperBound)

Trim or slice expression, one per updated coverage dimension

No

The following example shows how to update an existing coverage test_mr_metadata from a generated GML file by wcst_import tool:

http://localhost:8080/rasdaman/ows?SERVICE=WCS&version=2.0.1
&REQUEST=UpdateCoverage
&SUBSET=i(0,60)
&subset=j(0,40)
&INPUTCOVERAGEREF=file:///tmp/4514863c_55bb_462f_a4d9_5a3143c0e467.gml


#### 5.4.4.3. Deleting Coverages¶

The DeleteCoverage request type serves to delete a coverage (consisting of the underlying rasdaman collection, the associated WMS layer (if exists) and the petascope metadata). For example: The coverage test_mr can be deleted as follows:

http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1
&REQUEST=DeleteCoverage
&COVERAGEID=test_mr


### 5.4.5. Renaming a coverage¶

The /rasdaman/admin/coverage/update non-standard API allows to update a coverage id and the associated WMS layer if one exists (v10.0+). For example, the coverage test_mr can be renamed to test_mr_new as follows:

http://localhost:8080/rasdaman/admin/coverage/update
?COVERAGEID=test_mr
&NEWCOVERAGEID=test_mr_new


Coverage metadata can be updated through the interactive rasdaman WSClient on the OGC WCS > Describe Coverage tab, by selecting a text file (MIME type must be one of text/xml, application/json, or text/plain) containing the new metadata; Note that to be able to do this it is necessary to login first in the Admin tab.

The non-standard API for this feature is at /rasdaman/admin/coverage/update which operates through multipart/form-data POST requests. The request should contain 2 parts:

1. the coverageId to update, and

2. the path to a local text file to be uploaded to the server.

For example, the below request will update the metadata of coverage test_mr_metadata with the one in a local XML file at /home/rasdaman/Downloads/test_metadata.xml by using the curl tool:

curl -F "COVERAGEID=test_mr_metadata"


### 5.4.7. INSPIRE Coverages¶

The INSPIRE Download Service is an implementation of the Technical Guidance for the implementation of INSPIRE Download Services using Web Coverage Services (WCS) version 2.0+.

In order to achieve INSPIRE Download Service compliance, the following enhancements have been implemented in rasdaman for WCS GetCapabilities response.

• Under <ows:OperationsMetadata> there is a new section for INSPIRE metadata for the service. For example, the result below contains two INSPIRE coverages cov_1 and cov_2.

▶ show

• Service Metadata URL field (<inspire_common:URL>), a URL containing the location of the metadata associated with the WCS service which is configured by setting inspire_common_url in petascope.properties.

• Under <inspire_common:SupportedLanguages> section, the supported language is fixed to eng (English) only.

• A coverage is considered INSPIRE coverage, if it has a specific URL set by metadataURL attribute. All INSPIRE coverages is listed in the list of XML elements <inspire_dls:SpatialDataSetIdentifier>. The example above contains two INSPIRE coverages, each <inspire_dls:SpatialDataSetIdentifier> element containing an attribute metadataURL to provide more information about the coverages. The value for <inspire_common:Namespace> elements of each INSPIRE coverage is derived from the service endpoint.

#### 5.4.7.1. Create an INSPIRE coverage¶

Controlling whether a local coverage is treated as an INSPIRE coverage can be done by:

• Manually sending a request to /rasdaman/admin/inspire/metadata/update with two mandatory parameters:

• COVERAGEID - the coverage to be converted to an INSPIRE coverage

• METADATAURL - a URL to an INSPIRE-compliant catalog entry for this coverage; if set to empty, i.e. METADATAURL= then the coverage is marked as non-INSPIRE coverage.

For example, the coverage test_inspire_metadata can be marked as INSPIRE coverage as follows:

curl --user rasadmin:rasadmin -X POST \

• Via wcst_import.sh, in an ingredients files with inspire section contains the settings for importing INSPIRE coverage:

• metadata_url - If set to non-empty string, then the importing coverage will be marked as INSPIRE coverage. If set to empty string or omitted, then the coverage will be updated as non-INSPIRE coverage.

For example, the coverage cov_3 will be imported as INSPIRE coverage with this configuration in the ingredients file:

▶ show

### 5.4.8. Check if a coverage exists¶

In v10+, rasdaman offers non-standard API to check if a coverage exists in a simpler and faster way than doing a GetCapabilities or a DescribeCoverage request. The result is a true/false string literal.

Example:

http://localhost:8080/rasdaman/admin/coverage/exist?coverageId=cov1


### 5.4.9. GetCapabilities response extensions¶

The WCS GetCapabilities response contains some rasdaman-specific extensions, as documented below.

• The <ows:AdditionalParameters> element of each coverage contains some information which can be useful to clients:

• sizeInBytes - an estimated size (in bytes) of the coverage

• sizeInBytesWithPyramidLevels - an estimated size (in bytes) of the base coverage plus sizes of its pyramid coverages; only available if this coverage has pyramid

• axisList - the coverage axis labels in geo CRS order

Example:

▶ show

## 5.5. OGC Web Coverage Processing Service (WCPS)¶

The OGC Web Coverage Processing Service (WCPS) standard defines a protocol-independent language for the extraction, processing, analysis, and fusion of multi-dimensional gridded coverages, often called datacubes.

### 5.5.1. General¶

WCPS requests can be submitted in both abstract syntax (example) and in XML (example).

For example, using the WCS GET/KVP protocol binding, a WCPS request can be sent through the following ProcessCoverages request:

http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1
&request=ProcessCoverage&query=<wcps-query>


The following subsections list enhancements rasdaman offers over the OGC WCPS standard. A brief introduction to the WCPS language is given in the WCPS cheatsheet; further educational material is available on EarthServer.

### 5.5.2. Polygon/Raster Clipping¶

The non-standard clip() function enables clipping in WCPS. The signature is as follows:

clip( coverageExpression, wkt [, subsettingCrs ] )


where

• coverageExpression is an expression of result type coverage, e.g. dem + 10;

• wkt is a valid WKT (Well-Known Text) expression, e.g. POLYGON((...)), LineString(...);

• subsettingCrs is an optional CRS URL in which the wkt coordinates are expressed, e.g. "http://localhost:8080/rasdaman/def/crs/EPSG/0/4326".

#### 5.5.2.1. Clipping Examples¶

• Polygon clipping with coordinates in EPSG:4326 on coverage with native CRS EPSG:3857:

▶ show

• Linestring clipping on 3D coverage with axes X, Y, datetime.

▶ show

• Linestring clipping on 2D coverage with axes X, Y.

▶ show

In this case with WITH COORDINATES extra parameter, the geo coordinates of the values on the linestring will be included as well in the result. The first two bands of the result holds the coordinates (by geo CRS order), and the remaining bands the original cell values. Example output for the above query:

.. code-block:: text


“-28.975 119.975 90”,”-28.975 120.475 84”,”-28.475 120.975 80”, …

• Multipolygon clipping on 2D coverage.

▶ show

• Curtain clipping by a Linestring on 3D coverage

▶ show

• Curtain clipping by a Polygon on 3D coverage

▶ show

• Corridor clipping by a Linestring on 3D coverage

▶ show

• Corridor clipping by a Polygon on 3D coverage (geo CRS: EPSG:4326) with input geo coordinates in EPSG:3857.

▶ show

### 5.5.3. Auto-ratio for spatial scaling¶

The scale() function allows to specify the target extent of only one of the spatial horizontal axes, instead of requiring both. In such a case, the extent of the unspecified axis will be determined automatically while preserving the original ratio between the two spatial axes.

For example in the request below, the extent of Lat will be automatically set to a value that preserves the ratio in the output result:

▶ show

### 5.5.4. Non-scaled axes are optional¶

The scale() function will implicitly add the full domains of unspecified non-spatial axes of a given coverage, with the effect that they will not be scaled in the result. This deviates from the OGC WCPS standard, which requires all axes to be specified with target domains, even if the resolution of an axis should not be changed in the result.

In the example query below, a 3D coverage is scaled only spatially because only the spatial axes E and N are specified in the target scale intervals, while the ansi non-spatial axis is omitted.

▶ show

### 5.5.5. Extensions on domain functions¶

The domain interval can be extracted from a domain and imageCrsDomain. Both the interval - ie: [lowerBound:upperBound] - and lower as well as upper bound can be retrieved for each axis.

Syntax:

operator(.lo|.hi)?


with .lo or .hi returning the lower bound or upper bound of this interval.

Further, the third argument of the domain() operator, the CRS URL, is optional. If not specified, domain() will use the CRS of the selected axis (ie, the second argument) instead.

For example, the coverage AvgLandTemp has 3 dimensions with grid bounding box of (0:184, 0:1799, 0:3599), and a geo bounding box of ("2000-02-01:2015-06-01", -90:90, -180:180). The table below lists various expressions and their results:

Table 5.5 Non-standard domain operations

Expression

Result

imageCrsdomain($c, Long) (0:3599) imageCrsdomain($c, Long).lo

0

imageCrsdomain($c, Long).hi 3599 domain($c, Long)

(-180:180)

domain($c, Long).lo -180 domain($c, Long).hi

180

### 5.5.6. LET clause¶

An optional LET clause allows binding alias variables to valid WCPS query sub-expressions; subsequently the alias variables can be used in the return clause instead of repeating the aliased sub-expressions.

The syntax in context of a full query is as follows:

FOR-CLAUSE
LET $variable := assignment [ ,$variable := assignment ]
...
[ WHERE-CLAUSE ]
RETURN-CLAUSE


where

assignment ::= coverageExpression | [ dimensionalIntervalList ]


An example with the first case:

for $c in (test_mr) let$a := $c[i(0:50), j(0:40)],$b := avg($c) * 2 return encode( scale($c, { imageCrsDomain( $c ) } ) +$b, "image/png" )


The second case allows to conveniently specify domains which can then be readily used in subset expression, e.g:

for $c in (test_mr) let$dom := [i(20), j(40)]
return
encode( $c[$dom ] + 10, "itext/json" )


### 5.5.7. min and max functions¶

Given two coverage expressions A and B (resulting in compatible coverages, i.e. same domains and types), min(A, B) and max(A, B) calculate a result coverage with the minimum / maximum for each pair of corresponding cell values of A and B.

For multiband coverages, bands in the operands must be pairwise compatible; comparison is done in lexicographic order with the first band being most significant and the last being least significant.

The result coverage value has the same domain and type as the input operands.

### 5.5.8. Positional parameters¶

Positional parameters allow to reference binary or string values in a WCPS query, which are specified in a POST request in addition to the WCPS query. Each positional parameter must be a positive integer prefixed by a $, e.g. $1, $2, etc. The endpoint to send WCPS query by POST with extra values is: /rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=ProcessCoverages  with the mandatory parameter query and optional positional parameters 1, 2, etc. The value of a positional parameter can be either a binary file data or a string value. #### 5.5.8.1. Example¶ One can use the curl tool to send a WCPS request with positional parameters from the command line; it will read the contents of specified files automatically if they are prefixed with a @. For example, to combine an existing coverage $c with two temporary coverages $d and $e provided by positional parameters $1 and $2 into a result encoded in png format (specified by positional parameter $3): curl -s "http://localhost:8080/rasdaman/ows?SERVICE=WCS&VERSION=2.0.1&REQUEST=ProcessCoverages" \ -F 'query=for$c in (existing_coverage), $d in (decode($1)), $e in (decode($2))
return encode(($c +$d + $e)[Lat(0:90), Long(-180:180)], "$3"))' \
-F "1=@/home/rasdaman/file1.tiff" \
-F "2=@/home/rasdaman/file2.tiff" \
-F "3=png" > test.png


### 5.5.9. Decode Operator in WCPS¶

The non-standard decode() operator allows to combine existing coverages with temporary coverages created in memory from input files attached in the request body via POST.

Only 2D geo-referenced files readable by GDAL are supported. One way to check if a file $f is readable by GDAL is with gdalinfo$f. netCDF/GRIB files are not supported.

#### 5.5.9.1. Syntax¶

The syntax is

decode(${positional_parameter})  where ${positional_parameter) refers to files in the POST request. See the previous section for more details on positional parameters.

### 5.5.10. Case Distinction¶

Conditional evaluation based on the cell values of a coverage is possible with the switch expression. Although the syntax is a little different, the semantics is very much compatible to the rasql case statement, so it is recommended to additionally have a look at its corresponding documentation.

#### 5.5.10.1. Syntax¶

SWITCH
CASE condExp RETURN resultExp
[ CASE condExp RETURN resultExp ]*
DEFAULT RETURN resultExpDefault


where condExp and resultExp are either scalar-valued or coverage-valued expressions.

#### 5.5.10.2. Constraints¶

• All condExp must return either boolean values or boolean coverages

• All resultExp must return either scalar values, or coverages

• The domain of all condition expressions must be the same

• The domain of all result expressions must be the same (that means same extent, resolution/direct positions, crs)

#### 5.5.10.3. Evaluation Rules¶

If the result expressions return scalar values, the returned scalar value on a branch is used in places where the condition expression on that branch evaluates to True. If the result expressions return coverages, the values of the returned coverage on a branch are copied in the result coverage in all places where the condition coverage on that branch contains pixels with value True.

The conditions of the statement are evaluated in a manner similar to the IF-THEN-ELSE statement in programming languages such as Java or C++. This implies that the conditions must be specified by order of generality, starting with the least general and ending with the default result, which is the most general one. A less general condition specified after a more general condition will be ignored, as the expression meeting the less general expression will have had already met the more general condition.

Furthermore, the following hold:

• domainSet(result) = domainSet(condExp1)

• metadata(result) = metadata(condExp1)

• rangeType(result) = rangeType(resultExp1). In case resultExp1 is a scalar, the result range type is the range type describing the coverage containing the single pixel resultExp1.

#### 5.5.10.4. Examples¶

switch
case $c < 10 return {red: 0; green: 0; blue: 255} case$c < 20 return {red: 0;   green: 255; blue:   0}
case $c < 30 return {red: 255; green: 0; blue: 0} default return {red: 0; green: 0; blue: 0}  The above example assigns blue to all pixels in the$c coverage having a value less than 10, green to the ones having values at least equal to 10, but less than 20, red to the ones having values at least equal to 20 but less than 30 and black to all other pixels.

switch
case $c > 0 return log($c)
default     return 0


The above example computes log of all positive values in $c, and assigns 0 to the remaining ones. switch case$c < 10 return $c * {red: 0; green: 0; blue: 255} case$c < 20 return $c * {red: 0; green: 255; blue: 0} case$c < 30 return $c * {red: 255; green: 0; blue: 0} default return {red: 0; green: 0; blue: 0}  The above example assigns blue: 255 multiplied by the original pixel value to all pixels in the$c coverage having a value less than 10, green: 255 multiplied by the original pixel value to the ones having values at least equal to 10, but less than 20, red: 255 multiplied by the original pixel value to the ones having values at least equal to 20 but less than 30 and black to all other pixels.

### 5.5.11. CIS 1.0 to CIS 1.1 encoding¶

For output format application/gml+xml WCPS supports delivery as CIS 1.1 GeneralGridCoverage by specifying an additional proprietary parameter outputType in the encode() function, e.g:

for c in (test_irr_cube_2)
return encode( c, "application/gml+xml",
"{\"outputType\":\"GeneralGridCoverage\"}" )


### 5.5.12. Query Parameter¶

For specifying the WCPS query in a request, in addition to the query parameter the non-standard q parameter is also supported. A request must contain only one q or query parameter.

http://localhost:8080/rasdaman/ows?service=WCS&version=2.0.1
&REQUEST=ProcessCoverage&q=<wcps-query>


### 5.5.13. Describe Operator in WCPS¶

The non-standard describe() function delivers a “coverage description” of a given coverage without the range set, in either GML or JSON.

#### 5.5.13.1. Syntax¶

describe( coverageExpression, outputFormat [ , extraParameters ] )


where

• outputFormat is a string specifying the format encoding in which the result will be formatted. Formats are indicated through their MIME type identifier, just as in encode(). Formats supported:

• application/gml+xml or gml for GML

• application/json or json for JSON

• extraParameters is an optional string containing parameters for fine-tuning the output, just as in encode(). Options supported:

• "outputType=GeneralGridCoverage" to return a CIS 1.1 General Grid Coverage structure

#### 5.5.13.2. Semantics¶

A describe() operation returns a description of the coverage resulting from the coverage expression passed, consisting of domain set, range type, and metadata, but not the range set. As such, this operator is the WCPS equivalent to a WCS DescribeCoverage request, and the output adheres to the same WCS schema.

The coverage description generated will follow the coverage’s type, so one of Rectified Grid Coverage (CIS 1.0), ReferenceableGridCoverage (CIS 1.0), or General Grid Coverage (CIS 1.0).

By default, the coverage will be provided as Rectified or Referenceable Grid Coverage (in accordance with its type); optionally, a General Grid Coverage can be generated instead through "outputType=GeneralGridCoverage". As JSON is supported only from OGC CIS 1.1 onwards this format is only available (i) if the coverage is stored as a CIS 1.1 General Grid Coverage (currently not supported) or (ii) this output type is selected explicitly through an extraParameter.

Efficiency: The describe() operator normally does not materialize the complete coverage, but determines only the coverage description making this function very efficient. A full evaluation is only required if coverageExpression contains a clip() performing a curtain, corridor, or linestring operation.

#### 5.5.13.3. Examples¶

• Determine coverage description as a CIS 1.0 Rectified Grid Coverage in GML, without evaluating the range set:

for $c in (Cov) return describe($c.red[Lat(10:20), Long(30:40), "application/gml+xml" )

• Deliver coverage description as a CIS 1.1 General Grid Coverage in GML, where range type changes in the query:

for $c in (Cov) return describe( {$c.red; $c.green;$c.blue }, "application/gml+xml",
"outputType=GeneralGridCoverage" )

• Deliver coverage description as a CIS 1.1 General Grid Coverage, in JSON:

for $c in (Cov) return describe($c, "application/json", "outputType=GeneralGridCoverage" )


#### 5.5.13.4. Specific Exceptions¶

• Unsupported output format

• This format is only supported for General Grid Coverage

• Illegal extra parameter

## 5.6. OGC Web Map Service (WMS)¶

The OGC Web Map Service (WMS) standard provides a simple HTTP interface for requesting overlays of geo-registered map images, ready for display.

With petascope, geo data can be served simultaneously via WMS, WCS, and WCPS. Further information:

This section mainly covers rasdaman extensions of the OGC WMS standard.

### 5.6.1. GetMap extensions¶

#### 5.6.1.1. Transparency¶

By adding a parameter transparent=true to WMS requests the returned image will have NoData Value=0 in the metadata indicating to the client that all pixels with value 0 value should be considered transparent for PNG encoding format. Example:

▶ show

If in a GetMap request the output CRS requested is different from the coverage’s native CRS, petascope will duly reproject the map applying resampling and interpolation. The algorithm used can be controlled with the non-standard GetMap parameter interpolation=${method}; default is nearest-neighbour interpolation. See Geographic projection for the methods available and their meaning. Example: ▶ show ### 5.6.2. nD Coverages as WMS Layers¶ Petascope allows to import a 3D+ coverage as a WMS layer. To this end, the ingredients file used for wcst_import must contain wms_import": true. For 3D+ coverages this works with recipes regular_time_series, irregular_time_series, and general_coverage. This example shows how to define an irregular_time_series 3D coverage from 2D TIFF files. Once the coverage is created, GetMap requests can use the additional (non-horizontal) axes for subsetting according to the OGC WMS 1.3.0 standard. Table 5.6 WMS Subset Parameters for Different Axis Types Axis Type Subset parameter Time time=… Elevation elevation=… Other dim_AxisName=… (e.g dim_pressure=…) According to the WMS 1.3.0 specification, the subset for non-geo-referenced axes can have these formats: • Specific value (value1): time='2012-01-01T00:01:20Z' dim_pressure=20  • Range values (min/max): time='2012-01-01T00:01:20Z'/'2013-01-01T00:01:20Z' dim_pressure=20/30  • Multiple values (value1,value2,value3,…): time='2012-01-01T00:01:20Z','2013-01-01T00:01:20Z' dim_pressure=20,30,60,100  • Multiple range values (min1/max1,min2/max2,…): dim_pressure=20/30,40/60  A GetMap request always returns a 2D result. If a non-geo-referenced axis is omitted from the request it will be considered as a slice on the upper bound along this axis. For example, in a time-series the most recent timeslice will be delivered. Examples: ### 5.6.3. GetLegendGraphic request¶ WMS GetLegendGraphic allows to get a legend PNG/JPEG image associated with a style of a layer. Admin can set a legend image for a style via a style creation request. Required request parameters: • format - data format in which the legend image is returned; only image/png and image/jpeg are supported. • layer - the WMS layer which contains the specified style. • style - the style which contains the legend image. Note Any further extra parameters will be ignored by rasdaman. This request, for example, will return the legend image for style color of layer cov1: http://localhost:8080/rasdaman/ows?service=WMS&request=GetLegendGraphic &format=image/png&layer=cov1&style=color  When a style of a layer has an associated legend graphic, WMS GetCapabilities will have an additional <LegendURL> XML section for this style. For example: ▶ show ### 5.6.4. Layer Management¶ Non-standard API for WMS layer management are listed below. Layers can be easily created from existing WCS coverages in two ways: • Create a new WMS layer from an existing coverage MyCoverage: /rasdaman/admin/layer/activate?COVERAGEID=MyCoverage  During coverage import this can be done with the wms_import option in the ingredients file. • Remove a WMS layer directly: /rasdaman/admin/layer/deactivate&COVERAGEID=MyLayer  Indirectly a layer will be removed when deleting the associated WCS coverage ### 5.6.5. Style Behavior¶ When a client sends GetMap requests, the rules below define (in conformance with the WMS 1.3 standard) how a style is applied to the requested layers: • If no styles are defined then rasdaman returns the data as-is, encoded in the requested format. • If some styles are defined, e.g. X, Y, and Z, then: • If the client specifies a style Y, then Y is applied. • If the client does not specify a style, then: • If the admin has set a style as default, e.g. Z, then Z is applied. • Otherwise, if no style has been set as default, then the first style from the list of styles (X) is applied. ### 5.6.6. Style Management¶ Styles can be created for layers using rasql and WCPS query fragments. This allows users to define several visualization options for the same dataset in a flexible way. Examples of such options would be color classification, NDVI detection etc. The following HTTP request will create a style with the name, abstract and layer provided in the KVP parameters below Note Tomcat version 7+ requires the query (WCPS/rasql fragment) to be URL-encoded correctly. This site offers such an encoding service. #### 5.6.6.1. Style Definition¶ A style of a WMS layer can be created via the /rasdaman/admin/layer/style/add endpoint, while an existing style can be updated via the /rasdaman/admin/layer/style/update endpoint. Both endpoints understand the following parameters: • COVERAGEID - an existing WMS layer to which the style to be created or updated belongs (mandatory); • STYLEID - the style name, must be unique among all the styles of one layer (mandatory); • TITLE - an optional style title as human-understandable text; • ABSTRACT - an optional description of the what the style does • One of the following (optional): • RASQLTRANSFORMFRAGMENT - a rasql query expression applied to the map tiles before being returned to the client; • WCPSQUERYFRAGMENT - a WCPS query expression applied to the map tiles before being returned to the client; • COLORTABLETYPE + COLORTABLEDEFINITION - an optional color table for coloring the map tiles before returning to the client. At least a query fragment, or a color table, or both, must be specified in the request. Additionally the updating endpoint supports the following optional parameters: • NEWSTYLEID - the style specified with STYLEID will be renamed to the new id specified by this parameter. • DEFAULT - if set to true then this style is set as the default of the layer (more details here); if not specified, it is false by default. • LEGENDGRAPHIC - associate a PNG/JPEG legend image to this style, specified in Base64 string format; clients can get the legend with a GetLegendGraphic request (more details here). The legend can be removed by setting this parameter to empty, i.e. LEGENDGRAPHIC=. Below the supported values for COLORTABLETYPE are explained: • ColorMap: check Coloring Arrays for more details; the color table definition must be a JSON object, for example: ▶ show • GDAL: The color table definition must be a JSON object containing 256 color arrays in a colorTable array, example: ▶ show • SLD: The color table definition must be valid Styled Layer Descriptor XML and contain a ColorMap element. Note that rasdaman will only consider the first sld:ColorMap element in the SLD document, any other SLD elements will be ignored. Check Coloring Arrays for details about the supported types (ramp (default), values, intervals), example ColorMap with type="values": ▶ show #### 5.6.6.2. Style Removal¶ Removing a style from an existing WMS layer can be done via the /rasdaman/admin/layer/style/remove endpoint, e.g. /rasdaman/admin/layer/style/remove?COVERAGEID=MyCoverage&STYLEID=mystyle  #### 5.6.6.3. Examples¶ • Create a style with a WCPS query fragment and set this style as default style: ▶ show Variable $c will be replaced by a layer name when sending a GetMap request containing this layer’s style.

• Create a style with a rasql query fragment:

▶ show

Variable $Iterator will be replaced with the actual name of the rasdaman collection and the whole fragment will be integrated inside the regular GetMap request. • Multiple layers can be used in a style definition. Besides the iterators $c in WCPS query fragments and $Iterator in rasql query fragments, which always refer to the current layer, other layers can be referenced by name using an iterator of the form $LAYER_NAME in the style expression.

Example: create a WCPS query fragment style referencing 2 layers ($c refers to layer sentinel2_B4 which defines the style): ▶ show Then, in any GetMap request using this style the result will be obtained from the combination of the 2 layers sentinel2_B4 and sentinel2_B8: ▶ show • WMS styling supports colorizing the result of GetMap request when the style is requested by applying a color table definition to it. A style can contain either one or both a query fragment and color table definitions. The request supports two parameters for this purpose: COLORTABLETYPE with valid values ColorMap, GDAL and SLD, and COLORTABLEDEFINITION containing the corresponding definition. ▶ show ### 5.6.7. Pyramid Management¶ The following WMS requests are used to manage downscaled coverages, which are primarily created as pyramid levels of a particular base coverage. Internally they are used for efficient zooming in/out in WMS, and downscaling when using the scale() function in WCPS or scaling extension in WCS. Only regular axes, typically spatial X and Y, can be downscaled for this purpose. Below the API for pyramid management are covered: • Create a pyramid member coverage c for a base coverage b with given scale factors for each axis. Only regular axes can have a scale factor > 1. E.g. to create a downscaled coverage cov_3D_4 of a 3D coverage cov_3D that is 4x smaller for Lat and Long regular axes (Time is irregular axis, hence, scale factor must be 1): ▶ show wcst_import can execute create pyramid requests automatically when importing data with the scale_levels or scale_factors options in the ingredients file; more details here. • Add a list of existing coverage c, d, e, … as pyramid member coverages of a base coverage b. The scale factors for each axis of the pyramid member coverage will be calculated implicitly based on axis resolutions. If harvesting=true (default is false), recursively collect pyramid members of c, d, e, … and add them as pyramid member of b. E.g. to add a downscaled coverage cov_3D_4 (4x smaller) and its pyramid members recursively as pyramid member coverages of base coverage cov_3D: ▶ show wcst_import provides several options for conveniently adding pyramid members in the ingredients file. • Remove a list of existing pyramid member coverage c, d, e, … from a base coverage b. The coverages c, d, e, … will still exist, until they are removed with a WCS-T DeleteCoverage request. E.g. to remove pyramid member cov_3D_4 from base coverage cov_3D: ▶ show • List all pyramid member coverages associated with a base coverage in JSON-formatted output. E.g. to list the pyramid members of Sentinel2_10m: ▶ show Example output: ▶ show ### 5.6.8. Testing a WMS Setup¶ A rasdaman WMS service can be tested with any conformant client through a GetMap request like the following: ▶ show ### 5.6.9. Errors and Workarounds¶ #### 5.6.9.1. Cannot load new WMS layer in QGIS¶ In this case, the problem is due to QGIS caching the WMS GetCapabilities from the last request so the new layer does not exist (see clear cache solution). ## 5.7. Data Import¶ Raster data in a variety of formats, such as TIFF, netCDF, GRIB, etc. can be imported in rasdaman through the wcst_import.sh utility. Internally it is based on WCS-T requests, but hides the complexity and maintains the geo-related metadata in its so-called petascopedb while the raster data get imported into the rasdaman array store. Building large time-series / datacubes, mosaics, etc. and keeping them up-to-date as new data become available is supported for a large variety of data formats and file/directory organizations. The systemtest contains many examples for importing different types of data. ### 5.7.1. Introduction¶ The wcst_import.sh tool is based on two concepts: • Recipe - A recipe defines how a set of data files can be combined into a well-defined coverage (e.g. a 2-D mosaic, regular or irregular 3-D timeseries, etc.); • Ingredients - A JSON file that configures how the recipe should build the coverage (e.g. the server endpoint, the coverage name, which files to consider, etc.). To execute an ingredients file in order to import some data: $ wcst_import.sh path/to/my_ingredients.json


Alternatively, wcst_import.sh can be started in the background as a daemon:

$wcst_import.sh path/to/my_ingredients.json --daemon start  or as a daemon that is “watching” for new data at some interval (in seconds): $ wcst_import.sh path/to/my_ingredients.json --watch <interval>


For further informations regarding the usage of wcst_import.sh:

• Python expressions - The types above can be combined into any valid Python expression; this allows to do mathematical operations, string parsing, date/time manipulation, etc. E.g. ${gdal:minX} + 1/2 *${gdal:resolutionX} or datetime(${netcdf:variable:time:min} * 24 * 3600). Expressions can use functions from any Python library which just needs to be explicitly imported as explained in Using libraries in sentences. #### 5.7.6.3. Data expressions¶ Each driver allows expressions to extract information from input files. We will mark with capital letters things that vary in the expression. E.g. ${gdal:metadata:FIELD} means that you can replace FIELD with any valid gdal metadata tag such as TIFFTAG_DATETIME. Example ingredients where data expressions are used can be found in Examples.

##### 5.7.6.3.1. NetCDF¶

Type

Description

Examples

${netcdf:metadata:YOUR_METADATA_FIELD} ${netcdf:metadata:title}

Variable information

${netcdf:variable:VAR_NAME:MODIFIER} where VAR_NAME can be any variable in the file and MODIFIER can be one of: first|last|max|min; Any extra modifiers will return the corresponding metadata field on the given variable ${netcdf:variable:t:min} ${netcdf:variable:t:units} Dimension information ${netcdf:dimension:DIM_NAME} where DIM_NAME can be any dimension in the file. This will return the value on the selected dimension.

${netcdf:dimension:time} ##### 5.7.6.3.2. GDAL¶ Relevant for TIFF, PNG, JPEG, and other 2D data formats. Type Description Examples Metadata information ${gdal:metadata:METADATA_FIELD}

${gdal:metadata:TIFFTAG} Geo Bounds ${gdal:BOUND_NAME} where BOUND_NAME can be one of the minX|maxX|minY|maxY

${gdal:minX} Geo Resolution ${gdal:RESOLUTION_NAME} where RESOLUTION_NAME can be one of the resolutionX|resolutionY

${gdal:resolutionX} Origin ${gdal:ORIGIN_NAME} where ORIGIN_NAME can be one of the originX|originY

##### 5.7.6.3.4. File¶

Type

Description

Examples

File Information

${file:PROPERTY} where property can be one of path|name|dir_path|original_path|original_dir_path original_* allows to get the original input file’s path/directory. Used only in before_import hooks with replace_path to replace original input file paths with customized file paths. ${file:path}

Imported File Information

${imported_file:PROPERTY} where property can be one of path|name|dir_path|original_path|original_dir_path Files which were imported to rasdaman (excluding skipped files). This variable is used only in after_import hooks. ${imported_file:path}

##### 5.7.6.3.5. Special functions¶

A couple of special functions are available to help with more complicated expressions:

Function and Arguments

Description

Examples

grib_datetime

• date

• time

This function helps to deal with the usual grib date and time format. It returns back a datetime string in ISO format.

grib_datetime(${grib:dataDate},${grib:dataTime})


datetime

• date

• format

This function helps to deal with strange date time formats. It returns back a datetime string in ISO format.

datetime("20120101:1200",
"YYYYMMDD:HHmm")


regex_extract

• string

• regex

• group

This function extracts information from a string using regex; input is the string you parse, regex is the regular expression, group is the regex group you want to select

datetime(
regex_extract('${file:name}', '(.*)_(\\d*-\\d\\d)(.*)', 2), 'YYYY-MM')  replace • str • old • new Replaces all occurrences of a substring with another substring in the input string replace('${file:path}',
'.tiff', '.xml')


#### 5.7.6.4. Using libraries in sentences¶

In case the ingredient sentences require functionality from extra Python libraries, they can be imported with a statements option. For example, to calculate the lower bound and upper bound for the time axis ansi (starting days from 1978-12-31T12:00:00) one could use datetime and timedelta from the datatime library.

▶ show

Python functions imported in this way override the special functions provided by wcst_import. For example, the special utility function datetime(date_time_string, format) to convert a string of datetime to an ISO date time format will be overridden when the datetime module is imported with a statements setting.

Note

See details about potential issue for running python code in the ingredients file.

#### 5.7.6.5. Local metadata from input files¶

Beside the global metadata of a coverage, you can add local metadata for each file which is a part of the whole coverage (e.g. a 3D time-series coverage mosaiced from 2D GeoTiff files).

Under the metadata section add a “local” object with keys and values extracted by using format type expression. Example of extracting an attribute from a netCDF input file:

▶ show

Each file’s envelope (geo domain) and its local metadata will be added to the coverage metadata under <slice>...</slice> element if coverage metadata is imported in XML format. Example of a coverage containing local metadata in XML from 2 netCDF files:

▶ show

Since v10.0, local metadata for input files can be also fetched from corresponding external text files with the optional metadata_file option. For example:

▶ show

When subsetting a coverage which contains a local metadata section from input files (via WC(P)S requests), if the geo domains of subsetted coverage intersect with some input files’ envelopes, only local metadata of these files will be added to the output coverage metadata.

For example: a GetCoverage request with a trim such that crs axis subsets are within netCDF file 1:

▶ show

The coverage’s metadata result will contain local metadata only from netCDF file 1:

▶ show

#### 5.7.6.6. Customized axis labels¶

By default, the axes to be configured must be matched by their name as defined by the coverage CRS. For example, a CRS OGC/0/AnsiDate@EPSG:4326 defines three axes with labels ansi, Long, and Lat. To configure them, we would have a section as bellow:

▶ show

Since v9.8, one can change the default axis label defined by the CRS through indicating the axis index in the CRS (0-based) with the "crsOrder" setting. For example, to change the axis labels to MyDateTimeAxis, MyLatAxis, and MyLongAxis:

▶ show

#### 5.7.6.7. Group coverage slices¶

Since v9.8, wcst_import allows to group input files on irregular axes (with "dataBound": false) through the sliceGroupSize option, which would specify the group size as a positive number. E.g:

▶ show

If each input slice corresponds to index X, and one wants to have slice groups of size N, then the index would be translated with this option to X - (X % N).

Typical use case is importing 3D coverage from 2D satellite imagery where the time axis is irregular and its values are fetched from input files by regex expression. Then, all input files which belong to the same time window (e.g 7 days in AnsiDate CRS with "sliceGroupSize": 7) will have the same value, which is the first date of the week.

Metadata can be individually specified for each band and axis in the ingredient file. Example:

▶ show

Since v9.7, the following metadata can also be automatically derived from the input netCDF files.

• For netCDF: If "bands" is set to "auto" or does not exist under "metadata" in the ingredient file, all user-specified bands will have metadata which is fetched directly from the netCDF file. Metadata for one band is collected automatically if the band is not added or it is set to "auto".

• Otherwise, the user could specify metadata explicitly by a dictionary of keys/values. Example:

▶ show

• For netCDF: If "axes" is set to "auto" or does not exist under "metadata" in the ingredient file, all user-specified axes will have metadata which is fetched directly from the netCDF file. Metadata for one axis is collected automatically if: 1) the axis is not specified, 2) the axis is set to "auto", or 3) the axis is set to ${netcdf:variable:Name:metadata}. The axis label for variable is detected from the min or max value of CRS axis configuration under "slicer/axes" section. For example: ▶ show • Otherwise, the user could specify metadata explicitly as a dictionary of keys/values. ▶ show #### 5.7.6.9. Rotated CRS support¶ If rasdaman is compiled with GDAL v3.4.1+, importing and querying data with rotated CRS COSMO:101 is supported. The netCDF data usually has to be preprocessed before import: 1. Invert the latitude axis when it is south to north order (lower to upper coordinates): cdo invertlat input.nc inverted_input.nc  2. Swap the order of the rotated latitude (rlat) and rotated longitude (rlon) axes when the data variable has rlat,rlon order. For example, the float CAPE_ML(time, rlat, rlon) variable can be transformed to float CAPE_ML(time, rlon, rlat) with the following command: ncpdq --rdr=time,rlon,rlat inverted_input.nc correct_lon_lat.nc  Example ingredient file for importing the CAPE_ML variable from preprocessed COSMO netCDF data: ▶ show wcst_import automatically checks if the specified band variables (CAPE_ML in the above example) have a grid_mapping metadata entry (e.g. CAPE_ML:grid_mapping = "rotated_pole"), and adds all metadata from the grid mapping variable (rotated_pole) to the global metadata of the imported coverage. With the added grid_mapping section, the global metadata of the coverage might look as below, for example: ▶ show When encoding to netCDF in WCS or WCPS requests with the same COSMO:101 CRS, rasdaman will add this grid mapping metadata as a non-dimension variable in the output, so that it has the correct CRS information. The name of the non-dimension variable in the output is set from the identifier value (rotated_pole above). ### 5.7.7. Recipe wcs_extract¶ Allows to import a coverage from a remote petascope endpoint into the local petascope. Parameters are explained below. ▶ show ### 5.7.8. Recipe sentinel1¶ This is a convenience recipe for importing Sentinel 1 data in particular; currently only GRD/SLC product types are supported, and only geo-referenced tiff files. Below is an example: ▶ show The recipe extends general_coverage so the "recipe" section has the same structure. However, a lot of information is automatically filled in by the recipe now, so the ingredients file is much simpler as the example above shows. The other obvious difference is that the "coverage_id" is templated with several variables enclosed in ${ and } which are automatically replaced to generate the actual coverage name during import:

• modebeam - the mode beam of input files, e.g. IW/EW.

• polarisation - single polarisation of input files, e.g: HH/HV/VV/VH

If the files collected by "paths" are varying in any of these parameters, the corresponding variables must appear somewhere in the "coverage_id" (as for each combination a separate coverage will be constructed). Otherwise, the data import will either fail or result in invalid coverages. E.g. if all data’s mode beam is IW, but still different polarisations, the "coverage_id" could be "MyCoverage_${polarisation}"; In addition, the data to be imported can be optionally filtered with the following options in the "input" section: • modebeams - specify a subset of mode beams to import from the data, e.g. only the IW mode beam; if not specified, data of all supported mode beams will be ingested. • polarisations - specify a subset of polarisations to import, e.g. only the HH polarisation; if not specified, data of all supported polarisations will be imported. Limitations: • Only GRD/SLC products are supported. • Data must be geo-referenced. • Filenames are assumed to be of the format: s1[ab]-(.*?)-grd(.?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?).tiff or s1[ab]-(.*?)-slc(.?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?)-(.*?).tiff. ### 5.7.9. Recipe sentinel2¶ This is a convenience recipe for importing Sentinel 2 data in particular. It relies on support for Sentinel 2 in more recent GDAL versions. Importing zipped Sentinel 2 is also possible and automatically handled. Below is an example: ▶ show The recipe extends general_coverage so the "recipe" section has the same structure. However, a lot of information is automatically filled in by the recipe now, so the ingredients file is much simpler as the example above shows. The other obvious difference is that the "coverage_id" is templated with several variables enclosed in ${ and } which are automatically replaced to generate the actual coverage name during import:

• crsCode - the CRS EPSG code of the imported files, e.g. 32757 for WGS 84 / UTM zone 57S.

• resolution - Sentinel 2 products bundle several subdatasets of different resolutions:

• 10m - bands B4, B3, B2, and B8 (base type unsigned short)

• 20m - bands B5, B6, B7, B8A, B11, and B12 (base type unsigned short)

• 60m - bands B1, B8, and B10 (base type unsigned short)

• TCI - True Color Image (red, green, blue char bands); also 10m as it is derived from the B2, B3, and B4 10m bands.

• level - L1C or L2A

If the files collected by "paths" are varying in any of these parameters, the corresponding variables must appear somewhere in the "coverage_id" (as for each combination a separate coverage will be constructed). Otherwise, the import will either fail or result in invalid coverages. E.g. if all data is level L1C with CRS 32757, but still different resolutions, the "coverage_id" could be "MyCoverage_${resolution}"; the other variables can still be specified though, so "MyCoverage_${resolution}_${crsCode}" is valid as well. In addition, the data to be imported can be optionally filtered with the following options in the "input" section: • resolutions - specify a subset of resolutions to import from the data, e.g. only the “10m” subdataset; if not specified, data of all supported resolutions will be ingested. • levels - specify a subset of levels to import, so that files of other levels will be fully skipped; if not specified, data of all supported levels will be ingested. • crss - specify a list of CRSs (EPSG codes as strings) to import; if not specified or empty, data of any CRS will be imported. ### 5.7.10. Creating your own recipe¶ The recipes above cover a frequent but limited subset of what is possible to model using a coverage. WCSTImport allows to define your own recipes in order to fill these gaps. In this tutorial we will create a recipe that can construct a 3D coverage from 2D georeferenced files. The 2D files that we want to target have all the same CRS and cover the same geographic area. The time information that we want to retrieve is stored in each file in a GDAL readable tag. The tag name and time format differ from dataset to dataset so we want to take this information as an option to the recipe. We would also want to be flexible with the time crs that we require so we will add this option as well. Based on this usecase, the following ingredient file seems to fulfill our need: ▶ show To create a new recipe start by creating a new folder in the recipes folder. Let’s call our recipe my_custom_recipe: ▶ show The last command is needed to tell python that this folder is containing python sources, if you forget to add it, your recipe will not be automatically detected. Let’s first create an example of our ingredients file so we get a feeling for what we will be dealing with in the recipe. Our recipe will just request from the user two parameters Let’s now create our recipe, by creating a file called recipe.py ▶ show Use your favorite editor or IDE to work on the recipe (there are type annotations for most WCSTImport classes so an IDE like PyCharm would give out of the box completion support). First, let’s add the skeleton of the recipe (please note that in this tutorial, we will omit the import section of the files (your IDE will help you auto import them)): ▶ show The first thing you need to do is to make sure the get_name() method returns the name of your recipe. This name will be used to determine if an ingredient file should be processed by your recipe. Next, you will need to focus on the constructor. Let’s examine it. We get a single parameter called session which contains all the information collected from the user plus a couple more useful things. You can check all the available methods of the class in the session.py file, for now we will just save the options provided by the user that are available in session.get_recipe() in a class attribute. In the validate() method, you will validate the options for the recipe provided by the user. It’s generally a good idea to call the super method to validate some of the general things like the WCST Service availability and so on although it is not mandatory. We also want to validate our custom recipe options here. This is how the recipe looks like now: ▶ show Now that our recipe can validate the recipe options, let’s move to the describe() method. This method allows you to let your users know any relevant information about the data import before it actually starts. The irregular_timeseries recipe prints the timestamp for the first couple of slices for the user to check if they are correct. Similar behaviour should be done based on what your recipe has to do. Next, we should define the import behaviour. The framework does not make any assumptions about how the correct method of data import is, however it offers a lot of utility functionality that help you do it in a more standardized way. We will continue this tutorial by describing how to take advantage of this functionality, however, note that this is not required for the recipe to work. The first thing that you need to do is to define an importer object. This importer object, takes a coverage object and imports it using WCST requests. The object has two public methods, ingest(), which imports the coverage into the WCS-T service (note: this can be an insert operation when the coverage was not defined, or update if the coverage exists. The importer will handle both cases for you, so you don’t have to worry if the coverage already exists.) and get_progress() which returns a tuple containing the number of imported slices and the total number of slices. After adding the importer, the code should look like this: ▶ show In order to build the importer, we need to create a coverage object. Let’s see how we can do that. The coverage constructor requires a • coverage_id: the id of the coverage • slices: a list of slices that compose the coverage. Each slice defines the position in the coverage and the data that should be defined at the specified position • range_fields: the range fields for the coverage • crs: the crs of the coverage • pixel_data_type: the type of the pixel in gdal format, e.g. Byte, Float32 etc The coverage object can be built in many ways, we will present one such method. Let’s start from the crs of the coverage. For our recipe, we want a 3D crs, composed of the CRS of the 2D images and a time CRS as indicated. The following lines of code give us exactly this: ▶ show Let’s also get the range fields for this coverage. We can extract them again from the 2D image using a helper class that can use GDAL to get the relevant information: ▶ show Let’s also get the pixel base type, again using the gdal helper: pixel_type = gdal_dataset.get_band_gdal_type()  Let’s see what we have so far: ▶ show As you can notice, the only thing left to do is to implement the _get_slices() method. To do so we need to iterate over all the input files and create a slice for each. Here’s an example on how we could do that ▶ show And we are done we now have a valid coverage object. The last thing needed is to define the status method. This method need to provide a status update to the framework in order to display it to the user. We need to return the number of finished work items and the number of total work items. In our case we can measure this in terms of slices and the importer can already provide this for us. So all we need to do is the following: ▶ show We now have a functional recipe. You can try the ingredients file against it and see how it works. ▶ show ### 5.7.11. Importing many files¶ When an ingredient contains many paths to be imported, usually more than 1000, this may lead to hitting some system limits during the import. In particular when data is imported with the GDAL driver, wcst_import has a cache of open GDAL datasets to avoid reopening files, which is costly. With too many open GDAL datasets limit on max open files can be reached, which is often 1024 (see ulimit -n). wcst_import handles this case by clearing its cache; however, this may degrade import performance, so increasing the limit on open files should be considered. Furthermore, limits on maximum number of threads may be reached as well, as each open GDAL dataset creates several threads. This will lead to errors such as fork: retry: Resource temporarily unavailable. The maximum allowed number can be observed with cat /sys/fs/cgroup/pids/user.slice/user-<id>.slice/pids.max, where <id> can be found with id -u <user> for the user with which wcst_import is executed. Increasing to a larger value, e.g. 4194304, should solve this issue. Finally, wcst_import.sh allows to control the gdal cache size with the -c, --gdal-cache-size <size> option. The specified value can be one of: -1 (no limit, cache all files), 0 (fully disable caching), N (clear the cache whenever it has more than N datasets, N should be greater than 0). The default value is -1 if this option is not specified. ## 5.8. Data export¶ WCS formats are requested via the format KVP key (<gml:format> elements for XML POST requests), and take a valid MIME type as value. Output encoding is passed on to the the GDAL library, so the limitations on output formats are devised accordingly by the supported raster formats of GDAL. The valid MIME types which Petascope may support can be checked from the WCS 2.0.1 GetCapabilities response: ▶ show In case of encode processing expressions, besides MIME types WCPS (and rasql) can also accept GDAL format identifiers or other commonly-used format abbreviations like “CSV” for Comma-Separated-Values for instance. ### 5.8.1. Support for time in netCDF output¶ If the global metadata of a coverage contains "units" and "calendar" settings for the time axis, when encoding to netCDF rasdaman will adjust the coordinates of the time variable based on the origin specified in the "units" and "calendar" setting instead of the time CRS. Only standard and proleptic_gregorian calendars are currently supported. More details on these standard attributes of time variables can be found in the CF conventions docs. For example, a coverage might have this metadata for the ansi time axis: ▶ show The values of ansi variable in the output netCDF file will be based on the origin 2016-12-01 00:00:00 as specified by the <units> above, instead of 1600-12-31, the origin of the AnsiDate CRS associated with this axis. ## 5.9. rasdaman / petascope Geo Service Administration¶ The petascope conpoment, which geo services contact through its OGC APIs, uses rasdaman for storing the raster arrays; geo-related data parts (such as geo-referencing), as per coverage standard, are maintained by petascope itself. Petascope is implemented as a war file of Java servlets. Internally, incoming requests requiring coverage evaluation are translated by petascope, with the help of the coverage metadata, into rasql queries executed by rasdaman as the central workhorse. Results returned from rasdaman are forwarded by petascope to the client. Note rasdaman can maintain arrays not visible via petascope (such as non-geo objects like human brain images). Data need to be imported via Data Import, not rasql, for being visible as coverages. For further internal documentation on petascope see Developer introduction to petascope and its metadata database. ### 5.9.1. Service Startup and Shutdown¶ Depending of how java_server is configured in petascope.properties, starting the petascope Web application is different as follows: • If set to external, then managing the petascope Web application is done via the system Tomcat in which it is deployed, e.g. $ systemctl start tomcat
$systemctl stop tomcat$ systemctl restart tomcat

• If set to embedded then petascope is managed along with rasdaman; see this section for more details.

### 5.9.2. Configuration¶

The rasdaman-geo frontend (petascope) can be configured via changing settings in /opt/rasdaman/etc/petascope.properties. For changes to take effect, system Tomcat (if deployment is external) or rasdaman (if deployment is embedded) needs to be restarted after editing this file.

#### 5.9.2.1. Database¶

• spring.datasource.url set the connectivity string to the database administered by rasdaman-geo. Supported databases are PostgreSQL, H2, HSQLDB; for more details, see this section.

• Default: jdbc:postgresql://localhost:5432/petascopedb

• Need to change: YES when DMBS other than PostgreSQL is used

• spring.datasource.username set the username for connecting to the above database.

• Default: petauser

• Need to change: YES when changed in the above database

• spring.datasource.password set the password for the user specified by spring.datasource.username.

• Default: randomly generated password

• Need to change: YES when changed in the above database

• spring.datasource.jdbc_jar_path absolute path to the JDBC jar file for connecting to the database configured in setting spring.datasource.url. If left empty, the default PostgreSQL JDBC driver will be used. To use a different DBMS (e.g. H2), please download the corresponding JDBC driver, and set the path to it.

• Default: empty

• Need to change: YES when a DMBS other than PostgreSQL is used

• spring.datasource.tomcat.initial-size set the initial size for JDBC connections in pool.

• Default: 30 connections

• Need to change: NO

• spring.datasource.tomcat.max-active set the maximum number of active JDBC connections in pool.

• Default: 70 connections

• Need to change: NO

• spring.datasource.tomcat.max-idle set the maximum number of idle JDBC connections in pool.

• Default: 30 connections

• Need to change: NO

• metadata_url set the connectivity string to the database administered by rasdaman-geo. This setting is only used for database migration from one DBMS to another (e.g. PostgreSQL to H2) with migrate_petascopedb.sh; in this case metadata_url is used to connect to the source database, while spring.datasource.url is used to connect to the target database.

• Default: jdbc:postgresql://localhost:5432/petascopedb

• Need to change: YES when migrating from a DMBS different from PostgreSQL

• metadata_user set the username for the above database

• Default: petauser

• Need to change: YES when different in the above database

• metadata_pass set the password for the user specified by metadata_user

• Default: petapasswd

• Need to change: YES when different in the above database

• metadata_jdbc_jar_path absolute path to the JDBC jar file for connecting to the database configured in setting metadata_url. If left empty, the default PostgreSQL JDBC driver will be used. To use a different DBMS (e.g. H2), please download the corresponding JDBC driver, and set the path to it.

• Default: empty

• Need to change: YES when a DMBS other than PostgreSQL is used

#### 5.9.2.2. General¶

• server.contextPath when rasdaman-geo is running in embedded mode (setting java_server), this setting allows to control the prefix in the deployed web application URL, e.g. the /rasdaman in http://localhost:8080/rasdaman/ows.

• Default: /rasdaman

• Need to change: NO

• secore_urls set SECORE endpoints to be used by rasdaman-geo. Multiple endpoints for fail-safety can be specified as a comma-separated list, attempted in order as listed. By default, internal indicates that rasdaman-geo should use its own SECORE, which is more efficient as it avoids external HTTP requests.

• Default: internal

• Need to change: NO

• xml_validation if set to true, WCS POST/SOAP XML requests will be validated against OGC WCS 2.0.1 schema definitions; when starting Petascope it will take around 1-2 minutes to load the schemas from the OGC server.

Note

Passing the OGC CITE tests requires this parameter to be set to false.

• Default: false

• Need to change: NO

• ogc_cite_output_optimization if true, rasdaman-geo will optimize responses in order to pass a couple of invalid OGC CITE test cases. Indentation of WCS GetCoverage and WCS DescribeCoverage results, for example, will be trimmed.

• Default: false

• Need to change: NO, except when executing OGC CITE tests

• petascope_servlet_url set the service endpoint in <ows:HTTP> elements of the result of GetCapabilities. Change to your public service URL if rasdaman-geo runs behind a proxy; if not set then it will be automatically derived, usually to http://localhost:8080/rasdaman/ows.

• Default: empty

• Need to change: YES when rasdaman-geo runs behind a proxy

• max_wms_cache_size set the maximum amount of memory (in bytes) to use for caching WMS GetMap requests. This setting speeds up repeating WMS operaions over similar area/zoom level. It is recommended to consider increasing the parameter if the system has more RAM, but make sure to correspondingly update the -Xmx option for Tomcat as well. The cache evicts least recently inserted data when it reaches the maximum limit specified here.

• Default: 100000000 (100 MB)

• Need to change: NO

• uploaded_files_dir_tmp set an absolute path to a server directory where files uploaded to rasdaman-geo by a request will be temporarily stored; the user running rasdaman-geo (either tomcat or rasdaman) should have write permissions on the specified directory.

• Default: /tmp/rasdaman_petascope/upload

• Need to change: NO

• full_stacktraces log only stacktraces generated by rasdaman (false), or full stacktraces including all external libraries (true). It is recommended to keep this setting to false for shorter exception stacktraces in petascope.log.

• Default: false

• Need to change: NO

• inspire_common_url set the URL to an external catalog service for the INSPIRE standard, to be provided by the user. If not set then it will be automatically derived from the petascope_servlet_url setting.

• Default: empty

• Need to change: NO

#### 5.9.2.3. Deployment¶

• java_server specify how is rasdaman-geo deployed: embedded starts the Web application standalone with embedded Tomcat, listening on the server.port setting as configured below, while external indicates that rasdaman.war is deployed in the webapps dir of external Tomcat.

It is recommended to set embedded, as there is no dependency on external Tomcat server, petascope.log can be found in the rasdaman log directory /opt/rasdaman/log, and start/stop of rasdaman-geo is in sync with starting/stopping the rasdaman service. Setting to external on the other hand can be preferred when there is already an existing Tomcat server running other Web applications.

• Default: embedded

• Need to change: NO, unless rasdaman-geo is deployed in external Tomcat

• server.port set the port on which embedded rasdaman-geo (java_server=embedded above) will listen when rasdaman starts. This setting has no effect when java_server=external.

• Default: 8080

• Need to change: YES when port 8080 is occupied by another process, e.g. external Tomcat

• static_html_dir_path absolute path to a directory containing static demo Web pages (html/css/javascript). If set, rasdaman-geo will serve the index.html in this directory at the /rasdaman endpoint, e.g. http://localhost:8080/rasdaman/. Changes of files in this directory do not require a rasdaman-geo restart. The system user running Tomcat (if java_server=external) or rasdaman (if java_server=embedded) must have read permission on this directory.

• Default: empty

• Need to change: YES when demo web pages required under radaman-geo’s endpoint

#### 5.9.2.4. Rasdaman¶

• rasdaman_url set the connection URL to the rasdaman database. Normally rasdaman is installed on the same machine, so the bellow needs no changing (unless the default rasmgr port 7001 has changed).

• Default: http://localhost:7001

• Need to change: NO, unless changed in rasdaman (not recommended)

• rasdaman_database set the name of the rasdaman database (configured in /opt/rasdaman/etc/rasmgr.conf).

• Default: RASBASE

• Need to change: NO, unless changed in rasdaman (not recommended)

• rasdaman_user set the user for unauthenticated read-only access to rasdaman. Any request which does not provide credentials for a rasdaman user in basic authentication format in the HTTP Authorization header, will entail executing read-only operations with this user in rasdaman. It is best to limit this user to read-only access in rasdaman by granting the R permission to it.

• Default: rasguest

• Need to change: YES when changed in rasdaman

• rasdaman_pass set the password for the user set for rasdaman_user. It is recommended to change the default password for rasguest user in rasdaman and update the value here.

• Default: rasguest

• Need to change: YES when changed in rasdaman

• rasdaman_admin_user this user is used to map updating OGC requests (e.g. during data import, or deleting coverages) to updating rasql queries, for any request which does not provide credentials for a rasdaman user in basic authentication format in the HTTP Authorization header. Additionally, these credentials are used internally for various tasks which require admin access rights in rasdaman.

Generally, this user should be granted full admin permissions.

• Default: rasadmin

• Need to change: YES when changed in rasdaman

• rasdaman_admin_pass set the password for the user set for rasdaman_admin_user. It is recommended to change the default password for rasadmin user in rasdaman and update the value here.

• Default: rasadmin

• Need to change: YES when changed in rasdaman

• rasdaman_retry_attempts set the number of re-connect attempts to a rasdaman server in case a connection fails.

• Default: 5

• Need to change: NO

• rasdaman_retry_timeout set the wait time in seconds between re-connect attempts to a rasdaman server.

• Default: 10 (seconds)

• Need to change: NO

• rasdaman_bin_path absolute path to the rasdaman executables directory.

• Default: /opt/rasdaman/bin

• Need to change: NO

#### 5.9.2.5. Security¶

• allow_write_requests_from configure from which IP addresses (as a comma-separated list) should the server accept write requests such as WCS-T InsertCoverage, UpdateCoverage and DeleteCoverage. 127.0.0.1 will allow locally generated requests, usually needed to import data with wcst_import.sh; setting to empty will block all requests, while * will allow any IP address.

Note

This setting (i.e. the origin IP) is ignored when a request contains basic auth credentials for a valid rasdaman user with RW rights in the HTTP Authorization header.

• Default: 127.0.0.1

• Need to change: NO, unless more IP addresses should be allowed to execute write requests

• security.require-ssl allow embedded petascope to work with HTTPS requests from its endpoint.

• Default: false

• Need to change: NO

#### 5.9.2.6. Logging¶

rasdaman-geo uses the log4j library version 1.2.17 provided by Spring Boot version 1.5.2 to log information/errors in a petascope.log file. See the log4j 1.2 document for more details.

• Configuration for petascope logging; by default only level INFO or higher is logged to a file. The valid logging levels are TRACE, DEBUG, INFO, WARN, ERROR and FATAL.

log4j.rootLogger=INFO, rollingFile

• Configuration for reducing logs from external libraries: Spring, Hibernate, Liquibase, GRPC and Netty.

log4j.logger.org.springframework=WARN
log4j.logger.org.hibernate=WARN
log4j.logger.liquibase=WARN
log4j.logger.io.grpc=WARN
log4j.logger.io.netty=WARN
log4j.logger.org.apache=WARN

• Configure file logging. The paths for file logging specified below should be write-accessible by the system user running Tomcat. If running embedded Tomcat, then the files should be write accessible by the system user running rasdaman, which is normally rasdaman.

log4j.appender.rollingFile.layout=org.apache.log4j.PatternLayout
log4j.appender.rollingFile.layout.ConversionPattern=%6p [%d{yyyy-MM-dd HH:mm:ss}] %c{1}@%L: %m%n

• Select one strategy for rolling files and comment out the other. Default is rolling files by time interval.

# 1. Rolling files by maximum size and index
#log4j.appender.rollingFile.File=@LOG_DIR@/petascope.log
#log4j.appender.rollingFile.MaxFileSize=10MB
#log4j.appender.rollingFile.MaxBackupIndex=10
#log4j.appender.rollingFile=org.apache.log4j.RollingFileAppender

# 2. Rolling files by time interval (e.g. once a day, or once a month)
log4j.appender.rollingFile.rollingPolicy.ActiveFileName=@LOG_DIR@/petascope.log
log4j.appender.rollingFile.rollingPolicy.FileNamePattern=@LOG_DIR@/petascope.%d{yyyyMMdd}.log.gz
log4j.appender.rollingFile=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.rollingFile.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy


### 5.9.3. Security¶

By default only local IP addresses are allowed to make write requests to petascope (e.g. InsertCoverage and UpdateCoverage when importing data, or DeleteCoverage, etc). This is configured through the allow_write_requests_from setting in petascope.properties.

Any write requests from a non-listed IP address will be blocked. However, if one has a rasdaman user credentials with RW rights (see user rights), then one can send write requests with these credentials via basic authentication header. This authentication mechanism is used by the WSClient for example when logged in with the petascope admin credentials, to enable deleting coverages, updating metadata, styles, etc.

Note

Since v10+, the petascope admin user configured in petascope.properties by settings petascope_admin_user and petascope_admin_pass has no effect. One must use credentials of a rasdaman user with RW rights to perform a request with the basic header authentication method.

### 5.9.4. Meta Database Connectivity¶

Non-array data of coverages (here loosely called metadata) are stored in another database, separate from the rasdaman database. This backend is configured in petascope.properties.

As a first action it is highly recommended to substitute {db-username} and {db-password} by some safe settings; keeping obvious values constitutes a major security risk.

Note that the choice is exclusive: only one such database can be used at any time. Changing to another database system requires a database migration which is entirely the responsibility of the service operator and involves substantially more effort than just changing these entries; generally, it is strongly discouraged to change the meta database backend.

If necessary, add the path to the JDBC jar driver to petascope.properties using metadata_jdbc_jar_path and spring.datasource.jdbc_jar_path.

Several different systems are supported as metadata backends. Below is a list of petascope.properties settings for different systems that have been tested successfully with rasdaman.

#### 5.9.4.1. Postgresql (default)¶

The following configuration in petascope.properties enables PostgreSQL as metadata backend:

▶ show

#### 5.9.4.2. HSQLDB¶

The following configuration in petascope.properties enables HSQLDB as metadata backend:

▶ show

#### 5.9.4.3. H2¶

The following configuration in petascope.properties enables H2 as metadata backend:

▶ show

### 5.9.5. petascope Standalone Deployment¶

The petascope Web application can be deployed through any suitable servlet container, or (recommended) can be operated standalone using its built-in embedded container. The embedded variant is activated through setting java_server=embedded in $RMANHOME/etc/petascope.properties. To configure embedded mode, the following options will need to be checked and adjusted: • petascope.properties java_server=embedded server.port=8080 # a path writable by the rasdaman user log4j.appender.rollingFile.File=/opt/rasdaman/log/petascope.log # or log4j.appender.rollingFile.rollingPolicy.ActiveFileName=/opt/rasdaman/log/petascope.log  • secore.properties # paths writable by the rasdaman user secoredb.path=/opt/rasdaman/data/secore log4j.appender.rollingFile.File=/opt/rasdaman/log/secore.log log4j.appender.rollingFile.rollingPolicy.ActiveFileName=/opt/rasdaman/log/secore.log  In the standalone mode petascope can be started individually using the central startup/shutdown scripts of rasdaman: $ sudo -u rasdaman start_rasdaman.sh --service petascope
$sudo -u rasdaman stop_rasdaman.sh --service petascope  The Web application can be even be started from the command line: $ java -jar rasdaman.war [ --petascope.confDir={path-to-etc-dir} ]


The port required by the embedded tomcat will be fetched from the server.port setting in petascope.properties. Assuming the port is set to 8080, petascope can be accessed via URL http://localhost:8080/rasdaman/ows.

### 5.9.6. Serving Static Content¶

Serving external static content (such as HTML, CSS, and Javascript) residing outside rasdaman.war through petascope can be enabled with the following setting in petascope.properties:

static_html_dir_path={absolute-path-to-index.html}


with an absolute path to a directory readable by the user running petascope. The directory must contain an index.html, which will be served as the root, ie: at URL http://localhost:8080/rasdaman/.

### 5.9.7. Logging¶

Configuration file petascope.properties also defines logging. The log level can be adjusted in verbosity, log file path can be set, etc. Tomcat restart is required for new settings to become effective.

The user running Tomcat (tomcat or so) must have write permissions to the petascope.log file specified if java_server=external; usually the file should be placed in the Tomcat log directory in this case, e.g. /var/log/tomcat/petascope.log.

Otherwise, if java_server=embedded, then the user running rasdaman must have write permissions to the specified log file; usually the file would be placed in the rasdaman log directory in this case, e.g. /opt/rasdaman/log/petascope.log.

## 5.10. Geo Service Standards Compliance¶

rasdaman community is OGC WCS reference implementation and supports the following conformance classes:

• OGC CIS:

• CIS 1.0:

• Class GridCoverage

• Class RectifiedGridCoverage

• Class ReferenceableGridCoverage

• Class gml-coverage

• Class multipart-coverage

• CIS 1.1:

• Class grid-regular

• Class grid-irregular

• Class gml-coverage

• Class json-coverage

• Class other-format-coverage

• OGC WCS

• WCS 2.0:

• WCS Core

• WCS Range Subsetting

• WCS Processing (supporting WCPS 1.0)

• WCS Transaction

• WCS CRS

• WCS Scaling

• WCS Interpolation

• WMS 1.3.0:

• all raster functionality, including SLD ColorMap styling

Note

With WCS 2.1, petascope provides an additional proprietary parameter to request CIS 1.0 coverages to be returned as CIS 1.1 coverages. This is specified by adding parameter outputType=GeneralGridCoverage`.