# The data passport

Every answer the ICOS zarr proxy (<https://zarr.icos-cp.eu>) gives comes with
a **data passport**: a provenance record of exactly what was delivered, which
sources it came from, how to cite them and under which licence.  It is an
[RO-Crate 1.1](https://w3id.org/ro/crate/1.1) document, JSON-LD.

## Where you find it

| how you read the data | where the passport is |
|---|---|
| `GET /query?...&format=ndjson` (the default) | the last line of the answer, as `{"_passport": {...}}` |
| `format=arrow` | the stream's schema metadata, key `data_passport` |
| `format=parquet` | the file's key-value metadata, key `data_passport` |
| `format=csv` | not in the body, which stays plain data: the headers `X-Data-Citation`, `X-Data-License`, `X-Data-DOI` and `X-Selection-Id` carry its attribution, and the same query as NDJSON or Parquet carries the passport |
| the zarr store itself (`xarray.open_zarr`, zarr-python) | the proxy records every chunk a client reads, per client and store. `GET /<store>/session/passport` answers the passport for what has been read so far; `POST /<store>/session/close` mints it and ends the session. A session idle for 300 seconds is closed and minted by the server. |

## What it says

The graph holds, in this order (the order is stable; the first two places
are the ones clients read):

- **the root dataset** (`./`), first: the store's citation, its licence, and
  the release collection it is part of (`isPartOf`, a DOI);
- **the passport** (`icos:DataPassport`, a fresh `urn:uuid` per answer), second: when
  it was accessed, the client's address anonymised to its /24 (IPv4) network,
  the groups and arrays read, and
  - for a query: the selection as asked (`query`), the stations it covers,
    the rows and bytes delivered, and whether `max_rows` truncated it;
  - for a zarr session: per array, the number of chunks, their bytes and a
    SHA-256 over the chunks' own digests;
- **the arrays** delivered, each a `Dataset`; an array that measures a
  GCOS Essential Climate Variable names it in `variableMeasured`, a
  reference to the ECV's own entity (a `DefinedTerm`: its name and its NERC
  EXV term), which appears once however many arrays measure it.  The ECV is
  the array's own `ecv` / `ecv_uri` attributes, so a variable without one
  (the fluxes, radon, 14CO2) names none;
- **the sources**: each station (`icos:Station`, with its position and
  intake height) and each ICOS data object it is based on, by its handle
  PID with its own citation, and the licence;
- **the RO-Crate metadata descriptor** (`ro-crate-metadata.json`), last:
  the document declares itself an RO-Crate 1.1 about the root dataset.

Every object in the graph carries an `@id`, the nested ones too (`#agent`,
`#query/0`, `#station/<id>/geo`), so an RO-Crate reader such as the
`rocrate` Python library loads the passport as it is.

## Three digests you can check yourself

- **`storeVersion`** is the SHA-256 of the store's root consolidated metadata,
  `GET /<store>/.zmetadata`.  It changes exactly when the store does: a
  rebuild, a near-real-time append, an attribute change.
- **`selectionId`** is the first 32 hex digits of
  `sha256(storeVersion + "\n" + canonicalQuery)`, where `canonicalQuery` is the
  selection with sorted keys and no whitespace (for a zarr session, the
  sorted set of chunk keys read).  No clock and nothing random goes into it:
  the same selection on the same store version always has the same id, so it
  names a selection reproducibly.
- **`passportSha256`** is the SHA-256 of the whole passport rendered as
  `json.dumps(doc, sort_keys=True, separators=(",", ":"))` (ASCII-escaped)
  with this one field set to `null`.  Recomputing it shows the document has
  not been altered.

## Citing what you used

Cite the passport's `citation` (the collection) and, where you used single
stations, the `citation` of each data object it lists.  The licence is
`license` (CC BY 4.0 for ICOS data).  Keeping the passport beside a derived
result records which data, from which store version, went into it.

The same page as HTML, in the site's frame:
<https://zarr.icos-cp.eu/data-passport.html> (`data-passport.html` beside this
file; edit the two together).