Storage
========

Overview
---------
Players in strax's storage system take on one of three roles:
  * ``StorageFrontend``: Find data locations, and communicate this to one or more ``StorageBackend`` instances;
  * ``StorageBackend``: load pieces of data, and create instances of ``Saver``;
  * ``Saver``: save pieces of data to a specific location.

As an example, a ``StorageFrontend`` could talk to a database that tracks which data is stored where.
A ``StorageBackend`` then retrieves data from local disks, while another might retrieve it remotely using SSH or other transfer systems.
The front-end decides which backend is appropriate for a given request. Finally, a ``Savers`` guides the process of writing a particular
piece of data to disk or databases (potentially from multiple cores), compressing and rechunking as needed.

To implement a new way of storing and/or tracking data, you must implement (subclass) all or some of these classes.
This means subclassing them and overriding a few specific methods
(called 'abstract methods' because they ``raise NotImplementedError`` if they are not overridden).

Keys
-----
In strax, a piece of data is identified by a *DataKey*, consisting of three components:
  * The run id
  * The data type
  * The complete *lineage* of the data. This includes, for the data type itself, and all types it depends on (and their dependencies, and so forth):
    * The plugin class name that produced the data;
    * The version string of the plugin;
    * The values of all configuration options the plugin took (whether they were explicitly specified or left as default).

When you ask for data using ``Context.get_xxx``, the context will produce a key like this, and pass it to the ``StorageFrontend``.
It then looks for a filename or database collection name that matches this key -- something a ``StorageBackend`` understands. which is therefore generically called a *backend key*.
The matching between DataKey and backend key can be done very strictly, or more loosely, depending on how the context is configured.
This way you can choose to be completely sure about what data you get, or be more flexible and load whatever is available.
TODO: ref context documentation.


Run-level metadata
-------------------
Metadata can be associated with a run, but no particular data type. The ``StorageFrontend`` must take care of saving and loading these.

Such run-level metadata can be crucial in providing run-dependent default setting for configuration options, for example, calibrated quantities necessary
for data processing (e.g. electron lifetime and PMT gains).