Beacon Network¶

Beacon Network is a service for gathering multiple Beacons together for easy access.

Beacon Network as a service consists of:

A Registry for holding data regarding member Beacons
An Aggregator that serves as a gateway to query all known Beacons at once
A web portal that provides a GUI to use the APIs described above
An OIDC client used for authenticating users and authorizing access to protected data

This documentation covers the configuration, installation and deployment of the Registry and Aggregator APIs.

For information regarding the other relevant service components, see:

Table of Contents¶

Configuration¶

Aggregator¶

The aggregator can be configured with the configuration file located by default in beacon-network/aggregator/config/config.ini.

Configuration File¶

Configuration variables for setting up the web application are found in the [app] section.

[app]
# Application hostname, overwritten by ENV $APP_HOST
host=0.0.0.0

# Port for host, overwritten by ENV $APP_PORT
port=8080

# List of registries this aggregator is using
registries=aggregator/config/registries.json

# Boolean if this Aggregator wants to query Beacons
beacons=True

# Boolean if this Aggregator wants to query Aggregators
aggregators=True

# CORS domain, a single domain, or * for any domain. Leave empty for no CORS
cors=*

Configuration variables for defining the /service-info endpoint are found in the [info] section.

[info]
# Name of this service
name=ELIXIR-FI Beacon Aggregator

# GA4GH scoped service type with specification version
type_group=org.ga4gh
type_artifact=beacon-aggregator
type_version=1.0.0

# Description for this service
description=ELIXIR-FI Beacon Aggregator at CSC for Beacon network

# Location of technical documentation or user guide
documentation_url=https://beacon-network.readthedocs.io/en/latest/

# Name of organization
organization=CSC - IT Center for Science Ltd.

# URL to organization homepage
organization_url=https://www.csc.fi/

# URL for contact information for the maintainer of this service, or alternatively `mailto:person@place.org` notation
contact_url=https://www.csc.fi/contact-info

# Internal software version
version=1.1.0

# Time of server's first creation
create_time=2019-09-04T12:00:00Z

# Server environment. Possible values: prod, dev, test
environment=dev

Registries File¶

The registries.json file acts as the aggregator’s database. The array can be populated by multiple objects, and the aggregator will contact all of the listed url keys, which should point to /services endpoint at registries. If the aggregator is registered as a service at the registry, the serviceKey from the registration response can be put into the key key, which allows the registry to call the DELETE /cache endpoint at the aggregator to remove the cached list of beacons from the aggregator. This triggers the aggregator to request a new up-to-date list of beacons from the registry. This can be useful to let the aggregator know about changes in the registry’s service catalogue.

[
    {
        "url": "http://localhost:9090/services",
        "key": "secret"
    }
]

Environment Variables¶

ENV	Default	Description
CONFIG_FILE	config.ini	Location of configuration file.
DEBUG	False	Set to True to enable more debugging logs from functions.
APP_HOST	0.0.0.0	Application hostname.
APP_PORT	8080	Application port.
APPLICATION_SECURITY	0	Application security level, determines the SSL operating principle of the server. Possible values are 0-2, more information in SSL Context section below.
REQUEST_SECURITY	0	Request security level, determines the SSL operating principle of requests. Possible values are 0-2, more information in SSL Context section below.
PATH_SSL_CERT_FILE	/etc/ssl/certs	Path to certificate.pem file.
PATH_SSL_KEY_FILE	/etc/ssl/certs	Path to key.pem file.
PATH_SSL_CA_FILE	/etc/ssl/certs	Path to ca.pem file.
APP_CORS		CORS domain, either a single domain or * for any domain.

Registry¶

Configuration File¶

Configuration variables for setting up the web application and database connection are found in the [app] section.

[app]
# Application hostname, overwritten by ENV $APP_HOST
host=0.0.0.0

# Port for host, overwritten by ENV $APP_PORT
port=8080

# Database hostname, overwritten by ENV $DB_HOST
db_host=localhost

# Database port, overwritten by ENV $DB_PORT
db_port=5432

# Database username, overwritten by ENV $DB_USER
db_user=user

# Password for database user, overwritten by ENV $DB_PASS
db_pass=pass

# Database name, overwritten by ENV $DB_NAME
db_name=registry

# Is the API key for POST /services an OTP?
# If 'True', the API key is expired after use, if 'False', the API key can be used again
api_otp=True

# CORS domain, a single domain, or * for any domain
cors=*

Configuration variables for defining the /service-info endpoint are found in the [info] section.

# Set dev enviroment off ie. https checks are on
dev=False

[info]
# Name of this service
name=ELIXIR-FI Beacon Registry

# GA4GH scoped service type with specification version
type_group=org.ga4gh
type_artifact=service-registry
type_version=1.0.0

# Description for this service
description=ELIXIR-FI Beacon Registry at CSC for Beacon network

# Location of technical documentation or user guide
documentation_url=https://beacon-network.readthedocs.io/en/latest/

# Short identifier of host organization, preferably universally unique, e.g. reverse domain
organization=CSC - IT Center for Science Ltd.

# URL to organization homepage
organization_url=https://www.csc.fi/

# URL for contact information for the maintainer of this service, or alternatively `mailto:person@place.org` notation
contact_url=https://www.csc.fi/contact-info

# Internal software version
version=1.1.0

# Time of server's first creation
create_time=2019-09-04T12:00:00Z

Environment Variables¶

ENV	Default	Description
CONFIG_FILE	config.ini	Location of configuration file.
DEBUG	False	Set to True to enable more debugging logs from functions.
APP_HOST	0.0.0.0	Application hostname.
APP_PORT	8080	Application port.
APPLICATION_SECURITY	0	Application security level, determines the SSL operating principle of the server. Possible values are 0-2, more information in SSL Context section below.
REQUEST_SECURITY	0	Request security level, determines the SSL operating principle of requests. Possible values are 0-2, more information in SSL Context section below.
PATH_SSL_CERT_FILE	/etc/ssl/certs	Path to certificate.pem file.
PATH_SSL_KEY_FILE	/etc/ssl/certs	Path to key.pem file.
PATH_SSL_CA_FILE	/etc/ssl/certs	Path to ca.pem file.
DB_HOST	localhost	Database address.
DB_PORT	5432	Database port.
DB_USER	user	Username to access database.
DB_PASS	pass	Password for database user.
DB_NAME	db	Database name.
API_OTP	True	Boolean if API key at POST /services should be expired after use.
APP_CORS		CORS domain, either a single domain or * for any domain.

SSL¶

Experimental!! In production a reverse proxy is recommended.

Possible security levels for APPLICATION_SECURITY and REQUEST_SECURITY are 0-2.

Security Level	APPLICATION_SECURITY Behaviour	REQUEST_SECURITY Behaviour
0	Application is unsafe. API is served as HTTP.	Application can make requests to HTTP (unsafe) and HTTPS (safe) resources.
1	Application is safe. API is served as HTTPS. This requires the use of PATH_SSL_* ENVs.	Application can only make requests to HTTPS (safe) resources. Requests to HTTP (unsafe) resources are blocked.
2	Application belongs to a closed trust network. Applies same behaviour as security level 1. Application can only be requested from other applications that belong to the same trust network.	Application can only make requests to applications that belong to the same trust network (see previous cell description).

Installation and Deployment¶

Instructions for setting up the Aggregator and Registry APIs and their database.

Note

APIs and database can be set up in containers or locally. Requirements for installation:

Containerised

Docker

Local

Python 3.8+
PostgreSQL 12.6+

Installation¶

Install both Aggregator and Registry APIs with a single command.

git clone https://github.com/CSCfi/beacon-network
cd beacon-network
pip install .

The APIs can then be run with the following commands:

# Run aggregator
beacon_aggregator

# Run registry
beacon_registry

This starts the web applications with aiohttp.web.run_app using aiohttp’s default server. This is a lightweight way to run the apps, but for more stability see the Production Server section below.

Development Server¶

For development, both APIs can be run withouth installation using aiohttp’s default web.run_app method.

cd beacon-network
# If running without installation, install modules first
pip install -r requirements.txt

# Run aggregator
python -m aggregator.aggregator

# Run Registry
python -m registry.registry

Production Server¶

For production it is recommended to use gunicorn instead of aiohttp’s default server for stability.

Environment variables APP_HOST and APP_PORT can be used to designate a hostname.

# Run aggregator
gunicorn aggregator.aggregator:init_app --bind $APP_HOST:$APP_PORT \
                                        --worker-class aiohttp.GunicornUVLoopWebWorker \
                                        --workers 4

# Run registry
gunicorn registry.registry:init_app --bind $APP_HOST:$APP_PORT \
                                    --worker-class aiohttp.GunicornUVLoopWebWorker \
                                    --workers 4

Image Building¶

Aggregator and Registry can also be built into an image and then be deployed. First build the image. The same base image will be used for both services.

Either s2i or docker can be used to build an image from the source code. s2i builds very quickly, but is rather large in size ~700MB. docker builds the image with Dockerfile and results in an image roughly half the size of s2i at ~370MB, but takes considerably longer to build.

# Build using s2i
s2i build . centos/python-36-centos7 cscfi/beacon-network

# Build using docker
docker build -t cscfi/beacon-network .

Manual Container Deployment¶

Run the images with docker, specifying which service to run with the environment variable BEACON_RUN_APP which can take only two distinct values: aggregator and registry.

# Run aggregator
docker run -d -p 8080:8080 -e BEACON_RUN_APP=aggregator cscfi/beacon-network

# Run registry
docker run -d -p 8080:8080 -e BEACON_RUN_APP=registry cscfi/beacon-network

Other environment variables can also be passed here to overwrite the values given in the configuration file.

Database Container¶

Note

The Registry API is dependent on a PostgreSQL database. This can be easily set up in a container as well. The created database will be populated with the init.sql located at registry/db.

cd beacon-network/registry/db
  docker run -d --name registry-db \
  -e POSTGRES_USER=user \
  -e POSTGRES_PASSWORD=pass \
  -e POSTGRES_DB=registry \
  -v "$PWD"/docker-entrypoint-initdb.d/:/docker-entrypoint-initdb.d/ \
  -p 5432:5432 postgres:12.6

Docker Compose Deployment¶

docker-compose can be leveraged to launch both Aggregator and Registry APIs with a database for Registry simultaneously.

Simply:

docker-compose up

Note

The image must be built with docker in order for this to work, see Image Building section above.

API Examples¶

For reference on how to use the API endpoints, please refer to the specification below:

API Specification for the current version in master.

Examples¶

The following examples have been conducted against live APIs.

Aggregator Examples¶

The Aggregator serves as a single querying point that propagates the user query to multiple Beacons, and aggregates their responses.

Get API Info¶

API info is requested with GET method from the /service-info endpoint. The response follows GA4GH Service Info specification.

Request¶

curl localhost:5000/service-info

Response¶

{
    "id": "localhost:5000",
    "name": "ELIXIR-FI Beacon Aggregator",
    "type": {
        "group": "org.ga4gh",
        "artifact": "beacon-aggregator",
        "version": "1.1.0",
    },
    "description": "ELIXIR-FI Beacon Aggregator at CSC for Beacon network",
    "organization": {
        "name": "CSC - IT Center for Science Ltd.",
        "url": "https://csc.fi/"
    },
    "contactUrl": "https://www.csc.fi/contact-info",
    "documentationUrl": "https://beacon-network.readthedocs.io/en/latest/",
    "createdAt": "2019-09-04T12:00:00Z",
    "updatedAt": "2019-09-04T12:00:00Z",
    "environment": "dev",
    "version": "0.3.dev"
}

Query Beacons¶

Beacons are queried with GET method from the /query endpoint. The response follows GA4GH Beacon specification. The response is an array of these objects. The Aggregator can be queried synchronously with HTTPS and asynchronously with WSS (websocket).

Request¶

Standard HTTP query with synchronous answer

curl localhost:5000/query?assemblyId=GRCh38&referenceName=MT&start=9&referenceBases=T&alternateBases=C

For a websocket query, the following headers are required: Connection: Upgrade and Upgrade: Websocket. The requester should be a websocket client capable of receiving the data stream.

Response¶

[
    {
        "beaconId": "se.nbis.swefreq",
        "apiVersion": "1.1.0",
        "exists": false,
        "alleleRequest": {
            "referenceName": "MT",
            "referenceBases": "T",
            "assemblyId": "GRCh38",
            "includeDatasetResponses": "NONE",
            "alternateBases": "C",
            "start": 9
        },
        "datasetAlleleResponses": [],
        "beaconHandover": []
    },
    {
        "beaconId": "fi.rahtiapp.beaconpy-elixirbeacon",
        "apiVersion": "1.1.0",
        "exists": true,
        "alleleRequest": {
            "referenceName": "MT",
            "referenceBases": "T",
            "assemblyId": "GRCh38",
            "includeDatasetResponses": "NONE",
            "alternateBases": "C",
            "start": 9
        },
        "datasetAlleleResponses": []
    },
    {
        "beaconId": "fi.rahtiapp.staging-elixirbeacon",
        "apiVersion": "1.1.0",
        "exists": true,
        "alleleRequest": {
            "referenceName": "MT",
            "referenceBases": "T",
            "assemblyId": "GRCh38",
            "includeDatasetResponses": "NONE",
            "alternateBases": "C",
            "start": 9
        },
        "datasetAlleleResponses": []
    }
]

Delete Cached Beacons¶

Aggregator cache is deleted with DELETE method from the /cache endpoint. This endpoint is protected with a header API key named Authorization. This key is read from the registries.json file from the key key. More information below.

When the Aggregator is queried, the Aggregator requests a list of Beacons from a pre-configured Registry. This list is then cached for one hour for sequential queries. Should the service catalogue at the Registry change, a new list of Beacons should be supplied to the Aggregator. Registries can automatically call this endpoint if the Aggregator has been registered at the Registry, and the Aggregator has saved the service key to its registries.json in they key key.

[
    {
        "url": "http://localhost:9090/services",
        "key": "secret"
    }
]

Request¶

curl -x DELETE \
     localhost:5000/cache \
     -H 'Authorization: secret'

Response¶

Cache has been deleted.

Registry Examples¶

The Registry serves as a central hub of known Beacons. The Aggregator utilises this catalogue for queries.

Get API Info¶

API info is requested with GET method from the /service-info endpoint. The response follows GA4GH Service Info specification.

Request¶

curl localhost:8080/service-info

Response¶

{
    "id": "localhost:8080",
    "name": "ELIXIR-FI Beacon Registry",
    "type": {
        "group": "org.ga4gh",
        "artifact": "service-registry",
        "version": "1.1.0",
    },
    "description": "ELIXIR-FI Beacon Registry at CSC for Beacon network",
    "organization": {
        "name": "CSC - IT Center for Science Ltd.",
        "url": "https://csc.fi/"
    },
    "contactUrl": "https://www.csc.fi/contact-info",
    "documentationUrl": "https://beacon-network.readthedocs.io/en/latest/",
    "createdAt": "2019-09-04T12:00:00Z",
    "updatedAt": "2019-09-04T12:00:00Z",
    "environment": "dev",
    "version": "0.2.dev"
}

List Service Types¶

Service types are requested with GET method from the /services/types endpoint.

Request¶

curl localhost:8080/services/types

Response¶

[
    "service-registry",
    "beacon-aggregator",
    "beacon"
]

Register a New Service¶

A new service can be registered with POST method at the /services endpoint. This endpoint is protected with a header API key named Authorization. This key is read from the database table api_keys.

The registration process requires a maintainer email address, URL to service info endpoint and type of service. The Registry then attempts to pull the service information from the given URL. The endpoint should follow either GA4GH Beacon API specification / or GA4GH Service Info specification /service-info.

Request¶

curl -X POST \
localhost:8080/services \
-H 'Authorization: secret' \
-d '{
    "type": "beacon",
    "url": "http://localhost:3000/"
}'

Response¶

{
    "message": "Service has been registered. Service key and id for updating and deleting registration included in this response, keep them safe.",
    "serviceId": "localhost:3000",
    "serviceKey": "secret",
    "help": "https://beacon-network.readthedocs.io/en/latest/"
}

List Registered Services¶

Services can be listed with GET method on the /services endpoint.

Request¶

curl localhost:8080/services

Response¶

[
    {
        "id": "fi.rahtiapp.staging-elixirbeacon",
        "name": "GA4GHBeacon at CSC",
        "type": {
            "group": "org.ga4gh",
            "artifact": "beacon",
            "version": "1.1.0",
        },
        "description": "Beacon API Web Server based on the GA4GH Beacon API",
        "organization": {
            "name": "CSC - IT Center for Science",
            "logoUrl": "https://www.csc.fi/documents/10180/161914/CSC_2012_LOGO_RGB_72dpi.jpg",
            "url": "https://www.csc.fi/"
        }
        "createdAt": "2019-07-25 10:57:34.238533+00:00",
        "updatedAt": "2019-08-02 00:00:13.006256+00:00",
        "contactUrl": "https://www.csc.fi/contact-info",
        "environment": "dev",
        "version": "1.6.5",
        "url": "https://staging-elixirbeacon.rahtiapp.fi/"
    },
    {
        "id": "fi.rahtiapp.beaconpy-elixirbeacon",
        "name": "GA4GHBeacon at CSC",
        "type": {
            "group": "org.ga4gh",
            "artifact": "beacon",
            "version": "1.1.0",
        },
        "description": "Beacon API Web Server based on the GA4GH Beacon API",
        "organization": {
            "name": "CSC - IT Center for Science",
            "logoUrl": "https://www.csc.fi/documents/10180/161914/CSC_2012_LOGO_RGB_72dpi.jpg",
            "url": "https://www.csc.fi/"
        }
        "createdAt": "2019-07-25 10:57:28.863961+00:00",
        "updatedAt": "2019-08-02 00:00:13.016122+00:00",
        "contactUrl": "https://www.csc.fi/contact-info",
        "environment": "prod",
        "version": "1.6.5",
        "url": "https://staging-elixirbeacon.rahtiapp.fi/"
    },
    {
        "id": "fi.rahtiapp.dev-aggregator-beacon",
        "name": "ELIXIR-FI Beacon Aggregator",
        "type": {
            "group": "org.ga4gh",
            "artifact": "beacon-aggregator",
            "version": "1.1.0",
        },
        "description": "ELIXIR-FI Beacon Aggregator at CSC for Beacon network",
        "organization": {
            "name": "CSC - IT Center for Science",
            "logoUrl": "",
            "url": "https://www.csc.fi/"
        }
        "createdAt": "2019-07-25 10:57:11.340018+00:00",
        "updatedAt": "2019-08-05 00:00:10.960787+00:00",
        "contactUrl": "https://www.csc.fi/contact-info",
        "environment": "dev",
        "version": "0.2.dev",
        "url": "https://dev-aggregator-beacon.rahtiapp.fi/service-info",
    }
]

Find Service by ID¶

A specific service can be found with GET method on the /services endpoint by supplying the service ID as a path parameter in /services/<id>.

Request¶

curl localhost:8080/services/fi.rahtiapp.beaconpy-elixirbeacon

Response¶

{
    "id": "fi.rahtiapp.staging-elixirbeacon",
    "name": "GA4GHBeacon at CSC",
    "type": {
        "group": "org.ga4gh",
        "artifact": "beacon",
        "version": "1.1.0",
    },
    "description": "Beacon API Web Server based on the GA4GH Beacon API",
    "organization": {
        "name": "CSC - IT Center for Science",
        "logoUrl": "https://www.csc.fi/documents/10180/161914/CSC_2012_LOGO_RGB_72dpi.jpg",
        "url": "https://www.csc.fi/"
    }
    "createdAt": "2019-07-25 10:57:34.238533+00:00",
    "updatedAt": "2019-08-02 00:00:13.006256+00:00",
    "contactUrl": "https://www.csc.fi/contact-info",
    "environment": "dev",
    "version": "1.6.5",
    "url": "https://staging-elixirbeacon.rahtiapp.fi/"
}

Update Service¶

An existing service can be updated with PUT method at the /services endpoint with the service ID specified as a path parameter /service/<id>. This endpoint is protected with a header API key named Beacon-Service-Key which was given to the registrar at registration. This key is read from the database table service_keys.

The update process follows the same principles as the registration process.

Request¶

curl -X PUT \
localhost:8080/services/localhost:3000 \
-H 'Beacon-Service-Key: secret' \
-d '{
    "type": "beacon",
    "url": "http://localhost:3000/"
}'

Response¶

Service has been updated.

Delete Service¶

A registered service can be deleted from the Registry with DELETE method on the /services endpoint with the service ID specified as a path parameter /service/<id>. This endpoint is protected with header Beacon-Service-Key which was given to the registrar at registration.

Request¶

curl -X DELETE \
localhost:8080/services/localhost:3000 \
-H 'Beacon-Service-Key: secret'

Response¶

Service has been deleted.

Update Service Catalogue¶

The whole service catalogue at a Registry can be updated with GET method on the /update/services endpoint. This endpoint is protected with a header API key named Authorization. This key is read from the database table admin_keys.

This event will trigger the Registry to fetch up-to-date information frmo all registered services and to update the database accordingly.

Request¶

curl -X GET \
localhost:8080/update/services \
-H 'Authorization: secret'

Response¶

3 successful update(s). 0 failed update(s).