Platform aggregating electronic ressources usage statistics for the French researcher organizations. https://ezmesure.couperin.org
git clone https://github.com/ezpaarse-project/ezmesure.git
ezMESURE uses an Apache reverse proxy which communicates with HTTPS only. Put the SSL certificate (server.pem
) and private key (server.key
) in rp/apache2/ssl
. Optional if ezMESURE runs behind a proxy which takes care of handling HTTPS.
NB: you can use mkcert for local development
Create an environment file named ezmesure.local.env.sh
and export the following environment variables. You can then source ezmesure.env.sh
, which contains a set of predefined variables and is overriden by ezmesure.local.env.sh
.
NB: a helper script is available at tools/init_env.sh
.
name | description |
---|---|
EZMESURE_DOMAIN | the server domain |
APPLI_APACHE_SERVERADMIN | the admin of the server |
EZMESURE_AUTH_SECRET | secret for JWT signing |
ELASTICSEARCH_PASSWORD | password of the elastic user |
EZMESURE_SMTP_HOST | host of the SMTP server |
EZMESURE_NOTIFICATIONS_SENDER | the sender for emails issued by ezMESURE |
EZMESURE_NOTIFICATIONS_RECIPIENTS | recipients of the recent activity email |
EZMESURE_NOTIFICATIONS_SUPPORT_RECIPIENTS | recipients of the recent activity email |
EZMESURE_NOTIFICATIONS_CRON | cron for recent activity (defaults to daily midnight) |
REPORTING_SENDER | the sender for reporting emails |
If all services are not hosted on the same machine, set the following variables (URLs should not have trailing slashes) :
name | description |
---|---|
EZMESURE_FRONT_URL | Base URL of the front website. Must be reachable by the reverse proxy . |
EZMESURE_API_URL | Base URL of the API. Must be reachable by the reverse proxy . |
EZMESURE_REPORTING_URL | Base URL of the reporting service. Must be reachable by Kibana . |
KIBANA_INTERNAL_URL | Kibana URL used by the reporting service in order to access dashboards. Must pass through the reverse proxy (ie. RP address + /kibana ). |
EZMESURE_KIBANA_HOST | Kibana host. Must be reachable by the reverse proxy and the API . |
EZMESURE_KIBANA_PORT | Kibana port. |
EZMESURE_ELASTIC_HOST | Elasticsearch host. Must be reachable by the API and reporting service |
EZMESURE_ELASTIC_PORT | Elasticsearch port. |
Put the certificate (server.crt
) and private key (server.key
) used to declare the Shibboleth service provider in the fédération d'identités Education-Recherche in rp/shibboleth/ssl/
.
NB: the private key is critical and should not be shared.
Additionnaly, set the environment variables SHIBBOLETH_SP_URL
and SHIBBOLETH_DS_URL
with the URL of the service provider and discovery service. Those variables are not necessary if you disable Shibboleth authentication (see below).
If you don't need the shibboleth authentication, set the EZMESURE_DISABLE_SHIBBOLETH
environment variable to any value. If you already started ezMESURE, rebuild the front
service and restart front
and rp
:
docker-compose up -d --force-recreate front
docker-compose restart rp
For each node in the cluster, add certificates in elasticsearch/config/certificates/
. Kibana should also have certificates in kibana/config/certificates
. If you don't have them yet, you can generate them by following these steps :
- Open the
certs
directory. - Create an instances.yml file. A helper script is available at
tools/init_es_instances.sh
. - Run
docker-compose -f create-certs.yml run --rm create_certs
. - A
bundle.zip
file should be created, just unzip it in the certificates directory (NB: you may need tochown
it) :unzip bundle.zip -d ../elasticsearch/config/certificates/
unzip bundle.zip -d ../kibana/config/certificates/
Elasticsearch has some system requirements that you should check.
To avoid memory exceptions, you may have to increase mmaps count. Edit /etc/sysctl.conf
and add the following line :
# configuration needed for elastic search
vm.max_map_count=262144
Then apply the changes :
sysctl -p
Before you start ezMESURE, make sure all necessary environment variables are set.
# Start ezMESURE as daemon
docker-compose up -d
# Stop ezMESURE
docker-compose stop
# Get the status of ezMESURE services
docker-compose ps
Navigate to https://ezmesure-preprod.couperin.org/myspace and log in with your identity provider. The first time you log into ezMESURE, you'll get a mail with your Kibana credentials.
If the Shibboleth authentication is not enabled, users should be created via the Kibana management page. You can sign in with the elastic
superuser to achieve this. Users can then log into ezMESURE using their Kibana credentials.
An authentication token is required in order to use the API. Once logged, grab your token from the authentication tab.
To use your token, add the following header to your requests: Authorization: Bearer <token>
(replace <token>
with your actual token)
To upload an EC result file in elastic-search, you need to POST
it on the /api/logs/{index_name}
route. For example :
curl -v -X POST https://localhost/api/logs/test-index -F "files[]=@114ee1d0_2016-03-31_10h53.job-ecs.csv" -H "Authorization: Bearer <token>"
You can then issue a GET
request on the /api/logs
route to list your indices :
curl -X GET https://localhost/api/logs -H "Authorization: Bearer <token>"
Now you can access the Kibana instance on https://localhost/kibana/ and start building dashboards.
The ezMESURE API is documented here : https://localhost/api-reference
You should install local dependencies with npm in api
and front
directory;
ezmesure/api npm i
ezmesure/front npm i
You should source ezmesure.env.sh
for the following and before each start.
source ezmesure.env.sh
use mkcert for local development
ezMESURE uses an Apache reverse proxy which communicates with HTTPS only. Put the SSL certificate (server.pem
) and private key (server.key
) in rp/apache2/ssl
.
ezmesure request to elastic in https, to do that, you need to create certificate.
Before that, you need to create instances.yml
file, you need to use a script that will help you to create that in init_es_instance.sh
. This script will pre-fill the necessary fields.
ezmesure/tools/init_es_instances.sh
Adding new instance
Name: <Name>
IP: <IP>
Hostname: <Hostname>
Instance added to ./tools/../certs/instances.yml
Add another instance (Y/n) ? n
// TODO Add elastic in dns
Once the file is created, you can generate the certificates.
ezmesure/certs docker-compose -f create-certs.yml run --rm create_certs
Once the certificates are generated, they must be unzipped and placed in the right folders.
ezmesure/certs sudo unzip bundle.zip -d ../elasticsearch/config/certificates/
ezmesure/certs sudo unzip bundle.zip -d ../kibana/config/certificates/
The Shibboleth authentication process requires the user to be located at ezmesure-preprod.couperin.org
. If working on localhost, add the following line into /etc/hosts
:
127.0.0.1 ezmesure-preprod.couperin.org
On top of that, you have to override the environment variable EZMESURE_DISABLE_SHIBBOLETH
on ezmesure.local.env.sh
export EZMESURE_DISABLE_SHIBBOLETH=""
Don't forget to restore the environment variables after the modification.
Before launching ezmesure, you must create the elastic container, for that you must use this command.
docker-compose -f docker-compose.debug.yml run --rm elastic chown -R elasticsearch /usr/share/elasticsearch/
docker-compose -f docker-compose.debug.yml up -d