...
 
Commits (4)
......@@ -17,3 +17,5 @@ jest.ide.config
.githash
tmp
.cache
json-data
# dbnomics-docker
# dbnomics-docker (aka dbnomics-dev)
Docker Compose stack for DBnomics
Docker Compose stack for DBnomics development environment
* explain usage
* todo (migrate to nomics repos)
* limits (python3.5, ...)
*
\ No newline at end of file
## Prerequisite
Get a Linux computer with a running [docker](https://docs.docker.com/install/) environment.
## Install
Clone the repo on your local computer:
```
$ git clone https://git.nomics.world/dbnomics/dbnomics-docker.git
$ cd dbnomics-docker
$ chmod +x bin/dbnomics-dev*
```
## Usage
3 scripts will allow to graphically test, via a dbnomics-website locale instance, generated json-data.
Testing will be made in 3 steps:
* upload of the environment (solr, api, website)
* loading of data, one can load many provider data many times
* teardown of the environment
### Upload
```
$ dbnomics-docker/bin/dbnomics-dev-up
```
Synopsis:
* creates one docker volume for solr config
* creates one docker volume for json-data
* runs `solr`, available under http://localhost:18983
* runs `dbnomics-api` server, available under http://localhost:15000
* runs `dbnomics-website` server, available under http://localhost:13000
At this time, all servers are installed but with empty index and data.
You can now check `dbnomics-website`.
NB:
* wait for standard output to stop scrolling, before loading data.
* first launching will take more time than next ones due to the downloading of the docker images
## Load and test your json data
```
$ dbnomics-docker/bin/dbnomics-dev-load :somewhere/:provider-json-data
```
Beforehand, you managed to get one (or many) reachable provider's json data folders on your computer.
Just launch previous command, it will load data to the json-data docker volume, shares by `solr` and `dbnomics-api` and run `dbnomics-importer` script.
Check solr logs and dbnomics-dev-load output, and launch your favorite browser to http://localhost:13000, you may see your data !
You can can repeat previous operations for many other provider's data, loaded data are kept until dbnomics-dev is stopped.
You can even, rerun the loader for the same provider but data are copied over previous ones.
### Teardown
```
$ dbnomics-docker/bin/dbnomics-dev-down
```
This is the end, servers are shut down, dokers containers are removed, and volumes are dropped.
### Implementation details
`dbnomics-dev` is strongly based on docker (for linux). It uses docker-compose and docker commands to launch a local stack of dbnomics and solr servers.
First, script `dbnomics-dev-up` pull dbnomics-api and dbnomics-website images. Those images are built from `redpelicans/dnnomics-*` repos and pushed to redpelicans docker hub account.
In the short term, those images must be generated directly from dbnomics-api and dbnomics-website pipelines and pushed to dbnomics docker repos.
Then, `dbnomics-dev-up` init docker volumes. One, `solr-conf`, to setup solr with right `dbnomics` core and dedicated config files, the other, `json-data`, to store incoming json data. Latter is shared between `solr` and `api`.
Time comes to run docker images:
* `dbnomics-dev-solr`: based on solr:7 and extended with python 3.5 with embedded `dbnomics-importer` sources.
* `dbnomics-dev-api`: dockerisation of `dbnomics-api`, see https://gitlab.com/oecd/dbnomics-api
* `dbnomics-dev-websire`: dockerisation of `dbnomics-website`, see https://gitlab.com/oecd/dbnomics-ui
NB:
* python3.5 is used for importer because solr image is based on Debian 9, and only 3.5 is available in apt repos
That's all folks ...
#!/bin/sh
CURRENT_DIR=$(dirname $(readlink -f $0))
CONFIG="${CURRENT_DIR}"/../config
docker-compose -f $CONFIG/docker-compose.yaml down
docker volume rm -f solr-conf
docker volume rm -f json-data
#!/bin/sh
export JSON_DATA_PATH=$1
if [ ! -d $JSON_DATA_PATH ]
then
echo "Usage: dbnomics-dev-load PATH"
echo
echo "PATH is a relative or absolute path to a provider json-data folder"
exit
fi
JSON_DATA_DIR=$(basename $1)
echo "Copy json data from ${JSON_DATA_PATH} "
docker run -d --rm --name json-data-loader -v json-data:/home alpine tail -f /dev/null
docker cp ${JSON_DATA_PATH} json-data-loader:/home
docker exec json-data-loader chmod -R +rX /home
docker rm -f json-data-loader
echo "Import data from provider ${JSON_DATA_DIR} "
docker exec dbnomics-dev-solr bash -c "/opt/importer/import_storage_dir.py --full /opt/json-data/${JSON_DATA_DIR}"
#!/bin/sh
TMP=.cache
CURRENT_DIR=$(dirname $(readlink -f $0))
CONFIG="${CURRENT_DIR}"/../config
CACHE="${CONFIG}"/.cache
IMPORTER_REPO=dbnomics-importer
IMPORTER_PATH=$PWD/$TMP/$IMPORTER_REPO
IMPORTER_PATH=$CACHE/$IMPORTER_REPO
URL=https://git.nomics.world/dbnomics/$IMPORTER_REPO.git
echo $CONFIG
if [ ! -d $IMPORTER_PATH ]
then
echo "Cloning dbnomics-importer"
......@@ -16,12 +20,21 @@ else
cd ..
fi
echo "Creating SOLR data volume"
docker volume rm -f solr_data_vol
docker run --rm -v solr_data_vol:/opt/solr/server/solr/mycores solr:7 bash -c "precreate-core dbnomics"
docker run -d --rm --name init-solr -v solr_data_vol:/root alpine tail -f /dev/null
echo "Update images"
docker pull redpelicans/dbnomics-api
docker pull redpelicans/dbnomics-website
echo "Create Docker volumes"
docker volume rm -f solr-conf
docker volume rm -f json-data
docker volume create --name=solr-conf
docker volume create --name=json-data
echo "Precreate solr code"
docker run --rm -v solr-conf:/opt/solr/server/solr/mycores solr:7 bash -c "precreate-core dbnomics"
docker run -d --rm --name init-solr -v solr-conf:/root alpine tail -f /dev/null
docker exec -it init-solr rm /root/dbnomics/conf/managed-schema
docker cp $IMPORTER_PATH/solr_core_config/schema.xml init-solr:/root/dbnomics/conf
docker cp $IMPORTER_PATH/solr_core_config/solrconfig.xml init-solr:/root/dbnomics/conf
docker rm -f init-solr
docker-compose up
echo "Run docker composition setup"
docker-compose -f $CONFIG/docker-compose.yaml build
docker-compose -f $CONFIG/docker-compose.yaml up
#!/bin/bash
# mydir=$(dirname "$0")
mydir=$(dirname $(readlink -f $0))
echo $mydir
cat ${mydir}/../config/docker-compose.yaml
FROM python:3.8
RUN mkdir -p /opt
WORKDIR /opt
COPY .cache/dbnomics-importer .
RUN pip3 install --requirement requirements.txt
CMD ./import_storage_dir.py --full /opt/data/$PROVIDER-json-data
FROM solr:7
USER root
RUN apt-get update && \
apt-get install -yq python3 python3-pip && \
apt-get clean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*
RUN mkdir -p /opt/importer
RUN mkdir -p /opt/json-data
RUN chown -R solr:solr /opt/json-data
RUN pip3 install virtualenv
WORKDIR /opt/importer
COPY .cache/dbnomics-importer .
RUN pip3 install --requirement /opt/importer/requirements.txt
USER solr
WORKDIR /opt/solr
ENV SOLR_POST="/opt/solr/bin/post"
version: '3.7'
services:
solr:
container_name: dbnomics-dev-solr
build:
context: .
dockerfile: Dockerfile.solr
volumes:
- solr-conf:/opt/solr/server/solr/mycores
- json-data:/opt/json-data:rw
ports:
- "0.0.0.0:18983:8983"
website:
container_name: dbnomics-dev-website
image: redpelicans/dbnomics-website
ports:
- "0.0.0.0:13000:3000"
api:
container_name: dbnomics-dev-api
image: redpelicans/dbnomics-api
ports:
- "0.0.0.0:15000:5000"
volumes:
- json-data:/json-data:rw
volumes:
solr-conf:
external: true
json-data:
external: true
version: '3.7'
services:
solr:
image: solr:7
volumes:
- solr_data_vol:/opt/solr/server/solr/mycores
ports:
- "0.0.0.0:18983:8983"
volumes:
solr_data_vol:
external: true