run-fetcher-on-server.md

+# Run a fetcher on server
+
+Note: to be allowed to do this, your SSH public key must be added to the authorized keys of the `dev` user on the server.
+
+Sometimes running a fetcher on a local development machine may consume too much resources (CPU, RAM, disk).
+In this case it may be convenient to run the fetcher on a server.
+
+The `dev` unix user of `scylla.nomics.world` has been created to solve this problem.
+
+[Pyenv](https://github.com/pyenv/pyenv) is used to allow using any Python version, not especially the one shipped with the OS.
+
+## Usage
+
+On your local development machine:
+
+```bash
+ssh -A dev@scylla.nomics.world
+```
+
+Note: the `-A` option forwards the SSH identity, to allow pushing to Git repositories for example.
+
+You will immediately see a terminal session managed by [byobu](https://www.byobu.org/).
+
+To work on a fetcher (replace `foo` by the actual value):
+
+```bash
+cd fetchers
+git clone git@git.nomics.world:dbnomics-fetchers/foo-fetcher.git
+cd foo-fetcher
+
+# To install the Python version specified in the file `.python-version` (if it exists)
+pyenv install
+
+# To create a virtualenv with the Python version specified in the file `.python-version`
+# If it already exists, an error will be displayed, to be ignored
+pyenv virtualenv foo-fetcher
+
+# To activate the virtualenv
+pyenv activate foo-fetcher
+
+# To install Python dependencies (if `requirements.txt` exists)
+pip install -r requirements.txt
+
+# Run download, convert, validate...
+python download.py ...
+python convert.py ...
+dbnomics-validate-storage json-data
+```
+
+Note: `dbnomics-validate-storage` was installed with `pipx` using the system Python version.
+
+## Mount fetcher data
+
+You can access the fetcher directory using SFTP by using the URL `sftp://dev@scylla.nomics.world/home/dev/fetchers/foo-fetcher`.
+
+You can use that URL with the `sftp` command, or a file manager, for example.
+
+You can also use `sshfs` to mount that remote directory.