Skip to content
Snippets Groups Projects

Push json-data to Git

Passed Started by
Christophe Benz
@cbenz
1Running with gitlab-runner 17.8.3 (690ce25c)
2 on Run fetchers JBzuiMazz, system ID: s_33ef491069c6
3 Preparing the "docker" executor 00:10
4Using Docker executor with image git.nomics.world:4567/dbnomics/dbnomics-fetcher-pipeline/git-job:latest ...
5Authenticating with credentials from job payload (GitLab Registry)
6Pulling docker image git.nomics.world:4567/dbnomics/dbnomics-fetcher-pipeline/git-job:latest ...
7Using docker image sha256:afc3785ac7a4f3a034a20e87914f55a0bb100a12dc045e546d87be3a982d698a for git.nomics.world:4567/dbnomics/dbnomics-fetcher-pipeline/git-job:latest with digest git.nomics.world:4567/dbnomics/dbnomics-fetcher-pipeline/git-job@sha256:76b884efa708d8842c7a5310996ba87a6995d4cf2c5629b5e0f6cd69081003e1 ...
8 Preparing environment 00:04
9Running on runner-jbzuimazz-project-106-concurrent-1 via tyche...
10 Getting source from Git repository 00:03
11Skipping Git repository setup
12Skipping Git checkout
13Skipping Git submodules setup
14 Executing "step_script" stage of the job script 00:09
15Using docker image sha256:afc3785ac7a4f3a034a20e87914f55a0bb100a12dc045e546d87be3a982d698a for git.nomics.world:4567/dbnomics/dbnomics-fetcher-pipeline/git-job:latest with digest git.nomics.world:4567/dbnomics/dbnomics-fetcher-pipeline/git-job@sha256:76b884efa708d8842c7a5310996ba87a6995d4cf2c5629b5e0f6cd69081003e1 ...
16$ mkdir -p $PROVIDER_GIT_DIR
17$ git config --global push.default simple
18$ git config --global user.email "${PROVIDER_SLUG}-fetcher@localhost"
19$ git config --global user.name "${PROVIDER_SLUG} fetcher"
20$ eval $(ssh-agent -s)
21Agent pid 19
22$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
23Identity added: (stdin) (lbma-fetcher@localhost)
24$ mkdir -p ~/.ssh
25$ chmod 700 ~/.ssh
26$ ssh-keyscan -t rsa $CI_SERVER_HOST >> ~/.ssh/known_hosts
27# git.nomics.world:22 SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u5
28$ chmod 644 ~/.ssh/known_hosts
29$ PIPELINE_STATE_DIR="${PIPELINE_DIR}/state" # collapsed multi-line command
30$ if [ "$START_FROM" == "deploy" ] && [ ! -d "$WORKSPACE_JSON_DATA_DIR" ]; then # collapsed multi-line command
31$ cd "$PROVIDER_GIT_DIR"
32$ if [ $RM_GIT_LOCK == "1" ] && [ -f json-data/$GIT_LOCK_FILE ]; then # collapsed multi-line command
33$ echo "Updating json-data Git repository clone..."
34Updating json-data Git repository clone...
35$ time git-pull-or-clone -v --clone-url "$JSON_DATA_REMOTE_SSH_URL" json-data
362025-03-23 00:35:55,075 [27] INFO cli.py: 'json-data' already exists, pulling...
37Warning: Permanently added the RSA host key for IP address '51.159.101.244' to the list of known hosts.
38Auto packing the repository in background for optimum performance.
39See "git help gc" for manual housekeeping.
40 Command being timed: "git fetch"
41 User time (seconds): 0.01
42 System time (seconds): 0.03
43 Percent of CPU this job got: 2%
44 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.78
45 Average shared text size (kbytes): 0
46 Average unshared data size (kbytes): 0
47 Average stack size (kbytes): 0
48 Average total size (kbytes): 0
49 Maximum resident set size (kbytes): 6528
50 Average resident set size (kbytes): 0
51 Major (requiring I/O) page faults: 60
52 Minor (reclaiming a frame) page faults: 1938
53 Voluntary context switches: 229
54 Involuntary context switches: 96
55 Swaps: 0
56 File system inputs: 8144
57 File system outputs: 120
58 Socket messages sent: 0
59 Socket messages received: 0
60 Signals delivered: 0
61 Page size (bytes): 4096
62 Exit status: 0
63 Command being timed: "git reset --hard @{upstream} --quiet"
64 User time (seconds): 0.00
65 System time (seconds): 0.00
66 Percent of CPU this job got: 3%
67 Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.20
68 Average shared text size (kbytes): 0
69 Average unshared data size (kbytes): 0
70 Average stack size (kbytes): 0
71 Average total size (kbytes): 0
72 Maximum resident set size (kbytes): 5088
73 Average resident set size (kbytes): 0
74 Major (requiring I/O) page faults: 13
75 Minor (reclaiming a frame) page faults: 231
76 Voluntary context switches: 40
77 Involuntary context switches: 26
78 Swaps: 0
79 File system inputs: 568
80 File system outputs: 48
81 Socket messages sent: 0
82 Socket messages received: 0
83 Signals delivered: 0
84 Page size (bytes): 4096
85 Exit status: 0
86real 0m2.253s
87user 0m0.203s
88sys 0m0.084s
89$ echo "Synchronizing converted data to json-data Git repository..."
90Synchronizing converted data to json-data Git repository...
91$ time dbnomics-update-storage -v "$WORKSPACE_JSON_DATA_DIR" json-data
922025-03-23 00:35:58,081 [42] INFO dbnomics_data_model.storage.storage: Updating category tree of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) with the one of FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'merge'
932025-03-23 00:35:58,097 [42] INFO dbnomics_data_model.storage.storage: Updating releases metadata of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) with the one of FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'merge'
942025-03-23 00:35:58,097 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'platinum_M' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
952025-03-23 00:35:58,298 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.20 seconds.
962025-03-23 00:35:58,298 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
972025-03-23 00:35:58,298 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
982025-03-23 00:35:58,298 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'palladium_M' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
992025-03-23 00:35:58,344 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.04 seconds.
1002025-03-23 00:35:58,344 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1012025-03-23 00:35:58,344 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
1022025-03-23 00:35:58,344 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'palladium_D' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
1032025-03-23 00:35:58,410 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.06 seconds.
1042025-03-23 00:35:58,410 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1052025-03-23 00:35:58,410 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
1062025-03-23 00:35:58,410 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'silver_D' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
1072025-03-23 00:35:58,453 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.04 seconds.
1082025-03-23 00:35:58,453 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1092025-03-23 00:35:58,453 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
1102025-03-23 00:35:58,453 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'gold_D' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
1112025-03-23 00:35:58,557 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.10 seconds.
1122025-03-23 00:35:58,557 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1132025-03-23 00:35:58,557 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
1142025-03-23 00:35:58,558 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'gold_M' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
1152025-03-23 00:35:58,616 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.05 seconds.
1162025-03-23 00:35:58,616 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1172025-03-23 00:35:58,616 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
1182025-03-23 00:35:58,616 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'platinum_D' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
1192025-03-23 00:35:58,887 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.27 seconds.
1202025-03-23 00:35:58,887 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1212025-03-23 00:35:58,887 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
1222025-03-23 00:35:58,888 [42] INFO dbnomics_data_model.storage.storage: Updating dataset 'silver_M' of FileSystemStorage(storage_dir=PosixPath('json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) from the one in FileSystemStorage(storage_dir=PosixPath('/data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data'), get_series_offsets=None, storage_variant='detect', default_storage_variant='jsonl', warn_when_scanning_jsonl_series=False) using strategy 'replace'
1232025-03-23 00:35:58,896 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: dirsync finished in 0.00 seconds.
1242025-03-23 00:35:58,897 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync: 1 directories parsed, 0 files copied
1252025-03-23 00:35:58,897 [42] INFO dbnomics_data_model.storage.adapters.filesystem.storage.dirsync:
126$ cd json-data
127real 0m1.852s
128user 0m0.403s
129sys 0m0.082s
130$ echo "Committing data from $WORKSPACE_JSON_DATA_DIR, and pushing"
131Committing data from /data/dbnomics-prod/workspaces/lbma/pipelines/485466/json-data, and pushing
132$ time git add -A
133real 0m0.064s
134user 0m0.000s
135sys 0m0.004s
136$ time git commit -m "Convert data from provider" -m "CI_PIPELINE_ID:$CI_PIPELINE_ID" -m "CI_JOB_ID:$CI_JOB_ID" --quiet || true
137On branch master
138Your branch is up to date with 'origin/master'.
139nothing to commit, working tree clean
140real 0m0.038s
141user 0m0.003s
142sys 0m0.003s
143$ time git push
144Everything up-to-date
145real 0m0.398s
146user 0m0.012s
147sys 0m0.009s
148$ PIPELINE_STATE_FILE="${PIPELINE_STATE_DIR}/state.json" # collapsed multi-line command
149$ # bad strategy, to be removed # collapsed multi-line command
150$ # used by IMF, ISTAT, to be removed # collapsed multi-line command
151 Cleaning up project directory and file based variables 00:03
152Job succeeded