pipeline fails: git remote disappears

Description

At some point the cloned Git repository of {fetcher}-source-data, in the working directory of the fetcher, looses its remote "origin".

This can be seen in .git/refs:

root@debug-fetcher-data:/fetcher-data/enedis/enedis-json-data# ls -lR .git/refs
.git/refs:
total 8
drwxr-sr-x 2 root root 4096 Nov  9 01:08 heads
drwxr-sr-x 2 root root 4096 Nov  2 15:20 tags

.git/refs/heads:
total 4
-rw-r--r-- 1 root root 41 Nov  2 15:20 master

.git/refs/tags:
total 0

Normally it should be like:

root@debug-fetcher-data:/fetcher-data/oppins/oppins-json-data# ls -lR .git/refs
.git/refs:
total 12
drwxr-sr-x 2 root root 4096 Nov  7 08:16 heads
drwxr-sr-x 3 root root 4096 Oct  1 15:06 remotes
drwxr-sr-x 2 root root 4096 Sep 30 09:34 tags

.git/refs/heads:
total 4
-rw-r--r-- 1 root root 41 Nov  7 08:16 master

.git/refs/remotes:
total 4
drwxr-sr-x 2 root root 4096 Nov  7 08:17 origin

.git/refs/remotes/origin:
total 4
-rw-r--r-- 1 root root 41 Nov  7 08:17 master

.git/refs/tags:
total 0

I currently don't understand what makes the remotes directory disappear. When cloning the repository it is present.

Quick fix

This is not a real solution but it is possible to overcome this issue by using the WIPE_SOURCE_DATA=1 or WIPE_JSON_DATA=1 depending on the case.

Error example

https://git.nomics.world/dbnomics-fetchers/google-fetcher/-/jobs/293701

fatal: ambiguous argument 'origin/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 256, in <module>
    sys.exit(main())
  File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 117, in main
    pull_repo(args.target_dir, checkout=args.checkout, ref=args.ref)
  File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 228, in pull_repo
    run(["git", "reset", "--hard", f"{ORIGIN}/{ref}", "--quiet"], cwd=target_dir)
  File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 249, in run
    process = subprocess.run(command, cwd=cwd, check=check, encoding=encoding, **kwargs)
  File "/usr/local/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'reset', '--hard', 'origin/master', '--quiet']' returned non-zero exit status 128.
Edited Nov 09, 2020 by Christophe Benz
Assignee Loading
Time tracking Loading