pipeline fails: git remote disappears
Description
At some point the cloned Git repository of {fetcher}-source-data
, in the working directory of the fetcher, looses its remote "origin".
This can be seen in .git/refs
:
root@debug-fetcher-data:/fetcher-data/enedis/enedis-json-data# ls -lR .git/refs
.git/refs:
total 8
drwxr-sr-x 2 root root 4096 Nov 9 01:08 heads
drwxr-sr-x 2 root root 4096 Nov 2 15:20 tags
.git/refs/heads:
total 4
-rw-r--r-- 1 root root 41 Nov 2 15:20 master
.git/refs/tags:
total 0
Normally it should be like:
root@debug-fetcher-data:/fetcher-data/oppins/oppins-json-data# ls -lR .git/refs
.git/refs:
total 12
drwxr-sr-x 2 root root 4096 Nov 7 08:16 heads
drwxr-sr-x 3 root root 4096 Oct 1 15:06 remotes
drwxr-sr-x 2 root root 4096 Sep 30 09:34 tags
.git/refs/heads:
total 4
-rw-r--r-- 1 root root 41 Nov 7 08:16 master
.git/refs/remotes:
total 4
drwxr-sr-x 2 root root 4096 Nov 7 08:17 origin
.git/refs/remotes/origin:
total 4
-rw-r--r-- 1 root root 41 Nov 7 08:17 master
.git/refs/tags:
total 0
I currently don't understand what makes the remotes
directory disappear. When cloning the repository it is present.
Quick fix
This is not a real solution but it is possible to overcome this issue by using the WIPE_SOURCE_DATA=1
or WIPE_JSON_DATA=1
depending on the case.
Error example
https://git.nomics.world/dbnomics-fetchers/google-fetcher/-/jobs/293701
fatal: ambiguous argument 'origin/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 256, in <module>
sys.exit(main())
File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 117, in main
pull_repo(args.target_dir, checkout=args.checkout, ref=args.ref)
File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 228, in pull_repo
run(["git", "reset", "--hard", f"{ORIGIN}/{ref}", "--quiet"], cwd=target_dir)
File "/builds/dbnomics-fetchers/google-fetcher/git-pull-or-clone.py", line 249, in run
process = subprocess.run(command, cwd=cwd, check=check, encoding=encoding, **kwargs)
File "/usr/local/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'reset', '--hard', 'origin/master', '--quiet']' returned non-zero exit status 128.
Edited by Christophe Benz