How to clone all projects of a group at once in GitLab?
One liner with curl
, jq
, tr
:
for repo in $(curl -s --header "PRIVATE-TOKEN: your_private_token" https://<your-host>/api/v4/groups/<group_id> | jq -r ".projects[].ssh_url_to_repo"); do git clone $repo; done;
For Gitlab.com use https://gitlab.com/api/v4/groups/<group_id>
To include subgroups add include_subgroups=true
query param like
https://<your-host>/api/v4/groups/<group_id>?include_subgroups=true
Note: To clone with http url use
http_url_to_repo
instead ofssh_url_to_repo
in jq (Thanks @MattVon for the comment)
Not really, unless:
you have a 21st project which references the other 20 as submodules.
(in which case a clone followed by agit submodule update --init
would be enough to get all 20 projects cloned and checked out)or you somehow list the projects you have access (GitLab API for projects), and loop on that result to clone each one (meaning that can be scripted, and then executed as "one" command)
Since 2015, Jay Gabez mentions in the comments (August 2019) the tool gabrie30/ghorg
ghorg
allows you to quickly clone all of an org's or user's repos into a single directory.
Usage:
$ ghorg clone someorg
$ ghorg clone someuser --clone-type=user --protocol=ssh --branch=develop
$ ghorg clone gitlab-org --scm=gitlab --namespace=gitlab-org/security-products
$ ghorg clone --help
Also (2020): https://github.com/ezbz/gitlabber
usage: gitlabber [-h] [-t token] [-u url] [--debug] [-p]
[--print-format {json,yaml,tree}] [-i csv] [-x csv]
[--version]
[dest]
Gitlabber - clones or pulls entire groups/projects tree from gitlab
Here's an example in Python 3:
from urllib.request import urlopen
import json
import subprocess, shlex
allProjects = urlopen("https://[yourServer:port]/api/v4/projects?private_token=[yourPrivateTokenFromUserProfile]&per_page=100000")
allProjectsDict = json.loads(allProjects.read().decode())
for thisProject in allProjectsDict:
try:
thisProjectURL = thisProject['ssh_url_to_repo']
command = shlex.split('git clone %s' % thisProjectURL)
resultCode = subprocess.Popen(command)
except Exception as e:
print("Error on %s: %s" % (thisProjectURL, e.strerror))
There is a tool called myrepos, which manages multiple version controls repositories. Updating all repositories simply requires one command:
mr update
In order to register all gitlab projects to mr, here is a small python script. It requires the package python-gitlab installed:
import os
from subprocess import call
from gitlab import Gitlab
# Register a connection to a gitlab instance, using its URL and a user private token
gl = Gitlab('http://192.168.123.107', 'JVNSESs8EwWRx5yDxM5q')
groupsToSkip = ['aGroupYouDontWantToBeAdded']
gl.auth() # Connect to get the current user
gitBasePathRelative = "git/"
gitBasePathRelativeAbsolut = os.path.expanduser("~/" + gitBasePathRelative)
os.makedirs(gitBasePathRelativeAbsolut,exist_ok=True)
for p in gl.Project():
if not any(p.namespace.path in s for s in groupsToSkip):
pathToFolder = gitBasePathRelative + p.namespace.name + "/" + p.name
commandArray = ["mr", "config", pathToFolder, "checkout=git clone '" + p.ssh_url_to_repo + "' '" + p.name + "'"]
call(commandArray)
os.chdir(gitBasePathRelativeAbsolut)
call(["mr", "update"])