Efficiently Git Clone All Repositories from Azure DevOps using PowerShell: A Step-by-Step Guide

As a consultant, starting a new project with a client can be a daunting task. One way to make the transition smoother is by cloning all the repositories on your first day. This allows you to have quick access to all the necessary files and resources, enabling you to perform your job efficiently and effectively. In this blog post, we will explore the benefits of cloning repositories, a script for doing so, and some common pitfalls to avoid.

Skip to the code sample

Organizing your Git repos

When working for multiple clients or even just having private projects next to your client projects it can come in handy to organize your git repositories. For some Frontend repositories, the path with node_modules was too long and that forced me to place my folders on the Disk level. A path for a project for me would look like C:\Git\{ClientName}\{RepositoryName}.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

C:\Git
 ┣ Client1
 ┃ ┣ Client1.Repository1
 ┃ ┣ Client1.Repository2
 ┃ ┗ Client1.Repository3
 ┣ Client2
 ┃ ┣ Client2.Repository1
 ┃ ┗ Client2.Repository2
 ┗ private
 ┃ ┣ Blog
 ┃ ┗ Demo

Using workspaces in Git Fork

Fork is a tool that will help you focus on the right workload. Using the structure as discussed with Fork, you can focus on the right repositories. Cloned new repositories but not seen by Fork? Reload the whole folder using right-click and ‘Rescan repositories’. Get Git Fork from git-fork.com.

Use Fork Workspaces to focus on the current environment. It will also help you work on private projects outside of work hours on the same workstation. You can also create workspaces for different domains or teams if you are for example the lead or architect in a project.

Configure your git username

Depending on the network infra, you will need to configure your commit username to the email of your client. Some instances block all git pushes from committers with a different domain.

1

git config [--global] user.email "username@corperate.com"

In the script to clone all repositories, you can also enable the script to set the committer email for every repository.

Clone all repositories

To clone all repositories in Azure DevOps we can use the REST API to find all existing repositories. The code example consists of a Powershell script and a configuration file with settings and Authorization.

Configuration

Make sure to create a file named: CloneAllRepos.config with the contents written below. Make sure every parameter is configured as your workspace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

[General]
Url=https://dev.azure.com/MART/project

[LocalGitConfig]
GitPath=C:\Git\
OrgName=MART

[GitOptions]
PruneLocalBranches=false # Optional defaults to false
GitEmail=username@corperate.com

CloneAllRepos.ps1

When I first encountered the idea to clone all repos idea it was on a corporate wiki. After some backtracing, I found the source: Script to clone all Git repositories from your Azure DevOps collection.

The PowerShell script below does a git pull for existing repositories and performs a git clone on untracked repositories.

I edited the script to fit my needs with some extra parameters.

1. It puts the repos in the given directory in settings.
2. It prunes local branches when PruneLocalBranches is set to true.
3. It sets the git username email to the configured GitUsername under GitOptions, it’s ignored when empty.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

# Read configuration file
Get-Content "CloneAllRepos.config" | foreach-object -begin {$h=@{}} -process { 
    $k = [regex]::split($_,'='); 
    if(($k[0].CompareTo("") -ne 0) -and ($k[0].StartsWith("[") -ne $True)) { 
        $h.Add($k[0], $k[1]) 
    } 
}
#AzDO config
$url = $h.Get_Item("Url")
# LocalGitConfig
$gitPath = $h.Get_Item("GitPath")
$orgName = $h.Get_Item("OrgName")
$pruneLocalBranches = $h.Get_Item("PruneLocalBranches") -eq "true"
$gitEmail = $h.Get_Item("GitEmail")

# Get the access token from current az login session
# see https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/service-principal-managed-identity?toc=%2Fazure%2Fdevops%2Forganizations%2Fsecurity%2Ftoc.json&view=azure-devops#q-can-i-use-a-service-principal-or-managed-identity-with-azure-cli
$accessToken = az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query "accessToken" --output tsv
if ($null -eq $accessToken) {
    exit 1
}

$headers = @{
    "Authorization" = ("Bearer {0}" -f $accessToken)
    "Accept" = "application/json"
}

# Retrieve list of all repositories
$resp = Invoke-WebRequest -Headers $headers -Uri ("{0}/_apis/git/repositories?api-version=1.0" -f $url)
$json = convertFrom-JSON $resp.Content

# Clone or pull all repositories
$initpath =  ("{0}{1}" -f  $gitPath,$orgName)

foreach ($entry in $json.value) { 
    set-location $initpath
    $name = $entry.name 
    Write-Host $name -ForegroundColor Green

    if($entry.isDisabled){
        Write-Host "Skipping disabled repo: '$name'" -ForegroundColor Yellow
        continue;
    }

    $url = $entry.remoteUrl #-replace "://", ("://{0}@" -f $gitcred)
    if(!(Test-Path -Path $name)) {
        git clone $url
    } else {
        Write-Host "Directory '$name' exists lets pull"
        set-location $name
        git pull
        $defaultBranch = git symbolic-ref --short HEAD

        if($pruneLocalBranches){
            Write-Host "Pruning local branches $name" -ForegroundColor Yellow
            $branches = git branch -vv | Where-Object { $_ -notmatch "::" } | ForEach-Object { ($_ -split '\s+')[1] }

            foreach ($branch in $branches) {
                if ($branch -eq $defaultBranch) {
                    Write-Host "Skipping default branch '$branch'."
                    continue
                }
                if ((git branch -vv | Where-Object { $_ -match "$branch\s+\[origin\/" })) {
                    Write-Host "Skipping branch '$branch' as it has a remote reference."
                }
                else {
                    git branch -D $branch
                    Write-Host "Deleted local branch '$branch'." -ForegroundColor Green
                }
            }
        }
        if($gitEmail){
            git config user.email "$gitEmail"
        }
    }
}

Run it

Run the script it using a PowerShell prompt for example using for example Windows Terminal.

1

./CloneAllRepos.ps1

Using scripting for common tasks

In the world of microservices, we choose to duplicate some of the plumbing. When you want to change multiple repos knowledge on scripting can be helpful. In this series, I explored how to automate git tasks with PowerShell.

Some examples are:

• Updating multiple NuGet packages.
• Enforcing certain Nuget. config configurations.
• Renaming business terminology on multiple branches.

Automating

With this structure, you could automate actions over multiple repositories. In the code below I wrote an example of automating script for changing the Nuget.config file in every repository. If your packages have the same layout changes can be done easier and faster. Also, please check out my article using binary replace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

git checkout main
git pull
git checkout -b fix/nugetconfig

# DO THE NECESSARY CHANGE in nuget.config.

git mv -f NuGet.config nuget.config
git add *

git commit -m "Only use private Nuget upstream"
git push --set-upstream origin fix/nugetconfig
git checkout main

Conclusion and discussion

Make your workflow faster with scripting and your knowledge of the Git CLI. When you have to do repetitive tasks such as updating a single package on multiple (microservice-like) repositories, try to automate it. It may for the first occurrence not be profitable, but after three times, you will be faster than doing it manually. It can also help you clean up your workspace and be more tidy.