How to setup vcpkg with artifactory using a generic repository.

[Neumann-A]

Step 1: Setup a repository in artifactory

a) Go to the administration tab

b) Create a new repository

c) Select generic repository and set it up as you like

Step 2: Setup vcpkg for artifactory

<baseurl> = URL of the repository
<token> = API token for repository access

Setup for binary caching:
set VCPKG_BINARY_SOURCES=clear;http,<baseurl>/{name}/{version}/{triplet}/{sha},readwrite,Authorization: Bearer <token>

Setup for asset caching:
set X_VCPKG_ASSET_SOURCES=clear;x-script,pwsh <full_path_to_https_script>/https_assert.ps1 {url} {sha512} {dst};x-block-origin

https_assert.ps1 script:

param (
  [string]$Url,
  [string]$ExpectedSHA512,
  [string]$Destination
)

$baseurl = "<baseurl>/assets"
$token = "<token>" # read some env variable here instead of storing the token in the script!
$headers = @{ Authorization = "Bearer $token" }
$contentype = "multipart/form-data"
$filename = Split-Path -Path $Destination -Leaf
$correctFilename = $filename -replace '\.\d+\.part$', ''
$cacheurl = "$baseurl/$correctFilename"

$downloadsfolder = Split-Path -Path "$Destination" -Parent
if(-not (Test-Path $downloadsfolder)) {
  $out = New-Item -Path "$downloadsfolder" -ItemType Directory
}


function Verify-SHA512 {
    param (
        [string]$File,
        [string]$ExpectedHash
    )

    $hasher = [System.Security.Cryptography.SHA512]::Create()
    $stream = [System.IO.File]::OpenRead($File)
    $hashBytes = $hasher.ComputeHash($stream)
    $stream.Close()

    $fileHash = [BitConverter]::ToString($hashBytes) -replace '-', ''

    $hashesMatch = ($fileHash -eq $expectedHash)
    if(-not $hashesMatch) {
      Write-Error "SHA-512 hash verification failed for $File"
      Write-Error "Expected:   $expectedHash"
      Write-Error "Calculated: $fileHash"
    }

    return $hashesMatch
}

try {
  $response = Invoke-WebRequest -Uri $cacheurl -Headers $headers -OutFile $Destination -ContentType $contentype
} catch {
  if ($_.Exception.Response) {
      $statusCode = $_.Exception.Response.StatusCode
      if ($statusCode -eq "404") {
        Write-Warning "'$correctFilename' not available in cache at '$baseurl'"
        if (Test-Path $Destination) {
          $out = Remove-Item -Path $Destination
        }
      } else {
        Write-Error "Download failed with status code: $statusCode"
        exit 1
      }
  } else {
      Write-Output "Download failed with error: $_"
      exit 1
  }
}

# vcpkg will do SHA512 validation after download from cacheurl
if(-not (Test-Path $Destination)) {
  Write-Warning "Falling back to download from $Url"
  try {
    $response = Invoke-WebRequest -Uri $Url -OutFile $Destination
  } catch {
    if ($_.Exception.Response) {
        $statusCode = $_.Exception.Response.StatusCode
        if ($statusCode -eq "404") {
          Write-Error "Unable to download from '$Url'"
          exit 1
        } else {
          Write-Error "Download failed with status code: $statusCode"
          exit 1
        }
    } else {
        Write-Error "Download failed with error: $_"
        exit 1
  }
}
  # Need to verify here to upload the correct file to the cache
  if (-not (Verify-SHA512 -File $Destination -ExpectedHash $ExpectedSHA512)) {
      exit 1
  }
  Write-Output "Uploading to '$cacheurl'"
  $response = Invoke-WebRequest -Uri $cacheurl -Method Put -Headers $headers -InFile $Destination -ContentType $contentype
}

Resulting layout:

Conclusion:

X_VCPKG_ASSET_SOURCES=clear;x-script,pwsh <full_path_to_https_script>/https_assert.ps1 {url} {sha512} {dst};x-block-origin should have a x-script-upload and x-script-download option instead so that the script does not have to reimplement the SHA512 check. It would also be nice to store the assets in a more order way by also supplying {name|port} and {version} as url templates. The final storage name {filename} also should be passed to the script somehow instead of using implementation knowledge to recalculate it in the script. The download within vcpkg probably should be equal to working directory of parent call instead of switching between different downloads.

[retifrav]

This is fantastic, thanks a lot for describing the process and attaching the script. It just worked™ almost on the first try.

I said "almost" because I was getting downloads are blocked by x-block-origin error for assets (PowerShell, 7z and so on), and the error text turned out to be misleading, as the actual reason was that in my environment (Windows, believe it or not) there was no pwsh available in the PATH. So I just replaced it with PowerShell (could've made a symlink instead), and everything went fine after that.

Quite naturally, the script will only work on platforms which PowerShell interpreter is available for, and for instance ARM-based Ubuntu is apparently not one of those (I failed to install it there). But then of course it is rather trivial to re-implement this logic in BASH or some other language, so it is still very useful to have this example. Ah, just downloaded an ARM variant for Linux directly, and it worked out fine.

[Neumann-A]

@JavierMatosD: Maybe this needs a better error to re-order the errors or not show the first error at all.

[JavierMatosD]

I see that the setup instructions for asset caching here is passing x-block-origin. So, it's just saying that you are missing an asset and it can't download it. What kind of error would you like to see instead? I could also reorder the messages so that the script failure pop up before this message

[Neumann-A]

@ JavierMatosD. So the error is:

A suitable version of 7zip was not found (required v24.8.0).
error: Missing 7z2408-extra.7z and downloads are blocked by x-block-origin.
error: Launching <mirror-script>: error: calling CreateProcessW failed with 2 (The system cannot find the file specified.)

shouldn't it be:

A suitable version of 7zip was not found (required v24.8.0).
error: Launching <mirror-script>: error: calling CreateProcessW failed with 2 (The system cannot find the file specified.)
error: Missing 7z2408-extra.7z and downloads are blocked by x-block-origin.

and i am currently wondering if you also fell into the trap reading only the first error line.

[JavierMatosD]

Ok, this should print the messages before the blurb about x-block-origin microsoft/vcpkg-tool#1541

[retifrav]

In case anyone wanted to try asset caching with Gitea packages (the Generic variant) instead of Artifactory, the script works almost as is, only need to:

• replace Bearer with token for the authorization header;
• replace multipart/form-data with application/octet-stream.

The binary caching works too, but of course also requires replacing Bearer with token (and adjusting the URL/route pattern to match the expected "layout").

I wrote about this with some more details here.