Over the next month and going forward, requests for GitLab CI artifacts downloads may be redirected
to Google Cloud CDN instead of
Google Cloud Storage. We anticipate that GitLab CI users may benefit from faster
downloads from edge caches closest to your location.
Disclaimer: This blog contains information related to upcoming products, features, and functionality. It is important to note that the information in this blog post is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. As with all projects, the items mentioned in this blog and linked pages are subject to change or delay. The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab.
How will this work?
Currently when a CI runner or other client downloads a CI artifact,
GitLab.com responds with a 302 redirect to a time-limited, pre-signed URL with a domain of storage.googleapis.com.
After this change, the domain will change to cdn.artifacts.gitlab-static.net.
The exception is for requests originating from within the Google Cloud
Platform. These will continue to be redirected to Cloud Storage.
When will this change occur?
We expect to start the transition around the end of October 2022. This will be a
gradual transition using a percentage-based rollout, so we anticipate that you will see
an increasing number of your requests redirected to Google Cloud
CDN instead of Google Cloud Storage until all of the requests are served by the
former.
You can follow along with the progress of this initiative and raise any
questions in this issue. We
will post more detailed timelines in that issue as we refine the rollout
plan.
How does this change impact you?
Since GitLab CI runners and certain clients automatically handle URL
redirections already, we expect that downloads for CI artifacts should
continue to work without any action.
We encourage upgrading to the latest version of the GitLab Runner in
order to take advantage of the CDN. This feature was introduced in
GitLab Runner v13.1.0.
If a runner cannot download from the CDN host, it will retry without the
CDN and download the artifact directly through GitLab.com.
However, if you have a firewall that only allows
storage.googleapis.com, you will need to add
cdn.artifacts.gitlab-static.net (34.110.204.38) to the allow list.
What do these warning messages mean?
With this change, users may see warning messages in the CI job logs:
read: connection reset by peer
ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/<job id>/artifacts?direct_download=true: Get "https://cdn.artifacts.gitlab-static.net/...
read tcp 172.17.0.2:59332->34.110.204.38:443: read: connection reset by peer id=1234 token=<some token>
WARNING: Retrying... error=invalid argument
Downloading artifacts from coordinator... ok id=1234 responseStatus=200 OK token=<some token>
This error suggests the runner was not able to access the CDN. Check
your network firewalls and allow access to the IP 34.110.204.38.
Note that there are two Downloading artifacts from coordinator
messages. The second attempt succeeded because the runner retried
without the CDN.
x509: certificate signed by unknown authority
ERROR: Downloading artifacts from coordinator... error couldn't execute GET against https://gitlab.com/api/v4/jobs/<job id>/artifacts?direct_download=true: Get "https://storage.googleapis.com/gitlab-gprd-artifacts/...: x509: certificate signed by unknown authority id=1234 token=<some token>
If you see this error with a Windows runner, upgrade to v15.5.0 since it
is compiled with Go 1.18, which
supports using the system certificate pool.
Otherwise, this error suggests the runner is configured with custom SSL certificates.
You may need to update your certificates or include the certificates directly in the bundle.
Authentication required
Some clients may report a 401 error with Authentication required after
requesting to download a job artifact:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>
This error message suggests the HTTP client is following the 302
redirect and sending the Authorization header with the redirected
URL. This is a known issue with Java HTTP clients.
Update your client to drop the Authorization header the
redirect. Google Cloud Storage ignores this header if it were set, but
Cloud CDN rejects requests that have the Authorization header set.
