The Git project began nearly 15 years ago, on April 7,
2005, and is now the
version control system of choice for developers. Yet, there are certain types of projects that
often do not use Git, particularly projects that have many large binary files,
such as video games. One reason projects with large binary files don't use Git
is because, when a Git repository is cloned, Git will download every version of
every file in the repo. For most use cases, downloading this history is a
useful feature, but it slows cloning and fetching for projects with large binary
files, assuming the project even fits on your computer.
What is Partial Clone?
Partial Clone is a new feature of Git that replaces Git
LFS and makes working with very large repositories
better by teaching Git how to work without downloading every file. Partial Clone
has been
years
in the making, with code contributions from GitLab, GitHub, Microsoft and
Google. Today it is experimentally available in Git and GitLab, and can be
enabled by administrators
(docs).
Partial Clone speeds up fetching and cloning because less data is
transferred, and reduces disk usage on your local computer. For example, cloning
gitlab-com/www-gitlab-com
using Partial Clone (--filter=blob:none) is at least 50% faster, and transfers
70% less data.
Note: Partial Clone is one specific performance optimization for very large
repositories. Sparse
Checkout
is a related optimization that is particularly focused on repositories with
tremendously large numbers of files and revisions such as
Windows
code base.
A brief history of large files
"What about Git LFS?" you may ask. Doesn't LFS stand for "large file storage"?
Previously, extra tools were required to store large files in Git. In 2010,
git-annex was released, and five years
later in 2015, Git LFS was released. Both
git-annex and Git LFS added large file support to Git in a similar way: Instead
of storing a large file in Git, store a pointer file that links to the large
file. Then, when someone needs a large file, they can download it on-demand
using the pointer.
The criticism of this approach is that there are now two places to store files,
in Git or in Git LFS. Which means that everyone must remember that big files need
to go in Git LFS to keep the Git repo small and fast. There are downsides to
this approach. Besides being susceptible to human error, the pointer encodes
decisions based on bandwidth and file type into the structure of the repository
that influence all the people using the repository. Our assumptions about
bandwidth and storage are likely to change over time, and vary by the location,
but decisions encoded in the repository are not flexible. Administrators and
developers alike benefit from flexibility in where to store large files, and
which files to download.
Partial Clone solves these problems by removing the need for two classes of
storage, and special pointers. Let's walk through an example to understand how.
How to get started with Partial Clone
Let's continue to use gitlab-com/www-gitlab-com as an example project, since
it has quite a lot of images. For a larger repository, like a video game with
detailed textures and models that could take up a lot of disk space, the benefits will be even more significant.
Instead of a vanilla git clone, we will include a filter spec which controls
what is excluded when fetching data. In this situation, we just want to exclude
large binary files. I've included --no-checkout so we can more clearly observe
what is happening.
git clone --filter=blob:none --no-checkout [email protected]/gitlab-com/www-gitlab-com.git
# Cloning into 'www-gitlab-com'...
# remote: Enumerating objects: 624541, done.
# remote: Counting objects: 100% (624541/624541), done.
# remote: Compressing objects: 100% (151886/151886), done.
# remote: Total 624541 (delta 432983), reused 622339 (delta 430843), pack-reused 0
# Receiving objects: 100% (624541/624541), 74.61 MiB | 8.14 MiB/s, done.
# Resolving deltas: 100% (432983/432983), done.
Above we explicitly told Git not to checkout the default branch. Normally
checkout doesn't require fetching any data from the server, because we have
everything locally. When using Partial Clone, since we are deliberately not downloading everything, Git will need to fetch any missing files when doing a
checkout.
git checkout master
# remote: Enumerating objects: 12080, done.
# remote: Counting objects: 100% (12080/12080), done.
# remote: Compressing objects: 100% (11640/11640), done.
# remote: Total 12080 (delta 442), reused 9773 (delta 409), pack-reused 0
# Receiving objects: 100% (12080/12080), 1.10 GiB | 8.49 MiB/s, done.
# Resolving deltas: 100% (442/442), done.
# Updating files: 100% (12342/12342), done.
# Filtering content: 100% (3/3), 131.24 MiB | 4.73 MiB/s, done.
If we checkout a different branch or commit, we'll need to download more missing
files.
git checkout 92d1f39b60f957d0bc3c5621bb3e17a3984bdf72
# remote: Enumerating objects: 1968, done.
# remote: Counting objects: 100% (1968/1968), done.
# remote: Compressing objects: 100% (1953/1953), done.
# remote: Total 1968 (delta 23), reused 1623 (delta 15), pack-reused 0
# Receiving objects: 100% (1968/1968), 327.44 MiB | 8.83 MiB/s, done.
# Resolving deltas: 100% (23/23), done.
# Updating files: 100% (2255/2255), done.
# Note: switching to '92d1f39b60f957d0bc3c5621bb3e17a3984bdf72'.
Git remembers the filter spec we provided when cloning the repository so that
fetching updates will also exclude large files until we need them.
git config remote.origin.promisor
# true
git config remote.origin.partialclonefilter
# blob:none
When committing changes, you simply commit binary files like you would any other
file. There is no extra tool to install or configure, no need to treat big files
differently to small files.
Network and Storage
If you are already using Git LFS today, you might
be aware that large files are stored and transferred differently to regular Git
objects. On GitLab.com, Git LFS objects are stored in object storage (like AWS
S3) rather than fast attached storage (like SSD), and transferred over HTTP even
when using SSH for regular Git objects. Using object storage has the advantage
of reducing storage costs for large binary files, while using simpler HTTP
requests for large downloads allows the possibility of resumable and parallel
downloads.
Partial Clone
already
supports more than one remote, and work is underway to allow large files to be
stored in a different location such as object storage. Unlike Git LFS, however,
the repository or instance administrator will be able to choose which objects
should be stored where, and change this configuration over time if needed.
Follow the epic for improved large file
storage to learn more and
follow our progress.
Performance
When fetching new objects from the Git server using a filter
spec
to exclude objects from the response, Git will check each object and exclude
any that match the filter spec. In Git
2.25,
the most recent version, filtering has not been optimized for performance.
Jeff King (Peff) (GitHub) recently
contributed
performance improvements for blob size filtering, which will likely be included
in Git 2.26, and our plan is
to include it in GitLab 12.10 release.
Optimizing the sparse filter spec option (--filter:sparse), which filters
based on file path is more complex because blobs, which contain the file
content, do not include file path information. The directory structure of a
repository is stored in tree objects.
Follow the epic for Partial Clone performance
improvements to learn more
and follow our progress.
Usability
One of the drawbacks of Git LFS was that it required installing an additional
tool. In comparison, Partial Clone does not require any additional tools.
However, it does require learning new options and configurations, such as to
clone using the --filter option.
We want to make it easy for people get their work done, who simply desire Git to
just work. They shouldn't need to work out which is the optimal blob size filter
spec for a project? Or what even is a filter spec? While Partial Clone remains
experimental, we haven't made any changes to the GitLab interface to highlight
Partial Clone, but we are investigating this and welcome your feedback. Please
join the conversation on this
issue.
File locking and tool integrations
Any conversation of large binary files, particularly in regards to video
games is incomplete without discussing file locking and tooling integrations.
Unlike plain text source code, resolving conflicts between different versions of
a binary file is often impossible. To prevent conflicts in binary file editing,
an exclusive file lock is used, meaning only one person at a time can edit a
single file, regardless of branches. If conflicts can't be resolved, allowing multiple
versions of a individual file to be created in parallel on different branches is a bug, not
a feature. GitLab already has basic file locking support, but it is really only
useful for plain text because it only applies to the default branch, and is not
integrated with any local tools.
Local tooling integrations are important for binary asset workflows, to
automatically propagate file locks to the local development environment, and to
allow artists to work on assets without needing to use Git from the command
line. Propagating file locks quickly to local development environments is also
important because it prevents work from being wasted before it even happens.
Follow the file locking and
integrations epics for more
information about what we're working on.
Conclusion
Large files are necessary for many projects, and Git will soon support this
natively, without the need for extra tools. Although Partial Clone is still an
experimental feature, we are making improvements with every release and the
feature is now ready for testing.
Thank you to the Git community for your work over the past years on improving
support for enormous repositories. Particularly, thank you to Jeff
King (GitHub) and Christian
Couder (senior backend
engineer on Gitaly at GitLab) for your early experimentation with Partial Clone,
Jonathan Tan (Google) and Jeff Hostetler
(Microsoft) for contributing the first
implementation
of Partial Clone and promisor remotes, and the many others who've also
contributed.
If you are already using Partial Clone, or would like to help us test Partial
Clone on a large project, please get in touch with me, James
Ramsay (group manager, product
for Create at GitLab), Jordi
Mon (senior product marketing
manager for Dev at GitLab), or your account manager.
For more information on Partial Clone, check out the documentation.
Cover image by Simon Boxus on
Unsplash
