In my last infrastructure update, I documented our challenges with
storage as GitLab scales. We built a CephFS cluster to tackle both the capacity
and performance issues of NFS and decided to replace PostgreSQL standard Vacuum
with the pg_repack extension. Now, we're feeling the pain of running a high
performance distributed filesystem on the cloud.
Over the past month, we loaded a lot of projects, users, and CI artifacts onto
CephFS. We chose CephFS because it's a reliable distributed file system that can
grow capacity to the petabyte, making it virtually infinite, and we needed
storage. By going with CephFS, we could push the solution into the infrastructure
instead of creating a complicated application. The problem with CephFS is that
in order to work, it needs to have a really performant underlaying infrastructure
because it needs to read and write a lot of things really fast.
If one of the hosts delays writing to the journal, then the rest of the fleet is
waiting for that operation alone, and the whole file system is blocked. When this happens,
all of the hosts halt, and you have a locked file system; no one can read or
write anything and that basically takes everything down.
What we learned is that when you get into the consistency, accessibility, and
partition tolerance (CAP) of CephFS, it will just give away availability in
exchange for consistency. We also learned that when you put a lot of pressure on
the system, it will generate hot spots. For example, in specific places in the
cluster of machines hosting the GitLab CE repo, all the reads and
writes end up being on the same spot during high load times. This problem is
amplified because we hosted the system in the cloud where there is not a minimum
SLA for IO latency.
Performance Issues on the Cloud
By choosing to use the cloud, we are by default sharing infrastructure with a
lot of other people. The cloud is timesharing, i.e. you share the
machine with others on the providers resources. As such, the provider has to
ensure that everyone gets a fair slice of the time share. To do this, providers
place performance limits and thresholds on the services they provide.
On our server, GitLab can only perform 20,000 IOPS but the low limit is 0.
With this performance capacity, we became the "noisy neighbors" on the shared
machines, using all of the resources. We became the neighbor who plays their
music loud and really late. So, we were punished with latencies. Providers don't
provide a minimum IOPS, so they can just drop you. If we wanted to make the disk
reach something, we would have to wait 100 ms latency.
That's basically telling us to wait 8 years. What we found
is that the cloud was not meant to provide the level of IOPS performance we needed
to run an aggressive system like CephFS.
At a small scale, the cloud is cheaper and sufficient for many projects.
However, if you need to scale, it's not so easy. It's often sold as, "If you
need to scale and add more machines, you can spawn them because the cloud is
'infinite'". What we discovered is that yes, you can keep spawning more
machines but there is a threshold in time, particularly when you're adding heavy
IOPS, where it becomes less effective and very expensive. You'll still have to
pay for bigger machines. The nature of the cloud is time sharing so you still
will not get the best performance. When it comes down to it, you're paying a lot
of money to get a subpar level of service while still needing more performance.
So, what happens when the cloud is just not enough?
Moving to Bare Metal
At this point, moving to dedicated hardware makes sense for us. From a cost
perspective, it is more economical and reliable because of how the culture of
the cloud works and the level of performance we need. Of course hardware comes
with it's upfront costs: components will fail and need to be replaced. This
requires services and support that we currently don't have today. You have to
know the hardware you are getting into and put a lot more effort into keeping it
alive. But in the long run, it will make GitLab more efficient, consistent,
and reliable as we will have more ownership of the entire infrastructure.
How We Proactively Uncover Issues
At GitLab, we are able to proactively uncover issues like this because we are
building an observable system as a way to understand how
our system behaves. The machine is doing a lot of things, most of which we are
not even aware of. To get a deeper look at what's happening, we gather data and
metrics into Prometheus to build dashboards and observe trends.
These metrics are in the depth of the kernel and not readily visible to humans.
To see it, you need to build a system that allows you to pull, aggregate, and
graph this data in a way you can see it. Graphs are great because you can get a
lot of data in one screen and read it with a simple glance.
For example, our fleet overview dashboard shows how many different workers are
performing in one view:
How we used our dashboard to understand CephFS in the cloud
Below, you can see OSD Journal Latency. You can see how, over the last 7 days shown, we had a spike.
This is how much time we spent trying to write to this journal disk. In general,
we roughly perform commit data to this journal within 2 to 12 seconds. You can
see where it jumps to 42 seconds to complete -- that delay is where we are being
punished. The high spikes show GitLab.com is down.
What's great about having this dashboard is that there is a lot of data available
quickly, in one place. Non-technical people can understand this. This is the
level of insight into your system you want to aim for. You can build on your own
with Prometheus. We have been building this for the last month, it's close to the
end state. We're still working on it but to add more things.
This is how we make informed decisions to understand as best as we can what is
going on with our infrastructure. What we tend to do is whenever we see
a service failing or performing in a way that is unexpected, we pull together a
dashboard to highlight the underlaying data to help us understand what's happening,
and how things are being impacted on a larger scale. Usually monitoring is an afterthought
but we are changing this by shipping more and more detailed and comprehensive
monitoring with GitLab. Without detailed monitoring you are just guessing at
what is going on within your environment and systems.
The bottom line is that once you have moved beyond a handful of systems it is no
longer feasible to run one-off commands to try and understand what is happening
within your infrastructure. True insight can only be gained by having enough
data to make informed decisions with.
Recap: What We Learned
- CephFS gives us more scalability and ostensibly performance but did not work well in the cloud on shared resources, despite tweaking and tuning it to try to make it work.
- There is a threshold of performance on the cloud and if you need more, you will have to pay a lot more, be punished with latencies, or leave the cloud.
- Moving to dedicated hardware is more economical and reliable for the scale and performance of our application.
- Building an observable system by pulling and aggregating performance data into understandable dashboards helps us spot non-obvious trends and correlations, leading to addressing issues faster.
- Monitoring some things can be really application specific which is why we are building our own gitlab-monitor Prometheus exporter. We plan to ship this with GitLab CE soon.
