At cloudscale, we care deeply about data privacy and digital sovereignty. We have our own GPU instances which we fully control. When discussing coding agents, it therefore felt natural to compare self-hosting with hosted models and public providers like Anthropic’s Claude Code.

We spent several weeks evaluating this setup. The short answer: self-hosting works, but you inherit complexity that hosted providers hide from you.

The test setup

cloudscale.ch offers GPU instances equipped with NVIDIA RTX PRO 6000 Max-Q GPUs with 96GB VRAM each. Up to four GPUs can be attached to a single VM. Our setup consisted of one VM with one GPU attached:

                ┌───────────────┐
                │   Clients     │
                │ (opencode,    │
                │  agents, …)   │
                └──────┬────────┘
                       │
                ┌──────▼────────┐
                │    Caddy      │
                │    (TLS)      │
                └──────┬────────┘
                       │
                ┌──────▼────────┐
                │   LiteLLM     │
                │ (API gateway) │
                │ OpenAI compat │
                └──────┬────────┘
                       │
                ┌──────▼────────┐
                │     vLLM      │
                │  (inference)  │
                └──────┬────────┘
                       │
                ┌──────▼────────┐
                │     GPU       │
                │ RTX 6000 96GB │
                └───────────────┘

Caddy handles TLS termination. LiteLLM provides an API gateway with model routing. vLLM loads and serves the model using the underlying GPU.

Why LiteLLM

LiteLLM gives you a single API endpoint that can route requests across multiple backends: a local model on vLLM, an alternative local model, or a hosted provider as fallback. It also provides access control and usage tracking as well as many more features.

            ┌────────────────────┐
            │     LiteLLM        │
            │  (routing layer)   │
            └─────────┬──────────┘
                      │
        ┌─────────────┼─────────────┐
        │             │             │
 ┌──────▼──────┐ ┌────▼──────┐ ┌────▼────────┐
 │ Local 35B   │ │ Local alt │ │ Hosted LLM  │
 └─────────────┘ └───────────┘ └─────────────┘

What fits in 96GB

Very early on, we had to determine what fits into 96GB of VRAM. The sweet spot seems to be ~26-35B models and we had good success with FP8 quantization. A natural next step is NVFP4: our RTX PRO 6000 Max-Q GPUs use NVIDIA's Blackwell architecture, which has native hardware support for this 4-bit floating-point format. Compared to FP8, NVFP4 should either let us fit larger models into 96GB or leave more headroom for the KV cache and concurrent requests. We haven't tested it yet, but it's high on our list.

Our test setup was minimal: a single engineer comparing results between the self-hosted models and Claude Opus 4.6/4.7 using Claude Code. Towards the end of our testing period, we briefly tested concurrency, but only to a limited extent. A broader, production-like test is still pending.

Model evaluation

Initially we tested Qwen2.5-Coder-32B and deepseek-coder-33b-awq. This was not sufficient for our use case. Partly because our setup was still suboptimal at that stage compared to later in the test period, but also because newer models are just much better than older ones.

When Gemma 4 was released, we immediately switched to it (first gemma-4-31B-it, then gemma-4-26B-A4B-it). The improvement was night and day. This model worked much better in all use cases. However, compared to Claude Code, it still lagged significantly behind when it comes to code quality.

Qwen3.6-35B was finally the model which gave us confidence that self-hosting is viable. Coding quality is quite good, and reasoning ability is solid as well. Compared to Claude Opus 4.6/4.7, it's not quite there yet, but it's actually usable.

To compare more, we also configured LiteLLM to proxy models from Infomaniak: Qwen3-VL-235B, GPT-OSS-120B, Kimi-K2.5. We chose Infomaniak because it is a Swiss provider that hosts these models on its own infrastructure in Swiss data centers.

Comparing these three, there was a clear winner when it comes to reasoning and coding ability: Kimi-K2.5. We did notice a bit of an issue with latency though. During the day the latency (Time To First Token TTFT) was much higher than in the evening. This appears to be a provider-side issue and may improve with further usage or coordination.

The time-consuming part: Configuration & Iterations

The biggest surprise was not getting a model to run, but getting it to behave like a reliable coding assistant.

Early on, we noticed that configuring LiteLLM and vLLM is non-trivial. There are lots of knobs to tune to get a system running smoothly. Each model required its own set of parameters to be tuned and iterated on. Context window, max token length, KV cache, and VRAM usage were among the most challenging aspects to figure out. We do have a configuration which mostly works now, but it's not fully there yet. It still requires more iterations.

Example configuration

Example flags we used for Qwen3.6-35B-A3B-FP8:

--max-model-len 131072
--gpu-memory-utilization 0.92
--enable-prefix-caching
--enable-auto-tool-choice
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--kv-cache-dtype fp8
--max-num-seqs 8
--max-num-batched-tokens 8192

These parameters are highly model-specific, and finding a configuration requires still more experimentation. In particular, gpu-memory-utilization and kv-cache-dtype influenced how efficiently we could use the available VRAM, while max-model-len had a direct impact on memory usage and latency. These settings are not final, and we expect further improvements as our understanding of vLLM, LiteLLM, and agent behavior evolves.

Agent

The agent we used also has a significant impact on the overall experience. We initially wanted to test using the Claude code CLI in order to have the least difference between different options we tested. However it turned out that it's not quite trivial to make Claude code work against self-hosted systems because it uses a different API (Messages API and not Responses/Chat Completions API). While LiteLLM has a translation layer, it did not work reliably in our case. Most likely because the code path to decide on which translation layer to use is not configured for self-hosted models by default.

Beyond the privacy and sandbox trade-offs that come with any agent choice, agent compatibility turned out to be a bigger constraint than expected.

Opencode worked very well from the get-go. It's highly customizable and easy to configure different models for different agents. Still, it takes time to figure out which model to use for which agent (plan vs. build vs. explore vs. general), and settings such as temperature have a big effect on the quality.

Is self-hosting viable?

Getting a self-hosted system to perform well requires tuning across multiple layers—model, inference stack, and agent integration. There are many parameters to tune until you get a decent system. It still may not support all features provided by hosted solutions (e.g. features like exposing model reasoning are not consistently supported with all models).

What you get with self-hosted is full control, data privacy, and predictable cost. But it requires time to tune, debug, and maintain. Case in point: during our testing period, we upgraded vLLM and LiteLLM almost daily.

With hosted providers, you get good defaults, mostly stable behavior, and enjoy minimal setup. However, you also take on a heavy dependency on the reliability of that service: when the provider has an outage, degrades performance, or changes pricing or model behavior, your engineering workflow is directly affected.

To sum up, self-hosting is viable, but not plug-and-play. You inherit the complexity that hosted providers hide. This partially also applies to hosted models like those from Infomaniak. While you get an already optimized model setup, choosing which model to use for what task, and ensuring cost predictability is also non-trivial. In both cases, the number of users and expected context window are important parameters.

For cloudscale.ch, we have not yet reached a final conclusion. What is clear, however, is that self-hosting coding LLMs is viable but introduces a non-trivial operational and engineering cost that is often underestimated.

Let me explain what I mean.

At cloudscale, the belief has always been that where control and independence matter, operating one's own stack is the best approach: from our Linux-based OS to our Ceph storage and OpenStack cloud, all the way down. Self-hosted inference is simply the next logical layer of that same commitment: As AI becomes core infrastructure, relying on external providers would contradict the principles of digital sovereignty, as outlined in this article. cloudscale offers GPU infrastructure for its customers and self-hosted inference is putting it to use in-house.

The Clipboard Problem

Even on the most privacy-conscious engineering team, you are always one mishap away from a leak. This isn't hypothetical. We've all seen it happen with Google Translate. Someone works on a ticket written in a foreign language, pastes a block of customer data to translate it, and suddenly that data has touched a server none of us approved.

Coding agents introduce the same failure mode at higher velocity, and they don't wait to be asked. They reach for what they need: a schema, an env file, a support thread sitting open in context. The agent helpfully processes it. You get your answer. And somewhere, something you didn't intend to share has left the building.

These are the stakes: we're not just protecting source code. We're protecting proprietary logic, credentials, customer data, and whatever else ends up in that prompt. Self-hosting your inference layer is one of the most meaningful choices you can make here. But it's a starting point, not a finish line.

The Obvious Part: Self-Hosted Inference

If you decide to go down this road, you probably picture something like the Ollama experience: pull a model, run a command, done. The go-kart version. Clean, fast, fun. And for one engineer, on one machine, running one task at a time, it genuinely is.

Then the second engineer joins. Then the third. Then someone runs Claude Code with four parallel sub-agents on a long-context refactor, while two colleagues are doing the same. Suddenly you need to think about who gets GPU time, memory bandwidth, and when, how requests queue, whether one heavy session can starve everyone else. You didn't buy a bigger go-kart. That's how you end up with a commercial jetliner on the runway while barely understanding why you need flaps. Not because you made a bad decision, but because a tool that works beautifully for one person becomes infrastructure the moment it serves a team. And infrastructure has a completely different set of requirements than a local dev tool.

And here's the uncomfortable truth: if you actually care about safety, you need to go deep. Blindly accepting whatever your favorite AI suggests as a fix for your performance problem is exactly the behavior you're trying to protect against, so the same standard applies here. You need to actually understand what's happening, and that means going deep on dynamic and continuous batching, KV caching, and before long you're dusting off your understanding of attention mechanisms just to reason about how many long-context multi-agent users fit into a given amount of VRAM. The good news: this stuff is genuinely fascinating. The bad news: it's probably not what you had in mind when you started this undertaking on a Tuesday afternoon hoping to offload the boring parts of your development.

This isn't a reason to avoid it. I genuinely think it's worth it, if you value privacy. But not as a side project, not as a one-afternoon setup that then runs unattended.

Update: Michael explains more technical details in a separate post.

The Overlooked Part: The Chatty Client Problem

Here's the assumption that bites people: "I'm running the model myself, so I'm private."

Maybe. But the model and the client are two separate things, and they have two separate surfaces that can phone home.

Your inference backend, Ollama, vLLM, whatever you're running, may have diagnostics, telemetry, or update-check behavior baked in. That's worth auditing, but it's usually the easier half to control.

Your agent client, Claude Code, Cursor, Copilot, whatever your developers are actually typing into, is a different story entirely. The client can, and most of the well-known ones do, send telemetry, crash reports, usage analytics, and in some cases full prompt content to vendor servers, regardless of where the model lives. "Open source" on the label doesn't automatically mean audited data practices; some of the most popular tools in this space are more opaque than they appear.

The categories to look for when you audit:

• Usage analytics - what gets counted and reported
• Crash reporting - Sentry-style tools that may capture context around errors
• Prompt feedback loops - the big one: is the vendor training on your prompts? Read the terms carefully.
• License and update pings - lower risk, but worth knowing about

Now, the honest part: reading privacy policies is the boring part of this work. I know. We became engineers so we wouldn't have to do this. But for this specific use case, the privacy policy is load-bearing. It's the document that tells you what actually happens to your data.

The Danger Zone: What Developers Need to Understand Now

Everything we've discussed so far, the careful inference setup, the audited client, the dialed-in permissions, can be silently undone by what happens in this section. A single successful prompt injection can turn your sovereign, self-hosted, privacy-preserving setup into an exfiltration machine. All that careful work, on fire, from the inside.

Every single one of these risks applies equally if you're running Claude Code against Anthropic's API, Cursor against OpenAI, or any self-hosted solution. The threat model doesn't care where the model lives. It cares what the agent is allowed to do.

This is my current shortlist, not exhaustive, and guaranteed to look different in six months. But these are the concepts I keep coming back to. (As it turns out, OWASP agrees: they published an OWASP Top 10 for Agentic Applications for 2026, and the overlap is uncomfortably exact.)

Prompt injection. An attacker plants instructions inside something the agent will read, a README, a code comment, an API response, a file in the repo. The agent reads it and treats it as instruction, the same way it treats yours. Your codebase, and everything it touches, is now an attack surface. This is why tools like Context7 and other MCP servers that pull in external content by design give me cold sweats. The injection surface isn't a bug someone introduced, it's the feature.

Slopsquatting (supply chain attack via model hallucination). Models confidently suggest packages that don't exist. Attackers register those hallucinated package names with malicious payloads, waiting for the agent, or the developer approving it, to run the install. It's the npm install problem applied to the space between what the model knows and what it invents. Your agent doesn't know the difference between a real package and a plausible-sounding one.

Tool permissions and sandbox scope. An agent that can read files, run shell commands, and make outbound HTTP requests is extremely powerful, and default configurations are almost always too permissive, at least for me to sleep well. What makes this worse than a misconfiguration: agents don't shrug when a tool is denied. They optimize around the constraint, chaining whatever's available to reach the same destination. Before you trust that your setup is locked down, it's worth reading and testing (!) exactly what your sandbox setting actually covers. The answer is usually narrower than the name implies.

So: do you think the AI named after the father of information theory can read and exfiltrate /etc/passwd while running in sandboxed mode?

If we want to use these systems safely and well, we need to make space for people to learn how they fail, how to test their boundaries, and how to build with them responsibly.

The Mail Server Lesson

Running your own mail server never protected you from phishing. Sovereignty and security are related, but they're not the same thing, and if privacy is actually your goal, you need both. Sovereignty gives you control over where your data lives. Security determines whether it stays there. One without the other is a half-measure. You can self-host everything perfectly and still get hit through a prompt injection in a README. You can have airtight security practices and still leak prompts through a client you never audited.

I want to be clear about what this piece is not. It's not a warning to stop using agents. It's not a criticism of the tools or the teams building them. This is an exciting moment, I think this is one of the more interesting periods to be writing software.

But we as an industry have also quietly accumulated new problems most of us didn't know existed a year ago. And they are solvable, but not alone. The OWASP Top 10 for Agentic Applications is the industry's clearest current attempt to name these problems together. Within each organization it takes engineers willing to go deep, and management willing to make that depth possible. And it takes the security team treating developers as allies to educate, not risks to contain. The teams that treat this as a shared problem will be fine. The ones that throw it over a wall won't.

We've been here before.

When we introduced CSI snapshot support in our Kubernetes CSI driver, we stressed an important distinction: snapshots help with operational recovery, but they do not replace backups.

Snapshots are ideal for rolling back changes or recovering from failed deployments. However, because they remain tied to the original storage environment, they do not protect against the loss of a full Kubernetes setup.

In this post, we outline how to build a Disaster Recovery workflow for Kubernetes workloads on cloudscale infrastructure, combining CSI snapshots, Velero orchestration, and cross-zone Object Storage to restore applications and their data after a complete environment loss.

Snapshots vs Backups

Before touching YAML or CLI commands, we need a shared terminology.

| Term              | Meaning                                                 |
|-------------------|---------------------------------------------------------|
| Snapshot          | Point-in-time copy stored on the same storage cluster   |
| Backup            | Recoverable copy independent of original infrastructure |
| Disaster Recovery | Ability to rebuild workloads after infrastructure loss  |

Snapshots are fast because they live close to the source volume. On cloudscale, they are implemented using copy-on-write technology inside the storage cluster.

That makes them ideal for:

• Upgrade safety nets
• Migrations
• Quick rollback scenarios
• Cloning production data into test environments

But if the storage cluster itself disappears, snapshots disappear with it. This is why we explicitly recommend keeping a copy of your data at another geographic location. The remainder of this article shows how to implement exactly that, using standard Kubernetes tooling.

What CSI Snapshots Change

With the release of CSI snapshot support, Kubernetes gains native awareness of storage recovery points. The driver exposes the standard Kubernetes VolumeSnapshot API, which means snapshots are no longer something managed exclusively through a provider interface or our Control Panel. Instead, they become first-class Kubernetes resources.

At first glance this may look like a small technical addition. Operationally, however, it changes how backup and recovery workflows can be designed. Once snapshots exist as Kubernetes objects, ecosystem tools can interact with them directly. Backup software such as Velero can request snapshots, track them as part of a backup operation, and later use them during restores, all through standard Kubernetes APIs. The result is a workflow that remains portable, automation-friendly and aligned with upstream Kubernetes concepts.

This capability also highlights a limitation that often goes unnoticed. Many Kubernetes backup strategies rely on exported manifests combined with storage snapshots. As long as the cluster and its storage remain available, recovery appears straightforward, deleted namespaces or failed deployments can usually be restored without difficulty.

The situation changes once the underlying storage is no longer accessible. Recreating Kubernetes objects is rarely the challenge, but recovering the data they depend on, is.

Disaster Recovery therefore requires separating three independent concerns: Kubernetes resource state, a consistent recovery source for volume data, and a durable copy stored outside the original infrastructure. CSI snapshots address only one of these aspects. They provide fast recovery points, but they remain bound to the same storage environment.

This distinction leads directly to the hybrid approach described next.

Demo

Before starting, you need:

• A Kubernetes cluster (version 1.28 or newer)
• cloudscale CSI driver installed (at least v4.0.0)
• Velero CLI installed locally (tested with v1.18.0)
• S3-compatible Object Storage.

With that, we build a hybrid approach:

• CSI snapshots provide consistent recovery sources.
• Velero orchestrates backups and restores.
• Object Storage stores durable copies in another region.

In this example, the Kubernetes cluster runs in LPG, while backup data is written to Object Storage in RMA, separating recovery data from the original infrastructure. The same approach also works with other providers.

The core idea is simple: snapshots provide consistency, while Object Storage provides survivability.

1. Create Object Storage Backup Location

First, create an Object User and save its credentials to a file:

cat <<EOF > credentials-velero
[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
EOF

Then create a bucket in a different region than the cluster. In this example we name it velero-backups. As our Object Storage exposes an S3-compatible API, Velero's AWS plugin works without modifications.

2. Install Velero with CSI Support

The important part is enabling both CSI snapshots and the Data Mover.

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket velero-backups \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --backup-location-config \
  region=RMA,s3ForcePathStyle=true,s3Url=https://objects.rma.cloudscale.ch \
  --snapshot-location-config region=LPG \
  --features=EnableCSI,EnableCSIDataMover

What this configuration establishes:

• Snapshots are created in LPG (the region where the cluster is running)
• Backup data is stored in RMA (needs to be the site the bucket has been created)

This way, restores do not depend on the original storage cluster. This separation is the foundation of Disaster Recovery.

3. Create a Demo Workload

We deploy a minimal namespace containing a PVC and a Pod writing data to a file.

kubectl create ns backup-demo

Create a PersistentVolumeClaim:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc
  namespace: backup-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: cloudscale-volume-ssd
EOF

Writer Pod:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: writer
  namespace: backup-demo
spec:
  restartPolicy: Never
  containers:
    - name: writer
      image: busybox
      command:
        - sh
        - -c
        - |
          while true; do
            echo "cloudscale recovery demo \$(date '+%Y-%m-%d %H:%M:%S')" >> /data/value.txt
            sleep 5
          done
      volumeMounts:
        - mountPath: /data
          name: vol
  volumes:
    - name: vol
      persistentVolumeClaim:
        claimName: demo-pvc
EOF

Verify data exists:

kubectl exec -n backup-demo writer -- cat /data/value.txt

4. Create the Backup

Create the Disaster Recovery backup:

velero backup create dr-demo \
  --include-namespaces backup-demo \
  --snapshot-move-data \
  --wait

Velero now performs a coordinated backup workflow.

First, Velero stores Kubernetes resource metadata in Object Storage.

Next, Velero requests CSI snapshots for all PersistentVolumeClaims in the namespace. The cloudscale CSI driver creates the snapshots without interrupting running workloads.

Velero then prepares the data transfer by creating a temporary volume from each snapshot. The Velero Node Agent mounts this volume and reads its contents using the CSI Data Mover.

The snapshot data is copied into Object Storage in RMA, while the production PVC remains untouched throughout the process. The temporary volumes are removed again.

These steps will look like this in the Control Panel:

5. Simulate Catastrophic Failure

A backup that is never restored is only a theory, so let's simulate loss of the environment:

kubectl delete ns backup-demo

Everything disappears: Pods, PVCs, and Kubernetes objects are removed, leaving only the off-site backup. This can be verified in the Control Panel, where the PVCs are no longer present.

6. Restore the Namespace

We can now load the backup to recreate the workload.

velero restore create dr-demo-restore \
--from-backup dr-demo \
--wait

Velero recreates the namespace, creates a PVC, restores the volume content and starts a Pod. This can take a moment. After the process is done, you can verify data:

kubectl exec -n backup-demo writer -- tail /data/value.txt

If the value matches the original one, while new values are written, the Disaster Recovery test succeeded.

Conclusion

CSI snapshot support brings cloudscale storage into Kubernetes-native recovery workflows. Using standard APIs and tools like Velero, snapshots can evolve from simple rollback mechanisms into real recovery strategies.

The example shown here is intentionally minimal: back up a namespace, remove it, and restore it from scratch. The goal is not complexity, but proof that recovery works independently of the original environment.

The key takeaway is straightforward: Disaster Recovery begins where recovery no longer depends on the infrastructure that failed.

Before diving into our load mitigation strategy, let's introduce some RADOS Gateway (RGW) internals. When an S3 request hits objects.<region>.cloudscale.ch, it travels through two major layers before hitting our Ceph Storage Cluster.

Frontend: The part of RGW that accepts client HTTPS requests, parses S3/Swift APIs, and places those requests into the internal queue of RGW. It behaves essentially like the "public-facing door".

Backend: The part of RGW that executes the actual operations against Ceph, such as reading objects, writing data, listing bucket contents, and performing metadata lookups. Backends are the "workers" that do the heavy lifting.

We run multiple frontends and backends to handle high loads. When a client sends a request, the path looks like this:

                   ┌─────────────────────┐
Client  ──https─►  │    RGW Frontend     │
                   │ (HTTP server layer) │
                   └──────────┬──────────┘
                              ▼
                   ┌─────────────────────┐
                   │    RGW Backend      │
                   │ (RADOS operations)  │
                   └──────────┬──────────┘
                              ▼
                   ┌─────────────────────┐
                   │        Ceph         │
                   │  (storage cluster)  │
                   └─────────────────────┘

Load Handling Improvements

New customers bring new load patterns, and one such pattern has caused our RGW backend processes to be strained. Loki, an increasingly popular log aggregation system, uses S3 for storage and for log queries. These queries use concurrency to speed up retrieval, showing up as sharp spikes on our frontends.

While our services have been tuned for increasing amounts of traffic over the years, these tunings are never final and new load patterns require additional traffic-engineering. Very spiky peak loads could be handled by adding lots of backends to take on the load, but that is neither economical, nor sustainable: We want to target a certain base-load, with some room for peaks. But we should not provision our services to support loads magnitudes larger than our average.

Maybe counter to intuition, we achieved better peak-load behavior by limiting the number of requests we accept in the backend, instead of adding more backends to handle increased load.

Instead of accepting whatever peaks we get, we can also try and flatten the traffic curve, which is what we do now, limiting the number of concurrent requests at every step:

• The backends have a certain limit of requests they accept.
• Once the backends are at capacity, the frontend queues requests to it.
• Once the frontend is at capacity, the kernel queues requests to it.
• Once the kernel is at capacity, requests may fail.

As these queues fill, we are informed by our monitoring. So in practice, only the first limit is ever reached. When this happens, some interesting properties come into play:

• We start returning 503 Slow Down to the clients that cause the spikes: This is the standard HTTP code used by S3 to signal to clients to back off.
• We start preferring certain requests over others: For now we give priority to read over write requests, but we might consider giving priority to clients with few concurrent requests over clients with many concurrent requests (for increased fairness).

So far this has proven to be very successful in preventing spikes from overwhelming our infrastructure.

Handling RGW slow-downs in rgw-metrics

The rgw-metrics service periodically queries the RGW for usage information about every bucket. This data is used in the Control Panel in the Object Storage tab to display historical usage information and for the billing. For a deeper explanation, see our engineering blog Improving metrics collection for our Object Storage.

Normally RGW runs smoothly and rgw-metrics receives reliable responses. This had always been the case, so the system was never designed to handle transient slow-down responses.

When RGW became busy under heavy production load, it became less responsive and returned 503 Slow Down responses. You may recognize those from the chapter above. rgw-metrics interpreted these errors as hard failures. Unfortunately, this triggered alerts for the on-call team as the usage data collection service restarted repeatedly. This reduced the number of collected metrics, lowering the resolution of the usage curves.

We now treat 503 Slow Down responses as temporary conditions and instead of failing immediately, the service now backs off and retries after a longer interval. This allows the metrics service to recover automatically after high-load peaks and prevents waking on-call engineers in the middle of the night.

The core lesson: Load related optimizations don't only affect customers, they also impact our internal systems. Our metrics service was an unintended casualty of a change that otherwise improved production stability.

Outlook

We continue to monitor the system closely and extended our tests to include more concurrency and load-related scenarios, so that we can catch similar edge cases earlier and ensure a smooth experience for our users.

The Audit Log API

We recently added an endpoint to our public API that allows retrieving audit logs records for a specific project:

{
  "next": "https://api.cloudscale.ch/v1/project-logs?cursor=10225",
  "poll_more": "https://api.cloudscale.ch/v1/project-logs?cursor=10225",
  "results": [
    {
      "ip_address": "185.98.122.110",
      "action": "volume_create",
      "message": "Volume 'cache' has been created",
      "timestamp": "2025-09-03T11:15:19.391447Z",
      ...
    },
    ... // Up to 199 more records.
  ]
}

All actions performed in our control panel by logged-in users or using API tokens are recorded in the audit log. Because the audit log can contain an unbounded number of records, the API uses pagination to return the results.

Polling in the Audit Log API

The audit log API allows a client to poll for new records that were added since the last request. For this reason, it returns a URL in the response's poll_more field. Requesting that URL will return all records (possibly spread across multiple pages) that have not yet been returned.

The URLs in the poll_more field have a cursor parameter (I'll call those URLs cursor URLs). This parameter simply contains the primary key of the database row of the previous request's last record. In our case, the primary key column is called id:

dev=# \d log
                                 Table "public.log"
   Column   |           Type           | Nullable |             Default
------------+--------------------------+----------+----------------------------------
 id         | integer                  | not null | generated by default as identity
 ip_address | inet                     |          |
 action     | character varying(50)    | not null |
 timestamp  | timestamp with time zone | not null |
[some more fields omitted]
Indexes:
    "log_pkey" PRIMARY KEY, btree (id)

When the application receives a request for a cursor URL, it uses the cursor parameter to determine the set of records to return to the client.

The API guarantees that when polling for records using the poll_more URL, no records will be skipped or returned twice. This is easy to accomplish if the set of records have IDs that are monotonically increasing (foreshadowing): simply return all records with a primary key bigger than cursor, sorted by the primary key.

PostgreSQL Sequence Generators

As you have seen in the previous section, our primary key column id is set up to be filled in by Postgres automatically. The important bit is the generated by default as identity, which is documented in CREATE TABLE. This uses an implicitly created sequence generator, which is the standard way in Postgres to automatically fill an integer primary key column. Sequence generators are documented separately under CREATE SEQUENCE.

The implementation of Postgres' sequence generators needs to fulfill a few requirements:

1. The generated values need to be unique (meaning it can't e.g. use the current time, because due to granularity or system clock changes, it could return the same value twice).
2. The generated values have to fit in an integer or bigint column (meaning e.g. random UUIDs can't be used).
3. It needs to be fast (meaning it can't search the existing data for an unused ID).
4. It needs to be thread-safe without using locks that last until the end of the transaction (meaning select max(id) + 1 from ... can't be used).

To accomplish this, Postgres basically uses a global integer variable per sequence generator, accessed using a short-lived lock. Each database process can increment the corresponding variable and use the new value (the cache parameter of create sequence can be used to changes this to a more efficient but also more complicated scheme).

This implementation fulfills all the requirements listed above. There's a fifth requirement that the implementation appears to fulfill, but in reality doesn't: That generated IDs appear in increasing order in the database. Look at the following example of two concurrently processed transactions:

In the example, two transactions request a value from the sequence generator and are handed out consecutive values (which are then used to set the id column of rows inserted into our audit log record table). Both transactions commit, but in the reverse order than they requested a value from the sequence generator.

At the time indicated by the bold dashed line, the transaction that used ID "5" has already committed, and its result is visible to other transactions, but the transaction that used ID "4" has not yet committed. After that transaction has also committed, both rows will be visible.

If at the point in time of the bold dashed line, a client would request the most recent audit logs, the result's poll_more URL would have cursor set to "5", because that's the largest value in the id column. Even if the transaction that used ID "4" would later commit, that row would never be returned to the client. This would violate the guarantee that no rows are skipped when polling for records using the poll_more URLs.

Implementing truly Sequential IDs

So we went to look for a solution that matched the following criteria:

• Generate sequential IDs.
  Audit log records need to be assigned IDs in the order in which the transactions are committed, even if transactions are running in parallel. This is required by how we use cursor URLs to allow a client to poll for new records.

• Assign the sequential IDs within the transaction.
  Delaying assigning the sequential IDs to after committing the transaction would prevent requests immediately following the transaction from returning the new audit log records.

• Low contention between transactions.
  Some of our transactions can run for up to a few 100ms. This is because we have to talk to systems external to our control panel application within some transactions. Acquiring locks for the whole duration of the transaction would prevent processing these transactions in parallel, resulting in bad performance.

After looking at a lot of different approaches, we came up with the following solution. The implementation has 4 parts:

• An additional column sequence_id added to our log record table log.
• A sequence generator used to generate values for sequence_id.
• A table log_sequence_id_seq_lock, which is only used for locking and does not store data.
• A deferred trigger on log that populates log.sequence_id.

The implementation lives fully in the database schema. No code in our application that writes audit log records needed to be modified. This is the full implementation (explanations below):

-- Add new column used to store the sequential IDs.
alter table log add column sequence_id integer unique;

-- Sequence generator used to generated values for log.sequence_id.
create sequence log_sequence_id_seq;

-- Table that is only used for locking by
-- log_sequence_id_trigger_fn().
create table log_sequence_id_seq_lock();

-- Trigger function that assigns values to log.sequence_id.
create function log_sequence_id_trigger_fn()
    returns trigger
    language plpgsql
as $$ declare
begin
    -- Prevent the function from running in parallel.
    lock table log_sequence_id_seq_lock in exclusive mode;

    -- Populate sequence_id of the newly inserted row.
    update log l
    set sequence_id = nextval('log_sequence_id_seq')
    where l.id = new.id;

    -- Insert already happened, so no need to return the row.
    return null;
end $$;

-- Trigger that runs log_sequence_id_trigger_fn() at the
-- end of the transaction.
create constraint trigger log_sequence_id_trigger
    after insert on log
    initially deferred
    for each row
execute function log_sequence_id_trigger_fn();

log_sequence_id_trigger_fn() is set up to run whenever a row is inserted into log using a trigger. The trigger is set to deferred, so the function will run at the end of the transaction, instead of immediately when the row is inserted. The functions should complete very quickly, which is important because it acquires a lock that prevents the function from running in parallel.

The lock used by log_sequence_id_trigger_fn() is a relation-level lock on table log_sequence_id_seq_lock. That table is only used for this purpose of locking, no data is stored in that table. An alternative would have been to use Postgres' Advisory Locks, but these are less convenient because they only allow integer values to be used as lock names. Using a table with a descriptive name seemed less obtuse.

After acquiring the lock, log_sequence_id_trigger_fn() uses nextval() to get a value from the sequence generator log_sequence_id_seq and update sequence_id of the inserted row with it. The lock is automatically released when the transaction finally commits.

This sequence generator works exactly the same way as one that gets implicitly created by Postgres when creating a generated by default as identity column. But because of the explicit locking around its usage, the problems mentioned above are prevented.

When processing API requests, the implementation can reliably select records added since the last request using the values of sequence_id.

Conclusion

We've seen how Postgres' sequence generators, which work well in most circumstances, especially when high transaction throughput is necessary, can't directly be used to guarantee sequential IDs on inserted rows. Nonetheless, Postgres has all the tools necessary to implement a solution that provides this guarantee without sacrificing too much in terms of performance or complexity.

Unless you're willing to take rather extreme approaches, your choice of mobile phone operating system is one between Apple and Google. The cloud service market is dominated by Amazon, Microsoft, and Google. To fullfil your SaaS groupware needs, feel free to choose between Microsoft and Google.

U.S. Big Tech

While there are few determined souls not using any services of the "Big Five" (Amazon, Apple, Meta, Microsoft, and Google), most of us will happily rely on some of their free offerings for personal use. As for satisfying your business needs regarding groupware or cloud services, no one in IT procurement has ever been fired for choosing "industry standards" like AWS, GCP, or Azure.

Entrusting the large U.S. players with your own or your customers data comes with significant downsides and risks though.

Marketing ahoy

Social media platforms are lovely to stay in touch with people from around the world. Seeing Marco on his latest kite surfing adventure in Egypt, or Rachel relaxing on her beach vacation in Greece is heart-warming and allows me to stay in touch with friends I don't interact with very often. Though know that If You're Not Paying For It, You Become The Product.

Vendor lock-in

While migrating your services to any of the mentioned vendors is made easy by a plethora of migration tools, you'll often find yourself in quite some pain trying to migrate away. In combination with purposefully obscure cost and subscription models, these kinds of relationship may often leave you feeling somewhat frustrated and powerless.

U.S. Cloud Act

After Microsoft refused to hand over data related to an FBI investigation - arguing the emails in question were located in an Ireland data center and therefore outside of U.S. jurisdiction - congress passed the Cloud Act in 2018, which:

• Expands the reach of U.S. law enforcement over data held by U.S. providers abroad.
• Facilitates foreign law enforcement access to data from U.S. providers.

U.S. jurisdiction extended

With the Cloud Act extending U.S. jurisdiction to any server operated by a U.S. company, if you entrust them with your data, you'll find it to be under the control of the U.S. government, no matter where it's physically stored.

In a recent example, after the Trump administration passed sanctions against the chief prosecutor of the International Criminal Court (ICC), Mr. Khan had his email account suspended by Microsoft, forcing him to resort to Switzerland-based ProtonMail.

As for hosting your data in a location of your choice, in an sworn testimony before a French Senate inquiry, Microsoft France's director of public and legal affairs, was asked whether he could guarantee that French citizen data would never be transmitted to U.S. authorities without explicit French authorization. And, he replied, "No, I cannot guarantee it."

The same applies to any digital service offering by U.S. tech firms.

While entrusting U.S. tech companies with personal information is a choice everyone has to make for themselves, doing so with customer data is not only negligent, it is likely to be against the law.

Open Source to the Rescue

Considering the above, combined with our desire to maintain as much control over our infrastructure as possible, cloudscale has made a decision to rely on external service providers as little as possible. We design, build and operate our own services, on our own hardware, as much as we can.

FOSS FTW!

While Free and open source software (FOSS) can sometimes feel a bit chaotic, it promises to best fulfill specific use cases and is open to contributions by anyone, ensuring maximum amounts of transparency, trustworthiness, and adaptability. It allows us to customize and extend functionality based on our needs, and we strive to contribute our improvements back upstream for the greater good of other parties relying on these projects.

Because if many people and organizations share a common technological need - like running a web server for example - the most reliable, transparent and customizable solution will often end up being an open source project, where everyone collaborates to achieve the best possible result.

Microsoft spent hundreds of millions trying to establish their proprietary, commercial web server Internet Information Server (IIS) as the tool of choice, only to get pretty much wiped off the face of the internet by open source solutions.

FOSS @cloudscale

The vast majority of our customer-facing services are built from open source components managed by us, tailored to our specific needs. Our operating system of choice is Linux, with Debian being our favorite distribution. Front-facing web services are handled by HAproxy and nginx. We use Ceph for storage, and OpenStack for our cloud service offering.

Running Our Own Stack

While we do use some commercial software like an on-prem GitLab instance, we also try to rely on free open source components for our daily work wherever reasonably possible.

Centralized authentication of our employees happens through an OpenLDAP directory, with the credentials required to operate cloudscale infrastructure stored in an internal Vault-based setup. Our monitoring is based on Zabbix, and we rely on Loki and Grafana for log aggregation and analysis. We use Netbox to manage our hardware inventory, our database needs are fulfilled by PostgreSQL and MariaDB clusters. For our custom-tailored mail setup, we use components like Postfix, Dovecot and Rspamd.

Combined with our high standards regarding security, reliability and maintainability, the design, implementation and operation of these services takes significant time and effort. We like to call the extra work required to take a technical implementation from fullfilling the basic requirements to something everyone in the team can fully agree to and be proud of "vergolden" (gold plating).

And while doing so not only for customer-facing components, but applying the same rigorous standards to our internal service landscape - used by a fairly small number of employees - might seem extreme, we feel like the benefits outweigh the drawbacks, since this level of attention to detail usually guarantees we won't have to worry about a certain component for a long time.

Ownership and Responsibility

As a cloud service provider, self-hosting is the only approach giving us full control over our own and our customer digital assets.

The reasoning behind, and the benefits of this approach are:

Security and Privacy

• We can mitigate any threats as soon as we become aware.
• Our internal communications never pass through third-party systems.
• Our employee records aren't sitting in some overseas data center.
• Our customers can rest assured their information is only processed by systems managed by us.

Transparency

• Relying on open source software allows us to review any code we use.
• We know exactly what’s running on our systems, there are no black boxes or mystery outages.
• If something misbehaves or breaks, we can analyze the anomaly and learn from it, preventing similar kinds of incidents from happening in the future.

Adaptability

• We can customize and optimize any component to our specific needs, whether it's adapting it to our specific use case, or tweaking it for maximum performance.
• We’re not limited by someone else's roadmap.

Resilience and Independence

• When you rely on SaaS, you’ll find yourself at the mercy of someone else’s uptime, policy changes, or business decisions.
• By building our own infrastructure, we gain a higher degree of operational independence.
• We're not tied to the fate of a service or company that might get acquired, discontinued, or priced out of reach.

In for the long game

• While paying an external company for a service subscription might be attractive short-term, the cost benefits tend to diminish over time.
• Once we have our tools set up up and working the way we want, only little maintenance is required over the course of the next few years.
• While the initial effort required might be high, our total cost of ownership decreases over time.

Skill Development

• Managing our own infrastructure pushes us to develop deep technical skills. We stay sharp. We learn. We grow.
• That experience directly translates into better services for our customers, as well as a stronger internal culture of capability, confidence and ownership.

Not Always Easy - But Worth It

Self-hosting takes time, skills, and a willingness to put in the required work. But the return on investment — in control, security, and integrity — is priceless.

For us, it’s not about rejecting SaaS entirely, but about being intentional. Wherever control, customization or independence matters, we choose to own the stack. Where commodification makes sense, we might consider outsourcing.

Digital Sovereignty

Digital sovereignty means being in control over your digital destiny - your IT infrastructure, data, and operations. It ensures your authority over how your data is stored, who can access it, and how your systems are run.

Key Aspects

• Digital Ownership: Full control over where data is stored, who accesses it, and how it’s processed.
• Software Autonomy: Freedom to debug, modify, and adapt software to your needs without vendor-imposed limitations.
• Infrastructure Independence: Ability to host your services wherever you feel comfortable.

Final Thoughts

In a world dominated by a few large players, digital sovereignty might seem like a radical act. For us, it's merely a conscious commitment to ownership, responsibility, and ultimately freedom.

And that's a choice we’ll keep making.

Recently, I took the time to delve into the uv package and project manager, which you've likely heard of if you're involved in the Python community in any way. uv is maintained by the same company as the Ruff linter and code formatter. The command-line interface (CLI) command felt fast and intuitive. But of course, I wanted to use IntelliJ IDEA for my project and at the same time take advantage of uv's features. This process was much less straightforward than I had expected, since while there is uv support in IntelliJ IDEA, it's currently not as well integrated as it could be.

That's why I have decided to write a blog post about how I currently set up my uv-based projects, primarily for my future self as a reference - but I guess it could also be helpful to others even if you have no prior experience with uv. In that sense, this blog is opinionated and just reflects my current approach.

It's important to note that this is a snapshot, everything might change with future versions.

The Goal and Intro to this Guide

For me, these are the minimal requirements for a good development environment:

• No, or at least almost no, typing of commands into the terminal. Instead, I want to rely on Run Configurations and similar first-class IDE features.
• An exception to the above point is the initialization of a project, where I actually prefer the CLI.
• Imports must be correctly resolved within the IDE
• Debugger must work for the production as well as test code
• IDE is aware of the Python dependencies installed in the Project

If you want to follow along, please prepare the following:

• A uv binary in your PATH (see Installing uv)
• An IntelliJ IDEA installation with the following plugins installed: Python and Python Community Edition. PyCharm should work as well, but I did not test it.
• Installed Python interpreter is optional, we'll use uv to set one up.

As of this writing the uv version I am using is 0.8.0, together with IntelliJ IDEA 2025.2 and version 252.23892.458 of the two Python plugins.

Setting Up a Project with uv

In this section, we will set up a project with uv and add some code. The goal is to see uv and some of its highlights in action.

Let's go ahead and create a new project. I pass the --package flag to uv init since I prefer to use src-layout.

mkdir my-hello-project
cd my-hello-project
uv init --package

The working tree now looks like this:

.
|-- .git [truncated]
|-- .gitignore
|-- .python-version
|-- pyproject.toml
|-- README.md
`-- src
    `-- my_hello_project
        `-- __init__.py

Since this is pretty boring, let's add some code to the project, including two dependencies:

# Empty the __init__.py file
cat << EOF >  src/my_hello_project/__init__.py
EOF

# Add some non-trivial code that requires a dependency
cat << EOF > src/my_hello_project/hello.py
import cowsay
import sys
import random


def run_hello() -> None:
    characters = ["cow", "tux"]
    random_char = random.choice(characters)
    python_version, *_ = sys.version.split()
    print(cowsay.get_output_string(random_char, f"Running on {python_version}"))
EOF

# Make my_hello_project a module
cat << EOF > src/my_hello_project/__main__.py
from my_hello_project.hello import run_hello

if __name__ == "__main__":
    run_hello()
EOF

# Create tests
mkdir tests
cat << EOF > tests/test_hello.py
import sys

import pytest

from my_hello_project.hello import run_hello


def test_hello(capsys: pytest.CaptureFixture) -> None:
    # act
    run_hello()

    # assert
    major, minor, micro, *_ = sys.version_info
    expected = f"{major}.{minor}.{micro}"
    captured = capsys.readouterr()
    assert expected in captured.out
EOF

Now we use uv to add the two dependencies to pyproject.toml:

uv add cowsay
uv add --dev pytest

And now we are ready for some action. Let's run the tests:

uv run -m pytest      # or -m my_hello_project to run the application

The cool thing is that in this command multiple things happen:

• A python interpreter was downloaded if none was found in your PATH.
• The uv.lock file that pins the exact versions of all dependencies was created.
• A virtual env was created in .venv.
• The dependencies were installed.
• An editable install of my_hello_project was created (see uv pip freeze).
• The tests were run.

Running the Project from the IDE

Now let's get our project running smoothly from within the IDE! Here are my step-by-step instructions on how to run the project from the IDE:

Important: Do not confuse my-hello-project (the name of the project) and my_hello_project (the name of the package)

1. Open the main my-hello-project directory from within IntelliJ IDEA.
2. Set up a Python SDK (Screenshot 1):
   • Navigate to File > Project Structure > Project Settings > SDKs.
   • Click the + button and select Add Python SDK from Disk.
   • In the dialog, select the following values:
     • Environment: Select Existing
     • Type: uv
     • Path to uv: /path/to/your/uv
     • Uv venv use: /path/to/python3/in/your/.venv/directory
   • Remember the name of the SDK you have just created.
3. Set the SDK for the project (Screenshot 2):
   • Navigate to File > Project Structure > Project Settings > Project
   • Select the SDK you have just created in the SDK dropdown.
4. Set the SDK for the module (Screenshot 3):
   • Navigate to File > Project Structure > Modules > my_hello_project > Dependencies
   • Select the SDK you have just created in the Dependency dropdown.
5. Create a new Run Configuration (Screenshots 4 and 5):
   • Navigate to Run > Edit Configurations
   • Click the + button and select Python
   • Ensure the following values are set:
     • Use SDK of Module: my-hello-project
     • module: my_hello_project
   • Add a "Before Launch" task:
     • Click "Modify Options"
     • Choose "Add before launch task"
     • Choose "Run External tool" and click +:
     • Choose the following values and close the dialog:
       • Name: uv sync
       • Program: uv
       • Arguments: sync
       • Working directory: $ProjectFileDir$
     • Caveat: If your "Before Launch" task is not shown in the Run Configuration, make sure to not only set the checkbox but also select the uv sync entry before clicking OK.

Now you can use the Run Configuration to run the project as expected. You should see some ASCII art and the Python version printed in the Run Output.

  _________________
| Running on 3.13.5 |
  =================
                 \
                  \
                    ^__^
                    (oo)\_______
                    (__)\       )\/\
                        ||----w |
                        ||     ||

Probably, you are wondering what the "Before Launch" task is for? uv sync ensures that the virtual environment is up to date and uses a Python version that matches the specifier in .python-version. Let's see that in action by switching to a different Python version (a pre-release in this case):

echo '3.14.0b4' > .python-version

and run the Program again through the Run Configuration and you are now on the pre-release version:

  ___________________
| Running on 3.14.0b4 |
  ===================
                        \
                         \
                          \
                           .--.
                          |o_o |
                          |:_/ |
                         //   \ \
                        (|     | )
                       /'\_   _/`\
                       \___)=(___/

uv as successfully upgraded the Python version used in the virtual env, even without us thinking a single second about any CLI commands. Isn't that great? :-)

Setting up a Run Configuration for tests is straight forward, just right-click the tests folder and select "Run 'Python tests in tests'" and you are set (Screenshot 6). I'd suggest then also adding a "Before Launch" task to run uv sync as described above. An even better approach is to the "Before Launch" task to the relevant Run/Debug Configuration Templates.

Epilogue: The "uv run" Run Configuration and Setting up an Entrypoint Script

You might have noticed that there's also a Run Configuration called "uv run." So, why did I opt for the lower level "Python" type with a "Before Launch" task to synchronize the virtual environment and execute the program? I faced two challenges:

• I only could get it to work in "Debug" mode, but not in "Run" (see also this issue).
• When upgrading to Python 3.14.0b4 it stopped working altogether.

But I guess that's just a matter of time and the issue will be resolved.

Finally, if your application is a CLI program, it's a good practice to set up an entrypoint script. This allows users to run your program with a simple command when installed on their system. For example, if you want your program to be callable by the name hello, set up the following:

cat << EOF >> pyproject.toml
[project.scripts]
hello = "my_hello_project.hello:run_hello"
EOF

The program can now be run with uv run hello. Or if you build and install it:

uv build
pipx install dist/*.whl
hello

The Problem: Everything Feels Important

During our regular retrospectives, we frequently encounter various tech debt issues, such as bringing up older code to our current coding standards and style, overly complex code that could benefit from refactoring, and libraries that require replacement or upgrading.

As engineers, our instinct is to want to fix everything – it's in our nature. But with limited time and competing product priorities, we needed a better way to decide what actually deserved our attention first.

The Solution: A 4x4 matrix

That's when we decided to dedicate a focused workshop session to really understand our tech debt. I'd recently come across an idea from Dave Farley's "Modern Software Engineering" YouTube channel in his video "Why Software Estimations Are Always Wrong," and thought we could adapt it for our needs.

The concept is simple: a 4x4 matrix that frames tech debt prioritization through two key questions:

• Y-Axis (Value): How much would you pay a colleague if they solved this issue?
• X-Axis (Cost): How much would you want to get paid to solve this issue yourself?

Rather than falling into traditional estimation methods, we embraced deliberately imprecise but universally relatable categories: a beer, a vacation, a car, or a house. These rough buckets allowed the team to make quick value-versus-cost comparisons without getting caught in the false precision that typically derails prioritization discussions.

Setting Up the Workshop

We gathered the entire development team. I prepared a large 4x4 grid on our virtual whiteboard, with "beer" to "house" marked on both axes. We initially started with the same layout orientation shown in Dave Farley's original video, but after working with it for some time, we found we needed to change the approach. We switched to positioning the origin (0,0) a.k.a. (Beer,Beer) at the bottom-left corner – the standard Cartesian coordinate system we're all familiar with. This conventional layout proved much more intuitive when working with more than a few stickies.

Each team member wrote down tech debt items on sticky notes, and we placed them on the matrix based on our collective assessment. The discussions that emerged were incredibly valuable.

The Results Spoke for Themselves

What emerged was a clear visual representation of our priorities:

• Top-left: High value, low cost – our obvious quick wins
• Bottom-right: Low value, high cost – items we should probably ignore
• Bottom-left: Low value, low cost – nice-to-haves
• Top-right: High value, high cost - Strategic investments worth planning for

Beyond Prioritization: Building Shared Understanding

The workshop's real success wasn't just the prioritized items we created. It was the shared understanding we developed as a team. We had honest conversations about complexity and about the pain points we each face daily.

As an additional advantage, the resulting matrix serves as an effective tool for communicating with stakeholders, facilitating discussions about the trade-offs between competing priorities.

Lessons Learned

Sometimes the simplest tools are the most powerful. A few sticky notes, a virtual whiteboard, and some beer-to-house analogies gave us the clarity we'd been missing for months.

There is just one final thing I need to note: none of our developers actually owns or wants a car. Maybe we should go with a beer, GA Travelcard, a vacation, or a house next time ;-).

XSS in the Login Form

Let us start with the most critical finding. After user authentication, our code executed window.location = variable to redirect users to a predefined page. What we overlooked was that window.location accepts JavaScript URIs, creating an unexpected cross-site scripting vulnerability. While we assumed the worst-case scenario would be users landing on unintended pages, attackers could actually exploit this by setting values like this:

window.location = "javascript:alert('xss')"

The penetration testers demonstrated how crafting a malicious URL and sending it to legitimate users, and hope that he logs into our control panel using that link could execute code after authentication. This experience serves as an important reminder that even seemingly innocuous code patterns can harbor significant security risks.

We fixed this vulnerability in less than 24 hours and conducted a review of our logs to ensure no signs of exploitation were present.

Outdated Dependencies

Providing the source code allowed the testers to examine our dependencies in detail, uncovering several vulnerabilities related to known CVEs with recent fixes. Fortunately, we use Renovate to automatically keep our dependencies up to date. Our self-hosted Renovate bot continuously monitors for new CVEs and creates merge requests as soon as patches are available.

As a result, the vulnerabilities identified during the test were already addressed before the report reached our inbox.

Proxy Headers

Another feature that immediately caught the attention of the penetration testers was the Custom Image Import functionality. Instructing our servers to download content from “any” location seemed suspicious to them, we suppose. They discovered that our forward proxy, through which our internal systems perform downloads, included X-Forwarded-For headers that contained internal host names. Additionally, the versions of the requests and similar libraries were revealed.

As a consequence, we have disabled these headers in the meantime.

Staying Vigilant

External penetration tests are invaluable for uncovering blind spots and reinforcing security practices. While we are satisfied with the results of this test, we know that maintaining security is an ongoing effort. That is why we have also added a .well-known/security.txt file to our website, making it easy for security researchers to find contact information for responsible disclosure. A big thank-you goes out to Sebastian Pipping for nudging us to implement this.

If you host a site or service, we highly recommend adding a security.txt file as well. It is a simple but effective way to facilitate vulnerability reporting.

Stay safe!

Ich verwende eine typische, modernen Unix-Shell und habe ein einfaches Ziel: Ich möchte mithilfe von grep alle Skripts unter /etc/init.d finden, welche den String case "$1" in enthalten:

michi@myserver$ grep -RF 'case "$1" in' /etc/init.d
/etc/init.d/udev:case "$1" in
/etc/init.d/nullmailer:case "$1" in
/etc/init.d/dbus:case "$1" in
/etc/init.d/haveged:case "$1" in
...

Ich verwende Bash 5.2. Eine andere populäre Shell unter Linux-Benutzer*innen ist die Z shell (zsh). Alles hier beschriebene funktioniert auch unter der Z shell. Wo Unterschiede bestehen, werde ich das anmerken.

Das funktioniert gut. Nun möchte ich das Gleiche automatisiert auf mehreren Servern tun. OpenSSH erlaubt es, auf der Kommandozeile, nach allen anderen Argumenten, ein Kommando anzugeben. Das Kommando wird auf dem Server ausgeführt, dann wir die Verbindung wieder getrennt:

$ ssh myserver uname -sr
Linux 5.4.0-165-generic

Also kann ich folgendes tun:

$ ssh myserver grep -RF 'case "$1" in' /etc/init.d
grep: : No such file or directory
grep: in: No such file or directory
/etc/init.d/udev:    case "$type" in
/etc/init.d/udev:case "$1" in

Leider funktioniert das nicht, oder nur so halb. Es sieht aus, als wäre grep mit ganz anderen Argumenten aufgerufen worden, als ich angegeben habe. Um dem auf den Grund zu gehen, werde ich die folgende Shell-Funktion verwenden. Diese füge ich am Anfang von meinem ~/.bashrc (~/.zshrc unter der Z shell) ein, sowohl lokal als auch auf einem der Server, auf die ich mich verbinden möchte:

show-args() {
    printf "%s\n" "$@" | nl -ba
}

show-args gibt mir die Möglichkeit, statt ein Kommando auszuführen, zu sehen, welche Argumente an das Programm übergeben worden wären. Wenn ich das lokal teste, sehe ich folgendes:

$ show-args grep -RF 'case "$1" in' /etc/init.d
     1	grep
     2	-RF
     3	case "$1" in
     4	/etc/init.d

grep würde also mit 3 Argumenten aufgerufen. Den Flags, dem String, nach dem gesucht werden soll, und dem Verzeichnispfad, welcher durchsucht werden soll. Dass dieses Kommando an ssh übergeben wird, ändert daran nichts. ssh wird mit grep als Kommando und den gleichen 3 Argumenten aufgerufen:

$ show-args ssh myserver grep -RF 'case "$1" in' /etc/init.d
     1	ssh
     2	myserver
     3	grep
     4	-RF
     5	case "$1" in
     6	/etc/init.d

Also muss das Problem auf der anderen Seite liegen. Indem wir show-args und ssh myserver auf der Kommandozeile vertauschen, können wir überprüfen, wie grep auf dem Server aufgerufen wird:

$ ssh myserver show-args grep -RF 'case "$1" in' /etc/init.d
     1	grep
     2	-RF
     3	case
     4
     5	in
     6	/etc/init.d

Der Such-String case "$1" in wurde also aufgeteilt in 3 Argumente, als wäre dieser ohne Anführungszeichen geschrieben worden. Was ist hier passiert?

SSH und nicht-interaktive Kommandos

Die Manpage von ssh gibt uns einen Hinweis für das beobachtete Verhalten:

If a command is specified, it will be executed on the remote host instead of a login shell. A complete command line may be specified as command, or it may have additional arguments. If supplied, the arguments will be appended to the command, separated by spaces, before it is sent to the server to be executed.

Wenn ssh mit einem auszuführenden Kommando aufgerufen wird, verwendet der SSH-client den "exec"-Request (RFC 4254, Abschnitt 6.5). Dieser sieht vor, dass ein einzelner String als Kommando mitgegeben wird. Auf dem Server wird dieser dann an die Login-Shell übergeben und von dieser ausgeführt.

1. ssh erhält als Kommando 5 Argumente show-args, grep, -RF, case "$1" in und /etc/init.d. Hier ist wichtig zu bemerken, dass die einfachen Anführungszeichen ('') nicht teil des 3. Arguments sind. Diese wurden von der lokal laufenden Shell bereits entfernt.
2. ssh fügt die Argumente zu einem String zusammen: show-args grep -RF case "$1" in /etc/init.d. Dieser wird via der SSH-Verbindung an den Server geschickt.
3. Die Shell auf dem Server parst diesen String und macht daraus 7 Teile: show-args, grep, -RF, case, "$1", in und /etc/init.d.
4. Da die einfachen Anführungszeichen entfernt wurden, wird das $1 als Verwendung einer Variable behandelt. Da diese Variable nicht gesetzt ist, wird sie durch den leeren String ersetzt. Das Resultat ist ein Argument der Länge 0.
5. Die Funktion show-args wird mit den restlichen 6 Argumenten aufgerufen.

Teil des Problems ist, dass das eingetippte Kommando zweimal geparst wird, einmal von der lokalen Shell und ein zweites Mal von der Shell auf dem Server. Die einfachen Anführungszeichen um 'case "$1" in' reichen also nicht aus. Eine Möglichkeit ist, das Argument in eine zweite Klammerung an Anführungszeichen einzupacken. Zusätzlich müssen alle Zeichen, die für eine Unix-Shell eine spezielle bedeutung haben, mit \ escaped werden:

$ ssh myserver show-args grep -RF "'case \"\$1\" in'" /etc/init.d
     1	grep
     2	-RF
     3	case "$1" in
     4	/etc/init.d

Nun wird das Kommando beim Parsen auf dem Server also in die gewünschten 4 Teile unterteilt. Und wenn wir show-args aus dem Kommando wieder entfernen, funktioniert das Kommando auch wie gewünscht:

$ ssh myserver grep -RF "'case \"\$1\" in'" /etc/init.d
/etc/init.d/udev:case "$1" in
/etc/init.d/nullmailer:case "$1" in
/etc/init.d/dbus:case "$1" in
/etc/init.d/haveged:case "$1" in
...

Eine automatisierte Lösung

Das Quoting und Escaping im letzten Beispiel hat funktioniert, aber es kann mühsam und unübersichtlich werden.

Da ich häufiger in diese Situation komme, habe ich mir ein einfaches Werkzeug gebaut, dass das Problem etwas abschwächt: Ich verwende eine weitere Shell-Funktion, welche ich q (für "Quoting") nenne:

# Bash
q() { echo "${@@Q}"; }
# Z shell
q() { echo "${(q)@}"; }

Diese Funktion kann auch wieder in ~/.bashrc bzw. ~/.zshrc abgelegt werden.

Die Funktion verwendet ein Feature der jeweiligen Shell um Strings automatisch zu quoten (Bash Manual, Shell Parameter Expansion bzw. Z shell Manual, Parameter Expansion Flags). Der Q-Operator macht sozusagen das Entfernen von Anführungszeichen rückgängig, das beim Parsen durch die Shell passiert.

Um die Funktion anzuwenden, muss einfach das ganze Kommando, welches an ssh übergeben wird, in $(q ...) eingepackt werden. Keine zusätzlichen Anführungszeichen sind notwendig:

$ ssh myserver show-args $(q grep -RF 'case "$1" in' /etc/init.d)
     1	grep
     2	-RF
     3	case "$1" in
     4	/etc/init.d

$ ssh myserver $(q grep -RF 'case "$1" in' /etc/init.d)
/etc/init.d/udev:case "$1" in
/etc/init.d/nullmailer:case "$1" in
/etc/init.d/dbus:case "$1" in
/etc/init.d/haveged:case "$1" in
...

Ein Anwendungsfall, bei dem dies sehr praktisch sein kann, ist, wenn in das Kommando Variablen eingesetzt werden sollen:

$ for i in '##' '$1' 'a > b'; do
>   ssh myserver $(q echo "\$i = $i");
> done
$i = ##
$i = $1
$i = a > b

Viele weitere Anwendungen sind möglich, z.B. kann der Output von q in eine Datei geschrieben werden, welche später ausgeführt werden kann.

But why? There are already many free AI chatbots available…

With so many AI chatbots available for free, why go through the trouble of setting up a self-hosted one, beside technical curiosity? The answer lies in the often repeated and always neglected concerns about privacy, data protection, and trust:

• Privacy & data protection: Your chat conversations are often logged, analyzed, and stored. Even if providers claim to anonymize data, you can never be certain how your information is being used.
• Company secrets: Using an external AI chatbot risks exposing trade secrets, internal strategies, or confidential client information.
• If a service is free, you are the product: The meanwhile worn-out phrase still holds true.

Evaluation

Disclaimer: I did not spend too much time for the evaluation of the stack and got a lot of suggestions of my teammate Alain, which already has some more experience with LLMs (large language model).

TLDR:

• Web UI: Open WebUI
• LLM Management: Ollama
• LLM: DeepSeek-R1 70B

Web-Based Chat Interface

In my short research, I found two very promising Open-source tools as a AI chatbot Web UI:

• Open WebUI: Simpler, 2 Docker services
• LibreChat: More advanced, ~5 Docker services

I just went for to seemingly more light-weight solution, which is Open WebUI.

LLM Management Tool

Because Ollama is a really easy to use solution for managing LLMs and is natively supported by Open WebUI (no additional configuration needed) and also installs all necessary drivers and dependencies, I just went with this tool.

Large Language Model

Here comes the tricky part, selecting the right model. To ensure maximum performance, it needs to fit into the GPUs VRAM. The NVIDIA L40S GPUs, provided at cloudscale, has 48 GB of GDDR6 VRAM. Even though it is possible to use multiple L40S GPUs to run even larger models, I wanted to use a single GPU for this setup. The VRAM requirements can be found on the Ollama website or other model repositories. The DeepSeek-R1 70B model requires approximately 41 GB of VRAM, making it a great fit for this GPU.

Instructions

Setting up a GPU server on cloudscale

GPU servers are subject to Addendum for GPU servers / Vertragszusatz für GPU-Server. If you are interested or have any questions, please contact support.

I just created a GPU VM via the cloudscale control panel with the following specifications:

• Flavor: GPU1-160-20-1-400 (could also be a GPU flavor with less RAM or fewer CPU cores, the GPU is what matters)
• GPU Type: 1x NVIDIA L40S
• Scratch Disk: 400 GB on RAID 1 (this will be handy for persisting the LLM)
• Source Image: Debian 12 - Bookworm

If you want to follow this guide, I strongly recommend using Debian as the base image, as it ensures that all the steps will work as expected.

Mount the Scratch Disk

Even though we can load models up to 48 GB directly into the NVIDIA L40S's VRAM, we need to download and persist the models before we can run them. For this reason, at cloudscale, every GPU server comes equipped with an additional, local Scratch Disk. The Scratch Disk is tied directly to the server, but offers better performance than our usual volumes. Like other volumes, we have to mount it in our VM first:

# List all disks
lsblk -l

# Create a new folder for the mount
sudo mkdir -p /mnt/scratch

# Identify and mount Scratch Disk, e.g.: /dev/sdb
sudo mount /dev/sdb /mnt/scratch

# Get the device's UUID (note it somewhere or copy to clipboard)
sudo blkid /dev/sdb

# Persist the Scratch Disk mount
sudo vim /etc/fstab

# Add the following line
UUID=<device-uuid> /mnt/scratch ext4 defaults 0 0

Later, we will configure Ollama to use the Scratch Disk to download and persist the models.

Manually install NVIDIA drivers (optional)

This step can be skipped entirely, because the Ollama install script will install all necessary dependencies automatically. Follow this guide, if you are interested in how to manually install the NVIDIA GPU drivers on a VM.

In the following section, I will give you a step-by-step guide for installing the necessary NVIDIA GPU driver on Debian 12 - Bookworm.

# SSH into the rebooted server
ssh debian@<public-ip>

# Upgrade Debian to the latest version
# In the presented dialog, you can just confirm the default setting
sudo apt update && sudo apt upgrade -y

# Edit the sources list to enable non-free software (e.g. NVIDIA Drivers)
sudo vim /etc/apt/sources.list

The file should be updated as follows:

# See /etc/apt/sources.list.d/debian.sources
deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
deb http://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware

# Update the package list
sudo apt update

# Install the NVIDIA driver package
# Multiple dialogs will pop up, I just confirmed the default settings as they seemed reasonable enough
sudo apt install nvidia-driver

# Reboot VM as recommended
sudo reboot

# SSH into the rebooted server
ssh debian@<public-ip>

# Verify that the NVIDIA GPU drivers are installed and the GPU is correctly recognized by the VM
nvidia-smi

Install Ollama

# SSH into the server
ssh debian@<public-ip>

# Download and execute the Ollama install script
curl -fsSL https://ollama.com/install.sh | sh

# Reboot VM as recommended
sudo reboot

# SSH into the rebooted server
ssh debian@<public-ip>

# Verify that the NVIDIA GPU drivers are installed and the GPU is correctly recognized by the VM
nvidia-smi

# Verify that Ollama is installed successfully
ollama -v
sudo systemctl status ollama

# Configure Ollama to use the mounted Scratch Disk instead of the root volume
sudo mkdir -p /mnt/scratch/ollama_models
sudo chown ollama:ollama /mnt/scratch/ollama_models/

# Edit the ollama service configuration
sudo vim /etc/systemd/system/ollama.service

# Add the following line as the last one in the [Service] section
Environment="OLLAMA_MODELS=/mnt/scratch/ollama_models"

# Reload the ollama service
sudo systemctl daemon-reload
sudo systemctl restart ollama

# Download a small model to verify that it's stored on the Scratch Disk, the directory should not be empty any more
ollama pull smollm
ls /mnt/scratch/ollama_models/

# Cleanup the unused model
ollama rm smollm

# Install and run DeepSeek's R1
ollama run deepseek-r1:70b

# You can now test the LLM via CLI, exit with ctrl+d

Secure your service (optional)

If you plan to keep the service online, it's strongly advised to follow this section, or to implement alternative measures to protect it from unwanted access. This setup requires that you have a domain name.

Install nginx with certbot

In this guide, I will install and configure the web server nginx with certbot, to restrict the access and enable HTTPS for Open WebUI. Before your start, make sure that your domain's or subdomain's A and AAAA records are pointing to the GPU server's public IPv4 and IPv6 addresses respectively.

# Install nginx, certbot and certbot nginx plugin
sudo apt install nginx certbot python3-certbot-nginx

# Verify installation
sudo nginx -v

# Edit the nginx default configuration
sudo vim /etc/nginx/sites-available/default

# Replace <your-ip-v4> and <your-ip-v6> with the addresses you are using to access the internet.
# If you don't want to protect your service from unwanted access or you have other measures in place (e.g.: HTTP BasicAuth), the following three lines can be removed.
allow <your-ip-v4>;
allow <your-ip-v6>;
deny all;

upstream open_webui {
    # We will configure Open WebIU to listen on this port
    server 127.0.0.1:8080;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    root /var/www/html;

    index index.html;

    # Replace <your-domain> with the domamin where you configured the A and AAAA records, the certbot plugin will also use this to automatically extend this file with the HTTPS configuration
    server_name <your-domain>;

    # The proxy configuration is taken from: https://docs.openwebui.com/tutorials/https-nginx/#steps
    location / {
            proxy_pass http://open_webui;

            # Add WebSocket support (Necessary for version 0.5.0 and up)
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # (Optional) Disable proxy buffering for better streaming response from models
            proxy_buffering off;

            # (Optional) Increase max request size for large attachments and long audio messages
            client_max_body_size 20M;
            proxy_read_timeout 10m;
        }
}

# Verify that the nginx config is valid
sudo nginx -t

# Reload configuration
sudo nginx -s reload

# Setup certbot
sudo certbot --nginx -d <your-domain>

# Check the changes made by certbot:
# There should be new entries 'managed by certbot'
cat /etc/nginx/sites-available/default

Now you can open https://<your-domain>/ in a browser to verify if the TLS Certificate is correctly setup for your domain, and the page is served via HTTPS. The response will be "502 Bad Gateway", as the Open WebUI service is not reachable yet. If you receive a "403 Forbidden", this probably means you have to change the nginx configuration.

Install UFW (Uncomplicated Firewall)

# Install UFW
sudo apt install ufw

# Allow egress (we will monitor egress with OpenSnitch)
sudo ufw default allow outgoing

# Block ingress
sudo ufw default deny incoming

# Allow services, if you forget to allow SSH you will be blocked from accessing the VM!
sudo ufw allow ssh
sudo ufw allow vnc
sudo ufw allow http
sudo ufw allow https

# Enable UFW
sudo ufw enable

# Check status
sudo ufw status

# Verify that apt and certbot still can do their job
sudo apt update
sudo certbot renew --dry-run

Install Open WebUI

In this section, I will install Open WebUI into a Python Virtualenv, but there are other installation methods (e.g.: with Docker).

# Install dependencies
sudo apt install python3 python3-venv

# Verify python version is 3.11 to avoid compatibility issues
python3 --version

# Create a new user for running Open WebUI
sudo useradd -m open_webui

# Change directory and switch to user open_webui
cd /home/open_webui
sudo su open_webui

# Create a new directory and add a virtual environment
python3 -m venv venv

# Install Open WebUI in the virtual environment
venv/bin/pip install open-webui

# Start Open WebUI via the virtual environment, the default port is 8080
venv/bin/open-webui serve

You can now verify that your own Open WebUI is running, and create your admin account:

• If you skipped the "Secure your service" section: http://<public-ip>:8080
• Otherwise: https://<your-domain>

Now let's create a SystemD service, which runs Open WebUI in the background:

# Ctrl+c to stop Open WebUI and exit from the user open_webui
exit

# Create a new systemd service configuration
sudo vim /etc/systemd/system/open-webui.service

[Unit]
Description=Open WebUI Service
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/open_webui
ExecStart=/home/open_webui/venv/bin/open-webui serve
KillSignal=SIGTERM
KillMode=mixed
Restart=always
RestartSec=3
StandardOutput=syslog
StandardError=syslog
User=open_webui
Group=open_webui

[Install]
WantedBy=multi-user.target

# Reload systemd
sudo systemctl daemon-reload

# Enable and start open-webui service
sudo systemctl enable open-webui.service
sudo systemctl start open-webui.service

# Check for any errors
sudo systemctl status open-webui.service

In collaboration with Puzzle ITC's Yannik Dällenbach, I have recently worked to publish fleeting-plugin-cloudscale. With this plugin it is possible to dynamically scale Gitlab runner instances on cloudscale.ch.

While the bulk of the work is all thanks to Yannik's efforts, I had the chance to integrate his work into our open source landscape. This is what I learned.

GitLab Fleeting Works Well

Originally, GitLab used the Docker machine driver, to offer autoscaled runner instances. This tool has been deprecated for years. The alternative takes the form of a Go plugin.

With only a few short functions, Yannik was able to implement the necessary logic to launch, list, and cleanup runner instances. It's easy to write such a framework in a way that makes things cumbersome. GitLab has avoided that pitfall.

The GitLab runner process uses the plugin to react to tasks popping up in the CI pipeline, ensuring there is always enough capacity when it is actually needed.

Fleeting Plugin Packaging Is Different

I figured that publishing this plugin would involve a container image. This turned out to be right, but how GitLab does it was a bit unexpected:

Using a bespoke fleeting-artifact command, we build a container image, given a set of architecture-specific binaries. Go's cross-compilation story is top-notch, so providing these binaries is easy, but it is unexpected to not use generic container image build tools.

I think the approach has its merits, but I might have saved some time, had I known about it sooner. I only realized my container images did not work, after I built them using ko.

Go 1.24 Tools

This was my first project where I got to try out Go 1.24's go tool command. It is used to integrate tools related to the development of the project into dependency tracking. In our case, this enabled me to integrate go-releaser and the mentioned fleeting-artifact.

To install these tools, I ran the following:

go get -tool -modfile tool.mod github.com/goreleaser/goreleaser/v2@latest
go get -tool -modfile tool.mod gitlab.com/gitlab-org/fleeting/fleeting-artifact/cmd/fleeting-artifact
go mod tidy -modfile tool.mod

This makes these tools available as follows:

go tool -modfile tool.mod fleeting-artifact
go tool -modfile tool.mod goreleaser

What I like about this approach, though it is a bit verbose, is the fact that I can run the exact same tool locally, and in the CI, and across all our workstations.

Dev-tooling should be versioned as well, to keep its results stable, and go tool accomplishes that.

Zizmor

I like linters, static analyzers, language-servers, and so on. Anything that supplements my knowledge with community-wisdom. I didn't know about Zizmor before, but I rarely write GitHub workflows, and I figured: Someone must have written a validator for these.

Indeed someone has: Zizmors goal is to protect the user from the following (and more):

• Template injection vulnerabilities, leading to attacker-controlled code execution.
• Accidental credential persistence and leakage.
• Excessive permission scopes and credential grants to runners.
• Impostor commits and confusable git references.

It's not precisely a validator, or at least I did not come to rely on it as such, but it pointed out some potential security issues with my GitHub workflows, and I learned some newer directives I missed before.

As a result: The GitHub workflows in-place for fleeting-cloudscale-plugin are now hardened and checked for issues on each commit.

Conclusion

Adopting and packaging someone else's hard work was a somewhat new experience for me. With most of the groundwork already laid out, I was able to spend more time on packaging and testing, which the final result reflects I think.

If you want to check out our plugin, see:

https://github.com/cloudscale-ch/fleeting-plugin-cloudscale

If you want to demo it, use our Ansible playbook that configures a GitLab instance, adds a Gitlab runner, the fleeting-plugin-cloudscale plugin, and optionally a distributed S3 cache, all in one go:

https://github.com/cloudscale-ch/gitlab-runner

You'll be presented with a GitLab instance where CI jobs automatically work, scale, and share their cache. It's a great demo, and a good starting point to introduce GitLab into your own organization. Everything inside our infrastructure, far away from Five Eyes.

Cloudscale offers S3-compatible Object Storage built on top of our Ceph storage cluster with three-fold replication. To provide the S3 service, we use Ceph's RADOS Gateway (radosgw). While radosgw includes built-in usage tracking, we found its metrics insufficient for the needs of a public cloud provider like us.

In the Objects tab in our Control Panel, customers can view their exact usage over time and as an example see the number of requests on a specific date. This detailed data is not just for customer insights, it is also critical for accurate billing. Beside the number of requests, the metrics include the number of objects, the used storage and the network traffic.

To bridge the gap in capabilities, we developed our own solution: rgw-metrics.

What is rgw-metrics?

rgw-metrics is a microservice that repeatedly collects the current usage data for every bucket from radosgw. This data is aggregated into the current hourly segments, which are persisted. This usage data is then queried by the Control Panel through an API provided by rgw-metrics. This API is quite narrow and was stable over the years. It only allows to fetch metrics for a single or for multiple object users.

┌────────────────┐       ┌───────────────┐       ┌─────────────────┐
│  Contol Panel  ├──────►│  rgw-metrics  ├──────►│  RADOS Gateway  │
└────────────────┘       └───────────────┘       └─────────────────┘

Designed as a standalone microservice, running on both of our sites, means it operates independently of the Control Panel. This independence ensures metrics are consistently collected, even during extended maintenance periods.

A journey of evolution

The first version of rgw-metrics was written in Flask back in 2017 when we first introduced our S3 storage. While functional, the application had received little maintenance since its launch. Over time, this led to challenges, the outdated dependencies, manual deployment steps and the fact that the Control Panel is build with a different framework, Django, made engineers cautious about touching the application.

To address these issues, we decided on a black-box rewrite of rgw-metrics, transitioning it from Flask to Django.

The black-box rewrite approach

To ensure a seamless transition, we prioritized maintaining the existing API's behavior. That way we were able to create a collection of tests to validate the new service against the existing one. For instance, we compared the historical usage data from our public acceptance tests over the past year. Together with countless other internal projects using the Object Storage. During the development, we ran a script to compare the output of the new Django-based service with the original Flask-based implementation. This ensured the output of the new service matched the old one under various scenarios.

# essentially, it was automating these steps:
curl -H "$AUTH_HEADER" "https://old-api.cloudscale.ch/v1/metrics/buckets?start=2023-12-31&end=2024-01-01" > "export_flask/metrics.json"
curl -H "$AUTH_HEADER" "https://api.cloudscale.ch/v1/metrics/buckets?start=2023-12-31&end=2024-01-01" > "metrics.json"
diff export_flask/metrics.json metrics.json

Thanks to this test-driven method we acutely found multiple bugs, including one in our data migration scripts. An existing column was copied to the wrong target column.

What is up next for rgw-metrics?

With the rewrite complete, rgw-metrics now benefits from up-to-date dependencies, a container based deployment, similar to our main application, and a similar structure, which will help us develop additional features.

With the foundation strengthened, we are ready to tackle upcoming improvements like the efficient detection of large buckets: Each bucket has a limit of 10 million objects. Beyond this threshold, performance may degrade. Currently, we proactively contact users approaching this limit. However, gathering the necessary data through the current endpoints is suboptimal, as it requires iterating over every bucket for each object user. The amount of object users is growing every day, this forces us to extend the API to allow for more efficient queries on large buckets, reducing overhead and improving responsiveness.

Stay tuned as we continue to enhance our metrics system and provide an even better Object Storage experience for our users.

One of the great things about working as an engineer at cloudscale is that we get to work on many different products, technologies, and projects. When we began developing the load balancer as a service product, a crucial requirement was that it must be usable for creating highly available Kubernetes control planes. During the whole development process, I was looking forward to bootstrapping my first cluster using this new product. And now, that we have this blog, I want to share a few notes on how to create a highly available, stacked, Kubernetes control plane on three cloudscale Ubuntu VMs using containerd.

Provisioning the Cloud Infrastructure

The Kubernetes Documentation instructs us to set up the following:

In a cloud environment you should place your control plane nodes behind a TCP forwarding load balancer. This load balancer distributes traffic to all healthy control plane nodes in its target list. The health check for an apiserver is a TCP check on the port the kube-apiserver listens on (default value :6443).

This means that we'll need:

• A Load Balancer
• A Load Balancer Listener using port 6443
• A Load Balancer Pool with round-robin algorithm
• A Load Balancer Pool Member for each VM
• A Load Balancer Health Monitor checking Port 6443 on the VMs

Since the easiest way to set all this up and get all the VMs running is with Terraform, I have provided a Terraform file (see appendix) if you want a quick start. The Terraform file setups of the following:

Ensure Terraform is installed on your machine. Then, navigate to the Terraform file’s directory and run it.

terraform init

Once initialized, export a read/write API token from a, preferably, empty project and create the infrastructure by running:

export CLOUDSCALE_API_TOKEN="..."
terraform apply

Terraform will display a preview of the resources it plans to create or update and prompt you for confirmation. Type yes to proceed.

Terraform will also output three variables at the end: kube_api_lb_ip, server_ips_private, and server_ips_public. We'll need this information later on. Ensure that you can SSH into all VMs using the ubuntu user using the public IP addresses.

Installing kubeadm and containerd

Now, fasten your seatbelt. Here is a condensed summary of the following articles from the Kubernetes Documentation: Installing kubeadm, Container Runtimes, Creating Highly Available Clusters with kubeadm. I take some shortcuts and have left out some things I deem not relevant for non-production setups.

All commands must be run on all nodes.

Configure Kubernetes’ apt repository. Replace 1.32 with the desired Kubernetes version.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.32/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Download the necessary packages.

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Enable IP forwarding.

echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/k8s.conf
sudo sysctl --system

Install containerd and configure the systemd cgroup driver for runc.

sudo apt install -y containerd
sudo mkdir /etc/containerd
containerd config default | sed 's/SystemdCgroup = false/SystemdCgroup = true/' | sudo tee /etc/containerd/config.toml
sudo systemctl restart containerd

Initializing the Cluster and Installing a CNI Plugin

Next, we'll initialize the cluster from control-node-1 and install Cilium as a CNI (Container Network Interface) plugin. In the kubeadm init command, pass the IPv4 IP address of the load balancer as --control-plane-endpoint (shown as kube_api_lb_ip in terraform show) and the private IP address of node 1 as --apiserver-advertise-address.

sudo kubeadm init --control-plane-endpoint "KUBE_API_LB_IP:6443" --apiserver-advertise-address="10.11.12.21" --upload-certs

Next, set up your $HOME/.kube/config file as shown in the output of kubeadm init and keep the join commands in a safe place.

At this point, I usually check the nodes and pods in my cluster to see if everything looks good. It's perfectly normal that the node is NotYetReady and that coredns pods are not yet running. But the other pods should be running.

kubectl get nodes -o wide
kubectl get pods -A -o wide

In our experience, Cilium is the most worry-free CNI plugin to install, so let's do that and admire some colorful ASCII art until it is ready.

wget https://github.com/cilium/cilium-cli/releases/download/v0.16.22/cilium-linux-amd64.tar.gz
tar xvf cilium-linux-amd64.tar.gz
./cilium install
./cilium status --wait

Now the node should be ready and coredns pods should also come up within a short amount of time.

Joining the Other Nodes

Now join the other two nodes into the cluster using kubeadm. The command was shown in the kubeadm join output. Be sure to add the --apiserver-advertise-address="10.11.12.2x" using the respective private IPs (.22 and .23, shown as server_ips_private in terraform show).

sudo kubeadm join <your-kube-api-lb-ip>:6443 --token <your-token> \
  --discovery-token-ca-cert-hash sha256:<your-ca-cert-hash> \
  --control-plane --certificate-key <your-certificate-key> \
  --apiserver-advertise-address="10.11.12.<your-server-private-ip>"

After a certain period, all nodes listed in kubectl get nodes -o wide should be marked as Ready. And the coredns pods become running.

Seeing It in Action: Shutting Down a Node

Now, this blog post would, of course, not be complete without proofing that we can take down a control node. I suggest that you copy the $HOME/.kube/config to your local machine and use kubectl from there.

Let’s list all pods in the cluster. Pay attention to the coredns pods. They’re most likely scheduled on control-node-1.

kubectl get pods -A -o wide

Now drain control-node-1:

kubectl drain control-node-1 --ignore-daemonsets

After a few seconds, the coredns pods should have been successfully moved to the other control nodes.

kubectl get pods -A -o wide

It’s now safe to shut down the drained node.

sudo init 0

If everything worked, well, kubectl still works like a charm from your local machine and the other two control nodes. Another fun thing to do is to navigate to the private network named backend in the cloudscale Control Panel and look at the Ports tab. There, you'll see the network ports of the load balancer and the VMs. control-node-1 should be shown as down. The same applies to the "monitor_status" property if you query the pool members using our API.

After restarting the node, make it schedulable again.

kubectl uncordon control-node-1

And now you are ready to add worker nodes to the cluster. Or installing our Cloud Controller Manager (CCM) to configure a Load Balancer for managing external traffic, or setting up our Container Storage Interface (CSI) driver for persistent storage.

Lessons Learned and Final Words

In my initial test cluster, I made a mistake by not adding load balancer pool members for control-node-2 and control-node-3. As a result, when I shut down control-node-1, everything stopped working. So once, again I was reminded of: HA systems are worthless if failover testing is not done.

I hope this guide was interesting to you. If you find this type of content valuable, please send us an email because this could be the beginning of a small miniseries on running Kubernetes on cloudscale Infrastructure. As mentioned earlier, there’s much more to cover.

Have fun experimenting with Kubernetes on our infrastructure, but please read the complete documentation linked above before deploying production workloads!

Appendix: Terraform File

terraform {
  required_providers {
    cloudscale = {
      source  = "cloudscale-ch/cloudscale"
      version = "4.4.0"
    }
  }
}

provider "cloudscale" {
  # Add your provider configuration here if necessary
}

variable "control_node_count" {
  description = "Number of control nodes"
  type        = number
  default     = 3
}

variable "network_cidr" {
  description = "CIDR block for the backend network"
  type        = string
  default     = "10.11.12.0/24"
}

variable "zone_slug" {
  description = "Zone slug for the resources"
  type        = string
  default     = "lpg1"
}

variable "ssh_key_path" {
  description = "Path to the SSH public key file"
  type        = string
  default     = "~/.ssh/id_ed25519.pub" # Replace with your SSH key file path
}

# Create a network
resource "cloudscale_network" "backend" {
  name                    = "backend"
  zone_slug               = var.zone_slug
  auto_create_ipv4_subnet = "false"
}

# Create a subnet
resource "cloudscale_subnet" "backend-subnet" {
  cidr         = var.network_cidr
  network_uuid = cloudscale_network.backend.id
}

# Server Group for Control Nodes
resource "cloudscale_server_group" "control-plane-group" {
  name      = "control-plane-group"
  type      = "anti-affinity"
  zone_slug = var.zone_slug
}

# Control Nodes
resource "cloudscale_server" "control-nodes" {
  count            = var.control_node_count
  name             = "control-node-${count.index + 1}"
  flavor_slug      = "flex-8-4"
  image_slug       = "ubuntu-24.04"
  volume_size_gb   = 50
  ssh_keys         = [file(var.ssh_key_path)]
  server_group_ids = [cloudscale_server_group.control-plane-group.id]
  zone_slug        = var.zone_slug

  interfaces {
    type = "public"
  }

  interfaces {
    type = "private"
    addresses {
      subnet_uuid = cloudscale_subnet.backend-subnet.id
      address     = "10.11.12.${count.index + 21}"
    }
  }
}

# Kube-API Load Balancer
resource "cloudscale_load_balancer" "kube-api-lb" {
  name        = "kube-api-lb"
  flavor_slug = "lb-standard"
  zone_slug   = var.zone_slug
}


# Create a load balancer pool
resource "cloudscale_load_balancer_pool" "kube-api-pool" {
  name               = "kube-api-pool"
  algorithm          = "round_robin"
  protocol           = "tcp"
  load_balancer_uuid = cloudscale_load_balancer.kube-api-lb.id
}

# Create a load balancer listener
resource "cloudscale_load_balancer_listener" "kube-api-listener" {
  name          = "kube-api-listener"
  pool_uuid     = cloudscale_load_balancer_pool.kube-api-pool.id
  protocol      = "tcp"
  protocol_port = 6443
}


# Create a load balancer pool member
resource "cloudscale_load_balancer_pool_member" "kube-api-pool-member" {
  count         = var.control_node_count
  name          = "kube-api-${count.index}"
  pool_uuid     = cloudscale_load_balancer_pool.kube-api-pool.id
  protocol_port = 6443

  # Get the private IP address of the control node
  address = flatten([
    for iface in cloudscale_server.control-nodes[count.index].interfaces : [
      for addr in iface.addresses : addr.address
      if iface.type == "private"
    ]
  ])[0]
  subnet_uuid = cloudscale_subnet.backend-subnet.id
}

# Create a load balancer pool member
resource "cloudscale_load_balancer_health_monitor" "lb1-health-monitor" {
  pool_uuid = cloudscale_load_balancer_pool.kube-api-pool.id
  type      = "tcp"
}


output "kube_api_lb_ip" {
  value       = cloudscale_load_balancer.kube-api-lb.vip_addresses[0].address
  description = "IPv4 Address of the Load Balancer"
}

output "server_ips_public" {
  value = [
    for node in cloudscale_server.control-nodes :
    flatten([
      for iface in node.interfaces : [
        for addr in iface.addresses : addr.address
        if iface.type == "public"
      ]
    ])[0]
  ]
  description = "The public IP addresses of the control nodes."
}

output "server_ips_private" {
  value = [
    for node in cloudscale_server.control-nodes :
    flatten([
      for iface in node.interfaces : [
        for addr in iface.addresses : addr.address
        if iface.type == "private"
      ]
    ])[0]
  ]
  description = "The private IP addresses of the control nodes."
}

As a result, our playbooks require more and more knowledge about our environment, and the chance of "holding them wrong" increases.

How it Started

Like most Ansible users, we initially called all playbooks like this:

$ ansible-playbook playbooks/state/all.yml -i inventory/prod-rma1 -l www --diff

This works great, but it is more verbose than necessary. Some in our team, including - if not limited to - me, prefer short commands over long ones. Especially if they need to be used frequently.

So we took our existing playbooks like this one:

- name: Base network and boot setup
  hosts: '{{ playbook_hosts | default("ansible-managed") }}'

And added a shebang:

#!/usr/bin/env ansible-playbook
- name: Base network and boot setup
  hosts: '{{ playbook_hosts | default("ansible-managed") }}'

Then we made the file executable:

$ chmod +x playbooks/state/all.yml

And presto, we could drop the command name from our CLI, calling the same playbook as follows:

$ playbooks/state/all.yml -i inventory/prod-rma1 -l www

How it is Going

When integrating ARA logging, we discovered that operators would always have to use --diff / -D, for differences to actually be logged.

We figured that typing --diff is not something we wanted to add to our documentation. We wanted to enforce it. We also had some configuration we needed to apply, depending on the inventory selection.

Maybe there is another way, but we figured: Since we already use our playbooks somewhat like CLIs, why not wrap ansible-playbook and go all-in?

So that's what we did. We wrote our own Python CLI called ap, that would inspect the arguments destined for ansible-playbook, and introduce some of its own arguments for a better user experience.

The script uses Typer and looks roughly as follows (this is a minimized version to illustrate the concept):

import os
import subprocess
import sys

from typer import Context
from typer import Option
from typer import Typer
from typing import Annotated


cli = Typer(add_completion=False, add_help_option=False, no_args_is_help=True)


@cli.command(name='ap', context_settings={
    "allow_extra_args": True,
    "ignore_unknown_options": True
}, add_help_option=False, no_args_is_help=True)
def main(
    ctx: Context,
    inventory: Annotated[list[str], Option("--inventory", "-i", help=(
        "Inventories to use, each either a site or a full inventory path"
    ))] = [],
) -> None:
    args = ['ansible-playbook', *(a for a in ctx.args)]

    # Configure inventories
    for i in inventory:
        i = i.removeprefix('inventory/')

        args.append('-i')
        args.append(f'inventory/{i}')

    # Always use diff
    if '-D' not in args and '--diff' not in args:
        args.append('--diff')

    # Configure systems
    os.environ.update(ara_env(args))
    os.environ.update(netbox_env(args))
    os.environ.update(secrets_env(args))

    # Execute
    result = subprocess.run(args, env=os.environ)
    sys.exit(result.returncode)


if __name__ == '__main__':
    cli()

Aside from enforcing arguments, this also gave us the flexibility to shorten our inventory calls a little, as the inventory prefix is really always the same. So while we started with this:

$ ansible-playbook playbooks/state/all.yml -i inventory/prod-rma1 -l www --diff

We can now call this, which is equivalent:

$ playbooks/state/all.yml -i prod-rma1 -l www

And now we can use all these saved keystrokes on useful things, like our engineering blog!

Hier bei cloudscale arbeite ich hauptsächlich am Control-Panel, einer mittelgrossen Python/TypeScript-Applikation. Das Control-Panel ist die Applikation, welche unter control.cloudscale.ch erreichbar ist. Die Entwicklung begann vor 8 Jahren als Neu-Anfang und die Applikation wurde seither stetig weiterentwickelt und gepflegt.

In diesem Artikel konzentriere ich mich auf dem Python-Teil, welcher heute ~120 k Zeilen Code umfasst:

$ git ls-files '*.py' | wc -l
     674
$ git ls-files '*.py' | parallel -Xj1 cat | wc -l
  119248

Suchen von Code-Stellen

Wie bei jedem Software-Projekt, ist es bei der Arbeit am Control-Panel regelmässig notwendig, Code-Stellen, welche weit über die Applikation verteilt sind, zu finden oder anzupassen. Z.B. möchte ich herausfinden, ob eine interne API auf eine bestimmte Art verwendet wird oder ob diese überhaupt noch verwendet wird. Oder ich möchte prüfen, ob ein veraltetes oder problematisches Code-Muster, das ich entdeckt habe, noch an weiteren Stellen in der Applikation vorkommt. Das Ziel ist immer, den Code verständlicher und einfacher erweiterbar/wartbar zu machen.

Das erste Werkzeug, zu dem ich greife, ist "Suchen & Ersetzen" meiner IDE oder ein anderes, ähnlichen Werkzeug. Hier kann ich mit Regular Expressions mit wenig Aufwand nach Stellen im Code suchen. Dieser Ansatz ist sehr schnell und daher interaktiv. Die Geschwindigkeit von git grep wird wohl durch kein Werkzeug erreich, welches zuerst den Python-Source parsen müsste:

$ time git grep -P '\bsend_email\(' '*.py' > /dev/null

real	0m0.030s
user    0m0.014s
sys     0m0.098s

Aber Regular Expressions sind nicht geeignet, wenn komplexere zusammenhänge im Code erkennt werden müssen. Ein Beispiel wäre, alle Verwendungen einer Funktion zu finden, bei denen ein optionales Argument angegeben wird. Dafür gibt es besser geeignete Werkzeuge, welche allerdings auch mehr Vorbereitung bei der Anwendung benötigen. Im Folgenden Zeige ich eines der Werkzeuge, mit denen ich in den letzten Monaten viel gearbeitet habe.

pyastgrep

pyastgrep ist eine Library und eine CLI-Applikation, welche zum Durchsuchen von Python-Code anhand dessen AST (Abstract syntax tree) verwendet werden kann. pyastgrep stellt intern den Python-AST einer einzelnen Source-Datei oder eines ganzen Ordners als XML-Baum zur Verfügung. Diesen kann dann mit XPath-Ausdrücken durchsucht werden. Es ist dabei in jedem Fall sehr hilfreich, die Dokumentation des Python-Moduls ast und ein XPath Cheatsheet bereitzuhalten.

Als Beispiel zeige ich, wie ich alle Code-Stellen finden kann, bei denen die Funktion send_email() aufgerufen und ein Wert für das optionale Argument reply_to_address angegeben wird. Dazu baue ich schrittweise einen XPath-Ausdruck auf.

Im ersten Schritt selektiere ich alle Funktions-Aufrufe von Funktionen mit dem Namen send_email().

$ pyastgrep './/Call[func/Name[@id="send_email"]]' src
src/db/access/member_helper.py:57:9:        send_email(
src/services/openstack/functions.py:103:5:    send_email(

Call und Name sind die Knoten im Syntaxbaum, welche einen Funktionsaufruf respektive eine die Verwendung einer globalen oder lokalen Variable darstellen. Name[@id="send_email"] selektiert alle Variablen-Verwendungen von Variablen mit dem name send_email.

Soweit so gut! Ich weiss aber, dass weitaus mehr Code-Stellen diese Funktion aufrufen. Das Problem ist, dass die Funktion entweder als send_email() oder aber auch als email.send_email() aufgerufen werden kann. Da dies die einzige Funktion mit diesem Namen ist, kann ich etwas ungenau arbeiten, und alle Stellen selektieren, in denen auf einem beliebigen Objekt oder Modul eine Funktion mit diesem Namen aufgerufen wird:

$ pyastgrep './/Call[func[Name[@id="send_email"] or Attribute[@attr="send_email"]]]' src
src/panel/signals.py:18:13:            email.send_email(
src/panel/invoices/__init__.py:169:30:    to_address, cc_address = email.send_email(
src/panel/payment/__init__.py:46:9:        email.send_email(
src/panel/email/tests/test_template_rendering.py:15:5:    email.send_email(
src/panel/billing/notifications.py:28:30:    to_address, cc_address = email.send_email(
[... 10 weitere Resultate]

Attribute sind die Knoten, bei denen mit dem .-Operator auf ein Attribut eines anderen Objektes zugegriffen wird, wie z.B. in email.send_email. func[... or ...] selektiert die Funktionsaufrufe beider Varianten (lokale/globale Variable und Attribut).

Als Letztes schränke ich die Suche auf alle Stellen ein, an denen das Keyword-Only-Argument reply_to_address übergeben wird:

$ pyastgrep './/Call[func[Name[@id="send_email"] or Attribute[@attr="send_email"]] and keywords/keyword[@arg="reply_to_address"]]' src
src/panel/signals.py:18:13:            email.send_email(
src/panel/billing/notifications.py:51:5:    email.send_email(
src/project/tests/test_email_backend.py:30:5:    email.send_email(
src/db/access/member_helper.py:57:9:        send_email(
src/db/access/user/tickets.py:51:9:        email.send_email(
[... 5 weitere Resultate]

Call[... and ...] selektiert alle Funktionsaufrufe die beiden Bedingungen entsprechen (Funktionsname und Vorhandensein des Keyword-Arguments). keywords/keyword[...] iteriert über alle Keyword-Argumente des Funktionsaufrufs. @arg="reply_to_address" selektiert die Keyword-Argumente, die das Keyword reply_to_address verwenden (send_email(..., reply_to_address=...)).

Die Bedingung für das Keyword-Argument kann auch gut umgedreht werden. Als letztes Beispiel selektiere ich hier alle Aufrufe von send_email() bei denen das Argument reply_to_address nicht übergeben wird:

$ pyastgrep './/Call[func[Name[@id="send_email"] or Attribute[@attr="send_email"]] and not(keywords/keyword[@arg="reply_to_address"])]' src
src/panel/invoices/__init__.py:169:30:    to_address, cc_address = email.send_email(
src/panel/payment/__init__.py:46:9:        email.send_email(
src/panel/email/tests/test_template_rendering.py:15:5:    email.send_email(
src/panel/billing/notifications.py:28:30:    to_address, cc_address = email.send_email(
src/services/openstack/functions.py:103:5:    send_email(

Genauigkeit ist immer eine Abwägung

Als Abschluss möchte ich anmerken, dass das oben gezeigte Beispiel aus verschiedenen Gründen falsche Resultate liefern kann, also zu viele oder zu wenige. Jede dieser Abweichungen kann begegnet werden, mit jeweils unterschiedlichem Aufwand und nicht in jedem Fall perfekt. Dies sind ein paar Beispiele für Ungenauigkeiten, die ich im Beispiel oben zugelassen habe:

• Es könnte neben der gesuchten Funktion weitere Funktionen mit dem Namen send_mail geben. Um dem zu entgegnen, müssten die import-Anweisungen in jeder Source-Datei sowie das Vorhandensein von lokalen Variablen analysiert werden.
• Die Funktion send_mail könnte in einer Datei unter einem anderen Namen importiert worden sein, z.B. mit from panel.email import send_email as send_email_, vielleicht um einem Namenskonflikt aus dem Weg zu gehen. Auch hierfür müssten die import-Anweisungen analysiert werden.
• Das Argument reply_to_address könnte als Positional-Argument übergeben werden. In dem Fall müsste das Argument anhand der Position in der Argumentliste statt des Keywords reply_to_address selektiert werden.
• Das Argument reply_to_address könnte dynamisch via **kwargs übergeben werden. Dieser Fall ist sehr schwierig automatisiert vollständig zu erkennen. In unserem Fall wäre es am effektivsten gewesen, die Stellen, an denen **kwargs verwendet wird, automatisiert zu finden und diese dann manuell zu prüfen.

In den meisten Fällen gibt es keine perfekte Lösung, oder der Aufwand dafür ist grösser als der Nutzen. In diesen Fällen ist man gezwungen, eine Abwägung zwischen Genauigkeit, Flexibilität und Aufwand zu machen. Je grösser eine Applikation wird, je mehr gewinnen in meiner Erfahrung Genauigkeit und Flexibilität an Gewicht.

My onboarding started with a one on one session with the Team Lead and included a mix of setup activities:

• Unboxing my personal hardware, basic MacBook and Backup setup
• Create accounts for internal systems like LDAP, VPN, SSH, etc., according to the password policy
• Reading and signing papers and more

While commuting from Biel to Zürich and in the home office I could individualize my setup and familiarize myself with company tools and workflows. I also got time to start working through the cloud exercises available on GitHub, which gave me a better understanding how the API is working and how the customers are interacting with our system.

Let the Fridge Settle

Once I had everything set up, I was introduced to the software projects I would be working on at cloudscale. In one of the next daily standups I was also assigned my first small task: Include Server Name in Extra Traffic Transaction Description. Soon after, I was introduced into the code review / QA process, which enabled me to review and test the work of my teammates.

One by one I attended the companies different meeting formats:

• One-on-One: Weekly retrospective with my team lead
• Sprint planing: Every two weeks the team discusses and decides the scope of the next sprint
• Dev Team Insights: Each month our team presents an interesting insight, the last one was about how we use end-to-end testing in current projects
• All-Hands Retrospective: A workshop where the whole company sits together to identify road blocks and proposes solutions for resolving them
• Brownbag: A voluntary format in which an employee presents a topic (e.g.: Cloud Native Days), which usually takes place over lunchtime, hence the name

Set the Temperature

A key experience for understanding the company was the introduction day with Mänu, the CEO of cloudscale. We took a dive into the company’s history, structure, and where we are located in the market and the cloud pyramid, covering everything from our server centers to the networks and topologies that support our infrastructure. Alongside historical and technical insights, I learned about cloudscale's core values:

• Quality - go the extra mile
• 360° Transparency - Internally and externally
• Privacy / Security - From software to communication channels
• Swissness - Location, reliability and Secrecy
• Simplicity / Approachability - Being on eye level with customers, business partners and coworkers

Load Beverages Gradually

To not overload a fridge, the beverages should not be filled all at once. Analogous, I was gradually introduced in further important topics and given more responsibility/autonomy:

• Further sessions about implementation details in our projects
• Working on bigger tasks
• Writing this blog post
• Information Security Management System (ISMS) and ISO/IEC 27002
• Introduction to the system engineering team and which technologies they are using

Once the above topics are cooled down, the fridge can be loaded further:

• Visiting the server centers
• Introduction to Support
• Holding the next Dev Team Insights Presentation
• Organizing All-Hands Retrospective Meeting

Enjoy a cold one

For me, switching Jobs triggered some insecurities, but thanks to a well-structured onboarding process and a very supportive team, the transitioning to cloudscale has been a smooth experience. Cheers!

Disclaimer: We just got a shiny new fridge with tasty beverages. I was not involved in the actual fridge project, and it was filled by the CEO himself.

When customers use a disk in our cloud, they talk to one of our Ceph storage clusters. Every byte written is sent, every byte read received from one. The underlying physical hardware is abstracted away, shielding the VMs from unexpected disk failures and planned maintenance procedures.

To manage our clusters, we sometimes have to restart their services. Here's how we do that while minimizing customer impact.

Impact of Restarts

When a single disk dies, it typically takes one or two OSDs with it. This can rarely be detected by our customers - after all, this is what Ceph is all about.

However, restarting a lot of OSDs concurrently causes throughput drops and latency spikes. A drop in throughput can be noticed because a PostgreSQL server might suddenly be much slower in scanning its tables, a drop in latency is especially noticeable in clusters like Etcd, where high latency might cause leader elections.

💡 Tip

If you use Etcd in the cloud, tuning its time parameters may help avoid unnecessary leader elections:

• https://etcd.io/docs/v3.4/tuning/
• https://www.redhat.com/en/blog/introducing-selectable-profiles-for-etcd

If we restart as many OSDs as possible simultaneously (a third of a cluster), the impact on customer VMs is quite visible in benchmarks:

If we manually stagger the OSD restarts five seconds apart, we get more spread out numbers, especially when it comes to latency:

The effect on throughput is not too dramatic, but that's a function of the cluster size. On our smaller, internal clusters, the effect is more pronounced:

By staggering starts, we spread out the negative effects:

Note that staggering stops has no positive impact. Ceph uses kill -9 on its OSDs when stopping them and it is designed to deal well with suddenly disappearing OSDs.

Automating Staggered Restarts

While we are able to manually start OSDs in a staggered fashion, we have situations where we cannot do that. We want staggered OSD starts when a host unexpectedly reboots, when it gets reinstalled, when we do a package upgrade and so on. Ideally, we don't want to have to think about it.

To achieve this, we wrote a Python script that uses cluster-wide locking to ensure that OSDs are started slightly apart:

#!/usr/bin/env python3
"""

About
-----

A set of commands to force OSDs to start in a staggered fashion, avoiding
excessive peering load on the cluster.

Implementation
--------------

Uses an exclusive lock on an image in RADOS:

1. The script starts two processes: a main and a detached lock process.
2. The main process waits for the lock process to acquire a lock.
3. The main process calls `exec` on the additional commandline arguments.
4. The lock process runs until the lock expires.

Signals
-------

Signals are handled separately by the lock and the exec process.

It is possible to use a signal to kill the exec process before it runs the
command. In this case the command will not be executed. Once the command is
executing, signal handling is up to that command.

The lock process can also be killed, which causes it to release the lock
early.

Short-Lived Commands
--------------------

This command is meant for OSDs that may be up for weeks or months. If used
on a short-lived command (or an OSD that exits immediately), the lock is
not held for the whole duration.

The lock process detects if the parent pid has exited. In this case, the
lock is released early and the lock process exits.

This behavior relies on the fact that the exec process's pid is not used by
another process. This is generally no problem with short lock durations,
and a normal amount of processes. Should the pid be reused for some reason,
the effect will be a lock that is held for too long.

To avoid that race condition, we could use pidfd_open, but that is only
available in Python 3.9.

Requirements
------------

To run, this command expects the following packages to be present:

- librados for python: https://docs.ceph.com/en/reef/rados/api/librados-intro
- click
- pydantic

The current Python target is Python 3.8.

"""
from __future__ import annotations

import click
import code
import json
import logging
import mmap
import os
import random
import secrets
import shlex
import signal
import socket
import sys
import time
import yaml

from configparser import ConfigParser
from contextlib import contextmanager
from datetime import datetime
from datetime import timedelta
from pydantic import BaseModel
from pydantic import Field
from pydantic.types import FilePath
from rados import Ioctx
from rados import ObjectBusy
from rados import ObjectNotFound
from rados import Rados
from threading import Event
from types import FrameType
from typing import cast
from typing import Iterator
from typing import List
from typing import Optional

# The default config path
DEFAULT_CONFIG = '/etc/stagger-osds.yml'

# The ID of the client
CLIENT = 'stagger-osds'

config_option = click.option(
    '--config',
    default=DEFAULT_CONFIG,
    type=click.Path(exists=True, dir_okay=False),
)

log = logging.getLogger(CLIENT)


@click.group()
def cli() -> None:
    pass


class Config(BaseModel):

    # The path to the global ceph config
    ceph_config: FilePath

    # Keyring to authenticate with
    keyring: FilePath

    # Name of the pool to use
    pool: str = Field(
        ...,
        min_length=3,

        # Mypy 1.7+ fails here, the correct solution would be to use
        # Annotated[str, Field(min_length=3, strip_whitespace=True], but that
        # is not supported on the proxmox host.
        # type: ignore[call-arg]
        strip_whitespace=True,
    )

    # Key of the object in the selected pool
    key: str = Field(
        ...,
        min_length=3,

        # Mypy 1.7+ fails here, the correct solution would be to use
        # Annotated[str, Field(min_length=3, strip_whitespace=True], but that
        # is not supported on the proxmox host.
        # type: ignore[call-arg]
        strip_whitespace=True,
    )

    # How long to maximally keep the lock active (in seconds)
    duration: int = Field(..., gt=0, lt=61)

    # How long to try and acquire a lock (in seconds)
    timeout: int = Field(..., gt=0, lt=3601)

    @classmethod
    def from_yaml(cls, path: str) -> Config:
        with open(path, 'r') as f:
            return cls.parse_obj(yaml.safe_load(f))

    def configure_logging(self, process: str, verbose: bool) -> None:
        level = verbose and logging.DEBUG or logging.WARNING
        fmt = f"{process} [%(levelname)s] %(message)s"

        logging.basicConfig(format=fmt, level=level)

    @contextmanager
    def cluster_connection(self) -> Iterator[Rados]:
        """ Yields a connection to the cluster, by using the global ceph
        config for mon discovery, the keyring for client authentication.

        """
        conf = ConfigParser()
        conf.read(self.ceph_config)
        conf = dict(conf["global"].items())

        client = f"client.{CLIENT}"
        conffile = str(self.keyring)

        with Rados(name=client, conf=conf, conffile=conffile) as cluster:
            yield cluster

    @contextmanager
    def ioctx(self) -> Iterator[Ioctx]:
        """ Yields an ioctx. It is imperative that this is done on a new
        cluster connection each time, as there's some internal state in regards
        to application metadata, causing values to be cached agressively.

        """
        with self.cluster_connection() as cluster:
            with cluster.open_ioctx(self.pool) as ioctx:
                yield ioctx

    def metadata_get(self, ioctx: Ioctx, key: str, default: str) -> str:
        """ Gets the given metadata key, or a default value. """

        try:
            return cast(str, ioctx.application_metadata_get(CLIENT, key))
        except KeyError:
            return default

    def metadata_set(self, ioctx: Ioctx, key: str, val: str) -> None:
        """ Sets the given metadata key. Strings longer than 100 characters
        are truncated, as they otherwise cause errors.

        """
        ioctx.application_metadata_set(CLIENT, key, val[:100])

    def metadata_remove(self, ioctx: Ioctx, key: str) -> None:
        """ Deletes the given metadata key (idempotent). """

        ioctx.application_metadata_remove(CLIENT, key)

    def disable(self, ioctx: Ioctx) -> None:
        """ Disables lock acquisition with immediate effect. The current lock
        is released, and all OSDs waiting for a lock are started. As long
        as osd-staggering is disabled, locking is skipped completely.

        """
        self.metadata_set(ioctx, 'disabled', '1')

    def enable(self, ioctx: Ioctx) -> None:
        """ Removes the 'disabled' state. """

        self.metadata_remove(ioctx, 'disabled')

    @property
    def is_disabled(self) -> bool:
        """ True if locking is disabled. Uses its own connection to the cluster
        as the cache might otherwise return outdated values.

        """

        with self.ioctx() as ioctx:
            return self.metadata_get(ioctx, 'disabled', '0') == '1'


class LockHolder(BaseModel):
    """ Metadata about the current lock holder. """

    # The hostname of the lock holder
    host: str

    # The command being executed as an argument list as expected by exec*
    command: List[str]

    # The PID of the exec process
    ppid: int

    # The time in UTC when this lock expires
    expires: datetime

    @property
    def is_expired(self) -> bool:
        return datetime.utcnow() > self.expires


class SharedBool:
    """ Anonymous memory mapped bool, that can be used by multiple processes
    at the same time. Currently used for a single-writer model - not sure if
    there are race-conditions in a multi-writer scenario.

    """

    def __init__(self, default: bool = False):
        self.mmap = mmap.mmap(-1, 1)
        self.store(default)

    def __bool__(self) -> bool:
        self.mmap.seek(0)
        return self.mmap.read(1) == b'1'

    def __repr__(self) -> str:
        return f"SharedBool({self and 'True' or 'False'})"

    def store(self, value: bool) -> None:
        self.mmap.seek(0)
        self.mmap.write(value and b'1' or b'0')


def signal_handler() -> Event:
    """ Provides signal handling for each process. Should be called after
    forking, as each process must have its own signal handling.

    """
    interrupt = Event()

    def quit(signum: int, frame: Optional[FrameType]) -> None:
        interrupt.set()

    for sig in (signal.SIGTERM, signal.SIGHUP, signal.SIGINT):
        signal.signal(sig, quit)

    return interrupt


def ensure_session_leader() -> None:
    """ Ensures that the current process is promoted session leader, if it
    is not one already.

    """

    if os.getpid() != os.getpgid(0):
        os.setsid()


def is_pid_running(pid: int) -> bool:
    """ Returns True if the given PID is of a running process. """

    try:
        os.kill(pid, 0)
    except OSError:
        return False
    else:
        return True


def wait_for_lock_then_hold(start_osd: SharedBool, ppid: int, config: Config,
                            command: List[str]) -> None:
    """ Tries to acquire a lock on the object with the given name, then holds
    it for the given duration.

    The lock is released early under the following circumstances:
        - The process with pid `ppid` no longer exists.
        - An interrupt is received.

    Meant to be run in a separate process.

    """

    # Ensure we get separate signal handling
    ensure_session_leader()
    interrupt = signal_handler()

    # Lock configuration
    lock_cfg = {
        # The key of the object that is locked
        'key': config.key,

        # The name of the lock on the object
        'name': CLIENT,

        # Unique string belonging to the owner of the lock. This string is
        # required to unlock the lock pre-maturely.
        'cookie': secrets.token_hex(8),
    }

    try:
        with config.ioctx() as ioctx:

            # Exit condition for the acquisition and hold loops
            def keep_running(until: float, warn_on_timeout: bool) -> bool:
                if time.monotonic() > until:
                    if warn_on_timeout:
                        log.warning("Timeout expired")
                    return False

                if interrupt.is_set():
                    log.info("Interrupt received")
                    return False

                if not is_pid_running(ppid):
                    log.info("Parent process exited")
                    return False

                if config.is_disabled:
                    log.info("Stagger disabled globally")
                    return False

                # Randomize the wait time slightly, to be extra sure that we
                # are not creating a thundering herd (not that that is
                # likely).
                return not interrupt.wait(random.uniform(0.9, 1.1))

            # Try to acquire the lock
            log.info(f"Acquiring lock (timeout {config.timeout}s)")
            until = time.monotonic() + config.timeout

            while keep_running(until, warn_on_timeout=True):
                try:
                    ioctx.lock_exclusive(**lock_cfg, duration=config.duration)
                    log.info("Acquired lock, cleared for launch")
                    start_osd.store(True)
                    break
                except ObjectBusy:
                    continue
                except Exception:
                    log.exception("Unexpected error while trying to get lock")
                    break

            if not start_osd:

                # If we cannot get a lock due to a timeout or an error, we
                # signal to the OSD to start anyway.
                log.warning("Could not acquire lock, cleared for launch")
                start_osd.store(True)
                return

            # Write some information about the current lock holder
            holder = LockHolder(
                host=socket.gethostname(),
                command=command,
                ppid=ppid,
                expires=datetime.utcnow() + timedelta(seconds=config.duration)
            )

            ioctx.write_full(config.key, holder.json().encode('utf-8'))

            # Keep the lock for the given duration
            try:
                log.info(f"Holding lock for up to {config.duration}s")

                until = time.monotonic() + config.duration
                while keep_running(until, warn_on_timeout=False):
                    pass

            finally:

                # The lock vanishes on its own, if the duration is up. Since
                # we might be too late, we can only try to unlock, but may
                # be too late.
                try:
                    ioctx.unlock(**lock_cfg)
                except ObjectNotFound:
                    pass

            # Release the lock
            log.info("Released lock")

    except Exception:

        # If there are *any* issues, we let the OSDs start.
        start_osd.store(True)
        log.exception(f"Could not connect to cluster using {config.keyring}")
        sys.exit(1)


def wait_for_lock(start_osd: SharedBool, timeout: int) -> bool:
    """ Waits for the `start_osd` variable to change to `True`, returning
    `True` if the OSD should be started, `False` otherwise.
    """

    # Ensure we get separate signal handling
    ensure_session_leader()
    interrupt = signal_handler()

    # Wait to acquire the lock
    log.info("Awaiting lock")

    # The same timeout that eventually abandons the lock, can be used here,
    # so we don't get stuck if the lock process dies.
    until = time.monotonic() + timeout
    while not (start_osd or interrupt.is_set()) and time.monotonic() < until:
        interrupt.wait(0.5)

    # If the interrupt has been received, stop:
    if interrupt.is_set():
        log.info("Interrupt received: Aborting")
        return False

    return True


@cli.command(context_settings={
    "allow_extra_args": True,
    "ignore_unknown_options": True
})
@click.pass_context
@config_option
@click.option('--verbose', is_flag=True, default=False)
def start(ctx: click.Context, config: str, verbose: bool) -> None:
    """ Wraps the given command with a cluster-wide lock, forcing the command
    to start at a staggered interval (spread out), even if run on multiple
    machines.

    """

    # Load config
    config = Config.from_yaml(config)

    # Load command
    command = ctx.args

    if not command:
        print("You must specify a command to run, after '--'", file=sys.stderr)
        sys.exit(1)

    # Shared lock
    start_osd = SharedBool(False)

    # The PID of the process destined to be the command
    ppid = os.getpid()

    # Fork a process that launches the lock
    if os.fork() == 0:

        # Fork the lock process
        if os.fork() == 0:
            config.configure_logging(process="lock-process", verbose=verbose)
            wait_for_lock_then_hold(
                start_osd=start_osd,
                ppid=ppid,
                command=command,
                config=config)

        # Launcher and lock process are both awaited by the init process
        sys.exit(0)

    # Wait for the process that launches the lock to exit (this is quick)
    os.wait()

    config.configure_logging(process="exec-process", verbose=verbose)
    if wait_for_lock(start_osd=start_osd, timeout=config.timeout):
        log.info(shlex.join(command))
        os.execvp(command[0], command)


@cli.command()
@config_option
def status(config: str) -> None:
    """ Show the current status. """

    config = Config.from_yaml(config)
    print("Status:", config.is_disabled and "Disabled" or "Enabled")

    with config.ioctx() as ioctx:
        try:
            holder = ioctx.read(config.key)
        except KeyError:
            holder = None
        else:
            holder = holder and LockHolder.parse_obj(json.loads(holder))

    if not holder or holder.is_expired:
        print("Last Lock: Expired")
    else:
        print(f"Last Lock: Acquired on {holder.host}")
        print(f"PPID: {holder.ppid}")
        print(f"Command: {shlex.join(holder.command)}")


@cli.command()
@config_option
def enable(config: str) -> None:
    """ Enable OSD staggering. """

    config = Config.from_yaml(config)

    with config.ioctx() as ioctx:
        config.enable(ioctx)


@cli.command()
@config_option
def disable(config: str) -> None:
    """ Disable OSD staggering, immediately seizing all locking, even
    for locks that are currently in progress.

    """

    config = Config.from_yaml(config)

    with config.ioctx() as ioctx:
        config.disable(ioctx)


@cli.command(hidden=True)
@config_option
def shell(config: str) -> None:
    """ Drop into a Python shell, with an active ioctx. Only use if you know
    what you are doing!

    """

    config = Config.from_yaml(config)

    with config.ioctx() as ioctx:
        code.interact(local={**globals(), **locals()})


if __name__ == '__main__':
    cli()

This gives us the stagger-osds CLI tool, that we can use to wrap commands that are then run with a cluster-wide lock on a chosen object in a designated pool.

In a sense, it works a lot like flock. You can give it a command, that is then run with an exclusive lock on the cluster:

$ stagger-osds start --verbose -- sleep 5
exec-process [INFO] Awaiting lock
lock-process [INFO] Acquiring lock (timeout 50s)
lock-process [INFO] Acquired lock, cleared for launch
lock-process [INFO] Holding lock for up to 4s
exec-process [INFO] sleep 5
lock-process [INFO] Released lock

While the lock is used, it shows up in the status command:

$ stagger-osds status
Status: Enabled
Last Lock: Acquired on lab-nvme1-a-lpg1
PPID: 3927556
Command: sleep 5

With stagger-osds disable we can disable the lock, causing all processes currently waiting for the lock to be started instantly.

Systemd Integration

To start all ceph-osd@.service units wrapped in the stagger-osds wrapper, we use the following drop-in:

[Service]
ExecStart=
ExecStart=/usr/local/bin/stagger-osds start --verbose -- /usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
SyslogIdentifier=ceph-osd
TimeoutSec=148

The timeout needs to be configured to accommodate the start of 1/3 of all OSDs on the cluster (after the timeout is over, all OSDs are started right away). Here is the kind of config one might use on a cluster with 600 OSDs:

• 1/3 of the Cluster: 600 OSDs / 3 = 200 OSDs
• Timeout: 200 OSDs x 4s = 800s

ceph_config: /etc/ceph/ceph.conf
keyring: /etc/ceph/ceph.client.stagger-osds.keyring
pool: sys
key: stagger-osds
duration: 4
timeout: 800

The keyring was created as follows:

ceph auth get-or-create \
    client.stagger-osds \
    mon "allow rw" \
    osd "allow rwx pool stagger-osds" \
    -o /etc/ceph/ceph.client.stagger-osds.keyring

Conclusion

Stagger OSDs is active on all our Ceph clusters, and since we have started using it, clusters run with more predictable performance, even during major upgrades.

Using builtin Ceph tools, we are also able to use global locks without the likes of Zookeeper, Etcd or Redis.