Engineering Blog

Let me explain what I mean.

At cloudscale, the belief has always been that where control and independence matter, operating one's own stack is the best approach: from our Linux-based OS to our Ceph storage and OpenStack cloud, all the way down. Self-hosted inference is simply the next logical layer of that same commitment: As AI becomes core infrastructure, relying on external providers would contradict the principles of digital sovereignty, as outlined in this article. cloudscale offers GPU infrastructure for its customers and self-hosted inference is putting it to use in-house.

The Clipboard Problem

Even on the most privacy-conscious engineering team, you are always one mishap away from a leak. This isn't hypothetical. We've all seen it happen with Google Translate. Someone works on a ticket written in a foreign language, pastes a block of customer data to translate it, and suddenly that data has touched a server none of us approved.

Coding agents introduce the same failure mode at higher velocity, and they don't wait to be asked. They reach for what they need: a schema, an env file, a support thread sitting open in context. The agent helpfully processes it. You get your answer. And somewhere, something you didn't intend to share has left the building.

These are the stakes: we're not just protecting source code. We're protecting proprietary logic, credentials, customer data, and whatever else ends up in that prompt. Self-hosting your inference layer is one of the most meaningful choices you can make here. But it's a starting point, not a finish line.

The Obvious Part: Self-Hosted Inference

If you decide to go down this road, you probably picture something like the Ollama experience: pull a model, run a command, done. The go-kart version. Clean, fast, fun. And for one engineer, on one machine, running one task at a time, it genuinely is.

Then the second engineer joins. Then the third. Then someone runs Claude Code with four parallel sub-agents on a long-context refactor, while two colleagues are doing the same. Suddenly you need to think about who gets GPU time, memory bandwidth, and when, how requests queue, whether one heavy session can starve everyone else. You didn't buy a bigger go-kart. That's how you end up with a commercial jetliner on the runway while barely understanding why you need flaps. Not because you made a bad decision, but because a tool that works beautifully for one person becomes infrastructure the moment it serves a team. And infrastructure has a completely different set of requirements than a local dev tool.

And here's the uncomfortable truth: if you actually care about safety, you need to go deep. Blindly accepting whatever your favorite AI suggests as a fix for your performance problem is exactly the behavior you're trying to protect against, so the same standard applies here. You need to actually understand what's happening, and that means going deep on dynamic and continuous batching, KV caching, and before long you're dusting off your understanding of attention mechanisms just to reason about how many long-context multi-agent users fit into a given amount of VRAM. The good news: this stuff is genuinely fascinating. The bad news: it's probably not what you had in mind when you started this undertaking on a Tuesday afternoon hoping to offload the boring parts of your development.

This isn't a reason to avoid it. I genuinely think it's worth it, if you value privacy. But not as a side project, not as a one-afternoon setup that then runs unattended.

Update: Michael explains more technical details in a separate post.

The Overlooked Part: The Chatty Client Problem

Here's the assumption that bites people: "I'm running the model myself, so I'm private."

Maybe. But the model and the client are two separate things, and they have two separate surfaces that can phone home.

Your inference backend, Ollama, vLLM, whatever you're running, may have diagnostics, telemetry, or update-check behavior baked in. That's worth auditing, but it's usually the easier half to control.

Your agent client, Claude Code, Cursor, Copilot, whatever your developers are actually typing into, is a different story entirely. The client can, and most of the well-known ones do, send telemetry, crash reports, usage analytics, and in some cases full prompt content to vendor servers, regardless of where the model lives. "Open source" on the label doesn't automatically mean audited data practices; some of the most popular tools in this space are more opaque than they appear.

The categories to look for when you audit:

• Usage analytics - what gets counted and reported
• Crash reporting - Sentry-style tools that may capture context around errors
• Prompt feedback loops - the big one: is the vendor training on your prompts? Read the terms carefully.
• License and update pings - lower risk, but worth knowing about

Now, the honest part: reading privacy policies is the boring part of this work. I know. We became engineers so we wouldn't have to do this. But for this specific use case, the privacy policy is load-bearing. It's the document that tells you what actually happens to your data.

The Danger Zone: What Developers Need to Understand Now

Everything we've discussed so far, the careful inference setup, the audited client, the dialed-in permissions, can be silently undone by what happens in this section. A single successful prompt injection can turn your sovereign, self-hosted, privacy-preserving setup into an exfiltration machine. All that careful work, on fire, from the inside.

Every single one of these risks applies equally if you're running Claude Code against Anthropic's API, Cursor against OpenAI, or any self-hosted solution. The threat model doesn't care where the model lives. It cares what the agent is allowed to do.

This is my current shortlist, not exhaustive, and guaranteed to look different in six months. But these are the concepts I keep coming back to. (As it turns out, OWASP agrees: they published an OWASP Top 10 for Agentic Applications for 2026, and the overlap is uncomfortably exact.)

Prompt injection. An attacker plants instructions inside something the agent will read, a README, a code comment, an API response, a file in the repo. The agent reads it and treats it as instruction, the same way it treats yours. Your codebase, and everything it touches, is now an attack surface. This is why tools like Context7 and other MCP servers that pull in external content by design give me cold sweats. The injection surface isn't a bug someone introduced, it's the feature.

Slopsquatting (supply chain attack via model hallucination). Models confidently suggest packages that don't exist. Attackers register those hallucinated package names with malicious payloads, waiting for the agent, or the developer approving it, to run the install. It's the npm install problem applied to the space between what the model knows and what it invents. Your agent doesn't know the difference between a real package and a plausible-sounding one.

Tool permissions and sandbox scope. An agent that can read files, run shell commands, and make outbound HTTP requests is extremely powerful, and default configurations are almost always too permissive, at least for me to sleep well. What makes this worse than a misconfiguration: agents don't shrug when a tool is denied. They optimize around the constraint, chaining whatever's available to reach the same destination. Before you trust that your setup is locked down, it's worth reading and testing (!) exactly what your sandbox setting actually covers. The answer is usually narrower than the name implies.

So: do you think the AI named after the father of information theory can read and exfiltrate /etc/passwd while running in sandboxed mode?

If we want to use these systems safely and well, we need to make space for people to learn how they fail, how to test their boundaries, and how to build with them responsibly.

The Mail Server Lesson

Running your own mail server never protected you from phishing. Sovereignty and security are related, but they're not the same thing, and if privacy is actually your goal, you need both. Sovereignty gives you control over where your data lives. Security determines whether it stays there. One without the other is a half-measure. You can self-host everything perfectly and still get hit through a prompt injection in a README. You can have airtight security practices and still leak prompts through a client you never audited.

I want to be clear about what this piece is not. It's not a warning to stop using agents. It's not a criticism of the tools or the teams building them. This is an exciting moment, I think this is one of the more interesting periods to be writing software.

But we as an industry have also quietly accumulated new problems most of us didn't know existed a year ago. And they are solvable, but not alone. The OWASP Top 10 for Agentic Applications is the industry's clearest current attempt to name these problems together. Within each organization it takes engineers willing to go deep, and management willing to make that depth possible. And it takes the security team treating developers as allies to educate, not risks to contain. The teams that treat this as a shared problem will be fine. The ones that throw it over a wall won't.

We've been here before.

---

If you have comments or corrections to share, you can reach our engineers at engineering-blog@cloudscale.ch.