Engineering Blog

In a universe dominated by a few U.S. tech giants, it's not always easy maintaining decent control over your digital assets.

Unless you're willing to take rather extreme approaches, your choice of mobile phone operating system is one between Apple and Google. The cloud service market is dominated by Amazon, Microsoft, and Google. To fullfil your SaaS groupware needs, feel free to choose between Microsoft and Google.

U.S. Big Tech

While there are few determined souls not using any services of the "Big Five" (Amazon, Apple, Meta, Microsoft, and Google), most of us will happily rely on some of their free offerings for personal use. As for satisfying your business needs regarding groupware or cloud services, no one in IT procurement has ever been fired for choosing "industry standards" like AWS, GCP, or Azure.

Entrusting the large U.S. players with your own or your customers data comes with significant downsides and risks though.

Marketing ahoy

Social media platforms are lovely to stay in touch with people from around the world. Seeing Marco on his latest kite surfing adventure in Egypt, or Rachel relaxing on her beach vacation in Greece is heart-warming and allows me to stay in touch with friends I don't interact with very often. Though know that If You're Not Paying For It, You Become The Product.

Vendor lock-in

While migrating your services to any of the mentioned vendors is made easy by a plethora of migration tools, you'll often find yourself in quite some pain trying to migrate away. In combination with purposefully obscure cost and subscription models, these kinds of relationship may often leave you feeling somewhat frustrated and powerless.

U.S. Cloud Act

After Microsoft refused to hand over data related to an FBI investigation - arguing the emails in question were located in an Ireland data center and therefore outside of U.S. jurisdiction - congress passed the Cloud Act in 2018, which:

• Expands the reach of U.S. law enforcement over data held by U.S. providers abroad.
• Facilitates foreign law enforcement access to data from U.S. providers.

U.S. jurisdiction extended

With the Cloud Act extending U.S. jurisdiction to any server operated by a U.S. company, if you entrust them with your data, you'll find it to be under the control of the U.S. government, no matter where it's physically stored.

In a recent example, after the Trump administration passed sanctions against the chief prosecutor of the International Criminal Court (ICC), Mr. Khan had his email account suspended by Microsoft, forcing him to resort to Switzerland-based ProtonMail.

As for hosting your data in a location of your choice, in an sworn testimony before a French Senate inquiry, Microsoft France's director of public and legal affairs, was asked whether he could guarantee that French citizen data would never be transmitted to U.S. authorities without explicit French authorization. And, he replied, "No, I cannot guarantee it."

The same applies to any digital service offering by U.S. tech firms.

While entrusting U.S. tech companies with personal information is a choice everyone has to make for themselves, doing so with customer data is not only negligent, it is likely to be against the law.

Open Source to the Rescue

Considering the above, combined with our desire to maintain as much control over our infrastructure as possible, cloudscale has made a decision to rely on external service providers as little as possible. We design, build and operate our own services, on our own hardware, as much as we can.

FOSS FTW!

While Free and open source software (FOSS) can sometimes feel a bit chaotic, it promises to best fulfill specific use cases and is open to contributions by anyone, ensuring maximum amounts of transparency, trustworthiness, and adaptability. It allows us to customize and extend functionality based on our needs, and we strive to contribute our improvements back upstream for the greater good of other parties relying on these projects.

Because if many people and organizations share a common technological need - like running a web server for example - the most reliable, transparent and customizable solution will often end up being an open source project, where everyone collaborates to achieve the best possible result.

Microsoft spent hundreds of millions trying to establish their proprietary, commercial web server Internet Information Server (IIS) as the tool of choice, only to get pretty much wiped off the face of the internet by open source solutions.

FOSS @cloudscale

The vast majority of our customer-facing services are built from open source components managed by us, tailored to our specific needs. Our operating system of choice is Linux, with Debian being our favorite distribution. Front-facing web services are handled by HAproxy and nginx. We use Ceph for storage, and OpenStack for our cloud service offering.

Running Our Own Stack

While we do use some commercial software like an on-prem GitLab instance, we also try to rely on free open source components for our daily work wherever reasonably possible.

Centralized authentication of our employees happens through an OpenLDAP directory, with the credentials required to operate cloudscale infrastructure stored in an internal Vault-based setup. Our monitoring is based on Zabbix, and we rely on Loki and Grafana for log aggregation and analysis. We use Netbox to manage our hardware inventory, our database needs are fulfilled by PostgreSQL and MariaDB clusters. For our custom-tailored mail setup, we use components like Postfix, Dovecot and Rspamd.

Combined with our high standards regarding security, reliability and maintainability, the design, implementation and operation of these services takes significant time and effort. We like to call the extra work required to take a technical implementation from fullfilling the basic requirements to something everyone in the team can fully agree to and be proud of "vergolden" (gold plating).

And while doing so not only for customer-facing components, but applying the same rigorous standards to our internal service landscape - used by a fairly small number of employees - might seem extreme, we feel like the benefits outweigh the drawbacks, since this level of attention to detail usually guarantees we won't have to worry about a certain component for a long time.

Ownership and Responsibility

As a cloud service provider, self-hosting is the only approach giving us full control over our own and our customer digital assets.

The reasoning behind, and the benefits of this approach are:

Security and Privacy

• We can mitigate any threats as soon as we become aware.
• Our internal communications never pass through third-party systems.
• Our employee records aren't sitting in some overseas data center.
• Our customers can rest assured their information is only processed by systems managed by us.

Transparency

• Relying on open source software allows us to review any code we use.
• We know exactly what’s running on our systems, there are no black boxes or mystery outages.
• If something misbehaves or breaks, we can analyze the anomaly and learn from it, preventing similar kinds of incidents from happening in the future.

Adaptability

• We can customize and optimize any component to our specific needs, whether it's adapting it to our specific use case, or tweaking it for maximum performance.
• We’re not limited by someone else's roadmap.

Resilience and Independence

• When you rely on SaaS, you’ll find yourself at the mercy of someone else’s uptime, policy changes, or business decisions.
• By building our own infrastructure, we gain a higher degree of operational independence.
• We're not tied to the fate of a service or company that might get acquired, discontinued, or priced out of reach.

In for the long game

• While paying an external company for a service subscription might be attractive short-term, the cost benefits tend to diminish over time.
• Once we have our tools set up up and working the way we want, only little maintenance is required over the course of the next few years.
• While the initial effort required might be high, our total cost of ownership decreases over time.

Skill Development

• Managing our own infrastructure pushes us to develop deep technical skills. We stay sharp. We learn. We grow.
• That experience directly translates into better services for our customers, as well as a stronger internal culture of capability, confidence and ownership.

Not Always Easy - But Worth It

Self-hosting takes time, skills, and a willingness to put in the required work. But the return on investment — in control, security, and integrity — is priceless.

For us, it’s not about rejecting SaaS entirely, but about being intentional. Wherever control, customization or independence matters, we choose to own the stack. Where commodification makes sense, we might consider outsourcing.

Digital Sovereignty

Digital sovereignty means being in control over your digital destiny - your IT infrastructure, data, and operations. It ensures your authority over how your data is stored, who can access it, and how your systems are run.

Key Aspects

• Digital Ownership: Full control over where data is stored, who accesses it, and how it’s processed.
• Software Autonomy: Freedom to debug, modify, and adapt software to your needs without vendor-imposed limitations.
• Infrastructure Independence: Ability to host your services wherever you feel comfortable.

Final Thoughts

In a world dominated by a few large players, digital sovereignty might seem like a radical act. For us, it's merely a conscious commitment to ownership, responsibility, and ultimately freedom.

And that's a choice we’ll keep making.

---

Wenn du uns Kommentare oder Korrekturen mitteilen möchtest, kannst du unsere Engineers unter engineering-blog@cloudscale.ch erreichen.