Engineering Blog

At first glance, this looked like a straightforward CSS bug. We are a cloud infrastructure provider, so frontend styling is not exactly our daily bread and butter. Still, a UI that shouts at its users is not a good look. What followed was a reminder that the most perplexing bugs hide in bytes you cannot even see.

It Only Broke in Production

The annoying part: it only happened in production. Local builds were fine.

We checked font-weight declarations, CSS custom properties, font files. Everything looked correct. We were going in circles.

Then, in the browser's developer tools, we noticed something strange: commenting out the @font-face definition for our "demi" font weight and uncommenting it again temporarily fixed the rendering. But the fix never survived a fresh build.

That gave us a clue: the browser was doing what we told it to. We just weren't seeing the whole instruction.

The Culprit

We turned off CSS minification. Set minifyFontValues: false. Cleared every cache. Nothing. Eventually we ended up back in the browser's developer tools, looking at the raw CSS.

Right there, in front of the @font-face definition, sat a character that did not belong. It was invisible in most views, but present in the source: \ufeff.

For those not intimately familiar with Unicode trivia, \ufeff or EF BB BF is the UTF-8 Byte Order Mark (BOM). It is an invisible character used to indicate a file's encoding. Here, it was sitting in the middle of a CSS bundle, right before a @font-face declaration. The browser misparsed what followed, confusing the font weight selection and causing all text to render as bold.

So we had found what was breaking the CSS. We still had no idea what was putting it there. There was not a single BOM in our source code. It did not appear in the development build, or in production builds from a few months prior. So how did it get here?

Git Bisect

The BOM was easy to detect. The harder question was when it had first appeared in the build output. Since older production builds were unaffected, we had a good candidate for a search through our commit history.

Over four hundred commits in the last month. We did not want to check them by hand.

Git bisect performs a binary search through your commit history. You mark one commit as "good" and another as "bad", and git repeatedly checks out the midpoint between them until it narrows the problem down to a single commit.

The only thing we needed was an automated test for our known failure condition: does the production CSS contain the BOM?

#!/usr/bin/env bash
set -e

# --- Optional: only npm install when lockfile changed ---
LOCK_HASH=$(md5sum package-lock.json | awk '{print $1}')
HASH_FILE="/tmp/webpack_bisect_lock"
if [ ! -f "$HASH_FILE" ] || [ "$(cat "$HASH_FILE")" != "$LOCK_HASH" ]; then
echo "Lockfile changed — running npm install..."
npm install
echo "$LOCK_HASH" > "$HASH_FILE"
fi

# --- Clean stale artifacts ---
rm -rf node_modules/.cache/webpack/default-production gui/output/bundles

# --- Production build ---
npx webpack --config gui/webpack.config.js --mode=production

# --- Check CSS for BOM ---
CONTROL_PANEL_CSS=$(ls gui/output/bundles/control-panel-*.css | head -n 1)
BOM_COUNT=$(python3 -c "with open('$CONTROL_PANEL_CSS','rb') as f: print(f.read().count(b'\xef\xbb\xbf'))")

if [ "$BOM_COUNT" -eq 0 ]; then
echo "GOOD: No BOM in $(basename "$CONTROL_PANEL_CSS")"
exit 0
else
echo "BAD: BOM found ($BOM_COUNT) in $(basename "$CONTROL_PANEL_CSS")"
exit 1
fi

With this script, git bisect could run autonomously:

git bisect start
git bisect bad HEAD
git bisect good <commit-from-two-months-ago>
git bisect run ./bisect-test.sh

Git bisect found it in minutes: a PostCSS update.

At first, this looked like we'd found the culprit. But PostCSS 8.5.24 was not adding the BOM. It had simply stopped stripping it out. That was reasonable on their side. For us, it just stopped masking a problem that originated elsewhere.

That meant the next question was where the BOM had originally come from.

In our case, the answer was sass-loader. Updating to 17.0.1 ensured BOMs were not emitted in the first place, and the fonts went back to normal.

The Fix

In the end, the fix was almost disappointingly simple: npm update sass-loader.

The hard part was finding it. The PostCSS update was for a security issue and did not cause the bug, it just stopped hiding it. That is the annoying thing about dependency updates: sometimes they expose a mess that was always there.

We spend a lot of time on systems customers never see. The Control Panel is one they see every day. It should be just as predictable.

And, ideally, less bold.

We're hiring a Frontend Engineer

Do you enjoy tracking down invisible bugs and care about well-built user interfaces? We are currently looking for a frontend engineer to join our team in Zurich. If that sounds like your kind of problem, have a look at our open positions.

Wenn du uns Kommentare oder Korrekturen mitteilen möchtest, kannst du unsere Engineers unter engineering-blog@cloudscale.ch erreichen.