./ahmedhashim

I once spent a day debugging why a Python script wasn’t working in production. The code ran without issues on Ubuntu during development but crashed consistently when deployed to Alpine Linux servers.

Turns out the problem was because of libc. I was developing on a glibc system but deploying with musl-based Docker images.

When you install a Python package on Ubuntu, you get wheels compiled against glibc. Those same wheels fail on Alpine because musl handles dynamic linking differently.

It goes beyond binary compatibility: DNS resolution behaves differently between the two as well. Applications that resolved hostnames fine during development can start timing out in production because musl’s resolver has different fallback behavior.

Memory management and threading also differ, along with subtler things like character encoding. The bugs that come out of these differences can cause a few gray hairs.

My fix is to use the runtime versions from development through production. If you’re deploying on musl, try developing with it. If production runs Python 3.9.2, don’t develop on 3.9.7 because it’s slightly newer.

I don’t expect developers to abandon their Mac workflows, but Docker makes testing against a production image trivial. A quick docker run before deployment can save hours of production debugging.