I used to be a huge fan of creating light-weight container images that only had a single binary in them with FROM SCRATCH but after having a couple years of Kubernetes administrator support I can now say without hestitation that anyone who willfully creates a FROM SCRATCH image that is going to be supported at enterprise scale should be sternly discouraged from doing so.
Why? Because it is impossible to get inside a running container created in this way when things don’t work as planned. Containers made from images that are built on busybox, alpine, or ubuntu (all extremely minimal Linux “distros”) support the ability to kubectl exec -it <pod> -- /bin/sh into them to get a real intimate view of what is happening. All others force the creation of a sidecar or other pod that has to attempt to figure out what the application is doing. The extra size of a container with exec support is very much worth the minimal additional resource cost at enterprise scale.
Here’s the reality. Developers release buggy software all the time and practically never properly document the thing running inside the container. This is why interpreted languages have been so popular all these years. People can look at what is actually running and even fix it. And while I’m a Go fan, if the creators of the container don’t make it incredibly obvious where to go to determine what the program inside the container is doing—including its source code with Containerfile—then creating a compiled binary is the worst thing any developer team can do since it is completely and totally unsupportable.
To give you a real-world example of this consider the Kubernetes Dashboard project. It’s a disaster. (Just use k9s instead.) The Dashoard server does not support exec and is virtually impossible to know what is running inside the container and to debug it despite all the complicated proxy setup required to get a safe user-facing version of the dashboard running. The project has docs but they are out of sync with the releases and contain seriously breaking changes to the argument signature that is essential for Istio integration.