Docker in production
We have been using Docker 17.06 in production successfuly for more than 9 month already. I wanted to share some good practices, they are easy to find but having them all in one place is convenient. This is also reminder for me on the issues discovered and fixes applied.
While docker compose is okay solution for small services, if you have a team and you starting/migrating a project, please look into Kubernetes and CNCF it has many good tools already built by the community.
Docker on Windows
Docker runs on Windows 10 PRO because it uses HyperV virtualization.
- You cant run both DockerMachine and HyperV native docker at the same time (HyperV and Virtualbox together) (June 2018)
- HyperV requires some manual steps due to networking adapter setup
- While Docker itself works combining it with LSW (Linux Subsystem for Windows) is tricky
- Git (git-scm) needs to be installed from https://git-scm.com/ and ENABLE symlink support
- Would have to use PowerShell/CMD for git commands, Windows 10 NTFS symlinks are not supported under LSW
- Symlinks require enabling developer mode in Windows 10, and editing LocalSecurityPolicy to allow creating symlinks,
- Its better to disable UAC due to performance penalties working under LSW
- You would have to recreate local repository with
git clone -c core.symlinks=true https://github.com/myuser/repo.git
- Black magic in form of adding windows account environment variables:
- Would have to install cmder terminal emulator instead of LSW one
- Clipboard LSW copy-paste was only added in latest > 18xxx revision of Windows 10
Generally while Docker and LSW work great separately, combining them into environment is tricky. Especially if you plan to run Kubernetes at the same time (same machine), as Kubernetes doesnt support propert folder mounts still on Windows 10 (June 2018), even if you manage to mount some folders - you will start getting lock/error on file access and symlink issues.
- Docker Cheat Sheet
- Docker Engine Security Cheat Sheet
- Best practices for writing Dockerfiles
- Dockerfile reference
You cant specify resources limits in Dockerfile, you would have to use docker-compose or cli arguments. Practical workaround is launching docker containers as services with systemd.
Example systemd service
- Fetches latest image on start from AWS ECR
- ECR credentials are provided by EC2 role under root user
- Memory is limited to 500mb, swap usage is restricted
- Host networking is used for better performance
- We dont want kernel to kill processes inside our container in case host runs out of memory
- Delete container data when container stops, container is stateless
- Mount data folder readonly
- Explicitly specify ports and protocol
- Container is configured to react to SIGPWR as stop signal, we wait 45 seconds for gracefull stop
- We dont restart container if it dies abnormally, if there is problem we need to fix it
- We start our container/service after entering multi-user mode (often ‘default’ on normal boot)
Healthchecks with start period, docker fires up containers but they might take time to initialize. This is better handled in Kubernetes, docker only has HEALTHCHECK in Dockerfile.
EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime The EXPOSE instruction does not actually publish the port. EXPOSE is help for people trying to use your container.
FROM can be dynamic
Try to run your apps as non-root user, this user doesnt have to exist on host at all
Kubernetes/Minikube local setup on Windows (Virtualbox)
To install Minikube use Chocolatey
Or download directly from https://github.com/kubernetes/minikube/releases/download/v0.25.2/minikube-windows-amd64 and rename to minikube.exe and place inside PATH (i.e. your windows home folder)
Use 0.25.2 version (June 2018) as later versions switched to KubeAdm for creating local cluster, while 0.25.2 defaults to localcube.
Install Kubernetes CLI via Chocolatey
Or download directly from https://storage.googleapis.com/kubernetes-release/release/v1.10.0/bin/windows/amd64/kubectl.exe and place inside PATH (i.e. your windows home folder)
Setup minikube settings, recommended minimum memory >2.8Gb and k8s version 1.10.0 (LTS)
Start cluster by typing inside PowerShell,
- localkube bootstrapper is deprecated but stable for 1.10.0
- host-only-cidr is needed to fix cluster networking issues while restoring (wake-up)
After launching i recommend disabling all addons (requires restart), as many of them are not RBAC enabled
Or manually edit
~/.minikube/config/config.json before starting cluster
After cluster starts, use forwarding to access your services
Overall we didnt encountered any issues with running docker, it’s stable and we continue to migrate to K8S. If you dont use Docker i highly recommend it for your local environment and both beta/production. If you already use Puppet/Chef or other provisioning software, you can easily migrate to docker just by running i.e. standalone puppet provisioning inside Docker container.
Kubernetes 1.10.0 release is an LTS release and is mainstream now (AWS EKS GA in June 2018).