Docker Swarm Container Reconciliation Is Amazing

As I progress into porting my Orleans cluster to use Orleans 2.0 (thus .NET Core 2.0) giving me easier Docker support, I’m continued to be impressed with how awesome Docker actually is. I was really impressed by how Docker automatically initiates reconciliation of containers when nodes in your swarm cluster are dropped. I wanted to further dive into experimenting with this, and maybe explain a little of what is actually happening.

Similar to my previous post, I’ll be creating a swarm cluster of four nodes; one manager and three worker nodes. The cluster will have a docker registry service which will allow me to deploy my own application/docker image across all swarm nodes. A single ZooKeeper service for Orleans silo cluster membership, and twelve Orleans silos.

For reference on how I set all of this up, take a look at my previous post, it will guide you through how I set all of this up.

Say we have five nodes in our swarm cluster hosting ten containers. One of those nodes happen to fail, taking down three containers on that node — what happens? Swarm will notice that we’re missing three containers, and automatically schedule them across whatever machines are available in the cluster. What happens if my container fails due to some dependency on a container that failed or hasn’t started yet, or because of the container being unhealthy? Swarm will kill that container and reschedule it again.

This is pretty much the same with containers, our desired state would be whatever services we’ve said we want deployed to our cluster, and our actual state is well, what we have at that point in time.

I can explain, give links to other blogs & websites, but I always find it better to see it all in action, so lets do exactly that. Lets start a swarm cluster, deploy some containers to it and destroy machines/containers. So I have swarm running here, with one ZooKeeper container, and twelve silo containers.

So lets run

to see what silos are running on what machines.

For this, lets kill the node1 virtual machine, let’s just check what silos are running on that node by running

so these silos will inevitably die along with the virtual machine falling… Lets now kill the virtual machine

Now, we would expect swarm to notice that these three containers are missing, and move those three silos (silo.1, silo.2 and silo.8) onto another machine. Let’s check this by running this command again, to get currently running containers.

That is exactly what happened.

Pretty damn cool.

Related posts:

Falling in love is the easiest thing one can do, be it intentionally or unintentionally. However, getting away from those fluttery feelings is the most painful and gradually happening task. Even if…

This is one of the first poems I wrote after getting back into the writing game. I had been reading a lot of Robert Frost and there’s a lot of his influence in the piece. The poem itself only took…