Cluster And Scheduling

Cluster mode

Cluster mode

The Group can be set to ClusterMode . Once set to cluster mode, all the server under the group will be managed and scheduled by humpback center.

All managed nodes are registered in the cluster via Humpback Agent and zookeeper. The same node can be registered to multiple groups.

Humpback Center is only a scheduler, not responsible for container management.

How to start the cluster center service

Run Humpback Center in docker 

  1. $ docker pull humpbacks/humpback-center:latest
  2. $ docker run -d -ti --net=host --restart=always \
  3.  --name=humpback-center \
  4.  -e HUMPBACK_SITEAPI=http://192.168.2.80/api \
  5.  -e CENTER_LISTEN_PORT=:8589 \
  6.  -e CENTER_API_ENABLECORS=true \
  7.  -e DOCKER_CLUSTER_URIS=etcd://192.168.2.80:2379,192.168.2.81:2379,192.168.2.82:2379 \
  8.  -e DOCKER_CLUSTER_NAME=humpback/center \
  9.  -v /opt/app/humpback-center/cache:/opt/humpback-center/cache \
  10.  -v /opt/app/humpback-center/logs:/opt/humpback-center/logs \
  11.  -v /opt/app/humpback-center/data:/opt/humpback-center/data \
  12.  -v /opt/app/humpback-center/etc/config.yaml:/opt/humpback-center/etc/config.yaml \
  13.  humpbacks/humpback-center:latest
  14. $ docker ps -a
  15. CONTAINER ID    IMAGE                           COMMAND                  CREATED         STATUS         PORTS         NAMES
  16. a1640bf8c956    humpbacks/humpback-center:latest  "./humpback-center"     15 minutes ago  45 seconds ago              humpback-center

How to join the cluster

Run Humpback Agent in docker

  1. $ docker pull humpbacks/humpback-agent:latest
  2. $ docker run -d -ti --net=host --restart=always \
  3.  --name=humpback-agent \
  4.  -e DOCKER_API_VERSION=v1.21 \
  5.  -e DOCKER_AGENT_IPADDR=0.0.0.0 \
  6.  -e DOCKER_CLUSTER_ENABLED=true \
  7.  -e DOCKER_CLUSTER_URIS=etcd://192.168.2.80:2379,192.168.2.81:2379,192.168.2.82:2379 \
  8.  -e DOCKER_CLUSTER_NAME=humpback/center \
  9.  -v /var/run/:/var/run/:rw \
  10.  humpbacks/humpback-agent:latest
  11. $ docker ps -a
  12. CONTAINER ID    IMAGE                           COMMAND               CREATED        STATUS         PORTS         NAMES
  13. b1ac4a82c2dd    humpbacks/humpback-agent:latest   "./humpback-agent"   3 minutes ago  20 seconds ago               humpback-agent

Related environment variables

  • HUMPBACK_SITEAPI=http://192.168.2.80/api， Humpback-Web site address, pay attention to bring / api.

  • CENTER_LISTEN_PORT=:8589, The default port for the Humpback Center API is 8589.

  • CENTER_API_ENABLECORS=true, Whether the Humpback Center API supports cross-domain access.

  • DOCKER_API_VERSION=v1.21, Default v1.21, use thedocker version command to view the API Version in the client first.

  • DOCKER_CLUSTER_ENABLED=true, Cluster mode switch, After the container is started, it is automatically registered in the cluster

  • DOCKER_CLUSTER_URIS, Default to etcd cluster address, if it is zookeeper, consul or other found tools, please change the prefix to zk: // or consul: //.

  • DOCKER_CLUSTER_NAME=humpback/center, Set the humpback cluster name, and the configuration of Humpback Agent is consistent with theHumpback Center configuration

  • /opt/app/humpback-center/cache Cluster container information persistence directory, It is recommended that you do not manually change and delete.

  • /opt/app/humpback-center/logs System log directory.

  • /opt/app/humpback-center/data System data directory.

After successful startup of Humpback Agent, you can modify the Group attribute at the Humpback site and open the ClusterMode mode option. At this time all the servers under the Group will switch to cluster scheduling mode.

How to exit the cluster

When the ClusterMode option of Group is closed on the Humpback site, all services in the group will exit the cluster management mode and do not need to restart the Humpback Agent. Note: Once group exit cluster, all containers which are scheduled by the cluster will be deleted.

Scheduling strategy

At present, ClusterMode has only one built-in scheduling strategy. The strategy compute rank according to node’s CPU, memory usage, and number of containers.

Scheduling strategy

Scheduling a container will use following three steps:

  • Health Check：First choose the health nodes.

  • Weight Assessment：Compute the rank.

  • Node Filtering：Ignore the nodes which were selected before or create container failed.

About the container

Container migration

Container migration occurs when the node is down or Humpback Agent stopped. The scheduler will create new containers in the health nodes. It will make sure the number of running containers is same as you configured. But the migration action will not start as soon as node down, it has a delay time(default 45 seconds). Delay time is used for avoid network unstable.

Container recovery

Humpback Center will scan the number of container instances for all groups every 120s, in order to check the number of container instances. This situation generally occurs when the container is deleted manually. The container recovery algorithm is also same as scheduling strategy.

About WebHook

The WebHook is used to call back to notify the cluster container status changed. You can get the detail infomation about the IP address and name of the target host to be assigned.

WebHook event description:

  1. CreateMetaEvent    Triggers when the Containers are created for the first time
  2. RemoveMetaEvent    Triggered when removing Containers, including exit clustering mode
  3. OperateMetaEvent   Triggered when the Containers are manipulated, Start/Stop/Restart/Kill…
  4. UpdateMetaEvent    Triggered when modifying the number of cluster instances
  5. UpgradeMetaEvent   Triggered when upgrading Containers Tag
  6. MigrateMetaEvent   Triggered when a container migration occurs
  7. RecoveryMetaEvent  Triggered when the system recovers or shrinks the container

WebHook content format:

The following is the WebHook event for the container to migrate. The container CLUSTER-dc0d7d79-cluster-test-2 was migrated to the 192.168.0.1 node.

  1. {
  2.     "Timestamp": 1491976204448307844,
  3.     "Event": "MigrateMetaEvent",
  4.     "MetaBase": {
  5.         "GroupId": "dc0d7d79-e9a1-46aa-8d47-95335f061b60",
  6.         "MetaId": "0136b582-35ca-4755-8148-b7edde2a1127",
  7.         "Instances": 9,
  8.         "WebHooks": [
  9.             {
  10.                 "Url": "http://192.168.0.10:8513/mkkks4r/users",
  11.                 "SecretToken": "123456"
  12.             }
  13.         ],
  14.         "ImageTag": "latest",
  15.         "Config": {
  16.             "Id": "",
  17.             "Image": "RegistryServer/base/alpine-bash",
  18.             "Command": "date",
  19.             "Name": "cluster-test",
  20.             "Ports": [],
  21.             "Volumes": [],
  22.             "Dns": null,
  23.             "Env": [],
  24.             "HostName": "",
  25.             "NetworkMode": "host",
  26.             "RestartPolicy": "no",
  27.             "Extrahosts": null,
  28.             "Links": [],
  29.             "Ulimits": null
  30.         }
  31.     },
  32.     "HookContainers": [
  33.         {
  34.             "IP": "192.168.0.1",
  35.             "Name": "SERVER01",
  36.             "Container": {
  37.                 "Id": "8af07e8091aa34660e7c53b76083a36dd114250548c7df5670e948e7b15eae05",
  38.                 "Image": "RegistryServer/base/alpine-bash",
  39.                 "Command": "date",
  40.                 "Name": "CLUSTER-dc0d7d79-cluster-test-2",
  41.                 "Ports": null,
  42.                 "Volumes": null,
  43.                 "Dns": null,
  44.                 "Env": [],
  45.                 "HostName": "SCMISMDTSLR02",
  46.                 "NetworkMode": "host",
  47.                 "Status": {
  48.                     "Status": "exited",
  49.                     "Running": false,
  50.                     "Paused": false,
  51.                     "Restarting": false,
  52.                     "OOMKilled": false,
  53.                     "Dead": false,
  54.                     "Pid": 0,
  55.                     "ExitCode": 0,
  56.                     "Error": "",
  57.                     "StartedAt": "2017-04-12T03:33:31.216194816Z",
  58.                     "FinishedAt": "2017-04-12T03:33:31.230026534Z"
  59.                 },
  60.                 "RestartPolicy": "no",
  61.                 "Extrahosts": null,
  62.                 "Links": null,
  63.                 "Ulimits": null
  64.             }
  65.         }
  66.     ]
  67. }
  • Timestamp：WebHook triggers timestamps.
  • Event：Trigger event. See event description.
  • MetaBase：Meta represents the same batch of containers that contain multiple container instances. In cluster mode MetaID represents a unique batch of containers.
  • GroupID：is the group to which the container belongs.
  • MetaID：The container batch ID number is unique in the cluster. If you receive the same MetaID WebHook, you need to determine Timestamp, take the largest for the latest.
  • Instances：The number of container instances set for this batch.
  • WebHooks：All WenHook settings are set in the configuration.
  • ImageTag：The image tag used by the current batch container, ImageTag will change if the same upgrade is made.
  • Config：creates configuration information for the initial batch.
  • HookContainers：reports the target node information that the container is assigned: IP and Name, along with container information and status information.