MPI Training

Instructions for using MPI for training

This guide walks you through using MPI for training.

Installation

This section has not yet been converted to kustomize, please refer to kubeflow/manifests/issues/227.

If you haven’t already done so please follow the Getting Started Guide to deploy Kubeflow.

An alpha version of MPI support was introduced with Kubeflow 0.2.0. You must be using a version of Kubeflow newer than 0.2.0.

You can check whether the MPI Job custom resource is installed via:

  1. kubectl get crd

The output should include mpijobs.kubeflow.org like the following:

  1. NAME                                       AGE
  2. ...
  3. mpijobs.kubeflow.org                       4d
  4. ...

If it is not included you can add it as follows:

  1. cd ${KSONNET_APP}
  2. ks pkg install kubeflow/mpi-job
  3. ks generate mpi-operator mpi-operator
  4. ks apply ${ENVIRONMENT} -c mpi-operator

Alternatively, you can deploy the operator with default settings without using ksonnet by running the following from the repo:

  1. kubectl create -f deploy/

Creating an MPI Job

You can create an MPI job by defining an MPIJob config file. See TensorFlow benchmark example config file for launching a multi-node TensorFlow benchmark training job. You may change the config file based on your requirements.

  1. cat examples/tensorflow-benchmarks.yaml

Deploy the MPIJob resource to start training:

  1. kubectl create -f examples/tensorflow-benchmarks.yaml

Monitoring an MPI Job

Once the MPIJob resource is created, you should now be able to see the created pods matching the specified number of GPUs. You can also monitor the job status from the status section. Here is sample output when the job is successfully completed.

  1. kubectl get -o yaml mpijobs tensorflow-benchmarks-16
  1. apiVersion: kubeflow.org/v1alpha1
  2. kind: MPIJob
  3. metadata:
  4.   clusterName: ""
  5.   creationTimestamp: 2019-01-07T20:32:12Z
  6.   generation: 1
  7.   name: tensorflow-benchmarks-16
  8.   namespace: default
  9.   resourceVersion: "185051397"
  10.   selfLink: /apis/kubeflow.org/v1alpha1/namespaces/default/mpijobs/tensorflow-benchmarks-16
  11.   uid: 8dc8c044-127d-11e9-a419-02420bbe29f3
  12. spec:
  13.   gpus: 16
  14.   template:
  15.     metadata:
  16.       creationTimestamp: null
  17.     spec:
  18.       containers:
  19.       - image: mpioperator/tensorflow-benchmarks:latest
  20.         name: tensorflow-benchmarks
  21.         resources: {}
  22. status:
  23.   launcherStatus: Succeeded

Training should run for 100 steps and takes a few minutes on a GPU cluster. You can inspect the logs to see the training progress. When the job starts, access the logs from the launcher pod:

  1. PODNAME=$(kubectl get pods -l mpi_job_name=tensorflow-benchmarks-16,mpi_role_type=launcher -o name)
  2. kubectl logs -f ${PODNAME}
  1. TensorFlow:  1.10
  2. Model:       resnet101
  3. Dataset:     imagenet (synthetic)
  4. Mode:        training
  5. SingleSess:  False
  6. Batch size:  128 global
  7.              64 per device
  8. Num batches: 100
  9. Num epochs:  0.01
  10. Devices:     ['horovod/gpu:0', 'horovod/gpu:1']
  11. Data format: NCHW
  12. Optimizer:   sgd
  13. Variables:   horovod
  15. ...
  17. 40    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.146
  18. 40    images/sec: 132.1 +/- 0.0 (jitter = 0.1)    9.182
  19. 50    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.071
  20. 50    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.210
  21. 60    images/sec: 132.2 +/- 0.0 (jitter = 0.2)    9.180
  22. 60    images/sec: 132.2 +/- 0.0 (jitter = 0.2)    9.055
  23. 70    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.005
  24. 70    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.096
  25. 80    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.231
  26. 80    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.197
  27. 90    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.201
  28. 90    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.089
  29. 100    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.183
  30. ----------------------------------------------------------------
  31. total images/sec: 264.26
  32. ----------------------------------------------------------------
  33. 100    images/sec: 132.1 +/- 0.0 (jitter = 0.2)    9.044
  34. ----------------------------------------------------------------
  35. total images/sec: 264.26
  36. ----------------------------------------------------------------

Docker Images

Docker images are built and pushed automatically to mpioperator on Dockerhub. You can use the following Dockerfiles to build the images yourself: