Getting Started with kOps on Spot Ocean

Ocean by Spot simplifies infrastructure management for Kubernetes. With robust, container-driven infrastructure auto-scaling and intelligent right-sizing for container resource requirements, operations can literally “set and forget” the underlying cluster.

Ocean seamlessly integrates with your existing instance groups, as a drop-in replacement for AWS Auto Scaling groups, and allows you to streamline and optimize the entire workflow, from initially creating your cluster to managing and optimizing it on an ongoing basis.

Features

  • Simplify Cluster Management — Ocean’s Virtual Node Groups make it easy to run different infrastructure in a single cluster, which can span multiple AWS VPC availability zones and subnets for high-availability.

  • Container-Driven Autoscaling and Vertical Rightsizing — Auto-detect your container infrastructure requirements so the appropriate instance size or type will always be available. Measure real-time CPU/Memory consumption of your Pods for ongoing resource optimization.

  • Cloud-Native Showback — Gain a granular view of your cluster’s cost breakdown (compute and storage) for each and every one of the cluster’s resources such as Namespaces, Deployments, Daemon Sets, Jobs, and Pods.

  • Optimized Pricing and Utilization — Ocean not only intelligently leverages Spot Instances and reserved capacity to reduce costs, but also eliminates underutilized instances with container-driven autoscaling and advanced bin-packing.

Prerequisites

Make sure you have installed kOps and installed kubectl.

Setup your environment

Spot

Generate your credentials here. If you are not a Spot Ocean user, sign up for free here. For further information, please checkout our Spot API guide, available on the Spot Help Center website.

To use environment variables, run:

  1. export SPOTINST_TOKEN=<spotinst_token>
  2. export SPOTINST_ACCOUNT=<spotinst_account>

To use credentials file, run the spotctl configure command:

  1. spotctl configure
  2. ? Enter your access token [? for help] **********************************
  3. ? Select your default account [Use arrows to move, ? for more help]
  4. > act-01234567 (prod)
  5. act-0abcdefg (dev)

Or, manually create an INI formatted file like this:

  1. [default]
  2. token = <spotinst_token>
  3. account = <spotinst_account>

and place it in:

  • Unix/Linux/macOS:

    1. ~/.spotinst/credentials
  • Windows:

    1. %UserProfile%\.spotinst\credentials

AWS

Make sure to set up a dedicated IAM user, DNS records and cluster state storage. Please refer to setup your environment for further details.

Feature Flags

FlagDescription
+SpotinstEnables the use of the Spot integration.
+SpotinstOceanEnables the use of the Spot Ocean integration.
+SpotinstOceanTemplate - Enables the use of Spot Ocean object as a template for Virtual Node Groups.
+SpotinstHybridToggles between hybrid and full instance group implementations. Allows you to gradually integrate with Spot Ocean by continuing to use instance groups through AWS Auto Scaling groups, except for specific instance groups labeled with a predefined metadata label.
-SpotinstControllerToggles the installation of the Spot controller addon off. Please note that the feature flag must be prefixed with a minus (-) sign to set its value to false, which results in disabling the controller.

Creating a Cluster

You can add an Ocean instance group to new or existing clusters. To create a new cluster with a Ocean instance groups, run:

  1. # configure the feature flags
  2. export KOPS_FEATURE_FLAGS="Spotinst,SpotinstOcean,SpotinstOceanTemplate"
  3. # create the cluster
  4. kops create cluster --zones=us-west-2a example

Note

It’s possible to have a cluster with both Ocean-managed and unmanaged instance groups.

  1. # configure the feature flags
  2. export KOPS_FEATURE_FLAGS="Spotinst,SpotinstOcean,SpotinstHybrid"
  3. # create the instance groups
  4. kops create -f instancegroups.yaml
  1. # instancegroups.yaml
  2. # A cluster with both Ocean-managed and unmanaged instance groups.
  3. ---
  4. # Use Ocean in hybrid mode.
  5. apiVersion: kops.k8s.io/v1alpha2
  6. kind: InstanceGroup
  7. metadata:
  8. labels:
  9. kops.k8s.io/cluster: "example"
  10. spotinst.io/hybrid: "true"
  11. ...
  12. ---
  13. # Use AWS Auto Scaling group.
  14. apiVersion: kops.k8s.io/v1alpha2
  15. kind: InstanceGroup
  16. metadata:
  17. labels:
  18. kops.k8s.io/cluster: "example"
  19. ...

Creating an Instance Group

To create a new instance group, run:

  1. # configure the feature flags
  2. export KOPS_FEATURE_FLAGS="Spotinst,SpotinstOcean,SpotinstOceanTemplate"
  3. # create the instance group
  4. kops create instancegroup --role=node --name=example

To create a new instance group and have more control over the configuration, a config file can be used.

  1. # instancegroup.yaml
  2. # An instance group with Ocean configuration.
  3. ---
  4. apiVersion: kops.k8s.io/v1alpha2
  5. kind: InstanceGroup
  6. metadata:
  7. labels:
  8. kops.k8s.io/cluster: "example"
  9. spotinst.io/spot-percentage: "90"
  10. spec:
  11. image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20221206
  12. instanceMetadata:
  13. httpPutResponseHopLimit: 1
  14. httpTokens: required
  15. machineType: m5.large
  16. #define the max and min numbers of instances in the group
  17. maxSize: 3
  18. minSize: 1
  19. role: Node
  20. subnets:
  21. - us-west-2b

InstanceGroup Metadata Labels

LabelDescriptionDefault
spotinst.io/spot-percentageSpecify the percentage of Spot instances that should spin up from the target capacity.100
spotinst.io/utilize-reserved-instancesSpecify whether reserved instances should be utilized.true
spotinst.io/utilize-commitmentsSpecify whether reserved instance commitments should be utilized.none
spotinst.io/fallback-to-ondemandSpecify whether fallback to on-demand instances should be enabled.true
spotinst.io/draining-timeoutSpecify a period of time, in seconds, after a node is marked for termination during which on running pods remains active.none
spotinst.io/grace-periodSpecify a period of time, in seconds, that Ocean should wait before applying instance health checks.none
spotinst.io/ocean-default-launchspecSpecify whether to use the InstanceGroup’s spec as the default Launch Spec for the Ocean cluster.none
spotinst.io/ocean-instance-types-whitelistSpecify whether to whitelist specific instance types.none
spotinst.io/ocean-instance-types-blacklistSpecify whether to blacklist specific instance types.none
spotinst.io/ocean-instance-typesSpecify a list of instance types that should be used by the Ocean Launch Spec.none
spotinst.io/autoscaler-disabledSpecify whether the auto scaler should be disabled.false
spotinst.io/autoscaler-default-node-labelsSpecify whether default node labels should be set for the auto scaler.false
spotinst.io/autoscaler-auto-configSpecify whether headroom resources should be automatically configured and optimized.true
spotinst.io/autoscaler-auto-headroom-percentageSpecify the auto headroom percentage (a number in the range [0, 200]) which controls the percentage of headroom.none
spotinst.io/autoscaler-headroom-cpu-per-unitSpecify the number of CPUs to allocate for headroom. CPUs are denoted in millicores, where 1000 millicores = 1 vCPU.none
spotinst.io/autoscaler-headroom-gpu-per-unitSpecify the number of GPUs to allocate for headroom.none
spotinst.io/autoscaler-headroom-mem-per-unitSpecify the amount of memory (MB) to allocate for headroom.none
spotinst.io/autoscaler-headroom-num-of-unitsSpecify the number of units to retain as headroom, where each unit has the defined CPU and memory.none
spotinst.io/autoscaler-cooldownSpecify a period of time, in seconds, that Ocean should wait between scaling actions.300
spotinst.io/autoscaler-scale-down-max-percentageSpecify the maximum scale down percentage.none
spotinst.io/autoscaler-scale-down-evaluation-periodsSpecify the number of evaluation periods that should accumulate before a scale down action takes place.5
spotinst.io/autoscaler-resource-limits-max-vcpuSpecify the maximum number of virtual CPUs that can be allocated to the cluster.none
spotinst.io/autoscaler-resource-limits-max-memorySpecify the maximum amount of total physical memory (in GiB units) that can be allocated to the cluster.none
spotinst.io/restrict-scale-downSpecify whether the scale-down activities should be restricted.none
spotinst.io/other-architecture-imagesSpecify other architecture images.none

Cluster Metadata Labels

  1. # cluster.yaml
  2. # A Cluster with Ocean configuration.
  3. ---
  4. apiVersion: kops.k8s.io/v1alpha2
  5. kind: Cluster
  6. metadata:
  7. name: "example"
  8. labels:
  9. spotinst.io/strategy-cluster-spread-nodes-by: "count"
  10. spotinst.io/strategy-cluster-orientation-availability-vs-cost: "balanced"
  11. spotinst.io/resource-tag-specification-volumes: "true"
  12. ...

|| Label | Description | Default | |—-|————————————————————————————————————————————|—-| | spotinst.io/strategy-cluster-spread-nodes-by | Specify how Ocean will spread the nodes across markets by this value [vcpu,count]. | count | | spotinst.io/strategy-cluster-orientation-availability-vs-cost | Specify approach [cost,balanced,cheapest] that Ocean takes while launching nodes. | balanced | | spotinst.io/resource-tag-specification-volumes | Specify if Volume resources will be tagged with Virtual Node Group tags or Ocean tags. | false |

Documentation

If you’re new to Spot and want to get started, please checkout our Getting Started guide, available on the Spot Documentation website.

Getting Help

Please use these community resources for getting help: