Writing a New Stage

This doc covers the back-end changes you need to make to Orca to implement a new stage type.

Overview

To create a new stage, you need to make backend changes to orca to implement the logic of the stage, and the front-end changes to deck to implement the UI. Depending on what the stage does, you may need to implement new cloud provider-specific logic into clouddriver as well.

This doc currently only covers the backend changes made to orca.

Backend (orca)

For the backend, you need to define:

  • A stage class
  • One or more task classes associated with the stage

Stage class

A stage class must implement the com.netflix.spinnaker.orca.pipeline.StageDefinitionBuilder interface.

For providing additional functionality, it can also implement other interfaces:

  • A CancellableStage can be cancelled.
  • A RestartableStage can be restarted.
  • A CloudProviderAware stage exposes information about the cloud provider.
  • An AuthenticatedStage can perform custom authentication.

Here’s an example based on a stage that is used internally at Netflix to integrate with the Chaos Automation Platform (ChAP).

The stage is composed of two tasks:

  • beginChap starts a new ChAP run
  • monitorChap waits for the ChAP run to finish
  1. package com.netflix.spinnaker.orca.pipeline;
  3. import com.google.common.collect.ImmutableMap;
  4. import com.netflix.spinnaker.orca.CancellableStage;
  5. import com.netflix.spinnaker.orca.chap.ChapService;
  6. import com.netflix.spinnaker.orca.chap.Run;
  7. import com.netflix.spinnaker.orca.chap.tasks.BeginChapTask;
  8. import com.netflix.spinnaker.orca.chap.tasks.MonitorChapTask;
  9. import com.netflix.spinnaker.orca.pipeline.model.Execution;
  10. import com.netflix.spinnaker.orca.pipeline.model.Stage;
  11. import org.springframework.beans.factory.annotation.Autowired;
  12. import org.springframework.stereotype.Component;
  14. @Component
  15. public class ChapStage implements StageDefinitionBuilder, CancellableStage {
  17.   @Autowired
  18.   public ChapService chapService;
  20.   @Override
  21.   public <T extends Execution<T>> void taskGraph(Stage<T> stage, TaskNode.Builder builder {
  22.     builder
  23.       .withTask("beginChap", BeginChapTask.class)
  24.       .withTask("monitorChap", MonitorChapTask.class);
  25.   }
  27.   @Override
  28.   public CancellableStage.Result cancel(Stage stage) {
  29.     Run run = (Run) stage.getContext().get("run");
  30.     if (run != null) {
  31.       Run latestDetails = chapService.cancelChap(run.id.toString(), "");
  32.       return new CancellableStage.Result(stage, ImmutableMap.of("run", latestDetails));
  33.     }
  35.     return null;
  36.   }
  37. }

Task classes

A task class must implement a com.netflix.spinnaker.orca.Task , or an interface that extends it, such as:

  • A RetryableTask can be retried if it fails.
  • A PreconditionTask defines preconditions that the task will enforce.

To communicate that a task failed, throw a RuntimeException.

In our example, the ChapStage consists of two tasks:

  1. BeginChapTask
  2. MonitorChapTask

BeginChapTask

The BeginChapTask:

  1. retrieves the testCaseId specified by the user during the task configuration stage
  2. calls the ChAP service via REST API call (see ChapService.startChap)
  3. returns a DefaultTaskResult, passing it the response from the ChAP REST API call
  1. package com.netflix.spinnaker.orca.chap.tasks;
  3. import com.netflix.spinnaker.orca.*;
  4. import com.netflix.spinnaker.orca.chap.ChapService;
  5. import com.netflix.spinnaker.orca.chap.Run;
  6. import com.netflix.spinnaker.orca.pipeline.model.Stage;
  7. import org.springframework.beans.factory.annotation.Autowired;
  8. import org.springframework.stereotype.Component;
  10. import java.util.HashMap;
  11. import java.util.Map;
  12. import java.util.concurrent.TimeUnit;
  14. @Component
  15. public class BeginChapTask implements RetryableTask {
  17.   @Override
  18.   public TaskResult execute(Stage stage) {
  19.     Map<String, Object> ctx = stage.getContext();
  20.     Object testCaseId = ctx.get("testCaseId");
  22.     if(testCaseId == null || !(testCaseId instanceof String)) {
  23.       throw new RuntimeException("Cannot begin ChAP experiment without a testCaseId.");
  24.     }
  26.     Map<String, Object> params = new HashMap<>();
  27.     params.put("testCaseId", testCaseId);
  28.     Run chapRun = chapService.startChap(params);
  30.     Map<String, Object> map = new HashMap<>();
  31.     map.put("run", chapRun);
  32.     return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  33.   }
  35.   public ChapService getChapService() {
  36.     return chapService;
  37.   }
  39.   public void setChapService(ChapService chapService) {
  40.     this.chapService = chapService;
  41.   }
  43.   @Autowired
  44.   private ChapService chapService;
  46.   @Override
  47.   public long getBackoffPeriod() {
  48.     return TimeUnit.SECONDS.toMillis(5);
  49.   }
  51.   @Override
  52.   public long getTimeout() {
  53.     return TimeUnit.MINUTES.toMillis(1);
  54.   }
  55. }

MonitorChapTask

The MonitorChapTask polls the ChAP service for the status of the ChAP run.

  1. package com.netflix.spinnaker.orca.chap.tasks;
  3. import com.fasterxml.jackson.databind.ObjectMapper;
  4. import com.netflix.spinnaker.orca.DefaultTaskResult;
  5. import com.netflix.spinnaker.orca.ExecutionStatus;
  6. import com.netflix.spinnaker.orca.RetryableTask;
  7. import com.netflix.spinnaker.orca.TaskResult;
  8. import com.netflix.spinnaker.orca.chap.ChapService;
  9. import com.netflix.spinnaker.orca.chap.Run;
  10. import com.netflix.spinnaker.orca.pipeline.model.Stage;
  11. import org.springframework.beans.factory.annotation.Autowired;
  12. import org.springframework.stereotype.Component;
  14. import java.util.HashMap;
  15. import java.util.Map;
  16. import java.util.concurrent.TimeUnit;
  18. @Component
  19. public class MonitorChapTask implements RetryableTask {
  21.   @Autowired
  22.   private ObjectMapper objectMapper;
  24.   @Autowired
  25.   public ChapService chapService;
  27.   @Override
  28.   public TaskResult execute(Stage stage) {
  29.     Map<String, Object> ctx = stage.getContext();
  31.     Run run = objectMapper.convertValue(ctx.get("run"), Run.class);
  33.     if (run == null) {
  34.       throw new RuntimeException("Cannot monitor Chap task without a valid Run in the context.");
  35.     }
  37.     Run latestDetails = chapService.getChap(run.id.toString());
  39.     Map<String, Object> map = new HashMap<>();
  40.     map.put("run", latestDetails);
  42.     if(latestDetails.outcome == Run.Outcome.PASSED){
  43.       return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  44.     }
  46.     ExecutionStatus status;
  48.     switch (latestDetails.state) {
  49.       case COMPLETED:
  50.         //workflow is complete, but the outcome didnt pass, consider this a failure.
  51.       case FAILED:
  52.         throw new RuntimeException("ChAP experiment failed.");
  53.       case CANCELLED:
  54.         status = ExecutionStatus.CANCELED;
  55.         break;
  56.       default:
  57.         status = ExecutionStatus.RUNNING;
  58.         break;
  59.     }
  62.     return new DefaultTaskResult(status, map);
  63.   }
  65.   public ChapService getChapService() {
  66.     return chapService;
  67.   }
  69.   public void setChapService(ChapService chapService) {
  70.     this.chapService = chapService;
  71.   }
  73.   @Override
  74.   public long getBackoffPeriod() {
  75.     return TimeUnit.MINUTES.toMillis(1);
  76.   }
  78.   @Override
  79.   public long getTimeout() {
  80.     return TimeUnit.DAYS.toMillis(1);
  81.   }
  83.   public ObjectMapper getObjectMapper() {
  84.     return objectMapper;
  85.   }
  87.   public void setObjectMapper(ObjectMapper objectMapper) {
  88.     this.objectMapper = objectMapper;
  89.   }
  90. }

Other classes used

The details of the com.netflix.spinnaker.orca.chap.Run class and com.netflix.spinnaker.orca.chap.ChapService interface aren’t directly relevant to learning how to write a Spinnaker stage, but for completeness, here’s what those look like for this case:

Run

The Run class is a Java object that is deserialized from JSON.

  1. package com.netflix.spinnaker.orca.chap;
  3. import com.fasterxml.jackson.annotation.JsonAnyGetter;
  4. import com.fasterxml.jackson.annotation.JsonAnySetter;
  5. import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
  7. import java.util.HashMap;
  8. import java.util.Map;
  9. import java.util.UUID;
  11. @JsonIgnoreProperties(ignoreUnknown = true)
  12. public class Run {
  14.   public UUID id;
  15.   // Other properties not shown
  16.   // ...
  18.   // Support arbitrary properties without needing to define them explicitly
  19.   public Map<String, Object> properties = new HashMap<>();
  21.   @JsonAnySetter
  22.   public void set(String fieldName, Object value) {
  23.     this.properties.put(fieldName, value);
  24.   }
  26.   @JsonAnyGetter
  27.   public Object get(String fieldName) {
  28.     return this.properties.get(fieldName);
  29.   }
  30. }

ChapService

The ChapService defines a REST client API for talking to the ChAP service. It uses the Retrofit library.

  1. package com.netflix.spinnaker.orca.chap;
  3. import retrofit.http.Body;
  4. import retrofit.http.GET;
  5. import retrofit.http.POST;
  6. import retrofit.http.Path;
  8. import java.util.Map;
  10. public interface ChapService {
  11.   @POST("/v1/runs")
  12.   Run startChap(@Body Map params);
  14.   @GET("/v1/runs/{id}")
  15.   Run getChap(@Path("id") String id);
  17.   @POST("/v1/runs/{id}/stop")
  18.   Run cancelChap(@Path("id") String id, @Body String body);
  19. }

ChapConfig

To implement the ChapService interface, we do not define a class that extends the interface. Instead, we define a class named ChapConfig with a Spring @Configuration annotation. Note that this implementation uses the chap.baseUrl configuration value that is defined in a separate Spinnaker configuration file.

  1. package com.netflix.spinnaker.config;
  3. import com.fasterxml.jackson.databind.ObjectMapper;
  4. import com.netflix.spinnaker.orca.chap.ChapService;
  5. import com.netflix.spinnaker.retrofit.Slf4jRetrofitLogger;
  6. import org.springframework.beans.factory.annotation.Autowired;
  7. import org.springframework.beans.factory.annotation.Value;
  8. import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
  9. import org.springframework.context.annotation.Bean;
  10. import org.springframework.context.annotation.ComponentScan;
  11. import org.springframework.context.annotation.Configuration;
  12. import retrofit.Endpoint;
  13. import retrofit.RestAdapter;
  14. import retrofit.client.Client;
  15. import retrofit.converter.JacksonConverter;
  17. import static retrofit.Endpoints.newFixedEndpoint;
  19. @Configuration
  20. @ComponentScan({
  21.   "com.netflix.spinnaker.orca.chap.pipeline",
  22.   "com.netflix.spinnaker.orca.chap.tasks"
  23. })
  24. @ConditionalOnProperty(value = "chap.baseUrl")
  25. public class ChapConfig {
  27.   @Bean
  28.   Endpoint chapEndpoint(@Value("${chap.baseUrl}") String chapBaseUrl) {
  29.     return newFixedEndpoint(chapBaseUrl);
  30.   }
  32.   @Bean
  33.   ChapService chapService(Endpoint chapEndpoint,
  34.                           Client retrofitClient,
  35.                           RestAdapter.LogLevel retrofitLogLevel,
  36.                           ObjectMapper objectMapper) {
  37.     return new RestAdapter.Builder()
  38.       .setEndpoint(chapEndpoint)
  39.       .setClient(retrofitClient)
  40.       .setLogLevel(retrofitLogLevel)
  41.       .setLog(new Slf4jRetrofitLogger(ChapService.class))
  42.       .setConverter(new JacksonConverter(objectMapper))
  43.       .build()
  44.       .create(ChapService.class);
  45.   }
  46. }

Last modified June 24, 2021: Redesign Progress (#83) (8231bcf)