Workflow Package

Motivation

SQLFlow translates a SQL program, perhaps with extended SQL syntax for AI, into a workflow. Tekton/Argo are Kubernetes native workflow engine when deploying SQLFlow on Kubernetes, SQLFlow leverages Argo/Tekton to do the workflow management.

SQLFlow supports Argo/Tekton as the workflow backend and maybe more in the future. It’s different to communicate with the theses workflow engine, they are different CRD on Kubernetes, and they have different YAML spec, so it’s necessary to organize a separate package workflow to communicate the workflows with an uniform interface.

Design

To implement the above motivation, the workflow package should include the following functionalities:

  1. CodeGen: Go interface to generate the Fluid/Argo program to generate workflow YAML.
  2. Submit/Fetch: APIs to submit the workflow and trace the status of the workflow step.

We propose the following code structure:

  1. workflow/
  2.         |-workflow.go           # workflow interface
  3.         |-argo/                 # submit/trace argo workflow via k8s API
  4.         |-tekton/               # submit/trace tekton workflow via k8s API
  5.         `-codegen/
  6.                 |-fluid/        # generate Tekton YAML using Fluid
  7.                 `-couler/       # generate Argo YAML using Couler

Workflow Codegen

Couler/Fluid lets users write Argo and Tekton workflows in Python rather than YAML. Also, the Python code is easier to read and code review. SQLFlow implements Fluid Codegen to translate the []ir.SQLFlowStmt into Python code, the interface can be like：

  1. type Codegen interface {
  2.   GenCode([]ir.SQLFlowStmt) string
  3.   GenYAML(string) string
  4. }
  • GenCode inputs a SQL program and outputs the Fluid program in Python.
  • GenYAML compiles the Fluid program and outputs the workflow YAML.

Workflow Interface

  1. type Workflow interface {
  2.   Submit(yaml string) (workflowID string, err error)
  3.   Fetch(FetchRequest) FetchResponse
  4. }
  6. func New(backend string) (Codegen, Workflow, error) {
  7.   if backend == "tekton" {
  8.     return NewCodegen("fluid"), NewWorkflow("tekton"), nil
  9.   }
  10. }
  • Submit submits the input YAML content to a Kubernetes cluster, and returns the workflow.
  • Fetch fetches the step status and query result which packaged in FetchResponse.
  • New returns the corresponding Codegen and Workflow implementation.

Execution Example

  1. // New codegen and workflow operator
  2. cg, wf, e := workflow.New("tekton")
  4. // generate YAML file
  5. py := cg.GenCode(SQLProgram)
  6. yaml := cg.GenYAML(py)
  8. // submit the workflow YAML and retrieval workflow step status
  9. wfID := wf.Submit(yaml)
  10. fetchRequest := NewFetchRequest(wfID)
  11. for {
  12.   response := wf.Fetch(fetchRequest)
  13.   // deal with response.Message, response.Rows, e.g.
  14.   // Eof means the workflow completed, break the loop
  15.   if response.Eof {
  16.     break
  17.   }
  18.   fetchRequest = response.updated_request_since
  19. }