SCM Provider Generator

The SCM Provider generator uses the API of an SCMaaS provider (eg GitHub) to automatically discover repositories within an organization. This fits well with GitOps layout patterns that split microservices across many repositories.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       # Which protocol to clone using.
  9.       cloneProtocol: ssh
  10.       # See below for provider specific options.
  11.       github:
  12.         # ...
  • cloneProtocol: Which protocol to use for the SCM URL. Default is provider-specific but ssh if possible. Not all providers necessarily support all protocols, see provider documentation below for available options.

Note

Know the security implications of using SCM generators. Only admins may create ApplicationSets to avoid leaking Secrets, and only admins may create repos/branches if the project field of an ApplicationSet with an SCM generator is templated, to avoid granting management of out-of-bounds resources.

GitHub

The GitHub mode uses the GitHub API to scan an organization in either github.com or GitHub Enterprise.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       github:
  9.         # The GitHub organization to scan.
  10.         organization: myorg
  11.         # For GitHub Enterprise:
  12.         api: https://git.example.com/
  13.         # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
  14.         allBranches: true
  15.         # Reference to a Secret containing an access token. (optional)
  16.         tokenRef:
  17.           secretName: github-token
  18.           key: token
  19.         # (optional) use a GitHub App to access the API instead of a PAT.
  20.         appSecretName: gh-app-repo-creds
  21.   template:
  22.   # ...
  • organization: Required name of the GitHub organization to scan. If you have multiple organizations, use multiple generators.
  • api: If using GitHub Enterprise, the URL to access it.
  • allBranches: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a branchMatch filter.
  • tokenRef: A Secret name and key containing the GitHub access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
  • appSecretName: A Secret name containing a GitHub App secret in repo-creds format.

For label filtering, the repository topics are used.

Available clone protocols are ssh and https.

Gitlab

The GitLab mode uses the GitLab API to scan and organization in either gitlab.com or self-hosted GitLab.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       gitlab:
  9.         # The base GitLab group to scan.  You can either use the group id or the full namespaced path.
  10.         group: "8675309"
  11.         # For self-hosted GitLab:
  12.         api: https://gitlab.example.com/
  13.         # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
  14.         allBranches: true
  15.         # If true, recurses through subgroups. If false, it searches only in the base group. Defaults to false.
  16.         includeSubgroups: true
  17.         # Reference to a Secret containing an access token. (optional)
  18.         tokenRef:
  19.           secretName: gitlab-token
  20.           key: token
  21.   template:
  22.   # ...
  • group: Required name of the base GitLab group to scan. If you have multiple base groups, use multiple generators.
  • api: If using self-hosted GitLab, the URL to access it.
  • allBranches: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a branchMatch filter.
  • includeSubgroups: By default (false) the controller will only search for repos directly in the base group. If this is true, it will recurse through all the subgroups searching for repos to scan.
  • tokenRef: A Secret name and key containing the GitLab access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.

For label filtering, the repository tags are used.

Available clone protocols are ssh and https.

Gitea

The Gitea mode uses the Gitea API to scan organizations in your instance

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       gitea:
  9.         # The Gitea owner to scan.
  10.         owner: myorg
  11.         # The Gitea instance url
  12.         api: https://gitea.mydomain.com/
  13.         # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
  14.         allBranches: true
  15.         # Reference to a Secret containing an access token. (optional)
  16.         tokenRef:
  17.           secretName: gitea-token
  18.           key: token
  19.   template:
  20.   # ...
  • owner: Required name of the Gitea organization to scan. If you have multiple organizations, use multiple generators.
  • api: The URL of the Gitea instance you are using.
  • allBranches: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a branchMatch filter.
  • tokenRef: A Secret name and key containing the Gitea access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.
  • insecure: Allow for self-signed TLS certificates.

This SCM provider does not yet support label filtering

Available clone protocols are ssh and https.

Bitbucket Server

Use the Bitbucket Server API (1.0) to scan repos in a project. Note that Bitbucket Server is not to same as Bitbucket Cloud (API 2.0)

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       bitbucketServer:
  9.         project: myproject
  10.         # URL of the Bitbucket Server. Required.
  11.         api: https://mycompany.bitbucket.org
  12.         # If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
  13.         allBranches: true
  14.         # Credentials for Basic authentication. Required for private repositories.
  15.         basicAuth:
  16.           # The username to authenticate with
  17.           username: myuser
  18.           # Reference to a Secret containing the password or personal access token.
  19.           passwordRef:
  20.             secretName: mypassword
  21.             key: password
  22.         # Support for filtering by labels is TODO. Bitbucket server labels are not supported for PRs, but they are for repos
  23.   template:
  24.   # ...
  • project: Required name of the Bitbucket project
  • api: Required URL to access the Bitbucket REST api.
  • allBranches: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a branchMatch filter.

If you want to access a private repository, you must also provide the credentials for Basic auth (this is the only auth supported currently): * username: The username to authenticate with. It only needs read access to the relevant repo. * passwordRef: A Secret name and key containing the password or personal access token to use for requests.

Available clone protocols are ssh and https.

Azure DevOps

Uses the Azure DevOps API to look up eligible repositories based on a team project within an Azure DevOps organization. The default Azure DevOps URL is https://dev.azure.com, but this can be overridden with the field azureDevOps.api.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       azureDevOps:
  9.         # The Azure DevOps organization.
  10.         organization: myorg
  11.         # URL to Azure DevOps. Optional. Defaults to https://dev.azure.com.
  12.         api: https://dev.azure.com
  13.         # If true, scan every branch of eligible repositories. If false, check only the default branch of the eligible repositories. Defaults to false.
  14.         allBranches: true
  15.         # The team project within the specified Azure DevOps organization.
  16.         teamProject: myProject
  17.         # Reference to a Secret containing the Azure DevOps Personal Access Token (PAT) used for accessing Azure DevOps.
  18.         accessTokenRef:
  19.           secretName: azure-devops-scm
  20.           key: accesstoken
  21.   template:
  22.   # ...
  • organization: Required. Name of the Azure DevOps organization.
  • teamProject: Required. The name of the team project within the specified organization.
  • accessTokenRef: Required. A Secret name and key containing the Azure DevOps Personal Access Token (PAT) to use for requests.
  • api: Optional. URL to Azure DevOps. If not set, https://dev.azure.com is used.
  • allBranches: Optional, default false. If true, scans every branch of eligible repositories. If false, check only the default branch of the eligible repositories.

Bitbucket Cloud

The Bitbucket mode uses the Bitbucket API V2 to scan a workspace in bitbucket.org.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       bitbucket:
  9.         # The workspace id (slug).  
  10.         owner: "example-owner"
  11.         # The user to use for basic authentication with an app password.
  12.         user: "example-user"
  13.         # If true, scan every branch of every repository. If false, scan only the main branch. Defaults to false.
  14.         allBranches: true
  15.         # Reference to a Secret containing an app password.
  16.         appPasswordRef:
  17.           secretName: appPassword
  18.           key: password
  19.   template:
  20.   # ...
  • owner: The workspace ID (slug) to use when looking up repositories.
  • user: The user to use for authentication to the Bitbucket API V2 at bitbucket.org.
  • allBranches: By default (false) the template will only be evaluated for the main branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a branchMatch filter.
  • appPasswordRef: A Secret name and key containing the bitbucket app password to use for requests.

This SCM provider does not yet support label filtering

Available clone protocols are ssh and https.

Filters

Filters allow selecting which repositories to generate for. Each filter can declare one or more conditions, all of which must pass. If multiple filters are present, any can match for a repository to be included. If no filters are specified, all repositories will be processed.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.       filters:
  9.       # Include any repository starting with "myapp" AND including a Kustomize config AND labeled with "deploy-ok" ...
  10.       - repositoryMatch: ^myapp
  11.         pathsExist: [kubernetes/kustomization.yaml]
  12.         labelMatch: deploy-ok
  13.       # ... OR include any repository starting with "otherapp" AND a Helm folder and doesn't have file disabledrepo.txt.
  14.       - repositoryMatch: ^otherapp
  15.         pathsExist: [helm]
  16.         pathsDoNotExist: [disabledrepo.txt]
  17.   template:
  18.   # ...
  • repositoryMatch: A regexp matched against the repository name.
  • pathsExist: An array of paths within the repository that must exist. Can be a file or directory.
  • pathsDoNotExist: An array of paths within the repository that must not exist. Can be a file or directory.
  • labelMatch: A regexp matched against repository labels. If any label matches, the repository is included.
  • branchMatch: A regexp matched against branch names.

Template

As with all generators, several parameters are generated for use within the ApplicationSet resource template.

  1. apiVersion: argoproj.io/v1alpha1
  2. kind: ApplicationSet
  3. metadata:
  4.   name: myapps
  5. spec:
  6.   generators:
  7.   - scmProvider:
  8.     # ...
  9.   template:
  10.     metadata:
  11.       name: '{{ repository }}'
  12.     spec:
  13.       source:
  14.         repoURL: '{{ url }}'
  15.         targetRevision: '{{ branch }}'
  16.         path: kubernetes/
  17.       project: default
  18.       destination:
  19.         server: https://kubernetes.default.svc
  20.         namespace: default
  • organization: The name of the organization the repository is in.
  • repository: The name of the repository.
  • url: The clone URL for the repository.
  • branch: The default branch of the repository.
  • sha: The Git commit SHA for the branch.
  • short_sha: The abbreviated Git commit SHA for the branch (8 chars or the length of the sha if it’s shorter).
  • labels: A comma-separated list of repository labels.
  • branchNormalized: The value of branch normalized to contain only lowercase alphanumeric characters, ‘-‘ or ‘.’.