Implementing a controller

The basic logic of our CronJob controller is this:

• Load the named CronJob

• List all active jobs, and update the status

• Clean up old jobs according to the history limits

• Check if we’re suspended (and don’t do anything else if we are)

• Get the next scheduled run

• Run a new job if it’s on schedule, not past the deadline, and notblocked by our concurrency policy

• Requeue when we either see a running job (done automatically) or it’stime for the next scheduled run.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”);you may not use this file except in compliance with the License.You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an “AS IS” BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. We’ll start out with some imports. You’ll see below that we’ll need a few more importsthan those scaffolded for us. We’ll talk about each one when we use it.

package controllers
import (
    "context"
    "fmt"
    "sort"
    "time"
    "github.com/go-logr/logr"
    "github.com/robfig/cron"
    kbatch "k8s.io/api/batch/v1"
    corev1 "k8s.io/api/core/v1"
    apierrs "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ref "k8s.io/client-go/tools/reference"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    batch "tutorial.kubebuilder.io/project/api/v1"
)

Next, we’ll need a few more bits in our Reconciler:

We’ll need the Scheme, in order to call some helpers that set owner references,and we’ll need a Clock, which will allow us to fake timing in our tests.

// CronJobReconciler reconciles a CronJob object
type CronJobReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
    Clock
}

ClockWe’ll mock out the clock to make it easier to jump around in time while testing,the “real” clock just calls time.Now.

type realClock struct{}
func (_ realClock) Now() time.Time { return time.Now() }
// clock knows how to get the current time.
// It can be used to fake out timing for testing.
type Clock interface {
    Now() time.Time
}

ignoreNotFoundWe generally want to ignore (not requeue) NotFound errors, since we’ll get areconciliation request once the object exists, and requeuing in the meantimewon’t help.

func ignoreNotFound(err error) error {
    if apierrs.IsNotFound(err) {
        return nil
    }
    return err
}

Notice that we need a few more RBAC permissions — since we’re creating andmanaging jobs now, we’ll need permissions for those.

// +kubebuilder:rbac:groups=batch.tutorial.kubebuilder.io,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=batch.tutorial.kubebuilder.io,resources=cronjobs/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=batch,resources=jobs/status,verbs=get

Now, we get to the heart of the controller — the reconciler logic.

var (
    scheduledTimeAnnotation = "batch.tutorial.kubebuilder.io/scheduled-at"
)
func (r *CronJobReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("cronjob", req.NamespacedName)

1: Load the CronJob by name

We’ll fetch the CronJob using our client. All client methods take acontext (to allow for cancellation) as their first argument, and the objectin question as their last. Get is a bit special, in that it takes aNamespacedNameas the middle argument (most don’t have a middle argument, as we’ll seebelow).

Many client methods also take variadic options at the end.

    var cronJob batch.CronJob
    if err := r.Get(ctx, req.NamespacedName, &cronJob); err != nil {
        log.Error(err, "unable to fetch CronJob")
        // we'll ignore not-found errors, since they can't be fixed by an immediate
        // requeue (we'll need to wait for a new notification), and we can get them
        // on deleted requests.
        return ctrl.Result{}, ignoreNotFound(err)
    }

2: List all active jobs, and update the status

To fully update our status, we’ll need to list all child jobs in this namespace that belong to this CronJob.Similarly to Get, we can use the List method to list the child jobs. Notice that we use variadic options toset the namespace and field match (which is actually an index lookup that we set up below).

    var childJobs kbatch.JobList
    if err := r.List(ctx, &childJobs, client.InNamespace(req.Namespace), client.MatchingField(jobOwnerKey, req.Name)); err != nil {
        log.Error(err, "unable to list child Jobs")
        return ctrl.Result{}, err
    }

Once we have all the jobs we own, we’ll split them into active, successful,and failed jobs, keeping track of the most recent run so that we can record itin status. Remember, status should be able to be reconstituted from the stateof the world, so it’s generally not a good idea to read from the status of theroot object. Instead, you should reconstruct it every run. That’s what we’lldo here.

We can check if a job is “finished” and whether it succeeded or failed using statusconditions. We’ll put that logic in a helper to make our code cleaner.

    // find the active list of jobs
    var activeJobs []*kbatch.Job
    var successfulJobs []*kbatch.Job
    var failedJobs []*kbatch.Job
    var mostRecentTime *time.Time // find the last run so we can update the status

isJobFinishedWe consider a job “finished” if it has a “succeeded” or “failed” condition marked as true.Status conditions allow us to add extensible status information to our objects that otherhumans and controllers can examine to check things like completion and health.

    isJobFinished := func(job *kbatch.Job) (bool, kbatch.JobConditionType) {
        for _, c := range job.Status.Conditions {
            if (c.Type == kbatch.JobComplete || c.Type == kbatch.JobFailed) && c.Status == corev1.ConditionTrue {
                return true, c.Type
            }
        }
        return false, ""
    }

getScheduledTimeForJobWe’ll use a helper to extract the scheduled time from the annotation thatwe added during job creation.

    getScheduledTimeForJob := func(job *kbatch.Job) (*time.Time, error) {
        timeRaw := job.Annotations[scheduledTimeAnnotation]
        if len(timeRaw) == 0 {
            return nil, nil
        }
        timeParsed, err := time.Parse(time.RFC3339, timeRaw)
        if err != nil {
            return nil, err
        }
        return &timeParsed, nil
    }

    for i, job := range childJobs.Items {
        _, finishedType := isJobFinished(&job)
        switch finishedType {
        case "": // ongoing
            activeJobs = append(activeJobs, &childJobs.Items[i])
        case kbatch.JobFailed:
            failedJobs = append(failedJobs, &childJobs.Items[i])
        case kbatch.JobComplete:
            successfulJobs = append(successfulJobs, &childJobs.Items[i])
        }
        // We'll store the launch time in an annotation, so we'll reconsitute that from
        // the active jobs themselves.
        scheduledTimeForJob, err := getScheduledTimeForJob(&job)
        if err != nil {
            log.Error(err, "unable to parse schedule time for child job", "job", &job)
            continue
        }
        if scheduledTimeForJob != nil {
            if mostRecentTime == nil {
                mostRecentTime = scheduledTimeForJob
            } else if mostRecentTime.Before(*scheduledTimeForJob) {
                mostRecentTime = scheduledTimeForJob
            }
        }
    }
    if mostRecentTime != nil {
        cronJob.Status.LastScheduleTime = &metav1.Time{Time: *mostRecentTime}
    } else {
        cronJob.Status.LastScheduleTime = nil
    }
    cronJob.Status.Active = nil
    for _, activeJob := range activeJobs {
        jobRef, err := ref.GetReference(r.Scheme, activeJob)
        if err != nil {
            log.Error(err, "unable to make reference to active job", "job", activeJob)
            continue
        }
        cronJob.Status.Active = append(cronJob.Status.Active, *jobRef)
    }

Here, we’ll log how many jobs we observed at a slightly higher logging level,for debugging. Notice how instead of using a format string, we use a fixed message,and attach key-value pairs with the extra information. This makes it easier tofilter and query log lines.

    log.V(1).Info("job count", "active jobs", len(activeJobs), "successful jobs", len(successfulJobs), "failed jobs", len(failedJobs))

Using the date we’ve gathered, we’ll update the status of our CRD.Just like before, we use our client. To specifically update the statussubresource, we’ll use the Status part of the client, with the Updatemethod.

The status subresource ignores changes to spec, so it’s less likely to conflictwith any other updates, and can have separate permissions.

    if err := r.Status().Update(ctx, &cronJob); err != nil {
        log.Error(err, "unable to update CronJob status")
        return ctrl.Result{}, err
    }

Once we’ve updated our status, we can move on to ensuring that the status ofthe world matches what we want in our spec.

3: Clean up old jobs according to the history limit

First, we’ll try to clean up old jobs, so that we don’t leave too many lyingaround.

    // NB: deleting these is "best effort" -- if we fail on a particular one,
    // we won't requeue just to finish the deleting.
    if cronJob.Spec.FailedJobsHistoryLimit != nil {
        sort.Slice(failedJobs, func(i, j int) bool {
            if failedJobs[i].Status.StartTime == nil {
                return failedJobs[j].Status.StartTime != nil
            }
            return failedJobs[i].Status.StartTime.Before(failedJobs[j].Status.StartTime)
        })
        for i, job := range failedJobs {
            if err := r.Delete(ctx, job); err != nil {
                log.Error(err, "unable to delete old failed job", "job", job)
            }
            if int32(i) >= *cronJob.Spec.FailedJobsHistoryLimit {
                break
            }
        }
    }
    if cronJob.Spec.SuccessfulJobsHistoryLimit != nil {
        sort.Slice(successfulJobs, func(i, j int) bool {
            if successfulJobs[i].Status.StartTime == nil {
                return successfulJobs[j].Status.StartTime != nil
            }
            return successfulJobs[i].Status.StartTime.Before(successfulJobs[j].Status.StartTime)
        })
        for i, job := range successfulJobs {
            if err := r.Delete(ctx, job); err != nil {
                log.Error(err, "unable to delete old successful job", "job", job)
            }
            if int32(i) >= *cronJob.Spec.SuccessfulJobsHistoryLimit {
                break
            }
        }
    }

4: Check if we’re suspended

If this object is suspended, we don’t want to run any jobs, so we’ll stop now.This is useful if something’s broken with the job we’re running and we want topause runs to investigate or putz with the cluster, without deleting the object.

    if cronJob.Spec.Suspend != nil && *cronJob.Spec.Suspend {
        log.V(1).Info("cronjob suspended, skipping")
        return ctrl.Result{}, nil
    }

5: Get the next scheduled run

If we’re not paused, we’ll need to calculate the next scheduled run, and whetheror not we’ve got a run that we haven’t processed yet. getNextScheduleWe’ll calculate the next scheduled time using our helpful cron library.We’ll start calculating appropriate times from our last run, or the creationof the CronJob if we can’t find a last run.

If there are too many missed runs and we don’t have any deadlines set, we’llbail so that we don’t cause issues on controller restarts or wedges.

Otherwise, we’ll just return the missed runs (of which we’ll just use the latest),and the next run, so that we can know when it’s time to reconcile again.

    getNextSchedule := func(cronJob *batch.CronJob, now time.Time) (lastMissed *time.Time, next time.Time, err error) {
        sched, err := cron.ParseStandard(cronJob.Spec.Schedule)
        if err != nil {
            return nil, time.Time{}, fmt.Errorf("Unparseable schedule %q: %v", cronJob.Spec.Schedule, err)
        }
        // for optimization purposes, cheat a bit and start from our last observed run time
        // we could reconstitute this here, but there's not much point, since we've
        // just updated it.
        var earliestTime time.Time
        if cronJob.Status.LastScheduleTime != nil {
            earliestTime = cronJob.Status.LastScheduleTime.Time
        } else {
            earliestTime = cronJob.ObjectMeta.CreationTimestamp.Time
        }
        if cronJob.Spec.StartingDeadlineSeconds != nil {
            // controller is not going to schedule anything below this point
            schedulingDeadline := now.Add(-time.Second * time.Duration(*cronJob.Spec.StartingDeadlineSeconds))
            if schedulingDeadline.After(earliestTime) {
                earliestTime = schedulingDeadline
            }
        }
        if earliestTime.After(now) {
            return nil, sched.Next(now), nil
        }
        starts := 0
        for t := sched.Next(earliestTime); !t.After(now); t = sched.Next(t) {
            lastMissed = &t
            // An object might miss several starts. For example, if
            // controller gets wedged on Friday at 5:01pm when everyone has
            // gone home, and someone comes in on Tuesday AM and discovers
            // the problem and restarts the controller, then all the hourly
            // jobs, more than 80 of them for one hourly scheduledJob, should
            // all start running with no further intervention (if the scheduledJob
            // allows concurrency and late starts).
            //
            // However, if there is a bug somewhere, or incorrect clock
            // on controller's server or apiservers (for setting creationTimestamp)
            // then there could be so many missed start times (it could be off
            // by decades or more), that it would eat up all the CPU and memory
            // of this controller. In that case, we want to not try to list
            // all the missed start times.
            starts++
            if starts > 100 {
                // We can't get the most recent times so just return an empty slice
                return nil, time.Time{}, fmt.Errorf("Too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.")
            }
        }
        return lastMissed, sched.Next(now), nil
    }

    // figure out the next times that we need to create
    // jobs at (or anything we missed).
    missedRun, nextRun, err := getNextSchedule(&cronJob, r.Now())
    if err != nil {
        log.Error(err, "unable to figure out CronJob schedule")
        // we don't really care about requeuing until we get an update that
        // fixes the schedule, so don't return an error
        return ctrl.Result{}, nil
    }

We’ll prep our eventual request to requeue until the next job, and then figureout if we actually need to run.

    scheduledResult := ctrl.Result{RequeueAfter: nextRun.Sub(r.Now())} // save this so we can re-use it elsewhere
    log = log.WithValues("now", r.Now(), "next run", nextRun)

6: Run a new job if it’s on schedule, not past the deadline, and not blocked by our concurrency policy

If we’ve missed a run, and we’re still within the deadline to start it, we’ll need to run a job.

    if missedRun == nil {
        log.V(1).Info("no upcoming scheduled times, sleeping until next")
        return scheduledResult, nil
    }
    // make sure we're not too late to start the run
    log = log.WithValues("current run", missedRun)
    tooLate := false
    if cronJob.Spec.StartingDeadlineSeconds != nil {
        tooLate = missedRun.Add(time.Duration(*cronJob.Spec.StartingDeadlineSeconds) * time.Second).Before(r.Now())
    }
    if tooLate {
        log.V(1).Info("missed starting deadline for last run, sleeping till next")
        // TODO(directxman12): events
        return scheduledResult, nil
    }

If we actually have to run a job, we’ll need to either wait till existing ones finish,replace the existing ones, or just add new ones. If our information is out of date dueto cache delay, we’ll get a requeue when we get up-to-date information.

    // figure out how to run this job -- concurrency policy might forbid us from running
    // multiple at the same time...
    if cronJob.Spec.ConcurrencyPolicy == batch.ForbidConcurrent && len(activeJobs) > 0 {
        log.V(1).Info("concurrency policy blocks concurrent runs, skipping", "num active", len(activeJobs))
        return scheduledResult, nil
    }
    // ...or instruct us to replace existing ones...
    if cronJob.Spec.ConcurrencyPolicy == batch.ReplaceConcurrent {
        for _, activeJob := range activeJobs {
            // we don't care if the job was already deleted
            if err := r.Delete(ctx, activeJob); ignoreNotFound(err) != nil {
                log.Error(err, "unable to delete active job", "job", activeJob)
                return ctrl.Result{}, err
            }
        }
    }

Once we’ve figured out what to do with existing jobs, we’ll actually create our desired job constructJobForCronJobWe need to construct a job based on our CronJob’s template. We’ll copy over the specfrom the template and copy some basic object meta.

Then, we’ll set the “scheduled time” annotation so that we can reconstitute ourLastScheduleTime field each reconcile.

Finally, we’ll need to set an owner reference. This allows the Kubernetes garbage collectorto clean up jobs when we delete the CronJob, and allows controller-runtime to figure outwhich cronjob needs to be reconciled when a given job changes (is added, deleted, completes, etc).

    constructJobForCronJob := func(cronJob *batch.CronJob, scheduledTime time.Time) (*kbatch.Job, error) {
        // We want job names for a given nominal start time to have a deterministic name to avoid the same job being created twice
        name := fmt.Sprintf("%s-%d", cronJob.Name, scheduledTime.Unix())
        job := &kbatch.Job{
            ObjectMeta: metav1.ObjectMeta{
                Labels:      make(map[string]string),
                Annotations: make(map[string]string),
                Name:        name,
                Namespace:   cronJob.Namespace,
            },
            Spec: *cronJob.Spec.JobTemplate.Spec.DeepCopy(),
        }
        for k, v := range cronJob.Spec.JobTemplate.Annotations {
            job.Annotations[k] = v
        }
        job.Annotations[scheduledTimeAnnotation] = scheduledTime.Format(time.RFC3339)
        for k, v := range cronJob.Spec.JobTemplate.Labels {
            job.Labels[k] = v
        }
        if err := ctrl.SetControllerReference(cronJob, job, r.Scheme); err != nil {
            return nil, err
        }
        return job, nil
    }

    // actually make the job...
    job, err := constructJobForCronJob(&cronJob, *missedRun)
    if err != nil {
        log.Error(err, "unable to construct job from template")
        // don't bother requeuing until we get a change to the spec
        return scheduledResult, nil
    }
    // ...and create it on the cluster
    if err := r.Create(ctx, job); err != nil {
        log.Error(err, "unable to create Job for CronJob", "job", job)
        return ctrl.Result{}, err
    }
    log.V(1).Info("created Job for CronJob run", "job", job)

7: Requeue when we either see a running job or it’s time for the next scheduled run

Finally, we’ll return the result that we prepped above, that says we want to requeuewhen our next run would need to occur. This is taken as a maximum deadline — if somethingelse changes in between, like our job starts or finishes, we get modified, etc, we mightreconcile again sooner.

    // we'll requeue once we see the running job, and update our status
    return scheduledResult, nil
}

Setup

Finally, we’ll update our setup. In order to allow our reconciler to quicklylook up Jobs by their owner, we’ll need an index. We declare an index key thatwe can later use with the client as a pseudo-field name, and then describe how toextract the indexed value from the Job object. The indexer will automatically takecare of namespaces for us, so we just have to extract the owner name if the Job hasa CronJob owner.

Additionally, we’ll inform the manager that this controller owns some Jobs, so that itwill automatically call Reconcile on the underlying CronJob when a Job changes, isdeleted, etc.

var (
    jobOwnerKey = ".metadata.controller"
    apiGVStr    = batch.GroupVersion.String()
)
func (r *CronJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
    // set up a real clock, since we're not in a test
    if r.Clock == nil {
        r.Clock = realClock{}
    }
    if err := mgr.GetFieldIndexer().IndexField(&kbatch.Job{}, jobOwnerKey, func(rawObj runtime.Object) []string {
        // grab the job object, extract the owner...
        job := rawObj.(*kbatch.Job)
        owner := metav1.GetControllerOf(job)
        if owner == nil {
            return nil
        }
        // ...make sure it's a CronJob...
        if owner.APIVersion != apiGVStr || owner.Kind != "CronJob" {
            return nil
        }
        // ...and if so, return it
        return []string{owner.Name}
    }); err != nil {
        return err
    }
    return ctrl.NewControllerManagedBy(mgr).
        For(&batch.CronJob{}).
        Owns(&kbatch.Job{}).
        Complete(r)
}

That was a doozy, but now we’ve got a working controller. Let’s testagainst the cluster, then, if we don’t have any issues, deploy it!