12

EDIT: Question is solved, it was my mistake, i simply used the wrong cron settings. I assumed "* 2 * * *" would only run once per day at 2, but in fact it runs every minute past the hour 2. So Kubernetes behaves correctly.

I keep having multiple jobs running at one cron execution point. But it seems only if those jobs have a very short runtime. Any idea why this happens and how I can prevent it? I use concurrencyPolicy: Forbid, backoffLimit: 0 and restartPolicy: Never.

Example for a cron job that is supposed to run once per day, but runs multiple times just after its scheduled run time: 

job-1554346620                   1/1           11s        4h42m   
job-1554346680                   1/1           11s        4h41m                     
job-1554346740                   1/1           10s        4h40m

Relevant config:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: job
spec:
  schedule: "* 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: job
              image: job_image:latest
              command: ["rake", "run_job"]
          restartPolicy: Never
          imagePullSecrets:
            - name: regcred
      backoffLimit: 0
Improve this question

2 Answers 2

Reset to default
21

The most common problem of running CronJobs on k8s is:

spawning to many pods which consume all cluster resources

It is very important to set proper CronJob limitations

If you are not sure what you need - just take this example as a template:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-first-conjob
  namespace: devenv-admitriev
spec:
  schedule: "*/10 * * * *" # MM HH DD MM WKD -- Minutes, Hour, Day, Month, Weekday (eg. Sun, Mon)
  successfulJobsHistoryLimit: 3 # how many completed jobs should be kept
  failedJobsHistoryLimit: 1 # how many failed jobs should be kept
  suspend: false # Here you can suspend cronjob without deliting it
  concurrencyPolicy: Forbid # Choose Forbid if you don't want concurrent executions of your Job

  # The amount of time that Kubernetes can miss and still start a job.
  # If Kubernetes missed too many job starts (100)
  # then Kubernetes logs an error and doesn’t start any future jobs.
  startingDeadlineSeconds: 300 # if a job hasn't started in this many seconds, skip
  jobTemplate:
    spec:
      parallelism: 1 # How many pods will be instantiated at once.
      completions: 1 # How many containers of the job are instantiated one after the other (sequentially) inside the pod.
      backoffLimit: 3 # Maximum pod restarts in case of failure
      activeDeadlineSeconds: 1800 # Limit the time for which a Job can continue to run
      template:
        spec:
          restartPolicy: Never # If you want to restart - use OnFailure
          terminationGracePeriodSeconds: 30
          containers:
          - name: my-first-conjob
            image: busybox
            command:
              - /bin/sh
            args:
              - -c
              - date; echo sleeping....; sleep 90s; echo exiting...;
            resources:
              requests:
                memory: '128Mi'
              limits:
                memory: '1Gi'
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Excellent! But, as the docs warn, it's a risk limitation, not a risk-free guarantee: "even if you specify .spec.parallelism = 1 and .spec.completions = 1 and .spec.template.spec.restartPolicy = "Never", the same program may sometimes be started twice."
3

Hi it's not clear what you expected - looking into the question but if I understand correctly you mean not running all cronjobs at the same time:
1. First option - it's to change their schedule time,
2. Second option try to use in your spec template other options like - Parallel Jobs - described: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

"For a work queue Job, you must leave .spec.completions unset, and set .spec.parallelism to a non-negative integer"


  jobTemplate:
    spec:
      parallelism: 1
      template:

To recreate this task please provide more details.

In addition for "Jobs History" by default successfulJobsHistoryLimit and failedJobsHistoryLimit are set to 3 and 1 respectively.
Please take at: https://kubernetes.io/docs/tasks/job/ If you are interested you can set-up limit in "spec" section: 


successfulJobsHistoryLimit: 1

failedJobsHistoryLimit: 1

Hope this help.

Improve this answer

3 Comments

Hi Hanx, there should only be running one job for each cron execution, not 3 or more. If you look at that schedule, the job should run only once per day. But Kubernetes runs it multiple times for some reason, as you can see in the log snippet i provided. It runs multiple times with only a one minute interval: it ran 4h40m, 4h41m, 4h42m ago.
Edited the question slightly to make it more clear.
Sorry, my mistake! I got the cron syntax wrong. schedule: "* 2 * * *" means once every minute when the hour is 2. So Kubernetes behaves correctly. I simply should have set it to e.g. "0 2 * * *". Thanks Hanx for giving some more infos about Kubernetes Jobs!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.