Getting Started Tutorial: Create K8S Job in 5 Steps, Get Batch Processing

Kubernetes jobs are mainly for short-term and batch workloads. It runs for the end, not for continuous operation like deployment, replicasets, replication controllers, and DaemonSets.

This article will introduce how to create Kubernetes jobs and cronjobs, as well as some tips.

Kubernetes Jobs will run until the tasks specified in the Job are completed. In other words, if the pods give an exit code of 0, then the job will exit. In normal Kubernetes, no matter what the exit code is, the deployment object will create a new pod when it terminates or an error occurs to maintain the ideal state of the deployment.

During the job running process, if the node hosting the pod fails, the job pod will be automatically rescheduled to another node.

Kubernetes Jobs example

The best use case practice for Kubernetes Jobs is:

1. Batch processing task : For example, you want to run a batch processing task once a day or in a specified schedule. It may be like reading files from a repository or database and assigning them to a service to process files.

2. Operation and maintenance/ad-hoc tasks : For example, if you want to run a script/code, the script/code will run a database cleanup activity and even back up a Kubernetes cluster.

   How to create a Kubernetes job

   In this example, we will use the Ubuntu container to run a shell script with a for loop, and respond to messages based on the parameters you pass to the container. This parameter is a number that determines how many times the shell script loop should run.

For example, if you pass the parameter 100, the shell script will echo the message 100 times and the container will exit.

You can visit the following link to view the Dockerfile and shell script:
https://github.com/devopscube/Kubernetes-jobs-example/tree/master/Docker

Let's start with a simple job setup.

Step1 : Use a custom Docker image to create a job.yaml file, the command parameter is 100. 100 will be passed as a parameter to the docker ENTRYPOINT script.

apiVersion: batch/v1  kind: Job  metadata:      name: kubernetes-job-example      labels:     
    jobgroup: jobexample  spec:      template:     
    metadata:       
      name: kubejob       
      labels:         
        jobgroup: jobexample     
    spec:       
      containers:       
      - name: c         
        image: devopscube/kubernetes-job-demo:latest         
        args: ["100"]       
      restartPolicy: OnFailure   

Step2 : Use kubectl to create a job with job.yaml file

kubectl apply -f job.yam

Step3 : Use kubectl to check the status of the job

kubectl get jobs

Step4 : Use kubectl to get the pod list

kubectl get po

Step5 : Use kubectl to get job pod logs. Replace the original Pod name with the Pod name you see in the output.

kubectl logs kubernetes-job-example-bc7s9 -f

Run multiple Job pods in parallel

After a job is deployed, you can make it run in parallel on multiple Pods. For example, if you want to run 6 pods in a job and run 2 pods in parallel, you need to add the following 2 parameters to your job manifets:

completions: 6
parallelism: 2

The following is a manifest with those parameters:

apiVersion: batch/v1
kind: Job
metadata:
  name: kubernetes-parallel-job
  labels:
    jobgroup: jobexample
spec:
  completions: 5
  parallelism: 2
  template:
    metadata:
      name: kubernetes-parallel-job
      labels:
        jobgroup: jobexample
    spec:
      containers:
      - name: c
        image: devopscube/kubernetes-job-demo:latest
        args: ["100"]
      restartPolicy: OnFailure

Generate random name for Kubernetes Job

You cannot create multiple jobs from one job manifest file, because Kubernetes will report an error saying that there is a job with the same name. To circumvent this problem, you can add the generateName name parameter in the metadata .

E.g:

apiVersion: batch/v1
kind: Job
metadata:
  generateName: kube-job-
  labels:
    jobgroup: jobexample

In the above example, every time you run the manifest, the job will be created with kube-job- as the prefix, followed by a random string.

How to create a Kubernetes CronJob

If you want to run a batch job according to a specific schedule, for example, run once every 2 hours. You can create a Kubernetes cronjob with cron expressions . The job will start automatically according to the schedule you mentioned in the job.

Below we will introduce how to specify a cron plan, you can use the crontab generator (https://crontab-generator.org/ ) to generate your own time plan.

schedule: "0,15,30,45 * * * *"

The following figure shows the Kubernetes cronjob schedule syntax.
If we run our previous job every 15 minutes in the form of a cronjob, the manifest should look like the following. Create a file named cron-job.yaml and copy the following manifest:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
    name: kubernetes-cron-job
spec:
  schedule: "0,15,30,45 * * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: cron-batch-job
        spec:
          restartPolicy: OnFailure
          containers:
          - name: kube-cron-job
            image: devopscube/kubernetes-job-demo:latest
            args: ["100"]

Let's deploy cronjob using kubectl.
kubectl create -f cron-job.yaml
List cronjobs:

kubectl get cronjobs

You can check the cronjob log by listing cronjob pods and getting logs from pods that are running or completed.

Run Kubernetes CronJob manually

In some cases, you may want to execute cronjob in a temporary manner. You can do this by creating a job from an existing cronjob.

For example, if you want to manually trigger a cronjob, we should do this:

kubectl create job --from=cronjob/kubernetes-cron-job manual-cron-job

--from=cronjob/kubernetes-cron-job will copy the cronjob template and create a job named manual-cron- job.

Key parameters of Kubernetes Job

According to your needs, you can also use several key parameters of kubernetes jobs/cronjobs:

1. failedJobHistoryLimit &
   successfulJobsHistoryLimit: Delete failed and successful job history records based on the number of reservations you provide. This is very useful for reducing all failed entries when you try to list jobs. E.g:

2. backoffLimit: If your Pod fails, the total number of retries.

3. activeDeadlineSeconds: If you want to impose a hard limit on the running time of cronjob, you can use this parameter. For example, if you want to run a cronjob for only 1 minute, you can set it to 60.

Through this article, we have learned about the steps of creating Job and Cron Job and some detailed configuration procedures and key parameters. Hope this article can help you get started to understand K8S Job and Cron Job, and easily get batch processing tasks!

Original link:
https://devopscube.com/create-kubernetes-jobs-cron-jobs/