What is the job
For ReplicaSet, ReplicationController and other types of controllers, the number is expected to keep its hopes pod, lasting run down, unless the user explicitly deleted, these objects have been otherwise exist, they are durable for tasks such as web services. For non-durable tasks, such as file compression, the task is completed, pod need to end the run, without pod remain in the system, this time is necessary to use Job. So, to say Job complements the ReplicaSet, ReplicationController and other persistent controller.
Job nuances with other controllers
Job controller definition method and the like ReplicaSet similar, with only slight differences, as follows:
Job necessary to restart policy is "Never" or "OnFailure", this is well understood, because the pod to run to the end, rather than repeatedly restart.
Job does not need to choose, in which the pod does not need to tag, the system will automatically add relevant content when creating a Job. Of course, users can also organize resources for the purpose of labeling, but this has nothing to do with the realization of Job itself.
Job add two new fields: .spec.completions, .spec.parallelism. In an example for detailed instructions
backoffLimit field: the example described
Examples
Non-concurrent Job
Uncomplicated Job implication is that after Job start, run only one pod, the whole Job will end immediately after the end of the pod running.
The following is a simple Job profile contains only a pod, 2000 digits of pi decimal point output, running time approximately 10s:
apiVersion: batch/v1
kind: Job
metadata:
name: pi
spec:
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
backoffLimit: 4Selector without setting the above example, pod label. No need to set .spec.completions, .spec.parallelism, the default value of these two fields is 1. backoffLimit = 4, represents the number of allowed failures pod. Save the file and create more content to Job:
$ kubectl create -f https://k8s.io/examples/controllers/job.yaml
job "pi" createdConfirm Job status:
$ kubectl describe jobs/pi
Name: pi
Namespace: default
Selector: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495
Labels: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495
job-name=pi
Annotations: <none>
Parallelism: 1
Completions: 1
Start Time: Tue, 07 Jun 2016 10:56:16 +0200
Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
Pod Template:
Labels: controller-uid=b1db589a-2c8d-11e6-b324-0209dc45a495
job-name=pi
Containers:
pi:
Image: perl
Port:
Command:
perl
-Mbignum=bpi
-wle
print bpi(2000)
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 {job-controller } Normal SuccessfulCreate Created pod: pi-dtn4qCan be seen from the above output Selector, Pod labels automatically added. Note that the output of Events, the entire process only creates a pod.
List all of the pod Job:
$ pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name})
$ echo $pods
pi-aiw0aView Pod's output
$ kubectl logs $pods
3.14159265358979323846264......Job above is the simplest application example, 2000 output after the decimal point pi. However, consider another case, if we need to calculate pi after the decimal point 3000,4000,5000 bit how to do? One solution is to change the 3000 cases in 2000 on the first, create and run. And then changed to 4000, and then create and run until 5000. Clearly this approach is not wise: trouble, resource utilization is low. Another method is to create four Job, were calculated 2000,3000,4000,5000. Note 4 Job's name field can not conflict are pi-2000, pi-3000, pi-4000, pi-5000,
the file names are pi-2000.yaml, pi-3000.yaml, pi-4000.yaml, pi -5000.yaml, and stored in the directory / tmp / pi. Kubectl support for the use of one-time directory created four Job:
$ kubectl create -f /tmp/pi
job "pi-2000" created
job "pi-3000" created
job "pi-4000" created
job "pi-5000" createdThe above method is a pseudo-concurrent, only applies to less job situation. If the task we need to address from a pi-1 to pi-10000, then the above method does not apply:
We need to create a lot of Job and pod, bad management.
Resource quotas.
Crude concurrent Job
This example creates a Job, but Job To create multiple pod. For an example of finish after understand why called "rough concurrent."
This example requires a complex message queue and services, not described in detail how to deploy, padded message queue service. Suppose we have a RabbitMQ service, access the cluster address: amqp: // guest: guest @ rabbitmq-service: 5672. It has a queue named job1, there is apple banana cherry date fig grape lemon melon total of eight members of the queue.
Also assume that we have named gcr.io/
apiVersion: batch/v1
kind: Job
metadata:
name: job-wq-1
spec:
completions: 8
parallelism: 2
template:
metadata:
name: job-wq-1
spec:
containers:
- name: c
image: gcr.io/<project>/job-wq-1
env:
- name: BROKER_URL
value: amqp://guest:guest@rabbitmq-service:5672
- name: QUEUE
value: job1
restartPolicy: OnFailureIn the embodiment, a value of 8 Completions, job1 equal to the number of elements in the queue. Because each element of a successful pod processing, it is necessary to the success of eight, all members of job1 will be processed. In the concurrent coarse mode, the specified value of completions necessary, otherwise the default value is 1, the whole process only one member of the Job is completed.
In the above example, the value 2 is parallelism. Although success requires pod eight times, but at the same time, allowing only two concurrent pod. After a successful conclusion, and then start another. The main purpose of this control parameter is the number of concurrent pod can be adjusted according to actual situation. Of course, you can not specify, the default is the number of concurrent 1.
Env content in the image tells how to access the queue.
The above content stored in job.yaml file, run the Job:
kubectl create -f ./job.yamlWait a moment Job run is complete, view the results:
$ kubectl describe jobs/job-wq-1
Name: job-wq-1
Namespace: default
Selector: controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
Labels: controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
job-name=job-wq-1
Annotations: <none>
Parallelism: 2
Completions: 8
Start Time: Wed, 06 Sep 2017 16:42:02 +0800
Pods Statuses: 0 Running / 8 Succeeded / 0 Failed
Pod Template:
Labels: controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
job-name=job-wq-1
Containers:
c:
Image: gcr.io/causal-jigsaw-637/job-wq-1
Port:
Environment:
BROKER_URL: amqp://guest:guest@rabbitmq-service:5672
QUEUE: job1
Mounts: <none>
Volumes: <none>
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
───────── ──────── ───── ──── ───────────── ────── ────── ───────
27s 27s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-hcobb
27s 27s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-weytj
27s 27s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-qaam5
27s 27s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-b67sr
26s 26s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-xe5hj
15s 15s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-w2zqe
14s 14s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-d6ppa
14s 14s 1 {job } Normal SuccessfulCreate Created pod: job-wq-1-p17e0View Events, you can see created a total of 8 pod. In this case, a member of the queue for each treatment will need to create a pod, big overhead. If the number of members of the queue is very large, so this approach does not apply. We want to create a small pod, each pod can handle multiple records, consider the following example
Fine concurrent Job
redis:6379> lrange job2 0 -1
1) "apple"
2) "banana"
3) "cherry"
4) "date"
5) "fig"
6) "grape"
7) "lemon"
8) "melon"
9) "orange"Next, create image, detailed process is not described. Just make sure to run this image as a python program called worker.py
#!/usr/bin/env python
import time
import rediswq
host="redis"
# Uncomment next two lines if you do not have Kube-DNS working.
# import os
# host = os.getenv("REDIS_SERVICE_HOST")
q = rediswq.RedisWQ(name="job2", host="redis")
print("Worker with sessionID: " + q.sessionID())
print("Initial queue state: empty=" + str(q.empty()))
while not q.empty():
item = q.lease(lease_secs=10, block=True, timeout=2)
if item is not None:
itemstr = item.decode("utf=8")
print("Working on " + itemstr)
time.sleep(10) # Put your actual work here instead of sleep.
q.complete(item)
else:
print("Waiting for work")
print("Queue empty, exiting")First connect to the redis job2 queue. Then a while loop, each time a record is read and outputting job2, then sleep 10s. Loop exit condition is job2 queue is empty. This image different from the example 2, example 2 process one record this end, and this can handle multiple until the queue is empty.
Next, define the Job:
apiVersion: batch/v1
kind: Job
metadata:
name: job-wq-2
spec:
parallelism: 2
template:
metadata:
name: job-wq-2
spec:
containers:
- name: c
image: gcr.io/myproject/job-wq-2
restartPolicy: OnFailureIn the embodiment, as in the example 2 without completions as the specified value, since the end of that blank job2 been embedded in the logic of the image. parallelism = 2 represents two concurrent POD, the default is not set to 1, the self-adjusting according to the actual situation in the practical application.
Run Job:
kubectl create -f ./job.yamlView Job health after a while:
$ kubectl describe jobs/job-wq-2
Name: job-wq-2
Namespace: default
Selector: controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
Labels: controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
job-name=job-wq-2
Annotations: <none>
Parallelism: 2
Completions: <unset>
Start Time: Mon, 11 Jan 2016 17:07:59 -0800
Pods Statuses: 1 Running / 0 Succeeded / 0 Failed
Pod Template:
Labels: controller-uid=b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
job-name=job-wq-2
Containers:
c:
Image: gcr.io/exampleproject/job-wq-2
Port:
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
33s 33s 1 {job-controller } Normal SuccessfulCreate Created pod: job-wq-2-lglf8Although the maximum number of concurrent 2 is allowed, but only to create a successful Events show a pod, this is a normal situation, the maximum is not necessary, it may be because the system resources reach maximum.
View pod output:
$ kubectl logs pods/job-wq-2-7r7b2
Worker with sessionID: bbd72d0a-9e5c-4dd6-abf6-416cc267991f
Initial queue state: empty=False
Working on banana
Working on date
Working on lemonFine and coarse compared to concurrent concurrently, reducing the overhead of creating pod, each pod can handle many records, but to decide for themselves pod exit condition, if not quit, then Job never end.
About Recycling
pod Job created after the end of the run, regardless of success or failure, will not default deleted, still remain in the system, so users can view logs, status information, eliminate errors. Users need to manually run kubectl delete to delete all of the pod end of the run, in order to facilitate organizational resources, the Ministry will be deleted at a time pod, pod can be custom labels. Job After the run is complete and still remains in the system, deleted by the user. So the use of Job, the user should pay attention to recycling to avoid resources being depleted.