content

Manipulation of YAML files in Python

Manipulation of YAML files in Python

1. Summary

When developing the container publishing system, since it needs to call the API of k8s, it is necessary to deal with the YAML content format such as deployment and service of k8s.

Let's first introduce YAML

YAML full name YAML Ain't Markup Language, the main design goal is high human readability.

YAML 1.2 is a superset of JSON, which means that legal JSON thrown to the YAML 1.2 parser can be perfectly parsed

1. yaml file rules

case sensitive;
Use indentation to indicate hierarchical relationships;
Indent with the space bar instead of the tab key
The number of indented spaces is not fixed, only the elements of the same level need to be aligned to the left;
The strings in the file do not need to be marked with quotation marks, but if the strings contain special characters, they need to be marked with quotation marks;
Comments are marked with #

2. yaml file data structure

Object: A collection of key-value pairs ("map or dictionary" for short)
- The key-value pair is represented by a colon ":" structure, and the colon and the value need to be separated by a space
Array: An ordered set of values ("sequence or list" for short)
- The array is preceded by a "-" symbol, and the symbol and the value must be separated by a space
scalars: single, indivisible values (such as strings, bools, integers, floats, times, dates, null, etc.)
- The value of None can be represented by null and ~

3. Usage scenarios

YAML Generally speaking, there are many scenarios for serialization and configuration files.

In addition, YAML can be converted to Json format

Second, the use of Python

There are two ways to operate yaml in Python:

pyyamlpip3 install pyyaml
ruamel (recommended):pip3 install ruamel.yaml

1. pyyaml

pyyaml is the official package that comes with python. The official document is: https://pyyaml.org/wiki/PyYAML

Install:

pip3 install pyyaml

Read:yaml.load

import yaml

# 读取yaml内容
yaml_content = yaml.load("""
name: Vorlin Laruknuzum
sex: Male
class: Priest
title: Acolyte
hp: [32, 71]
sp: [1, 13]
gold: 423
inventory:
- a Holy Book of Prayers (Words of Wisdom)
- an Azure Potion of Cure Light Wounds
- a Silver Wand of Wonder
""", Loader=yaml.FullLoader)
print(yaml_content['name'])

# 读取yaml文件
with open(r'test.yaml', 'r') as f:
    yaml.load(f, Loader=yaml.FullLoader)

write:yaml.dump

import yaml

# 读取yaml内容
yaml_content = yaml.load("""
name: Vorlin Laruknuzum
sex: Male
class: Priest
title: Acolyte
hp: [32, 71]
sp: [1, 13]
gold: 423
inventory:
- a Holy Book of Prayers (Words of Wisdom)
- an Azure Potion of Cure Light Wounds
- a Silver Wand of Wonder
""", Loader=yaml.FullLoader)

yaml_content['name'] = 'New Name'
new_yaml = yaml.dump(yaml_content)
print(new_yaml)

However, there yaml.dumpis a problem:

Cannot handle data of dictionary nested dictionary
After saving, it is not the standard form of yaml
double quotes are lost

For example here is an example:

The yaml content is:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  annotations:
    creator: admin
    description: mbs-statement-service
    owner_name: mbs-statement-service
    sidecar.istio.io/inject: "false"
  creationTimestamp: null
  labels:
    app: mbs-statement-service
    version: "07071418"
  name: mbs-statement-service-07071418
  namespace: bankapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mbs-statement-service
      version: "07071418"
  strategy: {}
  template:
    metadata:
      annotations:
        creator: admin
        description: mbs-statement-service
        owner_name: mbs-statement-service
        sidecar.istio.io/inject: "false"
      creationTimestamp: null
      labels:
        app: mbs-statement-service
        version: "07071418"
      name: mbs-statement-service-07071418
      namespace: bankapp
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - /usr/local/jdk1.8.0_172/bin/java -jar  $(JAVA_OPTS) $(GCLOG_OPTS)  $(SHOOTING_OPTS)
          $(APOLLO_OPTS) 
          mbs-statement-service.jar
        image: vb-harbor.in.xx/xxx/mbs-statement-service:uat_20200528_v2
        imagePullPolicy: IfNotPresent

Using PyYaml to process:

import yaml

deployment_obj = yaml.load(yaml_content, Loader=yaml.FullLoader)
for key, containers in enumerate(deployment_obj['spec']['template']['spec']['containers']):
    deployment_obj['spec']['template']['spec']['containers'][key]['image'] = "$IMAGE_URL"


k8s_deployment = yaml.dump(deployment_obj)
print(k8s_deployment)

After printing it out, there is a problem with the format, the double quotes for version are missing:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  annotations:
    creator: admin
    description: mbs-statement-service
    owner_name: mbs-statement-service
    sidecar.istio.io/inject: 'false'
  creationTimestamp: null
  labels:
    app: mbs-statement-service
    version: 07071418
  name: mbs-statement-service-07071418
  namespace: bankapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mbs-statement-service
      version: 07071418
  strategy: {}
  template:
    metadata:
      annotations:
        creator: admin
        description: mbs-statement-service
        owner_name: mbs-statement-service
        sidecar.istio.io/inject: 'false'
      creationTimestamp: null
      labels:
        app: mbs-statement-service
        version: 07071418
      name: mbs-statement-service-07071418
      namespace: bankapp
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - /usr/local/jdk1.8.0_172/bin/java -jar  $(JAVA_OPTS) $(GCLOG_OPTS)  $(SHOOTING_OPTS)
          $(APOLLO_OPTS) mbs-statement-service.jar
        image: $IMAGE_URL
        imagePullPolicy: IfNotPresent

use default_styleto process

import yaml

deployment_obj = yaml.load(yaml_content, Loader=yaml.FullLoader)
for key, containers in enumerate(deployment_obj['spec']['template']['spec']['containers']):
    deployment_obj['spec']['template']['spec']['containers'][key]['image'] = "$IMAGE_URL"


k8s_deployment = yaml.dump(deployment_obj, default_style='"')
print(k8s_deployment)

It prints out that the problem is bigger, it will appear:"replicas": !!int "1"

"apiVersion": "apps/v1beta1"
"kind": "Deployment"
"metadata":
  "annotations":
    "creator": "admin"
    "description": "mbs-statement-service"
    "owner_name": "mbs-statement-service"
    "sidecar.istio.io/inject": "false"
  "creationTimestamp": !!null "null"
  "labels":
    "app": "mbs-statement-service"
    "version": "07071418"
  "name": "mbs-statement-service-07071418"
  "namespace": "bankapp"
"spec":
  "replicas": !!int "1"
  "selector":
    "matchLabels":
      "app": "mbs-statement-service"
      "version": "07071418"
  "strategy": {}
  "template":
    "metadata":
      "annotations":
        "creator": "admin"
        "description": "mbs-statement-service"
        "owner_name": "mbs-statement-service"
        "sidecar.istio.io/inject": "false"
      "creationTimestamp": !!null "null"
      "labels":
        "app": "mbs-statement-service"
        "version": "07071418"
      "name": "mbs-statement-service-07071418"
      "namespace": "bankapp"
    "spec":
      "containers":
      - "command":
        - "/bin/sh"
        - "-c"
        - "/usr/local/jdk1.8.0_172/bin/java -jar  $(JAVA_OPTS) $(GCLOG_OPTS)  $(SHOOTING_OPTS)\
          \ $(APOLLO_OPTS) mbs-statement-service.jar"
        "image": "$IMAGE_URL"
        "imagePullPolicy": "IfNotPresent"

So you have to use the following ruamel. Using the following ruamel will not remove the quotation marks after the version

2. ruamel

PyYaml is the most widely used, but the encapsulated API is not simple enough, and does not support the latest version of YAML 1.2, so you can use ruamel instead, ruamel.yaml is a derivative of PyYaml, the packaged API is simple and supports the latest version of YAML 1.2

The usage is basically the same as PyYAML

Install

pip3 install ruamel.yaml