Python configuration management in several ways

First, why should use the configuration

If we do not use profiles more complex projects, we may face the following situation:

• You decide to change your host database project because you want to transfer the project from the test environment to the actual production on the environment. If your project in multiple locations to use this host, you have to find them one by one position and then modified into a new host. It took a long time, and after a day, you find that some of the problems the project in a production environment, you need to move back again test environment, you have to modify again, this work is very tedious very elegant.
• You have developed a great open source project, you want to put it into version control systems such as on github, but host your server address, account number, password also upload up, but you do not realize until you have a bad guy got your information, steal information from your server, attack your server so that you had a great loss. Then you want to change about the program, the confidential information such as passwords addresses are deleted, but due to version control, others can still see your previous version of the code. So you have to get rid of your accounts, passwords, etc., it is really a sad experience of open source projects.

However, if you use a configuration management does, it will have the following advantages:

• This improves the reusability of the code, not each time to modify the internal code
• This means do not understand other people inside your code you can use your project, only to change according to your needs
• Conducive to teamwork
• Conducive to data security / secret data management

Two, Python carried out in several ways configuration management

Due to the use of more Python, so Python-based configuration management related explanation, of course, other languages ​​are similar, the main idea is still the same.

Python 2.1 using built-in data structures (such as a dictionary)

Single configuration in a single file 2.1.1

Naturally, we will be able to think of it, for example, the following code:

# main.py

import pymysql

DATABASE_CONFIG  = {
    'host':'localhost',
    'dbname':'db',
    'user':'user',
    'password':'pwd',
    'port':3306
}

def connect_db_do_something1(dbname):
    if dbname != config.DATABASE_CONFIG['dbname']:
        raise ValueError("Couldn't not find DB with given name")
    conn = pymysql.connect(host=config.DATABASE_CONFIG['host'],
                           user=config.DATABASE_CONFIG['user'],
                           password=config.DATABASE_CONFIG['password'],
                           db=config.DATABASE_CONFIG['dbname'])
    '''
    do something 1
    '''

def connect_db_do_something2(dbname):
    if dbname != config.DATABASE_CONFIG['dbname']:
        raise ValueError("Couldn't not find DB with given name")
    conn = pymysql.connect(host=config.DATABASE_CONFIG['host'],
                           user=config.DATABASE_CONFIG['user'],
                           password=config.DATABASE_CONFIG['password'],
                           db=config.DATABASE_CONFIG['dbname'])
    '''
    do something 2
    '''

connect_db_do_something1('db')
connect_db_do_something2('db')

In the above code, we can see that the same database configuration, we have repeatedly used twice, if we need to change the database-related data such as password, we do not need to modify the interior of the two methods, but only modify DATABASE_CONFIGthe dictionary correlation value. And when compared to previously configuration management, reducing the workload too much.

A plurality of individual configuration files 2.1.2

But when you start to get complicated project, your file is more than a simple, and this time if I need main2.pyinside need DATABASE_CONFIGtime is not very convenient, because if the direct import maintime, even though it can use main.DATABASE_CONFIG, but Meanwhile mian.pyin

connect_db_do_something1('db')
connect_db_do_something2('db')

Has also been carried out, this is not what we want to see, so we have a new demand, we can quickly and easily import in different database configuration files in the same project where DATABASE_CONFIG, so we came up with the following method solve this problem:

# config.py
DATABASE_CONFIG = {
    'host': 'localhost',
    'dbname': 'db',
    'user': 'user',
    'password': 'pwd',
    'port': 3306
}

# main1.py
import pymysql
import config

def connect_db_do_something1(dbname):
    if dbname != config.DATABASE_CONFIG['dbname']:
        raise ValueError("Couldn't not find DB with given name")
    conn = pymysql.connect(host=config.DATABASE_CONFIG['host'],
                           user=config.DATABASE_CONFIG['user'],
                           password=config.DATABASE_CONFIG['password'],
                           db=config.DATABASE_CONFIG['dbname'])
    '''
    do something 1
    '''
connect_db_do_something1('db')

# main2.py
import pymysql
import config

def connect_db_do_something2(dbname):
    if dbname != config.DATABASE_CONFIG['dbname']:
        raise ValueError("Couldn't not find DB with given name")
    conn = pymysql.connect(host=config.DATABASE_CONFIG['host'],
                           user=config.DATABASE_CONFIG['user'],
                           password=config.DATABASE_CONFIG['password'],
                           db=config.DATABASE_CONFIG['dbname'])
    '''
    do something 2
    '''
connect_db_do_something2('db')

According to the above code, we can in two different files main1.pyand main2.pythe references are config.pyconfigured in our configuration management look further.

A plurality of configurations in a single file 2.1.3

Our project likely need multiple configuration files, such as test and production environments. Let's talk about starting with a single file, we can use the following solution:

# config.py
class Config:
    APP_NAME = 'myapp'
    SECRET_KEY = 'secret-key-of-myapp'
    ADMIN_NAME = 'administrator'

    AWS_DEFAULT_REGION = 'ap-northeast-2'
    
    STATIC_PREFIX_PATH = 'static'
    ALLOWED_IMAGE_FORMATS = ['jpg', 'jpeg', 'png', 'gif']
    MAX_IMAGE_SIZE = 5242880 # 5MB


class DevelopmentConfig(Config):
    DEBUG = True
    
    AWS_ACCESS_KEY_ID = 'aws-access-key-for-dev'
    AWS_SECERT_ACCESS_KEY = 'aws-secret-access-key-for-dev'
    AWS_S3_BUCKET_NAME = 'aws-s3-bucket-name-for-dev'
    
    DATABASE_URI = 'database-uri-for-dev'


class TestConfig(Config):
    DEBUG = True
    TESTING = True
    
    AWS_ACCESS_KEY_ID = 'aws-access-key-for-test'
    AWS_SECERT_ACCESS_KEY = 'aws-secret-access-key-for-test'
    AWS_S3_BUCKET_NAME = 'aws-s3-bucket-name-for-test'
    
    DATABASE_URI = 'database-uri-for-dev'
  

class ProductionConfig(Config):
    DEBUG = False

    AWS_ACCESS_KEY_ID = 'aws-access-key-for-prod'
    AWS_SECERT_ACCESS_KEY = 'aws-secret-access-key-for-prod'
    AWS_S3_BUCKET_NAME = 'aws-s3-bucket-name-for-prod'

    DATABASE_URI = 'database-uri-for-dev'

# main.py
import sys
import config
import sys
import config

'''
do some important things
'''

if __name__ == '__main__':
    env = sys.argv[1] if len(sys.argv) > 2 else 'dev'
    
    if env == 'dev':
        app.config = config.DevelopmentConfig
    elif env == 'test':
        app.config = config.TestConfig
    elif env == 'prod':
        app.config = config.ProductionConfig
    else:
        raise ValueError('Invalid environment name')

So that we can get different levels of the different configurations from a configuration file.

Multiple configurations in multiple files 2.1.4

And similar to the above, but it replaced with a configuration file to read from different files in different configurations:

# config.py
class Config:
    APP_NAME = 'myapp'
    SECRET_KEY = 'secret-key-of-myapp'
    ADMIN_NAME = 'administrator'

    AWS_DEFAULT_REGION = 'ap-northeast-2'
    
    STATIC_PREFIX_PATH = 'static'
    ALLOWED_IMAGE_FORMATS = ['jpg', 'jpeg', 'png', 'gif']
    MAX_IMAGE_SIZE = 5242880 # 5MB

    
class DevelopmentConfig(Config):
    DEBUG = True
    
    AWS_ACCESS_KEY_ID = 'aws-access-key-for-dev'
    AWS_SECERT_ACCESS_KEY = 'aws-secret-access-key-for-dev'
    AWS_S3_BUCKET_NAME = 'aws-s3-bucket-name-for-dev'
    
    DATABASE_URI = 'database-uri-for-dev'
    
class TestConfig(Config):
    DEBUG = True
    TESTING = True
    
    AWS_ACCESS_KEY_ID = 'aws-access-key-for-test'
    AWS_SECERT_ACCESS_KEY = 'aws-secret-access-key-for-test'
    AWS_S3_BUCKET_NAME = 'aws-s3-bucket-name-for-test'
    
    DATABASE_URI = 'database-uri-for-dev'
  

class ProductionConfig(Config):
    DEBUG = False

    AWS_ACCESS_KEY_ID = 'aws-access-key-for-prod'
    AWS_SECERT_ACCESS_KEY = 'aws-secret-access-key-for-prod'
    AWS_S3_BUCKET_NAME = 'aws-s3-bucket-name-for-prod'

    DATABASE_URI = 'database-uri-for-dev'


class CIConfig:
    SERVICE = 'travis-ci'
    HOOK_URL = 'web-hooking-url-from-ci-service'


# main1.py
import config

dev_config = config.DevelopmentConfig
'''
do something
'''

# main2.py
import config

app.ci = config.CIConfig
'''
do otherthing
'''

Such use of more flexible, reading from different files in different configurations, but we only need to change the configuration for additions and deletions config.pyin progress, configuration management skills, advanced again!

2.2 Use external configuration file

Compared to using the built-in Python data structures, a more common approach is to use external configuration files because these files will be treated as configuration files, and not as config.pythe same have property code. External configuration file format variety, we have different ways of reading the time to use it according to the file format. For example: *.yaml, or *.yml, *.json, *.cfgor *.conf, *.inieven file your custom *.yourname.

2.2.1 yaml

YAML (/ jæməl /, similar to the tail camel camels) is a highly readable, the data used to express serialization format. YAML reference to other languages, including: C, Python, Perl, and get inspiration from XML, e-mail data format (RFC 2822) in. Clark Evans in 2001 published the first language [1], in addition Ingy döt Net co-designer and Oren Ben-Kiki is this language [2]. There have been several current programming or scripting language support (or resolve) this language.
----- Chinese Wikipedia

YAML looks like the following format:

mysql:
    host: localhost
    dbname: db
    user: user
    passwd: pwb
    port: 3306
other:
    host: other_host
    dbname: other_db
    user: other_user
    passwd: other_pwb
    port: 3306

Content which can be read by a code like:

import yaml

with open("config.yml", 'r') as ymlfile:
    cfg = yaml.load(ymlfile)

print(cfg['mysql'])

The following output

{'host': 'localhost',
 'dbname': 'db',
 'user': 'user',
 'password': 'pwd',
 'port': 3306}

If you need to be easily configured from python to write YAML, just use the yaml.dump(dict)can, dictrefer to the dictionary configuration. More detailed contents can be viewed PyYAML Documentation

THIS 2.2.2

INI file is a non-standard format of a fixed profile. It is a simple text composition and simple structure, often used in the Windows operating system, or other operating systems, many programs will be adopted as an INI file with the setup program. Windows operating system and later replace out INI file in the form of registry. INI file naming sources, is from the English "initial (Initial)" of acronyms, is working with its use - initialization procedures accordingly. Sometimes, INI files will take a different extension, such as ".CFG", ". CONF" , or ".TXT" instead.
----- Chinese Wikipedia

It grows like this:

[mysql]
host=localhost
dbname=db
user=user
passwd=pwd
port=3306

[other]
host=other_host
dbname=other_db
user=other_user
passwd=other_pwb
port=3306

It can be read by the following code:

import configparser

config = configparser.ConfigParser()
config.read("config.ini")
host = config['mysql']['host']
print(host)

This output INIprofile mysqlsection in hostvalue

To write INIthe configuration file is also very simple, refer to the following code:

import configparser
config = configparser.ConfigParser()
config.read("config.ini")
config['mysql']['test_str'] = 'a test string'
config.write(open("ini", "w"))

The profiles are now becomes:

[mysql]
host = localhost
dbname = db
user = user
passwd = pwd
port = 3306
test_str = a test string

[other]
host=other_host
dbname=other_db
user=other_user
passwd=other_pwb
port=3306

2.2.3 JSON

JSON is JavaScript Object Notation acronym. It is very broad, and therefore have very good support for many programming languages. It's the format we are all familiar, and looks a lot like Python in the dictionary:

{
    "mysql":{
        "host": "localhost",
        "dbname": "db",
        "user" : "user",
        "password": "pwd",
        "port": 3306
    },
    "other":{
        "host": "other_host",
        "dbname": "other_db",
        "user": "other_user",
        "passwd": "other_pwb",
        "port": 3306
    }

You can refer to the following code reads:

import json

with open('config.json') as json_data_file:
    config = json.load(json_data_file)

host = config['mysql']['host']
print(host)

# output: localhost

To write json configuration is simple, with reference to the following code:

import json

with open('config.json') as json_data_file:
    config = json.load(json_data_file)

config['mysql']['test_str'] = 'a test string'

with open('config.json', 'w') as outfile:
    json.dump(config, outfile)

This will increase the json file to get the configuration:

{
    "mysql":{
        "host": "localhost",
        "dbname": "db",
        "user" : "user",
        "password": "pwd",
        "port": 3306,
        "test_str" : "a test string"
    },
    "other":{
        "host": "other_host",
        "dbname": "other_db",
        "user": "other_user",
        "passwd": "other_pwb",
        "port": 3306
    }
}

Most other file formats so, not go into details. And the outer profile may be a plurality of configuration (mysql, other, etc.)

2.3 Environment Variables

But, back to our opening question to speak, two or more configuration management program (using the built-in Python data structures, using external configuration files) are ignored two questions:

First, how do we deal with security data directly exposed to possible problems it public, if we need to use a version control system such as Github, maybe we can try to config.pyfile into the .gitignoreinside, but if one day we modified the warehouse, forgot to config.pyignore and to push off on GitHub, then we will still be security-sensitive information leaked to the public, due to the presence of version control, even if you delete the record will have to submit this article to deal with them will be very troublesome.

Second, if we want to open a new project in our local, the project also need to refer to the same database configuration files, perhaps we can find the first item in the file folder, a copy config.pyto the new project folder. Ah, it seems feasible, however, if you want to open it more than a dozen projects, hundreds of projects it?

So we can introduce an arrangement under management approach, to solve two problems outlined above are relatively friendly solutions that use environment variables to set various development environments (Win, Mac, Linux) system environment variables differently, you can refer to this article .

Further PyCharm VS Code and has a more convenient arrangement, different items may be assigned different settings.

PyCharm, the menu Run-> Edit configurations, the manual providedEnvironment variables

VS Code, the Settingsearches envin the TerminalChoose your operating system-related Terminal > Integrated > Env: Your OS, point into the settings.jsonadd

Use environment variable configuration values ​​need not be managed as a separate file, so there is less of a security risk, it is very easy to use, you can use any items in your development environment of any code library, but it can be somewhat complex management . Some environment can not use environment variables, such as Apache, Nginx and other Web servers, this time we need to use other ways.

2.4 Use dynamic loading

This method is more advanced than Python using built-in data structures, methods, data structures built in claim profile must be directly on the path to the import. However, dynamic loading, the configuration file is not necessary on the path can be directly imported, even located in other repositories, so, the configuration file and project separated, and other items can also dynamically load the configuration file, for example:

# /opt/settings/config.py
DATABASE_CONFIG = {
    'host': 'localhost',
    'dbname': 'company',
    'user': 'user',
    'password': 'password',
    'port': 3306
}

# main.py
import sys
import pymysql

sys.path.append('/opt/settings')
import config

def connect_db(dbname):
    if dbname != config.DATABASE_CONFIG['dbname']:
        raise ValueError("Couldn't not find DB with given name")
    conn = pymysql.connect(host=config.DATABASE_CONFIG['host'],
                           user=config.DATABASE_CONFIG['user'],
                           password=config.DATABASE_CONFIG['password'],
                           db=config.DATABASE_CONFIG['dbname'])
    return conn

connect_db('company')

Third, the summary

Configuration management summarized above four ways, in general, not good or bad, depends on your individual needs, even the above methods can be mixed use, some software for the project, which itself might provide a relevant variable configuration entry such airbnb of Airflow. Moreover, when the system is very large, it is best to use third-party tools or services that mainly provides configuration management, related services can be found here .