Take you to master the use of Terraform different data sources to expand application scenarios

This article is shared from Huawei Cloud Community " Using Terraform Different Data Sources to Extend Application Scenarios ", author: kaliarch.

a background

In the production environment, Terraform is used for infrastructure orchestration. Usually, some information is imported through other external systems. The data source of this field is an interface. Terraform needs to have the ability to call remote interfaces to obtain data for target resource orchestration and process various clouds. In addition to the data data type of the provider interface provided by the manufacturer, the terraform community also has an http provider, which can be used to call third-party external system interfaces to achieve self-service external interface acquisition and arrangement.

In addition to Terraform directly calling the http provider interface to obtain data, you can also execute local Shell/Python scripts. The script internally implements calling external interfaces to obtain data, and then transfers the data to Terraform for use.

Two principles

2.1 Data source concept

Data sources allow data to be fetched and computed for use in code. Sources can be located within another Terraform configuration or externally. In contrast to resources, datasources are not managed by Terraform.

2.2 Advantages of Data Sources

Reduce coupling between modules and use your infrastructure as the source of truth.
Hide complexity from Terraform end users by reducing the number of variables

2.3 Extended data source method

2.3.1 http

Terraform downloads the http provider, which is an http client written in go, which can call the external interface.

https://registry.terraform.io/providers/hashicorp/http/latest/docs

2.3.2 Script mode

terraform acts as an executor, using external provider to execute each scripting language, so as to achieve the predetermined goal of executing the script content.

Three http data sources

3.1 Test scenario

Use Terraform to write orchestration files and get github profile information.

3.2 code

Directory Structure

├── main.tf // main file

├── outputs.tf // output file

└── variables.tf // variable file

code content

# main.tf

data "http" "get_method" {

url = "https://api.github.com/users/${var.gitName}"

request_headers = {

Accept = "application/json"

}

}

data "http" "post_method" {

url = "https://checkpoint-api.hashicorp.com/v1/check/terraform"

method = "POST"

# Optional request body

request_body = "request body"

}

# variables.tf

variable "gitName" {

type = string

default = "redhatxl"

}

# outputs.tf

output "resp" {

value = {

get = data.http.get_method.body

post = data.http.post_method.body

}

}

3.3 Testing

# init

$ terraform init

# plan

$ terraform plan

# Put the output file everywhere into the json file

$ terraform show --json github.out | > redhatxl.json

# apply application

$ terraform apply

3.4 Others

3.4.1 POST request

data "http" "example_post" {

url = "https://checkpoint-api.hashicorp.com/v1/check/terraform"

method = "POST"

# Optional request body

request_body = "request body"

}

3.4.2 Postconditions

data "http" "example" {

url = "https://checkpoint-api.hashicorp.com/v1/check/terraform"

# Optional request headers

request_headers = {

Accept = "application/json"

}

lifecycle {

postcondition {

condition = contains([201, 204], self.status_code)

error_message = "Status code invalid"

}

}

}

3.4.3 Preconditions

data "http" "example" {

url = "https://checkpoint-api.hashicorp.com/v1/check/terraform"

# Optional request headers

request_headers = {

Accept = "application/json"

}

}

resource "random_uuid" "example" {

lifecycle {

precondition {

condition = contains([201, 204], data.http.example.status_code)

error_message = "Status code invalid"

}

}

}

3.4.4 Using Provisioners

data "http" "example" {

url = "https://checkpoint-api.hashicorp.com/v1/check/terraform"

# Optional request headers

request_headers = {

Accept = "application/json"

}

}

resource "null_resource" "example" {

# On success, this will attempt to execute the true command in the

# shell environment running terraform.

# On failure, this will attempt to execute the false command in the

# shell environment running terraform.

provisioner "local-exec" {

command = contains([201, 204], data.http.example.status_code)

}

}

Four script execution

"External data sources allow external programs implementing a specific protocol (defined below) to act as data sources, exposing arbitrary data for consumption elsewhere in the Terraform configuration."

Sometimes my terraform modules depend on data not managed by a terraform provider, but by a build step or script in my repository. An external data source is an interface for running commands locally on the machine running Terraform and providing the program's console output as a data source.

This is a mechanism that allows local scripts to act as data sources. To be a valid data source, a local script simply prints JSON to standard output, like so:

4.1 Test scenario

Use Terraform to write orchestration files and get github profile information.

4.2 Code

├── main.tf

├── outputs.tf

├── scripts

│ └── py

│ └── fetch_githubinfo.py

└── variables.tf

main.tf

data "external" "githubinfo" {

program = ["python", "${path.module}/scripts/py/fetch_githubinfo.py"]

query = {

gitName = var.gitName

}

}

locals {

resp = data.external.githubinfo.result

}

variables.tf

variable "gitName" {

type = string

}

outputs.tf

output "resp" {

value = {

get = local.resp

}

}

fetch_githubinfo.py

#!/usr/bin/env python3

# coding: utf-8

import json

from terraform_external_data import terraform_external_data

import requests

import json

@terraform_external_data

def fetch(query):

# Terraform requires the values you return be strings,

# so terraform_external_data will error if they aren't.

gitName = query['gitName']

response = requests.get(f'https://api.github.com/users/{gitName}')

output_json = response.json()

return {str(key): str(value) for key, value in output_json.items()}

if __name__ == "__main__":

fetch()

4.3 Testing

Execute terraform init/terraform apply

4.4 Others

Terraform acts as an executor and can execute shell/js/golang/python and other language codes, but it needs an interpreter of the corresponding language in the execution environment.

Five Summary

It is very convenient to use the http/external of data to call the external interface to obtain data. However, the official positioning of External Data Source is the "escape window", so when considering using this solution, it is the last resort.

reference link

https://registry.terraform.io/providers/hashicorp/http/latest/docs

Click to follow and learn about Huawei Cloud's fresh technologies for the first time~