More and more operations and developers are discovering that infrastructure that cannot be automatically managed has become an obstacle to agile business rollout.
For infrastructure operation and maintenance personnel, tasks such as creating and recycling virtual machines are highly repetitive during daily maintenance. When creating business virtual machines, operation and maintenance personnel need to consider resource placement best practices, such as which server has more CPU and memory resources, so that they can be placed first. When provisioning virtual machines for business systems, it is also necessary to ensure that they are created on different cloud platforms as much as possible to avoid business abnormalities caused by cloud platform failures.
For application system developers, DevOps is now being used for agile development and testing of business systems. For testing a version, the CI/CD process can automatically package the application based on the source code and perform version testing. But virtual machine creation cannot be automated—developers who want to use virtual machines need to wait for operations personnel to successfully create virtual machines first, which will block the CI workflow. If you want to realize automated management of the virtual machine life cycle, you need to organize scripts for automatically creating virtual machines based on the private cloud vendor's API. However, if an enterprise has multiple cloud platforms, it needs to refer to the APIs of each vendor to develop infrastructure resource management functions, which will be cumbersome.
So, can the infrastructure also be handed over to code for automated management? This can not only reduce the work pressure of operation and maintenance personnel, but also allow developers to truly provision infrastructure resources on demand.
Readers may have a question here: Can't a commercial cloud management system meet these needs? In fact, when it comes to cloud platform management, almost every user has different needs. For example, a user may hope that the cloud management platform can be combined with its own CMDB for resource monitoring, while another user may hope that the cloud management platform can be combined with its own OA system for resource approval. Some users may also want to automatically deploy business systems on both VMware and SmartX clusters to achieve high availability at the infrastructure layer. But these are actually somewhat difficult to achieve for cloud management platforms. And there is a very critical issue here: the cloud management platform system needs to be charged, which will undoubtedly increase the cost of money.
In order to help more enterprises realize "infrastructure as code" at low cost and high efficiency, this article will introduce a powerful tool - Terraform, and explain how to use Terraform to implement SmartX hyper-converged infrastructure through the practice of financial institutions automated management .
Terraform Workflow and Benefits
Terraform is an infrastructure automation orchestration tool launched by HashiCorp. It can define infrastructure resources through configuration files and create, change and recycle resources based on configuration files, thereby helping users realize "infrastructure as code". Users can use the same workflow to configure and manage all infrastructure, and can also version control resources. Terraform not only supports public cloud vendors such as AWS, GCP, and Alibaba Cloud, but also supports private cloud vendors such as VMware, Lutanix, and SmartX. It can also provision infrastructure such as Kubernetes and Docker, covering public clouds, private clouds, and databases. , security, logging, monitoring and other infrastructure areas.
Terraform is not only powerful and capable of managing a large number of infrastructures, but its use is also very simple, mainly divided into three steps:
- Writing : Write configuration files in a readable language to define the resources required by the infrastructure.
- Plan : Combine the current state of the infrastructure with the predefined configuration files, initialize the configuration, and use the Terraform Plan command to preview the planning of infrastructure resources.
- Application : Apply the configuration file to complete the creation, update and destruction of resources.
Image source: “How does Terraform work?” by HashiCorp
When defining infrastructure, users can provision resources from multiple vendors at the same time, such as defining the elastic computing of AWS and Alibaba Cloud at the same time, so that the infrastructure can provision resources on different cloud vendors at the same time to meet the requirements of business requirements for infrastructure. Redundancy requirements. Furthermore, Terraform's configuration is declarative, so references to other resources and variables do not depend on the order in which they are defined in the configuration file.
For example, if a user wants to provision a set of resources for a business running on Alibaba Cloud, a series of resources such as VPC, security group, virtual machine, image, and object storage need to be created. The order of these resources in the configuration file is not important, Terraform will automatically follow the dependencies of the resources to ensure that the resources can be successfully created.
At this point, we summarize the advantages of Terraform:
- Essentially an infrastructure provisioning tool : Unlike virtual machine configuration tools such as Ansible or Saltstack, Terraform mainly helps users provision infrastructure, such as full lifecycle management of virtual machines. Ansible is better at configuring virtual machines, such as batch upgrading a batch of Linux Open SSH versions. These two types of tools do not conflict and users can also use them together.
- For managing immutable infrastructure : Most "infrastructure as code" tools can create mutable infrastructure to accommodate changes such as some middleware upgrades or disk additions. The disadvantage of variable infrastructure is that configurations can deviate. As multiple changes stack up, the actual provisioned infrastructure can deviate from the original configuration, making it difficult to diagnose, correct configuration errors, and troubleshoot performance issues when problems arise. Terraform provides an immutable infrastructure architecture. With each change in the environment, the current configuration is replaced with a new configuration responsible for the change, and the new infrastructure is re-provisioned. Moreover, the previous configuration can also be retained as a version so that it can be rolled back if needed later.
- Simple and easy to use : Terraform is not only simple to use, but also uses a declarative language, and the syntax is also very readable.
- "Cloud-agnostic" feature : Terraform is decoupled from cloud vendors, allowing the same workflow to be used to manage the infrastructure of different cloud vendors. After building a set of workflows in the early stage, you can still use this set of workflows when you introduce products from other cloud vendors in the later stage. You don’t need to learn the technologies of multiple cloud vendors to quickly maintain the infrastructure products of new vendors.
- Free and open source : Terraform's open source community is very active. Currently, more than 2,200 of the more than 2,400 Providers are co-built by the community. At the same time, contributors in the community have built a large number of plug-ins for this platform, and the activity of the entire community has also enabled the rapid development of the product, which can continuously introduce new functions and improve functions.
Implementing SmartX hyper-convergence "infrastructure as code" based on Terraform
As mentioned earlier, Terraform also supports SmartX hyper-converged automatic management, which needs to be implemented in conjunction with the SmartX native management platform CloudTower.
Taking SmartX hyperconverged infrastructure as an example, users first use code to declare resources in the configuration file. Terraform reads all configuration files with a .tf extension at runtime and concatenates them together. Subsequently, Terraform will call CloudTower's Provider based on the configuration file to create and manage corresponding resources on the SmartX hyper-converged infrastructure. Among them, Provider is a plug-in of Terraform. Different cloud vendors will provide corresponding Providers based on their own cloud platform APIs, and Terraform uses Providers provided by different cloud vendors to create and manage resources on different infrastructures.
This solution has the following advantages:
- Universal configuration : Suitable for multi-cloud scenarios, CloudTower resources can be quickly orchestrated in existing workflows. Other platforms can be managed by simply replacing the Provider.
- Reducing the difficulty of automation : Operation and maintenance personnel do not need to understand the details of the CloudTower API, which reduces the difficulty of deploying automated operation and maintenance. Business personnel only need to care about the configuration of resources, and do not need to care about the steps and processes of creating resources.
- Out-of-the-box : CloudTower provides native Modules and Examples, covering daily usage scenarios such as virtual machine management, virtual disk management, snapshot management, and content library management.
We use the following demo to demonstrate how to use Terraform CLI to manage CloudTower platform virtual machines in batches, and how to use Terraform to quickly provision virtual machines in Jenkins.
Financial institutions explore "infrastructure as code" based on Terraform
In the demo of Jenkins, we can see that users only need to fill in the resource parameters in Jenkins, and Jenkins can write these parameters into the configuration file through the pre-organized script in the background. Terraform will automatically generate a virtual machine based on the configuration file, and can notify the enterprise WeChat.
Following this idea, users can create an enterprise-level self-service management platform by adding some functions on this basis . For example, users can join the CMDB: before generating the configuration file, first query in the CMDB which servers have sufficient resources, so as to give priority to utilization, and place different virtual machines with the same function on different clusters. Users can also use Ansible to organize virtual machine application configuration playbooks (Playbooks) in advance. After the virtual machine is successfully created, users can also install applications such as MySQL, Redis or Kafka on the virtual machine.
In addition to this, users can also join the application, approval and delivery process. Application personnel only need to log in to the front-end page to apply for resources (pre-apply for the number of virtual machines, CPU, memory and storage requirements, etc.), choose whether to deploy applications, and submit resource applications. Then there is a series of automatic processes: the application will enter the company's OA process and start approval; after the approval is passed, the virtual machine will be automatically provisioned and the application will be deployed on demand; after the resource is successfully created, the key information of the virtual machine and the application will be automatically fed back, which is convenient The application staff can use it directly. This realizes an enterprise-level self-service management platform.
Financial industry user practice: self-service based on Terraform + CloudTower
Based on the above architecture, a user in the financial industry used Terraform + CloudTower to independently develop an infrastructure operation and maintenance platform that meets the company's needs, and combined with CMDB to realize resource usage, statistics, and planning.
The back end of the entire operation and maintenance platform uses Terraform to manage the life cycle of virtual machines, uses Ansible to manage the configuration of virtual machines, uses Python to write deployment scripts for enterprise applications, uses CMDB to store configuration records of resource life cycles, and configures virtual machines. Clusters, storage, and servers provide optimal placement recommendations. At the same time, users have added the functions they need according to the characteristics of the company's business model, including management of IP addresses, networks, users and permissions.
Through this operation and maintenance platform, users realize the automated delivery of virtual machines and enterprise-level applications on the infrastructure cloud platform. The entire delivery process has been standardized, automated, streamlined, and data-based, which greatly shortens the delivery cycle, reduces the workload of operation and maintenance personnel, and facilitates the audit and traceability of subsequent resources .
Want to try building your own enterprise self-service management platform? You can get the CloudTower Provider plug-in on the Terraform official website through the following link to easily automate infrastructure: https://registry.terraform.io/Providers/smartxworks/cloudtower
References:
1. How does Terraform work?
https://developer.hashicorp.com/terraform/intro#how-does-terraform-work
Click to download the community version of the hyper-converged suite , experience the CloudTower management platform for free, and start your infrastructure automation journey immediately.