In the previous article " Installing Standalone Elastic Agents and Collecting Data - Elastic Stack 8.0 ", we detailed how to use No Fleet Server to write data into Elasticsearch. In today's article, we will detail how to use Elastic Agents to collect data in standalone mode and finally write the data to Elasticsearch through Logstash.

In today's exercise, I will use the following architecture to set up my test environment:

I installed Elasticsearch and Kibana under macOS, Apache, Elastic Agent and Logstash under Ubuntu OS. For this demo, I'll be using the latest Elastic Stack 8.8.1 for the demo.

Install

Elasticsearch 及 Kibana

If you have not installed your own Elasticsearch and Kibana, please refer to the following articles to install:

When installing, please pay special attention to: Please choose the installation guide of Elastic Stack 8.x to install. By default, Elasticsearch access is with HTTPS access.

Elastic Agent

We can go to Elastic's official website Download Elastic Products | Elastic to download the installation package that is consistent with the version of Elasticsearch and matches the operating system of your own computer.

wget https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.8.1-linux-arm64.tar.gz

tar xzf elastic-agent-8.8.1-linux-arm64.tar.gz

cd elastic-agent-8.8.1-linux-arm64

We can see the following files in the current directory:

 pwd
/home/parallels/fleet/elastic-agent-8.8.1-linux-arm64
parallels@ubuntu2004:~/fleet/elastic-agent-8.8.1-linux-arm64$ ls 
data  elastic-agent  elastic-agent.reference.yml  elastic-agent.yml  fleet.enc.lock  LICENSE.txt  NOTICE.txt  README.m

The above elastic-agent.yml file is the configuration file of Elastic Agent. For the configuration of Elastic Agent in standalone mode, please refer to Elastic official link Install standalone Elastic Agents (advanced users) | Fleet and Elastic Agent Guide [8.8] | Elastic for details .

Let's first try to send data from Elastic Agent directly to Elasticsearch. If we open the default elastic-agent.yml file, we will find that, by default, it will collect system metrics. Here, we do not introduce any integration. We open the elastic-agent.yml file:

As shown above, for the convenience of demonstration, I use the super user elastic to configure. This is not recommended in actual use. You need to create an account with appropriate permissions according to your needs for configuration. In addition, the above fingerprint can also be found in the terminal output when Elasticsearch is started for the first time.

You can also find it at the end of the config/kibana.yml file. You can also learn how to get this fingerprint by reading the article " Elasticsearch: Everything you need to know about using Elasticsearch in Python - 8.x ".

Note : Above, we use the username and password to configure the elastic-agent.yml file. You can also use API Key to configure. See the article " Elasticsearch: Everything you need to know about using Elasticsearch in Python - 8.x " for how to get the API key.

After configuring the elastic-agent.yml file above, we run the following command:

sudo ./elastic-agent install

parallels@ubuntu2004:~/fleet/elastic-agent-8.8.1-linux-arm64$ sudo ./elastic-agent install
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
Do you want to enroll this Agent into Fleet? [Y/n]:n
Elastic Agent has been successfully installed.

We can check it in Kibana:

We can also go to Discover to check:

In Discover, we can see the metrics data that has just been ingested. Please note that so far, we have only ingested the default system metrics data.

Write elastic agent data to Logstash

Install Logstash and configure

We next write the output data from the elastic agent to Logstash. We can refer to Elastic's official documentation Logstash output | Fleet and Elastic Agent Guide [8.8] | Elastic . We first need to install Logstash. Please refer to the previous article " How to install Logstash in the Elastic stack ". We can also go directly to Elastic's official website Download Logstash Free | Get Started Now | Elastic to download the Logstash version that matches Elasticsearch.

wget https://artifacts.elastic.co/downloads/logstash/logstash-8.8.1-linux-aarch64.tar.gz
tar xzf logstash-8.8.1-linux-aarch64.tar.gz
cd logstash-8.8.1

For testing, we need to create a logstash.conf configuration file:

logstash.conf

input {
  elastic_agent {
    port => 5044
  }
}

output {
   stdout {}
}

Above, our Logstash pipeline is very simple. For the convenience of testing, we did not directly write to Elasticsearch, but used stdout to output for testing. Logstash listens on port 5044 of the elastic agent. We use the following command to start Logstash:

./bin/logstash -f logstash

$ pwd
/home/parallels/logstash/logstash-8.8.1
parallels@ubuntu2004:~/logstash/logstash-8.8.1$ ./bin/logstash -f logstash.conf

Reconfigure Elastic Agents

Above we have started the Elastic Agent. To be able to reconfigure the Elastic Agent, we need to stop its previous service and reinstall it. We first enter the following command:

parallels@ubuntu2004:~$ su
Password: 
root@ubuntu2004:/home/parallels# cd /opt/Elastic/Agent/
root@ubuntu2004:/opt/Elastic/Agent# ./elastic-agent uninstall
Elastic Agent will be uninstalled from your system at /opt/Elastic/Agent. Do you want to continue? [Y/n]:y
Elastic Agent has been uninstalled.

We can get the IP address of the current Logstash running through the following command:

 ip a | grep inet
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
    inet 192.168.0.8/24 brd 192.168.0.255 scope global noprefixroute eth0
    inet6 fe80::d246:4880:928b:f508/64 scope link noprefixroute

Above, we uninstalled the previously installed Elastic Agent. Let's reconfigure the elastic-agent.yml file next:

elastic-agent.yml

We again use the following command to start the elastic agent:

sudo ./elastic-agent install

parallels@ubuntu2004:~/fleet/elastic-agent-8.8.1-linux-arm64$ sudo ./elastic-agent install
[sudo] password for parallels: 
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
Do you want to enroll this Agent into Fleet? [Y/n]:n
Elastic Agent has been successfully installed.

We can use the following command to check:

./elastic-agent inspect

From the output above, we can see that outputs is written to port 5044 of Logstash.

We can switch to the terminal where Logstash is running to view:

From the terminal of Logstash, we can see that there are many outputs. We need to note that in our previous configuration, we did not write data to Elasticsearch. In order to write data to Elasticsearch, we must reconfigure the logstash.conf file. We stop Logstash from running. We can refer to the articles " Logstash: How to connect to a cluster with HTTPS access " and " Beats: Using fingerprint to connect Beats/Logstash and Elasticsearch ".

logstash.conf

input {
  elastic_agent {
    port => 5044
  }
}

output {
   stdout {}

   elasticsearch {
      hosts => ["https://192.168.0.3:9200"]
      index => "data-%{+YYYY.MM.dd}"
      ssl => true
      ilm_enabled => true
      user => "elastic"
      password => "z5nxTriCD4fi7jSS=GFM"
      ca_trusted_fingerprint => "783663875df7ae1daf3541ab293d8cd48c068b3dbc2d9dd6fa8a668289986ac2"
    }
}

For the user, password and ca_trusted_fingerprint here, please refer to the elastic-agent.yml configuration above.

We rerun Logstash again:

./bin/logstash -f logstash.conf

Let's go back to Kibana's interface to check:

We can see the newly generated data-2023.06.21 index file. In this way, we write the data into Logstash through Logstash, and then write it into Elasticsearch.

Configure the elastic-agent.yml file through Kibana

In our demonstration above, we used the default elastic-agent.yml file. It can only collect metrics information for the current machine. If we want to use it for many other integration operations, its manual configuration will be very troublesome, and we are not very good at it. You can study the above elastic-agent.reference.yml file in the elastic-agent installation directory in detail. Before performing the following exercises, we still download the previously installed elastic agent as before:

parallels@ubuntu2004:~$ su
Password: 
root@ubuntu2004:/home/parallels# cd /opt/Elastic/Agent/
root@ubuntu2004:/opt/Elastic/Agent# ls
data           elastic-agent.reference.yml  fleet.enc       LICENSE.txt  README.md
elastic-agent  elastic-agent.yml            fleet.enc.lock  NOTICE.txt   vault
root@ubuntu2004:/opt/Elastic/Agent# ./elastic-agent uninstall
Elastic Agent will be uninstalled from your system at /opt/Elastic/Agent. Do you want to continue? [Y/n]:y
Elastic Agent has been uninstalled.

For the integration of many replications, we need to use the tools integrated in Kibana to help us configure the elastic-agent.yml file.

We open Kibana:

As shown above, Elastic provides us with rich integrations to collect data conveniently, such as our common Nginx, Apache, etc. As an example, we choose the Apache server for demonstration. Follow the command below to install on Ubuntu OS:

sudo apt install apache2

From the above output, we can see that the Apache server has been successfully running.

Next let's create how to collect the elastic-agent.yml file of the Apache server.

Let's copy the above elastic-agent.yml content.

Let's go back to the root directory of the previously installed elastic agent:

parallels@ubuntu2004:~/fleet/elastic-agent-8.8.1-linux-arm64$ mv elastic-agent.yml elastic-agent.back.yml
parallels@ubuntu2004:~/fleet/elastic-agent-8.8.1-linux-arm64$ vi elastic-agent.yml

Let's paste the content of elastic-agent.yml copied above:

We modify it according to our own configuration:

We save the above elastic-agent.yml file and use the following command to install it:

sudo ./elastic-agent install

parallels@ubuntu2004:~/fleet/elastic-agent-8.8.1-linux-arm64$ sudo ./elastic-agent install
[sudo] password for parallels: 
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
Do you want to enroll this Agent into Fleet? [Y/n]:n
Elastic Agent has been successfully installed.

We click on the first link above:

We hit the refresh button several times on the Apache page:

Let's check this page:

We can go back to the Stack Management interface to view:

We can view it through Discover:

Write Apache data to Elasticsearch through Logstash

In order to be able to write Apache's daily data through Logstash, we only need to modify the elastic-agent.yml file:

elastic-agent.yml

We start Logstash. We first uninstall the previously installed elastic agent, and re-run the elastic agent again. We can see the following output in the terminal of Logstash:

We run the following command several times:

curl http://192.168.0.8:80

We can view it in Kibana:

We can see that the data is correctly written to Elasticsearch.

Observability: How to input the data collected by Elastic Agent into Logstash and finally write it into Elasticsearch