Kibana dashboards allow you to quickly and easily build visualizations on top of data indexed in Elasticsearch. However, the UI abstraction on Elasticsearch can make it tricky to diagnose performance issues in a single pane—especially for developers accustomed to digging into implementation details with debuggers and other profiling tools.
This blog describes four methods to help investigate and diagnose performance issues with Kibana dashboards. Specifically, we'll cover the use of Elastic Application Performance Monitoring (APM), debugging and profiling through the Chrome developer tools, inspecting individual dashboards through the Inspector, and the Elasticsearch Slow Log to determine why a dashboard is running slowly.
APM
When faced with slow dashboards, it is advisable to rule out performance hits caused by overall dashboard loading in Kibana itself. It could be a specific service or plugin that is causing the underlying issue. This can be achieved using Chrome DevTools debugging or using the Elastic APM Node.js agent, which is built into Kibana for debugging purposes. Follow the steps in the Debugging Kibana section of the developer guide to set up a locally running Kibana and APM configuration. Following these steps will set up a locally running Elasticsearch instance and APM server using Docker:
To ensure Kibana starts successfully, make sure to check out compatible versions of Elasticsearch and Kibana forks from each repository, and that the kibana.dev.yml configuration points to your locally running Elasticsearch cluster and APM server. Example user credentials for Kibana (including admin ) can be used to log in, as described in the @elastic/apm-integration-testing README. Using the yarn start command on the Kibana repository will give you a running instance of Kibana. Including non-performance dashboards can be achieved by exporting dashboards from the Save Objects screen in Kibana and importing them into a locally running instance via the same screen. Be sure to export dependent objects so that individual controls and indexes are exported with the dashboard:
Accessing your dashboard locally will allow the APM Node.js agent to capture any errors and traces in the dashboard navigation. You will see the /dashboard endpoint triggered to access your dashboard. APM will catch these requests in kibana-frontend service.
Selecting this trace will give you key trace information for the dashboard. The trace breakdown (shown in the screenshot below) enables you to closely examine the execution time of the dashboard execution.
A breakdown will show load times and percentages of total execution time for JavaScript file loads, such as Kibana plugins, or response times for key Kibana API calls. This makes APM information useful for identifying plugins that are slowing down your dashboard, especially if you're using custom or community-developed plugins. It can also help discover latencies in critical calls like Kibana security. Or, with runtime fields on the APM index, it can help answer usage questions such as which space is being used the most or which dashboard has the most page loads.
A key limitation of using APM for Kibana is that the captured traces only cover critical calls for a given logical unit at the page level. In the Metadata tab, the label information shows that these traces are captured at the dashboard level and do not provide information for each individual visualization, as shown in the trace metadata described below.
Chrome Developer Tools
With a version of Kibana running locally, you can also step through code using the Chrome developer tools. This is more useful when tracking bugs or unexpected behavior in Kibana. After following the same local setup steps as Elastic APM, you need to start Kibana with Node inspect support according to the Kibana tutorial documentation :
node --inspect-brk scripts/kibanaNavigating to chrome://inspect in your Chrome browser will allow you to attach native Node.js debugging tools to a running Kibana process:
Inspecting the Kibana process opens DevTools, giving access to all the most popular developer tools, such as the debugger. Setting a breakpoint in the bulk_resolve POST method found in the APM trace allows you to capture the dashboard object. This call can also be inspected using the Chrome DevTools network tab.
While the VEGA_DEBUG.view global variable enables us to inspect the status of any Vega chart via Chrome Developer Tools as outlined in the documentation , we need to be able to view details of other visualizations as well. The screenshot above shows the details of each panel in the dashboard, including the unique ID and type of the control. This recursive call will also allow you to view the result objects of the various panels, such as this map control, where we can see the JSON objects for the different layers:
Identify slow queries by searching the slow log
Expensive queries from the dashboard can negatively impact the Elasticsearch server. Elasticsearch provides a search slow log, which captures slow search queries into a dedicated log file. Pinpointing slow searches to raw visualizations can help identify which specific controls in the dashboard are affecting Elasticsearch performance.
Queries logged to the slow log are determined per-index by the thresholds set for the query and fetch phases. Check out this helpful blog and the search_type option in the Elasticsearch documentation for details on the default Query Then Fetch algorithm.
Set thresholds for query and fetch phases at four different log levels: warning ( warn ), information ( info ), debug ( debug ), and trace ( trace ). Thresholds are disabled with a value of -1 and can be changed at the index level in the developer console using the _settings API endpoint:
PUT /kibana_sample_data_logs/_settings
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.threshold.fetch.warn": "1s",
"index.search.slowlog.threshold.fetch.info": "800ms",
"index.search.slowlog.threshold.fetch.debug": "500ms",
"index.search.slowlog.threshold.fetch.trace": "200ms"
}When a threshold is configured, logs at the configured level are sent to the slow log location. The search log location depends on whether you are using a local Elasticsearch installation or Elastic Cloud. For local installations, the slow logs are located in the logs folder, which defaults to /var/log/elasticsearch/ unless you override the log location in your elasticsearch.yml configuration. Elastic Cloud deployments require enabling logs and metrics on your cluster by following the Enabling Logging and Monitoring steps in the documentation . Logs and metrics will be sent to the specified cluster.
Regardless of where your logs are located, the slow log will display messages similar to the following:
Looking at the messages in the log, it is difficult to correlate this source message with a single dashboard control. Each numbered item corresponds to the following entity. Element 1 contains the unique ID for the Kibana dashboard. Even more interesting is paragraph 2, which contains colon-separated information about a particular visualization:
agg_based:gauge:69a34b00-9ee8-11e7-8711-e7a007dcef99This example contains three key pieces of information:
- The type of visualization, in this case Lens
- The type of control, here is the Legacy Metric control
- Unique ID of the visualization that generated the slow query
Starting with Elastic Stack 8.x, these IDs are unique in the global space, unlike previous versions where they were unique in the Kibana space in which they reside. More details are provided in the documentation .
A more human-friendly format is obtained by cross-referencing the visualization IDs from elasticsearch.slowlog and kibana.log . The kibana.log message for the above gauge control shows the description highlighted in blue:
If the title is visible, or in the "Saved Object" screen, the decription can match the title in the dashboard itself:
The Inspect panel in Saved Objects is also available for cross-referencing IDs:
Inspector and Search Analyzer
Once the problematic visualization is identified, the Inspector will help you access the query. In edit mode on a dashboard, each control's submenu will have this option:
Selecting the Request view initially displays key statistics for the underlying query , including query time, number of hits returned, and number of documents matching the query. Returning hits is not the same as matching matches, not only because by default Elasticsearch queries return 10 hits, but also because many aggregations do not return documents to include in their views, since we are interested in visualizing the aggregation results.
The Request tab shows the query executed by this particular control. There are also options to copy the query to run in the developer console, or to use the Search Profiler to identify slow phases of the query's search and aggregation profile.
The functionality of each Inspect panel depends on the type of visualization used to provide appropriate debugging options. The Inspect panel for maps allows you to view queries per layer. Vega charts display values of datasets and key signals for debugging . These are useful for bird's-eye view of values in charts. For more advanced debugging of Vega charts using Google Chrome Developer Tools, follow this guide to inspect view objects.
Summarize
Elastic APM, Kibana Inspector and Search Profiler, and Elasticsearch Slow Log are powerful tools for diagnosing dashboard performance issues. After using these techniques and optimizing your queries, if you are still struggling with poor performance, try configuring Kibana to load dashboard panels when they are visible, as described in the Improving Dashboard Load Time documentation .
Happy debugging!