【译】保护 Consul 在特定设置中免受 RCE 风险的影响

2018年11月27日 Consul 团队

介绍

我们最近注意到了一组恶意软件，它们主要针对具有允许远程执行代码这一特定配置的 Consul nodes 。 我们的社区成员也 (负责任地) 报告了此恶意软件造成的事件, 并与我们合作, 在最近版本的 Consul 中添加了一个补丁, 以保护我们免受这种威胁。

这篇文章详细介绍了此恶意软件可能会如何影响用户, 具体取决于他们的配置, 并概述了我们已经采取的步骤, 来提供支持1.2.4、1.1.1、1.0.8 和0.9.4 版本的补丁, 使旧版本的 Consul 在没有重大版本升级的情况下也能很容易受到保护。

总结

如果您已将-enable-script-checks设置为 true", 或者正在运行 Consul 0.9.0 或更早版本, 并且Consul api 可通过网络访问的接口使用, 则应执行以下操作。

补救步骤:

1. 升级到下面链接的版本之一。
2. 如果需要脚本检查, 请将 Consul 代理上的 "-enable-script-checks" 更改为 "-enable-local-script-checks。
3. 禁用 Consul 服务器上的脚本检查。
4. 确保 Consul HTTP API 绑定到 loopback 接口, 而不是一个可公开访问的接口。
5. 如果尚未启用, 则启用 ACL 。

有关详细信息, 请继续阅读下面的内容。

如何利用脚本检查

脚本检查是 Consul 可以执行的健康检查的一种类型，用于确定目标服务的健康状态。支持的检查类型包括 HTTP、TCP、gRPC、Docker、Alias、TTL 和脚本检查。检查可以通过 Consul API 注册, 也可以由 Consul 代理从检查定义目录中加载。

脚本检查将在配置的时间间隔内执行 Consul 进程的任何命令或内联脚本。Docker 检查是相同的, 但通过 Docker daemon 的 API 在正在运行的 Docker 容器中执行脚本。默认情况下, 作为 Consul 0.9.0 的 Consul 安装中禁用脚本 (和 Docker) 检查, 尽管它们在许多情况下非常有用, 但存在潜在的安全风险。

Consul API 是用于与 Consul 代理交互的 HTTP 接口, 包括注册上面所述的运行状况检查定义。在大多数情况下, 我们建议将此 API 绑定到 loopback 接口, 以防止意外暴露。我们提供了一个广泛的 ACLl 系统, 可用于保护 API 交互, 包括注册检查。

使代理容易受到这种攻击的必要条件是:

• API 可通过网络访问的接口使用。
• 脚本检查已启用。
• ACL 被禁用或 ACL Token 被破坏。

在上述条件下, 攻击者可以对远程代理注册具有恶意负载检查。根据设计, 脚本检查允许任意执行代码, 因此允许通过公开的 API 启用检查的服务注册会带来 RCE (远程代码执行) 威胁。

Consul的早期设计假定HTTP API仅限于受信任的本地网络访问，直到后来才清楚许多用户由于各种原因不知道（或仍然选择）公开它, 这促使在 0.9.0 版本中更改为在默认情况下禁用脚本检查。

建议将启用 ACL 作为最佳做法, 但即使在限制性 ACL 到位的情况下, 如果攻击者用至少一个可以注册服务的 ACL 损害主机，则相同的ACL Token 将允许攻击者在任何公开的远程节点上注册脚本检查，因为根据设计，ACL的范围不限于单个代理。我们尚未发现恶意软件有利用本地可用的 ACL Token 针对受 ACL 保护的群集, 但主动指出此风险希望使用者们注意。

已知的影响

上个月我们有几份关于这个问题的报告, 并证实它正在变得更加普遍。我们要确保我们的 Consul 用户社区意识到这一威胁, 并知道如何保护自己免受这种攻击。

在每种已知情况下, 攻击者都通过 Consul 之外的一些不相关漏洞获得了对数据中心中一台主机的访问权限。然后, 攻击者在网络上检测到暴露的 Consul API (可能通过自动扫描), 并使用脚本检查将恶意软件传播到其他服务器。

修复

许多用户对其基础结构中的服务使用脚本检查功能, 因此完全禁用脚本检查将禁用他们所依赖的功能。

在本月初发布的领Consul 1.3.0 中, Consul 社区的一名成员提供了一个补丁, 添加了一个新的配置选项-- -enable-local-script-checks, 它允许注册脚本检查仅通过本地配置文件, 从而防止使用 HTTP API 注册恶意检查。我们建议所有需要脚本检查的用户切换到此选项, 并找到避免的方法, 具体取决于对代理的远程 API 访问。

除此之外，我们建议检查 ACL 系统和 Consul API 正在侦听的接口的使用。默认情况下脚本检查仍然完全禁用。

考虑到通过多个主要版本更改升级到 1.3.0 (在那里发布了此功能) 的挑战, 我们支持-enable-local-script-checks " 选项, 并发布了多个修补版本的 Consul, 以确保我们的用户可以以最小的干扰和努力来缓解这一威胁:

• 1.2.4
  • 更改日志链接
  • 下载链接
• 1.1.1
  • 更改日志链接
  • 下载链接
• 1.0.8
  • 更改日志链接
  • 下载链接
• 0.9.4
  • 更改日志链接
  • 下载链接

这些是该系列中的用户操作版本的替代。如果您当前使用的是 0.9.0 之前的某个版本, 我们建议升级到 0.9.4。与往常一样, 我们建议您在所有升级中遵循我们的升级指南。版本1.3.0 及以后已包含此功能。同一版本的企业版也已可用。

通常, 我们更喜欢看到用户升级, 很少出现这样的 backport 修复。由于此威胁的性质, 我们决定支持此修复程序, 以便用户可以快速保护自己, 而不需要可能带来重大更改的主要版本升级。

依赖通过 HTTP API 注册脚本检查的用户可以继续使用-enable-script-checks, 但应格外小心地启用 ACL, 并确保在网络上无法访问 API 侦听器。我们建议尽快找到不需要远程 HTTP API 访问的替代部署模式。

在 Consul 服务器上允许脚本检查尤其危险, 特别是为了全局工具或UI访问，在服务器上公开API是很常见的。服务器上的远程可注册脚本检查将面临从网络上的任何节点向攻击者暴露包括 ACL Token 在内的所有 Consul 状态的风险。我们正在考虑在未来版本的 Consul 中将在配置服务器上的脚本检查变成一个很难犯的错误。通常，Consul服务器无论如何都不会运行其他工作负载，因此禁用它们上的脚本检查不太可能会导致问题，但是请检查您的配置已确认没有启用它们，因为可能有其他客户端需要。

我们还在探索其他一些选项，以帮助通知用户这种潜在的不安全配置，并考虑对未来版本的脚本检查提供额外的保护。我们为脚本检查和代理配置在文档中添加了更加突出的警告，, 并在安全模型中添加了一个新的部分, 突出显示了配置 Consul 的已知不安全的方式。为了使运行中的 Consul 更易于安全访问, 我们在 Consul 1.4.0 中重新设计了ACL系统的接口,并且建议当前没有使用 ACL 的 Consul 用户使用这些接口。

---

【原文】Protecting Consul from RCE Risk in Specific Configurations

NOV 27 2018 THE CONSUL TEAM

Introduction

We’ve recently become aware of a set of malware targeting Consul nodes with a specific configuration which allows remote code execution. Members of our community also (responsibly) reported incidents caused by this malware, and worked with us to include a patch in a recent version of Consul that protects from this threat in the wild.

This post details how this malware may affect users, depending on their configuration, as well as outlines the steps we've taken to backport a patch for versions 1.2.4, 1.1.1, 1.0.8, and 0.9.4 to make it easy for older versions of Consul to be secured without a major version upgrade.

Summary

You should take action if you have -enable-script-checks set to true, or are running Consul 0.9.0 or earlier, and the Consul API is available on an interface that can be accessed over the network.

Steps to remediate:

1. Upgrade to one of the versions linked below.
2. Change -enable-script-checks to -enable-local-script-checks on Consul agents if script checks are required.
3. Disable script checks on Consul servers.
4. Ensure the Consul HTTP API is bound to a loopback interface instead of one publicly accessible.
5. Enable ACLs if not already enabled.

For full details, continue reading below.

How Script Checks Can Be Exploited

Script checks are one type of health check Consul can execute to determine the health status of a target service. Supported check types are HTTP, TCP, gRPC, Docker, Alias, TTL, and script checks. Checks can be registered via the Consul API or loaded from a directory of check definitions by the Consul agent.

Script checks will execute any command or inline script by the Consul process at the configured interval. Docker checks are the same, but execute the script within a running Docker container via the Docker daemon's API. Script (and Docker) checks are disabled by default in Consul installations as of Consul 0.9.0, and though they are useful for many cases present a potential security risk.

The Consul API is the HTTP interface for interacting with the Consul agent, including registering health check definitions as noted above. We recommend binding this API to the loopback interface in the majority of cases to prevent accidental exposure. We provide an extensive ACL system that can be utilized to protect API interactions, including registering checks.

The necessary conditions that make an agent vulnerable to this attack are:

• The API is available on an interface that can be accessed over the network.
• Script checks are enabled.
• ACLs are disabled or an ACL token is compromised.

Given the above conditions, an attacker can register a check on a remote agent with a malicious payload. By design, script checks allow arbitrary code execution, so allowing service registration with checks enabled via an exposed API presents an RCE (remote code execution) threat.

Consul's early design assumed that HTTP API was restricted to trusted local network access, and only later did it become clear that many users were not aware (or still chose) to expose it for a variety of reasons, prompting the change to disable script checks by default in 0.9.0.

Enabling ACLs is recommended as a best practice, but even with restrictive ACLs in place, if an attacker compromises a host with at least one ACL present that can register a service, that same ACL token will allow the attacker to register with a script check on any remote node that is exposed, since ACLs by design are not scoped to a single agent. We are not aware of malware exploiting locally-available ACL tokens against an ACL-protected cluster yet, but wish to proactively point out this risk.

Known Impact

We've had several reports of this in the last month and have confirmed that it is becoming more widespread. We want to ensure our community of Consul users is aware of this threat and knows how to protect themselves against this type of attack.

In each known case, the attacker gained access to one host in the datacenter via some unrelated vulnerability outside of Consul. The attacker then detected exposed Consul APIs on the network (likely though automated scanning) and used script checks to propagate the malware to other servers.

Remediation

Many users use the script check feature for services across their infrastructure, so disabling script checks altogether would disable functionality they rely on.

In Consul 1.3.0, released earlier this month, a member of the Consul community contributed a patch that adds a new configuration option, -enable-local-script-checks, which allows script checks to be registered only via local configuration files, thus preventing use of the HTTP API to register malicious checks. We recommend all users who need script checks switch to this option and find ways to avoid depending on remote API access to agents.

In addition to this, we recommend reviewing usage of the ACL system and interfaces that the Consul API is listening on. Script checks remain disabled entirely by default.

Given the challenge of upgrading through multiple major version changes to 1.3.0, where this feature was released, we have backported the -enable-local-script-checks option and released multiple patched versions of Consul to ensure our users can mitigate this threat with minimum disruption and effort:

• 1.2.4
  • Changelog link
  • Download link
• 1.1.1
  • Changelog link
  • Download link
• 1.0.8
  • Changelog link
  • Download link
• 0.9.4
  • Changelog link
  • Download link

These are drop-in replacements for users operating versions in that series. If you are currently using a version before 0.9.0, we recommend upgrading to 0.9.4 - there may be some breaking changes from earlier versions, but these are minimal compared with the much larger changes made in 1.0.0. As always, we recommend following our upgrade guidefor all upgrades. Version 1.3.0 and onwards already contain this feature. Enterprise binaries for the same versions are also available.

In general, we prefer to see users upgrade, and rarely backport fixes like this. Because of the nature of this threat, we decided to backport this fix so that users could protect themselves quickly, without needing a major version upgrade that might potentially introduce breaking changes.

Users that rely on registering script checks via the HTTP API can continue to use -enable-script-checks, but should take extra care to enable ACLs and to ensure the API listener is not accessible on the network. We recommend finding alternative deployment patterns where remote HTTP API access is not required as soon as possible.

It is especially dangerous to allow script checks on Consul servers, especially since it's common to expose the API on servers for global tooling or UI access. Remotely registerable script checks on servers risk exposing all Consul state including ACL tokens to the attacker from any node on the network. We are considering making it a hard error to configure script checks on servers in future versions of Consul. Generally Consul servers tend not to run other workloads anyway so disabling script checks on them isn't likely to cause a problem, but please check your configuration didn't enable them just because other clients need to.

We are also exploring several other options to help notify users of this potentially insecure configuration, as well as considering additional protections for script checks for future releases. We added more prominent warnings to our documentation for script checks and agent configuration as well as adding a new section to our security modelhighlighting known-insecure ways to configure Consul. As part of making running Consul securely more accessible, we redesigned the interfaces to our ACL system in Consul 1.4.0 and recommend usage of that for users not currently using ACLs with Consul.