Apache Flink 1.10 has made major changes to the task manager's memory model and the configuration options of the Flink application. These recently introduced changes make Flink more suitable for various deployment environments (e.g. Kubernetes, Yarn, Mesos), thereby strictly controlling its memory consumption. This article will introduce the Flink memory model in Flink 1.10, how to set and manage the memory consumption of Flink applications.
1. Introduction to Flink memory model
A clear understanding of Apache Flink's memory model can enable developers to more effectively manage the resources of various workloads. The following figure illustrates the main memory components in Flink: the
task manager process is a JVM process. From a high point of view, its memory is composed of JVM Heap and Off-Heap memory. These types of memory are used directly by Flink or used by the JVM for its specific purposes (for example, metaspace, etc.). There are two main memory users in Flink: the programmer's user code and the framework itself occupy internal data structures, network buffers, and other memory.
Please note : User code can directly access all memory types: JVM Heap, Direct and Native memory. Therefore, Flink cannot really control its distribution and use. However, there are two types of off-heap memory for tasks, and they are explicitly controlled by Flink:
(1) Managed memory (outside the heap)
(2) Network buffer
The latter is a part of the JVM's direct memory, allocated to the exchange of user records between programmer tasks.
2. How to set Flink memory
In Flink 1.10, in order to provide a better user experience, the framework provides advanced and fine-grained adjustments of memory components. There are basically three options for setting memory in the task manager.
The first two (and the simplest) alternatives are to configure one of the following two options for the total memory used by the task manager’s JVM process:
(1) Total process memory: Flink Java applications (including user code) and The total memory consumed by the JVM to run the entire process.
(2) Flink total memory: only the memory consumed by Flink Java applications, including user code, but excluding the memory allocated by the JVM for its operation.
It is recommended to configure "total Flink memory" for standalone mode . In this case, it is a common practice to explicitly declare how much memory is allocated for Flink, but the external JVM overhead is very small. For deploying Flink in a containerized environment (such as Kubernetes, Yarn, or Mesos), it is recommended to use the Total Process Memory option, because it becomes the total memory size of the requested container. Containerized environments usually strictly enforce this memory limit.
If you want more fine-grained control over the size of JVM heap and managed memory (non-heap), there is another way to configure Task Heap and Managed Memory at the same time. This choice allows a clear separation between heap memory and any other memory types.
Consistent with the flink community's efforts to unify batch and stream processing, this model is common in both cases. It allows the sharing of JVM heap memory between the user code of operator tasks in any workload and the heap state backend in the stream processing scheme. In a similar way, managed memory can be used for the RocksDB state backend in batch overflow and streaming.
The remaining memory components will be automatically adjusted according to their default values or other configured parameters. Flink also checks the overall consistency. You can find more information about the different memory components in the corresponding documentation.
3. Other components
When configuring Flink's memory, the size of different memory components can be fixed with the value of the corresponding option, or can be adjusted using multiple options. Below we provide more information about memory settings.
Score of total Flink memory
This method allows the total Flink memory to be subdivided proportionally, where managed memory (if not explicitly set) and network buffers can occupy part of it. Then, the remaining memory is allocated to the task heap (if not explicitly set) and other fixed JVM heap and off-heap components. The following figure represents an example of this setting:
Please note:
(1) Flink will verify that the size of the derived network memory is between its minimum and maximum values, otherwise Flink will fail to start. The maximum and minimum limits have default values, which can be overridden by the corresponding configuration options.
(2) Generally, Flink treats the configured score as a reminder. In some cases, the derived value may not match the score. For example, if "Total Flink Memory" and "Task Heap" are configured as fixed values, "Managed Memory" will get a certain percentage, and "Network Memory" will get the remaining memory that may not exactly match the ratio.
More tips to control the memory limit of the container
Heap and direct memory usage are managed by the JVM. There are many other possible sources of local memory consumption in Apache Flink or its user applications, which are not managed by Flink or JVM. It is often difficult to control their limits, which can complicate debugging of potential memory leaks. If the Flink process allocates too much memory in an unmanaged way, it can usually cause the Task Manager container to be killed in a containerized environment. In this case, it may be difficult to understand which type of memory consumption has exceeded its limit. Flink 1.10 introduced some specific adjustment options to clearly represent these components. Although Flink cannot always enforce strict limits and restrictions, the idea here is to explicitly plan memory usage.
(1) The status of RocksDB cannot become too large. The memory consumption of the RocksDB state backend is solved in managed memory. RocksDB obeys its restrictions by default (only since Flink 1.10). You can increase the size of managed memory to improve the performance of RocksDB, or you can reduce the size of managed memory to save resources.
(2) User code or its dependencies will consume a lot of off-heap memory. Adjust the "task outside heap" option to allocate other direct or native memory for user code or any of its dependencies. Flink cannot control native allocation, but it sets a limit for JVM Direct memory allocation. The direct memory limit is enforced by the JVM.
(3) JVM meta space requires additional memory. If you encounter OutOfMemoryError: Metaspace, Flink provides an option to increase its limit, and the JVM will ensure that the limit is not exceeded.
(4) JVM needs more internal memory. It is not possible to directly control certain types of JVM process allocation, but Flink provides JVM overhead options. These options allow you to declare the amount of additional memory that is expected for these allocations and is not covered by other options.
4 Conclusion
The Flink version (Flink 1.10) has made some major changes to Flink's memory configuration, so that it can manage application memory and debug Flink better than before. Future developments in this area also include the use of a similar memory model for the job manager process in FLIP-116, so please stay tuned for more new features in the upcoming version.
This article is translated from the official website of flink