Flink1.11 memory model and parameter adjustment

Flink TaskManager startup log

INFO  [] - Loading configuration property: taskmanager.memory.process.size, 2048m
INFO  [] - Loading configuration property: taskmanager.memory.managed.fraction, 0.3
INFO  [] - Loading configuration property: taskmanager.memory.jvm-overhead.fraction, 0.2
INFO  [] - Loading configuration property: taskmanager.memory.network.fraction, 0.1
INFO  [] - Loading configuration property: parallelism.default, 2
INFO  [] - Loading configuration property: task.cancellation.timeout, 0
INFO  [] - Final TaskExecutor Memory configuration:
INFO  [] -   Total Process Memory:          2.000gb (2147483648 bytes)
INFO  [] -     Total Flink Memory:          1.350gb (1449551456 bytes)
INFO  [] -       Total JVM Heap Memory:     701.440mb (735513127 bytes)
INFO  [] -         Framework:               128.000mb (134217728 bytes)
INFO  [] -         Task:                    573.440mb (601295399 bytes)
INFO  [] -       Total Off-heap Memory:     680.960mb (714038329 bytes)
INFO  [] -         Managed:                 414.720mb (434865454 bytes)
INFO  [] -         Total JVM Direct Memory: 266.240mb (279172875 bytes)
INFO  [] -           Framework:             128.000mb (134217728 bytes)
INFO  [] -           Task:                  0 bytes
INFO  [] -           Network:               138.240mb (144955147 bytes)
INFO  [] -     JVM Metaspace:               256.000mb (268435456 bytes)
INFO  [] -     JVM Overhead:                409.600mb (429496736 bytes)

Insert picture description here

  1. Total Process Memory: 2048m total resources of flink, parameter: taskmanager.memory.process.size
  2. JVM Metaspace: JVM metaspace, parameter: taskmanager.memory.jvm-metaspace.size, calculation: default value 256m
  3. JVM Overhead: JVM additional overhead, parameter: taskmanager.memory.jvm-overhead.fraction, calculation: 2048*0.2=409.6m
  4. Total Flink Memory: the number of resources available for flink, calculation: from 1-2-3=1.35g
  5. Total JVM Heap Memory: JVM heap memory, calculation: 4-6=701.44m
  6. Total Off-heap Memory: JVM off-heap memory, calculation: 7+8=701.44m
  7. Total JVM Direct Memory: Calculation: 9+10 (default is 0)+11=266.240m
  8. Managed: Managed memory, parameter: taskmanager.memory.flink.size, calculation: Total Flink Memory 0.6=1.35 0.3= 414.72m
  9. Network: network cache, parameter: taskmanager.memory.network.fraction, calculation: Total Flink Memory 0.1=1.35 0.1= 138.240m
  10. Framework: Flink framework memory, parameter: taskmanager.memory.framework.heap.size, calculation: default 128m
  11. Task: task memory, calculation: 5-10=573.440m

Guess you like

Origin blog.csdn.net/jklcl/article/details/112253510