1. Resource management configuration
The user can configure the number of resources that the resource manager can allocate through the following two parameters (in hawq-site.xml).
-
hawq_rm_memory_limit_perseg: Configure the memory size that each node can use, the default value is 64GB
-
hawq_rm_nvcore_limit_perseg: Configure the number of vcores that each node can use, the default value is 16
If we have 8 slave nodes and the configuration is as follows, the number of resources we can use in this OushuDB cluster is: 128GB * 8 = 1024GB memory, 16 * 8 = 128 vcores.
<property><name>hawq_rm_memory_limit_perseg</name><value>128GB</value></property><property><name>hawq_rm_nvcore_limit_perseg</name><value>16</value></property>
In the resource queue tree, only leaf nodes can be associated with users to accept queries. The resources of the child node are allocated from the parent node. The sum of the proportions of MEMORY_LIMIT_CLUSTER and CORE_LIMIT_CLUSTER for all child nodes of one of the parent nodes must be less than or equal to 100%.
2. Advanced configuration of resource queue
Usually, users only need to specify the following three parameters when creating a resource queue:
-
ACTIVE_STATEMENTS: The maximum number of concurrent queries allowed by the resource queue. Connections that exceed this number will be queued. The resource manager will evenly allocate resources to queue concurrent queries.
-
MEMORY_LIMIT_CLUSTER: The proportion of parent queue memory that can be used by resource queues
-
CORE_LIMIT_CLUSTER: The proportion of the parent queue CPU that the resource queue can use
But in some cases the user can specify some advanced parameters to configure the resource management queue.
3. Set Virtual Segment memory usage
VSEG_RESOURCE_QUOTA refers to the amount of memory used in a virtual segment. The default value is 256MB. For example, a resource management queue can use 8GB of resources. If VSEG_RESOURCE_QUOTA is 256MB, the maximum number of virtual segments that can be allocated by this resource management queue is 8GB/256MB = 32. If we use 512MB, we can allocate up to 8GB/512MB = 16.
VSEG_RESOURCE_QUOTA=‘mem:{128mb | 256mb | 512mb | 1024mb | 2048mb | 4096mb | 8192mb | 16384mb | 1gb | 2gb | 4gb | 8gb | 16gb}’
Using a large VSEG_RESOURCE_QUOTA can sometimes avoid the spilling of operations such as aggregations, joins, etc., because each segment can use more memory.
If you find that some statements frequently spill, you can appropriately increase VSEG_RESOURCE_QUOTA.
E.g:
CREATERESOURCEQUEUEadhoc2WITH (PARENT='department3', ACTIVE_STATEMENTS=3,MEMORY_LIMIT_CLUSTER=50%, CORE_LIMIT_CLUSTER=50%, VSEG_RESOURCE_QUOTA='mem:512MB');