Optimization parameters of kyuubi (full)
kyubi.dynamic.resource: Set to true to enable dynamic resource adjustment. This will allow Kyuubi to automatically adjust resource allocation for tasks based on current load and resource availability.
kyubi.server.endpoint: Set to the address and port number of the Kyuubi server so that Hue can connect to the Kyuubi server.
kyubi.server.heartbeat.interval: Set the heartbeat interval of Kyuubi server in milliseconds. Shorter intervals allow for faster failure detection and failover.
kyubi.server.session.idle.timeout: Set the session idle timeout in milliseconds. When a session has been inactive for a specified amount of time, it will be automatically closed.
kyubi.llap.max.executors: If LLAP (Low Latency Analytical Processing) service is used in your cluster, you can set this parameter to specify the maximum number of LLAP executors that Kyuubi can use.
kyubi.session.auto.spill.threshold: Set whether Kyubi will automatically enable the function of spilling to disk when the memory usage reaches a certain threshold. A lower threshold can effectively reduce memory pressure, but may affect query performance.
Here are some common parameters that can be used to optimize Kyuubi:
kyubi.server.maxConcurrency: Set the maximum number of concurrent connections allowed by the Kyuubi server. According to your hardware resources and load conditions, adjust this parameter appropriately to balance concurrent connections and performance.
kyubi.server.thrift.max.worker.threads: Set the maximum number of worker threads used by Kyuubi server Thrift service. Adjust this parameter appropriately to improve the processing capacity of concurrent requests.
kyubi.server.heartbeat.interval: Set the heartbeat interval between Kyuubi server and client. Shorter intervals allow for faster failure detection and failover.
kyubi.server.session.idle.timeout: Set the session idle timeout. When a session has been inactive for a specified amount of time, it will be automatically closed.
kyubi.server.task.commit.threads: Set the number of Kyuubi server task commit threads. Appropriately increasing the value of this parameter can improve the concurrency performance of task submission.
kyubi.server.task.execute.threads: Set the number of Kyuubi server task execution threads. Appropriately increasing the value of this parameter can improve the concurrent performance of task execution.
kyubi.server.task.result.cache.enable: Enables caching of Kyuubi server task results to speed up execution of repeated queries. Make sure to properly allocate enough memory to store the result cache.
kyubi.sql.legacy.engine.threshold: Sets the size threshold of queries that can be run using the Legacy Engine. For smaller queries, you can choose Legacy Engine to reduce the overhead of starting an additional Engine.
These parameters can be set by editing Kyuubi's configuration files (such as kyuubi.conf), and the specific file paths and parameter names may vary depending on the version of Kyuubi. Make sure to back up the relevant configuration files and restart the Kyuubi service before modifying any configuration parameters.
kyubi.server.max-connections: This parameter is used to configure the maximum number of connections allowed by the Kyuubi server. You can adjust this value according to your load and resource conditions to balance concurrent connections and resource consumption.
kyubi.server.max-threads: Used to configure the maximum number of threads for processing requests. Properly adjusting this parameter can improve concurrent processing capabilities, but be careful not to set it too high to avoid resource competition and performance degradation.
kyubi.server.max-execution-mem-bytes: This parameter configures the maximum execution memory per query. You can adjust based on the available memory of the cluster and query needs to avoid under-memory or over-allocation problems.
kyubi.server.fetch-result-page-size: Used to set the number of records fetched by each page. Properly adjusting this parameter can balance the overhead of network transmission and the real-time responsiveness of queries.
kyubi.server.result-set-buffer-size: This parameter sets the size of the result set buffer. You can adjust according to the size of the query data volume and available memory to optimize the data reading performance.
kyubi.server.query.max-execution-time: used to set the maximum execution time threshold of the query. Queries that exceed this threshold are canceled to prevent long-running queries from hogging resources and causing timeouts.
kyubi.server.enable-hive-authz: Enable this parameter to use Hive's permission control mechanism to authorize queries.
kyubi.server.maxTotal: Set the maximum number of connections to Kyuubi server. Increase this value to support more concurrent connections, the default value is 100.
kyubi.server.connectionTimeout: Set the Kyuubi server connection timeout in milliseconds. If the connection is not established within the specified time, the connection will be interrupted, the default value is 30000.
kyubi.session.concurrentMax: Set the maximum number of concurrent running sessions per user. This value can be adjusted according to cluster resources and load conditions, and the default is 100.
kyubi.session.openSessionTimeout: Set session open timeout in milliseconds. The session request will timeout if the session is not opened within the specified time, default is 60000.
kyubi.session.queryTimeout: Set the query timeout in seconds. If the query is not completed within the specified time, it will be canceled, the default is 0, which means no timeout limit.
kyubi.session.autoClearInterval: Set the interval for auto-clearing sessions, in seconds. This option enables the automatic cleaning function of the session, and the default value is 0, which means disabled.
kyubi.server.binaryTransfer: Set whether to enable binary transfer mode. Enabling this mode can improve the efficiency of data transfer, the default is false.
kyubi.session.enableHiveDynamicPartitionPruning: Enable Hive dynamic partition pruning. Enable this option to select the appropriate partition for optimization according to the metadata of the partition when executing the query. The default is true.
Kyuubi is an open source Spark SQL remote JDBC server that provides some parameters that can be used to optimize performance and tune behavior. Here are some common Kyuubi optimization parameters:
kyuubi.server.max.connections: Set the maximum number of connections allowed by the Kyuubi server. By increasing the value of this parameter, more concurrent connections can be supported. The default value is 100, adjust it according to your needs.
kyuubi.server.backlog: Set the maximum length of the request queue for the Kyuubi server to listen for connections. Larger values can support more concurrent connection requests. The default value is 1024.
kyuubi.server.session.check.interval: Set the time interval for Kyuubi server to check for idle sessions. A shorter interval releases idle sessions and resources more promptly. The default is 60000 milliseconds.
kyuubi.session.conf.sync.interval: Set the time interval for Kyuubi session configuration parameter synchronization. A shorter interval allows for more timely application of configuration changes. The default is 60000 milliseconds.
kyuubi.statement.max.result.size: Set the maximum number of rows returned by Kyuubi for each query result. By limiting the size of the result set, you can reduce the amount of data transferred over the network. The default value is -1, which means no limit.
kyuubi.operation.heartbeat.interval: Set the heartbeat interval between Kyuubi and the client. A shorter interval allows for more timely detection of the client's connection status. The default is 1000 milliseconds
kyubi.server.session.idle.timeout: Set the session idle timeout in milliseconds. Automatically closes a session when there is no activity for a specified amount of time