• -------------------------------------------------------------
  • ====================================

EsRejectedExecutionException排错与线程池类型

elasticsearch dewbay 5年前 (2019-05-08) 2336次浏览 已收录 0个评论 扫描二维码

1、EsRejectedExecutionException异常示例

java.util.concurrent.ExecutionException: RemoteTransportException[[node-client10][10.93.21.21:9300][indices:data/write/update]]; nested: RemoteTransportException[[node-client
09][10.93.18.35:9300][indices:data/write/update[s]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.netty.MessageChannelHandler$Reque
stHandler@661fbe1d on EsThreadPoolExecutor[index, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@637f6431[Running, pool size = 12, active
threads = 12, queued tasks = 200, completed tasks = 15656095]]];

2、EsRejectedExecutionException异常解释

EsRejectedExecutionException异常,从字面意思上看是 ES 拒绝执行请求。这个异常的触发场景如下。

使用 Elasticsearch 的时候,在并发查询量大的情况下,访问流量超过了集群中单个 Elasticsearch 实例的处理能力,Elasticsearch 服务端会触发保护性的机制,拒绝执行新的访问,并且抛出EsRejectedExecutionException异常。

这个保护机制与异常触发是由 Elasticsearch API 实现中的 thread pool 与配套的 queue 决定的。

在示例中,Elasticsearch 为 index 操作分配的线程池,pool size=12, queue capacity=200,当 12 个线程处理不过来,并且队列中缓冲的 tasks 超过 200 个,那么新的 task 就会被简单的丢弃掉,并且抛出EsRejectedExecutionException异常。

官方的详细解释链接在这里:

这里是几个重要线程池的默认配置,其中很多配置都与 processors(cpu core)有关。

1)processors

processors 指的是 cpu 的 core 数,这个核数可以被自动的探知并且在线程池的配置中自动引入,processors 不必在 elasitcsearch.yml 中配置,当然你可以重写配置。

重写方法:

 processor: 8

若在一台机器上部署多个 Elasticsearch node,可以将 cpu cores 平均分配给不同的 node,例如 2nodes 部署在 12 个 core 的机器上,可以配置 processors = 6。

2)线程池与队列

一个 Elasticsearch 节点会有多个线程池,但重要的是下面四个: 
索引(index):主要是索引数据和删除数据操作(默认是 cached 类型) 
搜索(search):主要是获取,统计和搜索操作(默认是 cached 类型) 
批量操作(bulk):主要是对索引的批量操作(默认是 cached 类型) 
更新(refresh):主要是更新操作(默认是 cached 类型) 

generic
For generic operations (e.g., background node discovery). Thread pool type is scaling.
index
For index/delete operations. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.
search
For count/search/suggest operations. Thread pool type is fixed with a size of int((# of available_processors * 3) / 2) + 1, queue_size of 1000.
get
For get operations. Thread pool type is fixed with a size of # of available processors, queue_size of 1000.
bulk
For bulk operations. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.
snapshot
For snapshot/restore operations. Thread pool type is scaling with a keep-alive of 5m and a max of min(5, (# of available processors)/2).
warmer
For segment warm-up operations. Thread pool type is scaling with a keep-alive of 5m and a max of min(5, (# of available processors)/2).
refresh
For refresh operations. Thread pool type is scaling with a keep-alive of 5m and a max of min(10, (# of available processors)/2).
listener
Mainly for java client executing of action when listener threaded is set to true. Thread pool type is scaling with a default max of min(10, (# of available processors)/2).

3)线程池类型

fixed

The fixed thread pool holds a fixed size of threads to handle the requests with a queue (optionally bounded) for pending requests that have no threads to service them.

The size parameter controls the number of threads, and defaults to the number of cores times 5.

The queue_size allows to control the size of the queue of pending requests that have no threads to execute them. By default, it is set to -1 which means its unbounded. When a request comes in and the queue is full, it will abort the request.

fixed 线程池保持固定个数的线程来处理请求队列。 

size 参数设置线程的个数,默认设置是 cpu 核心数的 5 倍。

queue_size 可以控制待处理请求队列的大小。默认是设置为-1,意味着无限制。当一个请求到来但队列满了的时候,reject_policy 参数可以控制它的行为。默认是 abort,会使那个请求失败。设置成 caller 会使该请求在 io 线程中执行。 

threadpool:   
    index:   
        type: fixed   
        size: 30   
        queue: 1000   
        reject_policy: caller  

scaling

The scaling thread pool holds a dynamic number of threads. This number is proportional to the workload and varies between the value of the core and max parameters.

The keep_alive parameter determines how long a thread should be kept around in the thread pool without it doing any work.

scaling 线程池保持着动态数量的线程。在 core 和 max 数量之间的动态变化,keep_alive 配置的时间,维持了线程长时间未被调用。

thread_pool:
    warmer:
        core: 1
        max: 8
        keep_alive: 2m

blocking

在不同的版本有不同的类型。blocking 类型的线程池特征是

blocking 线程池允许设置一个最小值(min,默认为 1)和线程池大小(size,默认为 cpu 核心数的 5 倍)。它也有一个等待队列,队列的大小(queue_size )默认是 1000,当这队列满了的时候。它会根据定好的等待时间(wait_time,默认是 60 秒)来调用 io 线程,如果没有执行就会报错。 

threadpool:   
    index:   
        type: blocking   
        min: 1   
        size: 30   
        wait_time: 30s  

露水湾 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:EsRejectedExecutionException排错与线程池类型
喜欢 (0)
[]
分享 (0)
关于作者:
发表我的评论
取消评论

表情 贴图 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址