OB删除1.5亿数据耗费2小时

目录

回顾：mysql是怎么删除数据的？

删除方案

代码实现

执行结果

结论

本篇是实际操作 批量处理数据以及线程池线程数设置 记录学习

背景：有一张用户标签表，存储数据量达4个亿，使用OceanBase存储，由于其中一种标签类型tagType1刷数时逻辑错误，所以需要将该类型数据全部删掉

回顾：mysql是怎么删除数据的？

首先我们得知道mysql是怎么删除数据的？

MySQL 删除操作使用延迟清理的方式。当使用 DELETE 语句删除数据时，MySQL 会将删除的数据标记为已删除，并在需要时将其从磁盘上清理掉。

如果我们要删除id=2这行数据，mysql执行流程如下

所以删除一行数据，需要先查询数据，然后再将数据标记已删除更新

删除方案

批量删除数据，按照以往经验，按照自增ID分页并发删除最快：1. 查询不需要回表；2. 更新可以避免数据库行锁

由于写数据时，是多个标签任务同时写入的，所以需要删掉的tagType1数据并不是连续的，意味着删除语句必须筛选tag_type = "tagType1"

可以查询出tagType1的最小ID和最大ID，然后进行分页查询

代码实现

public static final int endId = 237655461;
public static final int batchSize = 1000;
public static final int startId = 85427001;
public static final int threadNum = 6;

@Override
public void tmpDeleteClueTag() {
    log.info("deleteUserTag start");

    for (long i = startId; i < endId; i += batchSize * threadNum) {
        for (int j = 0; j < threadNum; j++) {
            final long from = i + j * batchSize;
            final long to = from + batchSize;
            executorService.submit(() -> {
                long l = System.currentTimeMillis();
                int result = clueUserTagGateway.deleteUserTag(from, to);
                log.info("deleteUserTag,{},{}->{},res:{},time:{}", Thread.currentThread().getName(), from, to,
                        result, System.currentTimeMillis() - l);
            });
        }
    }

    log.info("deleteUserTag success");
}

// 执行语句
DELETE FROM clue_user_tag where tag_type = 'spu' and id >= #{from} and id &lt; #{to}

// 线程池定义
public class CustomizeThreadPoolConfig {

    public static final int CORE_SIZE = 6;
    public static final int MAX_SIZE = 6;
    public static final int KEEP_ALIVE_TIME = 5;
    public static final int QUEUE_SIZE = 200;

    @Bean(name = "customerTagExecutor", destroyMethod = "shutdown")
    public ExecutorService CustomerTagExecutor() {
        return new TraceableThreadPoolExecutor(CORE_SIZE, MAX_SIZE, KEEP_ALIVE_TIME, TimeUnit.MINUTES,
                new LinkedBlockingQueue<>(QUEUE_SIZE), new CustomizableThreadFactory("pushExecutor-"),
                new ThreadPoolExecutor.CallerRunsPolicy());
    }

    public static class TraceableThreadPoolExecutor extends ThreadPoolExecutor {

        public TraceableThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,
                BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory, RejectedExecutionHandler handler) {
            super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory, handler);
        }
    }
}
  • 一次处理1000条，将id拆分开扔给线程池执行删除，可以避免数据库表行锁
  • 使用tag_type标签where删除无伤大雅，执行计划会根据id去优先查询的，查询出来后再从1000条中筛选出有tag_type匹配的情况，内存匹配是非常快的
  • 开启6个线程是因为我们数据平台只有6个线程去执行数据，如果线程数太多会造成数据平台侧资源争抢，一次删除请求会变慢

执行结果

刚开始，并不知道数据平台6个线程的限制，所以设置了20个线程

2024-12-16 15:47:46,478|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:4
2024-12-16 15:47:46,481|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:5
2024-12-16 15:47:46,481|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:3
2024-12-16 15:47:46,482|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:4
2024-12-16 15:47:46,483|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:2
2024-12-16 15:47:46,573|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:3098
2024-12-16 15:47:46,575|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:94
2024-12-16 15:47:46,577|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:95
2024-12-16 15:47:46,579|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:3295
2024-12-16 15:47:46,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:7
2024-12-16 15:47:46,581|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:98
2024-12-16 15:47:46,583|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:6
2024-12-16 15:47:46,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:5
2024-12-16 15:47:46,584|INFO |796ffea717b93262a1e1fb8799631878||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=,res:1000,time:3404
2024-12-16 15:47:46,589|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:1000,time:3008
2024-12-16 15:47:46,672|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:92
2024-12-16 15:47:46,673|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:98
2024-12-16 15:47:46,675|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:94
2024-12-16 15:47:46,675|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:3390
2024-12-16 15:47:46,676|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:93
2024-12-16 15:47:46,677|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:88
2024-12-16 15:47:46,677|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:4
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:95
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:3306
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:3372
2024-12-16 15:47:46,680|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:8
2024-12-16 15:47:46,680|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:5
2024-12-16 15:47:46,681|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:5
2024-12-16 15:47:46,682|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:7
2024-12-16 15:47:46,682|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:5
2024-12-16 15:47:46,774|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:95
2024-12-16 15:47:46,776|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:97
2024-12-16 15:47:46,778|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:98
2024-12-16 15:47:46,780|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:100
2024-12-16 15:47:46,781|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:100
2024-12-16 15:47:46,782|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:100
2024-12-16 15:47:46,784|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:102
2024-12-16 15:47:46,785|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:11
2024-12-16 15:47:47,079|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:303
2024-12-16 15:47:47,082|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:302
2024-12-16 15:47:47,082|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:304
2024-12-16 15:47:47,083|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:301
2024-12-16 15:47:47,084|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:303
2024-12-16 15:47:47,084|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:1000,time:3307
2024-12-16 15:47:47,085|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:300
2024-12-16 15:47:47,085|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:301
2024-12-16 15:47:47,184|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:105
2024-12-16 15:47:47,185|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:1000,time:3409
2024-12-16 15:47:47,185|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:103
2024-12-16 15:47:47,187|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:103
2024-12-16 15:47:47,188|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:105
2024-12-16 15:47:47,189|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:107
2024-12-16 15:47:47,273|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:188
2024-12-16 15:47:47,275|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:191
2024-12-16 15:47:47,277|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:192
2024-12-16 15:47:47,280|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:96
2024-12-16 15:47:47,281|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:96
2024-12-16 15:47:47,283|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:0,time:98
2024-12-16 15:47:47,285|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:98
2024-12-16 15:47:47,286|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:98
2024-12-16 15:47:47,288|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:99
2024-12-16 15:47:47,289|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:15
2024-12-16 15:47:47,374|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:99
2024-12-16 15:47:47,375|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:98
2024-12-16 15:47:47,485|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:0,time:202
2024-12-16 15:47:47,573|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:292
2024-12-16 15:47:47,575|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:295
2024-12-16 15:47:47,577|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:292
2024-12-16 15:47:47,578|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:292
2024-12-16 15:47:47,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:291
2024-12-16 15:47:47,581|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:292
2024-12-16 15:47:47,583|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:209
2024-12-16 15:47:47,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:209
2024-12-16 15:47:47,883|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-20|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-20,res:1000,time:3405
2024-12-16 15:47:47,987|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:403
2024-12-16 15:47:48,881|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-13|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-13,res:1000,time:3194
2024-12-16 15:47:49,184|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-7|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-7,res:1000,time:3211
2024-12-16 15:47:49,278|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-16|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-16,res:1000,time:3193
2024-12-16 15:47:49,278|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-4,res:0,time:3390
2024-12-16 15:47:49,286|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-11|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-11,res:1000,time:3312
2024-12-16 15:47:49,484|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-9|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-9,res:0,time:3112
2024-12-16 15:47:49,488|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-16|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-16,res:0,time:210
2024-12-16 15:47:49,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-10|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-10,res:1000,time:3205
2024-12-16 15:47:49,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-18|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-18,res:1000,time:3204

观察日志可见，刚开始执行，是个位数毫秒，耗时在逐步增加 ，询问数据平台得知，仅有6个线程在处理数据，所以造成资源争抢，产生请求堆积。

优化：修改线程池数量为6个（最终方案）

执行时间段：17:04 ~ 19:00

刚开始执行日志

2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85452001->85453001,res:0,time:5
2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85454001->85455001,res:0,time:3
2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85455001->85456001,res:0,time:3
2024-12-16 17:04:08,527|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85457001->85458001,res:0,time:3
2024-12-16 17:04:08,527|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85456001->85457001,res:0,time:3
2024-12-16 17:04:08,579|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85633001->85634001,res:0,time:75
2024-12-16 17:04:08,986|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,85458001->85459001,res:0,time:461
2024-12-16 17:04:09,011|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85461001->85462001,res:0,time:485
2024-12-16 17:04:09,079|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85459001->85460001,res:0,time:553
2024-12-16 17:04:09,083|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85466001->85467001,res:0,time:4
2024-12-16 17:04:09,084|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85460001->85461001,res:0,time:558
2024-12-16 17:04:09,089|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85462001->85463001,res:0,time:562
2024-12-16 17:04:09,102|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85463001->85464001,res:0,time:575
2024-12-16 17:04:09,582|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85665001->85666001,res:1000,time:1003
2024-12-16 17:04:09,893|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,85464001->85465001,res:1000,time:907
2024-12-16 17:04:09,972|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85465001->85466001,res:1000,time:961
2024-12-16 17:04:10,002|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85469001->85470001,res:1000,time:913
2024-12-16 17:04:10,006|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85467001->85468001,res:1000,time:923
2024-12-16 17:04:10,009|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85470001->85471001,res:1000,time:907
2024-12-16 17:04:10,011|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85468001->85469001,res:1000,time:927
2024-12-16 17:04:10,518|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85673001->85674001,res:1000,time:936

执行一个多小时后日志

2024-12-16 18:59:58,076|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134119001->134120001,res:0,time:2
2024-12-16 18:59:58,190|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134115001->134116001,res:0,time:966
2024-12-16 18:59:58,192|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,134114001->134115001,res:0,time:974
2024-12-16 18:59:58,217|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,134116001->134117001,res:0,time:918
2024-12-16 18:59:58,299|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,134117001->134118001,res:0,time:887
2024-12-16 18:59:58,300|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134117001->134118001,res:0,time:684
2024-12-16 18:59:58,540|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,134118001->134119001,res:1000,time:925
2024-12-16 18:59:58,629|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134205001->134206001,res:1000,time:1006
2024-12-16 18:59:58,632|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134207001->134208001,res:0,time:3
2024-12-16 18:59:59,000|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134120001->134121001,res:0,time:924
2024-12-16 18:59:59,183|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,134122001->134123001,res:1000,time:991
2024-12-16 18:59:59,185|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134121001->134122001,res:1000,time:995
2024-12-16 18:59:59,188|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134128001->134129001,res:0,time:3
2024-12-16 18:59:59,217|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,134123001->134124001,res:1000,time:1000
2024-12-16 18:59:59,220|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134124001->134125001,res:0,time:920
2024-12-16 18:59:59,273|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,134124001->134125001,res:1000,time:974
2024-12-16 18:59:59,573|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,134125001->134126001,res:1000,time:1033
2024-12-16 18:59:59,632|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134208001->134209001,res:1000,time:1000
2024-12-16 18:59:59,975|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134126001->134127001,res:1000,time:975

可以看到，请求耗时基本是平稳到 1s左右的，说明数据平台侧完全可以处理当前的并发量，资源是足够的

结论

由于我们对接OB没有监控平台，所以观察不了，本例通过最简单的日志打印观察执行情况

  • 如果处理耗时较一开始慢慢增大，表示并发量太大，处理不过来了，请求开始堆积了，那么就要减小并发量；
  • 如果处理耗时很平稳一直很快，可以尝试增加并发量观察，直到达到最大并发量即可。

本例没有其他操作，只是一直循环执行删除操作，那么业务并发量就与数据侧的处理线程持平上下浮动，对标CPU密集型任务；如果业务还需要查询第三方之类的操作再去删除，那么业务并发量就要先大设置观察，对标IO密集型任务

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/940449.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

Qt：QMetaObject::connectSlotsByName实现信号槽自动关联

Qt：QMetaObject::connectSlotsByName实现信号槽自动关联

简介 在Qt中&#xff0c;QMetaObject::connectSlotsByName 是一个便利的方法&#xff0c;它可以根据对象的对象名&#xff08;objectName&#xff09;自动将信号和槽连接起来。但是&#xff0c;要使用这个方法&#xff0c;必须确保&#xff1a; 1 控件&#xff08;如按钮&…
阅读更多...
记录仪方案_记录仪安卓主板定制_音视频记录仪PCBA定制开发

记录仪方案_记录仪安卓主板定制_音视频记录仪PCBA定制开发

记录仪主板采用了强大的联发科MTK8768处理器&#xff0c;拥有出色的性能表现。它搭载了四个主频为2.0GHz的Cortex-A53核心与四个主频为1.5GHz的Cortex-A53核心&#xff0c;确保了高效的处理速度。此外&#xff0c;主板配备了4GB的RAM(可选8GB)&#xff0c;并且内置64GB的ROM(可…
阅读更多...
Ubuntu 20.04 卸载和安装 MySQL8.0

Ubuntu 20.04 卸载和安装 MySQL8.0

卸载 首先&#xff0c;检查一下系统安装的软件包有哪些&#xff0c;使用dpkg -l | grep mysql命令&#xff1a; 为了将MySQL卸载干净&#xff0c;这些文件都需要被删除。 在Ubuntu20.04系统下&#xff0c;卸载干净MySQL8.0以确保下一次安装不会出错&#xff0c;可以按照以下…
阅读更多...
RCNN系列是如何逐步改善的

RCNN系列是如何逐步改善的

1、R-CNN的缺点&#xff1a; 1&#xff09;计算效率低下&#xff1a;RCNN需要为每一个候选框都提取特征&#xff0c;会导致大量重复的工作&#xff0c;因为候选框是原始图片的一部分&#xff0c;肯定是存在交集的。2&#xff09;需要大量的磁盘空间&#xff1a;在训练阶段&…
阅读更多...
数据结构day5：单向循环链表 代码作业

数据结构day5：单向循环链表 代码作业

一、loopLink.h #ifndef __LOOPLINK_H__ #define __LOOPLINK_H__#include <stdio.h> #include <stdlib.h>typedef int DataType;typedef struct node {union{int len;DataType data;};struct node* next; }loopLink, *loopLinkPtr;//创建 loopLinkPtr create();//…
阅读更多...
后摩尔定律时代，什么将推动计算机性能优化的发展？

后摩尔定律时代，什么将推动计算机性能优化的发展？

在摩尔定律时代&#xff0c;每两年芯片上的晶体管数量就会翻一番&#xff0c;这一看似不可避免的趋势被称为摩尔定律&#xff0c;它极大地促进了计算机性能的提高。然而&#xff0c;硅基晶体管不可能一直小下去&#xff0c;半导体晶体管的微型化推动了计算机性能的提升&#xf…
阅读更多...
LEAST-TO-MOST PROMPTING ENABLES COMPLEX REASONING IN LARGE LANGUAGE MODELS---正文

LEAST-TO-MOST PROMPTING ENABLES COMPLEX REASONING IN LARGE LANGUAGE MODELS---正文

题目 最少到最多的提示使大型语言模型能够进行复杂的推理 论文地址&#xff1a;https://arxiv.org/abs/2205.10625 摘要 思路链提示在各种自然语言推理任务中表现出色。然而&#xff0c;它在需要解决比提示中显示的示例更难的问题的任务上表现不佳。为了克服这种由易到难的概括…
阅读更多...
0101多级nginx代理websocket配置-nginx-web服务器

0101多级nginx代理websocket配置-nginx-web服务器

1. 前言 项目一些信息需要通过站内信主动推动给用户&#xff0c;使用websocket。web服务器选用nginx&#xff0c;但是域名是以前通过阿里云申请的&#xff0c;解析ip也是阿里云的服务器&#xff0c;甲方不希望更换域名。新的系统需要部署在内网服务器&#xff0c;简单拓扑图如…
阅读更多...
Mysql8版本的下载安装配置，无痛使用！！！！

Mysql8版本的下载安装配置，无痛使用！！！！

mysql8.x版本和msyql5.x版本zip安装的方式大同小异&#xff0c;但是在mysql8.0版本不用手动创建data数据目录&#xff0c;初始化的时候会自动安装的。而且mysql8.0版本性能官方表示比mysql 5.7的快两倍&#xff01; 可以查看文章看5.7版本的安装 MySql5.7安装、配置最新版_my…
阅读更多...
《商业模式2.0图鉴》读书笔记（如何构建创新驱动的商业模式：打破定律与重塑价值）

《商业模式2.0图鉴》读书笔记（如何构建创新驱动的商业模式：打破定律与重塑价值）

文章目录 引言一、构建创新商业模式的核心原则二、创新商业模式的构建维度三、商业模式设计的实践工具与方法四、从现状到未来&#xff1a;商业模式的演进路径结论附录标题图 引言 商业模式是企业连接资源与客户的桥梁&#xff0c;是价值创造与捕获的核心框架。随着市场需求和…
阅读更多...
【数据结构】数据结构整体大纲

【数据结构】数据结构整体大纲

数据结构用来干什么的&#xff1f;很简单&#xff0c;存数据用的。 &#xff08;这篇文章仅介绍数据结构的大纲&#xff0c;详细讲解放在后面的每一个章节中&#xff0c;逐个击破&#xff09; 那为什么不直接使用数组、集合来存储呢 ——> 如果有成千上亿条数据呢&#xff…
阅读更多...
Flutter组件————FloatingActionButton

Flutter组件————FloatingActionButton

FloatingActionButton 是Flutter中的一个组件&#xff0c;通常用于显示一个圆形的按钮&#xff0c;它悬浮在内容之上&#xff0c;旨在吸引用户的注意力&#xff0c;并代表屏幕上的主要动作。这种按钮是Material Design的一部分&#xff0c;通常放置在页面的右下角&#xff0c;但…
阅读更多...
python rabbitmq实现简单/持久/广播/组播/topic/rpc消息异步发送可配置Django

python rabbitmq实现简单/持久/广播/组播/topic/rpc消息异步发送可配置Django

windows首先安装rabbitmq 点击参考安装 1、环境介绍 Python 3.10.16 其他通过pip安装的版本(Django、pika、celery这几个必须要有最好版本一致) amqp 5.3.1 asgiref 3.8.1 async-timeout 5.0.1 billiard 4.2.1 celery 5.4.0 …
阅读更多...
【Verilog】期末复习

【Verilog】期末复习

数字逻辑电路分为哪两类&#xff1f;它们各自的特点是什么&#xff1f; 组合逻辑电路&#xff1a;任意时刻的输出仅仅取决于该时刻的输入&#xff0c;而与电路原来的状态无关 没有记忆功能&#xff0c;只有从输入到输出的通路&#xff0c;没有从输出到输入的回路 时序逻辑电路&…
阅读更多...
光伏电站无人机巡检都有哪些功能？

光伏电站无人机巡检都有哪些功能？

焱图慧云光伏智能巡检系统主要依托于先进的无人机技术、传感器技术、图像处理技术和智能分析技术。 一、无人机自主飞行与航迹控制 全自主飞行&#xff1a;无人机能够按照预设的飞行路线自主飞行&#xff0c;完成指定的巡检任务&#xff0c;无需人工干预&#xff0c;大大提高了…
阅读更多...
图书馆管理系统（三）基于jquery、ajax

图书馆管理系统（三）基于jquery、ajax

任务3.4 借书还书页面 任务描述 这部分主要是制作借书还书的界面&#xff0c;这里我分别制作了两个网页分别用来借书和还书。此页面&#xff0c;也是通过获取books.txt内容然后添加到表格中&#xff0c;但是借还的操作没有添加到后端中去&#xff0c;只是一个简单的前端操作。…
阅读更多...
如何使用 WebAssembly 扩展后端应用

如何使用 WebAssembly 扩展后端应用

1. WebAssembly 简介 随着互联网的发展&#xff0c;越来越多的应用借助 Javascript 转到了 Web 端&#xff0c;但人们也发现&#xff0c;随着移动互联网的兴起&#xff0c;需要把大量的应用迁移到手机端&#xff0c;随着手端的应用逻辑越来越复杂&#xff0c;Javascript 的解析…
阅读更多...
《鸿蒙HarmonyOS应用开发从入门到精通（第2版）》简介

《鸿蒙HarmonyOS应用开发从入门到精通（第2版）》简介

《鸿蒙HarmonyOS应用开发从入门到精通&#xff08;第2版&#xff09;》已于近日上市&#xff0c;该书由北京大学出版社出版。距离第1版上市已经过去二年半多。本文希望与读者朋友们分享下这本书里面的大致内容。 封面部分 首先是介绍封面部分。 《鸿蒙HarmonyOS应用开发从入门…
阅读更多...
Linux -- 线程控制相关的函数

Linux -- 线程控制相关的函数

目录 pthread_create -- 创建线程 参数 返回值 代码 -- 不传 args&#xff1a; 编译时带 -lpthread 运行结果 为什么输出混杂&#xff1f; 如何证明两个线程属于同一个进程&#xff1f; 如何证明是两个执行流&#xff1f; 什么是LWP&#xff1f; 代码 -- 传 args&a…
阅读更多...
VTK知识学习（26）- 图像基本操作（一）

VTK知识学习（26）- 图像基本操作（一）

1、前言 图像处理离不开一些基本的图像数据操作&#xff0c;例如获取和修改图像的基本信息、访问和修改图像像素值、图像显示、图像类型转换等。熟练掌握这些基本操作有助于使用 VTK进行图像处理应用程序的快速开发。 2、图像信息的访问与修改 1&#xff09;利用vtkIamgeData…
阅读更多...
最新文章

Copyright @ 2022~2023