OB删除1.5亿数据耗费2小时

目录

回顾：mysql是怎么删除数据的？

删除方案

代码实现

执行结果

结论

本篇是实际操作 批量处理数据以及线程池线程数设置 记录学习

背景：有一张用户标签表，存储数据量达4个亿，使用OceanBase存储，由于其中一种标签类型tagType1刷数时逻辑错误，所以需要将该类型数据全部删掉

回顾：mysql是怎么删除数据的？

首先我们得知道mysql是怎么删除数据的？

MySQL 删除操作使用延迟清理的方式。当使用 DELETE 语句删除数据时，MySQL 会将删除的数据标记为已删除，并在需要时将其从磁盘上清理掉。

如果我们要删除id=2这行数据，mysql执行流程如下

所以删除一行数据，需要先查询数据，然后再将数据标记已删除更新

删除方案

批量删除数据，按照以往经验，按照自增ID分页并发删除最快：1. 查询不需要回表；2. 更新可以避免数据库行锁

由于写数据时，是多个标签任务同时写入的，所以需要删掉的tagType1数据并不是连续的，意味着删除语句必须筛选tag_type = "tagType1"

可以查询出tagType1的最小ID和最大ID，然后进行分页查询

代码实现

public static final int endId = 237655461;
public static final int batchSize = 1000;
public static final int startId = 85427001;
public static final int threadNum = 6;@Override
public void tmpDeleteClueTag() {log.info("deleteUserTag start");for (long i = startId; i < endId; i += batchSize * threadNum) {for (int j = 0; j < threadNum; j++) {final long from = i + j * batchSize;final long to = from + batchSize;executorService.submit(() -> {long l = System.currentTimeMillis();int result = clueUserTagGateway.deleteUserTag(from, to);log.info("deleteUserTag,{},{}->{},res:{},time:{}", Thread.currentThread().getName(), from, to,result, System.currentTimeMillis() - l);});}}log.info("deleteUserTag success");
}// 执行语句
DELETE FROM clue_user_tag where tag_type = 'spu' and id >= #{from} and id &lt; #{to}// 线程池定义
public class CustomizeThreadPoolConfig {public static final int CORE_SIZE = 6;public static final int MAX_SIZE = 6;public static final int KEEP_ALIVE_TIME = 5;public static final int QUEUE_SIZE = 200;@Bean(name = "customerTagExecutor", destroyMethod = "shutdown")public ExecutorService CustomerTagExecutor() {return new TraceableThreadPoolExecutor(CORE_SIZE, MAX_SIZE, KEEP_ALIVE_TIME, TimeUnit.MINUTES,new LinkedBlockingQueue<>(QUEUE_SIZE), new CustomizableThreadFactory("pushExecutor-"),new ThreadPoolExecutor.CallerRunsPolicy());}public static class TraceableThreadPoolExecutor extends ThreadPoolExecutor {public TraceableThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit,BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory, RejectedExecutionHandler handler) {super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory, handler);}}
}
  • 一次处理1000条，将id拆分开扔给线程池执行删除，可以避免数据库表行锁
  • 使用tag_type标签where删除无伤大雅，执行计划会根据id去优先查询的，查询出来后再从1000条中筛选出有tag_type匹配的情况，内存匹配是非常快的
  • 开启6个线程是因为我们数据平台只有6个线程去执行数据，如果线程数太多会造成数据平台侧资源争抢，一次删除请求会变慢

执行结果

刚开始，并不知道数据平台6个线程的限制，所以设置了20个线程

2024-12-16 15:47:46,478|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:4
2024-12-16 15:47:46,481|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:5
2024-12-16 15:47:46,481|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:3
2024-12-16 15:47:46,482|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:4
2024-12-16 15:47:46,483|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:2
2024-12-16 15:47:46,573|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:3098
2024-12-16 15:47:46,575|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:94
2024-12-16 15:47:46,577|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:95
2024-12-16 15:47:46,579|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:3295
2024-12-16 15:47:46,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:7
2024-12-16 15:47:46,581|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:98
2024-12-16 15:47:46,583|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:6
2024-12-16 15:47:46,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:5
2024-12-16 15:47:46,584|INFO |796ffea717b93262a1e1fb8799631878||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=,res:1000,time:3404
2024-12-16 15:47:46,589|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:1000,time:3008
2024-12-16 15:47:46,672|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:92
2024-12-16 15:47:46,673|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:98
2024-12-16 15:47:46,675|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:94
2024-12-16 15:47:46,675|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:3390
2024-12-16 15:47:46,676|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:93
2024-12-16 15:47:46,677|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:88
2024-12-16 15:47:46,677|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-14|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-14,res:0,time:4
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-15|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-15,res:0,time:95
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:3306
2024-12-16 15:47:46,679|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:3372
2024-12-16 15:47:46,680|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:8
2024-12-16 15:47:46,680|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:5
2024-12-16 15:47:46,681|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:5
2024-12-16 15:47:46,682|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:7
2024-12-16 15:47:46,682|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:5
2024-12-16 15:47:46,774|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:95
2024-12-16 15:47:46,776|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:97
2024-12-16 15:47:46,778|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:98
2024-12-16 15:47:46,780|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:100
2024-12-16 15:47:46,781|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:100
2024-12-16 15:47:46,782|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:100
2024-12-16 15:47:46,784|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:102
2024-12-16 15:47:46,785|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:11
2024-12-16 15:47:47,079|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:303
2024-12-16 15:47:47,082|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:302
2024-12-16 15:47:47,082|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:304
2024-12-16 15:47:47,083|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:301
2024-12-16 15:47:47,084|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:303
2024-12-16 15:47:47,084|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:1000,time:3307
2024-12-16 15:47:47,085|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:300
2024-12-16 15:47:47,085|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:301
2024-12-16 15:47:47,184|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:105
2024-12-16 15:47:47,185|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:1000,time:3409
2024-12-16 15:47:47,185|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:103
2024-12-16 15:47:47,187|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:103
2024-12-16 15:47:47,188|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:105
2024-12-16 15:47:47,189|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:107
2024-12-16 15:47:47,273|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:188
2024-12-16 15:47:47,275|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:191
2024-12-16 15:47:47,277|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:192
2024-12-16 15:47:47,280|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:96
2024-12-16 15:47:47,281|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:96
2024-12-16 15:47:47,283|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:0,time:98
2024-12-16 15:47:47,285|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:98
2024-12-16 15:47:47,286|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:98
2024-12-16 15:47:47,288|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:99
2024-12-16 15:47:47,289|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:15
2024-12-16 15:47:47,374|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:99
2024-12-16 15:47:47,375|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:98
2024-12-16 15:47:47,485|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-2,res:0,time:202
2024-12-16 15:47:47,573|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-12|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-12,res:0,time:292
2024-12-16 15:47:47,575|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-5,res:0,time:295
2024-12-16 15:47:47,577|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-6,res:0,time:292
2024-12-16 15:47:47,578|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-17|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-17,res:0,time:292
2024-12-16 15:47:47,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-19|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-19,res:0,time:291
2024-12-16 15:47:47,581|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-3,res:0,time:292
2024-12-16 15:47:47,583|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-1,res:0,time:209
2024-12-16 15:47:47,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:209
2024-12-16 15:47:47,883|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-20|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-20,res:1000,time:3405
2024-12-16 15:47:47,987|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-8|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-8,res:0,time:403
2024-12-16 15:47:48,881|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-13|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-13,res:1000,time:3194
2024-12-16 15:47:49,184|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-7|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-7,res:1000,time:3211
2024-12-16 15:47:49,278|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-16|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-16,res:1000,time:3193
2024-12-16 15:47:49,278|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-4,res:0,time:3390
2024-12-16 15:47:49,286|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-11|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-11,res:1000,time:3312
2024-12-16 15:47:49,484|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-9|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-9,res:0,time:3112
2024-12-16 15:47:49,488|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-16|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-16,res:0,time:210
2024-12-16 15:47:49,580|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-10|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-10,res:1000,time:3205
2024-12-16 15:47:49,584|INFO |796ffea717b93262a1e1fb8799631878|pushExecutor-18|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,threadName=pushExecutor-18,res:1000,time:3204

观察日志可见，刚开始执行，是个位数毫秒，耗时在逐步增加 ，询问数据平台得知，仅有6个线程在处理数据，所以造成资源争抢，产生请求堆积。

优化：修改线程池数量为6个（最终方案）

执行时间段：17:04 ~ 19:00

刚开始执行日志

2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85452001->85453001,res:0,time:5
2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85454001->85455001,res:0,time:3
2024-12-16 17:04:08,526|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85455001->85456001,res:0,time:3
2024-12-16 17:04:08,527|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85457001->85458001,res:0,time:3
2024-12-16 17:04:08,527|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85456001->85457001,res:0,time:3
2024-12-16 17:04:08,579|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85633001->85634001,res:0,time:75
2024-12-16 17:04:08,986|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,85458001->85459001,res:0,time:461
2024-12-16 17:04:09,011|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85461001->85462001,res:0,time:485
2024-12-16 17:04:09,079|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85459001->85460001,res:0,time:553
2024-12-16 17:04:09,083|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85466001->85467001,res:0,time:4
2024-12-16 17:04:09,084|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85460001->85461001,res:0,time:558
2024-12-16 17:04:09,089|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85462001->85463001,res:0,time:562
2024-12-16 17:04:09,102|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85463001->85464001,res:0,time:575
2024-12-16 17:04:09,582|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85665001->85666001,res:1000,time:1003
2024-12-16 17:04:09,893|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,85464001->85465001,res:1000,time:907
2024-12-16 17:04:09,972|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,85465001->85466001,res:1000,time:961
2024-12-16 17:04:10,002|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,85469001->85470001,res:1000,time:913
2024-12-16 17:04:10,006|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,85467001->85468001,res:1000,time:923
2024-12-16 17:04:10,009|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,85470001->85471001,res:1000,time:907
2024-12-16 17:04:10,011|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,85468001->85469001,res:1000,time:927
2024-12-16 17:04:10,518|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,85673001->85674001,res:1000,time:936

执行一个多小时后日志

2024-12-16 18:59:58,076|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134119001->134120001,res:0,time:2
2024-12-16 18:59:58,190|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134115001->134116001,res:0,time:966
2024-12-16 18:59:58,192|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,134114001->134115001,res:0,time:974
2024-12-16 18:59:58,217|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,134116001->134117001,res:0,time:918
2024-12-16 18:59:58,299|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,134117001->134118001,res:0,time:887
2024-12-16 18:59:58,300|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134117001->134118001,res:0,time:684
2024-12-16 18:59:58,540|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,134118001->134119001,res:1000,time:925
2024-12-16 18:59:58,629|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134205001->134206001,res:1000,time:1006
2024-12-16 18:59:58,632|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134207001->134208001,res:0,time:3
2024-12-16 18:59:59,000|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134120001->134121001,res:0,time:924
2024-12-16 18:59:59,183|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-2|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-2,134122001->134123001,res:1000,time:991
2024-12-16 18:59:59,185|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134121001->134122001,res:1000,time:995
2024-12-16 18:59:59,188|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-3|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-3,134128001->134129001,res:0,time:3
2024-12-16 18:59:59,217|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-5|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-5,134123001->134124001,res:1000,time:1000
2024-12-16 18:59:59,220|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134124001->134125001,res:0,time:920
2024-12-16 18:59:59,273|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-1|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-1,134124001->134125001,res:1000,time:974
2024-12-16 18:59:59,573|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-4|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-4,134125001->134126001,res:1000,time:1033
2024-12-16 18:59:59,632|INFO |5e7bda947e8bce91351c806dcbe821da||c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,,134208001->134209001,res:1000,time:1000
2024-12-16 18:59:59,975|INFO |5e7bda947e8bce91351c806dcbe821da|pushExecutor-6|c.x.n.x.c.s.impl.CustomerClueServiceImpl|deleteUserTag,pushExecutor-6,134126001->134127001,res:1000,time:975

可以看到，请求耗时基本是平稳到 1s左右的，说明数据平台侧完全可以处理当前的并发量，资源是足够的

结论

由于我们对接OB没有监控平台，所以观察不了，本例通过最简单的日志打印观察执行情况

  • 如果处理耗时较一开始慢慢增大，表示并发量太大，处理不过来了，请求开始堆积了，那么就要减小并发量；
  • 如果处理耗时很平稳一直很快，可以尝试增加并发量观察，直到达到最大并发量即可。

本例没有其他操作，只是一直循环执行删除操作，那么业务并发量就与数据侧的处理线程持平上下浮动，对标CPU密集型任务；如果业务还需要查询第三方之类的操作再去删除，那么业务并发量就要先大设置观察，对标IO密集型任务

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.rhkb.cn/news/493513.html

如若内容造成侵权/违法违规/事实不符，请联系长河编程网进行投诉反馈email:809451989@qq.com，一经查实，立即删除！

相关文章

20241217使用M6000显卡在WIN10下跑whisper来识别中英文字幕

20241217使用M6000显卡在WIN10下跑whisper来识别中英文字幕

20241217使用M6000显卡在WIN10下跑whisper来识别中英文字幕 2024/12/17 17:21 缘起&#xff0c;最近需要识别法国电影《地下铁》的法语字幕&#xff0c;使用 字幕小工具V1.2【whisper套壳/GUI封装了】 无效。 那就是直接使用最原始的whisper来干了。 当你重装WIN10的时候&#…
阅读更多...
linux普通用户使用sudo不需要输密码

linux普通用户使用sudo不需要输密码

1.root用户如果没有密码&#xff0c;先给root用户设置密码 sudo passwd root #设置密码 2.修改visudo配置 su #切换到root用户下 sudo visudo #修改visudo配置文件 用户名 ALL(ALL) NOPASSWD: ALL #下图所示处新增一行配置 用户名需要输入自己当前主机的用户名
阅读更多...
【C++11】可变模板参数

【C++11】可变模板参数

目录 可变模板的定义方式 参数包的展开方式 递归的方式展开参数包 STL中的emplace相关接口函数 STL容器中emplace相关插入接口函数 ​编辑 模拟实现&#xff1a;emplace接口 C11的新特性可变参数模板能够让您创建可以接受可变参数的函数模板和类模板&#xff0c;相比 C9…
阅读更多...
python 曲线拟合，曲线拟合交点

python 曲线拟合，曲线拟合交点

目录 效果图： 源代码： 效果图： 源代码： import json import os import shutilimport cv2 import numpy as npfrom numpy.polynomial.polynomial import Polynomialdef calculate_distance(x1, y1, x2, y2):return np.sqrt((x2 - x1) ** 2 + (y2 - y1) ** 2)def get_new_g…
阅读更多...
Java模拟Mqtt客户端连接Mqtt Broker

Java模拟Mqtt客户端连接Mqtt Broker

Java模拟Mqtt客户端基本流程 引入Paho MQTT客户端库 <dependency><groupId>org.eclipse.paho</groupId><artifactId>org.eclipse.paho.mqttv5.client</artifactId><version>1.2.5</version> </dependency>设置mqtt配置数据 …
阅读更多...
圣诞快乐（h5 css js（圣诞树））

圣诞快乐（h5 css js（圣诞树））

一&#xff0c;整体设计思路 圣诞树h5&#xff08;简易&#xff09; 1.页面布局与样式&#xff1a; 页面使用了全屏的黑色背景&#xff0c;中央显示圣诞树&#xff0c;树形由三层绿色的三角形组成&#xff0c;每一层的大小逐渐变小。树干是一个棕色的矩形&#xff0c;位于三角…
阅读更多...
多音轨视频使用FFmpeg删除不要音轨方法

多音轨视频使用FFmpeg删除不要音轨方法

近期给孩子找宫崎骏动画&#xff0c;但是有很多是多音轨视频但是默认的都是日语&#xff0c;电视上看没办法所以只能下载后删除音轨文件只保留中文。 方法分两步&#xff0c;先安装FFmpeg在转文件即可。 第一步FFmpeg安装 FFmpeg是一个开源项目&#xff0c;包含了处理视频的…
阅读更多...
时空信息平台架构搭建：基于netty封装TCP通讯模块（IdleStateHandler网络连接监测,处理假死）

时空信息平台架构搭建：基于netty封装TCP通讯模块（IdleStateHandler网络连接监测,处理假死）

文章目录 引言I 异步TCP连接操作II 心跳机制：空闲检测（读空闲和写空闲）基于Netty的IdleStateHandler类实现心跳机制（网络连接监测）常规的处理假死健壮性的处理假死方案引言 基于netty实现TCP客户端：封装断线重连、连接保持 https://blog.csdn.net/z929118967/article/de…
阅读更多...
中国新能源汽车公共充电桩数据合集（2002-2023年）

中国新能源汽车公共充电桩数据合集（2002-2023年）

数据来源&#xff1a;全国各省市统计年鉴、统计公报、国家能源署、中国汽车行业协会&#xff0c;各类汽车统计年鉴、中国电动汽车充电基础设施促进联盟等 时间跨度&#xff1a;新能源汽车数据集&#xff1a;2002-2023年&#xff08;不同数据时间跨度有差异&#xff0c;详见数据…
阅读更多...
设计模式12：状态模式

设计模式12：状态模式

系列总链接&#xff1a;《大话设计模式》学习记录_net 大话设计-CSDN博客 参考&#xff1a;设计模式之状态模式 (C 实现)_设计模式的状态模式实现-CSDN博客 1.概述 状态模式允许一个对象在其内部状态改变时改变其行为。对象看起来像是改变了其类。使用状态模式可以将状态的相…
阅读更多...
国内网络在Ubuntu 22.04中在线安装Ollama并配置Open-WebuiDify

国内网络在Ubuntu 22.04中在线安装Ollama并配置Open-WebuiDify

配置docker科技网络 登录后复制 创建或编辑 Docker 配置文件 让docker使用代理&#xff1a; sudo mkdir /etc/systemd/system/docker.service.d -p sudo vim /etc/systemd/system/docker.service.d/http-proxy.conf 文件&#xff0c;并添加以下内容&#xff1a; [Service] En…
阅读更多...
【线性代数】理解矩阵乘法的意义（点乘）

【线性代数】理解矩阵乘法的意义（点乘）

刚接触线性代数时&#xff0c;很不理解矩阵乘法的计算规则&#xff0c;为什么规则定义的看起来那么有规律却又莫名其妙&#xff0c;现在参考了一些资料&#xff0c;回过头重新总结下个人对矩阵乘法的理解&#xff08;严格来说是点乘&#xff09;。 理解矩阵和矩阵的乘法&#x…
阅读更多...
国标GB28181协议平台Liveweb:搭建建筑工地无线视频联网监控系统方案

国标GB28181协议平台Liveweb:搭建建筑工地无线视频联网监控系统方案

随着科技高速发展&#xff0c;视频信号经过数字压缩&#xff0c;通过互联网宽带或者移动4G网络传递&#xff0c;可实现远程视频监控功能。将这一功能运用于施工现场安全管理&#xff0c;势必会大大提高管理效率&#xff0c;提升监管层次。而这些&#xff0c;通过Liveweb监控系统…
阅读更多...
SQL语句练习

SQL语句练习

阅读《SQL必知必会》&#xff08;第五版&#xff09;然后结合往常表做的练习记录 这里使用的数据库时sqlite3,使用的工具时navicat 表资源链接https://wenku.baidu.com/view/349fb3639b6648d7c1c74652.html 表录入后如上图所示。后面如果有多张表之间的操作&#xff0c;在引入…
阅读更多...
SAP RESTful架构和OData协议

SAP RESTful架构和OData协议

一、RESTful架构 RESTful 架构&#xff08;Representational State Transfer&#xff09;是一种软件架构风格&#xff0c;专门用于构建基于网络的分布式系统&#xff0c;尤其是在 Web 服务中。它通过利用 HTTP 协议和一组简单的操作&#xff08;如 GET、POST、PUT、DELETE&…
阅读更多...
基于MATLAB的图像增强

基于MATLAB的图像增强

&#x1f351;个人主页&#xff1a;Jupiter. &#x1f680; 所属专栏&#xff1a;传知代码 欢迎大家点赞收藏评论&#x1f60a; 目录 一、背景及意义介绍背景图像采集过程中的局限性 意义 二、概述三、代码结构及说明&#xff08;一&#xff09;整体结构&#xff08;二&#xf…
阅读更多...
通过阿里云 Milvus 与 PAI 搭建高效的检索增强对话系统

通过阿里云 Milvus 与 PAI 搭建高效的检索增强对话系统

背景介绍 阿里云向量检索服务Milvus版&#xff08;简称阿里云Milvus&#xff09;是一款云上全托管服务&#xff0c;确保了了与开源Milvus的100%兼容性&#xff0c;并支持无缝迁移。在开源版本的基础上增强了可扩展性&#xff0c;能提供大规模 AI 向量数据的相似性检索服务。相…
阅读更多...
滚珠花键的保养与维护方法

滚珠花键的保养与维护方法

滚珠花键作为关键的线性运动引导装置&#xff0c;以其高精度和高刚性在众多领域发挥着举足轻重的作用。然而&#xff0c;为了保持其卓越的性能&#xff0c;保养与维护措施不可或缺。 滚珠花键的保养与维护其实就是润滑与清洁&#xff0c;以下是一些具体的保养与维护方法&#x…
阅读更多...
Layui table不使用url属性结合laypage组件实现动态分页

Layui table不使用url属性结合laypage组件实现动态分页

从后台一次性获取所有数据赋值给 Layui table 组件的 data 属性&#xff0c;若数据量大时&#xff0c;很可能会超出浏览器字符串最大长度&#xff0c;导致渲染数据失败。Layui table 结合 laypage 组件实现动态分页可解决此问题。 HTML增加分页组件标签 在table后增加一个用于…
阅读更多...
fastdds:idl

fastdds:idl

1使用网络收发数据的最简单方式 在学习idl之前&#xff0c;先来看一下我们在开发中&#xff0c;通过网络收发数据时&#xff0c;常常怎么实现。 struct Student {char name[32];int age;char sex;// f 男&#xff0c;m 女 };//发送侧 struct Student s1 {"xiaoming&q…
阅读更多...
最新文章
推荐文章

Copyright @ 2022~2023