前言:
关于多线程的博客先前的博客如下:
python线程join方法_python 线程join-CSDN博客
【精选】Python GIL锁_python gil 锁-CSDN博客
python函数运行加速_python os.listdir速度慢_两只蜡笔的小新的博客-CSDN博客
只需下面的模版即可:
from multiprocessing import Pool
from functools import partial
# execute_one_file是需要多次运行的函数,使用partial对函数的其他参数进行传递
# 需要注意的是 多线程只支持 一个参数的传递,并且是第一个参数。如果是多个参数需要对函数进行改造。
'''
mess_input = []
num_index = 0
for _gt_list, _gp_list, _root_list in zip(gt_list, gp_list, root_list):
num_index = num_index + 1
mess_dict = {
'gt_':_gt_list,
'gp_':_gp_list,
'root_':_root_list,
'num_index':num_index,
'all_num':len(root_list)}
mess_input.append(mess_dict)
'''
execute_all_file = partial(execute_one_file, **{'output_name': args.output_name, 'im_loc_list': im_loc_list})
start_time = time.time()
all_execute_data = []
with Pool() as p:
for i, result in enumerate(p.imap_unordered(execute_all_file, mess_input), 1):
elapsed_time = time.time() - start_time
avg_time_per_task = elapsed_time / i
remaining_tasks = len(mess_input) - i
estimated_time = remaining_tasks * avg_time_per_task
print(f"Processed {i} tasks. Estimated time remaining: {estimated_time} seconds.")
all_execute_data.append(result)
# res_num = all_num - num_index
bar_mess = f"Processed {i} tasks. Estimated time remaining: {estimated_time} seconds."
with trange(len([1])) as tool_bar:
for _index in tool_bar:
tool_bar.set_description(bar_mess)
execute_one_file函数如下
关于设置 线程数量:
import time
def execute_all_file(file):
# 模拟一个需要一定时间的任务
time.sleep(1)
return file
if __name__ == "__main__":
mess_input = range(100) # 假设有100个任务需要处理
start_time = time.time()
results = [] # 用于收集所有任务的结果
num_processes = 4 # 设置进程数量
with Pool(num_processes) as p:
for i, result in enumerate(p.imap_unordered(execute_all_file, mess_input), 1):
results.append(result) # 收集任务结果
elapsed_time = time.time() - start_time
avg_time_per_task = elapsed_time / i
remaining_tasks = len(mess_input) - i
estimated_time = remaining_tasks * avg_time_per_task
print(f"Processed {i} tasks. Estimated time remaining: {estimated_time} seconds.")
print(f"All results: {results}")
在创建Pool
对象时传入了num_processes
参数,设置了进程的数量。可以根据你的需求和系统的能力来调整这个数量。
实际上multiprocessing
模块使用的是进程,而不是线程。Python的多线程因为全局解释器锁(GIL)的存在,无法真正实现并行,而多进程可以避免这个问题。