前言

       今天要跟大家分享的是一个导出数据进度条的简单实现，适用场景用在数据量大、组织数据耗时的情况下的简单实现。

---

一、设计思路

1、导出数据生成文件上传到OSS，
2、导出数据状态存redis缓存，
3、前端发导出请求后，返回的文件key
4、请求后端，后端查询缓存情况返回
5、前端解析是否完成标值，如果完成结束轮询，执行下载get下载，如果未完成，等待下一次轮询

---

二、设计时序图

三、核心代码

1.导出请求

下载请求

/**
     * 因子达标分析汇总表导出
     *
     * @param airEnvQualityQueryVo 因子达标分析汇总表导出
     * @return 统一出参
     */
    @PostMapping("/propSummaryData/export")
    @ApiOperation("因子达标分析汇总表导出")
    public RestMessage propSummaryData4Export(@RequestBody AirEnvQualityQueryVo airEnvQualityQueryVo) {
        Assert.notNull(airEnvQualityQueryVo, "查询参数不能为空");
        Assert.notNull(airEnvQualityQueryVo.getStartTime(), "开始时间不能为空");
        Assert.notNull(airEnvQualityQueryVo.getEndTime(),"结束时间不能为空");
        Assert.isTrue(StringUtils.isNotBlank(airEnvQualityQueryVo.getQueryType()),"查询类型不能为空");
        SimpleDateFormat formatter = new SimpleDateFormat("yyyyMMddHHmmss");
        String key = "propSummaryData:"+formatter.format(new Date());
        AsyncUtil.submitTask(key,() ->{
            //获取并组织excel数据
            String url;
            try {
                url = airEnvironmentExportService.propSummaryData4Export(airEnvQualityQueryVo,key);
            } catch (Exception e) {
                throw new BusinessException(e.getMessage());
            }
            return url;
        });
        return RestBuilders.successBuilder().data(key).build();
    }

serviceImpl

/**
     * 因子达标分析汇总表导出
     *
     * @param airEnvQualityQueryVo 因子达标分析汇总表导出
     * @return 统一出参
     */
    @Override
    public String propSummaryData4Export(AirEnvQualityQueryVo airEnvQualityQueryVo, String key) throws IOException {
        //获取汇聚数据
        AirEnvQualityResultOverviewVo resultOverviewVo = airEnvironmentQualityStatisticsService.getAirEnvQualityResultOverviewVo(airEnvQualityQueryVo);
        //数据转换
        resultOverviewVo.setTqRateCompStr(rateHandlerStr(resultOverviewVo.getTqRateComp()));
        resultOverviewVo.setSqRateCompStr(rateHandlerStr(resultOverviewVo.getSqRateComp()));
        //获取或者数据
        List<AirEnvQualityPropSummaryVo> airEnvQualityPropSummaryVos = airEnvironmentQualityStatisticsService.propSummaryData(airEnvQualityQueryVo);
        AtomicInteger done = new AtomicInteger();
        AsyncUtil.setTotal(key,airEnvQualityPropSummaryVos.size());
        airEnvQualityPropSummaryVos.forEach(vo ->{
            //数据转换
            vo.setBqReachRateStr(rateHandler(vo.getBqReachRate()));
            vo.setTqReachRateCompStr(rateHandlerStr(vo.getTqReachRateComp()));
            vo.setSqReachRateCompStr(rateHandlerStr(vo.getSqReachRateComp()));
            vo.setBqExceedRateStr(rateHandler(vo.getBqExceedRate()));
            vo.setTqExceedRateCompStr(rateHandlerStr(vo.getTqExceedRateComp()));
            vo.setSqExceedRateCompStr(rateHandlerStr(vo.getSqExceedRateComp()));
            done.getAndIncrement();
            AsyncUtil.setDone(key,done.get());
        });
        //组织导出数据
        Map<String,Object> map = new HashMap<>();
        map.put("p",resultOverviewVo);
        map.put("w",airEnvQualityPropSummaryVos);
        String url = getExcelUrl(map, "propSum.xlsx", "因子分析汇总");
        return url;
    }

2.核心工具类

AsyncUtil负责异步更新生成文件数据组织情况更新，存储到缓存

import cn.hutool.core.collection.CollectionUtil;
import com.easylinkin.oss.OSSBaseService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Component;

import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;

@Component
public class AsyncUtil implements ApplicationContextAware {

  static Logger LOG = LoggerFactory.getLogger(AsyncUtil.class);
  public static ExecutorService executor = Executors.newFixedThreadPool(40);
  public static ScheduledExecutorService ex = Executors.newScheduledThreadPool(1);
  static List<String> keys = new ArrayList<>();
  static boolean scheduleIsStart = false;

  private static OSSBaseService ossService;

  @Override
  public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    ossService = applicationContext.getBean(OSSBaseService.class);
  }

  public static RedisTemplate<String, RedisAsyResultData> getRedisTemplate() {
    return SpringUtils.getBean("redisTemplate", RedisTemplate.class);
  }

  static void updateKeyLiveTime() {
    if (!scheduleIsStart) {
      // 更新redis中缓存的过期时间
      ex.scheduleAtFixedRate(() -> {
        try {
          LOG.info("----- update AsyncResult keys length:{} -----",
              keys.size());
          if (CollectionUtil.isNotEmpty(keys)) {
            List<RedisAsyResultData> multiGet =
                getRedisTemplate().opsForValue().multiGet(keys);
            for (RedisAsyResultData result : multiGet) {
              if (result != null) {
                String key = result.getRedisKey();
                getRedisTemplate()
                    .expire(key, 5, TimeUnit.MINUTES);
              }
            }
          }
        } catch (Exception e) {
          scheduleIsStart = false;
          LOG.error(e.getMessage(), e);
        }
      }, 1, 3, TimeUnit.MINUTES);
      scheduleIsStart = true;
    }
  }

  public static RedisAsyResultData submitExportTask(String key, Supplier supplier) {
    RedisAsyResultData rs = new RedisAsyResultData();
    rs.setSuccess(false);
    rs.setRedisKey(key);
    rs.setDone(0);
    rs.setTotal(100);
    setToRedis(rs, key);
    if (!keys.contains(key)) {
      keys.add(key);
    }
    String finalKey = key;
    executor.submit(() -> {
      String msg = null;
      try {
        Object o = supplier.get();
        rs.setData(o);
        rs.setFlag(true);
      } catch (Exception e) {
        rs.setFlag(false);
        msg = e.getMessage();
        LOG.error(e.getMessage(), e);
      }
      rs.setSuccess(true);
      rs.setDone(rs.getTotal());
      if (null != msg) {
        rs.setError(msg);
      }
      keys.remove(finalKey);
      setToRedis(rs, finalKey);
    });
    updateKeyLiveTime();
    return rs;
  }

  /**
   * 设置进度
   * @param key
   * @param done
   * @return
   */
  public static void setDone(String key,Integer done){
    RedisAsyResultData result = getResult(key);
    Optional.ofNullable(result).ifPresent(re -> {
      re.setDone(done);
      saveResult(key,result);
    });
  }

  /**
   * 设置总数
   * @param key
   * @param total
   * @return
   */
  public static void setTotal(String key,Integer total){
    RedisAsyResultData result = getResult(key);
    Optional.ofNullable(result).ifPresent(re -> {
      re.setTotal(total);
      saveResult(key,result);
    });
  }

  public static RedisAsyResultData submitTask(String key, Supplier supplier) {
    AtomicReference<RedisAsyResultData> rs = new AtomicReference<>(new RedisAsyResultData());
    rs.get().setSuccess(false);
    rs.get().setRedisKey(key);
    rs.get().setDone(0);
    rs.get().setTotal(100);
    setToRedis(rs.get(), key);
    if (!keys.contains(key)) {
      keys.add(key);
    }
    String finalKey = key;
    executor.submit(() -> {
      String msg = null;
      try {
        Object o = supplier.get();
        RedisAsyResultData result = getResult(key);
        if (null != result){
          rs.set(result);
        }
        rs.get().setData(o);
        rs.get().setFlag(true);
      } catch (Exception e) {
        rs.get().setFlag(false);
        msg = e.getMessage();
        LOG.error(e.getMessage(), e);
      }
      rs.get().setSuccess(true);

      rs.get().setDone(rs.get().getTotal());
      if (null != msg) {
        rs.get().setError(msg);
      }
      keys.remove(finalKey);
      setToRedis(rs.get(), finalKey);
    });
    updateKeyLiveTime();
    return rs.get();
  }

  private static void setToRedis(RedisAsyResultData result, String redisKey) {
    getRedisTemplate().opsForValue().set(redisKey, result, 5, TimeUnit.MINUTES);
  }

  public static RedisAsyResultData getResult(String key) {
    RedisAsyResultData excelResult =
        getRedisTemplate().opsForValue().get(key);
    if (null != excelResult) {
      return excelResult;
    }
    return null;
  }

  public static void saveResult(String key, RedisAsyResultData result) {
    setToRedis(result, key);
  }

  public static byte[] FileToByte(String filePath) throws Exception{
    FileInputStream fis = null;
    BufferedInputStream bis = null;
    try {
      fis = new FileInputStream(filePath);
      bis = new BufferedInputStream(fis);
      ByteArrayOutputStream baos = new ByteArrayOutputStream();
      int c = bis.read();
      while (c != -1) {
        // 数据存储到ByteArrayOutputStream中
        baos.write(c);
        c = bis.read();
      }
      fis.close();
      bis.close();
      // 转换成二进制
      byte[] bytes = baos.toByteArray();
      return bytes;
    }catch (Exception e){
      e.printStackTrace();
      throw e;
    }finally {
      try {
        if (fis != null ) {
          fis.close();
        }
      } catch (IOException e) {
        e.printStackTrace();
        throw e;
      } finally {
        try {
          if (bis != null ) {
            bis.close();
          }
        } catch (IOException e) {
          e.printStackTrace();
          throw e;
        }
      }
    }
  }
}

3.查询导出文件生成情况接口

/**
   * 根据key获取导出接口
   * @param key
   * @return
   */
  @GetMapping("getRedisResult/{key}")
  public RestMessage getRedisResult(@PathVariable String key){
    Assert.hasLength(key,"key不能为空");
    return RestBuilders.successBuilder().data(AsyncUtil.getResult(key)).build();
  }

key为导出请求返回的

四、效果

前端进度条由每一次轮询请求返回的total、done计算

       最后一次轮询，判断flag的值true，或者自行判断total与done相等，又或者判断data是否又返回url表示是否生成完成，然后用返回的url进行get请求执行下载。

总结

• 简单的实现进度条，用在数据需要长时间一条条生成时，看进度条特别明显
• 轮询其实也可以用websocket替代（这样可以离开页面做其他操作，当然这样也是可以的，就是轮询要做到全局请求了，业务模块多的下载的时候前后端都压力变大）
• 这里其实还用到了easypoi的模板导出，大家可以自己看看api
        就写到这里，希望能帮到大家，uping！