在HarmonyOS下自渲染视频数据

在本文中，我们将介绍如何在HarmonyOS下自渲染视频数据。我们将实现包括创建本地窗口、设置缓冲区选项、请求缓冲区、处理视频帧数据以及刷新缓冲区等步骤。

环境准备

在开始之前，请确保您已经安装了HarmonyOS的开发环境，并且能够编译和运行C++代码。

注意⚠️

以下方案中用到的API，都不是线程安全的！！！我这里是通过其他手段有保护的，请自行上锁～

创建本地窗口

首先，我们需要根据surfaceId创建一个本地窗口。如何拿这个surfaceId，有两种方案:

• 一种是原生ArkTs UI开发从XComponent拿;
• 一种是跨平台框架比如 flutter通过TextureRegistry注册一个Texture拿,

NativeWindowRender 类的构造函数中实现了这一功能：

NativeWindowRender::NativeWindowRender(const RenderConfig &config) : render_config_(config) {
    int ret = OH_NativeWindow_CreateNativeWindowFromSurfaceId(config.surfaceId, &window_);
    if (window_ == nullptr || ret != 0) {
        LOG_ERROR("create native window failed: {}", ret);
        return;
    }
}

在这里，我们使用

OH_NativeWindow_CreateNativeWindowFromSurfaceId

函数从surfaceId创建一个本地窗口。如果创建失败，会记录错误日志。

设置缓冲区选项

创建本地窗口后，我们需要设置一些缓冲区选项，例如缓冲区使用情况、交换间隔、请求超时、颜色范围和变换等可以参考 NativeWindowOperation，我暂时只设置了这些，看看渲染效果和性能

bool NativeWindowRender::UpdateNativeBufferOptionsByVideoFrame(const VideoFrame *frame) {
#if 0
    // get current buffer geometry, if different, reset buffer geometry
    int32_t stride = 0;
    int32_t height = 0;
    // the fucking order of get is h, w, however, the fucking order of set is w, h
    int ret = OH_NativeWindow_NativeWindowHandleOpt(window_, GET_BUFFER_GEOMETRY, &height, &stride);
    if (ret != 0) {
        LOG_ERROR("get buffer geometry failed: {}", ret);
        return false;
    }

    // set buffer geometry if different
    if (stride != frame->yStride || height != frame->height) {
        ret = OH_NativeWindow_NativeWindowHandleOpt(window_, SET_BUFFER_GEOMETRY, frame->yStride, frame->height);
        if (ret != 0) {
            LOG_ERROR("set buffer geometry failed: {}", ret);
            return false;
        }
    }
#else
    int ret = OH_NativeWindow_NativeWindowHandleOpt(window_, SET_BUFFER_GEOMETRY, frame->yStride, frame->height);
    if (ret != 0) {
        LOG_ERROR("set buffer geometry failed: {}", ret);
        return false;
    }
#endif

    // set buffer format
    ret = OH_NativeWindow_NativeWindowHandleOpt(window_, SET_FORMAT, NATIVEBUFFER_PIXEL_FMT_YCBCR_420_P);
    if (ret != 0) {
        LOG_ERROR("set buffer format failed: {}", ret);
        return false;
    }

    // set buffer stride
    ret = OH_NativeWindow_NativeWindowHandleOpt(window_, SET_STRIDE, 4);
    if (ret != 0) {
        LOG_ERROR("set buffer stride failed: {}", ret);
        return false;
    }

    // set native source type to OH_SURFACE_SOURCE_VIDEO
    ret = OH_NativeWindow_NativeWindowHandleOpt(window_, SET_SOURCE_TYPE, OH_SURFACE_SOURCE_VIDEO);
    if (ret != 0) {
        LOG_ERROR("set source type failed: {}", ret);
        return false;
    }

    // set app framework type
    ret = OH_NativeWindow_NativeWindowHandleOpt(window_, SET_APP_FRAMEWORK_TYPE, "unknown");
    if (ret != 0) {
        LOG_ERROR("set app framework type failed: {}", ret);
        return false;
    }

    return true;
}

请求缓冲区并处理视频帧数据

在接收到视频帧数据时，我们需要请求缓冲区并将视频帧数据写入缓冲区, 坑点来了，UpdateNativeBufferOptionsByVideoFrame 你必须每次request前都要调用。。。。我反正没找到文档哪里有写，又或者我理解能力有问题，坑了一两个小时，最后lldb debug，发现如果只在构造函数中设置的话，只有第一次request出来的BufferHandle中的width height stride format等参数是符合预期的，后面就不对了，结果就只能画出来第一张图，后面没有任何报错但是就是不出图，所以切记切记， 每次都要调用 (吗？请帮忙指正)

void NativeWindowRender::OnVideoFrameReceived(const void *videoFrame, const VideoFrameConfig &config, bool resize) {
    const VideoFrame *frame = reinterpret_cast<const VideoFrame *>(videoFrame);

    // must call this every time, coz every time you request a buffer, the properties of BufferHandle may change to
    // default value.
    if (!UpdateNativeBufferOptionsByVideoFrame(frame)) {
        return;
    }

    // request buffer from native window
    OHNativeWindowBuffer *buffer = nullptr;
    int fence_fd = -1;
    int ret = OH_NativeWindow_NativeWindowRequestBuffer(window_, &buffer, &fence_fd);
    if (ret != 0 || buffer == nullptr) {
        LOG_ERROR("request buffer failed: {}", ret);
        return;
    }

    // get buffer handle from native buffer
    BufferHandle *handle = OH_NativeWindow_GetBufferHandleFromNative(buffer);

    // mmap buffer handle to write data
    void *data = mmap(handle->virAddr, handle->size, PROT_READ | PROT_WRITE, MAP_SHARED, handle->fd, 0);
    if (data == MAP_FAILED) {
        LOG_ERROR("mmap buffer failed");
        return;
    }

    // wait for fence fd to be signaled
    uint32_t timeout = 3000;
    if (fence_fd != -1) {
        struct pollfd fds = {.fd = fence_fd, .events = POLLIN};
        do {
            ret = poll(&fds, 1, timeout);
        } while (ret == -1 && (errno == EINTR || errno == EAGAIN));
        close(fence_fd);
    }

    // copy yuv420 data to buffer
    uint8_t *y = (uint8_t *)data;
    uint8_t *u = y + frame->yStride * frame->height;
    uint8_t *v = u + frame->uStride * frame->height / 2;
    memcpy(y, frame->yBuffer, frame->yStride * frame->height);
    memcpy(u, frame->uBuffer, frame->uStride * frame->height / 2);
    memcpy(v, frame->vBuffer, frame->vStride * frame->height / 2);

    // flush buffer
    Region region{.rects = nullptr, .rectNumber = 0};
    int acquire_fence_fd = -1;
    ret = OH_NativeWindow_NativeWindowFlushBuffer(window_, buffer, acquire_fence_fd, region);
    if (ret != 0) {
        LOG_ERROR("flush buffer failed: {}", ret);
    }

    // unmap buffer handle
    ret = munmap(data, handle->size);
    if (ret != 0) {
        LOG_ERROR("munmap buffer failed: {}", ret);
    }
}

在这里，我们首先调用 UpdateNativeBufferOptionsByVideoFrame 函数更新缓冲区选项，然后请求缓冲区并将视频帧数据写入缓冲区。最后，我们刷新缓冲区并解除映射。

结论

通过以上步骤，我们可以在HarmonyOS下实现自渲染视频数据。希望本文对您有所帮助。如果您有任何问题或建议，请随时与我联系。

PS

另外一种方案，是拿到NativeWindow后，用opengl自己画，但是opengl这个鬼你也知道的，一个context要一个线程，我要是十几路，几十路视频渲染，就得几十个线程？这在移动平台谁能忍？共享context的话，我自信我这种new opengler不能保证不出错。。。可以参考官方文档 自定义渲染 (XComponent) 。所以，这种方案交给鸿蒙UI框架说着说系统自己画的，叫懒人方案？