虚拟现实环境下的远程教育和智能评估系统(十二)

接下来,把实时注视点位置、语音文本知识点、帧知识点区域进行匹配;

首先,第一步是匹配语音文本知识点和帧知识点区域,我们知道教师所说的每句话对应的知识点,然后寻找当前时间段内,知识点对应的ppt中的区域,即得到学生应该看的知识点区域,后续我的队友进行可视化展示(视频生成和报告生成);

第二步,检测注视点位置是否在该区域;统计成功匹配的比例即可衡量该学生上课专注程度;

# -*- coding: utf-8 -*-
"""
@Time : 2024/6/22 14:45
@Auth : Zhao Yishuo
@File :pre_match.py
@IDE :PyCharm
"""
import cv2
import re
import io
from matplotlib import pyplot as plt

# Parse voice match final data
def parse_voice_match_final(file_path):
    knowledge_points = []
    with open(file_path, 'r', encoding='utf-8') as file:
        for line in file:
            match = re.search(r'range: (\d+)-(\d+); kp_id: (\w+)', line)
            if match:
                start_time = int(match.group(1))
                end_time = int(match.group(2))
                kp_id = match.group(3)
                knowledge_points.append((start_time, end_time, kp_id))
    return knowledge_points


import cv2
import pandas as pd
import re

# 解析 final_match_test.txt
def parse_final_match_test(file_path):
    ocr_data = []
    with open(file_path, 'r', encoding='utf-8') as file:
        timestamp = None
        for line in file:
            if 'Timestamp' in line:
                timestamp = int(line.split(': ')[1])
            elif 'Knowledge_point_id:' in line:
                match = re.search(r'\((\d+), (\d+), (\d+), (\d+)\): Knowledge_point_id: (\w+)', line)
                if match:
                    x1, y1, x2, y2 = map(int, match.groups()[:4])
                    kp_id = match.group(5)
                    ocr_data.append((timestamp, (x1, y1, x2, y2), kp_id))
    return ocr_data

# Match knowledge points with OCR/detection regions based on timestamps
def match_knowledge_points(voice_data, ocr_data):
    matches = []
    for (start_time, end_time, kp_id) in voice_data:
        for (timestamp, region, ocr_kp_id) in ocr_data:
            if kp_id == ocr_kp_id and start_time <= timestamp <= end_time:
                matches.append((start_time, end_time, region, kp_id))
    return matches

# Mark regions on the video
def mark_video(input_video_path, output_video_path, matches):
    cap = cv2.VideoCapture(input_video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    out = cv2.VideoWriter(output_video_path, fourcc, fps, (frame_width, frame_height))

    frame_count = 0

    current_region = None
    current_kp_id = None
    region_end_time = None

    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        timestamp = int(frame_count / fps * 1000)  # Convert frame count to milliseconds

        for match in matches:
            start_time, end_time, region, kp_id = match
            if start_time <= timestamp <= end_time:
                current_region = region
                current_kp_id = kp_id
                region_end_time = end_time
                break
        if current_region:
            cv2.rectangle(frame, (current_region[0], current_region[1]), (current_region[2], current_region[3]), (0, 255, 0), 2)
            cv2.putText(frame, current_kp_id, (current_region[0], current_region[1] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

        out.write(frame)
        frame_count += 1

    cap.release()
    out.release()

if __name__ == "__main__":
    # Paths to the files
    voice_match_final_path = 'voice_match_final.txt'
    final_match_test_path = 'final_match_test(1).txt'
    input_video_path = 'video_data/5.mp4'  # Path to the input video
    output_video_path = 'video_data/5_match.mp4'  # Path to save the output video with annotations

    # Parse the files
    voice_data = parse_voice_match_final(voice_match_final_path)
    ocr_data = parse_final_match_test(final_match_test_path)

    # Match the knowledge points with OCR/detection regions
    matches = match_knowledge_points(voice_data, ocr_data)

    # Print matches for debugging
    for match in matches:
        print(f"Start Time: {match[0]}, End Time: {match[1]}, Region: {match[2]}, KP_ID: {match[3]}")

    # Mark the video with matched regions
    # mark_video(input_video_path, output_video_path, matches)

 

# -*- coding: utf-8 -*-
"""
@Time : 2024/6/16 14:52
@Auth : Zhao Yishuo
@File :match.py
@IDE :PyCharm
"""
import cv2
import pandas as pd
import os
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from sklearn.preprocessing import StandardScaler,MinMaxScaler

plt.rcParams['font.sans-serif'] = ['SimHei']

# 手动读取和处理眼动数据文本文件
eyedata_path = 'eye_output_16.txt' # 文本文件路径
data = []
with open(eyedata_path, 'r') as file:
    for line in file:
        line = line.strip()
        if ':' in line:  # 检查是否存在冒号
            key, value = line.split(':', 1)
            data.append([key.strip(), value.strip()])
        # else:
            # print(f"Skipping malformed line: {line}")  # 记录格式不正确的行

data = pd.DataFrame(data, columns=['Type', 'Value'])

# 清洗数据
timestamps = data['Value'][data['Type'] == 'Timestamp'].astype(float).reset_index(drop=True)
videos = data['Value'][data['Type'] == 'Video'].reset_index(drop=True)
positions = data['Value'][data['Type'] == 'Relative Position'].str.extract(r'\[(.*?)\]')[0]  # 眼动位置
positions = positions.str.split(expand=True).astype(float).reset_index(drop=True)
positions[0] = round(positions[0])
positions[1] = round(-positions[1])

# 提取第1列和第2列
data = positions.iloc[:, [0, 1]]

# 确保数据为数值类型
data = data.apply(pd.to_numeric, errors='coerce')
# print(type(data))
x_values = data[0].tolist()
y_values = data[1].tolist()

eye_pos = np.vstack([x_values, y_values]).T  # df类型成功转换为np数组

eye_timestamps = np.array(timestamps.tolist())

# np.save('eye_positions.npy', eye_pos)
# np.save('eye_timestamps.npy', eye_timestamps)

eye_pos = np.load('eye_positions.npy')
eye_timestamps = np.load('eye_timestamps.npy')

# print(eye_pos,eye_timestamps)
text_path = 'final_match_test.txt'

import re

# Function to parse the text file and extract data
def parse_text_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        content = file.read()

    # Regular expressions to match timestamps, OCR, and detection positions
    timestamp_pattern = re.compile(r'Timestamp: (\d+)')
    ocr_pattern = re.compile(r'OCR \d+: \((\d+), (\d+), (\d+), (\d+)\) \(Knowledge_point_id: KP\d+\) (.+)')
    detection_pattern = re.compile(r'Detection \d+ \(Knowledge_pdoint_id: (KP\d+(?:, KP\d+)*)\): \((\d+), (\d+), (\d+), (\d+)\)')

    # Lists to store parsed data
    parsed_data = []

    # Current timestamp
    current_timestamp = None

    # Split content by lines
    lines = content.split('\n')
    for line in lines:
        # Check for a timestamp
        timestamp_match = timestamp_pattern.match(line)
        if timestamp_match:
            current_timestamp = int(timestamp_match.group(1))

        # Check for OCR match
        ocr_match = ocr_pattern.match(line)
        if ocr_match:
            x1, y1, x2, y2, ocr_text = ocr_match.groups()
            parsed_data.append((current_timestamp, ocr_text, (int(x1), int(y1), int(x2), int(y2))))

        # Check for detection match
        detection_match = detection_pattern.match(line)
        if detection_match:
            knowledge_points, x1, y1, x2, y2 = detection_match.groups()
            parsed_data.append((current_timestamp, f'Detection with {knowledge_points}', (int(x1), int(y1), int(x2), int(y2))))

    return parsed_data


# Parse the file and print the extracted data
parsed_data = parse_text_file(text_path)
text_timestamps = []
text_pos = []
for entry in parsed_data:
    # print(entry)
    text_timestamps.append(np.float32(entry[0])/1000)
    text_pos.append(np.array(entry[-1],dtype=np.float32))
text_timestamps = np.array(text_timestamps)
text_pos = np.array(text_pos)


def check_gaze_in_regions(gaze_timestamps, gaze_positions, parsed_data):
    results = []

    gaze_idx = 0

    num_gaze_points = len(gaze_timestamps)
    idx = 0

    while idx < len(parsed_data):
        temp_gaze = []
        temp_text = []
        # print('timestamp,rect_coords',timestamp,rect_coords)
        # Find gaze points that fall within the current timestamp range
        while (gaze_idx < num_gaze_points and gaze_timestamps[gaze_idx] >= text_timestamps[idx]):
            print(gaze_idx,num_gaze_points,gaze_timestamps[gaze_idx],text_timestamps[idx + 1])
            temp_gaze.append(gaze_idx)
            temp_text.append(idx)
            gaze_idx += 1
            idx += 1
            while gaze_timestamps[gaze_idx] >= text_timestamps[idx - 1] and gaze_timestamps[gaze_idx] <= text_timestamps[idx]:
                gaze_idx += 1
            gaze_idx -= 1

        # print(temp_text)
        print('gaze_idx,idx',gaze_idx,idx)
        if gaze_idx >= num_gaze_points:
            break

        # Check if gaze point is within rectangle region
        if gaze_idx < num_gaze_points and gaze_timestamps[gaze_idx] <= text_timestamps[idx + 1]:
            # print(1)
            for temp_gaze_idx in temp_gaze:
                gaze_x, gaze_y = gaze_positions[temp_gaze_idx]
                # print('gaze_x,gaze_y',gaze_x,gaze_y)
                for temp_text_idx in temp_text:
                    # print('text_pos[temp_text_idx]',text_pos[temp_text_idx])
                    x1, y1, x2, y2 = text_pos[temp_text_idx]
                    print('gaze_x,gaze_y,x1,y1,x2,y2',gaze_x,gaze_y,x1,y1,x2,y2)
                    if x1 <= gaze_x <= x2 and y1 <= gaze_y <= y2:
                        print('match found')
                        results.append(timestamp, gaze_positions[temp_gaze_idx])
                        break

        return results

results = check_gaze_in_regions(eye_timestamps, eye_pos, parsed_data)

# Print or process results
for result in results:
    print(result)

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:/a/736766.html

如若内容造成侵权/违法违规/事实不符,请联系我们进行投诉反馈qq邮箱809451989@qq.com,一经查实,立即删除!

相关文章

线程C++

#include <thread> #include <chrono> #include <cmath> #include <mutex> #include <iostream> using namespace std;mutex mtx; void threadCommunicat() {int ans 0;while (ans<3){mtx.lock();//上锁cout << "ans" <…

C++初学者指南第一步---14.函数调用机制

C初学者指南第一步—14.函数调用机制 文章目录 C初学者指南第一步---14.函数调用机制1.记住&#xff1a;内存的结构2.函数调用是如何工作的3. 不要引用局部变量4. 常见编译器优化5. Inlining内联 1.记住&#xff1a;内存的结构 堆&#xff08;自由存储&#xff09; 用于动态存…

Cesium如何高性能的实现上万条道路的流光穿梭效果

大家好&#xff0c;我是日拱一卒的攻城师不浪&#xff0c;专注可视化、数字孪生、前端、nodejs、AI学习、GIS等学习沉淀&#xff0c;这是2024年输出的第20/100篇文章&#xff1b; 前言 在智慧城市的项目中&#xff0c;经常会碰到这样一个需求&#xff1a;领导要求将全市的道路…

PADS学习笔记

1.PADS设计PCB流程 封装库&#xff08;layout&#xff09;&#xff0c;原理图库&#xff08;logic&#xff09;的准备原件封装的匹配&#xff08;logic&#xff09;原理图的绘制&#xff08;logic&#xff09;导网表操作&#xff08;logic&#xff09;导入结构&#xff08;lay…

SpringBoot + 虚拟线程,鸟枪换大炮!

“虚拟”线程&#xff0c;望文生义&#xff0c;它是“假”的&#xff0c;它不直接调度操作系统的线程&#xff0c;而是由JVM再提供一层线程的接口抽象&#xff0c;由普通线程调度&#xff0c;即一个普通的操作系统线程可以调度成千上万个虚拟线程。 虚拟线程比普通线程的消耗要…

Centos7安装自动化运维Ansible

自动化运维Devops-Ansible Ansible是新出现的自动化运维工具&#xff0c;基于Python 开发&#xff0c;集合了众多运维工具&#xff08;puppet 、cfengine、chef、func、fabric&#xff09;的优点&#xff0c;实现了批量系统配置 、批量程序部署、批量运行命令 等功能。Ansible…

Java8 --- Gradle7.4整合IDEA

目录 一、Gradle整合IDEA 1.1、Groovy安装 1.1.1、配置环境变量 ​编辑 1.2、创建项目 ​编辑 1.3、Groovy基本语法 1.3.1、基本语法 1.3.2、引号 1.3.3、语句结构 1.3.4、数据类型 1.3.5、集合操作 1.4、使用Gradle创建普通Java工程 1.5、使用Gradle创建Java ss…

JavaScript之类(1)

class基础语法结构&#xff1a; 代码&#xff1a; class MyClass {constructor() { ... }method1() { ... }method2() { ... }method3() { ... }... } 解释&#xff1a; 属性解释class是我们定义的类型(类)MyClass是我们定义的类的名称 constructor()我们可以在其中初始化对象m…

基于YOLOv5的PCB板缺陷检测系统的设计与实现(PyQT页面+YOLOv5模型+数据集)

简介 随着电子设备的广泛应用,PCB(印刷电路板)作为其核心部件,其质量和可靠性至关重要。然而,PCB生产过程中常常会出现各种缺陷,如鼠咬伤、开路、短路、杂散、伪铜等。这些缺陷可能导致设备故障,甚至引发严重的安全问题。为了提高PCB检测的效率和准确性,我们基于YOLOv…

2-14 基于matlab的GA优化算法优化车间调度问题

基于matlab的GA优化算法优化车间调度问题。n个工作在m个台机器上加工。已知每个工作中工序加工顺序、各工序的加工时间以及每个工件所包含的工序&#xff0c;在满足约束条件的前提下&#xff0c;目的是确定机器上各工件顺序&#xff0c;以保证某项性能指标最优。程序功能说明&a…

华为数通——单臂路由

单臂路由&#xff1a;指在三层设备路由器的一个接口上通过配置子接口&#xff08;或“逻辑接口”&#xff0c;并不存在真正物理接口&#xff09;的方式&#xff0c;实现原来相互隔离的不同VLAN&#xff08;虚拟局域网&#xff09;之间的互联互通。但是仅仅允许单播通信。 单臂路…

如何在Qt Designer中管理QSplitter

问题描述 当按下按钮时&#xff0c;我希望弹出一个对话框&#xff0c;用户可以在其中选择内容并最终按下 ‘Ok’ 按钮。我想在这个对话框中放置一个 QSplitter&#xff0c;左侧面板将显示树状结构&#xff0c;右侧将显示其他内容。如何正确实现这一点&#xff1f; 从 Qt 的示…

6.S081的Lab学习——Lab8: locks

文章目录 前言一、Memory allocator(moderate)提示&#xff1a;解析 二、Buffer cache(hard)解析&#xff1a; 三、Barrier (moderate)解析&#xff1a; 总结 前言 一个本硕双非的小菜鸡&#xff0c;备战24年秋招。打算尝试6.S081&#xff0c;将它的Lab逐一实现&#xff0c;并…

番外篇 | 基于YOLOv5-RCS的明火烟雾检测 | 源于RCS-YOLO

前言:Hello大家好,我是小哥谈。RCS-YOLO是一种目标检测算法,它是基于YOLOv3算法的改进版本。通过查看RCS-YOLO的整体架构可知,其中包括RCS-OSA模块。RCS-OSA模块在模型中用于堆叠RCS模块,以确保特征的复用并加强不同层之间的信息流动。本文就给大家详细介绍如何将RCS-YOLO…

【原创】springboot+mysql海鲜商城设计与实现

个人主页&#xff1a;程序猿小小杨 个人简介&#xff1a;从事开发多年&#xff0c;Java、Php、Python、前端开发均有涉猎 博客内容&#xff1a;Java项目实战、项目演示、技术分享 文末有作者名片&#xff0c;希望和大家一起共同进步&#xff0c;你只管努力&#xff0c;剩下的交…

yii2 ActiveForm使用技巧

持续更新&#xff1a; 1、搜索输入框&#xff1a;form-inline <?php $form ActiveForm::begin([action > [index],method > get,options > [class > form-inline] &#xff08;增加此行代码&#xff09; ]); ?>

大型企业网络DHCP服务器配置安装实践@FreeBSD

企业需求 需要为企业里的机器配置一台DHCP服务器。因为光猫提供DHCP服务的能力很差&#xff0c;多机器dhcp多机器NAT拓扑方式机器一多就卡顿。使用一台路由器来进行子网络的dhcp和NAT服务&#xff0c;分担光猫负载&#xff0c;但是还有一部分机器需要放到光猫网络&#xff0c;…

本地离线模型搭建指南-中文大语言模型底座选择依据

搭建一个本地中文大语言模型&#xff08;LLM&#xff09;涉及多个关键步骤&#xff0c;从选择模型底座&#xff0c;到运行机器和框架&#xff0c;再到具体的架构实现和训练方式。以下是一个详细的指南&#xff0c;帮助你从零开始构建和运行一个中文大语言模型。 本地离线模型搭…

【网络安全的神秘世界】已解决burpsuite报错Failed to start proxy service on 127.0.0.1:8080

&#x1f31d;博客主页&#xff1a;泥菩萨 &#x1f496;专栏&#xff1a;Linux探索之旅 | 网络安全的神秘世界 | 专接本 | 每天学会一个渗透测试工具 解决burpsuite无法在 127.0.0.1&#xff1a;8080 上启动代理服务端口被占用以及抓不到本地包的问题 Burpsuite无法启动proxy…