Kettle Local引擎使用记录(一)(基于Kettle web版数据集成开源工具data-integration源码)

Kettle Web

  • 📚第一章 前言
  • 📚第二章 demo源码
    • 📗pom.xml引入Kettle引擎核心文件
    • 📗java源码
      • 📕 controller
      • 📕 service
      • 📕 其它
      • 📕 maven settings.xml
    • 📗测试
      • 📕 测试文件
      • 📕 测试结果
  • ⁉️问题记录
    • ❓问题一:jar包冲突 - `An attempt was made to call the method javax.servlet.ServletContext.setInitParameter(Ljava/lang/String;Ljava/lang/String;)Z but it does not exist. Its class, javax.servlet.ServletContext, is available from the following locations:`
      • ❗解决方式一:Kettle排除掉冲突jar(因为没引入Kettle之前是没问题的)
      • ❕拓展:通过`Maven Helper`插件排除冲突jar包
    • ❓问题二:`Database type not found!`...`database type with plugin id [Oracle] couldn't be found!`
      • ❗解决思路一:该插件类型不存在,需要手动制作
      • ❗解决思路二:确保json转化的xml文件正确
      • ❗解决思路三:正确加载kettle

📚第一章 前言

在前面对data-integration做了一些简单了解,从部署到应用,今天尝试把后端运行作业代码拎出来,去真正运行一下,只有实操之后才会有更深刻的认识,有些看着简单的功能,实操过程中会遇到很多问题,这个时候你的想法也会发生改变,所以很多时候为什么开发人员痛恨做产品的同事,因为嘴上说的简单,真正去做一下,你自己感受一下,就知道了!

  • 基于Kettle开发的web版数据集成开源工具(data-integration)-部署篇
  • 基于Kettle开发的web版数据集成开源工具(data-integration)-介绍篇
  • 基于Kettle开发的web版数据集成开源工具(data-integration)-应用篇

data-integration可以理解为Kettleweb版本,详情可参照上面的文章,这里直接从代码层面开始

📚第二章 demo源码

直接在我之前个人项目上进行的韧小钊(纯个人项目,整理或者更新全靠一时兴起,如果对你有用,挺好,没用请PASS),现在想法就是后端实现运行作业的功能,至于保存作业,都是前端传给后台的,后台存的就是JSON串,所以后端改动应该不大,这里就是尝试先提取后端运行作业代码,直接运行前端传参,看能否成功
在这里插入图片描述

📗pom.xml引入Kettle引擎核心文件

引入之后,启动项目报jar包冲突,则在下面文件中进行exclusion配置排除

<!-- Kettle核心包 20240106 by rxz-->
<dependency>
	<groupId>pentaho-kettle</groupId>
	<artifactId>kettle-engine</artifactId>
	<version>${kettle.version}</version>
	<exclusions>
		<exclusion>
			<artifactId>servlet-api</artifactId>
			<groupId>javax.servlet</groupId>
		</exclusion>
	</exclusions>
</dependency>
<dependency>
	<groupId>pentaho-kettle</groupId>
	<artifactId>kettle-core</artifactId>
	<version>${kettle.version}</version>
	<exclusions>
		<exclusion>
			<artifactId>servlet-api</artifactId>
			<groupId>javax.servlet</groupId>
		</exclusion>
	</exclusions>
</dependency>

📗java源码

📕 controller

package com.renxiaozhao.api.controller;

import com.renxiaozhao.common.Result;
import com.renxiaozhao.service.inf.PdiUseDemoService;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

/**
 * kettle/pdi使用样例-控制层
 *
 * @author 韧小钊
 */
@RestController
@RequestMapping("/kettle")
@Api(tags = "KETTLE-PDI")
@Slf4j
public class PdiUseDemoController {
    /**
     * 服务对象
     */
    @Autowired
    private PdiUseDemoService pdiUseDemoService;
    
    @PostMapping("/run")
    @ApiOperation(value = "kettle作业运行", notes = "kettle作业运行")
    public Result<Void> executeJob(@RequestParam String jobJson) throws Exception {
        pdiUseDemoService.executeJob(jobJson);
        return Result.success();
    }

}


📕 service

这里就是对应基于Kettle开发的web版数据集成开源工具(data-integration)-应用篇中的后台服务流程章节ProjectExecutorController.executeById,目的是研究Kettle引擎的具体使用,所以这里是去掉了各种封装,直接前端传参(作业JSON串),后端直接执行

package com.renxiaozhao.service.inf;

import com.baomidou.mybatisplus.extension.service.IService;
import com.renxiaozhao.bean.entity.SportEntity;

public interface PdiUseDemoService extends IService<SportEntity> {
//直接复制的原来代码,有些实体类暂时用不到,请先忽略
    void executeJob(String jobJson) throws Exception;
}

package com.renxiaozhao.service.impl;

import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.renxiaozhao.bean.entity.SportEntity;
import com.renxiaozhao.dao.mapper.SportMapper;
import com.renxiaozhao.service.inf.PdiUseDemoService;
import com.renxiaozhao.service.util.JSONLinkedObject;
import com.renxiaozhao.service.util.XML;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringEscapeUtils;
import org.pentaho.di.core.exception.KettleMissingPluginsException;
import org.pentaho.di.core.exception.KettleXMLException;
import org.pentaho.di.core.logging.LogLevel;
import org.pentaho.di.core.logging.LoggingObjectType;
import org.pentaho.di.core.logging.SimpleLoggingObject;
import org.pentaho.di.core.variables.Variables;
import org.pentaho.di.core.xml.XMLHandler;
import org.pentaho.di.trans.Trans;
import org.pentaho.di.trans.TransAdapter;
import org.pentaho.di.trans.TransExecutionConfiguration;
import org.pentaho.di.trans.TransMeta;
import org.springframework.stereotype.Service;
import org.w3c.dom.Document;

import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.UUID;

@Slf4j
@Service
public class PdiUseDemoServiceImpl extends ServiceImpl<SportMapper,SportEntity> implements PdiUseDemoService {

    @Override
    public void executeJob(String jobJson) throws Exception {
        execute(jobJson);
    }
    private void execute(String jobJson) throws Exception {
        // 构建TransMeta 对象
        TransMeta transMeta = buildTransMeta(jobJson);
        TransExecutionConfiguration executionConfiguration = new TransExecutionConfiguration();
        // 设置默认值以便运行配置可以正确设置
        executionConfiguration.setExecutingLocally(true);
        executionConfiguration.setExecutingRemotely(false);
        executionConfiguration.setExecutingClustered(false);
        // 不启用安全模式
        executionConfiguration.setSafeModeEnabled(true);
        executionConfiguration.getUsedVariables(transMeta);
        executionConfiguration.setLogLevel(LogLevel.DEBUG);
        // 默认设置本地引擎执行
        executionConfiguration.setRunConfiguration("Pentaho local");
        //设置命令参数
        executionConfiguration.setVariables(new HashMap<>());
        //  创建trans
        Trans trans = new Trans(transMeta);
        String spoonLogObjectId = UUID.randomUUID().toString();
        SimpleLoggingObject spoonLoggingObject = new SimpleLoggingObject(Thread.currentThread().getName() + "-" + Thread.currentThread().getId()
                , LoggingObjectType.SPOON, null);
        spoonLoggingObject.setContainerObjectId(spoonLogObjectId);
        spoonLoggingObject.setLogLevel(executionConfiguration.getLogLevel());
        trans.setParent(spoonLoggingObject);
        trans.setLogLevel(executionConfiguration.getLogLevel());
        trans.setReplayDate(executionConfiguration.getReplayDate());
        trans.setRepository(executionConfiguration.getRepository());
        trans.setMonitored(false);
        // 启动转换
        trans.addTransListener(new TransAdapter() {
            @Override
            public void transFinished(Trans trans) {
                log.info("项目执行完成");
            }
        });
        trans.startThreads();

    }
    public TransMeta buildTransMeta(String jobJson) throws IOException, KettleXMLException, KettleMissingPluginsException {
        Document document;
        //json转xml
        if (!jobJson.startsWith("<?xml")) {
            // json转xml
            jobJson = StringEscapeUtils.unescapeXml(jobJson);
            jobJson = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + XML.toString(new JSONLinkedObject(jobJson));

            log.info("json转换成xml,转换后的xml:{}", jobJson);
        }

        // 写到临时目录
        File outFile = new File("D:\\tmp\\test", "test.xml");
        FileUtils.writeStringToFile(outFile, jobJson);

        // 加载xml
        document = XMLHandler.loadXMLString(jobJson);

        TransMeta transMeta = new TransMeta();
        transMeta.loadXML(
                document.getDocumentElement(), outFile.getPath(), null, null, true, new Variables(),
                (message, rememberText, rememberPropertyName) -> {
                    // Yes means: overwrite
                    return true;
                });

        if (transMeta.hasMissingPlugins()) {
            log.info("【{}】缺少执行插件。", jobJson);
        }

        return transMeta;
    }
}

📕 其它

在这里插入图片描述
代码已上传
在这里插入图片描述

📕 maven settings.xml

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd">

    <localRepository>D:\soft\apache-maven-3.8.5\apache-maven-3.8.5\repository</localRepository>
    <pluginGroups></pluginGroups>
    <proxies></proxies>
    <servers></servers>
    <mirrors>
        <mirror>
            <id>pentaho-public</id>
            <name>Pentaho Public Mirror</name>
            <url>https://repo.orl.eng.hitachivantara.com/artifactory/pnt-mvn/</url>
            <mirrorOf>*</mirrorOf>
        </mirror>
        <mirror>
            <id>aliyunmaven</id>
            <mirrorOf>*</mirrorOf>
            <name>阿里云公共仓库</name>
            <url>https://maven.aliyun.com/repository/public</url>
        </mirror>
    </mirrors>
    <profiles></profiles>

</settings>

📗测试

启动服务,通过swagger测试
在这里插入图片描述
在这里插入图片描述

📕 测试文件

传参对应的就是dp_portal_project_fileproject_file,详见基于Kettle开发的web版数据集成开源工具(data-integration)-应用篇
在这里插入图片描述
测试文件参考如下(直接使用的话,记得修改数据库信息):

{
    "transformation": {
        "attributes": "",
        "connection": [
            {
                "access": "Native",
                "attributes": {
                    "attribute": [
                        {
                            "attribute": "Y",
                            "code": "SUPPORTS_BOOLEAN_DATA_TYPE"
                        },
                        {
                            "attribute": "Y",
                            "code": "SUPPORTS_TIMESTAMP_DATA_TYPE"
                        },
                        {
                            "attribute": "N",
                            "code": "QUOTE_ALL_FIELDS"
                        },
                        {
                            "attribute": "N",
                            "code": "FORCE_IDENTIFIERS_TO_LOWERCASE"
                        },
                        {
                            "attribute": "N",
                            "code": "FORCE_IDENTIFIERS_TO_UPPERCASE"
                        },
                        {
                            "attribute": "Y",
                            "code": "PRESERVE_RESERVED_WORD_CASE"
                        },
                        {
                            "attribute": "N",
                            "code": "STRICT_NUMBER_38_INTERPRETATION"
                        },
                        {
                            "attribute": "",
                            "code": "PREFERRED_SCHEMA_NAME"
                        }
                    ]
                },
                "dataSourceId": "e2be98ed-9077-4802-8795-df45acb7b033",
                "dataSourceName": "dataintegration_db",
                "data_tablespace": "",
                "database": "dataintegration_db",
                "dsName": "dataintegration_db",
                "index_tablespace": "",
                "name": "e2be98ed-9077-4802-8795-df45acb7b033",
                "password": "Encrypted 2be98afc86ae49d94a40dab508cc2fd89",
                "port": "3306",
                "server": "192.168.17.10",
                "servername": "",
                "type": "MYSQL",
                "username": "stelladp"
            }
        ],
        "info": {
            "capture_step_performance": "N",
            "clusterschemas": "",
            "created_date": "2020/02/05 13:34:55.102",
            "created_user": "037de45a-e2ad-4d0a-943c-455db05161f4",
            "dependencies": "",
            "description": "",
            "directory": "/home/admin",
            "extended_description": "",
            "feedback_shown": "Y",
            "feedback_size": 50000,
            "is_key_private": "N",
            "key_for_session_key": "",
            "log": "",
            "maxdate": {
                "connection": "",
                "field": "",
                "maxdiff": 0,
                "offset": 0,
                "table": ""
            },
            "modified_date": "2020/02/19 17:08:25.150",
            "modified_user": "admin",
            "name": "执行SQL",
            "parameters": {
                "parameter": [
                ]
            },
            "partitionschemas": "",
            "shared_objects_file": "",
            "size_rowset": 10000,
            "slaveservers": "",
            "sleep_time_empty": 50,
            "sleep_time_full": 50,
            "step_performance_capturing_delay": 1000,
            "step_performance_capturing_size_limit": 100,
            "trans_type": "Normal",
            "trans_version": "",
            "unique_connections": "N",
            "using_thread_priorities": "Y"
        },
        "nodeList": {
            "flowInfo": {
                "Id": "fdf50d66dc7d",
                "Name": "我的流程",
                "Remark": ""
            },
            "lineList": [
            ],
            "nodeList": [
                {
                    "copy": 0,
                    "dataQuerySql": "",
                    "dataSourceId": "",
                    "datasourceId": "",
                    "errors": 0,
                    "icon": "iconfont icon-kongjian1",
                    "id": "ExecSQL6925c0883a9b",
                    "label": "执行SQL脚本",
                    "left": "341px",
                    "linesInput": 0,
                    "linesOutput": 0,
                    "linesRead": 0,
                    "linesRejected": 0,
                    "linesUpdated": 0,
                    "linesWritten": 0,
                    "pluginId": "ExecSQL",
                    "pluginOutput": "Y",
                    "priority": "-",
                    "projectError": 0,
                    "projectId": "",
                    "projectName": "",
                    "seconPluginType": "scripts",
                    "seconds": 0,
                    "speed": 0,
                    "status": false,
                    "statusDescription": "",
                    "stepExecutionStatu": "",
                    "stepExecutionStatus": "",
                    "stepMetaExecutionStatus": null,
                    "stepName": "",
                    "stepSql": "",
                    "top": "212px"
                }
            ],
            "statusListen": [
            ]
        },
        "notepads": "",
        "order": {
            "hop": [
            ]
        },
        "slave-step-copy-partition-distribution": "",
        "slave_transformation": "N",
        "step": [
            {
                "BindString": "",
                "Connectpool": false,
                "Excute_sql": "",
                "GUI": {
                    "draw": "Y",
                    "xloc": "160",
                    "yloc": "224"
                },
                "arguments": {
                    "argument": [
                    ]
                },
                "attributes": [
                ],
                "bool": true,
                "cluster": false,
                "cluster_schema": [
                ],
                "connecName": "",
                "connect_size": "",
                "connection": "e2be98ed-9077-4802-8795-df45acb7b033",
                "copies": "1",
                "custom_distribution": [
                ],
                "defalut_mode": "",
                "delete_field": "",
                "descripe": "The default catalog of connections created by this pool.",
                "description": [
                ],
                "distribute": "Y",
                "execute_each_row": "N",
                "fields": {
                    "field": [
                    ]
                },
                "free_space": "",
                "header": "N",
                "identifier": false,
                "identifier_captial": false,
                "identifier_lowercase": false,
                "initFlag": true,
                "insert_field": "",
                "name": "执行SQL脚本",
                "noempty": "N",
                "oldStepName": "执行SQL脚本",
                "outFields": [
                ],
                "partitioning": {
                    "method": "none",
                    "schema_name": [
                    ]
                },
                "preserve_case": true,
                "quoteString": "N",
                "read_field": "",
                "remotesteps": {
                    "input": "\n      ",
                    "output": "\n      "
                },
                "replace_variables": "N",
                "set_params": false,
                "single_statement": "Y",
                "sql": "select count(1) from dp_portal_role",
                "stoponempty": "N",
                "strict_number": false,
                "time_stamp": true,
                "type": "ExecSQL",
                "update_field": ""
            }
        ],
        "step_error_handling": {
            "error": [
            ]
        }
    }
}

📕 测试结果

报错了…😔详见问题二
在这里插入图片描述

⁉️问题记录

❓问题一:jar包冲突 - An attempt was made to call the method javax.servlet.ServletContext.setInitParameter(Ljava/lang/String;Ljava/lang/String;)Z but it does not exist. Its class, javax.servlet.ServletContext, is available from the following locations:

java.lang.BootstrapMethodError: java.lang.NoSuchMethodError: javax.servlet.ServletContext.setInitParameter(Ljava/lang/String;Ljava/lang/String;)Z
	at org.springframework.boot.web.servlet.server.AbstractServletWebServerFactory.lambda$mergeInitializers$0(AbstractServletWebServerFactory.java:253)
	at org.springframework.boot.web.embedded.jetty.ServletContextInitializerConfiguration.callInitializers(ServletContextInitializerConfiguration.java:66)
	at org.springframework.boot.web.embedded.jetty.ServletContextInitializerConfiguration.configure(ServletContextInitializerConfiguration.java:55)
	at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1454)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
	at org.eclipse.jetty.server.Server.start(Server.java:415)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.Server.doStart(Server.java:382)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.springframework.boot.web.embedded.jetty.JettyWebServer.initialize(JettyWebServer.java:108)
	at org.springframework.boot.web.embedded.jetty.JettyWebServer.<init>(JettyWebServer.java:86)
	at org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.getJettyWebServer(JettyServletWebServerFactory.java:410)
	at org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.getWebServer(JettyServletWebServerFactory.java:153)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:181)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:154)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:142)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)
	at com.renxiaozhao.api.ApiApplicationServer.main(ApiApplicationServer.java:21)
Caused by: java.lang.NoSuchMethodError: javax.servlet.ServletContext.setInitParameter(Ljava/lang/String;Ljava/lang/String;)Z
	at java.lang.invoke.MethodHandleNatives.resolve(Native Method)
	at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:975)
	at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1000)
	at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1394)
	at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1750)
	at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:477)
	... 29 common frames omitted
2024-01-06 13:40:50.233 [main] INFO  org.eclipse.jetty.server.AbstractConnector -
				Started ServerConnector@697a0948{HTTP/1.1,[http/1.1]}{0.0.0.0:9080}
2024-01-06 13:40:50.242 [main] INFO  org.eclipse.jetty.server.AbstractConnector -
				Stopped ServerConnector@697a0948{HTTP/1.1,[http/1.1]}{0.0.0.0:9080}
2024-01-06 13:40:50.243 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler -
				Stopped o.s.b.w.e.j.JettyEmbeddedWebAppContext@521441d5{application,/renxiaozhao,[file:///C:/Users/lenovo/AppData/Local/Temp/jetty-docbase.1734690716899826200.9080/, jar:file:/D:/soft/apache-maven-3.8.5/apache-maven-3.8.5/repository/io/springfox/springfox-swagger-ui/2.9.2/springfox-swagger-ui-2.9.2.jar!/META-INF/resources, jar:file:/D:/soft/apache-maven-3.8.5/apache-maven-3.8.5/repository/com/github/xiaoymin/knife4j-spring-ui/2.0.8/knife4j-spring-ui-2.0.8.jar!/META-INF/resources],UNAVAILABLE}
2024-01-06 13:40:50.247 [main] WARN  o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext -
				Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Jetty web server
2024-01-06 13:40:50.261 [main] INFO  o.s.b.a.l.ConditionEvaluationReportLoggingListener -
				

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2024-01-06 13:40:50.287 [main] ERROR o.s.b.diagnostics.LoggingFailureAnalysisReporter -
				

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call the method javax.servlet.ServletContext.setInitParameter(Ljava/lang/String;Ljava/lang/String;)Z but it does not exist. Its class, javax.servlet.ServletContext, is available from the following locations:

    jar:file:/D:/soft/apache-maven-3.8.5/apache-maven-3.8.5/repository/javax/servlet/servlet-api/2.4/servlet-api-2.4.jar!/javax/servlet/ServletContext.class
    jar:file:/D:/soft/apache-maven-3.8.5/apache-maven-3.8.5/repository/javax/servlet/javax.servlet-api/4.0.1/javax.servlet-api-4.0.1.jar!/javax/servlet/ServletContext.class

It was loaded from the following location:

    file:/D:/soft/apache-maven-3.8.5/apache-maven-3.8.5/repository/javax/servlet/servlet-api/2.4/servlet-api-2.4.jar


Action:

Correct the classpath of your application so that it contains a single, compatible version of javax.servlet.ServletContext

Disconnected from the target VM, address: '127.0.0.1:50771', transport: 'socket'

Process finished with exit code 1

❗解决方式一:Kettle排除掉冲突jar(因为没引入Kettle之前是没问题的)

在这里插入图片描述

<kettle.version>9.2.0.0-290</kettle.version>
<dependency>
	<groupId>pentaho-kettle</groupId>
	<artifactId>kettle-engine</artifactId>
	<version>${kettle.version}</version>
	<exclusions>
		<exclusion>
			<artifactId>servlet-api</artifactId>
			<groupId>javax.servlet</groupId>
		</exclusion>
	</exclusions>
</dependency>

<dependency>
	<groupId>pentaho-kettle</groupId>
	<artifactId>kettle-core</artifactId>
	<version>${kettle.version}</version>
	<exclusions>
		<exclusion>
			<artifactId>servlet-api</artifactId>
			<groupId>javax.servlet</groupId>
		</exclusion>
	</exclusions>
</dependency>

在这里插入图片描述

❕拓展:通过Maven Helper插件排除冲突jar包

使用IDEA工具,安装Maven Helper插件,进行冲突jar包排查,步骤如下:

  • 安装插件
    在这里插入图片描述
  • 安装完成之后,打开pom.xml文件,可以看到Dependency Analyzer
    在这里插入图片描述
  • 点击Dependency Analyzer进行分析
    在这里插入图片描述
    在这里插入图片描述
    在这里插入图片描述
  • 尽量定位修改的位置进行排查(找到修改的pom.xml进行定位)
    在这里插入图片描述
    冲突嘛(排除哪一方都行,根据报错排除对应的jar包,不行再试另一个jar包)
    在这里插入图片描述
  • 直接点击Exclude,帮你自动生成排除配置
    在这里插入图片描述
    在这里插入图片描述

❓问题二:Database type not found!database type with plugin id [Oracle] couldn't be found!

2024-01-05 16:54:41.190 [qtp1963906615-73] WARN  org.eclipse.jetty.server.HttpChannel -
				/renxiaozhao/kettle/run
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.pentaho.di.core.exception.KettleXMLException: 
错误从XML文件读取转换
Database type not found!

	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1013)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.pentaho.di.core.exception.KettleXMLException: 
错误从XML文件读取转换
Database type not found!

	at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3460)
	at com.renxiaozhao.service.impl.PdiUseDemoServiceImpl.buildTransMeta(PdiUseDemoServiceImpl.java:96)
	at com.renxiaozhao.service.impl.PdiUseDemoServiceImpl.execute(PdiUseDemoServiceImpl.java:41)
	at com.renxiaozhao.service.impl.PdiUseDemoServiceImpl.executeJob(PdiUseDemoServiceImpl.java:37)
	at com.renxiaozhao.service.impl.PdiUseDemoServiceImpl$$FastClassBySpringCGLIB$$56121f70.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)
	at com.renxiaozhao.service.impl.PdiUseDemoServiceImpl$$EnhancerBySpringCGLIB$$b8de3a8b.executeJob(<generated>)
	at com.renxiaozhao.api.controller.PdiUseDemoController.executeJob(PdiUseDemoController.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
	... 47 common frames omitted
Caused by: java.lang.RuntimeException: Database type not found!
	at org.pentaho.di.core.database.DatabaseMeta.setValues(DatabaseMeta.java:642)
	at org.pentaho.di.core.database.DatabaseMeta.setDefault(DatabaseMeta.java:525)
	at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:516)
	at org.pentaho.di.core.database.DatabaseMeta.<init>(DatabaseMeta.java:986)
	at org.pentaho.di.trans.TransMeta.loadXML(TransMeta.java:3030)
	... 68 common frames omitted
Caused by: org.pentaho.di.core.exception.KettleDatabaseException: 
database type with plugin id [Oracle] couldn't be found!

	at org.pentaho.di.core.database.DatabaseMeta.findDatabaseInterface(DatabaseMeta.java:592)
	at org.pentaho.di.core.database.DatabaseMeta.getDatabaseInterface(DatabaseMeta.java:566)
	at org.pentaho.di.core.database.DatabaseMeta.setValues(DatabaseMeta.java:640)
	... 72 common frames omitted

❗解决思路一:该插件类型不存在,需要手动制作

检查对应配置文件kettle-database-types.xml,发现配置是存在的,所以这里报错不是类型不存在所致

  <database-type id="ORACLE">
    <description>Oracle</description>
    <classname>org.pentaho.di.core.database.OracleDatabaseMeta</classname>
  </database-type>   

在这里插入图片描述

❗解决思路二:确保json转化的xml文件正确

在这里插入图片描述
在这里插入图片描述

❗解决思路三:正确加载kettle

现在怀疑就是kettle没有正确加载,但是如何排查,还是一头雾水,有知道的大佬欢迎告知,接下来再排查半小时,没结果就放弃了,大周六的,不要太卷!


问题解决已更新:Kettle Local引擎使用记录(二):问题记录及解决

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:/a/309819.html

如若内容造成侵权/违法违规/事实不符,请联系我们进行投诉反馈qq邮箱809451989@qq.com,一经查实,立即删除!

相关文章

Opencv实验合集——实验七:二维码和条形码匹配

1.概念 二维码&#xff08;QR码&#xff09; 概念&#xff1a; 二维码是一种矩阵式的二维条码&#xff0c;由黑白方块组成&#xff0c;可以存储大量的信息&#xff0c;包括文本、链接、数字等。QR码的编码方式是在矩阵中通过不同的黑白方块组合表示不同的信息。 特点&#xf…

PHP开发日志 ━━ 不同方法判断某个数组中是否存在指定的键名,测试哪种方法效率高

我们可以用isset($arr[a]) 或者 array_key_exists(a, $arr) 来判断a键名是否存在与$arr数组。 那么这两种方式哪个运行速度快呢&#xff1f; 不多废话了&#xff0c;现在我们写一段代码来测试一下&#xff1a; $array [a > 1, b > 2, c > 3];$start microtime(tru…

图像去雨-雨线清除-图像处理(计算机视觉作业-附代码)

图像去雨是一种图像处理技术&#xff0c;可以通过算法的方式去除图像中的雨滴或雨线等噪声。在实际应用中&#xff0c;图像去雨技术可以提高图像质量&#xff0c;使得图像更加清晰、自然、易于识别和理解。 通常&#xff0c;图像去雨技术基于计算机视觉和深度学习等技术&#…

算法与数据结构--最小生成树算法

一.应用的场景 类似于这种最小成本问题&#xff0c;实际上就是计算加权图把所有点连起来权重之和最小值的时候是怎么连接的。类似的问题还有最短耗时之类的问题。 二.最小生成树的定义 生成树&#xff1a; 图的生成树是它的一颗含有其所有顶点的无环连通子图。 【简单说就是所…

作业帮基于 DolphinScheduler 的数据开发平台实践

摘要 随着任务数量、任务类型需求不断增长&#xff0c;对我们的数据开发平台提出了更高的要求。本文主要分享我们将调度引擎升级到 Apache DolphinScheduler 的实践经验&#xff0c;以及对数据开发平台的一些思考。 1. 背景 首先介绍下我们的大数据平台架构&#xff1a; 数据…

在vue3+vite中使用svg-sprite-loader,antdv修改菜单icon

1. 安装 npm install vite-plugin-svg-icons -D 2. 在vite.config.js的plugins中添加配置项 import { createSvgIconsPlugin } from vite-plugin-svg-icons;createSvgIconsPlugin({iconDirs: [resolve(process.cwd(), src/components/svgIcon/svg)], // icon存放的目录&…

【Emgu.CV教程】4.4、无缝融合应用之TextureFlattening()纹理扁平化

这是无缝融合应用的最后一篇&#xff0c;TextureFlattening()函数&#xff0c;专门用于对图像指定部位进行纹理扁平化的。这个功能现在讲起来有点太早了&#xff0c;应该放到《图像的空间滤波--平滑》这一章节中才合适。因为它就是用Sobel算子进行平滑&#xff0c;也就是在保留…

API(Date类,SimpleDateFormat类,Calendar类,JDK8时间相关类,包装类,算法小题)

文章目录 【常用API】今日内容教学目标 第一章 Date类1.1 Date概述1.2 Date常用方法 第二章 SimpleDateFormat类2.1 构造方法2.2 格式规则2.3 常用方法2.4 练习1(初恋女友的出生日期)2.5 练习2(秒杀活动) 第三章 Calendar类3.1 概述3.2 常用方法3.3 get方法示例3.4 set方法示例…

EI级 | Matlab实现VMD-TCN-LSTM变分模态分解结合时间卷积长短期记忆神经网络多变量光伏功率时间序列预测

EI级 | Matlab实现VMD-TCN-LSTM变分模态分解结合时间卷积长短期记忆神经网络多变量光伏功率时间序列预测 目录 EI级 | Matlab实现VMD-TCN-LSTM变分模态分解结合时间卷积长短期记忆神经网络多变量光伏功率时间序列预测预测效果基本介绍程序设计参考资料 预测效果 基本介绍 1.【E…

CSS3实现轮播效果

在我们不使用JS的情况下&#xff0c;是否也可以实现轮播功能呢&#xff1f; 答应是可以的 上代码&#xff1a; <!DOCTYPE html> <html lang"en"> <head><meta charset"UTF-8"><title>轮播</title><style>.boss…

⭐Unity 将电脑打开的窗口画面显示在程序中

1.效果&#xff1a; 下载资源包地址&#xff1a; Unity中获取桌面窗口 2.下载uWindowCapturev1.1.2.unitypackage 放入Unity工程 3.打开Single Window场景&#xff0c;将组件UwcWindowTexture的PartialWindowTitle进行修改&#xff0c;我以腾讯会议为例 感谢大家的观看&#xf…

QT开发 2024最新版本优雅的使用vscode开发QT

▬▬▬▬▬▶VS开发QT◀▬▬▬▬▬ &#x1f384;先看效果 &#x1f384;编辑环境变量 如图添加环境变量&#xff01;&#xff01;&#xff01; 东西全在QT的安装目录&#xff01;&#xff01;&#xff01; 找到的按照我的教程再装一次&#xff01;&#xff01;&#xff01; 点…

如何给AVR16芯片解锁

AVRM16核心板本身集成了强大的芯片自解锁功能模块&#xff0c;当由于熔丝位设置错误&#xff0c;导致芯片锁死&#xff0c;无法正常使用时候&#xff0c;可以利用畅学AVR16核心板上的解锁功能给芯片解锁。 &#xff08;如果芯片没有锁死&#xff0c;可以跳过此步骤&#xff09…

yolov5无人机视频检测与计数系统(创新点和代码)

标题&#xff1a;基于YOLOv5的无人机视频检测与计数系统 摘要&#xff1a; 无人机技术的快速发展和广泛应用给社会带来了巨大的便利&#xff0c;但也带来了一系列的安全隐患。为了实现对无人机的有效管理和监控&#xff0c;本文提出了一种基于YOLOv5的无人机视频检测与计数系…

排序之插入排序

在计算机科学中&#xff0c;排序算法是一种将一组元素按照某种特定顺序排列的方法。插入排序是一种简单且易于理解的排序算法&#xff0c;它的基本思想是将一个记录插入到已经排好序的有序表中&#xff0c;从而得到一个新的、记录数增1的有序表。 基本思想 插入排序的基本思想…

Ubuntu 20.04 Intel RealSense D435i 相机标定教程

下载编译code_utils mkdir -p ~/imu_catkin_ws/src cd ~/imu_catkin_ws/src catkin_init_workspace source ~/imu_catkin_ws/devel/setup.bash git clone https://github.com/gaowenliang/code_utils.git cd .. catkin_make报错&#xff1a;sumpixel_test.cpp:2:10: fatal err…

计算机网络 物理层

文章目录 物理层物理层的基本概念数据通信的基础知识数据通信系统的模型有关信道的几个基本概念信道的极限容量 物理层下面的传输媒体导引型传输媒体非引导型传输媒体 信道复用技术波分复用码的复用 宽带接入技术ADSL 技术光纤同轴混合网 (HFC 网&#xff09;FTTx 技术 物理层 …

基于sprinmgboot实习管理系统源码和论文

随着信息化时代的到来&#xff0c;管理系统都趋向于智能化、系统化&#xff0c;实习管理也不例外&#xff0c;但目前国内仍都使用人工管理&#xff0c;市场规模越来越大&#xff0c;同时信息量也越来越庞大&#xff0c;人工管理显然已无法应对时代的变化&#xff0c;而实习管理…

新生儿成长的阳光之钙:补充注意事项指南

引言&#xff1a; 钙是新生儿骨骼发育不可或缺的重要元素&#xff0c;对于宝宝的生长发育起着至关重要的作用。本文将深入探讨钙的功能、补充时机&#xff0c;以及在给新生儿补充钙时应该注意的事项&#xff0c;为小天使们提供最贴心的呵护。 第一部分&#xff1a;钙的重要性与…

【python入门】day21:向文件输出“奋斗成就更好的你”、输出北京的天气预报

向文件输出“奋斗成就更好的你” #向文件输出‘奋斗成就更好的你’ 第一种方式&#xff1a;使用print方式进行输出&#xff08;输出目的地是文件&#xff09; fpopen(e:/text.txt,w)#w只写模式&#xff0c;也可以用a读写模式 print(奋斗成就更好的你,filefp) fp.close() 第二种…