hive on spark 的架构和常见问题 - hive on spark 使用的是 yarn client 模式还是 yarn cluster 模式？

hive on spark 的架构和常见问题 - hive on spark 使用的是 yarn client 模式还是 yarn cluster 模式？

1. 回顾下 spark 的架构图和部署模式

来自官方的经典的 spark 架构图如下：

image

image
上述架构图，从进程的角度来讲，有四个角色/组件：

  • Cluster manager：An external service for acquiring resources on the cluster (e.g. standalone manager, Mesos, YARN, Kubernetes)
  • Worker node: Any node that can run application code in the cluster
  • Driver program：The process running the main() function of the application and creating the SparkContext
  • Executor: A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.
  • 上述四个角色中，Cluster manager 和 Worker，是常驻的静态的资源管理器，是独立于 spark 应用程序之外的；
  • 上述四个角色中，driver 和 executor，是每个 spark 应用程序的运行时动态视图，用户提交的每个 spark 应用程序，都会向资源管理器申请资源，并在申请获得的资源中启动 driver 和 exector 对应的 jvm 进程；
  • spark 支持多种资源管理器，常见的有 standalone/yarn/k8s/mesos,其中 mesos 目前已经 deprecated 了；
  • spark deploy mode: spark部署模式：当用户（或框架）提交 spark 应用程序时(从client 客户端提交)，上述 driver 角色可以运行在客户端进程中，也可以运行在从资源管理器申请获得的资源中启动的 jvm 进程中，前者被 spark 称为 client 模式，而后者被 spark 称为 cluster 模式；
  • 用户可以在 spark-defaults.conf 等配置文件中，通过参数 spark.submit.deployMode 指定具体的部署模式；
  • 用户也可以在使用 spark-sumbmit 脚本提交 spark 作业时，通过参数 --deploy-mode，指定具体的部署模式；
  • Spark is agnostic to the underlying cluster manager. As long as it can acquire executor processes, and these communicate with each other, it is relatively easy to run it even on a cluster manager that also supports other applications (e.g. Mesos/YARN/Kubernetes).
  • Deploy mode: Distinguishes where the driver process runs. In “cluster” mode, the framework launches the driver inside of the cluster. In “client” mode, the submitter launches the driver outside of the cluster.(Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client),default: client);
  • spark.submit.deployMode：The deploy mode of Spark driver program, either “client” or “cluster”, Which means to launch driver program locally (“client”) or remotely (“cluster”) on one of the nodes inside the cluster；

2. hive on spark 使用的是 yarn client 模式还是 yarn cluster 模式？

  • hive 作为跟 hadoop 紧密耦合的的一个重要组件，其调度运行作业使用的资源管理器，就是 yarn;
  • hive 支持多种计算引擎，常见的有 mr/spark/tez,可以通过 set hive.execution.engine = mr/spark/tez 进行配置；
  • 当 hive 应用使用 spark 引擎运行时，我们称之为 hive on spark；相对应地，当 spark 应用访问 hvie 数据源时，我们称之为 spark on hive;
  • hive on spark 作业运行时，用户并没有明确指定部署模式，此时 hive on spark 作业，使用的是 client 模式还是 cluster 模式呢？
  • 查看 hs2 源码可知，hive on spark 作业，是 hs2 进程作为 spark 客户端，通过调用 spark-submit 脚本提交的，而 spark 相关的配置参数，有一部分是用户通过 spark-defaults.conf 配置的，也有一部分是 hs2 代码写死了的配置;
  • 查看 hs2 日志，可以看到动态生称的配置文件，比如 run/cloudera-scm-agent/process/5666-hive-HIVESERVER2/spark-defaults.conf，其中明确指定了spark.master=yarn, 而该配置文件中 spark.submit.deployMode 的值，取决于 spark-defaults.conf中的配置；
    image
    image
    image
  • 更进一步，在 beeline 中，通过命令 “set spark.submit.deployMode” 查看可知，hive on spark 的 deployMode 固定为 cluster，无论 spark-defauls.conf 中 spark.submit.deployMode 配置如何；
  • 更进一步，在 beeline 中，通过命令 “set spark.submit.deployMode=client” 手动指定 hive on spark 的deployMode 为 client ，然后提交 hive on spark 作业，则这些作业会因为 spark jvm 进程启动失败而报错;
    image
    综上所述：
  • hive on spark 模式下，hs2 承载了 spark 客户端的角色，hs2 通过 spark-submit脚本提交 spark 作业；
  • 无论用户如何配置 spark-defaults.conf, hive on spark 固定使用的都是 yarn cluster 模式；
  • 如果用户在 beeline 中，通过命令 “set spark.submit.deployMode=client” 手动指定 hive on spark 的deployMode 为 client 后，则此后提交的 hive on spark 作业会因为 spark jvm 进程启动失败而报错;
  • hs2 中，hive on spark 的两个核心的相关源码类是:org.apache.hive.spark.client.SparkClientImpl/org.apache.hive.spark.client.RemoteDriver;
  • 通过命令 “set spark.submit.deployMode=client” 手动指定 hive on spark 的deployMode 为 client 后， hive on spark 作业 失败的日志如下：
ERROR : FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 256f3dc9-c1a3-49f3-be2c-9ab81a8dd518_1: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error "Error: Could not create the Java Virtual Machine.","Error: A fatal exception has occurred. Program will exit."
INFO  : Completed executing command(queryId=hive_20240620154413_1ad26fe2-f2d5-4252-a609-b3b8d4ce2822); Time taken: 0.517 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 256f3dc9-c1a3-49f3-be2c-9ab81a8dd518_1: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error "Error: Could not create the Java Virtual Machine.","Error: A fatal exception has occurred. Program will exit." (state=42000,code=30041)
  • hs2 动态生成的 spark-default.conf，以及 hs2中启动 hive on spark 作业的相关日志，如下：
# hive on spark 相关配置-/run/cloudera-scm-agent/process/5666-hive-HIVESERVER2/spark-defaults.conf 包含：
spark.master=yarn
spark.submit.deployMode=cluster
spark.authenticate=true
spark.driver.log.dfsDir=/user/spark/driverLogs
spark.driver.log.persistToDfs.enabled=true
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.executorIdleTimeout=60
spark.dynamicAllocation.minExecutors=1
spark.dynamicAllocation.schedulerBacklogTimeout=1
spark.eventLog.enabled=true
spark.io.encryption.enabled=false
spark.network.crypto.enabled=false
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.shuffle.service.enabled=true
spark.shuffle.service.port=7337
spark.ui.enabled=true
spark.ui.killEnabled=true
spark.lineage.log.dir=/var/log/spark/lineage
spark.lineage.enabled=true
spark.eventLog.dir=hdfs://ns1/user/spark/applicationHistory
spark.yarn.historyServer.address=http://uf30-3:18088
spark.yarn.jars=local:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/jars/*,local:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/hive/*
spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/native
spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/native
spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/native
spark.yarn.config.gatewayPath=/opt/cloudera/parcels
spark.yarn.config.replacementPath={{HADOOP_COMMON_HOME}}/../../..
spark.yarn.historyServer.allowTracking=true
spark.yarn.appMasterEnv.MKL_NUM_THREADS=1
spark.executorEnv.MKL_NUM_THREADS=1
spark.yarn.appMasterEnv.OPENBLAS_NUM_THREADS=1
spark.executorEnv.OPENBLAS_NUM_THREADS=1
spark.extraListeners=com.cloudera.spark.lineage.NavigatorAppListener
spark.sql.queryExecutionListeners=com.cloudera.spark.lineage.NavigatorQueryListener

# hs2中，hive on spark 相关日志：
2024-06-20 09:43:30,902 INFO  org.apache.hive.spark.client.SparkClientImpl: [HiveServer2-Background-Pool: Thread-151785]: Loading spark defaults configs from: file:/run/cloudera-scm-agent/process/5666-hive-HIVESERVER2/spark-defaults.conf
2024-06-20 09:43:30,908 INFO  org.apache.hive.spark.client.SparkClientImpl: [HiveServer2-Background-Pool: Thread-151785]: Running client driver with argv: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/bin/spark-submit --executor-cores 4 --executor-memory 2147483648b --principal hive/uf30-1@CDH.COM --keytab hive.keytab --jars /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/hive-kryo-registrator-2.1.1-cdh6.3.2.jar --properties-file /tmp/spark-submit.6442647368541171349.properties --class org.apache.hive.spark.client.RemoteDriver /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/hive-exec-2.1.1-cdh6.3.2.jar --remote-host uf30-1 --remote-port 54208 --remote-driver-conf hive.spark.client.future.timeout=60000 --remote-driver-conf hive.spark.client.connect.timeout=1000 --remote-driver-conf hive.spark.client.server.connect.timeout=900000 --remote-driver-conf hive.spark.client.channel.log.level=null --remote-driver-conf hive.spark.client.rpc.max.size=52428800 --remote-driver-conf hive.spark.client.rpc.threads=8 --remote-driver-conf hive.spark.client.secret.bits=256 --remote-driver-conf hive.spark.client.rpc.server.address=null --remote-driver-conf hive.spark.client.rpc.server.port=null
2024-06-20 09:43:31,887 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: Warning: Ignoring non-spark config property: hive.spark.client.server.connect.timeout=900000
2024-06-20 09:43:31,887 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: Warning: Ignoring non-spark config property: hive.spark.client.rpc.threads=8
2024-06-20 09:43:31,887 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: Warning: Ignoring non-spark config property: hive.spark.client.future.timeout=60000
2024-06-20 09:43:31,887 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: Warning: Ignoring non-spark config property: hive.spark.client.connect.timeout=1000
2024-06-20 09:43:31,887 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: Warning: Ignoring non-spark config property: hive.spark.client.secret.bits=256
2024-06-20 09:43:31,888 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: Warning: Ignoring non-spark config property: hive.spark.client.rpc.max.size=52428800
2024-06-20 09:43:32,059 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:32 WARN spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead.
2024-06-20 09:43:32,059 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:32 WARN spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.driver.memoryOverhead' instead.
2024-06-20 09:43:33,044 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Kerberos credentials: principal = hive/uf30-1@CDH.COM, keytab = hive.keytab
2024-06-20 09:43:33,531 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm71
2024-06-20 09:43:33,585 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
2024-06-20 09:43:33,698 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO conf.Configuration: resource-types.xml not found
2024-06-20 09:43:33,698 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-06-20 09:43:33,719 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (20480 MB per container)
2024-06-20 09:43:33,720 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Will allocate AM container, with 2560 MB memory including 512 MB overhead
2024-06-20 09:43:33,720 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Setting up container launch context for our AM
2024-06-20 09:43:33,724 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Setting up the launch environment for our AM container
2024-06-20 09:43:33,745 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Preparing resources for our AM container
2024-06-20 09:43:33,805 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
2024-06-20 09:43:33,810 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:33 INFO yarn.Client: Uploading resource file:/run/cloudera-scm-agent/process/5666-hive-HIVESERVER2/hive.keytab -> hdfs://ns1/user/hive/.sparkStaging/application_1716544959017_1620/hive.keytab
2024-06-20 09:43:34,091 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/hive-exec-2.1.1-cdh6.3.2.jar -> hdfs://ns1/user/hive/.sparkStaging/application_1716544959017_1620/hive-exec-2.1.1-cdh6.3.2.jar
2024-06-20 09:43:34,434 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO yarn.Client: Uploading resource file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/hive-kryo-registrator-2.1.1-cdh6.3.2.jar -> hdfs://ns1/user/hive/.sparkStaging/application_1716544959017_1620/hive-kryo-registrator-2.1.1-cdh6.3.2.jar
2024-06-20 09:43:34,771 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO yarn.Client: Uploading resource file:/tmp/spark-6592d710-76fe-4804-9e14-6fa37e26747c/__spark_conf__4167604981797776583.zip -> hdfs://ns1/user/hive/.sparkStaging/application_1716544959017_1620/__spark_conf__.zip
2024-06-20 09:43:34,852 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO spark.SecurityManager: Changing view acls to: hive
2024-06-20 09:43:34,853 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO spark.SecurityManager: Changing modify acls to: hive
2024-06-20 09:43:34,854 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO spark.SecurityManager: Changing view acls groups to: 
2024-06-20 09:43:34,855 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO spark.SecurityManager: Changing modify acls groups to: 
2024-06-20 09:43:34,856 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO spark.SecurityManager: SecurityManager: authentication enabled; ui acls disabled; users  with view permissions: Set(hive); groups with view permissions: Set(); users  with modify permissions: Set(hive); groups with modify permissions: Set()
2024-06-20 09:43:34,893 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:34 INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive/hive-site.xml
2024-06-20 09:43:35,013 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO security.YARNHadoopDelegationTokenManager: Attempting to login to KDC using principal: hive/uf30-1@CDH.COM
2024-06-20 09:43:35,017 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO security.YARNHadoopDelegationTokenManager: Successfully logged into KDC.
2024-06-20 09:43:35,028 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-350949835_1, ugi=hive/uf30-1@CDH.COM (auth:KERBEROS)]] with renewer yarn/uf30-1@CDH.COM
2024-06-20 09:43:35,056 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO hdfs.DFSClient: Created token for hive: HDFS_DELEGATION_TOKEN owner=hive/uf30-1@CDH.COM, renewer=yarn, realUser=, issueDate=1718847815048, maxDate=1719452615048, sequenceNumber=489988, masterKeyId=1416 on ha-hdfs:ns1
2024-06-20 09:43:35,059 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-350949835_1, ugi=hive/uf30-1@CDH.COM (auth:KERBEROS)]] with renewer hive/uf30-1@CDH.COM
2024-06-20 09:43:35,060 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO hdfs.DFSClient: Created token for hive: HDFS_DELEGATION_TOKEN owner=hive/uf30-1@CDH.COM, renewer=hive, realUser=, issueDate=1718847815056, maxDate=1719452615056, sequenceNumber=489989, masterKeyId=1416 on ha-hdfs:ns1
2024-06-20 09:43:35,106 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO security.HadoopFSDelegationTokenProvider: Renewal interval is 86400044 for token HDFS_DELEGATION_TOKEN
2024-06-20 09:43:35,168 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 WARN conf.HiveConf: HiveConf of name hive.enforce.bucketing does not exist
2024-06-20 09:43:35,206 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO yarn.Client: Submitting application application_1716544959017_1620 to ResourceManager
2024-06-20 09:43:35,456 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO impl.YarnClientImpl: Submitted application application_1716544959017_1620
2024-06-20 09:43:35,461 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO yarn.Client: Application report for application_1716544959017_1620 (state: ACCEPTED)
2024-06-20 09:43:35,468 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO yarn.Client: 
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      diagnostics: AM container is launched, waiting for AM container to Register with RM
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      ApplicationMaster host: N/A
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      ApplicationMaster RPC port: -1
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      queue: root.users.dap
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      start time: 1718847815223
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      final status: UNDEFINED
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      tracking URL: http://uf30-3:8088/proxy/application_1716544959017_1620/
2024-06-20 09:43:35,469 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]:      user: hive
2024-06-20 09:43:35,474 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO util.ShutdownHookManager: Shutdown hook called
2024-06-20 09:43:35,476 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-c92c59a3-55ce-4aa1-9463-6ed42b4ddd99
2024-06-20 09:43:35,481 INFO  org.apache.hive.spark.client.SparkClientImpl: [spark-submit-stderr-redir-HiveServer2-Background-Pool: Thread-151785]: 24/06/20 09:43:35 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-6592d710-76fe-4804-9e14-6fa37e26747c
2024-06-20 09:43:35,865 INFO  org.apache.hive.spark.client.SparkClientImpl: [Driver]: Child process (spark-submit) exited successfully.
2024-06-20 09:43:40,157 INFO  org.apache.hive.spark.client.SparkClientImpl: [HiveServer2-Background-Pool: Thread-151785]: Successfully connected to Remote Spark Driver at: uf30-1:38642
2024-06-20 09:43:48,053 INFO  org.apache.hive.spark.client.SparkClientImpl: [Spark-Driver-RPC-Handler-0]: Received Spark job ID: 0 for client job c111431c-5a57-4bb4-9357-07d0e5948d79

3 hive on spark 常见问题

  • 问题现象：部分 HIVE ON SPARK 作业报错失败，但重试有时候又能够成功，客户端报错: java.sql.SQLException…failed to create spark client for spark session xxx: java.util.concurrent.TimeoutException: client xxx timed out waiting for connection from the remote spark driver.
  • 问题日志:hiveserver2 日志：“timed out waiting for remote spark driver to connect to hiveserver2. possible reasons include network issues, errors in remote driver, cluster has no available resources etc. please check yarn or spark driver;s logs for further information.”
  • 问题日志:yarn 日志：container exited with a non-zero exit code 13. Error file: prelaunch.err. …java.util.concurrent.ExecutionException:javax.security.sasl.saslException: client closed before SASL negotiation finished.
  • 问题原因：大量 HIVE ON SPARK 作业并发提交时, yarn 中对应队列上没有足够资源启动 spark 集群了，或者 hiveserver2/yarn/kdc 在高并发下的性能问题，造成了超时;
  • 问题解决：业务侧可以减少并发调度的作业数，或者调小申请的spark资源的大小（spark.driver.memory/spark.executor.memory/spark.executor.cores 等）；服务端可以增大对应队列的资源,也可以同步调大服务端超时参数以缓解问题，注意该超时参数 hive.spark.client.server.connect.timeout 需要在服务端更改并重启服务端才能生效，客户端更改时不会报错但是不会生效；
  • 另外如有必要，需要同步更改黑白名单参数：hive.conf.restricted.list/hive.security.authorization.sqlstd.confwhitelist/hive.security.authorization.sqlstd.confwhitelist.append）；
  • 相关参数：
服务端参数 hive.spark.client.server.connect.timeout:默认90秒：Timeout for handshake between Hive client and remote Spark driver.  Checked by both processes.
服务端参数 hive.spark.client.future.timeout: 默认 60秒：Timeout for requests from Hive client to remote Spark driver.
客户端参数 hive.spark.client.connect.timeout:默认一秒：Timeout for remote Spark driver in connecting back to Hive client

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/728628.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

[C++][数据结构][B-树][上]详细讲解

[C++][数据结构][B-树][上]详细讲解

目录 0.常见的搜索结构1.B树概念2.B-树的插入分析1.流程分析2.插入过程总结 0.常见的搜索结构 种类数据格式时间复杂度顺序查找无要求 O ( N ) O(N) O(N)二分查找有序 O ( l o g 2 N ) O(log_2 N) O(log2​N)二叉搜索树无要求 O ( N ) O(N) O(N)二叉平衡树无要求 O ( l o g 2 …
阅读更多...
20212416 2023-2024-2 《移动平台开发与实践》综合实践

20212416 2023-2024-2 《移动平台开发与实践》综合实践

移动平台开放综合实践 1.实验内容2.实验过程2.1 确定基础功能2.2 设计UI界面2.3 编写程序运行代码2.4 在基本功能的基础上丰富功能 3. 代码分析3.1设置按钮的点击事件监听器3.2 比分更新模块3.3 比分存储模块 4. 运行结果5.实践中遇到的问题及解决6.学习感悟与思考参考资料 1.实…
阅读更多...
k8s中 docker和containerd 镜像相互导入导出

k8s中 docker和containerd 镜像相互导入导出

containerd镜像导出并导入docker 1 查看containerd 本地镜像列表 crictl images 2 containerd 导出本地镜像到当前目录下（注意： 导出导入需要指定镜像平台类型 --platform） ctr -n k8s.io images export nacos-server-24-06-30-13-02-…
阅读更多...
【windows|007】DHCP服务详解

【windows|007】DHCP服务详解

🍁博主简介： 🏅云计算领域优质创作者 🏅2022年CSDN新星计划python赛道第一名 🏅2022年CSDN原力计划优质作者 ​ 🏅阿里云ACE认证高级工程师 ​ 🏅阿里云开发者社区专家博主 💊交流社…
阅读更多...
在阿里云服务器Linux系统上从头到尾实现Webapp的部署(安装卸载JDK、安装Tomcat、安装配置MySQL)

在阿里云服务器Linux系统上从头到尾实现Webapp的部署(安装卸载JDK、安装Tomcat、安装配置MySQL)

输入yum list | grep jdk 选择 devel是软件包中的典型命名格式 devel表示这个包是开发工具相关的 里面包含内容是最完整的 x86表示cpu架构是x86_64 还有openjdk表示开源版本 输入yum install java-1.8.0-openjdk-devel.x86_64 开始下载 遇到问你 is this ok? 输入y表示ok 输…
阅读更多...
计算机网络期末复习——简明扼要介绍考点及相关知识

计算机网络期末复习——简明扼要介绍考点及相关知识

期末复习的方法论：一般来说，期末复习都是“理论”结合“实践”。 理论，在于要对期末考点有基本的把握。可以询问老师或者师兄，总之要知道考试的重点在哪里。对照教材，勾画考试重点，删去不重要的琐碎知识点。…
阅读更多...
【机器学习】深度学习赋能：基于 LSTM 的智能日志异常检测

【机器学习】深度学习赋能：基于 LSTM 的智能日志异常检测

目录 1. LSTM 简介 2. 日志序列异常检测概述 3. 数据预处理 3.1 日志解析 3.2 数据清洗 3.3 序列化 3.4 特征提取 示例代码 4. 构建 LSTM 模型 4.1 模型结构 4.2 模型构建示例 5. 训练 LSTM 模型 5.1 数据准备 5.2 模型训练 示例代码 6. 异常检测 6.1 异常分数…
阅读更多...
QT截图程序三-截取自定义多边形

QT截图程序三-截取自定义多边形

上一篇文章QT截图程序，可多屏幕截图二，增加调整截图区域功能-CSDN博客描述了如何截取，具备调整边缘功能后已经方便使用了，但是与系统自带的程序相比，似乎没有什么特别，只能截取矩形区域。 如果可以按照自己…
阅读更多...
中欧科学家论坛暨第六届人工智能与先进制造国际会议（AIAM2024）

中欧科学家论坛暨第六届人工智能与先进制造国际会议（AIAM2024）

会议日期：2024年10月20-21日 会议地点：德国-法兰克福 会议官网：https://www.iaast.cn/meet/home/Bx130JiM 出版检索：EI、Scopus等数据库收录 【会议简介】 “中欧科学家论坛”由德国、法国、荷兰、瑞士、丹麦、意大利、西班牙…
阅读更多...
python爬虫之selenium自动化操作

python爬虫之selenium自动化操作

python爬虫之selenium自动化操作 需求：操作淘宝去掉弹窗广告搜索物品后进入百度回退又前进 selenium模块的基本使用 问题：selenium模块和爬虫之间具有怎样的关联? 1、便捷的获取网站中动态加载的数据 2、便捷实现模拟登录 什么是selenium模块&#x…
阅读更多...
java连接kerberos用户认证

java连接kerberos用户认证

文章目录 一、背景二、代码2.1目录2.2配置文件application.properties2.3pom依赖2.4代码AuthProviderConfig配置类CustomConfigurationByKeytab配置类CustomConfigurationByPassword配置类TestControllerMyCallbackHandlerDummyUserDetailsService实现类LdapTest2Application启…
阅读更多...
数据结构经典面试之数组——C#和C++篇

数据结构经典面试之数组——C#和C++篇

文章目录 1. 数组的基本概念与功能2. C#数组创建数组访问数组元素修改数组元素数组排序 3. C数组创建数组访问数组元素修改数组元素数组排序 4. 数组的实际应用与性能优化5. C#数组示例6. C数组示例总结 数组是编程中常用的数据结构之一，它用于存储一系列相同类型的…
阅读更多...
算法训练营day15--110.平衡二叉树+ 257. 二叉树的所有路径+ 404.左叶子之和+222.完全二叉树的节点个数

算法训练营day15--110.平衡二叉树+ 257. 二叉树的所有路径+ 404.左叶子之和+222.完全二叉树的节点个数

一、110.平衡二叉树 题目链接：https://leetcode.cn/problems/balanced-binary-tree/ 文章讲解：https://programmercarl.com/0110.%E5%B9%B3%E8%A1%A1%E4%BA%8C%E5%8F%89%E6%A0%91.html 视频讲解：https://www.bilibili.com/video/BV1Ug411S7m…
阅读更多...
React中的JSX应该怎么用

React中的JSX应该怎么用

什么是JSX JSX Javascript XML，JSX是一个 JavaScript 的语法扩展。 JSX可以很好地描述 UI 应该呈现出它应有交互的本质形式并且其完全可以和JavaScript融合在一起使用。而且具有 JavaScript 的全部功能。JSX 可以生成 React “元素”。 JSX代码示例： …
阅读更多...
编译原理：语法分析（语法制导翻译）、语义分析（类型检查、中间代码生成）

编译原理：语法分析（语法制导翻译）、语义分析（类型检查、中间代码生成）

编译器在做语法分析的过程中，除了回答程序代码的语法是否合法（LL,LR能否接收）外，还需要完成后续的工作（包括构建语法树、类型检查、中间代码生成、目标代码生成），这些后续工作一般都可以通过语法…
阅读更多...
板凳--------第60章 SOCKET：服务器设计

板凳--------第60章 SOCKET：服务器设计

60.1 迭代型和并发型服务器 1016 1.迭代型： 服务器每次只处理一个客户端，只有当完全处理完一个客户端的请求后才会去处理下一个客户端。只适用于快速处理客户端请求的场景，因为每个客户端都必须等待，直到前面所有的客户端都处理完…
阅读更多...
10年265倍！动态展示全球第一股英伟达10年股价走势

10年265倍！动态展示全球第一股英伟达10年股价走势

英伟达在过去十年的股价走势展示了其在市场上的强劲表现和显著增长。自1999年上市以来，英伟达的股价经历了多次显著的涨幅，并在2024年达到了历史新高。 从2023年6月的数据来看，英伟达的股价为386.54美元/股，市值为9548亿美元。然…
阅读更多...
redis以后台的方式启动

redis以后台的方式启动

文章目录 1、查看redis安装的目录2、Redis以后台的方式启动3、通过客户端连接redis4、连接后，测试与redis的连通性 1、查看redis安装的目录 [rootlocalhost ~]# cd /usr/local/redis/ [rootlocalhost redis]# ll 总用量 112 drwxr-xr-x. 2 root root 150 12月 6…
阅读更多...
Excel中插入的图片在不同电脑上消失的问题及解决方法

Excel中插入的图片在不同电脑上消失的问题及解决方法

在使用Excel时插入图片，然后在不同电脑上打开却发现图片消失并被替换为链接地址，这个问题通常出现于文件中的图片路径没有正确保存或者电脑上缺少相关的图片文件。下面让我们来详细解释这个问题以及可能的解决方法。 ### 问题原因分析1. **相对路径问题…
阅读更多...
数据结构—排序、查找、图论和字符串算法之Java实例

数据结构—排序、查找、图论和字符串算法之Java实例

一：引言 在编程的海洋中，算法是程序员的灵魂之光。它们不仅指引着代码的前进方向，更能解决难题，提升效率。虽然各式各样的算法琳琅满目，但其中有一些却是每位程序员必定会遇到且应当深刻掌握的。本文将带您走进这些至…
阅读更多...
最新文章

Copyright @ 2022~2023