在Centos7.9_2207安装CDH6.3.2

在Centos7.9_2207安装CDH6.3.2

背景

笔者做大数据开发，实时部分一般要用到HBase、Kudu、Redis等组件来保证幂等性，为了方便，还是选用老古董CDH6.3.2【最后的免费版】做一个单节点机器，方便随时挂起。多节点虚拟机由之前的双路E5做Frp后远程调用。

物理机

资源配置

机型：惠普战99锐龙版

CPU：7840HS 8c16t

RAM：2*48G

SSD：1T+4T

很普通的集显工作站。2条DDR5的48G内存条组成96G内存。

部署好CDH常用组件后，目前的资源占用情况大概是这样：

在这里插入图片描述

经过多年的发展，轻薄本用的7840HS已经远远超越了2016年的标压6700hq，如果没有便携或者大内存的需求，上个7945HX，可以和双路E5-2696V3打的有来有回。

安装Centos7.9_2207

下载ISO镜像

https://wiki.centos.org/Download.html
http://isoredirect.centos.org/centos/7/isos/x86_64/
https://mirrors.tuna.tsinghua.edu.cn/centos/7.9.2009/isos/x86_64/

下载CDH安装包：

https://archive.cloudera.com/

资源配置

CPU：4c

RAM：48G

硬盘：300G

因为要搭一个单节点CDH，所有组件会全部在一个虚拟机启动，所以尽可能分配充足的资源。

网络配置

IP：192.168.88.52

子网掩码：255.255.255.0

网关：192.168.88.2

DNS：8.8.8.8

host：bigdata-

在这里插入图片描述

禁用IPV6。

此时：

[root@bigdata-centos ~]# hostname
bigdata-centos
[root@bigdata-centos ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@bigdata-centos ~]#

部署虚拟机

zhiyong
Shyf@2019

正常部署即可。

CPU性能测试

在CentOS上安装UnixBench可以按照以下步骤进行：

  1. 打开终端，使用root用户或具有sudo权限的用户登录。

  2. 确保您的系统已安装了必要的依赖关系。运行以下命令来安装这些依赖关系：

sudo yum install make gcc
mkdir -p /home/bigdata/unixbench
cd /home/bigdata/unixbench
wget https://codeload.github.com/kdlucas/byte-unixbench/zip/refs/tags/v5.1.3
unzip byte-unixbench-5.1.3.zip
cd /home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench
make ./Run

耐心等待跑完后，可以看到跑分情况，能凑合用：

[root@bigdata-centos UnixBench]# ./Run
make all
make[1]: Entering directory `/home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench'
Checking distribution of files
./pgms  exists
./src  exists
./testdir  exists
./tmp  exists
./results  exists
make[1]: Leaving directory `/home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench'
sh: 3dinfo: command not found

   #    #  #    #  #  #    #          #####   ######  #    #   ####   #    #
   #    #  ##   #  #   #  #           #    #  #       ##   #  #    #  #    #
   #    #  # #  #  #    ##            #####   #####   # #  #  #       ######
   #    #  #  # #  #    ##            #    #  #       #  # #  #       #    #
   #    #  #   ##  #   #  #           #    #  #       #   ##  #    #  #    #
    ####   #    #  #  #    #          #####   ######  #    #   ####   #    #

   Version 5.1.3                      Based on the Byte Magazine Unix Benchmark

   Multi-CPU version                  Version 5 revisions by Ian Smith,
                                      Sunnyvale, CA, USA
   January 13, 2011                   johantheghost at yahoo period com


1 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10

1 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10

1 x Execl Throughput  1 2 3

1 x File Copy 1024 bufsize 2000 maxblocks  1 2 3

1 x File Copy 256 bufsize 500 maxblocks  1 2 3

1 x File Copy 4096 bufsize 8000 maxblocks  1 2 3

1 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10

1 x Pipe-based Context Switching  1 2 3 4 5 6 7 8 9 10

1 x Process Creation  1 2 3

1 x System Call Overhead  1 2 3 4 5 6 7 8 9 10

1 x Shell Scripts (1 concurrent)  1 2 3

1 x Shell Scripts (8 concurrent)  1 2 3

4 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10

4 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10

4 x Execl Throughput  1 2 3

4 x File Copy 1024 bufsize 2000 maxblocks  1 2 3

4 x File Copy 256 bufsize 500 maxblocks  1 2 3

4 x File Copy 4096 bufsize 8000 maxblocks  1 2 3

4 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10

4 x Pipe-based Context Switching  1 2 3 4 5 6 7 8 9 10

4 x Process Creation  1 2 3

4 x System Call Overhead  1 2 3 4 5 6 7 8 9 10

4 x Shell Scripts (1 concurrent)  1 2 3

4 x Shell Scripts (8 concurrent)  1 2 3

========================================================================
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: bigdata-centos: GNU/Linux
   OS: GNU/Linux -- 3.10.0-1160.71.1.el7.x86_64 -- #1 SMP Tue Jun 28 15:37:28 UTC 2022
   Machine: x86_64 (x86_64)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 1: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 2: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 3: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:36:48 up 43 min,  3 users,  load average: 0.00, 0.01, 0.05; runlevel 5

------------------------------------------------------------------------
Benchmark Run: 三 10月 18 2023 01:36:48 - 02:05:51
4 CPUs in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       45522028.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3556.8 MWIPS (15.8 s, 7 samples)
Execl Throughput                               3186.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        792934.6 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          215891.9 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2162307.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1915569.4 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  20637.0 lps   (10.0 s, 7 samples)
Process Creation                               8614.4 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7681.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   2594.1 lpm   (60.0 s, 2 samples)
System Call Overhead                        2884075.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   45522028.9   3900.8
Double-Precision Whetstone                       55.0       3556.8    646.7
Execl Throughput                                 43.0       3186.7    741.1
File Copy 1024 bufsize 2000 maxblocks          3960.0     792934.6   2002.4
File Copy 256 bufsize 500 maxblocks            1655.0     215891.9   1304.5
File Copy 4096 bufsize 8000 maxblocks          5800.0    2162307.5   3728.1
Pipe Throughput                               12440.0    1915569.4   1539.8
Pipe-based Context Switching                   4000.0      20637.0     51.6
Process Creation                                126.0       8614.4    683.7
Shell Scripts (1 concurrent)                     42.4       7681.2   1811.6
Shell Scripts (8 concurrent)                      6.0       2594.1   4323.4
System Call Overhead                          15000.0    2884075.7   1922.7
                                                                   ========
System Benchmarks Index Score                                        1252.4

------------------------------------------------------------------------
Benchmark Run: 三 10月 18 2023 02:05:51 - 02:34:44
4 CPUs in system; running 4 parallel copies of tests

Dhrystone 2 using register variables      151547416.3 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                    14886.9 MWIPS (13.9 s, 7 samples)
Execl Throughput                              15018.6 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        829648.1 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          217171.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2729918.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                             6317809.1 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 846011.1 lps   (10.0 s, 7 samples)
Process Creation                              54076.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                  20820.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   3192.6 lpm   (60.1 s, 2 samples)
System Call Overhead                        8667731.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0  151547416.3  12986.1
Double-Precision Whetstone                       55.0      14886.9   2706.7
Execl Throughput                                 43.0      15018.6   3492.7
File Copy 1024 bufsize 2000 maxblocks          3960.0     829648.1   2095.1
File Copy 256 bufsize 500 maxblocks            1655.0     217171.6   1312.2
File Copy 4096 bufsize 8000 maxblocks          5800.0    2729918.6   4706.8
Pipe Throughput                               12440.0    6317809.1   5078.6
Pipe-based Context Switching                   4000.0     846011.1   2115.0
Process Creation                                126.0      54076.5   4291.8
Shell Scripts (1 concurrent)                     42.4      20820.9   4910.6
Shell Scripts (8 concurrent)                      6.0       3192.6   5321.0
System Call Overhead                          15000.0    8667731.7   5778.5
                                                                   ========
System Benchmarks Index Score                                        3863.4

[root@bigdata-centos UnixBench]#

准备工作

配置SSH免密

要配置CentOS上的免密登录并将密钥发送到其他机器，可以按照以下步骤进行：

  1. 在bigdata-centos机器上生成SSH密钥对。打开终端并运行以下命令：
[root@bigdata-centos ~]# ssh-keygen -t rsa	#按3次回车
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:MLaVLFiyOmRjJ3awp9Kb3kh2Y53wSf8P3hUojvZyeho root@bigdata-centos
The key's randomart image is:
+---[RSA 2048]----+
|  . . .          |
|   o = . .       |
|  O * = +        |
| * O . *     .   |
|. = . o S . . .  |
| . + = + o .   . |
|  = + = E o   .  |
| + = . ..=oo .   |
|  o .   o*+.o    |
+----[SHA256]-----+
[root@bigdata-centos ~]# ssh-copy-id root@bigdata-centos
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'bigdata-centos (192.168.88.52)' can't be established.
ECDSA key fingerprint is SHA256:jO6jVOMifwO+jzliWqHgSF2LU39q4UtooXWKEWsVVgc.
ECDSA key fingerprint is MD5:78:5d:df:f1:53:79:f3:73:de:ac:9d:be:dc:c4:3c:40.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@bigdata-centos's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@bigdata-centos'"
and check to make sure that only the key(s) you wanted were added.

[root@bigdata-centos ~]# ssh bigdata-centos
Last login: Wed Oct 18 02:09:26 2023 from 192.168.88.1
[root@bigdata-centos ~]# pwd
/root
[root@bigdata-centos ~]# exit
登出
Connection to bigdata-centos closed.
[root@bigdata-centos ~]# pwd
/root
[root@bigdata-centos ~]#

此时SSH免密配置成功。

配置NTP时钟同步

为了使用Kudu，一定要配置好NTP。

[root@bigdata-centos ~]# yum install -y ntp
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
软件包 ntp-4.2.6p5-29.el7.centos.2.x86_64 已安装并且是最新版本
无须任何处理
[root@bigdata-centos ~]# vim /etc/ntp.conf
[root@bigdata-centos ~]# cat /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
#restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

restrict 192.168.88.0 mask 255.255.255.0 nomodify notrap
#server ntp4.aliyun.com
#server 202.112.10.36
server 127.127.1.0 iburst# localclock
fudge 127.127.1.0 stratum 8

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# hwclock --systohc --localtime
[root@bigdata-centos ~]# timedatectl
      Local time: 四 2023-10-19 15:45:54 CST
  Universal time: 四 2023-10-19 07:45:54 UTC
        RTC time: 四 2023-10-19 07:45:54
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: no
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@bigdata-centos ~]# vim /etc/ntp.conf
您在 /var/spool/mail/root 中有新邮件
[root@bigdata-centos ~]# vim /etc/ntp.conf
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# timedatectl
      Local time: 四 2023-10-19 16:09:23 CST
  Universal time: 四 2023-10-19 08:09:23 UTC
        RTC time: 四 2023-10-19 08:09:23
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@bigdata-centos kudu]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*LOCAL(0)        .LOCL.           8 l    2   64    1    0.000    0.000   0.000
[root@bigdata-centos kudu]# rm -rf /home/kududata

主要是Kudu的时钟比较麻烦，所以单节点可以只配置和本机同步。

配置JDK

[root@bigdata-centos ~]# java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode)
[root@bigdata-centos ~]#

这个版本的Centos7.9本身自带了JDK1.8，可以直接使用。

关闭透明大页

[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
[root@bigdata-centos ~]# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@bigdata-centos ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
[root@bigdata-centos ~]#

显然当前版本的Centos开启了透明大页功能，需要关闭：

[root@bigdata-centos UnixBench]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@bigdata-centos UnixBench]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@bigdata-centos UnixBench]# vim /etc/default/grub
[root@bigdata-centos UnixBench]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

关闭后还需要reboot重启生效。

减少swap

[root@bigdata-centos ~]# sysctl -w vm.swappiness=0
vm.swappiness = 0
[root@bigdata-centos ~]# echo 'vm.swappiness=0' >> /etc/sysctl.conf
[root@bigdata-centos ~]# sysctl -p
vm.swappiness = 0
[root@bigdata-centos ~]#

0表示只有当可⽤物理内存⼩于最⼩阈值vm.min_free_kbytes时才使⽤交换空间。

关闭安全配置

[root@bigdata-centos yum.repos.d]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@bigdata-centos yum.repos.d]# systemctl stop firewalld
[root@bigdata-centos yum.repos.d]# vim /etc/selinux/config
[root@bigdata-centos yum.repos.d]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disable
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@bigdata-centos yum.repos.d]#

此时关闭了SELinux与防火墙。

安装Http与Repo

yum install -y httpd
yum install -y createrepo
systemctl start httpd
systemctl enable httpd

# Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

此时安装好Http与Repo。浏览器打开：

192.168.88.52:80

看到：

在这里插入图片描述

说明Http已经启动。

制作离线Yum源

[root@bigdata-centos yum.repos.d]# mkdir -p /home/bigdata/yum/offline/zhiyong-bigdata-centos
[root@bigdata-centos yum.repos.d]# vi /etc/yum.repos.d/zhiyong-bigdata-centos.repo
[root@bigdata-centos yum.repos.d]# cat /etc/yum.repos.d/zhiyong-bigdata-centos.repo
[offline]
name=zhiyong-bigdata-centos
baseurl=http://192.168.88.52:80/yum/	# file:///home/bigdata/yum/offline/zhiyong-bigdata-centos/
enabled=1
gpgcheck=0
[root@bigdata-centos repodata]# cd /home/bigdata/yum/offline/zhiyong-bigdata-centos
[root@bigdata-centos zhiyong-bigdata-centos]# ll
总用量 3599392
-rw-r--r--. 1 root root     277604 10月 18 03:40 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root    2237116 10月 18 03:40 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root    2112700 10月 18 03:40 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root   25034716 10月 18 03:40 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root  159295840 10月 18 03:40 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root      14040 10月 18 03:10 allkeys.asc
-rw-r--r--. 1 root root 2082186246 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r--. 1 root root         40 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
-rw-r--r--. 1 root root         64 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
-rw-r--r--. 1 root root   10483568 10月 18 03:10 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root 1203832464 10月 18 03:10 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      11488 10月 18 03:10 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      10996 10月 18 03:10 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root   14209868 10月 18 03:10 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      33887 10月 18 03:10 manifest.json
-rw-r--r--. 1 root root    1006956 10月 18 03:40 mysql-connector-java-5.1.48.jar
-rw-r--r--. 1 root root  184988341 10月 18 03:10 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
drwxr-xr-x. 2 root root       4096 10月 18 03:40 repodata
[root@bigdata-centos zhiyong-bigdata-centos]# createrepo .
[root@bigdata-centos zhiyong-bigdata-centos]# yum clean all
[root@bigdata-centos zhiyong-bigdata-centos]# yum makecache
[root@bigdata-centos zhiyong-bigdata-centos]# mkdir -p /var/www/html/yum
[root@bigdata-centos zhiyong-bigdata-centos]# cp /home/bigdata/yum/offline/zhiyong-bigdata-centos/* /var/www/html/yum

将离线的repo文件使用MobaXterm放置到/home/bigdata/yum/offline/zhiyong-bigdata-centos路径。此时：

http://192.168.88.52/yum/

在这里插入图片描述

已经可以看到这些文件。此时可以离线部署【由于Cloudera关闭了官网的下载链接，目前也只能离线部署】。

放置parcel包

/opt/cloudera/parcel-repo
/opt/cloudera/parcels

将CDH组件的4个包在该路径也放置一份。

安装MySQL

[root@bigdata-centos zhiyong-bigdata-centos]# pwd
/home/bigdata/yum/offline/zhiyong-bigdata-centos

rpm -qa|grep mariadb | xargs -I {} rpm -e --nodeps {}
rpm -ivh 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
rpm -ivh 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
rpm -ivh 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
rpm -ivh 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
rpm -ivh 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm

systemctl start mysqld
systemctl enable mysqld

[root@bigdata-centos zhiyong-bigdata-centos]# grep 'temporary password' /var/log/mysqld.log
2023-10-17T19:46:44.706162Z 1 [Note] A temporary password is generated for root@localhost: #DR2-C#qp1wK
[root@bigdata-centos zhiyong-bigdata-centos]# mysql -uroot -p
#DR2-C#qp1wK
alter user root@localhost identified by 'Zhiyong123#';
flush privileges;

[root@bigdata-centos yum]# pwd
/var/www/html/yum
[root@bigdata-centos yum]# cp mysql-connector-java-5.1.48.jar /usr/share/java/
[root@bigdata-centos java]# pwd
/usr/share/java
[root@bigdata-centos java]# mv mysql-connector-java-5.1.48.jar mysql-connector-java.jar

此时MySQL的密码已经修改。

创建元数据库表

mysql -uroot -pZhiyong123# -e 'create database metastore default character set utf8;'
mysql -uroot -pZhiyong123# -e "create user 'hive'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database hive default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on hive.* to 'hive'@'%'";
mysql -uroot -pZhiyong123# -e "grant all privileges on metastore.* to 'hive'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'amon'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database amon default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on amon.* to 'amon'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'rman'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database rman default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on rman.* to 'rman'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'sentry'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database sentry default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on sentry.* to 'sentry'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'nav'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database nav default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on nav.* to 'nav'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'navms'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database navms default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on navms.* to 'navms'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'cm'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database cm default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on cm.* to 'cm'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'oos'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database oos default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on oos.* to 'oos'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'hue'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database hue default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on hue.* to 'hue'@'%'";

创建成功。此处建库根据需求。

安装CDH

安装CM

yum install cloudera-manager-daemons cloudera-manager-server

初始化CM

[root@bigdata-centos java]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm Zhiyong123#
JAVA_HOME=/usr/lib/jvm/jre-openjdk
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/lib/jvm/jre-openjdk/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Wed Oct 18 04:53:43 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

此时初始化CM的元数据成功。要注意密码不能有!，可以有#。血的教训。

启动CM

systemctl start cloudera-scm-server.service

稍等片刻即可进入熟悉的CM界面。

安装组件

http://192.168.88.52:7180/cmf/login
admin
admin

在这里插入图片描述

在这里插入图片描述

当然是选择体验版。。。

在这里插入图片描述

未在已配置的存储库中找到任何 parcel。尝试在更多选项下添加一个自定义存储库。否则，您可能只能继续使用包。

还需要：

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
systemctl restart cloudera-scm-server

在这里插入图片描述

重置Hash以后才可以下一步。

在这里插入图片描述

出现无法复制安装文件的情况。。。要取消掉存储库的所有配置项，只保留本地源，这样才能保证不会从已过时的Cloudera官网下载安装包。不然会出现：

/tmp/scm_prepare_node.k8fwJHf3
https://archive.cloudera.com/cm6/6.3.1/allkeys.asc/

类似这种的报错。

此外，由于是单节点，本身是不合理的，所以后续所有检查、告警能跳过的一律跳过。

在这里插入图片描述

在这里插入图片描述

原则就是能凑合着用即可。

后续操作

允许远程登陆MySQL

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Zhiyong123#' WITH GRANT OPTION;
FLUSH PRIVILEGES;

这样就可以远程登陆MySQL，方便后续查看Hive的元数据。

配置Jdbc连接

在这里插入图片描述

找一个Impala的JDBC41驱动包，传入url：

jdbc:impala://192.168.88.52:21050

后续就可以用这种方式访问Impala，间接操作Kudu和Hive，从而忽视Hue不正常的问题。

调度工具Oozie已经要淘汰了，开发环境也没有研究的必要，直接停掉。

FRP端口暴露

使用路由侠，暴露Impala的端口给外部访问。这样就可以让别的只会写Sql的Sql Boy用公网的方式连接到Impala：

jdbc:impala://zhiyong.xxxx这里是具体的网址xxxxxxx.luyouxia.net:11701
jdbc:impala://zhiyong.e3.luyouxia.net:11701

这种方式比租一个ECS要便宜些。延时90ms左右。

此时完成了CDH6.3.2单节点部署的主要操作。

Kudu无法启动问题

出现Kudu无法启动的问题，reboot重置NTP：

[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# ntpstat
synchronised to local net (127.127.1.0) at stratum 11
   time correct to within 7948 ms
   polling server every 64 s

之后：

gflagfile 的 Tablet Server 高级配置代码段（安全阀）	--max_clock_sync_error_usec=20000000
gflagfile 的 Master 高级配置代码段（安全阀）	--max_clock_sync_error_usec=20000000
block_cache_capacity_mb=1G
memory_limit_hard_bytes=8G

即可解决如下报错：

E1019 19:13:24.295385 15847 block_cache.cc:97] Block cache capacity exceeds the memory pressure threshold (8589934592 bytes vs. 5153960755 bytes). This will cause instability and harmful flushing behavior. Lower --block_cache_capacity_mb or raise --memory_limit_hard_bytes.

CM的报错在如下Log查看：

/var/log/cloudera-scm-server

此时可以正常使用。

也可以在Idea中使用JDBC方式访问：

package com.zhiyong.kudu;

import java.sql.*;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.Map;

public class KuduImpalaJdbcDemo {
    public static void main(String[] args) throws ClassNotFoundException, SQLException {
        System.out.println("使用Impala操作Kudu");
        Class.forName("com.cloudera.impala.jdbc41.Driver");
        String url = "jdbc://192.168.88.52:21050";

        Connection connection = DriverManager.getConnection(url);

        ArrayList<Map> result = new ArrayList<>();

        Statement statement = connection.createStatement();
        ResultSet resultSet;
        ResultSetMetaData metaData;
        String sql = "select * from table_name ";

        try {

            resultSet = statement.executeQuery(sql);//结果集
            metaData = resultSet.getMetaData();//元数据集

            while (resultSet.next()){
                LinkedHashMap<Object, Object> map = new LinkedHashMap<>();
                for (int i = 0; i < metaData.getColumnCount(); i++) {
                    map.put(metaData.getColumnName(i+1),resultSet.getObject(metaData.getColumnName(i+1)));
                }
                result.add(map);
            }
        } catch (SQLException e) {
            e.printStackTrace();
        } finally {
            try {
                statement.close();
                connection.close();

            } catch (Exception e){
                e.printStackTrace();
            }
        }

        System.out.println("结束");

    }
}

基本搭建完成。

转载请注明出处：https://lizhiyong.blog.csdn.net/article/details/134433148

在这里插入图片描述

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/151544.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

【UE5】显示或隐藏物体轮廓线

【UE5】显示或隐藏物体轮廓线

效果 步骤 1. 先下载所需的材质文件“M_Highlight.uasset” 材质下载链接&#xff1a; 链接&#xff1a;https://pan.baidu.com/s/1rxmRhkUoXVq6-DkIKyBhAQ 提取码&#xff1a;55bv 2. 在视口中拖入后期处理体积 根据需求设置后期处理体积的大小或者直接设置无限范围&…
阅读更多...
Lobatto Quadrature-数值积分

Lobatto Quadrature-数值积分

See https://mathworld.wolfram.com/LobattoQuadrature.html
阅读更多...
053-第三代软件开发-元对象系统

053-第三代软件开发-元对象系统

第三代软件开发-元对象系统 文章目录 第三代软件开发-元对象系统项目介绍元对象系统Qt的元对象系统如何实现对象的属性系统&#xff1f;请问Qt的元对象系统还有哪些功能&#xff1f;Qt的元对象系统支持哪些类型的事件&#xff1f;元对象系统中的属性可以用于实现哪些功能&#…
阅读更多...
MySQL 分库分表与 TiDB（平凯数据库），如何选择？

MySQL 分库分表与 TiDB（平凯数据库），如何选择？

随着互联网行业的飞速发展&#xff0c;数据量不断增长&#xff0c;传统的关系型数据库已经无法满足大规模数据处理的需求。为了解决这一问题&#xff0c;分库分表和分布式数据库应运而生。本文将对比分析 MySQL 分库分表和 TiDB 这两种解决方案&#xff0c;帮助大家更好地选择适…
阅读更多...
栈的三道oj【C++】

栈的三道oj【C++】

栈和队列的相关oj 最小栈思路解决代码 栈的压入弹出序列思路解决代码 逆波兰表达式思路&#xff1a;解决代码 这里就挑了三道题用来熟悉栈 最小栈 力扣链接 咱们已经是高贵的C使用者了&#xff0c;不用像C语言一样从头开始造轮子了 这里我们调用了stack后&#xff0c;就会发…
阅读更多...
一个22届被裁前端思想上得转变

一个22届被裁前端思想上得转变

距离上篇文章已经过去了三个多月&#xff0c;这个三个月&#xff0c;经历了技术攻坚&#xff0c;然后裁员&#xff0c;退房&#xff0c;回老家&#xff0c;找工作。短短的几个月&#xff0c;就经历社会的一次次毒打&#xff0c;特别是找工作&#xff0c;虽然算上实习我也有两年…
阅读更多...
Since Maven 3.8.1 http repositories are blocked

Since Maven 3.8.1 http repositories are blocked

原因 高版本的maven不支持http的存储库。 解决方案 其实方法有好几种&#xff0c;比如降级maven版本至3.6.3(之前一直用的都是这个版本)&#xff0c;我选择了一种比较快(但不一定安全)的方式&#xff0c;因为3.6.3版本被我卸载了&#xff0c;这里直接修改idea的setting配置&…
阅读更多...
算法分析与设计考前冲刺 阅读

算法分析与设计考前冲刺 阅读

拜读我胡哥的精品复习资料 acmack 胡哥发表重要讲话&#xff0c;强调算法的重要性&#xff0c;我等深受触动。 Map&#xff1a;底层是红黑树&#xff0c;按照key自动进行排序 list&#xff1a; 线性链表 我一直单纯的觉得list是列表&#xff0c;这不仅说明了胡哥与我的技术…
阅读更多...
（十一）Flask模板引擎jinja2

（十一）Flask模板引擎jinja2

模板引擎Jinja2 一、简介及基本使用&#xff1a; Flask使用Jinja2作为默认的模板引擎。Jinja2是一个功能强大且易于使用的模板引擎&#xff0c;它允许我们在HTML中嵌入Python代码&#xff0c;并通过将模板和数据进行渲染来生成动态内容。 实战之在Flask中使用Jinja2模板引擎…
阅读更多...
Python | 机器学习之逻辑回归

Python | 机器学习之逻辑回归

​&#x1f308;个人主页&#xff1a;Sarapines Programmer&#x1f525; 系列专栏&#xff1a;《人工智能奇遇记》&#x1f516;少年有梦不应止于心动&#xff0c;更要付诸行动。 目录结构 1. 机器学习之逻辑回归概念 1.1 机器学习 1.2 逻辑回归 2. 逻辑回归 2.1 实验目的…
阅读更多...
不可错过的10本架构师必读书籍，带你嗨翻架构师之路，三连评论送书！

不可错过的10本架构师必读书籍，带你嗨翻架构师之路，三连评论送书！

书籍目录 一&#xff1a;书前开胃菜 二&#xff1a;高并发架构实战 三&#xff1a;架构师的自我修炼 四&#xff1a;中台架构与实现 五&#xff1a;分布式系统架构 六&#xff1a;流程自动化实战 七&#xff1a;分布式系统架构与开发 八&#xff1a;服务端开发 九&am…
阅读更多...
代码随想录图论|130. 被围绕的区域 417太平洋大西洋水流问题

代码随想录图论|130. 被围绕的区域 417太平洋大西洋水流问题

130. 被围绕的区域 **题目&#xff1a;**给你一个 m x n 的矩阵 board &#xff0c;由若干字符 ‘X’ 和 ‘O’ &#xff0c;找到所有被 ‘X’ 围绕的区域&#xff0c;并将这些区域里所有的 ‘O’ 用 ‘X’ 填充。 题目链接&#xff1a;130. 被围绕的区域 解题思路&#xff1a…
阅读更多...
基于springboot实现“漫画之家”系统项目【项目源码+论文说明】

基于springboot实现“漫画之家”系统项目【项目源码+论文说明】

基于springboot实现“漫画之家”系统演示 摘要 随着信息技术和网络技术的飞速发展&#xff0c;人类已进入全新信息化时代&#xff0c;传统管理技术已无法高效&#xff0c;便捷地管理信息。为了迎合时代需求&#xff0c;优化管理效率&#xff0c;各种各样的管理系统应运而生&am…
阅读更多...
FreeSWITCH案例跟踪之一，sip bye发不出去

FreeSWITCH案例跟踪之一，sip bye发不出去

报故障的说&#xff0c;网关呼叫fs&#xff0c;网关收不到fs的sip bye Wireshark看call-flow, 是这样的&#xff1a; INVITE里面的contact是<sip:172.23.4.109:5060;transporttcp> 于是Wireshark设置过滤条件为ip.addr 172.23.4.109 and tcp.port 5060 fs tcp连网关被…
阅读更多...
提高生存能力的7个关键技巧！

提高生存能力的7个关键技巧！

作为一款备受热议和玩家喜爱的多人在线射击游戏&#xff0c;《绝地求生》中生存能力的提高是取得胜利的关键。在这篇实用干货分享中&#xff0c;我们将详细说明7个关键技巧&#xff0c;帮助你在游戏中提高生存能力&#xff0c;获得更多胜利。 1.选择降落点&#xff1a;选择适合…
阅读更多...
make和makefile

make和makefile

一、认识make和Makefile 1、会不会写makefile&#xff0c;从一个侧面说明了一个人是否具备完成大型工程的能力 2、一个工程中的源文件不计数&#xff0c;其按类型、功能、模块分别放在若干个目录中&#xff0c;makefile定义了一系列的规则来指定&#xff0c;哪些文件需要先编译…
阅读更多...
java版直播商城平台规划及常见的营销模式 电商源码/小程序/三级分销+商城免费搭建

java版直播商城平台规划及常见的营销模式 电商源码/小程序/三级分销+商城免费搭建

1. 涉及平台 平台管理、商家端&#xff08;PC端、手机端&#xff09;、买家平台&#xff08;H5/公众号、小程序、APP端&#xff08;IOS/Android&#xff09;、微服务平台&#xff08;业务服务&#xff09; 2. 核心架构 Spring Cloud、Spring Boot、Mybatis、Redis 3. 前端框架…
阅读更多...
Windows10关闭系统自动更新

Windows10关闭系统自动更新

1.背景 2.步骤 第一步: 第二步: 完美
阅读更多...
笔记本电脑没有声音？几招恢复声音流畅！

笔记本电脑没有声音？几招恢复声音流畅！

笔记本电脑已经成为我们日常生活和工作的重要工具&#xff0c;而其中的声音是其功能之一。然而&#xff0c;有时您可能会遇到笔记本电脑没有声音的问题&#xff0c;这可能是由多种原因引起的。在本文中&#xff0c;我们将深入探讨笔记本电脑没有声音的常见原因&#xff0c;并提…
阅读更多...
jbase实现通用码表

jbase实现通用码表

没有通用码表的体系是不完美的&#xff0c;当年我用C#能实现的通用码表&#xff0c;现在在java一样的实现了&#xff0c;通用码表对提高开发效率和降低开发成本的作用巨大&#xff0c;开发可以专注写业务&#xff0c;而不必被太多的维护界面束缚。进而体现在产品竞争力上面&…
阅读更多...
最新文章

Copyright @ 2022~2023