1 部署nfs
# 在k8s的主节点上执行
mkdir -p /appdata/download
cd /appdata/download
git clone https://github.com/nacos-group/nacos-k8s.git
将nacos部署到middleware的命名空间中
kubectl create namespace middleware
cd /appdata/download/nacos-k8s
# 创建角色
kubectl create -f deploy/nfs/rbac.yaml -n middleware
# 创建 ServiceAccount 和部署 NFS-Client Provisioner
kubectl apply -f deploy/nfs/deployment.yaml -n middleware
# 创建 NFS StorageClass并不隶属于任何特定的命名空间
kubectl apply -f deploy/nfs/class.yaml
# 验证NFS部署成功
kubectl get pod -l app=nfs-client-provisioner -n middleware
注意deployment.yaml中要填写自己的nfs服务器地址,不能使用默认的
2 部署mysql
调整mysql-nfs.yaml文件中nfs的地址路径,且需要再10.101.12.95这个nfs上先创建好目录,否则不成功。
cd /appdata/download/nacos-k8s
kubectl apply -f deploy/mysql/mysql-nfs.yaml -n middleware
这个mysql容器的创建不是一下就创建完成的,需要一点时间,估计还有
mysql虽然看不到,但是容器组中是存在的
3 初始化nacos配置信息
https://github.com/alibaba/nacos/blob/master/distribution/conf/mysql-schema.sql
因为上面mysql都看不到,我怎么执行呢?脚本自动创建了。
4 部署nacos
首次还花点时间,因为要下载镜像。
# 默认创建的就是集群环境
kubectl apply -f deploy/nacos/nacos-pvc-nfs.yaml -n middleware
大概用了7分钟。
5 常见问题
5.1 nfs重复创建的一些问题
如果中间出现了下面一些错误,重复执行的话,可以按照下面的操作排查问题
-
rbac重复创建的问题
[root@master1 nacos-k8s]# kubectl create -f deploy/nfs/rbac.yaml
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": clusterroles.rbac.authorization.k8s.io "nfs-client-provisioner-runner" already exists
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": clusterrolebindings.rbac.authorization.k8s.io "run-nfs-client-provisioner" already exists
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": roles.rbac.authorization.k8s.io "leader-locking-nfs-client-provisioner" already exists
Error from server (AlreadyExists): error when creating "deploy/nfs/rbac.yaml": rolebindings.rbac.authorization.k8s.io "leader-locking-nfs-client-provisioner" already exists
因为脚本是幂等的,所以修改脚本后,使用apply而不是create,即可重复执行。
5.2 mysql删除重建
执行kubectl describe pod mysql-72cxs,发现下面存储路径不正确,我已经调整为nacos_prod,看来这个数据也需要删除。
这个时候执行删除pod,
kubectl delete pod mysql-72cxs -n middleware
因为pod会自动重建,系统才会按照新的来。再看挂载是ok的了
但还有异常信息,说明nfs挂载目录,没有/data/k8s_storage/nacos/mysql这个路径,删掉重建就可以了
我的数据名,已经改了,但是数据库还是之前的,看来这个也要删除.
5.3 nacos启动UnknownHostException: jmenv.tbsite.net
安装coredns之后,就没有这个问题了
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serverMemberManager' defined in URL [jar:file:/home/nacos/target/nacos-server.jar!/BOOT-INF/lib/nacos-core-2.0.3.jar!/com/alibaba/nacos/core/cluster/ServerMemberManager.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.core.cluster.ServerMemberManager]: Constructor threw exception; nested exception is ErrCode:500, ErrMsg:jmenv.tbsite.net
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:304)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:285)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1338)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:554)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:514)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:319)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1196)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760)
... 98 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.nacos.core.cluster.ServerMemberManager]: Constructor threw exception; nested exception is ErrCode:500, ErrMsg:jmenv.tbsite.net
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:187)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:300)
... 112 common frames omitted
Caused by: com.alibaba.nacos.api.exception.NacosException: java.net.UnknownHostException: jmenv.tbsite.net
at com.alibaba.nacos.core.cluster.lookup.AddressServerMemberLookup.run(AddressServerMemberLookup.java:150)
at com.alibaba.nacos.core.cluster.lookup.AddressServerMemberLookup.doStart(AddressServerMemberLookup.java:98)
at com.alibaba.nacos.core.cluster.AbstractMemberLookup.start(AbstractMemberLookup.java:53)
at com.alibaba.nacos.core.cluster.ServerMemberManager.initAndStartLookup(ServerMemberManager.java:181)
at com.alibaba.nacos.core.cluster.ServerMemberManager.init(ServerMemberManager.java:161)
at com.alibaba.nacos.core.cluster.ServerMemberManager.<init>(ServerMemberManager.java:142)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:175)
5.4 No DataSource set
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'asyncNotifyService': Unsatisfied dependency expressed through field 'dumpService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'externalDumpService': Invocation of init method failed; nested exception is ErrCode:500, ErrMsg:Nacos Server did not start because dumpservice bean construction failure :
No DataSource set
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:598)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1402)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:591)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:514)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:321)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:319)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:277)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1276)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1196)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:857)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:760)
... 27 common frames omitted
查看mysql的日志,发现nacos还是连接的nacos_devtest,虽然我已经创建好了nacos_prod
开始我以为是nacos权限问题
GRANT ALL PRIVILEGES ON nacos.* TO 'nacos'@'%' IDENTIFIED BY 'nacos';
FLUSH PRIVILEGES;
查看configmap才发现配置有问题。
可以手工更新一下,然后重启nacos就可以了
kubectl edit configmap nacos-cm -n middleware
5.5 应用如何访问
# 通过这个地址可以访问
nacos-headless.middleware.svc.cluster.local:8848