大纲图
流量方向
如上图, 当用户or 别的service 从k8s 集群外部访问 集群内的services 流量方向有两种
一种是垂直方向, 通过域名 -> Load Balancer -> gateway -> services , 在k8s 一般是通过ingress 来实现, 而ingress 不是本文的范围内
另一种是横向流量, 因为每个service 基本都在不同的pods, 甚至是不同的nodes, 它们如何互相沟通, 这就是k8s service 的负责范围.
Service 的基本定义
严格来说,Kubernetes中的Service仅仅包括ClusterIP、NodePort和LoadBalancer这三种类型。
ClusterIP:
是默认的Service类型,为 Pod集合分配一个仅集群内部可达的虚拟IP地址。只能从集群内访问。
通常用来定义内部微服务。
NodePort:
除了分配 ClusterIP 外,还会在每一个 Node 上享有一个高端口号暴露服务。
可以通过 : 从集群外访问服务。但不经常使用,局限性较大。
LoadBalancer:
使用云平台本身的负载均衡功能,会在平台上创建一个负载均衡器。
外部流量进入负载均衡器,会转发到集群中能处理请求的节点。
这种类型的Service需要云平台原生支持。
所以总体来说,ClusterIP用于内部服务发现和通信,NodePort和LoadBalancer可以在某种程度上从外部访问服务,但不如Ingress功能全面。
NodePort 基本介绍
NodePort类型的Service是Kubernetes中最常见的一种服务类型。让我来详细介绍一下NodePort Service:
NodePort Service会在每个NodeNode上随机分配一个端口(默认范围30000-32767),通过这个端口就可以从集群外访问Service。
与ClusterIP不同,NodePort Service不仅分配一个内部ClusterIP,还会将服务暴露到每个Node的同一端口上。
客户端可以通过任何一个Node的IP和端口进行访问,例如:NodeIP:NodePort。
NodePort使得外部应用能够找到服务,但不提供负载均衡。流量直接到达Node上服务的首个Pod。
这种类型适用于测试和非生产环境,在生产环境一般不推荐直接使用。
它可以作为集群负载测试的简单方式,也可用于开发和调试服务。
和LoadBalancer相比,NodePort不需要外部负载均衡器支持,在任何云环境下都可以使用。
所以总结来说,NodePort Service通过在集群中的每个节点上映射一个端口,来暴露服务至外部,实现了较为简单的外部访问能力。
即使NodePort 更常用与测试环境, 但是学习nodePort 对k8s 可以加深对k8s 内部网络访问规则的理解
NodePort 的个例子
构造1个接口用于return hostname信息
/actuator/info
@Configuration
@Slf4j
public class MainConfigs {
@Bean("hostname")
public String hostname() {
return getHostName();
}
private String getHostName() {
log.info("MainConfigs: getHostName ...");
String hostName = "unknown";
try {
hostName = java.net.InetAddress.getLocalHost().getHostName();
} catch (Exception e) {
log.error("Error in getting hostname...", e);
}
return hostName;
}
}
@Component
@Slf4j
public class AppVersionInfo implements InfoContributor {
@Value("${pom.version}") // https://stackoverflow.com/questions/3697449/retrieve-version-from-maven-pom-xml-in-code
private String appVersion;
@Autowired
private String hostname;
@Override
public void contribute(Info.Builder builder) {
log.info("AppVersionInfo: contribute ...");
builder.withDetail("app", "Sales API")
.withDetail("version", appVersion)
.withDetail("hostname",hostname)
.withDetail("description", "This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP.");
}
}
编写yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels: # label of this deployment
app: bq-api-service # custom defined
author: Jason
name: bq-api-service-deploy-sample
namespace: default
spec:
replicas: 3 # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.
revisionHistoryLimit: 10
selector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error
# error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..
matchLabels:
app: bq-api-service
strategy: # Strategy of upodate
type: RollingUpdate # RollingUpdate or Recreate
rollingUpdate:
maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the update
maxUnavailable: 25% # The maximum number of Pods that can be unavailable during the update
template: # Pod template
metadata:
labels:
app: bq-api-service # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`
spec:
containers:
- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.4 # image of the container
imagePullPolicy: IfNotPresent
name: bq-api-service-container
restartPolicy: Always # Restart policy for all containers within the Pod
terminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully
---
apiVersion: v1 # api version can be v1 or apps/v1
kind: Service
metadata:
name: bq-api-service-1 # name of the service
labels:
app: bq-api-service # label of the service
spec:
selector: # Label of the Pod that the Service is selecting
app: bq-api-service
ports:
- port: 32111 # port of the service
targetPort: 8080 # port of the Pod
name: 8080-port # name of the port
type: NodePort # type of the service, NodePort, ClusterIP, LoadBalancer
在这个yaml 中
我们构建了1个deployment, 包括3个pod 副本
还有1个nodePort service, 定义了service pod 是 32111 (随便定义, 一般跟target port一样), target port 是 8080 (必须跟docker file 暴露的端口一至)
执行yaml
正常用kubectl 命令执行
root@k8s-master:~/k8s-s/services# kubectl apply -f bq-api-service-sample.yaml
deployment.apps/bq-api-service-deploy-sample created
service/bq-api-service-1 created
查看resource
先看 nodes
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS
k8s-master Ready control-plane,master 43d v1.23.6 192.168.0.3 <none> Debian GNU/Linux 11 (bullseye) 5.10.0-28-cloud-amd64 docker://25.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
k8s-node0 Ready <none> 43d v1.23.6 192.168.0.6 <none> Debian GNU/Linux 11 (bullseye) 5.10.0-28-cloud-amd64 docker://25.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node0,kubernetes.io/os=linux
k8s-node1 Ready <none> 43d v1.23.6 192.168.0.44 <none> Debian GNU/Linux 11 (bullseye) 5.10.0-28-cloud-amd64 docker://25.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node1,kubernetes.io/os=linux
k8s-node3 Ready <none> 23d v1.23.6 192.168.0.45 <none> Debian GNU/Linux 11 (bullseye) 5.10.0-28-cloud-amd64 docker://25.0.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node3,kubernetes.io/os=linux
可以查看各个node的ip 我们有1个master 和3 台nodes
再看pods
root@k8s-master:~/k8s-s/services# kubectl get pods -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
bq-api-service-deploy-sample-f448cf6ff-c8zlp 1/1 Running 0 6m47s 10.244.2.104 k8s-node0 <none> <none> app=bq-api-service,pod-template-hash=f448cf6ff
bq-api-service-deploy-sample-f448cf6ff-tkxck 1/1 Running 0 6m47s 10.244.1.45 k8s-node1 <none> <none> app=bq-api-service,pod-template-hash=f448cf6ff
bq-api-service-deploy-sample-f448cf6ff-vvkt9 1/1 Running 0 6m47s 10.244.3.54 k8s-node3 <none> <none> app=bq-api-service,pod-template-hash=f448cf6ff
root@k8s-master:~/k8s-s/services#
可以见3个pods 被create, 分布在3个不同的nodes node0, node1 和 node3
再看service
root@k8s-master:~/k8s-s/services# kubectl get svc -o wide --show-labels
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR LABELS
bq-api-service-1 NodePort 10.105.158.88 <none> 32111:31372/TCP 7m58s app=bq-api-service app=bq-api-service
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21h <none> component=apiserver,provider=kubernetes
可见1个名字是 bq-api-service-1的NodePort service 被创建, 在PORT(S) 属性里 的值是 32111:31372
其中这个32111是service 本身的port, 而31372 是1个随机生成的port, 可以让k8s外部通过 node ip:31372 去访问这个service
而kubectl get svc 这个命令是不能显示target port 信息
要查看taget port则要查看yaml 信息
root@k8s-master:~/k8s-s/services# kubectl get svc bq-api-service-1 -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"bq-api-service"},"name":"bq-api-service-1","namespace":"default"},"spec":{"ports":[{"name":"8080-port","port":32111,"targetPort":8080}],"selector":{"app":"bq-api-service"},"type":"NodePort"}}
creationTimestamp: "2024-04-07T14:44:56Z"
labels:
app: bq-api-service
name: bq-api-service-1
namespace: default
resourceVersion: "2301903"
uid: 6dc89e70-6d6f-4965-a613-246c085ca1ef
spec:
clusterIP: 10.105.158.88
clusterIPs:
- 10.105.158.88
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: 8080-port
nodePort: 31372
port: 32111
protocol: TCP
targetPort: 8080
selector:
app: bq-api-service
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
再看endpoint
root@k8s-master:~/k8s-s/services# kubectl get endpoints -o wide --show-labels
NAME ENDPOINTS AGE LABELS
bq-api-service-1 10.244.1.45:8080,10.244.2.104:8080,10.244.3.54:8080 17m app=bq-api-service
kubernetes 192.168.0.3:6443 21h endpointslice.kubernetes.io/skip-mirror=true
重点来了, 一个与service 同名的endpoint 被创建出来, 其实service 与 endpoint 是1 1 对应的, 可以把endpoint 看作service 的属性
endpoints 有3个
10.244.1.45:8080,10.244.2.104:8080,10.244.3.54:8080
其实他们正是 是3个nodes的内部ip + target port
测试
正常我们要通过随机生成的端口nodePort: 31372 来访问
我们先登陆一台k8s 集群外的server
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000
link/ether 42:01:c0:a8:00:23 brd ff:ff:ff:ff:ff:ff
altname enp0s4
inet 192.168.0.35/32 brd 192.168.0.35 scope global dynamic ens4
valid_lft 53636sec preferred_lft 53636sec
inet6 fe80::4001:c0ff:fea8:23/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP>
...
用curl 命令来测试
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6:31372/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-tkxck","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6:31372/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-c8zlp","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6:31372/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-tkxck","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6:31372/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-c8zlp","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6:31372/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-c8zlp","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$ curl 192.168.0.6:31372/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-vvkt9","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
gateman@tf-vpc0-subnet0-main-server:~$
在多次执行的情况下, 虽然我们curl 指定是 node0 的ip , 但是实际上, 返回的hostname 是随机的, 包含其他node 的结果, 所以nodeport service 是包含Loadbalance 的功能的
NodePort, service port, target port的作用
从service 的yaml 可以看出, 当这个service 被创建后, 3个ports 出现了
分别是
nodePort: 31372
port: 32111
targetPort: 8080
其中nodePort 31372 是用于被集群外访问的端口, targetPort 是Pod 的端口 , 置于port 32111 是什么来的
其实我之前的文章已经介绍过, 这里的32111 port 是service 本身的port
https://editor.csdn.net/md/?articleId=131278799
也就是讲 service 有两个端口, 1个是被外部访问的, 另1个是service自己内部端口
其实如果我们在集群内任何台主机, 是可以同过service 的ip 和 service 的port去访问的
root@k8s-master:~/k8s-s/services# kubectl get svc bq-api-service-1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bq-api-service-1 NodePort 10.105.158.88 <none> 32111:31372/TCP 47m
root@k8s-master:~/k8s-s/services# curl 10.105.158.88:32111/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-tkxck","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
如何我们进入某个pod的容器
甚至可以用service name 和 service port 去访问
root@k8s-master:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
bq-api-service-deploy-sample-f448cf6ff-c8zlp 1/1 Running 0 56m
bq-api-service-deploy-sample-f448cf6ff-tkxck 1/1 Running 0 56m
bq-api-service-deploy-sample-f448cf6ff-vvkt9 1/1 Running 0 56m
root@k8s-master:~# kubectl exec -it bq-api-service-deploy-sample-f448cf6ff-c8zlp -- /bin/bash
root@bq-api-service-deploy-sample-f448cf6ff-c8zlp:/app# curl http://bq-api-service-1:32111/actuator/info
{"app":"Sales API","version":"1.1.4","hostname":"bq-api-service-deploy-sample-f448cf6ff-tkxck","description":"This is a simple Spring Boot application to demonstrate the use of BigQuery in GCP."}
所以看出service port 32111 是被pod 容器内部用的
nodePort 图例
还是重新画张图加深理解