1. 故障现象
2022年10月25日，登陆VC报错。
按照报错信息，结合官方文档，判断为STS证书过期导致。
vCenter Server Appliance (VCSA) 6.5.x, 6.7.x or vCenter Server 7.0.x
在/var/log/vmware/vpxd-svcs/vpxd-svcs.log看到类似报错:
ERROR com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Server rejected the provided time range. Cause:ns0:InvalidTimeRange: The token authority rejected an issue request for TimePeriod [startTime=Thu Oct 02 09:22:13 EST 2022, endTime=Fri Oct 03 09:22:13 EST 2022] :: Signing certificate is not valid at Thu Jan 02 09:22:13 EST 2020, cert validity: TimePeriod [startTime=Wed Jan 06 20:44:39 EST 2010, endTime=Wed Jan 01 20:54:23 EST 2020]
Note: The endTime should be a date in the past if the certificate is expired.
These issue occurs when the Security Token Service (STS) certificate has expired. This causes internal services and solution users to not be able to acquire valid tokens and as a result fails to function as expected.
2. 查看证书过期情况
root@dxcvcsa [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
证书的确已经过期。
3. 更新证书
root@dxcvcsa [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.7 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 4
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y
Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:Administrator@vsphere.local
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : y
Press Enter key to skip optional parameters or use Previous value.
Enter proper value for 'Country' [Previous value : US] : cn
Enter proper value for 'Name' [Previous value : CA] : CA
Enter proper value for 'Organization' [Previous value : VMware] : VMware
Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] : VMware Engineering
Enter proper value for 'State' [Previous value : California] : GuangDong
Enter proper value for 'Locality' [Previous value : Palo Alto] : Guangzhou
Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 127.0.0.1
Enter proper value for 'Email' [Previous value : email@acme.com] : email@acme.com
Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : dxcvcsa.localdns.com
Enter proper value for VMCA 'Name' :dxcVMCA
You are going to regenerate Root Certificate and all other certificates using VMCA
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Replacing Machine SSL Cert...]
default-site
Lookup all services
Get service default-site:45ee0951-9cf9-4c22-8641-a791f5e935c8
Don't update service default-site:45ee0951-9cf9-4c22-8641-a791f5e935c8
Get service default-site:adf34f62-1d81-467b-9f76-59304c504388
Don't update service default-site:adf34f62-1d81-467b-9f76-59304c504388
Get service default-site:452dfd21-741a-4286-b59f-e4479fd73d02
Don't update service default-site:452dfd21-741a-4286-b59f-e4479fd73d02
Get service 9356d7ff-5045-4720-a142-3e1561dc2caa
Update service 9356d7ff-5045-4720-a142-3e1561dc2caa; spec: /tmp/svcspec_o29ann0i
Get service eb760607-6057-4c8f-bffe-c4459a23361a
Update service eb760607-6057-4c8f-bffe-c4459a23361a; spec: /tmp/svcspec_f9a6t5iv
Get service e72dc500-379b-445c-a6a2-934980d7697f
Update service e72dc500-379b-445c-a6a2-934980d7697f; spec: /tmp/svcspec_q745wbdl
Get service cc66bae3-9a81-4a47-bfc2-f56b521a3491
Update service cc66bae3-9a81-4a47-bfc2-f56b521a3491; spec: /tmp/svcspec_h6wiab6b
Get service ff3c666a-8048-401c-8e5d-3cc29d783d5f
Update service ff3c666a-8048-401c-8e5d-3cc29d783d5f; spec: /tmp/svcspec_734jtjut
Get service 47bbd5fd-cdd8-4c43-a839-9cac3c4ffb14_kv
Update service 47bbd5fd-cdd8-4c43-a839-9cac3c4ffb14_kv; spec: /tmp/svcspec_5q6r0b9z
Get service 0d2020df-096e-401f-bfbe-22ab3c73e321
Update service 0d2020df-096e-401f-bfbe-22ab3c73e321; spec: /tmp/svcspec_rnepbocv
Get service 40d4c99b-3840-4e75-ae9f-01c1a1d51693
Update service 40d4c99b-3840-4e75-ae9f-01c1a1d51693; spec: /tmp/svcspec_2ej9pwvm
Get service f9210573-346b-48c1-a0f4-57e469eed937
Update service f9210573-346b-48c1-a0f4-57e469eed937; spec: /tmp/svcspec_rgu720he
Get service 18db73cb-840d-4dc9-b591-af78cb26699d
Update service 18db73cb-840d-4dc9-b591-af78cb26699d; spec: /tmp/svcspec_vhd1si6e
Get service 447163a3-d02e-41cb-bedf-6bb6bc52c882
Update service 447163a3-d02e-41cb-bedf-6bb6bc52c882; spec: /tmp/svcspec_2vt5_pkn
Get service 1f305057-ad6e-46f2-816f-b638cbe5f8cc
Update service 1f305057-ad6e-46f2-816f-b638cbe5f8cc; spec: /tmp/svcspec_ed9zzks0
Get service 47bbd5fd-cdd8-4c43-a839-9cac3c4ffb14
Update service 47bbd5fd-cdd8-4c43-a839-9cac3c4ffb14; spec: /tmp/svcspec_uu_hj1bs
Get service 81ef1813-f5da-4a52-bf5e-730b0d76c45b
Update service 81ef1813-f5da-4a52-bf5e-730b0d76c45b; spec: /tmp/svcspec_o9q1aqf5
Get service 9968f0d6-7c05-4b00-a0bf-61cd8138c29f
Update service 9968f0d6-7c05-4b00-a0bf-61cd8138c29f; spec: /tmp/svcspec_332zqona
Get service 2472164c-9862-4209-9377-e6c9310bf544
Update service 2472164c-9862-4209-9377-e6c9310bf544; spec: /tmp/svcspec_vllnxe3y
Get service e8e5ba87-5834-40e3-8697-7524754dba64
Update service e8e5ba87-5834-40e3-8697-7524754dba64; spec: /tmp/svcspec_ytjr_fpf
Get service f351ae3e-99db-4cb6-b559-2afe53406c8d
Update service f351ae3e-99db-4cb6-b559-2afe53406c8d; spec: /tmp/svcspec_ahxrtfp2
Get service 81bd2bd9-9fc1-481f-bf8f-744a54e0fb76
Update service 81bd2bd9-9fc1-481f-bf8f-744a54e0fb76; spec: /tmp/svcspec_b9p8e9r_
Get service 87a6c98a-046f-46ec-9aba-d66a30c0a91b
Update service 87a6c98a-046f-46ec-9aba-d66a30c0a91b; spec: /tmp/svcspec_l5nahdu6
Get service b496d4b6-7560-4f58-9129-ce594ee96778
Update service b496d4b6-7560-4f58-9129-ce594ee96778; spec: /tmp/svcspec_qy6458zi
Get service 3888acd4-aa58-4c5f-8b43-30f454f4d97f
Update service 3888acd4-aa58-4c5f-8b43-30f454f4d97f; spec: /tmp/svcspec_tgdq0mzy
Get service d690b63c-6105-4411-8e14-1d10259b812f
Update service d690b63c-6105-4411-8e14-1d10259b812f; spec: /tmp/svcspec_95zuwvcb
Get service 174b1a17-b44b-4967-bb94-4f7c531ba800
Update service 174b1a17-b44b-4967-bb94-4f7c531ba800; spec: /tmp/svcspec_crrn4enf
Get service 47bbd5fd-cdd8-4c43-a839-9cac3c4ffb14_authz
Update service 47bbd5fd-cdd8-4c43-a839-9cac3c4ffb14_authz; spec: /tmp/svcspec_s6zjph53
Get service 34585982-ec94-4a93-bc1f-f80eecdaf88d
Update service 34585982-ec94-4a93-bc1f-f80eecdaf88d; spec: /tmp/svcspec_p_xvj30r
Get service f8a197a6-4fdb-4dcb-baa7-cc4825f824dc
Update service f8a197a6-4fdb-4dcb-baa7-cc4825f824dc; spec: /tmp/svcspec_mnjwbgp6
Get service dfa6cc50-dbe5-4997-bd8d-949e75be87e8
Update service dfa6cc50-dbe5-4997-bd8d-949e75be87e8; spec: /tmp/svcspec_fzje6ttg
Get service eb760607-6057-4c8f-bffe-c4459a23361a_com.vmware.vsphere.client
Don't update service eb760607-6057-4c8f-bffe-c4459a23361a_com.vmware.vsphere.client
Get service bc5ba386-ce79-42de-a8f9-67c6b8f03bf1
Update service bc5ba386-ce79-42de-a8f9-67c6b8f03bf1; spec: /tmp/svcspec_40_4ncxp
Get service 024591a5-3492-4567-81d7-0439f2113196
Update service 024591a5-3492-4567-81d7-0439f2113196; spec: /tmp/svcspec__s5my1_r
Get service 5944fc2d-78d7-42f1-9a17-efc9fa0bbff3
Update service 5944fc2d-78d7-42f1-9a17-efc9fa0bbff3; spec: /tmp/svcspec_wnt0axw7
Get service eb760607-6057-4c8f-bffe-c4459a23361a_com.commvault.vsa
Don't update service eb760607-6057-4c8f-bffe-c4459a23361a_com.commvault.vsa
Updated 31 service(s)
Status : 60% Completed [Replace vpxd-extension Cert...]
2022-10-26T00:46:00.988Z Updating certificate for "com.vmware.imagebuilder" extension
Status : 85% Completed [starting services...]
Status : 100% Completed [All tasks completed successfully]
3.1更新完毕，查看服务状态
service-control --stop –-all
service-control --start --all
3.2更新完毕，查看证书状态
root@dxcvcsa [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Oct 26 00:54:00 2024 GMT
STORE TRUSTED_ROOTS
Alias : 50b4e9c55d6b2db1034e66bfc38a01e2767c5137
Not After : Oct 14 03:02:08 2030 GMT
Alias : 450298f685afd4f275d79a596fa4ec42a8d38fc8
Not After : Oct 19 01:38:45 2032 GMT
Alias : 92e2f9521f9c605fb523b539e877a795a2f4d7b5
Not After : Oct 20 00:44:35 2032 GMT
STORE TRUSTED_ROOT_CRLS
Alias : 7f39f6f28fdfb986ca190af6fafe42eaf534d304
Alias : d7fafe3b63ce838a05e20f65d87de85c7010f40e
Alias : ba124fb88dd50bf2878bcc5dbb75d5bf0b4ee7dc
STORE machine
Alias : machine
Not After : Oct 26 00:54:05 2024 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
Not After : Oct 26 00:54:06 2024 GMT
STORE vpxd
Alias : vpxd
Not After : Oct 26 00:54:07 2024 GMT
STORE vpxd-extension
Alias : vpxd-extension
Not After : Oct 26 00:54:10 2024 GMT
STORE APPLMGMT_PASSWORD
STORE data-encipherment
Alias : data-encipherment
Not After : Oct 19 02:54:13 2022 GMT
STORE SMS
Alias : sms_self_signed
Not After : Oct 19 03:05:10 2030 GMT
STORE BACKUP_STORE
Alias : bkp___MACHINE_CERT
Not After : Oct 26 00:38:48 2024 GMT
Alias : bkp_machine
Not After : Oct 26 00:38:56 2024 GMT
Alias : bkp_vsphere-webclient
Not After : Oct 26 00:39:01 2024 GMT
Alias : bkp_vpxd
Not After : Oct 26 00:39:05 2024 GMT
Alias : bkp_vpxd-extension
Not After : Oct 26 00:39:12 2024 GMT
STORE BACKUP_STORE_H5C
Alias : bkp__MACHINE_CERT
Not After : Oct 25 00:34:35 2024 GMT
Alias : bkpmachine
Not After : Oct 25 00:35:58 2024 GMT
Alias : bkpvsphere-webclient
Not After : Oct 25 00:35:59 2024 GMT
Alias : bkpvpxd
Not After : Oct 25 00:35:59 2024 GMT
Alias : bkpvpxd-extension
Not After : Oct 25 00:35:59 2024 GMT
root@dxcvcsa [ ~ ]#
3.3正常登录VC 查看证书信息
重新生成证书所用信息，已在证书体现，有个细节就是country填的是cn，这里显示的还是US。
有专用脚本检测证书状态。
3.4新生成证书存放位置
root@dxcvcsa [ /usr/lib/vmware-vmca/share/config ]# cat /var/tmp/vmware/certool.cfg
Country = cn
Name = CA
Organization = VMware
OrgUnit = VMware Engineering
State = GuangDong
Locality = Guangzhou
IPAddress = 127.0.0.1
Email = email@acme.com
Hostname = dxcvcsa.localdns.com
root@dxcvcsa [ /usr/lib/vmware-vmca/share/config ]#
3.5默认证书存放位置
The Certool.cfg is located at:
vCenter Server Appliance: /usr/lib/vmware-vmca/share/config/certool.cfg
External Platform Service Controller Appliance: /usr/lib/vmware-vmca/share/config/certool.cfg
root@dxcvcsa [ ~ ]# cat /usr/lib/vmware-vmca/share/config/certool.cfg
#
# Template file for a CSR request
#
# Country is needed and has to be 2 characters
Country = US
Name = CA
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
IPAddress = 127.0.0.1
Email = email@acme.com
Hostname = server.acme.com
root@dxcvcsa [ ~ ]# cat /usr/lib/vmware-vmca/share/config/certool.cfg
#
# Template file for a CSR request
#
# Country is needed and has to be 2 characters
Country = US
Name = CA
Organization = VMware
OrgUnit = VMware Engineering
State = California
Locality = Palo Alto
IPAddress = 127.0.0.1
Email = email@acme.com
Hostname = server.acme.com
Tips:
如果不知道PNID可以用下面命令查一下：
/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
参考文献
1. Checking Expiration of STS Certificate on vCenter Servers (79248)
2. How to use vSphere Certificate Manager to Replace SSL Certificates (2097936)
作者：samyang2558
链接：https://www.jianshu.com/p/af415de235f5
来源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。
