1. 问题背景:Rancher升级后Fleet SSH GitRepo连接失败现象
上周三凌晨,我们按计划将生产环境的Rancher从2.6.5升级到2.7.5版本后,监控系统突然开始报警——所有通过Fleet管理的GitRepo仓库同步任务全部失败。查看Fleet控制器日志时,发现大量类似如下的报错:
code复制Host key verification failed for git@github.com.
No matching host key found in known_hosts file
这个错误直接导致集群中300+应用的GitOps流程中断。更棘手的是,同样的GitRepo配置在升级前完全正常,且Rancher官方升级文档中根本没有提到任何关于SSH连接变更的说明。
2. 主机密钥验证机制深度解析
2.1 SSH known_hosts文件工作原理
当客户端首次通过SSH连接服务器时,OpenSSH会记录服务器的公钥指纹到~/.ssh/known_hosts文件中。后续每次连接时,客户端都会比对该指纹,防止中间人攻击。其验证流程如下:
- 客户端发起SSH连接请求
- 服务器返回自己的公钥指纹
- 客户端检查
known_hosts中是否存在该主机记录- 存在且匹配 → 建立连接
- 存在但不匹配 → 报错"Host key verification failed"
- 不存在 → 询问用户是否信任该主机(交互式场景)
2.2 Fleet的SSH连接特殊之处
Fleet作为Rancher的GitOps组件,在非交互式环境中运行SSH连接。这意味着:
- 无法弹出交互式提示要求确认新主机密钥
- 默认情况下如果
known_hosts不存在或缺少对应记录,连接会直接失败 - 在Rancher升级过程中,Fleet控制器的
known_hosts文件可能被重置或丢失
3. 问题排查全记录
3.1 验证已知主机密钥状态
首先登录Fleet控制器Pod检查known_hosts文件:
bash复制kubectl -n cattle-fleet-system exec -it fleet-controller-xxx -- bash
cat ~/.ssh/known_hosts
发现文件确实为空。手动测试SSH连接:
bash复制ssh -T git@github.com
# 输出:
# The authenticity of host 'github.com (20.205.243.166)' can't be established.
# ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
# This host key is known but does not match.
3.2 对比升级前后的配置差异
通过对比升级前后的Fleet Deployment配置,发现环境变量SSH_KNOWN_HOSTS在2.7.5版本中被移除了。该变量原本用于预置Git服务商的主机密钥:
yaml复制# Rancher 2.6.5中的配置片段
env:
- name: SSH_KNOWN_HOSTS
value: |
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
3.3 定位Rancher升级的影响点
查阅Rancher 2.7的变更日志发现,Fleet的SSH连接处理逻辑进行了重构:
- 移除了硬编码的
SSH_KNOWN_HOSTS环境变量 - 改为依赖Pod内
/etc/ssh/ssh_known_hosts系统级文件 - 默认不再自动注入常见Git服务商密钥
4. 解决方案与实施步骤
4.1 临时解决方案:手动注入known_hosts
对于急需恢复的场景,可以手动创建ConfigMap并挂载:
yaml复制apiVersion: v1
kind: ConfigMap
metadata:
name: fleet-ssh-known-hosts
namespace: cattle-fleet-system
data:
known_hosts: |
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
然后修改Fleet Deployment添加volumeMount:
yaml复制volumeMounts:
- name: ssh-known-hosts
mountPath: /root/.ssh/known_hosts
subPath: known_hosts
volumes:
- name: ssh-known-hosts
configMap:
name: fleet-ssh-known-hosts
4.2 长期解决方案:使用GitRepo的spec配置
Rancher 2.7.5+推荐在GitRepo资源中直接指定knownHosts字段:
yaml复制apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
name: my-repo
spec:
repo: git@github.com:myorg/myrepo.git
knownHosts: |
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
4.3 自动化方案:使用准入控制器
对于大规模环境,可以开发Mutating Admission Webhook自动为GitRepo注入knownHosts:
go复制func mutateGitRepo(ar *v1.AdmissionReview) *v1.AdmissionResponse {
gitRepo := /* 解析请求中的GitRepo */
if gitRepo.Spec.Repo == "git@github.com:*/*" && gitRepo.Spec.KnownHosts == "" {
gitRepo.Spec.KnownHosts = githubKnownHosts
return createPatchResponse(gitRepo)
}
}
5. 关键注意事项与经验分享
5.1 主机密钥的安全考量
- 不要禁用验证:虽然可以设置
StrictHostKeyChecking=no,但这会带来安全风险 - 定期更新密钥:Git服务商可能更换密钥,建议每季度检查一次
- 多环境隔离:生产与非生产环境应使用不同的密钥配置
5.2 调试SSH连接的技巧
当SSH连接出现问题时,可以添加-v参数获取详细日志:
bash复制ssh -vT git@github.com
关键日志点:
debug1: Host 'github.com' is known→ 主机验证通过debug1: Found key in /root/.ssh/known_hosts:1→ 找到匹配记录Offending key in /root/.ssh/known_hosts:42→ 密钥不匹配的具体位置
5.3 企业级GitOps的最佳实践
- 统一密钥管理:通过Vault等工具集中管理SSH密钥
- 变更预检:升级前使用
kubectl diff检查配置变更 - 监控方案:对Fleet同步状态设置Prometheus监控指标
6. 扩展知识:主流Git服务商的SSH指纹
以下是2023年最新验证过的主机密钥(建议使用时再次确认):
GitHub:
code复制github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
GitLab:
code复制gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
Bitbucket:
code复制bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==
