1. 为什么需要HTTPS黑盒监控?
在分布式系统架构中,服务可用性监控是最基础的运维需求。传统的白盒监控(如Prometheus的Node Exporter)通过暴露系统内部指标来工作,但它无法回答一个关键问题:从外部用户视角看,我的服务真的可用吗?这就是黑盒监控的价值所在。
HTTPS协议作为现代Web服务的标准,其监控有三大特殊挑战:
- 证书有效性:过期或配置错误的证书会导致服务不可用
- 协议握手:TLS版本不匹配、密码套件配置错误等问题
- 内容校验:返回内容需要包含特定关键字或状态码
我曾遇到一个典型案例:某金融服务的Prometheus监控显示所有指标正常,但实际用户却无法访问。最终发现是CDN边缘节点的TLS 1.3配置与部分老旧客户端不兼容。这种问题只有通过黑盒监控才能及时发现。
2. Prometheus黑盒监控方案选型
2.1 Blackbox Exporter核心架构
Prometheus官方推荐的blackbox_exporter采用模块化设计:
code复制 +-------------------+
| |
+---------->+ Probe Target |
| | |
| +---------+---------+
| |
+---------+---------+ |
| | v
| Blackbox | +-------+-------+
| Exporter | | |
| (HTTP/HTTPS +--->+ Probe Module |
| Module) | | |
| | +-------+-------+
+---------+---------+ |
| |
| +---------v---------+
| | |
+-----------+ Prometheus |
| Server |
| |
+-------------------+
关键配置参数示例(blackbox.yml):
yaml复制modules:
https_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [200,301,302]
method: GET
tls_config:
insecure_skip_verify: false
preferred_ip_protocol: "ip4"
2.2 与Pushgateway的集成模式
对于短暂任务或批处理作业,标准的pull模式不适用。这时需要Pushgateway作为中间层:
- 短期任务将监控指标推送到Pushgateway
- Prometheus定期从Pushgateway拉取数据
- Blackbox Exporter同时监控Pushgateway的可用性
典型工作流:
bash复制# 任务完成后推送指标
echo "some_metric 3.14" | curl --data-binary @- http://pushgateway:9091/metrics/job/some_job
# Blackbox配置检查Pushgateway
- targets:
- http://pushgateway:9091/metrics
labels:
module: http_2xx
3. HTTPS监控的进阶配置
3.1 证书过期监控
在blackbox.yml中添加:
yaml复制 https_cert:
prober: http
http:
fail_if_not_ssl: true
tls_config:
insecure_skip_verify: false
preferred_ip_protocol: "ip4"
对应的告警规则(alert.rules):
yaml复制- alert: SSLCertExpiringSoon
expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30 # 30天
for: 5m
labels:
severity: critical
annotations:
summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
description: "SSL certificate expires in {{ $value | humanizeDuration }}"
3.2 多阶段内容校验
对于关键业务页面,需要验证页面内容和加载时间:
yaml复制 https_shop_check:
prober: http
http:
valid_status_codes: [200]
method: GET
headers:
Host: "shop.example.com"
fail_if_body_not_matches_regexp:
- "Available"
- "Add to Cart"
fail_if_not_ssl: true
preferred_ip_protocol: "ip4"
4. 生产环境部署实践
4.1 高可用部署方案
建议的拓扑结构:
code复制 +-----------------+
| Load |
| Balancer |
+-------+---------+
|
+---------------+---------------+
| |
+----------+---------+ +-----------+----------+
| Blackbox | | Blackbox |
| Exporter 1 | | Exporter 2 |
| (region-a) | | (region-b) |
+--------------------+ +---------------------+
| |
+---------------+---------------+
|
+-------+---------+
| Prometheus |
| Server |
+-----------------+
4.2 性能调优参数
在大量监控目标场景下,需要调整这些参数(blackbox.yml):
yaml复制# 每个探针的超时时间
timeout: 10s
# 最大并发探针数
max_concurrency: 50
# HTTP探针的keepalive设置
http:
ip_protocol_fallback: false
preferred_ip_protocol: "ip4"
tls_config:
insecure_skip_verify: false
transport:
dial_timeout: 5s
dial_keep_alive: 30s
max_idle_conns_per_host: 5
5. 常见问题排查指南
5.1 证书验证失败
错误现象:
code复制probe_ssl_verification_error 1
排查步骤:
- 检查证书链是否完整:
bash复制
openssl s_client -connect example.com:443 -showcerts - 验证证书是否过期:
bash复制openssl x509 -enddate -noout -in cert.pem - 检查中间证书是否缺失
5.2 Pushgateway数据过期
典型告警规则:
yaml复制- alert: StalePushgatewayMetrics
expr: time() - push_time_seconds > 3600
for: 5m
labels:
severity: warning
annotations:
summary: "Metrics stale in pushgateway (job {{ $labels.job }})"
处理方案:
- 设置自动清理旧数据:
bash复制# 启动Pushgateway时添加参数 --persistence.file=/data/metrics.store \ --persistence.interval=5m \ --web.enable-admin-api - 通过API强制清理:
bash复制
curl -X PUT http://pushgateway:9091/api/v1/admin/wipe
6. 监控指标深度解析
6.1 关键HTTPS指标
| 指标名称 | 类型 | 说明 |
|---|---|---|
| probe_success | Gauge | 探针是否成功(0/1) |
| probe_duration_seconds | Gauge | 探针执行耗时 |
| probe_http_status_code | Gauge | HTTP状态码 |
| probe_ssl_earliest_cert_expiry | Gauge | 最近到期证书的时间戳 |
| probe_http_content_length | Gauge | 响应内容长度 |
| probe_http_redirects | Gauge | 重定向次数 |
6.2 黄金告警规则
yaml复制groups:
- name: https.rules
rules:
- alert: HTTPSDown
expr: probe_success == 0
for: 1m
labels:
severity: critical
annotations:
summary: "HTTPS probe failed (instance {{ $labels.instance }})"
- alert: HighLatency
expr: probe_duration_seconds > 2
for: 5m
labels:
severity: warning
annotations:
summary: "High latency detected (instance {{ $labels.instance }})"
value: "{{ $value }}s"
7. Grafana可视化实战
推荐使用ID 7587(Blackbox Exporter)和 1860(SSL Certificate)仪表盘:
- HTTPS可用性看板配置:
json复制{
"panels": [
{
"title": "HTTPS Success Rate",
"type": "stat",
"targets": [{
"expr": "avg(probe_success) by (instance)",
"legendFormat": "{{instance}}"
}]
},
{
"title": "Certificate Expiry",
"type": "gauge",
"targets": [{
"expr": "(probe_ssl_earliest_cert_expiry - time()) / 86400",
"legendFormat": "Days remaining"
}]
}
]
}
- Pushgateway监控看板关键查询:
sql复制sum(push_time_seconds) by (job) // 各job最后推送时间
count(metrics) by (job) // 各job指标数量
8. 安全加固建议
- Blackbox Exporter访问控制:
yaml复制# 启用TLS
web:
tls_server_config:
cert_file: /certs/server.crt
key_file: /certs/server.key
client_auth_type: RequireAndVerifyClientCert
client_ca_file: /certs/ca.crt
# 启用HTTP基本认证
basic_auth_users:
prometheus: "$2y$05$xxxxxxxxxxxxxxxxxxxx"
- Pushgateway数据隔离:
bash复制# 按项目使用不同路径
/metrics/job/<job_name>/project/<project_id>
# 启用删除保护
--web.enable-admin-api=false
