Python操作MySQL：从基础到高级实战指南-代码聚汇网

Python操作MySQL：从基础到高级实战指南

DR阿福·2026-08-05·阅读 50 GEO检测（免费）

PythonMySQL数据库操作

|难度：普通|阅读约 65 分钟

1. Python操作MySQL的核心价值与应用场景

MySQL作为最流行的开源关系型数据库之一，与Python的结合堪称数据处理领域的黄金搭档。在实际项目中，我经常遇到需要从Web应用、数据分析平台到自动化脚本等各种场景下操作MySQL的需求。Python通过多种驱动库提供了灵活的数据访问方式，能够满足从简单查询到复杂事务处理的全套需求。

为什么选择Python操作MySQL？首先，Python简洁的语法和丰富的第三方库大幅降低了数据库操作门槛。相比Java的JDBC或PHP的PDO，Python的数据库接口更加人性化。其次，Python强大的数据处理能力（如Pandas、NumPy）与MySQL的结合，为数据分析和业务智能提供了完整解决方案。我在金融风控系统中就曾用这种组合处理过千万级交易记录。

当前主流的技术栈中，Django ORM、SQLAlchemy等ORM框架提供了高级抽象，而PyMySQL、mysql-connector-python等底层驱动则适合需要精细控制的场景。根据项目经验，中小型项目可以直接使用ORM提高开发效率，而高性能要求的系统则需要权衡ORM的便利性和原生SQL的执行效率。

需要模型API调用？免费领10W Token，多模型网关一键接入 Claude、DeepSeek 等主流模型。

2. 环境准备与驱动选型

2.1 驱动库比较与选择

Python操作MySQL主要有以下几种驱动选择：

mysql-connector-python：MySQL官方出品，纯Python实现，兼容性好但性能中等
PyMySQL：纯Python实现，轻量级且支持Python3新特性
MySQLdb：C语言扩展，性能最好但已停止维护（Python3需用mysqlclient分支）

bash复制# 安装示例（推荐PyMySQL）
pip install pymysql cryptography  # cryptography用于密码加密

注意：生产环境务必安装cryptography包，避免密码明文传输风险

2.2 连接池配置最佳实践

高频访问场景下，直接创建连接会导致严重性能问题。建议使用DBUtils或SQLAlchemy的连接池：

python复制from dbutils.pooled_db import PooledDB
import pymysql

pool = PooledDB(
    creator=pymysql,
    maxconnections=20,
    mincached=5,
    host='localhost',
    user='root',
    password='safe_password',
    database='test',
    charset='utf8mb4'
)

关键参数说明：

maxconnections：根据服务器内存设置（建议每GB内存对应10-15个连接）
wait_timeout：应小于MySQL的wait_timeout参数（默认8小时）

3. 核心操作全解析

3.1 连接管理与异常处理

规范的连接操作应使用context manager模式：

python复制import pymysql
from contextlib import contextmanager

@contextmanager
def get_connection():
    conn = pymysql.connect(
        host='127.0.0.1',
        user='dev_user',
        password='encrypted_pwd',
        database='production_db',
        charset='utf8mb4',
        cursorclass=pymysql.cursors.DictCursor  # 返回字典形式结果
    )
    try:
        yield conn
    except pymysql.Error as e:
        conn.rollback()
        print(f"Database error: {e}")
        raise
    finally:
        conn.close()

重要：必须显式处理事务回滚，避免部分失败导致数据不一致

3.2 CRUD操作进阶技巧

3.2.1 高效批量插入

使用executemany()比循环插入快10倍以上：

python复制data = [(f'product_{i}', i*10) for i in range(1000)]
with get_connection() as conn:
    with conn.cursor() as cursor:
        sql = "INSERT INTO products (name, price) VALUES (%s, %s)"
        cursor.executemany(sql, data)
    conn.commit()

3.2.2 安全参数化查询

永远不要拼接SQL字符串！正确做法：

python复制# 错误示范（SQL注入风险）
f"SELECT * FROM users WHERE id = {user_input}"

# 正确做法
cursor.execute("SELECT * FROM users WHERE id = %s", (user_input,))

3.2.3 事务隔离级别实战

MySQL默认使用REPEATABLE READ，某些场景需要调整：

python复制with get_connection() as conn:
    with conn.cursor() as cursor:
        # 设置读已提交隔离级别
        cursor.execute("SET TRANSACTION ISOLATION LEVEL READ COMMITTED")
        # 业务操作...
        cursor.execute("UPDATE accounts SET balance = balance - 100 WHERE user_id = 1")
        cursor.execute("UPDATE accounts SET balance = balance + 100 WHERE user_id = 2")
    conn.commit()

4. 性能优化实战方案

4.1 索引使用与EXPLAIN分析

通过EXPLAIN诊断慢查询：

python复制with get_connection() as conn:
    with conn.cursor() as cursor:
        cursor.execute("EXPLAIN SELECT * FROM orders WHERE user_id = 123")
        result = cursor.fetchone()
        print(f"查询类型: {result['type']}")  # 理想情况应出现index或range

常见问题处理：

出现ALL类型：考虑为WHERE条件字段添加索引
出现filesort：检查ORDER BY字段是否有合适索引

4.2 连接查询优化

多表连接时注意：

小表驱动大表原则
确保关联字段有索引
避免SELECT * 只查询必要字段

python复制# 优化后的联表查询
sql = """
SELECT o.order_id, u.username, p.product_name 
FROM orders o
JOIN users u ON o.user_id = u.id  -- 确保这两个字段都有索引
JOIN products p ON o.product_id = p.id
WHERE o.create_time > %s
LIMIT 100
"""

5. ORM框架深度使用

5.1 SQLAlchemy核心模式

SQLAlchemy提供两种使用方式：

Core API：类似传统SQL但更安全
ORM：完全的面向对象方式

python复制from sqlalchemy import create_engine, Table, Column, Integer, String, MetaData

# 引擎配置（连接池已内置）
engine = create_engine(
    'mysql+pymysql://user:password@localhost/dbname',
    pool_size=10,
    max_overflow=20,
    echo_pool=True  # 开发时开启监控
)

metadata = MetaData()
users = Table('users', metadata,
    Column('id', Integer, primary_key=True),
    Column('name', String(30)),
    Column('email', String(100))
)

# 执行原生SQL仍是最灵活的方式
with engine.connect() as conn:
    result = conn.execute("SELECT * FROM users WHERE status=:status", 
                         {'status': 'active'})
    for row in result:
        print(row)

5.2 Django ORM最佳实践

Django项目中的数据库操作要点：

python复制# 使用select_related/prefetch_related优化关联查询
orders = Order.objects.select_related('user').prefetch_related('items')[:100]

# 批量操作使用bulk_create
Product.objects.bulk_create([
    Product(name=f"Item {i}", price=i*10)
    for i in range(1000)
])

# 原生SQL仍有用武之地
from django.db import connection
with connection.cursor() as cursor:
    cursor.callproc('calculate_revenue', [2023])

6. 生产环境问题排查

6.1 连接泄露检测

通过SHOW PROCESSLIST发现未关闭连接：

python复制def check_leaked_connections():
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute("SHOW PROCESSLIST")
            processes = cursor.fetchall()
            leaked = [p for p in processes 
                    if p['db'] == 'your_db' 
                    and p['Command'] == 'Sleep'
                    and p['Time'] > 300]  # 5分钟以上空闲连接
            if leaked:
                print(f"发现{len(leaked)}个可能泄露的连接")

6.2 死锁分析与处理

MySQL死锁日志分析：

python复制with get_connection() as conn:
    with conn.cursor() as cursor:
        cursor.execute("SHOW ENGINE INNODB STATUS")
        status = cursor.fetchone()
        print(status['Status'])  # 包含最近的死锁信息

处理建议：

事务尽量短小
多表操作保持一致的顺序
合理设置innodb_lock_wait_timeout

7. 高级特性应用

7.1 存储过程调用

Python调用MySQL存储过程示例：

python复制with get_connection() as conn:
    with conn.cursor() as cursor:
        cursor.callproc('calculate_monthly_report', [2023, 5])
        results = list(cursor.stored_results())[0].fetchall()
        for row in results:
            print(f"部门: {row['dept']}, 销售额: {row['amount']}")

7.2 JSON字段操作

MySQL 5.7+支持JSON类型字段：

python复制# 插入JSON数据
data = {
    "name": "智能设备",
    "specs": {
        "color": "black",
        "weight": 350
    }
}
sql = "INSERT INTO products (id, attributes) VALUES (%s, %s)"
cursor.execute(sql, (101, json.dumps(data)))

# 查询JSON字段
sql = """
SELECT 
    id,
    JSON_EXTRACT(attributes, '$.name') as product_name,
    JSON_EXTRACT(attributes, '$.specs.weight') as weight
FROM products
WHERE JSON_EXTRACT(attributes, '$.specs.color') = 'black'
"""

8. 安全加固方案

8.1 凭据管理规范

绝对不要在代码中硬编码数据库密码！推荐方案：

使用环境变量
密钥管理服务（如AWS KMS）
配置文件加密

python复制# 使用python-dotenv管理环境变量
from dotenv import load_dotenv
import os

load_dotenv()
DB_CONFIG = {
    'host': os.getenv('DB_HOST'),
    'user': os.getenv('DB_USER'),
    'password': os.getenv('DB_PASSWORD'),
    'database': os.getenv('DB_NAME')
}

8.2 SQL注入防御体系

多层级防护策略：

永远使用参数化查询
最小权限原则（应用账号只赋予必要权限）
输入验证（如使用Pydantic模型）
定期扫描（使用sqlmap等工具测试）

python复制# 使用Pydantic进行输入验证
from pydantic import BaseModel, constr

class UserQuery(BaseModel):
    user_id: int
    status: constr(regex='^(active|inactive)$')

def get_users(query: UserQuery):
    with get_connection() as conn:
        with conn.cursor() as cursor:
            sql = "SELECT * FROM users WHERE id=%s AND status=%s"
            cursor.execute(sql, (query.user_id, query.status))
            return cursor.fetchall()

9. 监控与性能分析

9.1 慢查询日志分析

配置MySQL慢查询日志后，可以用Python进行分析：

python复制def analyze_slow_log(log_path):
    from collections import defaultdict
    import re
    
    pattern = re.compile(r'Query_time: (\d+\.\d+).*?SET timestamp=\d+;\n(.*?);', re.DOTALL)
    queries = defaultdict(list)
    
    with open(log_path) as f:
        for match in pattern.finditer(f.read()):
            time, query = match.groups()
            queries[query.strip()].append(float(time))
    
    for query, times in queries.items():
        avg_time = sum(times)/len(times)
        print(f"平均耗时{avg_time:.3f}s\n{query}\n{'-'*50}")

9.2 实时性能监控

使用PyMySQL结合prometheus-client实现：

python复制from prometheus_client import Gauge
import time

DB_QUERY_TIME = Gauge('db_query_seconds', 'Database query time')

def monitored_query(sql, params=None):
    start = time.time()
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute(sql, params or ())
            result = cursor.fetchall()
    DB_QUERY_TIME.set(time.time() - start)
    return result

10. 实战案例：电商订单系统

10.1 分库分表策略

当日单量超过百万时的解决方案：

python复制def get_order_table(order_id):
    """根据订单ID路由到具体分表"""
    table_num = order_id % 16  # 16个分表
    return f"orders_{table_num:02d}"

def save_order(order_data):
    table = get_order_table(order_data['id'])
    sql = f"""
    INSERT INTO {table} 
    (id, user_id, amount, status)
    VALUES (%(id)s, %(user_id)s, %(amount)s, %(status)s)
    """
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute(sql, order_data)
        conn.commit()

10.2 分布式事务处理

使用XA协议保证跨库操作一致性：

python复制def transfer_funds(from_acc, to_acc, amount):
    xid = str(uuid.uuid4())
    try:
        with get_connection() as conn:
            with conn.cursor() as cursor:
                # 开启XA事务
                cursor.execute(f"XA START '{xid}'")
                
                # 扣款方操作
                cursor.execute(
                    "UPDATE account SET balance=balance-%s WHERE id=%s",
                    (amount, from_acc)
                )
                
                # 收款方操作
                cursor.execute(
                    "UPDATE account SET balance=balance+%s WHERE id=%s",
                    (amount, to_acc)
                )
                
                # 准备阶段
                cursor.execute(f"XA END '{xid}'")
                cursor.execute(f"XA PREPARE '{xid}'")
                
                # 提交阶段
                cursor.execute(f"XA COMMIT '{xid}'")
    except Exception as e:
        # 回滚处理
        with get_connection() as conn:
            with conn.cursor() as cursor:
                cursor.execute(f"XA ROLLBACK '{xid}'")
        raise

11. 新型MySQL特性应用

11.1 窗口函数实战

MySQL 8.0+支持的窗口函数：

python复制sql = """
SELECT 
    product_id,
    sale_date,
    amount,
    SUM(amount) OVER (PARTITION BY product_id ORDER BY sale_date) AS running_total,
    RANK() OVER (PARTITION BY YEAR(sale_date) ORDER BY amount DESC) AS yearly_rank
FROM sales
WHERE sale_date BETWEEN %s AND %s
"""

11.2 CTE递归查询

处理树形结构数据：

python复制sql = """
WITH RECURSIVE category_tree AS (
    -- 基础查询：选择根节点
    SELECT id, name, parent_id, 1 AS level
    FROM categories
    WHERE parent_id IS NULL
    
    UNION ALL
    
    -- 递归部分：连接子节点
    SELECT c.id, c.name, c.parent_id, ct.level + 1
    FROM categories c
    JOIN category_tree ct ON c.parent_id = ct.id
)
SELECT * FROM category_tree ORDER BY level, id;
"""

12. 调试与开发技巧

12.1 查询日志记录

开发环境建议开启完整查询日志：

python复制import logging

# 配置SQL日志
logging.basicConfig()
logger = logging.getLogger('sql')
logger.setLevel(logging.DEBUG)

# PyMySQL调试模式
conn = pymysql.connect(
    host='localhost',
    user='dev',
    password='dev_pwd',
    database='test',
    cursorclass=pymysql.cursors.DictCursor,
    autocommit=False
)
conn.debug = True  # 输出执行的SQL语句

12.2 单元测试策略

使用unittest.mock测试数据库操作：

python复制from unittest.mock import MagicMock

def test_query_user():
    # 创建mock连接
    mock_conn = MagicMock()
    mock_cursor = mock_conn.cursor.return_value
    mock_cursor.fetchall.return_value = [{'id': 1, 'name': 'Test'}]
    
    # 注入mock对象
    result = get_users(mock_conn, 1)
    
    # 验证行为
    mock_cursor.execute.assert_called_once_with(
        "SELECT * FROM users WHERE id=%s", (1,)
    )
    assert result[0]['name'] == 'Test'

13. 备份与恢复方案

13.1 逻辑备份实现

使用Python调用mysqldump：

python复制import subprocess
from datetime import datetime

def backup_database():
    timestamp = datetime.now().strftime("%Y%m%d_%H%M")
    dump_file = f"backup_{timestamp}.sql"
    
    cmd = [
        "mysqldump",
        "--single-transaction",
        "--routines",
        "--triggers",
        "--host=localhost",
        "--user=backup_user",
        "--password=backup_pwd",
        "production_db"
    ]
    
    with open(dump_file, 'w') as f:
        subprocess.run(cmd, stdout=f, check=True)
    
    print(f"备份成功: {dump_file}")

13.2 增量备份策略

结合binlog的增量备份方案：

python复制def take_binlog_backup():
    # 获取当前binlog位置
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute("SHOW MASTER STATUS")
            status = cursor.fetchone()
            print(f"当前binlog位置: {status['File']} {status['Position']}")
    
    # 使用mysqlbinlog工具备份
    cmd = [
        "mysqlbinlog",
        "--read-from-remote-server",
        "--host=localhost",
        "--user=repl",
        "--password=repl_pwd",
        "--result-file=binlog_backup.sql",
        "--start-position=123456",
        "mysql-bin.000001"
    ]
    subprocess.run(cmd, check=True)

14. 高可用架构集成

14.1 读写分离实现

使用ProxySQL中间件配置：

python复制def get_connection(read_only=False):
    host = 'read-replica' if read_only else 'master'
    return pymysql.connect(
        host=host,
        user='app_user',
        password='app_pwd',
        database='sharded_db',
        charset='utf8mb4'
    )

# 读操作自动路由到从库
def query_products():
    with get_connection(read_only=True) as conn:
        with conn.cursor() as cursor:
            cursor.execute("SELECT * FROM products")
            return cursor.fetchall()

14.2 故障自动转移

结合HAProxy的健康检查：

python复制import socket
from retrying import retry

@retry(stop_max_attempt_number=3, wait_fixed=1000)
def failover_connect():
    try:
        return pymysql.connect(
            host='haproxy-vip',
            user='app_user',
            password='app_pwd',
            database='critical_db',
            connect_timeout=3  # 快速失败
        )
    except (pymysql.Error, socket.timeout) as e:
        print(f"连接失败: {e}")
        raise

15. 云数据库适配

15.1 AWS RDS连接最佳实践

使用IAM数据库认证：

python复制import boto3
from pymysql import connect

def get_aws_connection():
    client = boto3.client('rds')
    token = client.generate_db_auth_token(
        DBHostname='mydb.cluster-123.us-east-1.rds.amazonaws.com',
        Port=3306,
        DBUsername='iam_user',
        Region='us-east-1'
    )
    
    return connect(
        host='mydb.cluster-123.us-east-1.rds.amazonaws.com',
        user='iam_user',
        password=token,
        database='production',
        ssl={'ca': '/path/to/rds-combined-ca-bundle.pem'}
    )

15.2 连接云数据库常见问题

云数据库的特殊注意事项：

公网连接需要配置安全组规则
建议启用SSL加密连接
注意云厂商的连接数限制
跨区域访问会有明显延迟

python复制# 安全的云数据库连接配置
conn = pymysql.connect(
    host='cloud-instance.rds.aliyuncs.com',
    user='cloud_user',
    password='complex_pwd',
    database='cloud_db',
    charset='utf8mb4',
    ssl={
        'ca': '/path/to/ca.pem',
        'check_hostname': True
    },
    connect_timeout=5,
    read_timeout=30  # 云环境网络可能不稳定
)

16. 数据迁移策略

16.1 全量迁移方案

使用Python实现可控迁移：

python复制def migrate_table(source_conn, target_conn, table_name, batch_size=1000):
    with source_conn.cursor() as src_cur:
        with target_conn.cursor() as tgt_cur:
            # 获取源表结构
            src_cur.execute(f"SHOW CREATE TABLE {table_name}")
            create_sql = src_cur.fetchone()['Create Table']
            
            # 在目标库创建表
            tgt_cur.execute(f"DROP TABLE IF EXISTS {table_name}")
            tgt_cur.execute(create_sql)
            
            # 分批次迁移数据
            offset = 0
            while True:
                src_cur.execute(f"SELECT * FROM {table_name} LIMIT %s OFFSET %s", 
                              (batch_size, offset))
                rows = src_cur.fetchall()
                if not rows:
                    break
                    
                tgt_cur.executemany(
                    f"INSERT INTO {table_name} VALUES ({','.join(['%s']*len(rows[0]))})",
                    rows
                )
                offset += batch_size
                print(f"已迁移 {offset} 条记录")
            
            target_conn.commit()

16.2 增量同步实现

基于binlog位置的点对点同步：

python复制import pymysqlreplication

def start_binlog_sync():
    stream = pymysqlreplication.BinLogStreamReader(
        connection_settings={
            'host': 'source_db',
            'port': 3306,
            'user': 'replicator',
            'passwd': 'repl_pwd'
        },
        server_id=100,  # 唯一ID
        blocking=True,
        resume_stream=True,
        log_file='mysql-bin.000001',
        log_pos=4
    )
    
    target_conn = pymysql.connect(host='target_db', user='writer', passwd='writer_pwd')
    
    try:
        for binlogevent in stream:
            if isinstance(binlogevent, pymysqlreplication.row_event.RowsEvent):
                apply_event_to_target(binlogevent, target_conn)
    finally:
        stream.close()
        target_conn.close()

17. 监控告警体系

17.1 关键指标采集

使用Prometheus exporter模式：

python复制from prometheus_client import start_http_server, Gauge
import time

# 定义监控指标
DB_CONNECTIONS = Gauge('mysql_connections', 'Current connections')
DB_QUERY_TIME = Gauge('mysql_query_seconds', 'Query execution time')

def monitor_mysql():
    start_http_server(8000)
    while True:
        with get_connection() as conn:
            with conn.cursor() as cursor:
                # 获取连接数
                cursor.execute("SHOW STATUS LIKE 'Threads_connected'")
                DB_CONNECTIONS.set(cursor.fetchone()['Value'])
                
                # 测试查询性能
                start = time.time()
                cursor.execute("SELECT 1")
                DB_QUERY_TIME.set(time.time() - start)
        
        time.sleep(15)

17.2 慢查询告警

结合ELK实现实时告警：

python复制from elasticsearch import Elasticsearch

def index_slow_query(query, duration):
    es = Elasticsearch(['http://elk:9200'])
    doc = {
        'query': query,
        'duration': duration,
        'timestamp': datetime.now().isoformat(),
        'severity': 'high' if duration > 5 else 'medium'
    }
    es.index(index='mysql-slowlogs', document=doc)

18. 数据加密方案

18.1 透明数据加密

使用MySQL企业版TDE功能：

python复制def enable_table_encryption(table_name):
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute(f"ALTER TABLE {table_name} ENCRYPTION='Y'")
            print(f"表 {table_name} 已启用加密")

18.2 应用层加密

敏感字段使用AES加密：

python复制from cryptography.fernet import Fernet

key = Fernet.generate_key()
cipher = Fernet(key)

def encrypt_data(data: str) -> bytes:
    return cipher.encrypt(data.encode())

def decrypt_data(token: bytes) -> str:
    return cipher.decrypt(token).decode()

# 数据库操作示例
user_ssn = encrypt_data('123-45-6789')
cursor.execute("INSERT INTO users (name, ssn) VALUES (%s, %s)", 
              ('John Doe', user_ssn))

19. 版本兼容性处理

19.1 多版本适配方案

针对不同MySQL版本的特性检测：

python复制def check_mysql_version(conn):
    with conn.cursor() as cursor:
        cursor.execute("SELECT VERSION()")
        version_str = cursor.fetchone()[0]
        version = tuple(map(int, version_str.split('.')[:2]))
        
        features = {
            'window_functions': version >= (8, 0),
            'json_support': version >= (5, 7),
            'cte': version >= (8, 0)
        }
        return features

19.2 降级兼容写法

JSON字段的兼容处理：

python复制def get_product_attributes(product_id):
    with get_connection() as conn:
        features = check_mysql_version(conn)
        with conn.cursor() as cursor:
            if features['json_support']:
                cursor.execute("""
                    SELECT id, JSON_EXTRACT(attrs, '$.color') as color
                    FROM products WHERE id = %s
                """, (product_id,))
            else:
                cursor.execute("SELECT id, attrs FROM products WHERE id = %s", (product_id,))
                row = cursor.fetchone()
                if row:
                    import json
                    attrs = json.loads(row['attrs'])
                    row['color'] = attrs.get('color')
                    return [row]
            return cursor.fetchall()

20. 扩展与集成

20.1 与Pandas无缝对接

将查询结果直接转为DataFrame：

python复制import pandas as pd

def query_to_dataframe(sql, params=None):
    with get_connection() as conn:
        return pd.read_sql(sql, conn, params=params)

# 使用示例
df = query_to_dataframe("""
    SELECT 
        DATE(create_time) as day,
        COUNT(*) as order_count,
        SUM(amount) as total_amount
    FROM orders
    WHERE status = 'completed'
    GROUP BY day
    ORDER BY day
""")
print(df.describe())

20.2 异步IO支持

使用aiomysql进行异步操作：

python复制import asyncio
import aiomysql

async def async_query():
    conn = await aiomysql.connect(
        host='localhost',
        user='async_user',
        password='async_pwd',
        db='async_db'
    )
    
    async with conn.cursor() as cursor:
        await cursor.execute("SELECT * FROM async_table")
        result = await cursor.fetchall()
        print(f"获取到 {len(result)} 条记录")
    
    conn.close()

# 在事件循环中运行
loop = asyncio.get_event_loop()
loop.run_until_complete(async_query())

21. 资源管理与连接治理

21.1 连接泄露检测

自动化检测未正确关闭的连接：

python复制import weakref
from collections import defaultdict

class ConnectionTracker:
    _instances = defaultdict(list)
    
    def __init__(self, conn):
        self.conn = conn
        self.trace = traceback.extract_stack()[:-1]
        self._instances[self.conn].append(weakref.ref(self))
    
    @classmethod
    def report_leaks(cls):
        for conn, refs in cls._instances.items():
            if conn.open:  # 连接仍未关闭
                print(f"发现泄露连接，创建堆栈:")
                for ref in refs:
                    if obj := ref():
                        for line in obj.trace:
                            print(f"  {line}")

# 使用装饰器包装连接
def track_connections(func):
    def wrapper(*args, **kwargs):
        conn = func(*args, **kwargs)
        return ConnectionTracker(conn)
    return wrapper

21.2 查询超时控制

防止长时间运行的查询：

python复制from concurrent.futures import ThreadPoolExecutor, TimeoutError

def execute_with_timeout(sql, params=None, timeout=30):
    with ThreadPoolExecutor(max_workers=1) as executor:
        future = executor.submit(
            lambda: get_connection().cursor().execute(sql, params or ())
        )
        try:
            return future.result(timeout=timeout)
        except TimeoutError:
            print(f"查询超时: {sql[:100]}...")
            raise

22. 数据校验与清洗

22.1 入库前数据验证

使用Pydantic模型验证：

python复制from pydantic import BaseModel, conint, constr

class ProductModel(BaseModel):
    id: conint(gt=0)
    name: constr(min_length=2, max_length=100)
    price: confloat(gt=0)
    category: constr(regex='^[A-Z][a-z]+$')

def save_product(data: dict):
    # 验证数据
    try:
        product = ProductModel(**data)
    except ValidationError as e:
        print(f"数据验证失败: {e}")
        raise
    
    # 保存到数据库
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute("""
                INSERT INTO products (id, name, price, category)
                VALUES (%s, %s, %s, %s)
            """, (product.id, product.name, product.price, product.category))
        conn.commit()

22.2 脏数据修复

批量修复错误数据：

python复制def clean_invalid_prices():
    with get_connection() as conn:
        with conn.cursor() as cursor:
            # 查找异常数据
            cursor.execute("""
                SELECT id, price FROM products
                WHERE price < 0 OR price > 1000000
            """)
            bad_records = cursor.fetchall()
            
            # 批量修复
            updates = []
            for row in bad_records:
                if row['price'] < 0:
                    new_price = 0
                else:
                    new_price = 1000000
                updates.append((new_price, row['id']))
            
            if updates:
                cursor.executemany(
                    "UPDATE products SET price = %s WHERE id = %s",
                    updates
                )
                conn.commit()
                print(f"修复了 {len(updates)} 条异常价格记录")

23. 复杂查询优化

23.1 分页查询优化

避免使用LIMIT offset：

python复制def get_products_optimized(last_id=0, limit=100):
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute("""
                SELECT id, name, price
                FROM products
                WHERE id > %s
                ORDER BY id
                LIMIT %s
            """, (last_id, limit))
            results = cursor.fetchall()
            
            # 返回结果及最后ID用于下次查询
            return {
                'data': results,
                'next_id': results[-1]['id'] if results else last_id
            }

23.2 大数据量导出

流式处理避免内存溢出：

python复制def export_large_table(table_name, output_file):
    with get_connection() as conn:
        with conn.cursor() as cursor:
            # 使用SSCursor进行流式读取
            cursor = conn.cursor(pymysql.cursors.SSCursor)
            cursor.execute(f"SELECT * FROM {table_name}")
            
            with open(output_file, 'w') as f:
                writer = csv.writer(f)
                # 写入标题行
                writer.writerow([col[0] for col in cursor.description])
                
                # 分批读取
                batch_size = 1000
                while True:
                    rows = cursor.fetchmany(batch_size)
                    if not rows:
                        break
                    writer.writerows(rows)
                    print(f"已导出 {len(rows)} 行")

24. 地理空间数据处理

24.1 GIS数据存储与查询

MySQL的空间扩展使用：

python复制def find_nearby_stores(lat, lng, radius_km):
    with get_connection() as conn:
        with conn.cursor() as cursor:
            cursor.execute("""
                SELECT 
                    id, 
                    name,
                    ST_Distance_Sphere(
                        point(%s, %s),
                        location
                    ) / 1000 AS distance_km
                FROM stores
                WHERE ST_Distance_Sphere(
                    point(%s, %s),
                    location
                ) <= %s * 1000
                ORDER BY distance_km
                LIMIT 10
            """, (lng, lat, lng, lat, radius_km))
            return cursor.fetchall()

24.2 空间索引优化

为GIS数据创建空间索引：

python复制def optimize_geo_queries():
    with get_connection() as conn:
        with conn.cursor() as cursor:
            # 检查是否已有空间索引
            cursor.execute("""
                SELECT INDEX_NAME 
                FROM INFORMATION_SCHEMA.STATISTICS
                WHERE TABLE_NAME = 'stores' 
                AND INDEX_TYPE = 'SPATIAL'
            """)
            if not cursor.fetchone():
                print("创建空间索引...")
                cursor.execute("ALTER TABLE stores ADD SPATIAL INDEX(location)")
                conn.commit()

25. 全文检索实现

25.1 全文索引配置

MySQL全文搜索设置：

python复制def setup_fulltext_index():
    with get_connection() as conn:
        with conn.cursor() as cursor:
            # 检查表引擎是否为InnoDB（5.6+支持）
            cursor.execute("""
                SELECT ENGINE 
                FROM INFORMATION_SCHEMA.TABLES
                WHERE TABLE_NAME = 'products'
            """)
            if cursor.fetchone()['ENGINE'] != 'InnoDB':
                print("需要转换为InnoDB引擎以支持全文索引")
                cursor.execute("ALTER TABLE products ENGINE=InnoDB")
            
            # 添加全文索引
            cursor.execute("""
                ALTER TABLE products