Python与SQL交互实战：从基础连接到高级查询优化

1. Python与SQL交互的核心价值

在数据处理领域，Python和SQL堪称黄金搭档。SQL作为结构化查询语言，擅长高效管理关系型数据；而Python凭借其丰富的生态库，为数据操作提供了灵活的工具链。两者结合既能发挥数据库的性能优势，又可利用Python进行复杂的数据处理和业务逻辑实现。

我经手的项目中，90%的数据处理场景都会涉及Python与数据库的交互。最常见的需求包括：从数据库提取原始数据、将处理结果回写数据库、执行动态查询等。不同于ORM框架的封装，直接使用Python操作SQL能获得更精细的控制权，特别适合需要优化查询性能或处理特殊数据结构的场景。

2. 环境准备与基础连接

2.1 数据库驱动选择

Python连接不同数据库需要对应的驱动：

• MySQL/MariaDB：mysql-connector-python 或 PyMySQL
• PostgreSQL：psycopg2
• SQLite：内置sqlite3模块
• Oracle：cx_Oracle
• SQL Server：pyodbc

以MySQL为例，安装驱动：

bash复制pip install mysql-connector-python
# 或
pip install pymysql

2.2 建立数据库连接

创建连接对象的通用模式：

python复制import mysql.connector

config = {
    'host': 'localhost',
    'user': 'username',
    'password': 'password',
    'database': 'dbname',
    'port': 3306
}

try:
    conn = mysql.connector.connect(**config)
    print("连接成功")
except mysql.connector.Error as err:
    print(f"连接失败: {err}")
finally:
    if 'conn' in locals() and conn.is_connected():
        conn.close()

重要提示：始终使用try-except处理连接异常，并在finally块中确保连接关闭，避免资源泄漏

3. 基础查询操作详解

3.1 使用游标执行查询

游标(Cursor)是数据库操作的核心接口，典型查询流程：

python复制import sqlite3

def query_data(db_path):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    
    try:
        cursor.execute("SELECT id, name, price FROM products WHERE stock > ?", (0,))
        rows = cursor.fetchall()
        
        for row in rows:
            print(f"ID: {row[0]}, 名称: {row[1]}, 价格: {row[2]}")
            
    except sqlite3.Error as e:
        print(f"查询出错: {e}")
    finally:
        cursor.close()
        conn.close()

3.2 参数化查询的三种方式

防止SQL注入的必备技能：

1. 位置参数（推荐）：

python复制cursor.execute("SELECT * FROM users WHERE username = %s AND password = %s", (user, pwd))

2. 命名参数：

python复制cursor.execute(
    "SELECT * FROM users WHERE username = %(user)s AND active = %(active)s",
    {'user': 'admin', 'active': True}
)

3. 字符串格式化（危险！仅用于演示错误做法）：

python复制# 绝对不要这样写！存在SQL注入风险
cursor.execute(f"SELECT * FROM users WHERE username = '{user_input}'")

4. 高级查询技巧

4.1 分页查询优化

常见但容易出错的场景，推荐两种实现方式：

方法一：LIMIT-OFFSET（适合小数据量）

python复制page_size = 20
page_num = 3
cursor.execute(
    "SELECT * FROM orders ORDER BY create_time DESC LIMIT %s OFFSET %s",
    (page_size, (page_num-1)*page_size)
)

方法二：游标分页（大数据量优化）

python复制last_id = 100  # 上一页最后一条记录的ID
cursor.execute(
    "SELECT * FROM orders WHERE id > %s ORDER BY id ASC LIMIT %s",
    (last_id, page_size)
)

4.2 批量操作提升性能

处理大量数据时，批量操作比单条执行效率高10-100倍：

python复制data = [
    ('product1', 10.99, 100),
    ('product2', 20.50, 200),
    # ...更多数据
]

query = "INSERT INTO products (name, price, stock) VALUES (%s, %s, %s)"
cursor.executemany(query, data)
conn.commit()  # 不要忘记提交事务！

5. 结果集处理艺术

5.1 灵活获取查询结果

根据需求选择不同的获取方式：

python复制# 获取所有行（列表形式）
all_rows = cursor.fetchall()

# 逐行获取（内存友好）
row = cursor.fetchone()
while row:
    process(row)
    row = cursor.fetchone()

# 获取前N行
first_5 = cursor.fetchmany(5)

# 字典形式结果（需配置游标类型）
dict_cursor = conn.cursor(dictionary=True)
dict_cursor.execute("SELECT * FROM users")
for row in dict_cursor:
    print(row['username'], row['email'])

5.2 数据类型转换处理

数据库与Python类型自动转换的常见问题：

python复制# 处理DECIMAL类型（转为float可能丢失精度）
from decimal import Decimal
cursor.execute("SELECT price FROM products")
row = cursor.fetchone()
price = float(row[0]) if isinstance(row[0], Decimal) else row[0]

# 处理日期时间
from datetime import datetime
cursor.execute("SELECT create_time FROM orders")
create_time = cursor.fetchone()[0]
if isinstance(create_time, str):  # 某些驱动返回字符串
    create_time = datetime.strptime(create_time, '%Y-%m-%d %H:%M:%S')

6. 实战：封装通用查询工具类

基于经验的实用封装：

python复制class DBHelper:
    def __init__(self, db_config):
        self.config = db_config
        self.conn = None
        
    def __enter__(self):
        self.conn = mysql.connector.connect(**self.config)
        return self
    
    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.conn and self.conn.is_connected():
            self.conn.close()
    
    def query(self, sql, params=None, fetch_type='all'):
        """通用查询方法
        
        :param sql: SQL语句
        :param params: 参数元组/字典
        :param fetch_type: 'all'|'one'|'many'
        :return: 查询结果
        """
        cursor = self.conn.cursor(dictionary=True)
        try:
            cursor.execute(sql, params or ())
            
            if fetch_type == 'all':
                return cursor.fetchall()
            elif fetch_type == 'one':
                return cursor.fetchone()
            elif fetch_type == 'many':
                return cursor.fetchmany()
            else:
                raise ValueError("不支持的fetch_type")
                
        finally:
            cursor.close()

# 使用示例
with DBHelper(config) as db:
    products = db.query(
        "SELECT * FROM products WHERE category = %s",
        ('electronics',),
        fetch_type='all'
    )

7. 性能优化与问题排查

7.1 连接池管理

高频查询场景应使用连接池：

python复制from mysql.connector import pooling

connection_pool = pooling.MySQLConnectionPool(
    pool_name="mypool",
    pool_size=5,
    **config
)

def get_data():
    conn = connection_pool.get_connection()
    try:
        cursor = conn.cursor()
        cursor.execute("SELECT ...")
        return cursor.fetchall()
    finally:
        conn.close()  # 实际是返回到连接池

7.2 常见错误处理

python复制try:
    cursor.execute(bad_sql)
except mysql.connector.Error as err:
    if err.errno == 1064:  # 语法错误
        print("SQL语法错误:", err.msg)
    elif err.errno == 1146:  # 表不存在
        print("表不存在，请检查表名")
    elif err.errno == 2006:  # 服务器连接断开
        print("连接断开，尝试重连...")
        conn.reconnect()
    else:
        print("未知错误:", err)

8. 安全最佳实践

1. 永远使用参数化查询：这是防止SQL注入的第一道防线
2. 最小权限原则：数据库账号只授予必要权限
3. 敏感数据加密：密码等敏感信息不应明文存储
4. 连接字符串保护：不要将含密码的配置硬编码或提交到版本库
5. 定期审计SQL：检查是否有潜在注入风险的代码

python复制# 安全示例：使用环境变量存储配置
import os
from dotenv import load_dotenv

load_dotenv()

safe_config = {
    'host': os.getenv('DB_HOST'),
    'user': os.getenv('DB_USER'),
    'password': os.getenv('DB_PASS'),
    'database': os.getenv('DB_NAME')
}

9. 扩展应用场景

9.1 与Pandas集成

python复制import pandas as pd

def query_to_dataframe(sql, params=None):
    with mysql.connector.connect(**config) as conn:
        return pd.read_sql(sql, conn, params=params)

# 使用示例
df = query_to_dataframe(
    "SELECT date, SUM(amount) FROM sales GROUP BY date",
    fetch_type='all'
)
print(df.head())

9.2 异步查询实现

使用aiomysql进行异步操作：

python复制import asyncio
import aiomysql

async def async_query():
    conn = await aiomysql.connect(
        host='localhost', user='user',
        password='pwd', db='mydb'
    )
    
    async with conn.cursor() as cursor:
        await cursor.execute("SELECT * FROM large_table")
        result = await cursor.fetchall()
    
    conn.close()
    return result

# 调用
loop = asyncio.get_event_loop()
results = loop.run_until_complete(async_query())

10. 不同数据库的特殊处理

10.1 SQLite内存数据库

python复制import sqlite3

# 内存数据库（适合测试和小型应用）
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()

# 创建表
cursor.execute("CREATE TABLE test (id INTEGER PRIMARY KEY, name TEXT)")

# 特殊命令（SQLite特有）
cursor.execute("PRAGMA journal_mode=WAL")  # 设置写入日志模式

10.2 PostgreSQL JSON支持

python复制import psycopg2

conn = psycopg2.connect("dbname=test user=postgres")
cursor = conn.cursor()

# 查询JSON字段
cursor.execute("""
    SELECT data->>'name' FROM users 
    WHERE data @> '{"age": 30}'
""")

# 插入JSON
import json
data = {'name': 'John', 'age': 30}
cursor.execute(
    "INSERT INTO users (data) VALUES (%s)",
    (json.dumps(data),)
)

11. 调试与日志记录

11.1 查询日志记录

python复制import logging
from mysql.connector import connect

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(levelname)s - %(message)s'
)

def get_connection():
    conn = connect(**config)
    conn.set_trace_callback(
        lambda e: logging.debug("Executing: %s", e)
    )
    return conn

# 所有执行的SQL将被记录到日志

11.2 性能分析

使用Python内置分析工具：

python复制import cProfile

def test_query():
    conn = connect(**config)
    cursor = conn.cursor()
    for _ in range(1000):
        cursor.execute("SELECT * FROM products WHERE price > 10")
        cursor.fetchall()
    conn.close()

# 性能分析
cProfile.run('test_query()', sort='cumtime')

12. 实际项目经验分享

在电商项目中，我们处理过包含百万级商品数据的数据库查询。通过以下优化将响应时间从2秒降至200毫秒：

1. 添加合适索引：特别是WHERE和JOIN条件中的字段
2. 只查询必要字段：避免SELECT *
3. 使用连接池：减少连接建立开销
4. 应用层缓存：对热点数据使用Redis缓存
5. 分批处理：大结果集分多次获取

python复制# 优化后的分页查询示例
def get_products(page, per_page=50):
    with connection_pool.get_connection() as conn:
        cursor = conn.cursor()
        cursor.execute("""
            SELECT id, name, price 
            FROM products 
            WHERE status = 'active'
            ORDER BY sales DESC
            LIMIT %s OFFSET %s
        """, (per_page, (page-1)*per_page))
        return [
            dict(zip(('id','name','price'), row))
            for row in cursor.fetchall()
        ]

13. 现代ORM与原生SQL的平衡

虽然SQLAlchemy等ORM很强大，但某些场景仍需原生SQL：

python复制# 混合使用示例
from sqlalchemy import create_engine

engine = create_engine("mysql+mysqlconnector://user:pass@host/db")

# 复杂查询使用原生SQL
def get_sales_report(start_date):
    sql = """
        SELECT p.name, SUM(oi.quantity) as total_qty
        FROM order_items oi
        JOIN products p ON oi.product_id = p.id
        JOIN orders o ON oi.order_id = o.id
        WHERE o.order_date >= %s
        GROUP BY p.name
        ORDER BY total_qty DESC
    """
    with engine.connect() as conn:
        return conn.execute(sql, (start_date,)).fetchall()

# 简单CRUD使用ORM
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

class Product(Base):
    __tablename__ = 'products'
    id = Column(Integer, primary_key=True)
    name = Column(String(100))
    price = Column(Numeric(10,2))

14. 数据库迁移与版本控制

使用Alembic管理数据库变更：

python复制# alembic/env.py 配置
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from alembic import context

config = context.config
fileConfig(config.config_file_name)
target_metadata = Base.metadata

def run_migrations_online():
    connectable = engine_from_config(
        config.get_section(config.config_ini_section),
        prefix='sqlalchemy.',
        poolclass=pool.NullPool
    )
    
    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            target_metadata=target_metadata
        )
        
        with context.begin_transaction():
            context.run_migrations()

run_migrations_online()

15. 测试策略与实践

15.1 单元测试数据库操作

python复制import unittest
from unittest.mock import Mock, patch

class TestDBOperations(unittest.TestCase):
    @patch('mysql.connector.connect')
    def test_query_products(self, mock_connect):
        # 配置mock
        mock_cursor = Mock()
        mock_connect.return_value.cursor.return_value = mock_cursor
        mock_cursor.fetchall.return_value = [
            (1, 'Product1', 10.99),
            (2, 'Product2', 20.50)
        ]
        
        # 调用被测函数
        from myapp import get_products
        products = get_products()
        
        # 验证
        self.assertEqual(len(products), 2)
        self.assertEqual(products[0][1], 'Product1')
        mock_cursor.execute.assert_called_once_with(
            "SELECT id, name, price FROM products"
        )

15.2 集成测试使用测试数据库

python复制import tempfile
import sqlite3
import pytest

@pytest.fixture
def test_db():
    # 创建临时数据库
    fd, path = tempfile.mkstemp()
    conn = sqlite3.connect(path)
    yield conn
    conn.close()
    os.unlink(path)

def test_insert_product(test_db):
    test_db.execute("CREATE TABLE products (id INTEGER, name TEXT, price REAL)")
    test_db.execute(
        "INSERT INTO products VALUES (?, ?, ?)",
        (1, 'Test Product', 9.99)
    )
    row = test_db.execute("SELECT * FROM products").fetchone()
    assert row[1] == 'Test Product'

16. 性能对比：不同驱动与连接方式

通过基准测试比较不同实现的性能（单位：毫秒）：

测试环境：Python 3.9，本地MySQL 8.0，普通机械硬盘

17. 特殊场景处理技巧

17.1 大文本字段处理

python复制# 流式读取大文本
def read_large_text(id):
    conn = connect(**config)
    cursor = conn.cursor()
    
    try:
        cursor.execute(
            "SELECT large_text FROM documents WHERE id = %s",
            (id,)
        )
        
        # 分批读取
        text_buffer = []
        while True:
            chunk = cursor.fetchone()
            if not chunk:
                break
            text_buffer.append(chunk[0])
            
        return "".join(text_buffer)
    finally:
        cursor.close()
        conn.close()

17.2 二进制数据存取

python复制# 存储图片
def save_image(product_id, image_path):
    with open(image_path, 'rb') as f:
        image_data = f.read()
    
    with connect(**config) as conn:
        cursor = conn.cursor()
        cursor.execute(
            "UPDATE products SET image = %s WHERE id = %s",
            (image_data, product_id)
        )
        conn.commit()

# 读取图片
def get_image(product_id, save_path):
    with connect(**config) as conn:
        cursor = conn.cursor()
        cursor.execute(
            "SELECT image FROM products WHERE id = %s",
            (product_id,)
        )
        image_data = cursor.fetchone()[0]
        
    with open(save_path, 'wb') as f:
        f.write(image_data)

18. 数据库设计影响查询模式

良好的设计能显著简化Python代码：

python复制# 反例：糟糕的设计导致复杂查询
cursor.execute("""
    SELECT u.name, 
           GROUP_CONCAT(p.name SEPARATOR ', ') as products
    FROM users u
    LEFT JOIN orders o ON u.id = o.user_id
    LEFT JOIN order_items oi ON o.id = oi.order_id
    LEFT JOIN products p ON oi.product_id = p.id
    GROUP BY u.id
""")

# 正例：合理设计后简化查询
cursor.execute("SELECT name FROM users")
users = cursor.fetchall()

for user in users:
    cursor.execute("""
        SELECT p.name 
        FROM user_products up
        JOIN products p ON up.product_id = p.id
        WHERE up.user_id = %s
    """, (user[0],))
    products = [row[0] for row in cursor.fetchall()]

19. 跨数据库兼容性处理

python复制def get_products(conn, category):
    # 根据数据库类型调整SQL语法
    if isinstance(conn, sqlite3.Connection):
        sql = "SELECT * FROM products WHERE category = ?"
    elif 'mysql' in str(conn.__class__).lower():
        sql = "SELECT * FROM products WHERE category = %s"
    else:
        sql = "SELECT * FROM products WHERE category = :1"
    
    cursor = conn.cursor()
    cursor.execute(sql, (category,))
    return cursor.fetchall()

20. 资源管理与上下文协议

正确管理数据库资源的关键：

python复制from contextlib import contextmanager

@contextmanager
def db_connection(config):
    conn = None
    try:
        conn = connect(**config)
        yield conn
    except Exception as e:
        print(f"操作失败: {e}")
        if conn:
            conn.rollback()
        raise
    finally:
        if conn:
            conn.close()

# 使用示例
with db_connection(config) as conn:
    cursor = conn.cursor()
    cursor.execute("...")
    results = cursor.fetchall()
    # 不需要手动关闭连接