Python数据库操作全指南：从基础到ORM框架

1. Python数据库操作模块全景概览

作为一门广泛应用于数据处理领域的编程语言，Python生态中存在着丰富多样的数据库连接模块。这些模块大致可分为三类：关系型数据库驱动（如MySQL、PostgreSQL）、NoSQL数据库驱动（如MongoDB、Redis）以及ORM框架（如SQLAlchemy、Django ORM）。每个模块都有其特定的适用场景和性能特征，开发者需要根据项目需求进行合理选择。

在Python 3.x环境中，大多数数据库模块都遵循PEP 249规范（Python数据库API规范v2.0），这为不同数据库提供了统一的接口标准。规范中定义了Connection、Cursor等核心对象，以及execute()、fetchone()等标准方法，使得不同数据库间的代码迁移成本大大降低。

提示：虽然PEP 249提供了接口标准，但各数据库驱动在实现细节上仍存在差异，特别是在事务处理、连接池管理和数据类型映射等方面。

2. 关系型数据库操作实战

2.1 MySQL数据库连接与操作

MySQL作为最流行的开源关系型数据库，在Python中有多个驱动选择。mysql-connector-python是MySQL官方提供的纯Python驱动，而PyMySQL则是社区维护的替代方案。以下是使用PyMySQL连接MySQL的典型示例：

python复制import pymysql

# 创建数据库连接
connection = pymysql.connect(
    host='localhost',
    user='username',
    password='password',
    database='test_db',
    charset='utf8mb4',
    cursorclass=pymysql.cursors.DictCursor
)

try:
    with connection.cursor() as cursor:
        # 执行SQL查询
        sql = "SELECT * FROM users WHERE email=%s"
        cursor.execute(sql, ('user@example.com',))
        
        # 获取结果
        result = cursor.fetchone()
        print(result)
finally:
    connection.close()

在实际项目中，直接管理数据库连接会带来资源泄漏风险。更推荐使用连接池技术，如DBUtils或SQLAlchemy提供的连接池功能：

python复制from dbutils.pooled_db import PooledDB
import pymysql

pool = PooledDB(
    creator=pymysql,
    maxconnections=10,
    host='localhost',
    user='username',
    password='password',
    database='test_db'
)

# 从连接池获取连接
connection = pool.connection()
try:
    # 执行数据库操作...
finally:
    connection.close()  # 实际是返回到连接池

2.2 SQLite轻量级数据库集成

SQLite作为嵌入式数据库，在Python中有着原生支持。其最大的优势在于零配置和单文件存储，非常适合小型应用或原型开发：

python复制import sqlite3

# 连接数据库（不存在则自动创建）
conn = sqlite3.connect('example.db')

# 创建游标对象
cursor = conn.cursor()

# 创建表
cursor.execute('''CREATE TABLE IF NOT EXISTS stocks
             (date text, trans text, symbol text, qty real, price real)''')

# 插入数据
cursor.execute("INSERT INTO stocks VALUES ('2023-06-01','BUY','RHAT',100,35.14)")

# 提交事务
conn.commit()

# 查询数据
for row in cursor.execute("SELECT * FROM stocks"):
    print(row)

conn.close()

SQLite虽然轻量，但在并发写入性能上存在局限。当应用需要高并发写入时，应考虑使用客户端-服务器架构的数据库如PostgreSQL。

3. NoSQL数据库操作详解

3.1 MongoDB文档数据库操作

MongoDB作为领先的文档型数据库，在Python中主要通过PyMongo驱动进行操作。与关系型数据库不同，MongoDB使用BSON（二进制JSON）格式存储数据，提供了更灵活的数据模型：

python复制from pymongo import MongoClient
from pymongo.errors import ConnectionFailure

try:
    # 创建MongoDB客户端
    client = MongoClient('mongodb://localhost:27017/')
    
    # 选择数据库和集合
    db = client['example_db']
    collection = db['users']
    
    # 插入文档
    user_data = {
        "name": "John Doe",
        "email": "john@example.com",
        "roles": ["user", "admin"],
        "metadata": {
            "created_at": "2023-06-01",
            "last_login": "2023-06-15"
        }
    }
    result = collection.insert_one(user_data)
    print(f"插入文档ID: {result.inserted_id}")
    
    # 查询文档
    for user in collection.find({"name": "John Doe"}):
        print(user)
        
except ConnectionFailure as e:
    print(f"MongoDB连接失败: {e}")

MongoDB的聚合管道功能特别适合复杂的数据分析场景：

python复制pipeline = [
    {"$match": {"status": "A"}},
    {"$group": {
        "_id": "$cust_id",
        "total": {"$sum": "$amount"}
    }},
    {"$sort": {"total": -1}}
]

results = collection.aggregate(pipeline)
for doc in results:
    print(doc)

3.2 Redis内存数据库应用

Redis作为高性能的键值存储，常被用作缓存、消息队列和会话存储。Python通过redis-py模块与Redis交互：

python复制import redis

# 创建Redis连接
r = redis.Redis(host='localhost', port=6379, db=0)

# 字符串操作
r.set('foo', 'bar')
value = r.get('foo')
print(value.decode('utf-8'))  # 输出: bar

# 哈希操作
r.hset('user:1000', mapping={
    'name': 'John',
    'age': '30',
    'email': 'john@example.com'
})
user = r.hgetall('user:1000')
print(user)  # 输出: {b'name': b'John', b'age': b'30', b'email': b'john@example.com'}

# 发布/订阅模式
pubsub = r.pubsub()
pubsub.subscribe('news')

# 在另一个客户端发布消息: r.publish('news', 'hello world')
for message in pubsub.listen():
    if message['type'] == 'message':
        print(message['data'].decode('utf-8'))

4. ORM框架高级应用

4.1 SQLAlchemy核心功能解析

SQLAlchemy是Python中最强大的ORM框架之一，提供了从基础SQL操作到高级ORM映射的全套解决方案。其核心架构分为两部分：Core（SQL表达式语言）和ORM（对象关系映射）。

使用SQLAlchemy ORM定义数据模型的典型方式：

python复制from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

Base = declarative_base()

class User(Base):
    __tablename__ = 'users'
    
    id = Column(Integer, primary_key=True)
    name = Column(String(50))
    email = Column(String(120), unique=True)
    
    def __repr__(self):
        return f"<User(name='{self.name}', email='{self.email}')>"

# 创建引擎和会话
engine = create_engine('sqlite:///example.db')
Base.metadata.create_all(engine)

Session = sessionmaker(bind=engine)
session = Session()

# 添加新用户
new_user = User(name='Alice', email='alice@example.com')
session.add(new_user)
session.commit()

# 查询用户
user = session.query(User).filter_by(name='Alice').first()
print(user)

SQLAlchemy的关系功能支持多种关联类型：

python复制from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship

class Address(Base):
    __tablename__ = 'addresses'
    
    id = Column(Integer, primary_key=True)
    email_address = Column(String(50), nullable=False)
    user_id = Column(Integer, ForeignKey('users.id'))
    
    user = relationship("User", back_populates="addresses")

User.addresses = relationship("Address", order_by=Address.id, back_populates="user")

# 现在可以方便地操作关联对象
user = User(name='Bob', email='bob@example.com')
user.addresses = [
    Address(email_address='bob@work.com'),
    Address(email_address='bob@home.com')
]
session.add(user)
session.commit()

4.2 Django ORM特性与应用

Django自带的ORM以其简洁的API和强大的功能著称，特别适合Web开发场景。定义模型：

python复制from django.db import models

class Author(models.Model):
    name = models.CharField(max_length=100)
    email = models.EmailField(unique=True)
    
    def __str__(self):
        return self.name

class Book(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)
    publish_date = models.DateField()
    price = models.DecimalField(max_digits=5, decimal_places=2)
    
    class Meta:
        indexes = [
            models.Index(fields=['title']),
            models.Index(fields=['author', 'publish_date']),
        ]

Django ORM提供了丰富的查询API：

python复制# 基本查询
books = Book.objects.filter(
    publish_date__year=2023
).exclude(
    price__lt=20
).order_by('-publish_date')

# 聚合查询
from django.db.models import Avg, Max, Min
stats = Book.objects.aggregate(
    avg_price=Avg('price'),
    max_price=Max('price'),
    min_price=Min('price')
)

# 关联查询
authors = Author.objects.prefetch_related('book_set').filter(
    book__title__icontains='python'
)

5. 性能优化与高级特性

5.1 批量操作与事务管理

数据库操作的性能瓶颈往往出现在网络IO上，批量操作可以显著提高性能。以下是几种常见数据库的批量操作示例：

MySQL批量插入：

python复制data = [
    ('Product A', 10.99),
    ('Product B', 24.50),
    ('Product C', 35.75)
]

# 使用executemany批量插入
cursor.executemany("INSERT INTO products (name, price) VALUES (%s, %s)", data)
connection.commit()

MongoDB批量写入：

python复制from pymongo import InsertOne

requests = [
    InsertOne({"name": "Item 1", "value": 10}),
    InsertOne({"name": "Item 2", "value": 20}),
    InsertOne({"name": "Item 3", "value": 30})
]

result = collection.bulk_write(requests)
print(result.inserted_count)

事务管理是保证数据一致性的关键。SQL数据库通常支持ACID事务：

python复制try:
    connection.begin()
    # 执行多个SQL操作...
    connection.commit()
except Exception as e:
    connection.rollback()
    print(f"事务失败: {e}")

MongoDB从4.0版本开始支持多文档事务：

python复制with client.start_session() as session:
    session.start_transaction()
    try:
        collection_one.insert_one({"key": "value"}, session=session)
        collection_two.delete_one({"key": "value"}, session=session)
        session.commit_transaction()
    except Exception as e:
        session.abort_transaction()
        print(f"事务中止: {e}")

5.2 连接池与异步IO

数据库连接是昂贵的资源，连接池可以显著提高应用性能。SQLAlchemy的连接池配置示例：

python复制from sqlalchemy import create_engine

engine = create_engine(
    "mysql+pymysql://user:password@localhost/dbname",
    pool_size=10,
    max_overflow=20,
    pool_recycle=3600,
    pool_pre_ping=True
)

异步IO可以进一步提高高并发场景下的性能。使用aiomysql和aiopg进行异步MySQL/PostgreSQL操作：

python复制import asyncio
import aiomysql

async def fetch_data():
    conn = await aiomysql.connect(
        host='localhost',
        user='user',
        password='password',
        db='test'
    )
    
    async with conn.cursor() as cursor:
        await cursor.execute("SELECT * FROM users")
        result = await cursor.fetchall()
        print(result)
    
    conn.close()

asyncio.run(fetch_data())

对于MongoDB，motor提供了异步支持：

python复制import motor.motor_asyncio

client = motor.motor_asyncio.AsyncIOMotorClient('mongodb://localhost:27017')
db = client['test_db']

async def insert_document():
    result = await db.users.insert_one({"name": "Async User"})
    print(f"插入文档ID: {result.inserted_id}")

asyncio.run(insert_document())

5.3 高级查询技巧

SQLAlchemy混合属性与关联代理：

python复制from sqlalchemy.ext.hybrid import hybrid_property
from sqlalchemy.ext.associationproxy import association_proxy

class User(Base):
    __tablename__ = 'users'
    
    id = Column(Integer, primary_key=True)
    first_name = Column(String(50))
    last_name = Column(String(50))
    
    @hybrid_property
    def full_name(self):
        return f"{self.first_name} {self.last_name}"
    
    # 允许在查询中使用
    @full_name.expression
    def full_name(cls):
        return cls.first_name + ' ' + cls.last_name

class Order(Base):
    __tablename__ = 'orders'
    
    id = Column(Integer, primary_key=True)
    user_id = Column(Integer, ForeignKey('users.id'))
    user = relationship("User")
    
    # 通过代理直接访问用户属性
    user_name = association_proxy('user', 'full_name')

Django ORM注解与子查询：

python复制from django.db.models import Count, F, Subquery, OuterRef

# 使用annotate添加计算字段
authors = Author.objects.annotate(
    book_count=Count('book')
).filter(
    book_count__gt=5
)

# 使用F()引用字段值
Book.objects.update(price=F('price') * 1.1)

# 复杂子查询
newest = Book.objects.filter(
    author=OuterRef('pk')
).order_by('-publish_date')
authors = Author.objects.annotate(
    newest_book_title=Subquery(newest.values('title')[:1])
)

6. 实战：构建数据库工具类

结合上述知识，我们可以构建一个实用的数据库工具类，封装常见操作：

python复制import contextlib
from typing import Iterator, Any, Dict, List
import pymysql
from pymysql.cursors import DictCursor

class MySQLDatabase:
    def __init__(self, config: Dict[str, Any]):
        self.config = config
    
    @contextlib.contextmanager
    def get_connection(self) -> Iterator[pymysql.Connection]:
        conn = pymysql.connect(
            host=self.config['host'],
            user=self.config['user'],
            password=self.config['password'],
            database=self.config['database'],
            charset='utf8mb4',
            cursorclass=DictCursor
        )
        try:
            yield conn
        finally:
            conn.close()
    
    def execute_query(self, sql: str, params=None) -> List[Dict[str, Any]]:
        with self.get_connection() as conn:
            with conn.cursor() as cursor:
                cursor.execute(sql, params or ())
                return cursor.fetchall()
    
    def execute_update(self, sql: str, params=None) -> int:
        with self.get_connection() as conn:
            with conn.cursor() as cursor:
                affected_rows = cursor.execute(sql, params or ())
                conn.commit()
                return affected_rows
    
    def batch_insert(self, table: str, columns: List[str], data: List[tuple]) -> int:
        placeholders = ', '.join(['%s'] * len(columns))
        columns_str = ', '.join(columns)
        sql = f"INSERT INTO {table} ({columns_str}) VALUES ({placeholders})"
        
        with self.get_connection() as conn:
            with conn.cursor() as cursor:
                affected_rows = cursor.executemany(sql, data)
                conn.commit()
                return affected_rows

# 使用示例
db_config = {
    'host': 'localhost',
    'user': 'root',
    'password': 'password',
    'database': 'test_db'
}

db = MySQLDatabase(db_config)
users = db.execute_query("SELECT * FROM users WHERE age > %s", (18,))
print(users)

对于MongoDB，可以构建类似的工具类：

python复制from typing import Dict, Any, List, Optional
from pymongo import MongoClient
from pymongo.collection import Collection
from pymongo.database import Database

class MongoDBClient:
    def __init__(self, connection_string: str, db_name: str):
        self.client = MongoClient(connection_string)
        self.db = self.client[db_name]
    
    def get_collection(self, name: str) -> Collection:
        return self.db[name]
    
    def insert_document(self, collection: str, document: Dict[str, Any]) -> Any:
        col = self.get_collection(collection)
        return col.insert_one(document).inserted_id
    
    def find_documents(
        self,
        collection: str,
        query: Optional[Dict[str, Any]] = None,
        projection: Optional[Dict[str, Any]] = None,
        limit: int = 0
    ) -> List[Dict[str, Any]]:
        col = self.get_collection(collection)
        return list(col.find(query or {}, projection or {}).limit(limit))
    
    def update_documents(
        self,
        collection: str,
        query: Dict[str, Any],
        update: Dict[str, Any],
        upsert: bool = False
    ) -> int:
        col = self.get_collection(collection)
        result = col.update_many(query, {'$set': update}, upsert=upsert)
        return result.modified_count

# 使用示例
mongo_client = MongoDBClient("mongodb://localhost:27017/", "test_db")
user_id = mongo_client.insert_document("users", {"name": "John", "age": 30})
users = mongo_client.find_documents("users", {"age": {"$gt": 25}})
print(users)

在实际项目中，这些工具类可以根据需求进一步扩展，添加连接池、重试机制、日志记录等功能。