随缘惜情1. MySQL数据脚本概述与核心价值
MySQL数据脚本是数据库开发中最基础却最实用的技能之一。作为从业12年的数据库工程师，我处理过上千个数据脚本项目，从简单的表结构创建到复杂的ETL流程，脚本始终是效率提升的关键。不同于图形化工具操作，脚本化方式具有可追溯、可复用、可版本控制的天然优势。
数据脚本的核心应用场景包括：
- 数据库初始化：新建项目时快速构建表结构、视图、存储过程
- 数据迁移：在不同环境间同步数据结构与内容
- 批量操作：高效执行重复性数据变更
- 自动化部署：与CI/CD流程集成实现数据库变更自动化
新手常犯的错误是直接使用Navicat等工具导出SQL后不经优化就直接使用。实际上，专业的MySQL脚本需要遵循以下原则：
- 显式声明字符集（推荐utf8mb4）
- 包含完整的注释说明
- 使用事务保证原子性
- 处理异常情况
- 考虑执行环境差异
重要提示：永远不要在生成环境直接执行未经测试的脚本，建议先在沙箱环境验证
需要模型API调用？ 免费领10W Token，多模型网关一键接入 Claude、DeepSeek 等主流模型。
2. MySQL脚本核心组件详解
2.1 基础结构定义
规范的MySQL脚本通常包含以下结构：
sql复制-- 脚本说明：用户表初始化脚本
-- 作者：YourName
-- 创建日期：2023-07-20
-- 版本：v1.0
SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;
-- 数据库创建
CREATE DATABASE IF NOT EXISTS `sample_db`
DEFAULT CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci;
USE `sample_db`;
-- 表结构定义
DROP TABLE IF EXISTS `users`;
CREATE TABLE `users` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`username` varchar(50) NOT NULL COMMENT '登录账号',
`password` varchar(100) NOT NULL COMMENT '加密密码',
`email` varchar(100) DEFAULT NULL COMMENT '电子邮箱',
`created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_username` (`username`),
KEY `idx_email` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='用户基本信息表';
关键设计要点：
- 字符集统一使用utf8mb4以支持完整Unicode（包括emoji）
- 显示设置FOREIGN_KEY_CHECKS避免外键约束报错
- 表字段使用反引号包裹避免关键字冲突
- 为每个字段添加COMMENT说明
- 合理设置索引（主键、唯一索引、普通索引）
2.2 数据操作脚本
数据操作脚本需要特别注意事务处理和数据验证：
sql复制START TRANSACTION;
-- 用户数据初始化
INSERT INTO `users` (`username`, `password`, `email`)
VALUES
('admin', '$2a$10$xJwL5vW18Uz4B1ZyvYsZ.eqBZ7IjBHAjwXgTcQN7JzQGqTj4LQdmW', 'admin@example.com'),
('user1', '$2a$10$N.kcB6Q4UQdG5pWYVn5X.OVUJjQ9hCQH7rYd6LsJfLs5JQ7X1zO7C', 'user1@example.com');
-- 验证插入数量
SELECT ROW_COUNT() INTO @insert_count;
IF @insert_count != 2 THEN
ROLLBACK;
SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = '用户数据初始化失败';
END IF;
COMMIT;
事务使用技巧：
- 明确划分事务边界（START TRANSACTION...COMMIT/ROLLBACK）
- 关键操作后验证ROW_COUNT()
- 使用SIGNAL抛出可读性强的错误信息
- 大事务拆分为小事务（建议单事务不超过1000行）
3. 高级脚本开发技巧
3.1 动态SQL生成
对于需要批量操作的情况，可以使用存储过程生成动态SQL：
sql复制DELIMITER //
CREATE PROCEDURE `batch_insert_users`(IN count INT)
BEGIN
DECLARE i INT DEFAULT 1;
SET @sql = 'INSERT INTO users (username, password) VALUES ';
WHILE i <= count DO
SET @sql = CONCAT(@sql,
IF(i>1, ',', ''),
'(',
QUOTE(CONCAT('user', i)), ',',
QUOTE('$2a$10$defaultpasswordhash'),
')'
);
SET i = i + 1;
END WHILE;
PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END //
DELIMITER ;
-- 调用示例
CALL batch_insert_users(1000);
性能提示：大批量插入建议使用LOAD DATA INFILE替代INSERT
3.2 数据迁移脚本模板
典型的数据迁移脚本应包含以下要素：
sql复制-- 数据迁移日志表
CREATE TABLE IF NOT EXISTS `migration_log` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`migration_name` varchar(255) NOT NULL,
`start_time` datetime NOT NULL,
`end_time` datetime DEFAULT NULL,
`status` enum('running','success','failed') NOT NULL,
`affected_rows` int(11) DEFAULT NULL,
`error_message` text DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- 迁移脚本示例
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
GET DIAGNOSTICS CONDITION 1 @sqlstate = RETURNED_SQLSTATE,
@errno = MYSQL_ERRNO, @text = MESSAGE_TEXT;
INSERT INTO `migration_log`
(`migration_name`, `start_time`, `status`, `error_message`)
VALUES
('202307_user_data_migration', NOW(), 'failed', CONCAT(@errno, ': ', @text));
ROLLBACK;
END;
START TRANSACTION;
INSERT INTO `migration_log`
(`migration_name`, `start_time`, `status`)
VALUES
('202307_user_data_migration', NOW(), 'running');
-- 实际迁移操作
INSERT INTO new_users (user_id, username, email)
SELECT id, username, email FROM legacy_users
WHERE registered = 1;
UPDATE `migration_log`
SET
`end_time` = NOW(),
`status` = 'success',
`affected_rows` = ROW_COUNT()
WHERE `migration_name` = '202307_user_data_migration';
COMMIT;
END;
4. 脚本优化与调试
4.1 性能优化策略
- 索引优化：
sql复制-- 创建索引前分析
EXPLAIN SELECT * FROM users WHERE email LIKE '%@example.com';
-- 添加覆盖索引
ALTER TABLE users ADD INDEX `idx_email_domain` ((SUBSTRING_INDEX(email, '@', -1)));
-- 验证索引效果
EXPLAIN SELECT * FROM users WHERE SUBSTRING_INDEX(email, '@', -1) = 'example.com';
- 批量操作优化：
sql复制-- 低效方式（逐行更新）
UPDATE products SET price = price * 1.1 WHERE category = 'electronics';
-- 高效方式（单语句完成）
UPDATE products
SET price = CASE
WHEN category = 'electronics' THEN price * 1.1
WHEN category = 'clothing' THEN price * 1.05
ELSE price
END;
4.2 调试技巧
- 使用SQL_DEBUG模式：
sql复制SET SESSION sql_mode = 'TRADITIONAL,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO';
SET @@session.sql_log_bin = 0; -- 临时禁用binlog
SET @@foreign_key_checks = 0; -- 临时禁用外键检查
- 脚本日志记录：
sql复制-- 创建临时日志表
CREATE TEMPORARY TABLE IF NOT EXISTS `script_log` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`log_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`message` text NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- 在脚本中插入日志
INSERT INTO `script_log` (`message`) VALUES ('开始处理用户数据');
5. 企业级实践方案
5.1 版本控制集成
推荐的项目目录结构：
code复制/database
/migrations
20230701000000_create_users_table.sql
20230702000000_add_user_avatar.sql
/seeds
dev
init_users.sql
prod
admin_users.sql
/functions
calculate_stats.sql
/procedures
monthly_report.sql
Git集成规范：
- 每个脚本文件包含头部注释说明变更目的
- 使用时间戳前缀保证迁移顺序（YYYYMMDDHHMMSS_description.sql）
- 禁止直接修改已提交的迁移脚本
5.2 自动化部署流程
典型CI/CD集成配置（以GitLab为例）：
yaml复制stages:
- deploy
db_migration:
stage: deploy
image: mysql:8.0
script:
- mysql --host=$DB_HOST --user=$DB_USER --password=$DB_PASSWORD
--port=$DB_PORT $DB_NAME < migrations/latest.sql
only:
- main
environment:
name: production
安全注意事项：
- 使用环境变量存储凭据
- 限制生产环境执行权限
- 实施变更评审流程
- 保留回滚脚本
6. 常见问题解决方案
6.1 字符集问题排查
典型错误现象：
code复制ERROR 1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x8D' for column 'emoji'
解决方案：
sql复制-- 检查当前字符集设置
SHOW VARIABLES LIKE 'character_set%';
SHOW VARIABLES LIKE 'collation%';
-- 修正字符集（需重建表）
ALTER TABLE `problem_table` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
6.2 大表变更策略
安全的大表结构变更流程：
- 创建新表（带新结构）
- 分批拷贝数据（使用WHERE条件分片）
- 建立完整索引
- 原子切换（RENAME TABLE）
sql复制-- 步骤1：创建新表
CREATE TABLE `users_new` LIKE `users`;
ALTER TABLE `users_new` ADD COLUMN `phone` varchar(20);
-- 步骤2：分批拷贝（每次1万条）
INSERT INTO `users_new` (`id`, `username`, `password`, `email`)
SELECT `id`, `username`, `password`, `email`
FROM `users`
WHERE `id` BETWEEN 1 AND 10000;
-- 步骤4：原子切换
RENAME TABLE `users` TO `users_old`, `users_new` TO `users`;
6.3 密码安全处理
避免在脚本中明文存储密码：
sql复制-- 不安全做法
INSERT INTO users (username, password) VALUES ('admin', '123456');
-- 正确做法（使用应用层hash）
INSERT INTO users (username, password)
VALUES ('admin', '$2a$10$N.kcB6Q4UQdG5pWYVn5X.OVUJjQ9hCQH7rYd6LsJfLs5JQ7X1zO7C');
-- 或者使用MySQL内置函数（需启用caching_sha2_password插件）
CREATE USER 'app_user'@'%' IDENTIFIED WITH caching_sha2_password BY 'StrongPassword123!';
7. 工具链推荐
7.1 开发工具
-
VS Code插件：
- MySQL：官方语法支持
- SQLTools：连接管理+智能提示
- Database Client：可视化操作
-
命令行工具：
- mycli：自动补全的MySQL客户端
- pgloader：异构数据库迁移
- sqldump：逻辑备份工具
7.2 校验工具
- 语法检查：
bash复制mysql -e "SOURCE script.sql" --force --show-warnings
- 差异比对：
bash复制mysqldiff --server1=user:pass@host1 --server2=user:pass@host2 db1:db2
- 执行计划分析：
sql复制EXPLAIN ANALYZE SELECT * FROM large_table WHERE create_date > '2023-01-01';
8. 性能基准测试
建立性能基准的推荐方法：
sql复制-- 创建测试表
CREATE TABLE `perf_test` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`data` varchar(255) DEFAULT NULL,
`create_time` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idx_create_time` (`create_time`)
) ENGINE=InnoDB;
-- 填充测试数据（100万行）
DELIMITER //
CREATE PROCEDURE `generate_test_data`(IN rows INT)
BEGIN
DECLARE i INT DEFAULT 1;
WHILE i <= rows DO
INSERT INTO `perf_test` (`data`)
VALUES (CONCAT('sample_data_', FLOOR(RAND()*1000)));
SET i = i + 1;
END WHILE;
END //
DELIMITER ;
CALL generate_test_data(1000000);
-- 执行基准测试
SET PROFILING = 1;
SELECT COUNT(*) FROM `perf_test` WHERE `create_time` > DATE_SUB(NOW(), INTERVAL 1 MONTH);
SHOW PROFILE;
关键指标监控：
- 查询响应时间
- 锁等待时间
- 临时表使用情况
- 文件排序操作
9. 安全最佳实践
9.1 权限控制原则
最小权限分配示例：
sql复制-- 应用账号权限
CREATE USER 'app_user'@'192.168.1.%' IDENTIFIED BY 'ComplexPassword123!';
GRANT SELECT, INSERT, UPDATE ON `app_db`.* TO 'app_user'@'192.168.1.%';
-- 报表账号权限
CREATE USER 'report_user'@'10.0.0.%' IDENTIFIED BY 'AnotherPassword456!';
GRANT SELECT ON `app_db`.report_views TO 'report_user'@'10.0.0.%';
-- 管理员权限
CREATE USER 'dba_admin'@'localhost' IDENTIFIED BY 'SuperSecure789!';
GRANT ALL PRIVILEGES ON *.* TO 'dba_admin'@'localhost' WITH GRANT OPTION;
9.2 敏感数据处理
加密存储方案：
sql复制-- 创建加密函数
CREATE FUNCTION `aes_encrypt`(p_text TEXT, p_key VARCHAR(32))
RETURNS TEXT DETERMINISTIC
BEGIN
RETURN TO_BASE64(AES_ENCRYPT(p_text, p_key));
END;
-- 查询时解密
CREATE FUNCTION `aes_decrypt`(p_text TEXT, p_key VARCHAR(32))
RETURNS TEXT DETERMINISTIC
BEGIN
RETURN AES_DECRYPT(FROM_BASE64(p_text), p_key);
END;
-- 使用示例
INSERT INTO `customers` (`name`, `phone_encrypted`)
VALUES ('张三', aes_encrypt('13800138000', 'encryption_key_123'));
SELECT `name`, aes_decrypt(`phone_encrypted`, 'encryption_key_123') AS phone
FROM `customers`;
10. 复杂场景解决方案
10.1 分库分表策略
水平分片路由表示例：
sql复制-- 分片规则表
CREATE TABLE `shard_rules` (
`shard_key` varchar(50) NOT NULL COMMENT '分片键值',
`shard_id` tinyint(4) NOT NULL COMMENT '物理分片ID',
`db_host` varchar(100) NOT NULL COMMENT '数据库主机',
`db_name` varchar(50) NOT NULL COMMENT '数据库名',
PRIMARY KEY (`shard_key`),
KEY `idx_shard_id` (`shard_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
-- 分片查询存储过程
DELIMITER //
CREATE PROCEDURE `query_user_shard`(IN p_user_id BIGINT)
BEGIN
DECLARE v_shard_id INT;
DECLARE v_db_host VARCHAR(100);
DECLARE v_db_name VARCHAR(50);
-- 计算分片ID（示例使用取模算法）
SET v_shard_id = MOD(p_user_id, 16);
-- 获取分片信息
SELECT db_host, db_name INTO v_db_host, v_db_name
FROM shard_rules
WHERE shard_id = v_shard_id
LIMIT 1;
-- 构建动态SQL
SET @sql = CONCAT('SELECT * FROM ', v_db_name, '.users WHERE id = ', p_user_id);
-- 准备并执行
PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END //
DELIMITER ;
10.2 数据归档方案
自动化归档实现：
sql复制-- 归档表结构（与源表相同）
CREATE TABLE `orders_archive` LIKE `orders`;
-- 归档存储过程
DELIMITER //
CREATE PROCEDURE `archive_old_data`(IN p_months INT)
BEGIN
DECLARE EXIT HANDLER FOR SQLEXCEPTION
BEGIN
GET DIAGNOSTICS CONDITION 1 @sqlstate = RETURNED_SQLSTATE,
@errno = MYSQL_ERRNO, @text = MESSAGE_TEXT;
INSERT INTO `archive_log` (`status`, `message`)
VALUES ('failed', CONCAT(@errno, ': ', @text));
ROLLBACK;
END;
START TRANSACTION;
-- 归档数据
INSERT INTO `orders_archive`
SELECT * FROM `orders`
WHERE `create_time` < DATE_SUB(NOW(), INTERVAL p_months MONTH);
-- 删除已归档数据
DELETE FROM `orders`
WHERE `create_time` < DATE_SUB(NOW(), INTERVAL p_months MONTH);
-- 记录日志
INSERT INTO `archive_log` (`status`, `affected_rows`)
VALUES ('success', ROW_COUNT());
COMMIT;
END //
DELIMITER ;
-- 创建定时事件
CREATE EVENT `auto_archive`
ON SCHEDULE EVERY 1 MONTH
STARTS CURRENT_TIMESTAMP
DO
CALL archive_old_data(12);
11. 监控与维护脚本
11.1 健康检查脚本
sql复制-- 数据库健康检查
SELECT
table_schema AS '数据库',
table_name AS '表名',
table_rows AS '行数',
ROUND(data_length/1024/1024, 2) AS '数据大小(MB)',
ROUND(index_length/1024/1024, 2) AS '索引大小(MB)',
ROUND((data_length+index_length)/1024/1024, 2) AS '总大小(MB)',
engine AS '引擎'
FROM
information_schema.tables
WHERE
table_schema NOT IN ('information_schema', 'mysql', 'performance_schema')
ORDER BY
(data_length+index_length) DESC;
-- 索引使用情况监控
SELECT
object_schema AS '数据库',
object_name AS '表名',
index_name AS '索引名',
COUNT_READ AS '读取次数',
COUNT_FETCH AS '取数据次数',
COUNT_INSERT AS '插入次数',
COUNT_UPDATE AS '更新次数',
COUNT_DELETE AS '删除次数'
FROM
performance_schema.table_io_waits_summary_by_index_usage
WHERE
index_name IS NOT NULL
ORDER BY
COUNT_READ DESC;
11.2 自动化维护脚本
sql复制-- 表优化脚本
SELECT CONCAT(
'OPTIMIZE TABLE `',
table_schema, '`.`',
table_name, '`;'
) AS optimize_commands
FROM information_schema.tables
WHERE
table_schema NOT IN ('information_schema', 'mysql', 'performance_schema')
AND engine = 'InnoDB'
AND data_free > 1024*1024*100 -- 碎片超过100MB
INTO OUTFILE '/tmp/optimize_commands.sql';
-- 索引重建脚本
SELECT CONCAT(
'ALTER TABLE `',
table_schema, '`.`',
table_name,
'` ENGINE=InnoDB;'
) AS rebuild_commands
FROM information_schema.tables
WHERE
table_schema = 'your_database'
AND engine = 'InnoDB'
INTO OUTFILE '/tmp/rebuild_commands.sql';
12. 灾难恢复方案
12.1 备份策略实现
sql复制-- 全量备份脚本
mysqldump --single-transaction --master-data=2 --routines --triggers
--all-databases | gzip > /backups/mysql/full_$(date +%Y%m%d).sql.gz
-- 增量备份脚本
# 获取binlog位置
mysql -e "SHOW MASTER STATUS" > /backups/mysql/binlog_pos_$(date +%Y%m%d).log
# 备份binlog
cp $(grep "log_bin_basename" /etc/mysql/my.cnf | cut -d= -f2)* /backups/mysql/
-- 自动清理旧备份
find /backups/mysql -type f -name "*.sql.gz" -mtime +30 -delete
find /backups/mysql -type f -name "mysql-bin.*" -mtime +7 -delete
12.2 恢复流程设计
sql复制-- 全量恢复
zcat /backups/mysql/full_20230701.sql.gz | mysql
-- 时间点恢复
# 找出需要应用的binlog
mysqlbinlog --start-datetime="2023-07-01 00:00:00" \
--stop-datetime="2023-07-02 12:00:00" \
/var/lib/mysql/mysql-bin.000123 | mysql
-- 数据校验脚本
SELECT
(SELECT COUNT(*) FROM production.users) AS prod_count,
(SELECT COUNT(*) FROM restored.users) AS restored_count,
(SELECT COUNT(*) FROM production.users p LEFT JOIN restored.users r
ON p.id = r.id WHERE r.id IS NULL) AS missing_records;
13. 性能调优实战
13.1 慢查询优化
sql复制-- 启用慢查询日志
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1; -- 超过1秒的查询
SET GLOBAL log_queries_not_using_indexes = 'ON';
-- 分析慢查询
pt-query-digest /var/lib/mysql/mysql-slow.log
-- 优化案例：大分页查询
-- 原始低效查询
SELECT * FROM large_table ORDER BY id LIMIT 1000000, 10;
-- 优化方案1：使用覆盖索引
SELECT * FROM large_table
WHERE id >= (SELECT id FROM large_table ORDER BY id LIMIT 1000000, 1)
ORDER BY id LIMIT 10;
-- 优化方案2：使用游标分页
SELECT * FROM large_table
WHERE id > :last_id
ORDER BY id LIMIT 10;
13.2 连接池配置
推荐配置参数：
ini复制[mysqld]
# 连接相关
max_connections = 500
wait_timeout = 300
interactive_timeout = 300
# 内存配置
key_buffer_size = 256M
innodb_buffer_pool_size = 4G
innodb_log_file_size = 512M
query_cache_size = 0 # MySQL 8.0已移除
# 并发控制
innodb_thread_concurrency = 0
thread_cache_size = 100
table_open_cache = 2000
应用层连接池配置（以HikariCP为例）：
properties复制# 连接池大小
maximumPoolSize=20
minimumIdle=5
# 超时设置
connectionTimeout=30000
idleTimeout=600000
maxLifetime=1800000
# 健康检查
connectionTestQuery=SELECT 1
healthCheckRegistry=com.zaxxer.hikari.metrics.prometheus.PrometheusMetricsTracker
14. 高级特性应用
14.1 窗口函数实战
sql复制-- 销售排名分析
SELECT
product_id,
sale_date,
amount,
SUM(amount) OVER (PARTITION BY product_id ORDER BY sale_date) AS running_total,
RANK() OVER (PARTITION BY YEAR(sale_date), MONTH(sale_date) ORDER BY amount DESC) AS monthly_rank,
LAG(amount, 1) OVER (PARTITION BY product_id ORDER BY sale_date) AS prev_day_sales
FROM sales
WHERE sale_date BETWEEN '2023-01-01' AND '2023-12-31';
-- 同比环比计算
WITH monthly_sales AS (
SELECT
product_id,
YEAR(sale_date) AS year,
MONTH(sale_date) AS month,
SUM(amount) AS total
FROM sales
GROUP BY product_id, YEAR(sale_date), MONTH(sale_date)
)
SELECT
product_id,
CONCAT(year, '-', LPAD(month, 2, '0')) AS year_month,
total,
total - LAG(total, 1) OVER (PARTITION BY product_id ORDER BY year, month) AS mom_diff,
total - LAG(total, 12) OVER (PARTITION BY product_id ORDER BY year, month) AS yoy_diff
FROM monthly_sales;
14.2 JSON功能应用
sql复制-- JSON字段操作
ALTER TABLE products ADD COLUMN attributes JSON;
-- 插入JSON数据
UPDATE products SET attributes = JSON_OBJECT(
'color', 'red',
'size', 'XL',
'tags', JSON_ARRAY('new', 'sale'),
'specs', JSON_OBJECT('weight', 1.5, 'material', 'cotton')
) WHERE id = 1001;
-- 查询JSON字段
SELECT
id,
name,
attributes->>'$.color' AS color,
JSON_EXTRACT(attributes, '$.specs.weight') AS weight,
JSON_CONTAINS(attributes->>'$.tags', '"sale"') AS is_on_sale
FROM products
WHERE attributes->>'$.color' = 'red';
-- 更新JSON字段
UPDATE products
SET attributes = JSON_SET(
attributes,
'$.color', 'blue',
'$.tags', JSON_ARRAY_APPEND(attributes->>'$.tags', '$', 'featured')
)
WHERE id = 1001;
15. 云数据库实践
15.1 AWS RDS最佳实践
sql复制-- 创建RDS实例参数组
CREATE DATABASE `app_prod` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- 配置参数组
CALL mysql.rds_set_configuration('binlog retention hours', 24);
-- 创建只读副本账号
CREATE USER 'readonly_user'@'%' IDENTIFIED BY 'ReadOnlyPass123!';
GRANT SELECT ON app_prod.* TO 'readonly_user'@'%';
-- 跨区域复制配置
CALL mysql.rds_set_external_master(
'source.rds.amazonaws.com',
3306,
'repl_user',
'ReplPass123!',
'mysql-bin-changelog.123',
107,
1
);
-- 启动复制
CALL mysql.rds_start_replication;
15.2 阿里云RDS优化
sql复制-- 使用连接池代理
SET @proxy = 'proxysql-cluster.proxy.rds.aliyuncs.com';
SET @user = 'proxy_user';
SET @pass = 'ProxyPass123!';
-- 读写分离配置
-- 写节点
INSERT INTO mysql_servers(hostgroup_id,hostname,port) VALUES (10,'primary.rds.aliyuncs.com',3306);
-- 读节点
INSERT INTO mysql_servers(hostgroup_id,hostname,port) VALUES (20,'replica1.rds.aliyuncs.com',3306);
INSERT INTO mysql_servers(hostgroup_id,hostname,port) VALUES (20,'replica2.rds.aliyuncs.com',3306);
-- 规则配置
INSERT INTO mysql_query_rules (rule_id,active,match_pattern,destination_hostgroup,apply)
VALUES
(1,1,'^SELECT.*FOR UPDATE',10,1),
(2,1,'^SELECT',20,1),
(3,1,'^INSERT',10,1),
(4,1,'^UPDATE',10,1),
(5,1,'^DELETE',10,1);
16. 数据仓库集成
16.1 ETL流程实现
sql复制-- 增量抽取存储过程
DELIMITER //
CREATE PROCEDURE `incremental_etl`(IN p_last_etl_time DATETIME)
BEGIN
-- 创建临时表存储增量数据
CREATE TEMPORARY TABLE IF NOT EXISTS `temp_incremental` (
`id` INT,
`change_type` ENUM('insert','update','delete'),
`change_time` DATETIME,
PRIMARY KEY (`id`)
);
-- 识别变更记录
INSERT INTO `temp_incremental`
SELECT id, 'insert', create_time
FROM source_table
WHERE create_time > p_last_etl_time;
INSERT INTO `temp_incremental`
SELECT id, 'update', update_time
FROM source_table
WHERE update_time > p_last_etl_time
AND create_time <= p_last_etl_time
ON DUPLICATE KEY UPDATE
change_type = 'update',
change_time = VALUES(change_time);
-- 同步到数据仓库
INSERT INTO dw_fact_table (id, col1, col2, etl_time)
SELECT s.id, s.col1, s.col2, NOW()
FROM source_table s
JOIN temp_incremental t ON s.id = t.id
WHERE t.change_type IN ('insert','update')
ON DUPLICATE KEY UPDATE
col1 = VALUES(col1),
col2 = VALUES(col2),
etl_time = VALUES(etl_time);
-- 记录日志
INSERT INTO etl_log (start_time, end_time, processed_count)
VALUES (p_last_etl_time, NOW(), (SELECT COUNT(*) FROM temp_incremental));
END //
DELIMITER ;
16.2 数据质量检查
sql复制-- 数据质量验证脚本
SELECT
'orders' AS table_name,
COUNT(*) AS total_rows,
SUM(CASE WHEN order_date IS NULL THEN 1 ELSE 0 END) AS null_dates,
SUM(CASE WHEN amount <= 0 THEN 1 ELSE 0 END) AS invalid_amounts,
SUM(CASE WHEN customer_id NOT IN (SELECT id FROM customers) THEN 1 ELSE 0 END) AS orphan_records
FROM orders
UNION ALL
SELECT
'customers' AS table_name,
COUNT(*) AS total_rows,
SUM(CASE WHEN registration_date IS NULL THEN 1 ELSE 0 END) AS null_dates,
SUM(CASE WHEN email NOT LIKE '%@%.%' THEN 1 ELSE 0 END) AS invalid_emails,
0 AS orphan_records
FROM customers;
-- 定时质量检查事件
CREATE EVENT `daily_data_quality_check`
ON SCHEDULE EVERY 1 DAY
STARTS CURRENT_TIMESTAMP
DO
BEGIN
INSERT INTO data_quality_results
SELECT *, NOW() AS check_time
FROM (
-- 上述数据质量查询
) AS quality_metrics;
END;
17. 新版本特性应用
17.1 MySQL 8.0新功能
sql复制-- 公用表表达式(CTE)
WITH regional_sales AS (
SELECT region, SUM(amount) AS total_sales
FROM orders
GROUP BY region
), top_regions AS (
SELECT region
FROM regional_sales
WHERE total_sales > (SELECT SUM(total_sales)/10 FROM regional_sales)
)
SELECT
r.region,
p.category,
SUM(o.amount) AS category_sales
FROM orders o
JOIN products p ON o.product_id = p.id
JOIN regional_sales r ON o.region = r.region
WHERE r.region IN (SELECT region FROM top_regions)
GROUP BY r.region, p.category;
-- 窗口函数增强
SELECT
product_id,
sale_date,
amount,
FIRST_VALUE(amount) OVER (PARTITION BY product_id ORDER BY sale_date) AS first_sale,
NTH_VALUE(amount, 3) OVER (PARTITION BY product_id ORDER BY sale_date) AS third_sale,
NTILE(4) OVER (PARTITION BY YEAR(sale_date) ORDER BY amount DESC) AS quartile
FROM sales;
-- 原子DDL支持
START TRANSACTION;
CREATE TABLE new_table1 (id INT PRIMARY KEY);
CREATE TABLE new_table2 (id INT PRIMARY KEY);
-- 两条DDL作为一个原子单元执行
COMMIT;
17.2 性能优化器改进
sql复制-- 直方图统计
ANALYZE TABLE orders UPDATE HISTOGRAM ON amount, order_date WITH 100 BUCKETS;
-- 查看优化器跟踪
SET optimizer_trace="enabled=on";
SELECT * FROM orders WHERE amount > 1000 AND order_date > '2023-01-01';
SELECT * FROM information_schema.optimizer_trace;
SET optimizer_trace="enabled=off";
-- 资源组管理
CREATE RESOURCE GROUP report_group
TYPE = USER
VCPU = 2-3
THREAD_PRIORITY = 5;
SET RESOURCE GROUP report_group;
SELECT * FROM large_report_view;
SET RESOURCE GROUP DEFAULT;
18. 异构数据库集成
18.1 与MongoDB交互
sql复制-- 创建FEDERATED表连接MongoDB
CREATE TABLE `mongo_products` (
`_id` varchar(24) NOT NULL,
`name` varchar(100),
`price` decimal(10,2),
`category` varchar(50),
`attributes` json,
PRIMARY KEY (`_id`)
) ENGINE=FEDERATED
CONNECTION='mongodb://mongo_user:mongo_pass@mongo_host:27017/ecommerce/products';
-- 查询示例
SELECT p._id, p.name, p.price,
p.attributes->>'$.color' AS