Flink消费Kafka集成DolphinScheduler：实时数据管道搭建与配置详解

2. Flink消费Kafka的核心代码与参数配置

2.1 Maven依赖与连接器选择

Flink消费Kafka需要引入连接器依赖，这一步选错版本后面全是坑。以我当时用的Flink 1.17.1为例，需要在pom.xml中加入flink-connector-kafka依赖。这里要特别强调，从Flink 1.15开始，Kafka连接器从flink-connector-kafka-0.x系列改名并统一为flink-connector-kafka，artifactId不再带版本号后缀。网上很多老教程还在用flink-connector-kafka-0.11这种写法，用在新版本项目上直接编译不过。

我实际使用的依赖配置如下：

xml复制<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka</artifactId>
    <version>1.17.1</version>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>1.17.1</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients</artifactId>
    <version>1.17.1</version>
    <scope>provided</scope>
</dependency>

这里有个重要的经验：flink-streaming-java和flink-clients这两个依赖在打包时要设置为provided，因为Flink集群环境里本身就有这些类，打进去反而会因为版本冲突引发各种奇怪问题。但flink-connector-kafka需要打成JAR包，因为默认的Flink发行版里并没有内置Kafka连接器。如果用了其他第三方依赖，比如连接数据库的JDBC驱动，同样需要打进去。

2.2 核心消费端代码实现

用DataStream API写一个消费Kafka、处理数据、再输出的任务，核心代码并不复杂。我以一个常见的实时订单数据清洗场景为例，代码结构大概是这样的：

java复制import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.connector.kafka.source.KafkaSource;
import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.ProcessFunction;
import org.apache.flink.util.Collector;

public class KafkaToConsoleJob {
    public static void main(String[] args) throws Exception {
        // 创建执行环境，开启Checkpoint
        Configuration config = new Configuration();
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(config);
        env.enableCheckpointing(60000);
        
        // 构建Kafka Source
        KafkaSource<String> source = KafkaSource.<String>builder()
                .setBootstrapServers("kafka-1:9092,kafka-2:9092,kafka-3:9092")
                .setTopics("ods_order_info")
                .setGroupId("flink-order-consumer")
                .setStartingOffsets(OffsetsInitializer.latest())
                .setValueOnlyDeserializer(new SimpleStringSchema())
                .build();
        
        DataStream<String> stream = env.fromSource(source, 
                WatermarkStrategy.noWatermarks(), "kafka-order-source");
        
        stream.process(new ProcessFunction<String, String>() {
            @Override
            public void processElement(String value, Context ctx, C