Skip to main content

​
Introduction

Each streaming pipeline has three components:
  • Source: The data stream source to consume message from.
  • Transformer: Write python code to transform the message before writing to destination.
  • Sink (destination): The data stream destination to write message to.
Check out this tutorial to set up an example streaming pipeline.
In version 0.8.80 or greater.
You can build a streaming pipeline with one source, multiple transformers, and multiple sinks. Here is an example streaming pipeline:
Complex streaming pipeline

​
Supported sources

​
Supported sinks (destinations)

​
Test pipeline execution

After finishing configuring the streaming pipeline, you can click the button Execution pipeline to test streaming pipeline execution.

​
Run pipeline in production

Create the trigger in triggers page to run streaming pipelines in production.

​
Executor count

If you want to run multiple executors at the same time to scale the streaming pipeline execution, you can set the executor_count variable in the pipeline’s metadata.yaml file. Here is an example: 
blocks:
- ...
- ...
executor_count: 10
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline

​
Executor type

When running Mage on Kubernetes cluster, you can also configure streaming pipeline to be run on separate k8s pods by setting executor_type field in the pipeline’s metadata.yaml to k8s. Example config: 
blocks:
- ...
- ...
executor_type: k8s
name: test_streaming_pipeline
type: streaming
uuid: test_streaming_pipeline

​
Contributing guide

Follow this doc to add a new source or destination (sink) to Mage streaming pipeline.
⌘I