Operators transform one or more DataStreams into a new DataStream. Programs can combine multiple transformations into sophisticated dataflow topologies.
DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., mapping, filtering, reducing). Please see operators for an overview of the available stream transformations in Python DataStream API.
Most transformations require a user-defined function as input to define the functionality of the transformation. The following describes different ways of defining user-defined functions.
Different Function interfaces are provided for different transformations in the Python DataStream API. For example,
MapFunction is provided for the
FilterFunction is provided for the
filter transformation, etc.
Users can implement the corresponding Function interface according to the type of the transformation. Take MapFunction for
Note In Python DataStream API, users can specify the output type information of the transformation explicityly. If not
specified, the output type will be
Types.PICKLED_BYTE_ARRAY so that data will be in a form of byte array generated by
the pickle seriallizer. For more details about the
Pickle Serialization, please refer to DataTypes.
As shown in the following example, all the transformations can also accept a lambda function to define the functionality of the transformation:
Note Operations ConnectedStream.map() and ConnectedStream.flat_map() do not support
lambda function and must accept
Users can also use Python function: