@Internal public class PythonPartitionCustomOperator<IN,OUT> extends OneInputPythonFunctionOperator<IN,OUT,IN,OUT>
PythonPartitionCustomOperator
enables us to set the number of partitions for current
operator dynamically when generating the StreamGraph
before executing the job. The number of partitions will be set in environment variables for
python Worker, so that we can obtain the number of partitions when executing user defined
partitioner function.bais, baisWrapper, baos, baosWrapper, collector, DATA_STREAM_STATELESS_FUNCTION_URN, jobOptions
elementCount, maxBundleSize, pythonFunctionRunner
chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService
Constructor and Description |
---|
PythonPartitionCustomOperator(Configuration config,
TypeInformation<IN> inputTypeInfo,
TypeInformation<OUT> outputTypeInfo,
DataStreamPythonFunctionInfo pythonFunctionInfo) |
Modifier and Type | Method and Description |
---|---|
void |
emitResult(Tuple2<byte[],Integer> resultTuple)
Sends the execution result to the downstream operator.
|
String |
getCoderUrn() |
Map<String,String> |
getInternalParameters() |
void |
setNumPartitions(int numPartitions) |
createPythonFunctionRunner, getFunctionUrn, getPythonEnv, open, processElement
endInput
checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, dispose, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey, setPythonConfig
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processWatermark
public PythonPartitionCustomOperator(Configuration config, TypeInformation<IN> inputTypeInfo, TypeInformation<OUT> outputTypeInfo, DataStreamPythonFunctionInfo pythonFunctionInfo)
public void emitResult(Tuple2<byte[],Integer> resultTuple) throws Exception
AbstractPythonFunctionOperator
emitResult
in class AbstractPythonFunctionOperator<OUT>
Exception
public Map<String,String> getInternalParameters()
getInternalParameters
in class OneInputPythonFunctionOperator<IN,OUT,IN,OUT>
public String getCoderUrn()
getCoderUrn
in class OneInputPythonFunctionOperator<IN,OUT,IN,OUT>
public void setNumPartitions(int numPartitions)
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.