IN
- Type of the input elements.OUT
- Type of the output elements.UDFIN
- Type of the UDF input type.@Internal public abstract class AbstractPythonScalarFunctionOperator<IN,OUT,UDFIN> extends AbstractStatelessFunctionOperator<IN,OUT,UDFIN>
ScalarFunction
s. It executes the
Python ScalarFunction
s in separate Python execution environment.
The inputs are assumed as the following format: {{{ +------------------+--------------+ | forwarded fields | extra fields | +------------------+--------------+ }}}.
The Python UDFs may take input columns directly from the input row or the execution result of Java UDFs: 1) The input columns from the input row can be referred from the 'forwarded fields'; 2) The Java UDFs will be computed and the execution results can be referred from the 'extra fields'.
The outputs will be as the following format: {{{ +------------------+-------------------------+ | forwarded fields | scalar function results | +------------------+-------------------------+ }}}.
Modifier and Type | Field and Description |
---|---|
protected int[] |
forwardedFields
The offset of the fields which should be forwarded.
|
protected PythonFunctionInfo[] |
scalarFunctions
The Python
ScalarFunction s to be executed. |
bais, baisWrapper, baos, baosWrapper, forwardedInputQueue, inputType, outputType, userDefinedFunctionInputOffsets, userDefinedFunctionInputType, userDefinedFunctionOutputType
elementCount, maxBundleSize, pythonFunctionRunner
chainingStrategy, latencyStats, LOG, metrics, output, processingTimeService
Modifier and Type | Method and Description |
---|---|
String |
getFunctionUrn() |
String |
getInputOutputCoderUrn() |
PythonEnv |
getPythonEnv()
Returns the
PythonEnv used to create PythonEnvironmentManager.. |
FlinkFnApi.UserDefinedFunctions |
getUserDefinedFunctionsProto()
Gets the proto representation of the Python user-defined functions to be executed.
|
void |
open()
This method is called immediately before any elements are processed, it should contain the
operator's initialization logic, e.g.
|
bufferInput, createPythonFunctionRunner, getFunctionInput, processElement, processElementInternal
endInput
checkInvokeFinishBundleByCount, close, createPythonEnvironmentManager, dispose, emitResult, emitResults, getConfig, getFlinkMetricContainer, getPythonConfig, invokeFinishBundle, isBundleFinished, prepareSnapshotPreBarrier, processWatermark, setCurrentKey, setPythonConfig
getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, notifyCheckpointComplete, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, processWatermark1, processWatermark2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, setup, snapshotState, snapshotState
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
setKeyContextElement
close, dispose, getMetricGroup, getOperatorID, initializeState, prepareSnapshotPreBarrier, setKeyContextElement1, setKeyContextElement2, snapshotState
notifyCheckpointAborted, notifyCheckpointComplete
getCurrentKey, setCurrentKey
processLatencyMarker, processWatermark
protected final PythonFunctionInfo[] scalarFunctions
ScalarFunction
s to be executed.protected final int[] forwardedFields
public void open() throws Exception
AbstractStreamOperator
The default implementation does nothing.
open
in interface StreamOperator<OUT>
open
in class AbstractStatelessFunctionOperator<IN,OUT,UDFIN>
Exception
- An exception in this method causes the operator to fail.public PythonEnv getPythonEnv()
AbstractPythonFunctionOperator
PythonEnv
used to create PythonEnvironmentManager..getPythonEnv
in class AbstractPythonFunctionOperator<OUT>
public FlinkFnApi.UserDefinedFunctions getUserDefinedFunctionsProto()
getUserDefinedFunctionsProto
in class AbstractStatelessFunctionOperator<IN,OUT,UDFIN>
public String getFunctionUrn()
getFunctionUrn
in class AbstractStatelessFunctionOperator<IN,OUT,UDFIN>
public String getInputOutputCoderUrn()
getInputOutputCoderUrn
in class AbstractStatelessFunctionOperator<IN,OUT,UDFIN>
Copyright © 2014–2022 The Apache Software Foundation. All rights reserved.