T
- The type of the elements that result from this StreamTransformation
@Internal public abstract class StreamTransformation<T> extends Object
StreamTransformation
represents the operation that creates a
DataStream
. Every
DataStream
has an underlying
StreamTransformation
that is the origin of said DataStream.
API operations such as DataStream.map(org.apache.flink.api.common.functions.MapFunction<T, R>)
create
a tree of StreamTransformation
s underneath. When the stream program is to be executed
this graph is translated to a StreamGraph
using
StreamGraphGenerator
.
A StreamTransformation
does not necessarily correspond to a physical operation
at runtime. Some operations are only logical concepts. Examples of this are union,
split/select data stream, partitioning.
The following graph of StreamTransformations
:
Source Source
+ +
| |
v v
Rebalance HashPartition
+ +
| |
| |
+------>Union<------+
+
|
v
Split
+
|
v
Select
+
v
Map
+
|
v
Sink
Would result in this graph of operations at runtime:
Source Source
+ +
| |
| |
+------->Map<-------+
+
|
v
Sink
The information about partitioning, union, split/select end up being encoded in the edges that connect the sources to the map operation.
Modifier and Type | Field and Description |
---|---|
protected long |
bufferTimeout |
protected int |
id |
protected static Integer |
idCounter |
protected String |
name |
protected TypeInformation<T> |
outputType |
protected boolean |
typeUsed |
Constructor and Description |
---|
StreamTransformation(String name,
TypeInformation<T> outputType,
int parallelism)
Creates a new
StreamTransformation with the given name, output type and parallelism. |
Modifier and Type | Method and Description |
---|---|
boolean |
equals(Object o) |
long |
getBufferTimeout()
Returns the buffer timeout of this
StreamTransformation . |
int |
getId()
Returns the unique ID of this
StreamTransformation . |
int |
getMaxParallelism()
Gets the maximum parallelism for this stream transformation.
|
ResourceSpec |
getMinResources()
Gets the minimum resource of this stream transformation.
|
String |
getName()
Returns the name of this
StreamTransformation . |
static int |
getNewNodeId() |
TypeInformation<T> |
getOutputType()
Returns the output type of this
StreamTransformation as a TypeInformation . |
int |
getParallelism()
Returns the parallelism of this
StreamTransformation . |
ResourceSpec |
getPreferredResources()
Gets the preferred resource of this stream transformation.
|
String |
getSlotSharingGroup()
Returns the slot sharing group of this transformation.
|
abstract Collection<StreamTransformation<?>> |
getTransitivePredecessors()
Returns all transitive predecessor
StreamTransformation s of this StreamTransformation . |
String |
getUid()
Returns the user-specified ID of this transformation.
|
String |
getUserProvidedNodeHash()
Gets the user provided hash.
|
int |
hashCode() |
void |
setBufferTimeout(long bufferTimeout)
Set the buffer timeout of this
StreamTransformation . |
abstract void |
setChainingStrategy(ChainingStrategy strategy)
Sets the chaining strategy of this
StreamTransformation . |
void |
setMaxParallelism(int maxParallelism)
Sets the maximum parallelism for this stream transformation.
|
void |
setName(String name)
Changes the name of this
StreamTransformation . |
void |
setOutputType(TypeInformation<T> outputType)
Tries to fill in the type information.
|
void |
setParallelism(int parallelism)
Sets the parallelism of this
StreamTransformation . |
void |
setResources(ResourceSpec minResources,
ResourceSpec preferredResources)
Sets the minimum and preferred resources for this stream transformation.
|
void |
setSlotSharingGroup(String slotSharingGroup)
Sets the slot sharing group of this transformation.
|
void |
setUid(String uid)
Sets an ID for this
StreamTransformation . |
void |
setUidHash(String uidHash)
Sets an user provided hash for this operator.
|
String |
toString() |
protected static Integer idCounter
protected final int id
protected String name
protected TypeInformation<T> outputType
protected boolean typeUsed
protected long bufferTimeout
public StreamTransformation(String name, TypeInformation<T> outputType, int parallelism)
StreamTransformation
with the given name, output type and parallelism.name
- The name of the StreamTransformation
, this will be shown in Visualizations and the LogoutputType
- The output type of this StreamTransformation
parallelism
- The parallelism of this StreamTransformation
public static int getNewNodeId()
public int getId()
StreamTransformation
.public void setName(String name)
StreamTransformation
.public String getName()
StreamTransformation
.public int getParallelism()
StreamTransformation
.public void setParallelism(int parallelism)
StreamTransformation
.parallelism
- The new parallelism to set on this StreamTransformation
.public int getMaxParallelism()
public void setMaxParallelism(int maxParallelism)
maxParallelism
- Maximum parallelism for this stream transformation.public void setResources(ResourceSpec minResources, ResourceSpec preferredResources)
minResources
- The minimum resource of this transformation.preferredResources
- The preferred resource of this transformation.public ResourceSpec getMinResources()
public ResourceSpec getPreferredResources()
public void setUidHash(String uidHash)
The user provided hash is an alternative to the generated hashes, that is considered when identifying an operator through the default hash mechanics fails (e.g. because of changes between Flink versions).
Important: this should be used as a workaround or for trouble shooting. The provided hash needs to be unique per transformation and job. Otherwise, job submission will fail. Furthermore, you cannot assign user-specified hash to intermediate nodes in an operator chain and trying so will let your job fail.
A use case for this is in migration between Flink versions or changing the jobs in a way that changes the automatically generated hashes. In this case, providing the previous hashes directly through this method (e.g. obtained from old logs) can help to reestablish a lost mapping from states to their target operator.
uidHash
- The user provided hash for this operator. This will become the JobVertexID, which is shown in the
logs and web ui.public String getUserProvidedNodeHash()
public void setUid(String uid)
StreamTransformation
. This is will later be hashed to a uidHash which is then used to
create the JobVertexID (that is shown in logs and the web ui).
The specified ID is used to assign the same operator ID across job submissions (for example when starting a job from a savepoint).
Important: this ID needs to be unique per transformation and job. Otherwise, job submission will fail.
uid
- The unique user-specified ID of this transformation.public String getUid()
public String getSlotSharingGroup()
setSlotSharingGroup(String)
public void setSlotSharingGroup(String slotSharingGroup)
Initially, an operation is in the default slot sharing group. This can be explicitly
set using setSlotSharingGroup("default")
.
slotSharingGroup
- The slot sharing group name.public void setOutputType(TypeInformation<T> outputType)
outputType
- The type information to fill in.IllegalStateException
- Thrown, if the type information has been accessed before.public TypeInformation<T> getOutputType()
StreamTransformation
as a TypeInformation
. Once
this is used once the output type cannot be changed anymore using setOutputType(org.apache.flink.api.common.typeinfo.TypeInformation<T>)
.StreamTransformation
public abstract void setChainingStrategy(ChainingStrategy strategy)
StreamTransformation
.public void setBufferTimeout(long bufferTimeout)
StreamTransformation
. The timeout defines how long data
may linger in a partially full buffer before being sent over the network.
Lower timeouts lead to lower tail latencies, but may affect throughput. For Flink 1.5+, timeouts of 1ms are feasible for jobs with high parallelism.
A value of -1 means that the default buffer timeout should be used. A value of zero indicates that no buffering should happen, and all records/events should be immediately sent through the network, without additional buffering.
public long getBufferTimeout()
StreamTransformation
.setBufferTimeout(long)
public abstract Collection<StreamTransformation<?>> getTransitivePredecessors()
StreamTransformation
s of this StreamTransformation
. This
is, for example, used when determining whether a feedback edge of an iteration
actually has the iteration head as a predecessor.Copyright © 2014–2018 The Apache Software Foundation. All rights reserved.