public class HiveTableSource extends Object implements StreamTableSource<RowData>, PartitionableTableSource, ProjectableTableSource<RowData>, LimitableTableSource<RowData>, LookupableTableSource<RowData>
Constructor and Description |
---|
HiveTableSource(org.apache.hadoop.mapred.JobConf jobConf,
ReadableConfig flinkConf,
ObjectPath tablePath,
CatalogTable catalogTable) |
Modifier and Type | Method and Description |
---|---|
TableSource<RowData> |
applyLimit(long limit)
Check and push down the limit to the table source.
|
TableSource<RowData> |
applyPartitionPruning(List<Map<String,String>> remainingPartitions)
Applies the remaining partitions to the table source.
|
String |
explainSource()
Describes the table source.
|
AsyncTableFunction<RowData> |
getAsyncLookupFunction(String[] lookupKeys)
Gets the
AsyncTableFunction which supports async lookup one key at a time. |
DataStream<RowData> |
getDataStream(StreamExecutionEnvironment execEnv)
Returns the data of the table as a
DataStream . |
TableFunction<RowData> |
getLookupFunction(String[] lookupKeys)
Gets the
TableFunction which supports lookup one key at a time. |
List<Map<String,String>> |
getPartitions()
Returns all the partitions of this
PartitionableTableSource . |
DataType |
getProducedDataType()
Returns the
DataType for the produced data of the TableSource . |
TableSchema |
getTableSchema()
Returns the schema of the produced table.
|
boolean |
isAsyncEnabled()
Returns true if async lookup is enabled.
|
boolean |
isBounded()
Returns true if this is a bounded source, false if this is an unbounded source.
|
boolean |
isLimitPushedDown()
Return the flag to indicate whether limit push down has been tried.
|
TableSource<RowData> |
projectFields(int[] fields)
Creates a copy of the
TableSource that projects its output to the given field
indexes. |
static HiveTablePartition |
toHiveTablePartition(List<String> partitionKeys,
String[] fieldNames,
DataType[] fieldTypes,
HiveShim shim,
Properties tableProps,
String defaultPartitionName,
org.apache.hadoop.hive.metastore.api.Partition partition) |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getReturnType
public HiveTableSource(org.apache.hadoop.mapred.JobConf jobConf, ReadableConfig flinkConf, ObjectPath tablePath, CatalogTable catalogTable)
public boolean isBounded()
StreamTableSource
isBounded
in interface StreamTableSource<RowData>
public DataStream<RowData> getDataStream(StreamExecutionEnvironment execEnv)
StreamTableSource
DataStream
.
NOTE: This method is for internal use only for defining a TableSource
. Do not use
it in Table API programs.
getDataStream
in interface StreamTableSource<RowData>
public TableSchema getTableSchema()
TableSource
getTableSchema
in interface TableSource<RowData>
TableSchema
of the produced table.public DataType getProducedDataType()
TableSource
DataType
for the produced data of the TableSource
.getProducedDataType
in interface TableSource<RowData>
DataSet
or DataStream
.public boolean isLimitPushedDown()
LimitableTableSource
LimitableTableSource.applyLimit(long)
.isLimitPushedDown
in interface LimitableTableSource<RowData>
public TableSource<RowData> applyLimit(long limit)
LimitableTableSource
applyLimit
in interface LimitableTableSource<RowData>
limit
- the value which limit the number of records.TableSource
.public List<Map<String,String>> getPartitions()
PartitionableTableSource
PartitionableTableSource
.getPartitions
in interface PartitionableTableSource
public TableSource<RowData> applyPartitionPruning(List<Map<String,String>> remainingPartitions)
PartitionableTableSource
remainingPartitions
is the
remaining partitions of PartitionableTableSource.getPartitions()
after partition pruning applied.
After trying to apply partition pruning, we should return a new TableSource
instance which holds all pruned-partitions.
applyPartitionPruning
in interface PartitionableTableSource
remainingPartitions
- Remaining partitions after partition pruning applied.TableSource
holds all pruned-partitions.public TableSource<RowData> projectFields(int[] fields)
ProjectableTableSource
TableSource
that projects its output to the given field
indexes. The field indexes relate to the physical poduced data type (TableSource.getProducedDataType()
) and not to the table schema (TableSource.getTableSchema()
of the TableSource
.
The table schema (TableSource.getTableSchema()
of the TableSource
copy must
not be modified by this method, but only the produced data type (TableSource.getProducedDataType()
) and the produced DataSet
(BatchTableSource#getDataSet(
) or DataStream
(StreamTableSource#getDataStream
).
If the TableSource
implements the DefinedFieldMapping
interface, it might
be necessary to adjust the mapping as well.
IMPORTANT: This method must return a true copy and must not modify the original table source object.
projectFields
in interface ProjectableTableSource<RowData>
fields
- The indexes of the fields to return.TableSource
that projects its output.public static HiveTablePartition toHiveTablePartition(List<String> partitionKeys, String[] fieldNames, DataType[] fieldTypes, HiveShim shim, Properties tableProps, String defaultPartitionName, org.apache.hadoop.hive.metastore.api.Partition partition)
public String explainSource()
TableSource
explainSource
in interface TableSource<RowData>
TableSource
.public TableFunction<RowData> getLookupFunction(String[] lookupKeys)
LookupableTableSource
TableFunction
which supports lookup one key at a time.getLookupFunction
in interface LookupableTableSource<RowData>
lookupKeys
- the chosen field names as lookup keys, it is in the defined orderpublic AsyncTableFunction<RowData> getAsyncLookupFunction(String[] lookupKeys)
LookupableTableSource
AsyncTableFunction
which supports async lookup one key at a time.getAsyncLookupFunction
in interface LookupableTableSource<RowData>
lookupKeys
- the chosen field names as lookup keys, it is in the defined orderpublic boolean isAsyncEnabled()
LookupableTableSource
The lookup function returned by LookupableTableSource.getAsyncLookupFunction(String[])
will be used if
returns true. Otherwise, the lookup function returned by LookupableTableSource.getLookupFunction(String[])
will be used.
isAsyncEnabled
in interface LookupableTableSource<RowData>
Copyright © 2014–2021 The Apache Software Foundation. All rights reserved.