org.apache.flink.table.catalog.hive.client

Interface HiveShim

All Superinterfaces:

Serializable

All Known Implementing Classes:

HiveShimV100, HiveShimV101, HiveShimV110, HiveShimV111, HiveShimV120, HiveShimV121, HiveShimV122, HiveShimV200, HiveShimV201, HiveShimV210, HiveShimV211, HiveShimV220, HiveShimV230, HiveShimV231, HiveShimV232, HiveShimV233, HiveShimV234, HiveShimV235, HiveShimV236, HiveShimV310, HiveShimV311, HiveShimV312

public interface HiveShim
extends Serializable

A shim layer to support different versions of Hive.

Method Summary

Modifier and Type	Method and Description
void	alterPartition(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String databaseName, String tableName, org.apache.hadoop.hive.metastore.api.Partition partition)
void	alterTable(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String databaseName, String tableName, org.apache.hadoop.hive.metastore.api.Table table) Alters a Hive table.
BulkWriter.Factory<RowData>	createOrcBulkWriterFactory(Configuration conf, String schema, LogicalType[] fieldTypes) Create orc BulkWriter.Factory for different hive versions.
void	createTableWithConstraints(org.apache.hadoop.hive.metastore.IMetaStoreClient client, org.apache.hadoop.hive.metastore.api.Table table, Configuration conf, UniqueConstraint pk, List<Byte> pkTraits, List<String> notNullCols, List<Byte> nnTraits) Creates a table with PK and NOT NULL constraints.
org.apache.hadoop.hive.ql.udf.generic.SimpleGenericUDAFParameterInfo	createUDAFParameterInfo(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector[] params, boolean isWindowing, boolean distinct, boolean allColumns) Creates SimpleGenericUDAFParameterInfo.
Optional<org.apache.hadoop.hive.ql.exec.FunctionInfo>	getBuiltInFunctionInfo(String name) Get a Hive built-in function by name.
Class<?>	getDateDataTypeClass() Hive Date data type class was changed in Hive 3.1.0.
List<org.apache.hadoop.hive.metastore.api.FieldSchema>	getFieldsFromDeserializer(Configuration conf, org.apache.hadoop.hive.metastore.api.Table table, boolean skipConfError) Get Hive table schema from deserializer.
org.apache.hadoop.hive.metastore.IMetaStoreClient	getHiveMetastoreClient(org.apache.hadoop.hive.conf.HiveConf hiveConf) Create a Hive Metastore client based on the given HiveConf object.
Class<?>	getHiveMetaStoreUtilsClass() Get the class of Hive's HiveMetaStoreUtils as it was split from MetaStoreUtils class in Hive 3.1.0.
Class	getHiveOutputFormatClass(Class outputFormatClz) For a given OutputFormat class, get the corresponding HiveOutputFormat class.
org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter	getHiveRecordWriter(org.apache.hadoop.mapred.JobConf jobConf, Class outputFormatClz, Class<? extends org.apache.hadoop.io.Writable> outValClz, boolean isCompressed, Properties tableProps, org.apache.hadoop.fs.Path outPath) Get Hive's FileSinkOperator.RecordWriter.
Class<?>	getMetaStoreUtilsClass() Get the class of Hive's MetaStoreUtils because its package name was changed in Hive 3.1.0.
Set<String>	getNotNullColumns(org.apache.hadoop.hive.metastore.IMetaStoreClient client, Configuration conf, String dbName, String tableName) Get the set of columns that have NOT NULL constraints.
Optional<UniqueConstraint>	getPrimaryKey(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String dbName, String tableName, byte requiredTrait) Get the primary key of a Hive table and convert it to a UniqueConstraint.
Class<?>	getTimestampDataTypeClass() Hive Timestamp data type class was changed in Hive 3.1.0.
List<String>	getViews(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String databaseName) Get a list of views in the given database from the given Hive Metastore client.
org.apache.hadoop.io.Writable	hivePrimitiveToWritable(Object value) Converts a Hive primitive java object to corresponding Writable object.
boolean	isDateStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData colStatsData) Whether a Hive ColumnStatisticsData is for DATE columns.
Set<String>	listBuiltInFunctions() List names of all built-in functions.
java.time.LocalDate	toFlinkDate(Object hiveDate) Converts a hive date instance to LocalDate which is expected by DataFormatConverter.
CatalogColumnStatisticsDataDate	toFlinkDateColStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData hiveDateColStats) Generate Flink CatalogColumnStatisticsDataDate from Hive ColumnStatisticsData for DATE columns.
java.time.LocalDateTime	toFlinkTimestamp(Object hiveTimestamp) Converts a hive timestamp instance to LocalDateTime which is expected by DataFormatConverter.
Object	toHiveDate(Object flinkDate) Converts a Flink date instance to what's expected by Hive.
org.apache.hadoop.hive.metastore.api.ColumnStatisticsData	toHiveDateColStats(CatalogColumnStatisticsDataDate flinkDateColStats) Generate Hive ColumnStatisticsData from Flink CatalogColumnStatisticsDataDate for DATE columns.
Object	toHiveTimestamp(Object flinkTimestamp) Converts a Flink timestamp instance to what's expected by Hive.

Method Detail

getHiveMetastoreClient

org.apache.hadoop.hive.metastore.IMetaStoreClient getHiveMetastoreClient(org.apache.hadoop.hive.conf.HiveConf hiveConf)

Create a Hive Metastore client based on the given HiveConf object.

Parameters:

hiveConf - HiveConf instance

Returns:

an IMetaStoreClient instance

getViews

List<String> getViews(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                      String databaseName)
               throws org.apache.hadoop.hive.metastore.api.UnknownDBException,
                      org.apache.thrift.TException

Get a list of views in the given database from the given Hive Metastore client.

Parameters:

client - Hive Metastore client

databaseName - the name of the database

Returns:

A list of names of the views

Throws:

org.apache.hadoop.hive.metastore.api.UnknownDBException - if the database doesn't exist

org.apache.thrift.TException - for any other generic exceptions caused by Thrift

alterTable

void alterTable(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                String databaseName,
                String tableName,
                org.apache.hadoop.hive.metastore.api.Table table)
         throws org.apache.hadoop.hive.metastore.api.InvalidOperationException,
                org.apache.hadoop.hive.metastore.api.MetaException,
                org.apache.thrift.TException

Alters a Hive table.

Parameters:

client - the Hive metastore client

databaseName - the name of the database to which the table belongs

tableName - the name of the table to be altered

table - the new Hive table

Throws:

org.apache.hadoop.hive.metastore.api.InvalidOperationException

org.apache.hadoop.hive.metastore.api.MetaException

org.apache.thrift.TException

alterPartition

void alterPartition(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                    String databaseName,
                    String tableName,
                    org.apache.hadoop.hive.metastore.api.Partition partition)
             throws org.apache.hadoop.hive.metastore.api.InvalidOperationException,
                    org.apache.hadoop.hive.metastore.api.MetaException,
                    org.apache.thrift.TException

Throws:

org.apache.hadoop.hive.metastore.api.InvalidOperationException

org.apache.hadoop.hive.metastore.api.MetaException

org.apache.thrift.TException

createUDAFParameterInfo

org.apache.hadoop.hive.ql.udf.generic.SimpleGenericUDAFParameterInfo createUDAFParameterInfo(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector[] params,
                                                                                             boolean isWindowing,
                                                                                             boolean distinct,
                                                                                             boolean allColumns)

Creates SimpleGenericUDAFParameterInfo.

getMetaStoreUtilsClass

Class<?> getMetaStoreUtilsClass()

Get the class of Hive's MetaStoreUtils because its package name was changed in Hive 3.1.0.

Returns:

MetaStoreUtils class

getHiveMetaStoreUtilsClass

Class<?> getHiveMetaStoreUtilsClass()

Get the class of Hive's HiveMetaStoreUtils as it was split from MetaStoreUtils class in Hive 3.1.0.

Returns:

HiveMetaStoreUtils class

getDateDataTypeClass

Class<?> getDateDataTypeClass()

Hive Date data type class was changed in Hive 3.1.0.

Returns:

Hive's Date class

getTimestampDataTypeClass

Class<?> getTimestampDataTypeClass()

Hive Timestamp data type class was changed in Hive 3.1.0.

Returns:

Hive's Timestamp class

toHiveDateColStats

org.apache.hadoop.hive.metastore.api.ColumnStatisticsData toHiveDateColStats(CatalogColumnStatisticsDataDate flinkDateColStats)

Generate Hive ColumnStatisticsData from Flink CatalogColumnStatisticsDataDate for DATE columns.

isDateStats

boolean isDateStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData colStatsData)

Whether a Hive ColumnStatisticsData is for DATE columns.

toFlinkDateColStats

CatalogColumnStatisticsDataDate toFlinkDateColStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData hiveDateColStats)

Generate Flink CatalogColumnStatisticsDataDate from Hive ColumnStatisticsData for DATE columns.

getHiveRecordWriter

org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter getHiveRecordWriter(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                 Class outputFormatClz,
                                                                                 Class<? extends org.apache.hadoop.io.Writable> outValClz,
                                                                                 boolean isCompressed,
                                                                                 Properties tableProps,
                                                                                 org.apache.hadoop.fs.Path outPath)

Get Hive's FileSinkOperator.RecordWriter.

getHiveOutputFormatClass

Class getHiveOutputFormatClass(Class outputFormatClz)

For a given OutputFormat class, get the corresponding HiveOutputFormat class.

getFieldsFromDeserializer

List<org.apache.hadoop.hive.metastore.api.FieldSchema> getFieldsFromDeserializer(Configuration conf,
                                                                                 org.apache.hadoop.hive.metastore.api.Table table,
                                                                                 boolean skipConfError)

Get Hive table schema from deserializer.

listBuiltInFunctions

Set<String> listBuiltInFunctions()

List names of all built-in functions.

getBuiltInFunctionInfo

Optional<org.apache.hadoop.hive.ql.exec.FunctionInfo> getBuiltInFunctionInfo(String name)

Get a Hive built-in function by name.

getNotNullColumns

Set<String> getNotNullColumns(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                              Configuration conf,
                              String dbName,
                              String tableName)

Get the set of columns that have NOT NULL constraints.

getPrimaryKey

Optional<UniqueConstraint> getPrimaryKey(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                                         String dbName,
                                         String tableName,
                                         byte requiredTrait)

Get the primary key of a Hive table and convert it to a UniqueConstraint. Return empty if the table doesn't have a primary key, or the constraint doesn't satisfy the desired trait, e.g. RELY.

toHiveTimestamp

@Nullable
Object toHiveTimestamp(@Nullable
                                 Object flinkTimestamp)

Converts a Flink timestamp instance to what's expected by Hive.

toFlinkTimestamp

java.time.LocalDateTime toFlinkTimestamp(Object hiveTimestamp)

Converts a hive timestamp instance to LocalDateTime which is expected by DataFormatConverter.

toHiveDate

@Nullable
Object toHiveDate(@Nullable
                            Object flinkDate)

Converts a Flink date instance to what's expected by Hive.

toFlinkDate

java.time.LocalDate toFlinkDate(Object hiveDate)

Converts a hive date instance to LocalDate which is expected by DataFormatConverter.

hivePrimitiveToWritable

@Nullable
org.apache.hadoop.io.Writable hivePrimitiveToWritable(@Nullable
                                                                Object value)

Converts a Hive primitive java object to corresponding Writable object.

createTableWithConstraints

void createTableWithConstraints(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                                org.apache.hadoop.hive.metastore.api.Table table,
                                Configuration conf,
                                UniqueConstraint pk,
                                List<Byte> pkTraits,
                                List<String> notNullCols,
                                List<Byte> nnTraits)

Creates a table with PK and NOT NULL constraints.

createOrcBulkWriterFactory

BulkWriter.Factory<RowData> createOrcBulkWriterFactory(Configuration conf,
                                                       String schema,
                                                       LogicalType[] fieldTypes)

Create orc BulkWriter.Factory for different hive versions.