org.apache.flink.table.catalog.hive.client

Interface HiveShim

All Superinterfaces:

Serializable

All Known Implementing Classes:

HiveShimV100, HiveShimV101, HiveShimV110, HiveShimV111, HiveShimV120, HiveShimV121, HiveShimV122, HiveShimV200, HiveShimV201, HiveShimV210, HiveShimV211, HiveShimV220, HiveShimV230, HiveShimV231, HiveShimV232, HiveShimV233, HiveShimV234, HiveShimV235, HiveShimV236, HiveShimV310, HiveShimV311, HiveShimV312

public interface HiveShim
extends Serializable

A shim layer to support different versions of Hive.

Method Summary

Modifier and Type	Method and Description
void	alterPartition(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String databaseName, String tableName, org.apache.hadoop.hive.metastore.api.Partition partition)
void	alterTable(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String databaseName, String tableName, org.apache.hadoop.hive.metastore.api.Table table) Alters a Hive table.
org.apache.hadoop.hive.ql.udf.generic.SimpleGenericUDAFParameterInfo	createUDAFParameterInfo(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector[] params, boolean isWindowing, boolean distinct, boolean allColumns) Creates SimpleGenericUDAFParameterInfo.
Optional<org.apache.hadoop.hive.ql.exec.FunctionInfo>	getBuiltInFunctionInfo(String name) Get a Hive built-in function by name.
Class<?>	getDateDataTypeClass() Hive Date data type class was changed in Hive 3.1.0.
List<org.apache.hadoop.hive.metastore.api.FieldSchema>	getFieldsFromDeserializer(Configuration conf, org.apache.hadoop.hive.metastore.api.Table table, boolean skipConfError) Get Hive table schema from deserializer.
org.apache.hadoop.fs.FileStatus[]	getFileStatusRecurse(org.apache.hadoop.fs.Path path, int level, org.apache.hadoop.fs.FileSystem fs) The return type of HiveStatsUtils.getFileStatusRecurse was changed from array to List in Hive 3.1.0.
org.apache.hadoop.hive.metastore.IMetaStoreClient	getHiveMetastoreClient(org.apache.hadoop.hive.conf.HiveConf hiveConf) Create a Hive Metastore client based on the given HiveConf object.
Class<?>	getHiveMetaStoreUtilsClass() Get the class of Hive's HiveMetaStoreUtils as it was split from MetaStoreUtils class in Hive 3.1.0.
org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter	getHiveRecordWriter(org.apache.hadoop.mapred.JobConf jobConf, String outputFormatClzName, Class<? extends org.apache.hadoop.io.Writable> outValClz, boolean isCompressed, Properties tableProps, org.apache.hadoop.fs.Path outPath) Get Hive's FileSinkOperator.RecordWriter.
Class<?>	getMetaStoreUtilsClass() Get the class of Hive's MetaStoreUtils because its package name was changed in Hive 3.1.0.
Set<String>	getNotNullColumns(org.apache.hadoop.hive.metastore.IMetaStoreClient client, Configuration conf, String dbName, String tableName) Get the set of columns that have NOT NULL constraints.
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector	getObjectInspectorForConstant(org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo primitiveTypeInfo, Object value) Get ObjectInspector for a constant value.
Optional<UniqueConstraint>	getPrimaryKey(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String dbName, String tableName, byte requiredTrait) Get the primary key of a Hive table and convert it to a UniqueConstraint.
Class<?>	getTimestampDataTypeClass() Hive Timestamp data type class was changed in Hive 3.1.0.
List<String>	getViews(org.apache.hadoop.hive.metastore.IMetaStoreClient client, String databaseName) Get a list of views in the given database from the given Hive Metastore client.
boolean	isDateStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData colStatsData) Whether a Hive ColumnStatisticsData is for DATE columns.
Set<String>	listBuiltInFunctions() List names of all built-in functions.
void	makeSpecFromName(Map<String,String> partSpec, org.apache.hadoop.fs.Path currPath) The signature of HiveStatsUtils.makeSpecFromName() was changed in Hive 3.1.0.
boolean	moveToTrash(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, Configuration conf, boolean purge) Moves a particular file or directory to trash.
java.time.LocalDate	toFlinkDate(Object hiveDate) Converts a hive date instance to LocalDate which is expected by DataFormatConverter.
CatalogColumnStatisticsDataDate	toFlinkDateColStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData hiveDateColStats) Generate Flink CatalogColumnStatisticsDataDate from Hive ColumnStatisticsData for DATE columns.
java.time.LocalDateTime	toFlinkTimestamp(Object hiveTimestamp) Converts a hive timestamp instance to LocalDateTime which is expected by DataFormatConverter.
Object	toHiveDate(Object flinkDate) Converts a Flink date instance to what's expected by Hive.
org.apache.hadoop.hive.metastore.api.ColumnStatisticsData	toHiveDateColStats(CatalogColumnStatisticsDataDate flinkDateColStats) Generate Hive ColumnStatisticsData from Flink CatalogColumnStatisticsDataDate for DATE columns.
Object	toHiveTimestamp(Object flinkTimestamp) Converts a Flink timestamp instance to what's expected by Hive.

Method Detail

getHiveMetastoreClient

org.apache.hadoop.hive.metastore.IMetaStoreClient getHiveMetastoreClient(org.apache.hadoop.hive.conf.HiveConf hiveConf)

Create a Hive Metastore client based on the given HiveConf object.

Parameters:

hiveConf - HiveConf instance

Returns:

an IMetaStoreClient instance

getViews

List<String> getViews(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                      String databaseName)
               throws org.apache.hadoop.hive.metastore.api.UnknownDBException,
                      org.apache.thrift.TException

Get a list of views in the given database from the given Hive Metastore client.

Parameters:

client - Hive Metastore client

databaseName - the name of the database

Returns:

A list of names of the views

Throws:

org.apache.hadoop.hive.metastore.api.UnknownDBException - if the database doesn't exist

org.apache.thrift.TException - for any other generic exceptions caused by Thrift

moveToTrash

boolean moveToTrash(org.apache.hadoop.fs.FileSystem fs,
                    org.apache.hadoop.fs.Path path,
                    Configuration conf,
                    boolean purge)
             throws IOException

Moves a particular file or directory to trash. The file/directory can potentially be deleted (w/o going to trash) if purge is set to true, or if it cannot be moved properly.

This interface is here because FileUtils.moveToTrash in different Hive versions have different signatures.

Parameters:

fs - the FileSystem to use

path - the path of the file or directory to be moved to trash.

conf - the Configuration to use

purge - whether try to skip trash and directly delete the file/directory. This flag may be ignored by old Hive versions prior to 2.3.0.

Returns:

true if the move is successful, and false otherwise

Throws:

IOException - if the file/directory cannot be properly moved or deleted

alterTable

void alterTable(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                String databaseName,
                String tableName,
                org.apache.hadoop.hive.metastore.api.Table table)
         throws org.apache.hadoop.hive.metastore.api.InvalidOperationException,
                org.apache.hadoop.hive.metastore.api.MetaException,
                org.apache.thrift.TException

Alters a Hive table.

Parameters:

client - the Hive metastore client

databaseName - the name of the database to which the table belongs

tableName - the name of the table to be altered

table - the new Hive table

Throws:

org.apache.hadoop.hive.metastore.api.InvalidOperationException

org.apache.hadoop.hive.metastore.api.MetaException

org.apache.thrift.TException

alterPartition

void alterPartition(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                    String databaseName,
                    String tableName,
                    org.apache.hadoop.hive.metastore.api.Partition partition)
             throws org.apache.hadoop.hive.metastore.api.InvalidOperationException,
                    org.apache.hadoop.hive.metastore.api.MetaException,
                    org.apache.thrift.TException

Throws:

org.apache.hadoop.hive.metastore.api.InvalidOperationException

org.apache.hadoop.hive.metastore.api.MetaException

org.apache.thrift.TException

createUDAFParameterInfo

org.apache.hadoop.hive.ql.udf.generic.SimpleGenericUDAFParameterInfo createUDAFParameterInfo(org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector[] params,
                                                                                             boolean isWindowing,
                                                                                             boolean distinct,
                                                                                             boolean allColumns)

Creates SimpleGenericUDAFParameterInfo.

getMetaStoreUtilsClass

Class<?> getMetaStoreUtilsClass()

Get the class of Hive's MetaStoreUtils because its package name was changed in Hive 3.1.0.

Returns:

MetaStoreUtils class

getHiveMetaStoreUtilsClass

Class<?> getHiveMetaStoreUtilsClass()

Get the class of Hive's HiveMetaStoreUtils as it was split from MetaStoreUtils class in Hive 3.1.0.

Returns:

HiveMetaStoreUtils class

getDateDataTypeClass

Class<?> getDateDataTypeClass()

Hive Date data type class was changed in Hive 3.1.0.

Returns:

Hive's Date class

getTimestampDataTypeClass

Class<?> getTimestampDataTypeClass()

Hive Timestamp data type class was changed in Hive 3.1.0.

Returns:

Hive's Timestamp class

getFileStatusRecurse

org.apache.hadoop.fs.FileStatus[] getFileStatusRecurse(org.apache.hadoop.fs.Path path,
                                                       int level,
                                                       org.apache.hadoop.fs.FileSystem fs)
                                                throws IOException

The return type of HiveStatsUtils.getFileStatusRecurse was changed from array to List in Hive 3.1.0.

Parameters:

path - the path of the directory

level - the level of recursion

fs - the file system of the directory

Returns:

an array of the entries

Throws:

IOException - in case of any io error

makeSpecFromName

void makeSpecFromName(Map<String,String> partSpec,
                      org.apache.hadoop.fs.Path currPath)

The signature of HiveStatsUtils.makeSpecFromName() was changed in Hive 3.1.0.

Parameters:

partSpec - partition specs

currPath - the current path

getObjectInspectorForConstant

org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector getObjectInspectorForConstant(org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo primitiveTypeInfo,
                                                                                            Object value)

Get ObjectInspector for a constant value.

toHiveDateColStats

org.apache.hadoop.hive.metastore.api.ColumnStatisticsData toHiveDateColStats(CatalogColumnStatisticsDataDate flinkDateColStats)

Generate Hive ColumnStatisticsData from Flink CatalogColumnStatisticsDataDate for DATE columns.

isDateStats

boolean isDateStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData colStatsData)

Whether a Hive ColumnStatisticsData is for DATE columns.

toFlinkDateColStats

CatalogColumnStatisticsDataDate toFlinkDateColStats(org.apache.hadoop.hive.metastore.api.ColumnStatisticsData hiveDateColStats)

Generate Flink CatalogColumnStatisticsDataDate from Hive ColumnStatisticsData for DATE columns.

getHiveRecordWriter

org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter getHiveRecordWriter(org.apache.hadoop.mapred.JobConf jobConf,
                                                                                 String outputFormatClzName,
                                                                                 Class<? extends org.apache.hadoop.io.Writable> outValClz,
                                                                                 boolean isCompressed,
                                                                                 Properties tableProps,
                                                                                 org.apache.hadoop.fs.Path outPath)

Get Hive's FileSinkOperator.RecordWriter.

getFieldsFromDeserializer

List<org.apache.hadoop.hive.metastore.api.FieldSchema> getFieldsFromDeserializer(Configuration conf,
                                                                                 org.apache.hadoop.hive.metastore.api.Table table,
                                                                                 boolean skipConfError)

Get Hive table schema from deserializer.

listBuiltInFunctions

Set<String> listBuiltInFunctions()

List names of all built-in functions.

getBuiltInFunctionInfo

Optional<org.apache.hadoop.hive.ql.exec.FunctionInfo> getBuiltInFunctionInfo(String name)

Get a Hive built-in function by name.

getNotNullColumns

Set<String> getNotNullColumns(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                              Configuration conf,
                              String dbName,
                              String tableName)

Get the set of columns that have NOT NULL constraints.

getPrimaryKey

Optional<UniqueConstraint> getPrimaryKey(org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                                         String dbName,
                                         String tableName,
                                         byte requiredTrait)

Get the primary key of a Hive table and convert it to a UniqueConstraint. Return empty if the table doesn't have a primary key, or the constraint doesn't satisfy the desired trait, e.g. RELY.

toHiveTimestamp

Object toHiveTimestamp(Object flinkTimestamp)

Converts a Flink timestamp instance to what's expected by Hive.

toFlinkTimestamp

java.time.LocalDateTime toFlinkTimestamp(Object hiveTimestamp)

Converts a hive timestamp instance to LocalDateTime which is expected by DataFormatConverter.

toHiveDate

Object toHiveDate(Object flinkDate)

Converts a Flink date instance to what's expected by Hive.

toFlinkDate

java.time.LocalDate toFlinkDate(Object hiveDate)

Converts a hive date instance to LocalDate which is expected by DataFormatConverter.