Apache Hive has established itself as a focal point of the data warehousing ecosystem.
It serves as not only a SQL engine for big data analytics and ETL, but also a data management platform, where data is discovered, defined, and evolved.
Flink offers a two-fold integration with Hive.
The first is to leverage Hive’s Metastore as a persistent catalog for storing Flink specific metadata across sessions.
The second is to offer Flink as an alternative engine for reading and writing Hive tables.
The hive catalog is designed to be “out of the box” compatible with existing Hive installations.
You do not need to modify your existing Hive Metastore or change the data placement or partitioning of your tables.
Flink supports Hive 2.3.4 and 1.2.1 and relies on Hive’s compatibility guarantee’s for other minor versions.
If you use a different minor Hive version such as 1.2.2 or 2.3.1, it should also be ok to
choose the closest version 1.2.1 (for 1.2.2) or 2.3.4 (for 2.3.1) to workaround. For
example, you want to use Flink to integrate 2.3.1 hive version in sql client, just set the
hive-version to 2.3.4 in YAML config. Similarly pass the version string when creating
HiveCatalog instance via Table API.
Users are welcome to try out different versions with this workaround. Since only 2.3.4 and 1.2.1 have been tested, there might be unexpected issues. We will test and support more versions in future releases.
To integrate with Hive, users need the following dependencies in their project.
Connecting To Hive
Connect to an existing Hive installation using the Hive Catalog through the table environment or YAML configuration.
Currently HiveCatalog supports most Flink data types with the following mapping:
Flink Data Type
Hive Data Type
The following limitations in Hive’s data types impact the mapping between Flink and Hive:
CHAR(p) has a maximum length of 255
VARCHAR(p) has a maximum length of 65535
Hive’s MAP only supports primitive key types while Flink’s MAP can be any data type
Hive’s UNION type is not supported
Flink’s INTERVAL type cannot be mapped to Hive INTERVAL type
Flink’s TIMESTAMP_WITH_TIME_ZONE and TIMESTAMP_WITH_LOCAL_TIME_ZONE are not supported by Hive
Flink’s TIMESTAMP_WITHOUT_TIME_ZONE type cannot be mapped to Hive’s TIMESTAMP type due to precision difference.