Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package flink
    Definition Classes
    apache
  • package api
    Definition Classes
    flink
  • package scala

    The Flink Scala API.

    The Flink Scala API. org.apache.flink.api.scala.ExecutionEnvironment is the starting-point of any Flink program. It can be used to read from local files, HDFS, or other sources. org.apache.flink.api.scala.DataSet is the main abstraction of data in Flink. It provides operations that create new DataSets via transformations. org.apache.flink.api.scala.GroupedDataSet provides operations on grouped data that results from org.apache.flink.api.scala.DataSet.groupBy().

    Use org.apache.flink.api.scala.ExecutionEnvironment.getExecutionEnvironment to obtain an execution environment. This will either create a local environment or a remote environment, depending on the context where your program is executing.

    Definition Classes
    api
    Deprecated

    All Flink Scala APIs are deprecated and will be removed in a future Flink version. You can still build your application in Scala, but you should move to the Java version of either the DataStream and/or Table API.

    See also

    FLIP-265 Deprecate and remove Scala API support

  • package extensions

    acceptPartialFunctions extends the original DataSet with methods with unique names that delegate to core higher-order functions (e.g.

    acceptPartialFunctions extends the original DataSet with methods with unique names that delegate to core higher-order functions (e.g. map) so that we can work around the fact that overloaded methods taking functions as parameters can't accept partial functions as well. This enables the possibility to directly apply pattern matching to decompose inputs such as tuples, case classes and collections.

    The following is a small example that showcases how this extensions would work on a Flink data set:

    object Main {
      import org.apache.flink.api.scala.extensions._
      case class Point(x: Double, y: Double)
      def main(args: Array[String]): Unit = {
        val env = ExecutionEnvironment.getExecutionEnvironment
        val ds = env.fromElements(Point(1, 2), Point(3, 4), Point(5, 6))
        ds.filterWith {
          case Point(x, _) => x > 1
        }.reduceWith {
          case (Point(x1, y1), (Point(x2, y2))) => Point(x1 + y1, x2 + y2)
        }.mapWith {
          case Point(x, y) => (x, y)
        }.flatMapWith {
          case (x, y) => Seq('x' -> x, 'y' -> y)
        }.groupingBy {
          case (id, value) => id
        }
      }
    }

    The extension consists of several implicit conversions over all the data set representations that could gain from this feature. To use this set of extensions methods the user has to explicitly opt-in by importing org.apache.flink.api.scala.extensions.acceptPartialFunctions.

    For more information and usage examples please consult the Apache Flink official documentation.

    Definition Classes
    scala
    Deprecated

    All Flink Scala APIs are deprecated and will be removed in a future Flink major version. You can still build your application in Scala, but you should move to the Java version of either the DataStream and/or Table API.

    See also

    FLIP-265 Deprecate and remove Scala API support

  • package impl
    Definition Classes
    extensions
  • package acceptPartialFunctions
    Definition Classes
    impl
  • OnCoGroupDataSet
  • OnCrossDataSet
  • OnDataSet
  • OnGroupedDataSet
  • OnHalfUnfinishedKeyPairOperation
  • OnJoinFunctionAssigner
  • OnUnfinishedKeyPairOperation

class OnDataSet[T] extends AnyRef

Wraps a data set, allowing to use anonymous partial functions to perform extraction of items in a tuple, case class instance or collection

T

The type of the data set items

Annotations
@deprecated
Deprecated

(Since version 1.18.0)

See also

FLIP-265 Deprecate and remove Scala API support

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OnDataSet
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OnDataSet(ds: DataSet[T])

    ds

    The wrapped data set

    Deprecated

    All Flink Scala APIs are deprecated and will be removed in a future Flink major version. You can still build your application in Scala, but you should move to the Java version of either the DataStream and/or Table API.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def filterWith(fun: (T) ⇒ Boolean): DataSet[T]

    Applies a predicate fun to each item of the data set, keeping only those for which the predicate holds

    Applies a predicate fun to each item of the data set, keeping only those for which the predicate holds

    fun

    The predicate to be tested on each item

    returns

    A dataset of R

    Annotations
    @PublicEvolving()
  9. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. def flatMapWith[R](fun: (T) ⇒ TraversableOnce[R])(implicit arg0: TypeInformation[R], arg1: ClassTag[R]): DataSet[R]

    Applies a function fun to each item of the dataset, producing a collection of items that will be flattened in the resulting data set

    Applies a function fun to each item of the dataset, producing a collection of items that will be flattened in the resulting data set

    R

    The type of the items in the returned data set

    fun

    The function to be applied to each item

    returns

    A dataset of R

    Annotations
    @PublicEvolving()
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def groupingBy[K](fun: (T) ⇒ K)(implicit arg0: TypeInformation[K]): GroupedDataSet[T]

    Groups the items according to a grouping function fun

    Groups the items according to a grouping function fun

    K

    The return type of the grouping function, for which type information must be known

    fun

    The grouping function

    returns

    A grouped data set of Ts

    Annotations
    @PublicEvolving()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. def mapPartitionWith[R](fun: (Stream[T]) ⇒ R)(implicit arg0: TypeInformation[R], arg1: ClassTag[R]): DataSet[R]

    Applies a function fun to a partition as a whole

    Applies a function fun to a partition as a whole

    R

    The type of the items in the returned data set

    fun

    The function to be applied on the whole partition

    returns

    A dataset of R

    Annotations
    @PublicEvolving()
  16. def mapWith[R](fun: (T) ⇒ R)(implicit arg0: TypeInformation[R], arg1: ClassTag[R]): DataSet[R]

    Applies a function fun to each item of the data set

    Applies a function fun to each item of the data set

    R

    The type of the items in the returned data set

    fun

    The function to be applied to each item

    returns

    A dataset of R

    Annotations
    @PublicEvolving()
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. def reduceGroupWith[R](fun: (Stream[T]) ⇒ R)(implicit arg0: TypeInformation[R], arg1: ClassTag[R]): DataSet[R]

    Applies a reducer fun to a grouped data set

    Applies a reducer fun to a grouped data set

    R

    The type of the items in the returned data set

    fun

    The function to be applied to the whole grouping

    returns

    A dataset of Rs

    Annotations
    @PublicEvolving()
  21. def reduceWith(fun: (T, T) ⇒ T): DataSet[T]

    Applies a reducer fun to the data set

    Applies a reducer fun to the data set

    fun

    The reducing function to be applied on the whole data set

    returns

    A data set of Rs

    Annotations
    @PublicEvolving()
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. def toString(): String
    Definition Classes
    AnyRef → Any
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped