Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-16806

AEL Spark: Support for Event Time and Windowed Aggregations

    Details

    • Type: New Feature
    • Status: Open
    • Severity: Unknown
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      0
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • PM Ranking:
      High

      Description

      As a PDI developer I want to pick a timestamp from the data stream and define it as event time. Next I want to use a window to aggregated events (e.g. Show me the count of users in the last 10 minutes). Support sliding and tumbling windows.

      Using event time in windowed aggregations is one of the most important features in Spark Structured Streaming as well as Flink and PDI AEL should support it.

        Attachments

          Activity

            People

            • Assignee:
              rsaha Rakesh Saha
              Reporter:
              diddy Diethard Steiner
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: