Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-8715

PDI Operations Mart: Out of memory exception on loading the data mart on larger log tables

    Details

    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.

      Description

      In the 4 transformations to load the data mart (ETL - Operations Mart ...), the start date is not taken into account in the SELECT query (Read log table information) and the rows are filtered out later by the last run date.
      The "Get Last Run Date" should be taken into account in the SELECT query to avoid unnecessary workload.

      The JavaScript "Read log table information" has an issue since it loads all rows to process into memory and throws an OOME on larger log tables.

        Attachments

          Activity

            People

            • Assignee:
              jbleuel Jens Bleuel
              Reporter:
              jbleuel Jens Bleuel
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: