Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-18930

JSON Input Step Performance Degradation and Memory Leak

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Urgent
    • Resolution: Fixed
    • Affects Version/s: 8.3.0 GA, 9.0.0 GA
    • Fix Version/s: Backlog, 9.3 GA
    • Component/s: Step
    • Labels:
    • Story Points:
      0
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Tatooine (Maint)
    • Steps to Reproduce:
      Hide
      • Run the provided transformation 2_JIRA_parsing.ktr with test data (see ESR-7184)
      • Compare the results of the performance between versions 7.1, 8.2, 8.3 and 9.0
      • When executing the transformation a second time on versions 8.3 and 9.0 you run into an OOME
      • Details in attached document
      Show
      Run the provided transformation 2_JIRA_parsing.ktr with test data (see ESR-7184) Compare the results of the performance between versions 7.1, 8.2, 8.3 and 9.0 When executing the transformation a second time on versions 8.3 and 9.0 you run into an OOME Details in attached document

      Description

      This has been reported by a customer who is using a smaller data volume with the JSON step and scheduling this transformation every 5 seconds on the Pentaho Server. After a couple of days (>10) the Pentaho server got unresponsive with high CPU and memory utilization. This is most likely related to this memory leak in the JSON step.

      FYI: PDI-18211 was fixed in 9.0, but it seems there is a memory leak left.

      Summary of performance compare:
      7.1 --> about 30 seconds, subsequent runs finish with the same time
      8.2 --> about 50 seconds, subsequent runs finish with the same time
      8.3 --> about 11 minutes, subsequent runs end with OOME
      9.0 --> about 90 seconds, subsequent runs end with OOME

        Attachments

        1. 9.0_2Gb.PNG
          9.0_2Gb.PNG
          138 kB
        2. 9.0_4Gb.PNG
          9.0_4Gb.PNG
          145 kB
        3. 9.1_2Gb.PNG
          9.1_2Gb.PNG
          145 kB
        4. 9.1_4Gb.PNG
          9.1_4Gb.PNG
          156 kB
        5. JSON_Regression.docx
          553 kB
        6. JSON_Regression - 9.1 and SP tests.docx
          1.08 MB

          Issue Links

            Activity

              People

              Assignee:
              htarannum Hina Tarannum
              Reporter:
              jbleuel Jens Bleuel
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: