Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-7453

Java Heap Errors - Looping Over A Sub Job

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Severity: Medium
    • Resolution: Unresolved
    • Affects Version/s: 4.2.1 (4.1.0 GA Suite Release)
    • Fix Version/s: Backlog
    • Component/s: Job, Kitchen, Step, Transformation
    • Labels:
      None
    • Environment:
      Ubuntu Lucid 10.04
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Browser:
      Mozilla Firefox 9.0.x
    • Operating System/s:
      Ubuntu 10.x (32-bit)

      Description

      I am using Kitchen (Kettle version 4.2.1-stable, build 15952, build date : 2011-10-25 15.27.10) to run a job that processes a potentially large number of CSV files and inserts them into a MySQL (5.14.1) database. An outer job which manipulates all the files, then calls a sub job which processes one CSV file at a time. While running the job it is possible to see that the memory usage slowly increases until eventually a heap error is thrown. Increasing the memory available to PDI only delays matters and the job eventually crashes anyway.

      I have tried manipulating the variables:
      KETTLE_MAX_LOG_SIZE_IN_LINES
      KETTLE_MAX_LOG_TIMEOUT_IN_MINUTES
      KETTLE_MAX_JOB_ENTRIES_LOGGED
      KETTLE_MAX_JOB_TRACKER_SIZE
      KETTLE_MAX_LOGGING_REGISTRY_SIZE

      The effect of changing these appears to be quite minimal to the issue I am experiencing.

      I have tried these transformations using a snapshot build of 4.3.0 (Kettle version 4.3.0, build 1, build date : 2012-02-23 11.19.41) and have noted the same issue.

      This case was first raised in the forum by me here: http://forums.pentaho.com/showthread.php?90529-Memory-Problems-Looping-a-Sub-Job

        Attachments

        1. job_and_files_repro.zip
          534 kB
        2. job and files.zip
          486 kB
        3. PDI-7453_Heap.PNG
          PDI-7453_Heap.PNG
          48 kB
        4. PDI-7453-classes.PNG
          PDI-7453-classes.PNG
          36 kB

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              lostinthewoods Stephen Barton
              Votes:
              4 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated: