Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-1463

Loops in Jobs lead to StackOverflowError after many loops (possible also OutOfMemoryError: Java heap space)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: High
    • Resolution: Fixed
    • Affects Version/s: 2.5.2, 3.0.0 GA, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0 GA, 4.2.0 GA (4.0.0 GA Suite Release), 5.1.0 GA
    • Fix Version/s: 6.0.0 GA
    • Component/s: Job
    • Labels:
      None
    • Story Points:
      2
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      R2D2

      Description

      Loops in Jobs lead to StackOverflowError, even when the StackSize is highered. A very simplified example is attached that errors after some minutes/seconds depending on the machine. This can happen in the field after some hours, days or weeks.

      After the ending of the job the JVM is mostly working on cleaning the stack and looks like it is dead with 100% of CPU - but it survives after some minutes.

      Circumvention: Externalize the loop to the operating system.

      This is because of a design issue in Method Job.execute() that calls the next job entry recursive.
      I don't have a better idea, but one could think of changing this method
      private Result execute(final int nr, Result prev_result, final JobEntryCopy startpoint, JobEntryCopy previous, String reason) throws KettleException
      to
      private Result executeInner(final int nr, Result prev_result, final JobEntryCopy startpoint, JobEntryCopy previous, String reason) throws KettleException

      and do a small exeute method with only the loop inside and call the executeInner in this loop, thus the recursive calls can be avoided.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              elopes Eduardo Lopes (Inactive)
              Reporter:
              jbleuel Jens Bleuel
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h
                  8h