Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-1756

Parallel execution of transformations is not working (with repository)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Medium
    • Resolution: Fixed
    • Affects Version/s: 3.1.0 GA
    • Fix Version/s: 3.1.1, 3.2.0 GA
    • Component/s: Job
    • Labels:
      None
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.

      Description

      If we select an option "Launch next entries in parallel" in the job step then PDI fails with repository exceptions. Example: simple Start step connected to "Concurrent 1" and "Concurrent 2" transformations (that are simple generators of 1000 rows) gives us this:

      2008/10/03 11:36:52 - Concurent 1 - Dispatching started for transformation [Concurent 1]
      2008/10/03 11:36:52 - Concurent 1 - This transformation can be replayed with replay date: 2008/10/03 11:36:52
      2008/10/03 11:36:52 - Generate Rows.0 - Finished processing (I=0, O=0, R=0, W=1000, U=1000, E=0)
      2008/10/03 11:36:52 - Test Concurency - Finished jobentry [START] (result=[

      {1}

      ])
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error occurred while launching entry [Concurent 2.0]
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error during transformation metadata load
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error occurred while launching entry [Concurent 2.0]
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error during transformation metadata load
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job$1.run(Job.java:492)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at java.lang.Thread.run(Unknown Source)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Caused by: org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error during transformation metadata load
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:800)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:434)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job.execute(Job.java:410)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job.access$0(Job.java:375)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job$1.run(Job.java:486)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : ... 1 more
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Caused by: org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2476)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2286)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:778)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : ... 5 more
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Caused by: org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2460)
      2008/10/03 11:36:52 - Test Concurency - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : ... 7 more
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : A serious error occurred during job execution:
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error occurred while launching entry [Concurent 2.0]
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error during transformation metadata load
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error occurred while launching entry [Concurent 2.0]
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error during transformation metadata load
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job$1.run(Job.java:492)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at java.lang.Thread.run(Unknown Source)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Caused by: org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Unexpected error during transformation metadata load
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:800)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.entries.trans.JobEntryTrans.execute(JobEntryTrans.java:434)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job.execute(Job.java:410)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job.access$0(Job.java:375)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.Job$1.run(Job.java:486)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : ... 1 more
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Caused by: org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : An error occured reading a transformation from the repository
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2476)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2286)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.job.entries.trans.JobEntryTrans.getTransMeta(JobEntryTrans.java:778)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : ... 5 more
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : Caused by: org.pentaho.di.core.exception.KettleException:
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : This file doesn't exist : Concurent 2
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) :
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : at org.pentaho.di.trans.TransMeta.<init>(TransMeta.java:2460)
      2008/10/03 11:36:52 - Thread[Test Concurency (Test Concurency (Thread-258)),5,main] - ERROR (version 3.1.0, build 826 from 2008/09/30 11:32:36) : ... 7 more
      2008/10/03 11:36:52 - Spoon - Job has ended.
      2008/10/03 11:37:02 - Spoon - edit job graph entry: Concurent 2
      2008/10/03 11:37:08 - Spoon - edit job graph entry: Concurent 1

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gdavid Golda Thomas
                Reporter:
                len0x Anton Oleynikov
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: