Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-18205

'Copy Rows to Result' step is not passing rows to later transformation in job if transformations in between do not use 'Copy Rows to Result'

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Urgent
    • Resolution: Fixed
    • Affects Version/s: 8.2.0.6 GA, 8.3.0 GA
    • Fix Version/s: 9.0.0 GA
    • Component/s: Job
    • Labels:
    • Story Points:
      0
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Tatooine (Maint)
    • Steps to Reproduce:
      Hide
      • Download the provided transformations and jobs to one folder
      • in version 8.2.0.6 or 8.2.0.7 or 8.3.0 run the job & in the Execution Results see the results described in the Description (the row from transformation 1 is not received by transformation 4)
      • In a version prior to 8.2.0.6 run the job & in the Execution Results see the results described in the Description  (the row from transformation 1 is received by transformation 4)
      Show
      Download the provided transformations and jobs to one folder in version 8.2.0.6 or 8.2.0.7 or 8.3.0 run the job & in the Execution Results see the results described in the Description (the row from transformation 1 is not received by transformation 4) In a version prior to 8.2.0.6 run the job & in the Execution Results see the results described in the Description  (the row from transformation 1 is received by transformation 4)

      Description

      In a Job, rows from previous transformations (which use 'Copy Rows to Result' step) are not being passed to later transformations (which use 'Get rows from result') if there are multiple transformations in the chain in between (even if the transformations in the 'middle' of the chain do nothing).

      Example testcase has been uploaded. It is defined like this:

      Job:
       Start-> Transformation1> Transformation 2> Transformation 3> Transformation 4> Success
      
      Transformation 1: 
       Data Grid - outputs text abc.txt
       Copy rows to results
      
      Transformation 2: 
       Data Grid - contains no data
       Dummy
      
      Transformation 3: 
       Data Grid - contains no data
       Dummy
      
      Transformation 4:
       Get rows from result-->Dummy--> Write to log
       (We also feed a Data Grid into the dummy step, which sends text 'test.txt' just to test it is receiving something)
      

      Result:

      When the job is run in a version prior to 8.2.0.3 or 8.1.0.6 (we suspect in any version prior to 8.2.0.6) then this is the result in the Transformation 4 section:

      2019/07/17 16:13:23 - test_job - Starting entry [Transformation 4]
       2019/07/17 16:13:23 - Transformation 4 - Using run configuration [Pentaho local]
       2019/07/17 16:13:23 - Transformation 4 - Using legacy execution engine
       2019/07/17 16:13:23 - test4 - Dispatching started for transformation [test4]
       2019/07/17 16:13:24 - Get rows from result.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
       2019/07/17 16:13:24 - Data grid.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
       2019/07/17 16:13:24 - Write to log.0 - 
       2019/07/17 16:13:24 - Write to log.0 - ------------> Linenr 1------------------------------
       2019/07/17 16:13:24 - Write to log.0 - filename = abc.txt
       2019/07/17 16:13:24 - Write to log.0 - 
       2019/07/17 16:13:24 - Write to log.0 - ====================
       2019/07/17 16:13:24 - Write to log.0 - 
       2019/07/17 16:13:24 - Write to log.0 - ------------> Linenr 2------------------------------
       2019/07/17 16:13:24 - Write to log.0 - filename = test.txt
       2019/07/17 16:13:24 - Write to log.0 - 
       2019/07/17 16:13:24 - Write to log.0 - ====================
      

      When the job is run in version 8.2.0.7 or 8.3.0 (we suspect in 8.2.0.6 or any version after) then this is the result in the Transformation 4 section:

      2019/07/17 16:18:26 - test_job - Starting entry [Transformation 4]
       2019/07/17 16:18:26 - Transformation 4 - Using run configuration [Pentaho local]
       2019/07/17 16:18:26 - Transformation 4 - Using legacy execution engine
       2019/07/17 16:18:26 - test4 - Dispatching started for transformation [test4]
       2019/07/17 16:18:26 - Data grid.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
       2019/07/17 16:18:26 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
       2019/07/17 16:18:26 - Write to log.0 - 
       2019/07/17 16:18:26 - Write to log.0 - ------------> Linenr 1------------------------------
       2019/07/17 16:18:26 - Write to log.0 - filename = test.txt
       2019/07/17 16:18:26 - Write to log.0 - 
       2019/07/17 16:18:26 - Write to log.0 - ====================
      

      Notes:

      • Note that 'abc.txt' is NOT received by Transformation 4 in version 8.2.0.7 or 8.3.0
      • In 8.2.0.7 & 8.3.0 if we edit the job so that Transformations 2 & 3 are skipped (so it goes straight from transformation1 to transformation 4) then it works as in version 8.2.0.3 and 8.1.0.6 (abc.txt is received by transformation 4).
      • We suspect this has been caused by the change in behaviour from this bug:
        PDI-17926
        Title: Copy Rows to Results: Clear Result Rows at initialization
      • Bug PDI-17926 was reported in version 8.2.0GA and has been 'fixed' in versions 8.2.0.6 and 8.3.0GA

        Attachments

        1. image-2019-07-23-14-41-31-373.png
          image-2019-07-23-14-41-31-373.png
          52 kB
        2. test_job.kjb
          15 kB
        3. test1.ktr
          13 kB
        4. test2.ktr
          12 kB
        5. test3.ktr
          12 kB
        6. test4.ktr
          14 kB

          Issue Links

            Activity

              People

              Assignee:
              rrosinha Ricardo Rosinha
              Reporter:
              sfrench Stuart French
              Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: