Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-18954

Mapping (sub-transformation) tries to verify variables before they are passed

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Severity: High
    • Resolution: Unresolved
    • Affects Version/s: 8.3.0 GA, 9.0.0.5 GA
    • Fix Version/s: Backlog
    • Component/s: Step
    • Labels:
      None
    • Story Points:
      0
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Steps to Reproduce:
      Hide
      1. Install PDI 8.3 or 9.0
      2. Open PDI and set up a Hadoop cluster of your choice.  This was tested with CDH 5.13 and CDH 6.1.
      3. Download the zip file PMR_Sample.zip
      4. Open new_sample_emp_job.kjb and Edit the Copy Files to HDFS.
      5. Configure the files to valid locations on the HDFS
      6. Edit emp_val
      7. Edit the transformation to point to the valid path for new_sample_emp_dtl.ktr
      8. Also edit the Cluster tab and select the proper Hadoop cluster
      9. Open the Job Properties.
      10. Change the parameters KETTLE_LOOKUP and KETTLE_SUBTRANS to the proper locations on hdfs where you are copying the files with Copy to HDFS step.
      11. Open new_sample_emp_dtl.ktr
      12. Open the Transformation Properties.
      13. Change the default value to the location of the dummy_trans2.ktr
      14. Run the Job.  It should succeed successfully.
      15. Open new_sample_emp_dtl.ktr again.
      16. Open Transformation properties and remove the default value.
      17. Open the Mapping (sub-transformation) step, change nothing, and click ok.
      18. Notice the error that the transformation path is now invalid.
      19. Cancel out of the message and be sure to leave the default value of the parameter in new_sample_emp_dtl.ktr blank, and save the transformation.
      20. Run the new_sample_emp_job.kjb

      Actual Results:  The job will error due to the Mapping (sub-transformation) step not being able to validate that transformation being passed as a parameter through PMR

      Expected Results:  The step should be able to take into account parameters that will be passed into it at run time.

      Show
      Install PDI 8.3 or 9.0 Open PDI and set up a Hadoop cluster of your choice.  This was tested with CDH 5.13 and CDH 6.1. Download the zip file  PMR_Sample.zip Open new_sample_emp_job.kjb and Edit the Copy Files to HDFS. Configure the files to valid locations on the HDFS Edit emp_val Edit the transformation to point to the valid path for new_sample_emp_dtl.ktr Also edit the Cluster tab and select the proper Hadoop cluster Open the Job Properties. Change the parameters KETTLE_LOOKUP and KETTLE_SUBTRANS to the proper locations on hdfs where you are copying the files with Copy to HDFS step. Open new_sample_emp_dtl.ktr Open the Transformation Properties. Change the default value to the location of the dummy_trans2.ktr Run the Job.  It should succeed successfully. Open new_sample_emp_dtl.ktr again. Open Transformation properties and remove the default value. Open the Mapping (sub-transformation) step, change nothing, and click ok. Notice the error that the transformation path is now invalid. Cancel out of the message and be sure to leave the default value of the parameter in new_sample_emp_dtl.ktr blank, and save the transformation. Run the new_sample_emp_job.kjb Actual Results:   The job will error due to the Mapping (sub-transformation) step not being able to validate that transformation being passed as a parameter through PMR Expected Results:   The step should be able to take into account parameters that will be passed into it at run time.

      Description

      When using PMR to pass variables to your mapping transformation, you need to pass these through the User Defined tab, prefixed with KETTLE_.

      If you use a variable in the Mapping (sub-transformation) step in the mapper transformation, this variable must be able to be resolved prior to runtime in order to set the variable in the step.  If not, you will receive a similar error.

      However, if you remove the default value of the variable later, since PMR will be passing that value to the step, the job will fail at runtime before even executing the transformation.

      You must set the parameter in the mapping transformation to a dummy mapping transformation that it will be able to locate on runtime (local to the disk) for the step to validate and allow the job to run.   The correct parameter does eventually get passed and the dummy ktr won't be the transformation that is ran, but without using a dummy ktr the job will never get that far.

      Because of this, customers will have to edit every mapping transformation they use parameters in and point it to a dummy mapping ktr, which there may be hundreds of these.

        Attachments

        1. PMR_Sample.zip
          57 kB
        2. PMR_USER_DEFINED.PNG
          PMR_USER_DEFINED.PNG
          12 kB
        3. Spoon_Error.txt
          4 kB
        4. Variable_Error.PNG
          Variable_Error.PNG
          33 kB
        5. workaround.PNG
          workaround.PNG
          20 kB

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bowens Brittany Owens
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated: