Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-15196

JSON Input Step does not ignore missing paths

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: High
    • Resolution: Fixed
    • Affects Version/s: 6.1.0.1 GA
    • Fix Version/s: 7.1.0 GA
    • Component/s: Step
    • Labels:
      None
    • Story Points:
      0
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Tatooine (Maint)
    • Steps to Reproduce:
      Hide
      • Create a transformation that includes 2 steps: a Data grid, and a JSON input. In the data grid, create several records. Some input records should contain a given JSON path, and others should not. Also insert a JSON input step that attempts to parse all JSON paths
        • Be sure to check the "Ignore missing path" option on the Content tab
        • The attached transformation is an example
      • Run the transformation in a version of PDI previous to 6.1. The transformation will not produce errors
      • Run the transformation in Spoon 6.1, all records that do not contain this path will not be processed, and will produce an error
      Show
      Create a transformation that includes 2 steps: a Data grid, and a JSON input. In the data grid, create several records. Some input records should contain a given JSON path, and others should not. Also insert a JSON input step that attempts to parse all JSON paths Be sure to check the "Ignore missing path" option on the Content tab The attached transformation is an example Run the transformation in a version of PDI previous to 6.1. The transformation will not produce errors Run the transformation in Spoon 6.1, all records that do not contain this path will not be processed, and will produce an error

      Description

      As a user of Pentaho Spoon, I expect the "Ignore missing path" option to function as follows:

      • If the JSON path exists, the field should contain its value
      • If the JSON path does not exist, the field should be null

      This functionality was introduced with PDI-4701, but was regressed in the 6.1 release.

        Attachments

        1. csv_output_6010.csv
          14 kB
        2. csv_output_70_QAT428.csv
          14 kB
        3. JSON_Input_CSV_output.ktr
          14 kB
        4. JSON_Input_ktr_with_ignore_missing_paths_checked.png
          JSON_Input_ktr_with_ignore_missing_paths_checked.png
          39 kB
        5. JSON_input_Test_out_with_checked_unchecked_option.png
          JSON_input_Test_out_with_checked_unchecked_option.png
          57 kB
        6. JSON_Input_Transformation.ktr
          11 kB
        7. JSON_Testcase_output_with_checked_unchecked_option.png
          JSON_Testcase_output_with_checked_unchecked_option.png
          106 kB
        8. Json Input Test.ktr
          13 kB
        9. jSON Testcase.ktr
          12 kB
        10. test_against_6010.png
          test_against_6010.png
          67 kB
        11. test_against_6101.png
          test_against_6101.png
          93 kB
        12. test_against_70_QAT428.png
          test_against_70_QAT428.png
          47 kB
        13. testing_PDI-15196_1.png
          testing_PDI-15196_1.png
          32 kB
        14. testing_PDI-15196_2.png
          testing_PDI-15196_2.png
          108 kB

          Issue Links

            Activity

              People

              Assignee:
              dkrauchanka Dzmitry Krauchanka (Inactive)
              Reporter:
              joseph.bradley Joseph Bradley
              Votes:
              12 Vote for this issue
              Watchers:
              24 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: