Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-19064

The Append Option in JSON Output will create new files when the number of rows exceed the number of blocs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Urgent
    • Resolution: Fixed
    • Affects Version/s: 8.1.0 GA, 8.2.0 GA, 8.3.0.1 GA, 9.1.0 GA
    • Fix Version/s: 9.3 GA
    • Component/s: Step
    • Labels:
    • Story Points:
      0
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Tatooine (Maint)
    • Steps to Reproduce:
      Hide
      1. Install PDI Client 8.0
      2. Download the sample KTR:  JSON_Output_Append_1bloc.ktr
      3. Open Spoon 8.0 and open the KTR
      4. Edit the JSON Output step and modify the filename to a valid path
      5. Run the KTR
      6. Notice that each line was appended to the same file
      7. Install PDI Client 8.3
      8. Open Spoon 8.3 and open the KTR
      9. Run the KTR
      10. Notice that 12 separate files were written with one row each

      Actual Result:  Separate files will be written for each bloc

      Expected Result:  Each bloc is appended to the same file.

      Show
      Install PDI Client 8.0 Download the sample KTR:   JSON_Output_Append_1bloc.ktr Open Spoon 8.0 and open the KTR Edit the JSON Output step and modify the filename to a valid path Run the KTR Notice that each line was appended to the same file Install PDI Client 8.3 Open Spoon 8.3 and open the KTR Run the KTR Notice that 12 separate files were written with one row each Actual Result:   Separate files will be written for each bloc Expected Result:   Each bloc is appended to the same file.

      Description

      When writing to a JSON Output, if the number of rows of data exceeds the "Nr rows in a bloc" separate files will be created for those rows even if the Append Option is checked off.

      For example, if there are 12 rows of data and the "Nr rows in a bloc" value is 1, then 12 separate files will be created for each row of data.

      This is extremely problematic for customers processing thousands of rows of data as PDI will generate thousands of files.

      In builds 8.0 and earlier, the Append option would allow for each bloc to append to the same file.  This regression was introduced in 8.1.0.0 by BACKLOG-21738

      Example output in 8.0, which is all in a single file:  out_version80.json

      Example output in 8.3, which is 12 separate files:  out_version83.zip

      Please see ESR-7294 for more details.

        Attachments

          Activity

            People

            Assignee:
            htarannum Hina Tarannum
            Reporter:
            bowens Brittany Owens
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: