Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-7924

As an ETL developer I want to have built-in support for load balancing in Kettle transformations

    XMLWordPrintable

    Details

    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • CI Build:
      pdi-ce-5.0.0-M1-r17378

      Description

      There are2 methods of passing rows between steps: copy and distribute.
      It should be possible to make the distribution of rows optionally "smarter" by looking at the size of the output row set buffer.
      The new "load balancing" option should work exactly like distribution unless the target step is too slow and the associated row set fills up.
      In that case the algorithm should switch to the next row set and so on effectivelly implementing load balancing.

        Attachments

        1. copy-distribute-loadbalance-dialog.png
          copy-distribute-loadbalance-dialog.png
          52 kB
        2. copy-distribute-loadbalance-hops.png
          copy-distribute-loadbalance-hops.png
          19 kB
        3. copy-distribute-loadbalance-menu.png
          copy-distribute-loadbalance-menu.png
          41 kB
        4. slave1.PNG
          slave1.PNG
          41 kB
        5. slave2.PNG
          slave2.PNG
          41 kB
        6. slave3.PNG
          slave3.PNG
          41 kB
        7. test-case-7924.ktr
          30 kB
        8. test-case-7924-Carte.ktr
          16 kB

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                mcasters Matt Casters (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: