Pentaho Data Integration - Kettle
  1. Pentaho Data Integration - Kettle
  2. PDI-9148

As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)

    Details

    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • QA Validation Status:
      Not Yet Validated

      Description

      The idea would be to have a PMR job launched from spoon contain callback information to an embedded Carte server running inside Spoon.
      Then we can have the mappers, reducers and combiners phone home and notify Spoon that they started processing.
      At that point you could somehow get hold of the transformation running on the node, gather data from the steps and push it back to Spoon.
      We could get hold of the transformation again by starting up a copy of Carte for the duration of the transformation, implement RMI ... or we could use some Hadoop specific interface not yet known to me right now.

        Issue Links

          Activity

          Hide
          Matt Casters added a comment -
          This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes.
          We could even push logging information periodically directly into a specific Spoon perspective, ...
          Show
          Matt Casters added a comment - This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes. We could even push logging information periodically directly into a specific Spoon perspective, ...
          Hide
          Jens Bleuel added a comment -
          This is in progress by MattC, Spike version 0.3 on the Vizor PMR :
          http://www.youtube.com/watch?v=0R7oMt3EoXQ
          Show
          Jens Bleuel added a comment - This is in progress by MattC, Spike version 0.3 on the Vizor PMR : http://www.youtube.com/watch?v=0R7oMt3EoXQ
          Hide
          Matt Casters added a comment - - edited
          Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
          Show
          Matt Casters added a comment - - edited Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
          Hide
          Matt Casters added a comment -
          I'm more than done with this.
          Show
          Matt Casters added a comment - I'm more than done with this.

            People

            • Assignee:
              Unassigned User
              Reporter:
              Matt Casters
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: