Pentaho Data Integration - Kettle
  1. Pentaho Data Integration - Kettle
  2. PDI-9148

As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)

    Details

    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • QA Validation Status:
      Not Yet Validated

      Description

      The idea would be to have a PMR job launched from spoon contain callback information to an embedded Carte server running inside Spoon.
      Then we can have the mappers, reducers and combiners phone home and notify Spoon that they started processing.
      At that point you could somehow get hold of the transformation running on the node, gather data from the steps and push it back to Spoon.
      We could get hold of the transformation again by starting up a copy of Carte for the duration of the transformation, implement RMI ... or we could use some Hadoop specific interface not yet known to me right now.

        Issue Links

          Activity

          Matt Casters created issue -
          Hide
          Matt Casters added a comment -
          This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes.
          We could even push logging information periodically directly into a specific Spoon perspective, ...
          Show
          Matt Casters added a comment - This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes. We could even push logging information periodically directly into a specific Spoon perspective, ...
          Jared Cornelius made changes -
          Field Original Value New Value
          Status Open [ 1 ] Open [ 1 ]
          Priority Unknown [ 7 ] Severe [ 3 ]
          Assignee Triage [ project admin ] Unassigned User [ unassigned ]
          Fix Version/s 5.0 BD Hadoop Ease Of Use [ 11535 ]
          Doug Moran made changes -
          Status Open [ 1 ] Open [ 1 ]
          Priority Severe [ 3 ] Unknown [ 7 ]
          Assignee Unassigned User [ unassigned ] Triage [ project admin ]
          Fix Version/s 5.0 BD Hadoop Ease Of Use [ 11535 ]
          Jared Cornelius made changes -
          Status Open [ 1 ] Open [ 1 ]
          Priority Unknown [ 7 ] Severe [ 3 ]
          Assignee Triage [ project admin ] Unassigned User [ unassigned ]
          Fix Version/s 5.1.0 GA [ 11632 ]
          Hide
          Jens Bleuel added a comment -
          This is in progress by MattC, Spike version 0.3 on the Vizor PMR :
          http://www.youtube.com/watch?v=0R7oMt3EoXQ
          Show
          Jens Bleuel added a comment - This is in progress by MattC, Spike version 0.3 on the Vizor PMR : http://www.youtube.com/watch?v=0R7oMt3EoXQ
          Jens Bleuel made changes -
          Summary As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)
          Jens Bleuel made changes -
          Labels q_bad
          Jens Bleuel made changes -
          Component/s Big Data (Hadoop etc.) [ 10501 ]
          Hide
          Matt Casters added a comment - - edited
          Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
          Show
          Matt Casters added a comment - - edited Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
          Jens Bleuel made changes -
          Labels q_bad q_bad q_ee
          Hide
          Matt Casters added a comment -
          I'm more than done with this.
          Show
          Matt Casters added a comment - I'm more than done with this.
          Matt Casters made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jens Bleuel made changes -
          Fix Version/s 5.0.0 GA (5.0.0 GA Suite Release) [ 11433 ]
          Fix Version/s 5.1.0 GA [ 11632 ]
          Doug Moran made changes -
          Link This issue is related to PDI-10325 [ PDI-10325 ]
          Brandon Bruce made changes -
          Fix Version/s 5.1.0 Backlog [ 11632 ]
          Fix Version/s 5.0.0 GA (5.0.0 GA Suite Release) [ 11433 ]
          Jens Bleuel made changes -
          Fix Version/s 5.1.0 GA [ 12126 ]
          Fix Version/s 5.1.0 Backlog [ 11632 ]
          David Kincade made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Open Open
          7d 2h 20m 3 Jared Cornelius 24/Jan/13 9:23 AM
          Open Open Resolved Resolved
          102d 7h 2m 1 Matt Casters 06/May/13 5:25 PM
          Resolved Resolved Closed Closed
          359d 23h 1 David Kincade 01/May/14 4:25 PM

            People

            • Assignee:
              Unassigned User
              Reporter:
              Matt Casters
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: