Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-9148

As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)

    Details

    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.

      Description

      The idea would be to have a PMR job launched from spoon contain callback information to an embedded Carte server running inside Spoon.
      Then we can have the mappers, reducers and combiners phone home and notify Spoon that they started processing.
      At that point you could somehow get hold of the transformation running on the node, gather data from the steps and push it back to Spoon.
      We could get hold of the transformation again by starting up a copy of Carte for the duration of the transformation, implement RMI ... or we could use some Hadoop specific interface not yet known to me right now.

        Activity

        mcasters Matt Casters created issue -
        Hide
        mcasters Matt Casters added a comment -

        This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes.
        We could even push logging information periodically directly into a specific Spoon perspective, ...

        Show
        mcasters Matt Casters added a comment - This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes. We could even push logging information periodically directly into a specific Spoon perspective, ...
        jcornelius Jared Cornelius made changes -
        Field Original Value New Value
        Status Open [ 1 ] Open [ 1 ]
        Severity Unknown [ 7 ] Severe [ 3 ]
        Assignee Triage [ project admin ] Unassigned User [ unassigned ]
        Fix Version/s 5.0 BD Hadoop Ease Of Use [ 11535 ]
        dmoran Doug Moran made changes -
        Status Open [ 1 ] Open [ 1 ]
        Severity Severe [ 3 ] Unknown [ 7 ]
        Assignee Unassigned User [ unassigned ] Triage [ project admin ]
        Fix Version/s 5.0 BD Hadoop Ease Of Use [ 11535 ]
        jcornelius Jared Cornelius made changes -
        Status Open [ 1 ] Open [ 1 ]
        Severity Unknown [ 7 ] Severe [ 3 ]
        Assignee Triage [ project admin ] Unassigned User [ unassigned ]
        Fix Version/s 5.1.0 GA [ 11632 ]
        Hide
        jbleuel Jens Bleuel added a comment -

        This is in progress by MattC, Spike version 0.3 on the Vizor PMR :
        http://www.youtube.com/watch?v=0R7oMt3EoXQ

        Show
        jbleuel Jens Bleuel added a comment - This is in progress by MattC, Spike version 0.3 on the Vizor PMR : http://www.youtube.com/watch?v=0R7oMt3EoXQ
        jbleuel Jens Bleuel made changes -
        Summary As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)
        jbleuel Jens Bleuel made changes -
        Labels q_bad
        jbleuel Jens Bleuel made changes -
        Component/s Big Data (Hadoop etc.) [ 10501 ]
        Hide
        mcasters Matt Casters added a comment - - edited

        Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project

        Show
        mcasters Matt Casters added a comment - - edited Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
        jbleuel Jens Bleuel made changes -
        Labels q_bad q_bad q_ee
        Hide
        mcasters Matt Casters added a comment -

        I'm more than done with this.

        Show
        mcasters Matt Casters added a comment - I'm more than done with this.
        mcasters Matt Casters made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        jbleuel Jens Bleuel made changes -
        Fix Version/s 5.0.0 GA (5.0.0 GA Suite Release) [ 11433 ]
        Fix Version/s 5.1.0 GA [ 11632 ]
        dmoran Doug Moran made changes -
        Link This issue is related to PDI-10325 [ PDI-10325 ]
        bbruce Brandon Bruce (Inactive) made changes -
        Fix Version/s 5.1.0 Backlog [ 11632 ]
        Fix Version/s 5.0.0 GA (5.0.0 GA Suite Release) [ 11433 ]
        jbleuel Jens Bleuel made changes -
        Fix Version/s 5.1.0 GA [ 12126 ]
        Fix Version/s 5.1.0 Backlog [ 11632 ]
        dkincade David Kincade made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        sbaca Sarah Baca (Inactive) made changes -
        Link This issue is related to PDI-12939 [ PDI-12939 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Open Open
        7d 2h 20m 3 Jared Cornelius 24/Jan/13 9:23 AM
        Open Open Resolved Resolved
        102d 7h 2m 1 Matt Casters 06/May/13 5:25 PM
        Resolved Resolved Closed Closed
        359d 23h 1 David Kincade 01/May/14 4:25 PM

          People

          • Assignee:
            Unassigned
            Reporter:
            mcasters Matt Casters
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: