Pentaho Data Integration - Kettle

As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)

Details

  • QA Validation Status:
    Not Yet Validated

Description

The idea would be to have a PMR job launched from spoon contain callback information to an embedded Carte server running inside Spoon.
Then we can have the mappers, reducers and combiners phone home and notify Spoon that they started processing.
At that point you could somehow get hold of the transformation running on the node, gather data from the steps and push it back to Spoon.
We could get hold of the transformation again by starting up a copy of Carte for the duration of the transformation, implement RMI ... or we could use some Hadoop specific interface not yet known to me right now.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Casters added a comment -
This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes.
We could even push logging information periodically directly into a specific Spoon perspective, ...
Show
Matt Casters added a comment - This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes. We could even push logging information periodically directly into a specific Spoon perspective, ...
Jared Cornelius made changes -
Field Original Value New Value
Status Open [ 1 ] Open [ 1 ]
Priority Unknown [ 7 ] Severe [ 3 ]
Assignee Triage [ project admin ] Unassigned User [ unassigned ]
Fix Version/s 5.0 BD Hadoop Ease Of Use [ 11535 ]
Doug Moran made changes -
Status Open [ 1 ] Open [ 1 ]
Priority Severe [ 3 ] Unknown [ 7 ]
Assignee Unassigned User [ unassigned ] Triage [ project admin ]
Fix Version/s 5.0 BD Hadoop Ease Of Use [ 11535 ]
Jared Cornelius made changes -
Status Open [ 1 ] Open [ 1 ]
Priority Unknown [ 7 ] Severe [ 3 ]
Assignee Triage [ project admin ] Unassigned User [ unassigned ]
Fix Version/s 5.1.0 GA [ 11632 ]
Hide
Permalink
Jens Bleuel added a comment -
This is in progress by MattC, Spike version 0.3 on the Vizor PMR :
http://www.youtube.com/watch?v=0R7oMt3EoXQ
Show
Jens Bleuel added a comment - This is in progress by MattC, Spike version 0.3 on the Vizor PMR : http://www.youtube.com/watch?v=0R7oMt3EoXQ
Jens Bleuel made changes -
Summary As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)
Jens Bleuel made changes -
Labels q_bad
Jens Bleuel made changes -
Component/s Big Data (Hadoop etc.) [ 10501 ]
Hide
Permalink
Matt Casters added a comment - - edited
Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
Show
Matt Casters added a comment - - edited Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
Jens Bleuel made changes -
Labels q_bad q_bad q_ee
Hide
Permalink
Matt Casters added a comment -
I'm more than done with this.
Show
Matt Casters added a comment - I'm more than done with this.
Matt Casters made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Jens Bleuel made changes -
Fix Version/s 5.0.0 GA (5.0.0 GA Suite Release) [ 11433 ]
Fix Version/s 5.1.0 GA [ 11632 ]
Transition Time In Source Status Execution Times Last Executer Last Execution Date
Open Open Open Open
7d 2h 20m 3 Jared Cornelius 24/Jan/13 9:23 AM
Open Open Resolved Resolved
102d 7h 2m 1 Matt Casters 06/May/13 5:25 PM

People

  • Assignee:
    Unassigned User
    Reporter:
    Matt Casters
Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: