Pentaho Data Integration - Kettle

As a Pentaho Map/Reduce developer I want to remotely debug mappers, reducers and combiners (Vizor PMR)

Details

  • QA Validation Status:
    Not Yet Validated

Description

The idea would be to have a PMR job launched from spoon contain callback information to an embedded Carte server running inside Spoon.
Then we can have the mappers, reducers and combiners phone home and notify Spoon that they started processing.
At that point you could somehow get hold of the transformation running on the node, gather data from the steps and push it back to Spoon.
We could get hold of the transformation again by starting up a copy of Carte for the duration of the transformation, implement RMI ... or we could use some Hadoop specific interface not yet known to me right now.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Casters added a comment -
This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes.
We could even push logging information periodically directly into a specific Spoon perspective, ...
Show
Matt Casters added a comment - This would also provide more transparency in Spoon as to which transformations are being started as Mapper, reducer, combiner and on which nodes. We could even push logging information periodically directly into a specific Spoon perspective, ...
Hide
Permalink
Jens Bleuel added a comment -
This is in progress by MattC, Spike version 0.3 on the Vizor PMR :
http://www.youtube.com/watch?v=0R7oMt3EoXQ
Show
Jens Bleuel added a comment - This is in progress by MattC, Spike version 0.3 on the Vizor PMR : http://www.youtube.com/watch?v=0R7oMt3EoXQ
Hide
Permalink
Matt Casters added a comment - - edited
Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
Show
Matt Casters added a comment - - edited Project code copied to the private github.com/pentaho/pentaho-pmr-vizor project
Hide
Permalink
Matt Casters added a comment -
I'm more than done with this.
Show
Matt Casters added a comment - I'm more than done with this.

People

  • Assignee:
    Unassigned User
    Reporter:
    Matt Casters
Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: