Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-17918

HBase input failure on missing mapping when using "store mapping in step"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: High
    • Resolution: Fixed
    • Affects Version/s: 8.1.0 GA
    • Fix Version/s: 9.1.0 GA
    • Component/s: Big Data (Hadoop etc.)
    • Labels:
    • Environment:
      PDI 8.1, CentOS 7, HDP 26, OpenJDK 1.8.0-191
    • Story Points:
      8
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Tatooine (Maint)
    • Operating System/s:
      CentOS 7.x
    • Steps to Reproduce:
      Hide
      • create a hbase table
      • create a fresh transformation, and add an hbase input step
      • Add a valid URL to hbase-site.xml and a HBase table name ( of an existing table )
      • Select hadoop cluster
      • leave mapping name empty
      • check "store mapping in step meta data"
      • change to "Create/Edit mappings" and press "Get table names", choose the table that was typed into the field on previous page
      • add rows to the mapping manually
      • get back to previous page ( "Configure query" ) and press "Get/Key / Fields info"
      • Run transformation often
      Show
      create a hbase table create a fresh transformation, and add an hbase input step Add a valid URL to hbase-site.xml and a HBase table name ( of an existing table ) Select hadoop cluster leave mapping name empty check "store mapping in step meta data" change to "Create/Edit mappings" and press "Get table names", choose the table that was typed into the field on previous page add rows to the mapping manually get back to previous page ( "Configure query" ) and press "Get/Key / Fields info" Run transformation often

      Description

      When using the HBase input step with the "Store mapping info in step meta data" option checked to on we face this error occasionally :

      org.pentaho.di.core.exception.KettleException:
      Problem getting mapping information: Mapping "xml_events,null" does not exist!
      Mapping "xml_events,null" does not exist!
              at org.pentaho.big.data.kettle.plugins.hbase.output.HBaseOutput.processRow(HBaseOutput.java:218)
              at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Mapping "xml_events,null" does not exist!
              at org.pentaho.big.data.kettle.plugins.hbase.mapping.MappingAdmin.getMapping(MappingAdmin.java:771)
              at org.pentaho.big.data.kettle.plugins.hbase.output.HBaseOutput.processRow(HBaseOutput.java:215)
              ... 2 more
      

      errors from time to time. The engine is able to use the mapping that is stored in transformation file most of the time but under some ( unknown ) circumstances the engine queries the HBase databases and then complains about the missing mapping.

      Parallelism is increasing the risk of failure - if we start similar jobs that access HBase with in parallel with pan the error occurs more often.

      When storing the mapping in Hbase table "pentaho_mapping" the error never occurs even when the option is used.

        Attachments

        1. hbase_83010_50.log
          81 kB
        2. hbase_90GA_50.log
          50 kB
        3. hbase.ktr
          23 kB
        4. hbase-83010-797.kjb
          11 kB
        5. hbase-83010-797.ktr
          21 kB
        6. job_test.kjb
          39 kB
        7. manyKitchens_hbase_50.log
          52 kB

          Issue Links

            Activity

              People

              Assignee:
              rrosinha Ricardo Rosinha
              Reporter:
              pfabrici Peter Fabricius
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: