Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-11046

Cassandra: Input step with ReversedType Comparator

    Details

    • Type: Bug
    • Status: Closed
    • Severity: High
    • Resolution: Fixed
    • Affects Version/s: 5.0.0 GA (5.0.0 GA Suite Release)
    • Fix Version/s: 5.1.0 GA
    • Component/s: Big Data (Hadoop etc.)
    • Labels:
      None
    • Environment:
      DSE Cassandra
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.

      Description

      When using a "ReversedType" comparator for a Cassandra keyspace, the following error occurs on a Cassandra Input step:

      Cassandra Input.0 - ERROR (version 5.0.0.1, build 1 from 2013-09-11_16-51-19 by buildguy) : Unexpected error
      Cassandra Input.0 - ERROR (version 5.0.0.1, build 1 from 2013-09-11_16-51-19 by buildguy) : org.pentaho.di.core.exception.KettleException:
      2013/11/05 16:22:35 - Cassandra Input.0 -
      2013/11/05 16:22:35 - Cassandra Input.0 - Cant find a deserializer for type "

      {0}

      "

      Reproduction steps:

      Create table with CQL3:

      create table cf_topher(id int primary key, location text, created 'ReversedType(org.apache.cassandra.db.marshal.DateType)');
      insert into cf_topher(id, location, created) values (123, 'here', '2013-1-1');

      Then point the cassandra input step at this table. Without the 'ReversedType' it works fine. With the type id does not.

      Attaching screen shots for reference.

      1. test1.PNG
        67 kB
      2. test2.PNG
        87 kB
      3. Unknown.png
        47 kB
      4. Unknown-1.png
        215 kB
      5. Unknown-2.png
        146 kB

        Issue Links

          Activity

          Hide
          dmoran Doug Moran added a comment - - edited

          Mark,

          What would it take to do this?

          Show
          dmoran Doug Moran added a comment - - edited Mark, What would it take to do this?
          Hide
          mhall Mark Hall added a comment -

          Shouldn't be too much effort. An approach similar to that used for CompositeType should do the trick.

          Show
          mhall Mark Hall added a comment - Shouldn't be too much effort. An approach similar to that used for CompositeType should do the trick.
          Hide
          dmoran Doug Moran added a comment - - edited

          Make it so - I'll get it into a sprint. I assume it will be detected automatically and just handled. Fix the error message while in there too.

          Are there any other common types we are missing? Can we look at the schema and dynamically load and use whatever classes are specified to deserialize as long as they are available on the class path?

          Show
          dmoran Doug Moran added a comment - - edited Make it so - I'll get it into a sprint. I assume it will be detected automatically and just handled. Fix the error message while in there too. Are there any other common types we are missing? Can we look at the schema and dynamically load and use whatever classes are specified to deserialize as long as they are available on the class path?
          Hide
          mhall Mark Hall added a comment -

          I'll check for any other missing types. We might be able to do some stuff dynamically. We have to explicitly check for some types however in order to do something that is Kettle-compatible (e.g. CompositeType, DynamicCompositeType). The same will be true for ReversedType - to determine the outgoing field meta data, before actually deserializing any data, the ReversedType schema string will have to be parsed to find the base type that it wraps.

          Show
          mhall Mark Hall added a comment - I'll check for any other missing types. We might be able to do some stuff dynamically. We have to explicitly check for some types however in order to do something that is Kettle-compatible (e.g. CompositeType, DynamicCompositeType). The same will be true for ReversedType - to determine the outgoing field meta data, before actually deserializing any data, the ReversedType schema string will have to be parsed to find the base type that it wraps.
          Show
          mhall Mark Hall added a comment - master: https://github.com/pentaho/pentaho-cassandra-plugin/pull/26
          Show
          mhall Mark Hall added a comment - master: https://github.com/pentaho/pentaho-cassandra-plugin/pull/26
          Hide
          mhall Mark Hall added a comment -

          To validate - create a CQL 3 table as outlined in the description of this case. Previewing a "select *" in CassandraInput should show:

          id created location
          123 2013/01/01 00:00:00.000 here

          Show
          mhall Mark Hall added a comment - To validate - create a CQL 3 table as outlined in the description of this case. Previewing a "select *" in CassandraInput should show: id created location 123 2013/01/01 00:00:00.000 here
          Show
          mhall Mark Hall added a comment - 5.0: https://github.com/pentaho/pentaho-cassandra-plugin/pull/27
          Hide
          adoudkina Antonina Doudkina (Inactive) added a comment -

          Tested on PDI 5.1 nightly build # 659 against DataStax Cassandra 2.0.7 Community Edition.

          Test table was created as described in description section:

          create table cf_topher(id int primary key, location text, created 'ReversedType(org.apache.cassandra.db.marshal.DateType)');
          insert into cf_topher(id, location, created) values (123, 'here', '2013-1-1');

          Running transformation with Cassandra Input step against test table was successful. Data was retrieved correctly. Previewing step data also worked fine.

          Closing.

          Show
          adoudkina Antonina Doudkina (Inactive) added a comment - Tested on PDI 5.1 nightly build # 659 against DataStax Cassandra 2.0.7 Community Edition. Test table was created as described in description section: create table cf_topher(id int primary key, location text, created 'ReversedType(org.apache.cassandra.db.marshal.DateType)'); insert into cf_topher(id, location, created) values (123, 'here', '2013-1-1'); Running transformation with Cassandra Input step against test table was successful. Data was retrieved correctly. Previewing step data also worked fine. Closing.

            People

            • Assignee:
              adoudkina Antonina Doudkina (Inactive)
              Reporter:
              sandeepck Sandeep Kemparaju
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile