Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-19070

Jobs using a Kafka consumer ends when is expected to never end

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Urgent
    • Resolution: Fixed
    • Affects Version/s: 8.3.0.3 GA
    • Fix Version/s: Backlog
    • Component/s: Step
    • Labels:
      None
    • Story Points:
      13
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Ackbar
    • Steps to Reproduce:
      Hide

      This is non-deterministic behavior without sufficient error logging so replication and root cause analysis is difficult or impossible.

      Show
      This is non-deterministic behavior without sufficient error logging so replication and root cause analysis is difficult or impossible.

      Description

      The transformations that a contains Kafka consumer step is expected to run indefinitely until explicitly terminated by human intervention. However, the customer is experiencing a condition where the transformation will terminate unexpectedly and there is no trace errors presented in the log for root cause analysis at the BASIC level. 
      Instead, the transformation terminates with:

      Kafka murex.0 - Finished processing (I=4193, O=0, R=0, W=4193, U=0, E=0)
      KafkaPersister_Executor_Murex - Starting entry [Success]

      This is the error. The customer expects a Kafka-based solution to run indefinitely and never "Finish processing."

      The customer is resistant to increase logging level because this only occurs in productions and that would have a business impact. They have replaced Pentaho in Production with another product until we can implement and deliver a fix to this condition.

       

      Logging files are provided is support case https://pentaho.zendesk.com/agent/tickets/99153.

        Attachments

        1. dockerComposeKafkaCluster.zip
          3 kB
        2. KafkaConsumerNoDelay.ktr
          13 kB
        3. kafkaConsumerSubNoDelay.ktr
          14 kB
        4. KafkaPersister.zip
          33 kB
        5. kafkaProducer.ktr
          102 kB
        6. logs(2).zip
          83 kB
        7. Screenshot from 2021-02-18 17-14-23.png
          Screenshot from 2021-02-18 17-14-23.png
          133 kB
        8. Screenshot from 2021-02-18 17-15-26.png
          Screenshot from 2021-02-18 17-15-26.png
          76 kB
        9. Screenshot from 2021-02-18 17-18-04.png
          Screenshot from 2021-02-18 17-18-04.png
          145 kB
        10. Screenshot from 2021-02-18 17-20-43.png
          Screenshot from 2021-02-18 17-20-43.png
          175 kB
        11. Screenshot from 2021-02-23 11-08-04.png
          Screenshot from 2021-02-23 11-08-04.png
          41 kB

          Activity

            People

            Assignee:
            jberdecia Jose Berdecia
            Reporter:
            sbrown Steven Brown
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: