Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-19045

AMQP Producer fails to write to queue due to missing configure permissions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Urgent
    • Resolution: Fixed
    • Affects Version/s: 9.1.0 GA
    • Fix Version/s: 9.2.0 GA
    • Component/s: Step
    • Labels:
      None
    • Story Points:
      3
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Ackbar
    • Steps to Reproduce:
      Hide

      Setup RabbitMQ

      1. Install Erlang (MS Windows: https://www.erlang.org/downloads) (otp_win64_23.1.exe)
      2. Install RabbitMQ (MS Windows use this link: https://www.rabbitmq.com/install-windows.html#installer) (rabbitmq-server-3.8.9.exe)
      3. After installation is complete; check the RabbitMQ log located here:
        C:\Users\carlopezx\AppData\Roaming\RabbitMQ\log\rabbit@PC0FTPBT.log

      Setup Spoon 9.1

      1. Download attached transformations
      2. Run the attached AMQP_Direct_Producer.ktr
      3. Open the command prompt and browse to this location:
        cd C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.9\sbin
      4. Notice we connect with the following entries:
        2020-12-14 16:34:31.045 [info] <0.1902.0> accepting AMQP connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672)
        2020-12-14 16:34:31.061 [info] <0.1902.0> connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
        
      5. Run the attached: AMQP_Direct_Consumer.ktr
      6. Notice we connect with the following entries:
        2020-12-14 16:38:25.580 [info] <0.1963.0> accepting AMQP connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672)
        2020-12-14 16:38:25.584 [info] <0.1963.0> connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
        
      7. Stop both the producer and consumer transformations

      Setup user with full access: Configure, Write and Read permissions

      1. Create a virtual host for Pentaho; on the command prompt type the following:
        rabbitmqctl.bat add_vhost ptho
      2. Run the following command to add the user pentaho:
        rabbitmqctl.bat add_user "pentaho" "pentaho123"
      3. Run the following command to set permissions to pentaho:
        rabbitmqctl.bat set_permissions -p "ptho" "pentaho" ".*" ".*" ".*"
      4. Edit the AMQP_Direct_Producer.ktr
        Setup > Connection: amqp://localhost/ptho
        Security: pentaho:pentaho123
      5. Check the logs you should see something like this:
        2020-12-15 09:20:25.803 [info] <0.899.0> accepting AMQP connection <0.899.0> (127.0.0.1:51651 -> 127.0.0.1:5672)
        2020-12-15 09:20:25.845 [info] <0.899.0> connection <0.899.0> (127.0.0.1:51651 -> 127.0.0.1:5672): user 'pentaho' authenticated and granted access to vhost 'ptho'
        
      6. Edit the AMQP_Direct_Consumer.ktr
        Setup > Connection: amqp://localhost/ptho
        Security: pentaho:pentaho123
      7. Check the logs you should see something like this:
        2020-12-15 09:22:16.130 [info] <0.946.0> accepting AMQP connection <0.946.0> ([::1]:51682 -> [::1]:5672)
        2020-12-15 09:22:16.135 [info] <0.946.0> connection <0.946.0> ([::1]:51682 -> [::1]:5672): user 'pentaho' authenticated and granted access to vhost 'ptho'
        
      8. Stop both the producer and consumer transformations

      Setup user with limited access: Read and Write

      1. Run the following command to add the user pentaho-client:
        rabbitmqctl.bat add_user "pentaho-client" "pentaho123"
      2. Run the following command to set permissions to pentaho-client:
        rabbitmqctl.bat set_permissions -p "ptho" "pentaho-client" "^$" ".*" ".*"
      3. Edit the AMQP_Direct_Producer.ktr
        Setup > Connection: amqp://localhost/ptho
        Security: pentaho-client:pentaho123
      4. Check the RabbitMQ log and you should see something like this:
        2020-12-15 09:24:20.468 [info] <0.985.0> accepting AMQP connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672)
        2020-12-15 09:24:20.473 [info] <0.985.0> connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672): user 'pentaho-client' authenticated and granted access to vhost 'ptho'
        2020-12-15 09:24:20.533 [error] <0.993.0> Channel error on connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client'), channel 1:
        operation queue.declare caused a channel exception access_refused: access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client'
        2020-12-15 09:24:20.535 [info] <0.985.0> closing AMQP connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client')
        
      5. Check the Spoon log and you should see the following:
        2020/12/15 09:24:20 - AMQP Producer.0 - Starting to run...
        2020/12/15 09:24:20 - AMQP_Direct_Producer - Transformation has allocated 3 threads and 2 rowsets.
        2020/12/15 09:24:20 - AMQP Producer.0 - Attempting to connect to the amqp broker, please wait
        2020/12/15 09:24:20 - AMQP Producer.0 - Successfully connected to the amqp broker
        2020/12/15 09:24:20 - AMQP Producer.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Error declaring exchange or queue
        2020/12/15 09:24:20 - AMQP Producer.0 - java.io.IOException
        2020/12/15 09:24:20 - AMQP Producer.0 - com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client', class-id=50, method-id=10)
        2020/12/15 09:24:20 - AMQP Producer.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
        2020/12/15 09:24:22 - The transformation has finished!!
        

      Actual Results: When attempting to write to a queue we assume all permissions are granted to the user such as configure, read, write
      Expected Results: We should be able to write to a queue using the producer step without configure access right.

      Show
      Setup RabbitMQ Install Erlang (MS Windows: https://www.erlang.org/downloads ) (otp_win64_23.1.exe) Install RabbitMQ (MS Windows use this link: https://www.rabbitmq.com/install-windows.html#installer ) (rabbitmq-server-3.8.9.exe) After installation is complete; check the RabbitMQ log located here: C:\Users\carlopezx\AppData\Roaming\RabbitMQ\log\rabbit@PC0FTPBT.log Setup Spoon 9.1 Download attached transformations Run the attached AMQP_Direct_Producer.ktr Open the command prompt and browse to this location: cd C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.9\sbin Notice we connect with the following entries: 2020-12-14 16:34:31.045 [info] <0.1902.0> accepting AMQP connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672) 2020-12-14 16:34:31.061 [info] <0.1902.0> connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' Run the attached: AMQP_Direct_Consumer.ktr Notice we connect with the following entries: 2020-12-14 16:38:25.580 [info] <0.1963.0> accepting AMQP connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672) 2020-12-14 16:38:25.584 [info] <0.1963.0> connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' Stop both the producer and consumer transformations Setup user with full access: Configure, Write and Read permissions Create a virtual host for Pentaho; on the command prompt type the following: rabbitmqctl.bat add_vhost ptho Run the following command to add the user pentaho: rabbitmqctl.bat add_user "pentaho" "pentaho123" Run the following command to set permissions to pentaho: rabbitmqctl.bat set_permissions -p "ptho" "pentaho" ".*" ".*" ".*" Edit the AMQP_Direct_Producer.ktr Setup > Connection: amqp://localhost/ptho Security: pentaho:pentaho123 Check the logs you should see something like this: 2020-12-15 09:20:25.803 [info] <0.899.0> accepting AMQP connection <0.899.0> (127.0.0.1:51651 -> 127.0.0.1:5672) 2020-12-15 09:20:25.845 [info] <0.899.0> connection <0.899.0> (127.0.0.1:51651 -> 127.0.0.1:5672): user 'pentaho' authenticated and granted access to vhost 'ptho' Edit the AMQP_Direct_Consumer.ktr Setup > Connection: amqp://localhost/ptho Security: pentaho:pentaho123 Check the logs you should see something like this: 2020-12-15 09:22:16.130 [info] <0.946.0> accepting AMQP connection <0.946.0> ([::1]:51682 -> [::1]:5672) 2020-12-15 09:22:16.135 [info] <0.946.0> connection <0.946.0> ([::1]:51682 -> [::1]:5672): user 'pentaho' authenticated and granted access to vhost 'ptho' Stop both the producer and consumer transformations Setup user with limited access: Read and Write Run the following command to add the user pentaho-client: rabbitmqctl.bat add_user "pentaho-client" "pentaho123" Run the following command to set permissions to pentaho-client: rabbitmqctl.bat set_permissions -p "ptho" "pentaho-client" "^$" ".*" ".*" Edit the AMQP_Direct_Producer.ktr Setup > Connection: amqp://localhost/ptho Security: pentaho-client:pentaho123 Check the RabbitMQ log and you should see something like this: 2020-12-15 09:24:20.468 [info] <0.985.0> accepting AMQP connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672) 2020-12-15 09:24:20.473 [info] <0.985.0> connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672): user 'pentaho-client' authenticated and granted access to vhost 'ptho' 2020-12-15 09:24:20.533 [error] <0.993.0> Channel error on connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client'), channel 1: operation queue.declare caused a channel exception access_refused: access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client' 2020-12-15 09:24:20.535 [info] <0.985.0> closing AMQP connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client') Check the Spoon log and you should see the following: 2020/12/15 09:24:20 - AMQP Producer.0 - Starting to run... 2020/12/15 09:24:20 - AMQP_Direct_Producer - Transformation has allocated 3 threads and 2 rowsets. 2020/12/15 09:24:20 - AMQP Producer.0 - Attempting to connect to the amqp broker, please wait 2020/12/15 09:24:20 - AMQP Producer.0 - Successfully connected to the amqp broker 2020/12/15 09:24:20 - AMQP Producer.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Error declaring exchange or queue 2020/12/15 09:24:20 - AMQP Producer.0 - java.io.IOException 2020/12/15 09:24:20 - AMQP Producer.0 - com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client', class-id=50, method-id=10) 2020/12/15 09:24:20 - AMQP Producer.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0) 2020/12/15 09:24:22 - The transformation has finished!! Actual Results: When attempting to write to a queue we assume all permissions are granted to the user such as configure, read, write Expected Results: We should be able to write to a queue using the producer step without configure access right.

      Description

      Currently when we attempt to write to a RabittMQ we are expecting to have all the permissions on the queue: Configure, Read and Write in some cases the configure set of permissions is not granted to users; if this is the case the AMQP producer fails to write to the queue. Showing the following error on the rabbitmq.log

      2020-12-15 09:24:20.468 [info] <0.985.0> accepting AMQP connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672)
      2020-12-15 09:24:20.473 [info] <0.985.0> connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672): user 'pentaho-client' authenticated and granted access to vhost 'ptho'
      2020-12-15 09:24:20.533 [error] <0.993.0> Channel error on connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client'), channel 1:
      operation queue.declare caused a channel exception access_refused: access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client'
      2020-12-15 09:24:20.535 [info] <0.985.0> closing AMQP connection <0.985.0> (127.0.0.1:51705 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client')
      

      The Spoon log shows the following error as well:

      2020/12/15 09:24:20 - AMQP Producer.0 - Starting to run...
      2020/12/15 09:24:20 - AMQP_Direct_Producer - Transformation has allocated 3 threads and 2 rowsets.
      2020/12/15 09:24:20 - AMQP Producer.0 - Attempting to connect to the amqp broker, please wait
      2020/12/15 09:24:20 - AMQP Producer.0 - Successfully connected to the amqp broker
      2020/12/15 09:24:20 - AMQP Producer.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Error declaring exchange or queue
      2020/12/15 09:24:20 - AMQP Producer.0 - java.io.IOException
      2020/12/15 09:24:20 - AMQP Producer.0 - com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client', class-id=50, method-id=10)
      2020/12/15 09:24:20 - AMQP Producer.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
      2020/12/15 09:24:20 - Generate random value.0 - Signaling 'output done' to 1 output rowsets.
      2020/12/15 09:24:20 - Generate random value.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
      2020/12/15 09:24:22 - Generate Rows.0 - Signaling 'output done' to 1 output rowsets.
      2020/12/15 09:24:22 - Generate Rows.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
      2020/12/15 09:24:22 - AMQP_Direct_Producer - searching for annotations
      2020/12/15 09:24:22 - AMQP_Direct_Producer - no annotations found
      2020/12/15 09:24:22 - Size of ServiceMap:  2
      2020/12/15 09:24:22 - Size of ServiceMap:  3
      2020/12/15 09:24:22 - Size of ServiceMap:  3
      2020/12/15 09:24:22 - The transformation has finished!!
      

      This issue should be worked at the same time as PDI-19015

        Attachments

        1. AMQP_Direct_Consumer.ktr
          17 kB
        2. AMQP_Direct_Producer.ktr
          17 kB
        3. AMQPSub-Trans.ktr
          16 kB
        4. rabbit@PC0FTPBT.log
          17 kB
        5. spoon.log
          210 kB

          Issue Links

            Activity

              People

              Assignee:
              jberdecia Jose Berdecia
              Reporter:
              clopez Carlos Lopez
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: