Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-19015

AMQP Consumer fails to read from queue due to missing configure permissions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: Urgent
    • Resolution: Fixed
    • Affects Version/s: 9.1.0 GA
    • Fix Version/s: Backlog
    • Component/s: Step
    • Labels:
      None
    • Story Points:
      8
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Sprint Team:
      Ackbar
    • Steps to Reproduce:
      Hide

      Setup RabbitMQ

      1. Install Erlang (MS Windows: https://www.erlang.org/downloads) (otp_win64_23.1.exe)
      2. Install RabbitMQ (MS Windows use this link: https://www.rabbitmq.com/install-windows.html#installer) (rabbitmq-server-3.8.9.exe)
      3. After installation is complete; check the RabbitMQ log located here:
        C:\Users\carlopezx\AppData\Roaming\RabbitMQ\log\rabbit@PC0FTPBT.log

      Setup Spoon 9.1

      1. Download attached transformations
      2. Run the attached AMQP_Direct_Producer.ktr
      3. Open the command prompt and browse to this location:
        cd C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.9\sbin
      4. Notice we connect with the following entries:
        2020-12-14 16:34:31.045 [info] <0.1902.0> accepting AMQP connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672)
        2020-12-14 16:34:31.061 [info] <0.1902.0> connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
        
      5. Run the attached: AMQP_Direct_Consumer.ktr
      6. Notice we connect with the following entries:
        2020-12-14 16:38:25.580 [info] <0.1963.0> accepting AMQP connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672)
        2020-12-14 16:38:25.584 [info] <0.1963.0> connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
        
      7. Stop both the producer and consumer transformations

      Setup user with full access: Configure, Write and Read permissions)

      1. Create a virtual host for Pentaho; on the command prompt type the following:
        rabbitmqctl.bat add_vhost ptho
      2. Run the following command to add the user pentaho:
        rabbitmqctl.bat add_user "pentaho" "pentaho123"
      3. Run the following command to set permissions to pentaho:
        rabbitmqctl.bat set_permissions -p "ptho" "pentaho" ".*" ".*" ".*"
      4. Edit the AMQP_Direct_Producer:
        Setup > Connection: amqp://localhost/ptho
        Security: pentaho:pentaho123
      5. Check the logs you should see something like this:
        2020-12-14 17:09:28.465 [info] <0.2532.0> accepting AMQP connection <0.2532.0> (127.0.0.1:55223 -> 127.0.0.1:5672)
        2020-12-14 17:09:28.478 [info] <0.2532.0> connection <0.2532.0> (127.0.0.1:55223 -> 127.0.0.1:5672): user 'pentaho' authenticated and granted access to vhost 'ptho'
        
      6. Edit the AMQP_Direct_Consumer:
        Setup > Connection: amqp://localhost/ptho
        Security: pentaho:pentaho123
      7. Check the logs you should see something like this:
        2020-12-14 17:11:05.009 [info] <0.2576.0> accepting AMQP connection <0.2576.0> (127.0.0.1:55241 -> 127.0.0.1:5672)
        2020-12-14 17:11:05.013 [info] <0.2576.0> connection <0.2576.0> (127.0.0.1:55241 -> 127.0.0.1:5672): user 'pentaho' authenticated and granted access to vhost 'ptho'
        

      Setup user with limited access: Write and Read permissions

      1. Run the following command to add the user pentaho-client:
        rabbitmqctl.bat add_user "pentaho-client" "pentaho123"
      2. Run the following command to set permissions to pentaho-client:
        rabbitmqctl.bat set_permissions -p "ptho" "pentaho-client" "^$" ".*" ".*"
      3. Edit the AMQP_Direct_Consumer:
        Setup > Connection: amqp://localhost/ptho
        Security: pentaho-client:pentaho123
      4. Check the RabbitMQ log and you should see something like this:
        2020-12-14 18:07:06.826 [error] <0.4132.0> Channel error on connection <0.4124.0> (127.0.0.1:54695 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client'), channel 1:
        operation queue.declare caused a channel exception access_refused: access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client'
        2020-12-14 18:07:06.829 [info] <0.4124.0> closing AMQP connection <0.4124.0> (127.0.0.1:54695 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client')
        
      5. Check the Spoon log and you should see the following:
        2020/12/14 18:14:22 - AMQP_Direct_Consumer - Dispatching started for transformation [AMQP_Direct_Consumer]
        2020/12/14 18:14:23 - AMQP Consumer.0 - Attempting to connect to the amqp broker, please wait
        2020/12/14 18:14:23 - AMQP Consumer.0 - Successfully connected to the amqp broker
        2020/12/14 18:14:23 - AMQP Consumer.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Error declaring exchange or queue
        2020/12/14 18:14:23 - AMQP Consumer.0 - java.io.IOException
        2020/12/14 18:14:23 - AMQP Consumer.0 - com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client', class-id=50, method-id=10)
        2020/12/14 18:14:23 - The transformation has finished!!
        

      Actual Results: When accessing a queue we assume all permissions are granted to the user such as configure, read, write
      Expected Results: We should be able to connect to an existing queue using the consumer step without configure access right.

      Show
      Setup RabbitMQ Install Erlang (MS Windows: https://www.erlang.org/downloads ) (otp_win64_23.1.exe) Install RabbitMQ (MS Windows use this link: https://www.rabbitmq.com/install-windows.html#installer ) (rabbitmq-server-3.8.9.exe) After installation is complete; check the RabbitMQ log located here: C:\Users\carlopezx\AppData\Roaming\RabbitMQ\log\rabbit@PC0FTPBT.log Setup Spoon 9.1 Download attached transformations Run the attached AMQP_Direct_Producer.ktr Open the command prompt and browse to this location: cd C:\Program Files\RabbitMQ Server\rabbitmq_server-3.8.9\sbin Notice we connect with the following entries: 2020-12-14 16:34:31.045 [info] <0.1902.0> accepting AMQP connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672) 2020-12-14 16:34:31.061 [info] <0.1902.0> connection <0.1902.0> (127.0.0.1:54640 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' Run the attached: AMQP_Direct_Consumer.ktr Notice we connect with the following entries: 2020-12-14 16:38:25.580 [info] <0.1963.0> accepting AMQP connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672) 2020-12-14 16:38:25.584 [info] <0.1963.0> connection <0.1963.0> (127.0.0.1:54705 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/' Stop both the producer and consumer transformations Setup user with full access: Configure, Write and Read permissions) Create a virtual host for Pentaho; on the command prompt type the following: rabbitmqctl.bat add_vhost ptho Run the following command to add the user pentaho: rabbitmqctl.bat add_user "pentaho" "pentaho123" Run the following command to set permissions to pentaho: rabbitmqctl.bat set_permissions -p "ptho" "pentaho" ".*" ".*" ".*" Edit the AMQP_Direct_Producer: Setup > Connection: amqp://localhost/ptho Security: pentaho:pentaho123 Check the logs you should see something like this: 2020-12-14 17:09:28.465 [info] <0.2532.0> accepting AMQP connection <0.2532.0> (127.0.0.1:55223 -> 127.0.0.1:5672) 2020-12-14 17:09:28.478 [info] <0.2532.0> connection <0.2532.0> (127.0.0.1:55223 -> 127.0.0.1:5672): user 'pentaho' authenticated and granted access to vhost 'ptho' Edit the AMQP_Direct_Consumer: Setup > Connection: amqp://localhost/ptho Security: pentaho:pentaho123 Check the logs you should see something like this: 2020-12-14 17:11:05.009 [info] <0.2576.0> accepting AMQP connection <0.2576.0> (127.0.0.1:55241 -> 127.0.0.1:5672) 2020-12-14 17:11:05.013 [info] <0.2576.0> connection <0.2576.0> (127.0.0.1:55241 -> 127.0.0.1:5672): user 'pentaho' authenticated and granted access to vhost 'ptho' Setup user with limited access: Write and Read permissions Run the following command to add the user pentaho-client: rabbitmqctl.bat add_user "pentaho-client" "pentaho123" Run the following command to set permissions to pentaho-client: rabbitmqctl.bat set_permissions -p "ptho" "pentaho-client" "^$" ".*" ".*" Edit the AMQP_Direct_Consumer: Setup > Connection: amqp://localhost/ptho Security: pentaho-client:pentaho123 Check the RabbitMQ log and you should see something like this: 2020-12-14 18:07:06.826 [error] <0.4132.0> Channel error on connection <0.4124.0> (127.0.0.1:54695 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client'), channel 1: operation queue.declare caused a channel exception access_refused: access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client' 2020-12-14 18:07:06.829 [info] <0.4124.0> closing AMQP connection <0.4124.0> (127.0.0.1:54695 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client') Check the Spoon log and you should see the following: 2020/12/14 18:14:22 - AMQP_Direct_Consumer - Dispatching started for transformation [AMQP_Direct_Consumer] 2020/12/14 18:14:23 - AMQP Consumer.0 - Attempting to connect to the amqp broker, please wait 2020/12/14 18:14:23 - AMQP Consumer.0 - Successfully connected to the amqp broker 2020/12/14 18:14:23 - AMQP Consumer.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Error declaring exchange or queue 2020/12/14 18:14:23 - AMQP Consumer.0 - java.io.IOException 2020/12/14 18:14:23 - AMQP Consumer.0 - com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client', class-id=50, method-id=10) 2020/12/14 18:14:23 - The transformation has finished!! Actual Results: When accessing a queue we assume all permissions are granted to the user such as configure, read, write Expected Results: We should be able to connect to an existing queue using the consumer step without configure access right.

      Description

      We are using AMQP consumer over TSL and getting the following error in the RabbitMQ Server:

      2020-12-14 18:07:06.826 [error] <0.4132.0> Channel error on connection <0.4124.0> (127.0.0.1:54695 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client'), channel 1:
      operation queue.declare caused a channel exception access_refused: access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client'
      2020-12-14 18:07:06.829 [info] <0.4124.0> closing AMQP connection <0.4124.0> (127.0.0.1:54695 -> 127.0.0.1:5672, vhost: 'ptho', user: 'pentaho-client')
      

      and the following error when running the consumer step in PDI

      2020/12/14 18:14:22 - AMQP_Direct_Consumer - Dispatching started for transformation [AMQP_Direct_Consumer]
      2020/12/14 18:14:23 - AMQP Consumer.0 - Attempting to connect to the amqp broker, please wait
      2020/12/14 18:14:23 - AMQP Consumer.0 - Successfully connected to the amqp broker
      2020/12/14 18:14:23 - AMQP Consumer.0 - ERROR (version 9.1.0.0-324, build 9.1.0.0-324 from 2020-09-07 05.09.05 by buildguy) : Error declaring exchange or queue
      2020/12/14 18:14:23 - AMQP Consumer.0 - java.io.IOException
      2020/12/14 18:14:23 - AMQP Consumer.0 - com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - access to queue 'test.routing.key' in vhost 'ptho' refused for user 'pentaho-client', class-id=50, method-id=10)
      2020/12/14 18:14:23 - The transformation has finished!!
      

       

      The reply from the customer was that: The AMQPS client should avoid doing a "queue.declare" 
      This action requires "configure" access rights (which have not been granted) and will lead to an ACCESS_REFUSED error.  

      Does the AMQP Consumer always do a queue.declare even accessing to an existing message queue?

      According to https://www.rabbitmq.com/amqp-0-9-1-reference.html#queue.declare.passive is it possible to have an option in the step to set queue.declare.passive to true?

        Attachments

        1. AMQP_Direct_Consumer.ktr
          17 kB
        2. AMQP_Direct_Producer.ktr
          17 kB
        3. AMQPSub-Trans.ktr
          16 kB
        4. rabbit@PC0FTPBT.log
          38 kB
        5. spoon.log
          36 kB

          Issue Links

            Activity

              People

              Assignee:
              jberdecia Jose Berdecia
              Reporter:
              ppires@pentaho.com Paulo Pires
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: