Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-17190

Filling in the HTTP Post Step's HTTP Login and HTTP Password fields cause an error to be thrown

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Severity: High
    • Resolution: Fixed
    • Affects Version/s: 8.0.0 GA
    • Fix Version/s: 8.2.0 GA
    • Component/s: Step
    • Labels:
    • Story Points:
      0
    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Steps to Reproduce:
      Hide

      Test Setup:

      Test for 7.1

      1. Start a working copy of 7.1 server.
      2. In the browser, get the jobID for the UpdateAuditData http://localhost:8080/pentaho/api/scheduler/getJobs.
      3. Open the ktr.
      4. In the Generate Rows step, modify the job request in the body so that the number reflects the number that you got from the getJobs call. (Just update the number; you'll want to make sure to preserve the tabs that are in the jobRequest so that you don't get a 400 error when you run the REST call.)
      5. If needed, update the url string as well.
      6. For now, leave the Authorization string as is.
      7. In the HTTP Login and HTTP Password field's in the HTTP Post step, add a valid login and password.
      8. Run the job, making sure that the log level is set to Row Level. You should see a return code of 200 indicated that job has been triggered.
      9. Log into the server and note that the job has run. You can also check the catalina log to verify the run.
      10. Stop the server.

      Test for 8.0

      1. Start a working copy of 8.0 server.
      2. In the browser, get the jobID for the UpdateAuditData http://localhost:8080/pentaho/api/scheduler/getJobs.
      3. Open the ktr.
      4. In the Generate Rows step, modify the job request in the body so that the number reflects the number that you got from the getJobs call. (Just update the number; you'll want to make sure to preserve the tabs that are in the jobRequest so that you don't get a 400 error when you run the REST call.)
      5. If needed, update the url string as well.
      6. Change the Authorization to a Base 64 Authorization header that matches your username and password. You can use a tool like this to accomplish that: https://www.blitter.se/utils/basic-authentication-header-generator/.
      7. In the HTTP Login and HTTP Password field's in the HTTP Post step, add a valid login and password.
      8. Run the job, making sure that the log level is set to Row Level. You should see an error.
      9. Remove login and username.
      10. In the Fields tab, add the Authorization, name the parameter "Authorization" and select Y in the Put in the Header column.
      11. Run again, it should complete successfully.

      Please note that there are other test cases that can be used to further explore the problem. They are these.
      Test: Passing admin/password using HTTP Login and HTTP Password fields but inserting Fiddler as a Socks Proxy (Passed 8.0)
      Test: Passing Admin/password using HTTP Login and HTTP Password fields but inserting Fiddler as a Socks Proxy (Passed 8.0)
      Test: Passing Admin/password using HTTP Login and HTTP Password fields but inserting nginx as a proxy (Failed 8.0)

      Show
      Test Setup: Test for 7.1 1. Start a working copy of 7.1 server. 2. In the browser, get the jobID for the UpdateAuditData http://localhost:8080/pentaho/api/scheduler/getJobs . 3. Open the ktr. 4. In the Generate Rows step, modify the job request in the body so that the number reflects the number that you got from the getJobs call. (Just update the number; you'll want to make sure to preserve the tabs that are in the jobRequest so that you don't get a 400 error when you run the REST call.) 5. If needed, update the url string as well. 6. For now, leave the Authorization string as is. 7. In the HTTP Login and HTTP Password field's in the HTTP Post step, add a valid login and password. 8. Run the job, making sure that the log level is set to Row Level. You should see a return code of 200 indicated that job has been triggered. 9. Log into the server and note that the job has run. You can also check the catalina log to verify the run. 10. Stop the server. Test for 8.0 1. Start a working copy of 8.0 server. 2. In the browser, get the jobID for the UpdateAuditData http://localhost:8080/pentaho/api/scheduler/getJobs . 3. Open the ktr. 4. In the Generate Rows step, modify the job request in the body so that the number reflects the number that you got from the getJobs call. (Just update the number; you'll want to make sure to preserve the tabs that are in the jobRequest so that you don't get a 400 error when you run the REST call.) 5. If needed, update the url string as well. 6. Change the Authorization to a Base 64 Authorization header that matches your username and password. You can use a tool like this to accomplish that: https://www.blitter.se/utils/basic-authentication-header-generator/ . 7. In the HTTP Login and HTTP Password field's in the HTTP Post step, add a valid login and password. 8. Run the job, making sure that the log level is set to Row Level. You should see an error. 9. Remove login and username. 10. In the Fields tab, add the Authorization, name the parameter "Authorization" and select Y in the Put in the Header column. 11. Run again, it should complete successfully. Please note that there are other test cases that can be used to further explore the problem. They are these. Test: Passing admin/password using HTTP Login and HTTP Password fields but inserting Fiddler as a Socks Proxy (Passed 8.0) Test: Passing Admin/password using HTTP Login and HTTP Password fields but inserting Fiddler as a Socks Proxy (Passed 8.0) Test: Passing Admin/password using HTTP Login and HTTP Password fields but inserting nginx as a proxy (Failed 8.0)

      Description

      Pentaho Product: Spoon
      Version: 8.0
      Operating System: Windows
      Memory Available/Used: 32/2
      Hardware: Lenovo Thinkpad P50
      Clustered/No: No
      Virtualized?: No
      Database for repository: N/A
      Database for data: N/A
      Browser type and version (If Applicable): N/A
      Did you attach logs? Yes

      When I use the HTTP Post step and fill in a valid username and password in the HTTP Login and HTTP Password fields, an error is thrown:

      2018/04/16 16:43:43 - HTTP Post.0 - ERROR (version 8.0.0.0-28, build 8.0.0.0-28 from 2017-11-05 07.27.50 by buildguy) : Because of an error, this step can't continue: 
2018/04/16 16:43:43 - HTTP Post.0 - Can not result from [http://localhost:8080/pentaho/api/scheduler/triggerNow]
2018/04/16 16:43:43 - HTTP Post.0 -  at java.lang.Thread.run (null:-1)

      When we observe the call in the catalina access file or look in the chrome web tools, we see a 401 Authorization Error. Yet, when we pass the credentials using the Base 64 authorization header that is generated by a third party tool, this works perfectly. Strangely, if we insert a socks proxy server, like Fiddler, the transformation works.
      This seems to be a regression because performing the exact same test in 7.1 works perfectly.

      Closer inspection of the code seems to suggest that there might have been some underlying HTTP Client Library changes that might account for this issue, rather than pentaho code changes.

      Stack Trace:

      2018/04/16 16:43:43 - HTTP Post.0 - ERROR (version 8.0.0.0-28, build 8.0.0.0-28 from 2017-11-05 07.27.50 by buildguy) : Because of an error, this step can't continue: 
2018/04/16 16:43:43 - HTTP Post.0 - Can not result from [http://localhost:8080/pentaho/api/scheduler/triggerNow]
2018/04/16 16:43:43 - HTTP Post.0 -  at java.lang.Thread.run (null:-1)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.pentaho.di.trans.step.RunThread.run (RunThread.java:62)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.pentaho.di.trans.steps.httppost.HTTPPOST.processRow (HTTPPOST.java:468)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.pentaho.di.trans.steps.httppost.HTTPPOST.callHTTPPOST (HTTPPOST.java:237)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:108)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:83)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.apache.http.impl.client.InternalHttpClient.doExecute (InternalHttpClient.java:187)
2018/04/16 16:43:43 - HTTP Post.0 - ERROR (version 8.0.0.0-28, build 8.0.0.0-28 from 2017-11-05 07.27.50 by buildguy) : org.pentaho.di.core.exception.KettleException: 
2018/04/16 16:43:43 - HTTP Post.0 - Can not result from [http://localhost:8080/pentaho/api/scheduler/triggerNow]
2018/04/16 16:43:43 - HTTP Post.0 -  at java.lang.Thread.run (null:-1)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.pentaho.di.trans.step.RunThread.run (RunThread.java:62)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.pentaho.di.trans.steps.httppost.HTTPPOST.processRow (HTTPPOST.java:468)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.pentaho.di.trans.steps.httppost.HTTPPOST.callHTTPPOST (HTTPPOST.java:237)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:108)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.apache.http.impl.client.CloseableHttpClient.execute (CloseableHttpClient.java:83)
2018/04/16 16:43:43 - HTTP Post.0 -  at org.apache.http.impl.client.InternalHttpClient.doExecute (InternalHttpClient.java:187)
2018/04/16 16:43:43 - HTTP Post.0 - 
2018/04/16 16:43:43 - HTTP Post.0 -     at org.pentaho.di.trans.steps.httppost.HTTPPOST.callHTTPPOST(HTTPPOST.java:327)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.pentaho.di.trans.steps.httppost.HTTPPOST.processRow(HTTPPOST.java:468)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/04/16 16:43:43 - HTTP Post.0 -     at java.lang.Thread.run(Unknown Source)
2018/04/16 16:43:43 - HTTP Post.0 - Caused by: org.apache.http.client.ClientProtocolException
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.pentaho.di.trans.steps.httppost.HTTPPOST.callHTTPPOST(HTTPPOST.java:237)
2018/04/16 16:43:43 - HTTP Post.0 -     ... 3 more
2018/04/16 16:43:43 - HTTP Post.0 - Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity.
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:226)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
2018/04/16 16:43:43 - HTTP Post.0 -     at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
2018/04/16 16:43:43 - HTTP Post.0 -     ... 6 more

      Here is the list of tests we performed on both 7.1 and 8.0. Test details appear in the steps to reproduce section.

      Test 1: Passing admin/password using HTTP Login and HTTP Password fields. (Passed 7.1, Failed 8.0)
      Test 2: Passing Admin/password using HTTP Login and HTTP Password fields. (Passed 7.1, Failed 8.0)
      Test 3: Omitting HTTP Login and HTTP Password fields and passing Authorization only (Basic QWRtaW46cGFzc3dvcmQ= or Basic YWRtaW46cGFzc3dvcmQ=). (Passed 8.0).
      Test 4: Passing admin/password using HTTP Login and HTTP Password fields but inserting Fiddler as a Socks Proxy (Passed 8.0)
      Test 5: Passing Admin/password using HTTP Login and HTTP Password fields but inserting Fiddler as a Socks Proxy (Passed 8.0)
      Test 6: Passing Admin/password using HTTP Login and HTTP Password fields but inserting nginx as a proxy (Failed 8.0)

        PractiTest Integration




          Attachments

            Issue Links

            Cloned from

            Bug - A problem which impairs or prevents the functions of the product. PDI-17911 Filling in the HTTP Client Step's HTTP Login and HTTP Password fields cause an error to be thrown

            • High - An issue that impacts and reduces ability to use Pentaho in a production environment. Business operations are able to continue, however they may be highly impacted and a viable documented workaround is not available.
            • Closed
            is related to

            Bug - A problem which impairs or prevents the functions of the product. PDI-17108 HTTP Post step fails Digest Based Authentication

            • High - An issue that impacts and reduces ability to use Pentaho in a production environment. Business operations are able to continue, however they may be highly impacted and a viable documented workaround is not available.
            • Closed

              Activity

                People

                Assignee:
                mleite Margarida Leite
                Reporter:
                cbrathwaite Chantel Brathwaite (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved: