Uploaded image for project: 'Pentaho Data Integration - Kettle'
  1. Pentaho Data Integration - Kettle
  2. PDI-8202

Allow MongoDB Input to read from secondaries/slaves

    Details

    • PDI Sub-component:
    • Notice:
      When an issue is open, the "Fix Version/s" field conveys a target, not necessarily a commitment. When an issue is closed, the "Fix Version/s" field conveys the version that the issue was fixed in.
    • Operating System/s:
      RedHat Enterprise Linux 5

      Description

      If I attempt to connect to a secondary (slave) Mongo instance and issue a query using the MongoDB Input step, I get the following error:

      INFO 17-07 13:17:28,459 - etl_test - Dispatching started for transformation [etl_test]
      ERROR 17-07 13:17:29,341 - MongoDb Input - Unexpected error
      ERROR 17-07 13:17:29,341 - MongoDb Input - com.mongodb.MongoException: not talking to master and retries used up
      at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:246)
      at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:248)
      at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:248)
      at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:305)
      at com.mongodb.DBCursor._check(DBCursor.java:369)
      at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
      at com.mongodb.DBCursor.hasNext(DBCursor.java:523)
      at org.pentaho.di.trans.steps.mongodbinput.MongoDbInput.processRow(MongoDbInput.java:72)
      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:50)
      at java.lang.Thread.run(Thread.java:662)

      I get similar errors if I use the mongo client to connect to the secondary and issue queries. The problem goes away in the client if I issue the command:

      rs.slaveOk()

      and then run my queries. Unfortunately, I see no way to execute this command when using the MongoDB Input step, so I am unable to query any data from the secondary MongoDB instance. In production environments, I would expect most queries to be run against secondaries rather than the master, especially for ETL, so I imagine this would be a blocker for many people. Hopefully it can be addressed soon.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ksheth Kaushal Sheth
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: