Pentaho BI Server

Exceptions in Tomcat and error dialog interacting with Mantle after leaving client and server alone overnight

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 2.0.GA
  • Fix Version/s: 2.0.GA
  • Component/s: None
  • Customer Case:
  • QA Validation Status:
    Validated by QA
  • Description:
    Hide
    I was testing late one night, and left my server running with a client logged in. The next morning (~9 hours later) I tried interacting with the client (just toggled the toolbar buttons that show the file browser and My Workspace) and I got an error dialog showing HTML in the client and a boatload of exceptions in the Tomcat console.

    Clearly the client was in a sketchy state w.r.t. being logged in, but we should behave more gracefully. For instance if we detect that we're in a logged out state we should disable the UI or redirect to a "You've been logged out due to inactivity" page. And we shouldn't make the server go nuts just by interacting with a stale client.
    Show
    I was testing late one night, and left my server running with a client logged in. The next morning (~9 hours later) I tried interacting with the client (just toggled the toolbar buttons that show the file browser and My Workspace) and I got an error dialog showing HTML in the client and a boatload of exceptions in the Tomcat console. Clearly the client was in a sketchy state w.r.t. being logged in, but we should behave more gracefully. For instance if we detect that we're in a logged out state we should disable the UI or redirect to a "You've been logged out due to inactivity" page. And we shouldn't make the server go nuts just by interacting with a stale client.
  • Environment:
    WinXP, Firefox, MySQL

Attachments

  1. Text File
    backtrace.txt
    (31 kB)
    Jared Pshedesky
    06/Oct/08 8:47 AM
  2. Text File
    mantle-backtrace.txt
    (87 kB)
    Jared Pshedesky
    17/Sep/08 8:18 AM
  1. screenshot-1.jpg
    (186 kB)

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Jared Pshedesky added a comment - 17/Sep/08 8:18 AM
Attaching screen capture of the error dialog.
Show
Jared Pshedesky added a comment - 17/Sep/08 8:18 AM Attaching screen capture of the error dialog.
Jared Pshedesky made changes - 17/Sep/08 8:18 AM
Field Original Value New Value
Attachment screenshot-1.jpg [ 14353 ]
Hide
Permalink
Jared Pshedesky added a comment - 17/Sep/08 8:18 AM
Attaching console output containing the exception spewage.
Show
Jared Pshedesky added a comment - 17/Sep/08 8:18 AM Attaching console output containing the exception spewage.
Jared Pshedesky made changes - 17/Sep/08 8:18 AM
Attachment mantle-backtrace.txt [ 14354 ]
Hide
Permalink
Jared Pshedesky added a comment - 17/Sep/08 8:21 AM
Not sure if this is a separate issue from this, but with a stale client I was able to browse the Mantle UI and even successfully run xactions. The only things getting in my way were the frequent error dialogs like the one in the attached screenshot. We definitely need to handle automatic logout in a better way...
Show
Jared Pshedesky added a comment - 17/Sep/08 8:21 AM Not sure if this is a separate issue from this, but with a stale client I was able to browse the Mantle UI and even successfully run xactions. The only things getting in my way were the frequent error dialogs like the one in the attached screenshot. We definitely need to handle automatic logout in a better way...
Jake Cornelius made changes - 18/Sep/08 12:02 PM
Status Open [ 1 ] Open [ 1 ]
Fix Version/s 2.0.RC1 [ 10501 ]
Assignee Triage [ project admin ] Ravi Hasija [ rhasija ]
Priority Unknown [ 7 ] Critical [ 2 ]
Ravi Hasija made changes - 25/Sep/08 9:17 AM
Status Open [ 1 ] In Progress [ 3 ]
Hide
Permalink
Ravi Hasija added a comment - 25/Sep/08 9:23 AM
MySql closes the connection if there has been no activity for more than 8 hours. This is default behavior of MySQL which can be altered by setting an option to reconnect automatically after loosing connection due to inactivity or to set the timeout to a given time. So this is pertaining just with MySQL and there is a workaround on the database side.

However, we still need to manage loosing connection with database in a more refined manner.

Posting my findings and still working the issue.
Show
Ravi Hasija added a comment - 25/Sep/08 9:23 AM MySql closes the connection if there has been no activity for more than 8 hours. This is default behavior of MySQL which can be altered by setting an option to reconnect automatically after loosing connection due to inactivity or to set the timeout to a given time. So this is pertaining just with MySQL and there is a workaround on the database side. However, we still need to manage loosing connection with database in a more refined manner. Posting my findings and still working the issue.
Hide
Permalink
Ravi Hasija added a comment - 25/Sep/08 2:13 PM
Talked with Marc on this. This is a known issue and there is a fix for this. I am pasting the solution posted on the forum by Marc B sometime back.

Final Solution

    * Download C3P0 from SourceForge
    * Extract c3p0-0.9.1.2.jar from the .zip file, and place it in jboss/server/default/lib
    * Remove all hibernate.dbcp.* properties from your hibernate.cfg.xml - don't forget to remove the property connection.provider_class because we're replacing that line with a new connection provider
    * Add the following XML fragments to your hibernate.cfg.xml file

Code:

<property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
<property name="hibernate.c3p0.acquire_increment">3</property>
<property name="hibernate.c3p0.idle_test_period">14400</property>
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">75</property>
<property name="hibernate.c3p0.max_statements">0</property>
<property name="hibernate.c3p0.timeout">25200</property>
<property name="hibernate.c3p0.preferredTestQuery">select 1</property>
<property name="hibernate.c3p0.testConnectionOnCheckout">true</property>

OR

You can read the entire thread here for more information on the issue.

http://forums.pentaho.org/showthread.php?p=178067&highlight=C3P0
Show
Ravi Hasija added a comment - 25/Sep/08 2:13 PM Talked with Marc on this. This is a known issue and there is a fix for this. I am pasting the solution posted on the forum by Marc B sometime back. Final Solution     * Download C3P0 from SourceForge     * Extract c3p0-0.9.1.2.jar from the .zip file, and place it in jboss/server/default/lib     * Remove all hibernate.dbcp.* properties from your hibernate.cfg.xml - don't forget to remove the property connection.provider_class because we're replacing that line with a new connection provider     * Add the following XML fragments to your hibernate.cfg.xml file Code: <property name="connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property> <property name="hibernate.c3p0.acquire_increment">3</property> <property name="hibernate.c3p0.idle_test_period">14400</property> <property name="hibernate.c3p0.min_size">5</property> <property name="hibernate.c3p0.max_size">75</property> <property name="hibernate.c3p0.max_statements">0</property> <property name="hibernate.c3p0.timeout">25200</property> <property name="hibernate.c3p0.preferredTestQuery">select 1</property> <property name="hibernate.c3p0.testConnectionOnCheckout">true</property> OR You can read the entire thread here for more information on the issue. http://forums.pentaho.org/showthread.php?p=178067&highlight=C3P0
Ravi Hasija made changes - 25/Sep/08 2:33 PM
Status In Progress [ 3 ] Open [ 1 ]
Hide
Permalink
Ravi Hasija added a comment - 25/Sep/08 2:37 PM
Known issue. Needs to be documented. Created BISERVER-2032 documentation issue for Marina, provided information in the issue in the form of a word document. Over to Marina to include the info in the README doc.
Show
Ravi Hasija added a comment - 25/Sep/08 2:37 PM Known issue. Needs to be documented. Created BISERVER-2032 documentation issue for Marina, provided information in the issue in the form of a word document. Over to Marina to include the info in the README doc.
Ravi Hasija made changes - 25/Sep/08 2:37 PM
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Hide
Permalink
Jared Pshedesky added a comment - 26/Sep/08 3:48 PM
So Ravi... Has there been a code fix here, or is this just a workaround that you got from Mark? Isn't the larger issue here that the client should be gracefully logged out after a period of inactivity much less than 8 hours?
Show
Jared Pshedesky added a comment - 26/Sep/08 3:48 PM So Ravi... Has there been a code fix here, or is this just a workaround that you got from Mark? Isn't the larger issue here that the client should be gracefully logged out after a period of inactivity much less than 8 hours?
Hide
Permalink
Ravi Hasija added a comment - 26/Sep/08 5:12 PM
No, we do not have a code fix. This situation is specific to MySQL and the solution from Mark is our resolution. The line of thought seems to be that if we do not come across this scenario outside of this specific config, then we do not need to code for this. I am thinking we could probably request for graceful handling of logouts as a feature for a future release. Mark and John can be the best judge on this.
Show
Ravi Hasija added a comment - 26/Sep/08 5:12 PM No, we do not have a code fix. This situation is specific to MySQL and the solution from Mark is our resolution. The line of thought seems to be that if we do not come across this scenario outside of this specific config, then we do not need to code for this. I am thinking we could probably request for graceful handling of logouts as a feature for a future release. Mark and John can be the best judge on this.
Jared Pshedesky made changes - 06/Oct/08 8:32 AM
Resolution Fixed [ 1 ]
Status Resolved [ 5 ] Reopened [ 4 ]
Hide
Permalink
Jared Pshedesky added a comment - 06/Oct/08 8:33 AM
Reopened to mark this closed with a resolution of Won't Fix, since we did not fix anything, just supplied a workaround.
Show
Jared Pshedesky added a comment - 06/Oct/08 8:33 AM Reopened to mark this closed with a resolution of Won't Fix, since we did not fix anything, just supplied a workaround.
Jared Pshedesky made changes - 06/Oct/08 8:33 AM
Status Reopened [ 4 ] Closed [ 6 ]
Resolution Won't Fix [ 2 ]
Hide
Permalink
Jared Pshedesky added a comment - 06/Oct/08 8:46 AM - edited
I'm reopening this. It looks like the problem might be worse than just having to relogin. I just encountered this again and so I tried to logout and relogin, but the server seems hosed. So it looks like if anyone were to leave a client connected overnight, the entire server is hosed. Given that MySQL is the "official" DB on which we're releasing Enterprise Edition, I think this should be reconsidered as a bug, and a showstopper for GA. We shouldn't put out an EE release on which a user would need to implement a workaround right away!

I also see a different error dialog and different backtrace now. Including those as attachments shortly...
Show
Jared Pshedesky added a comment - 06/Oct/08 8:46 AM - edited I'm reopening this. It looks like the problem might be worse than just having to relogin. I just encountered this again and so I tried to logout and relogin, but the server seems hosed. So it looks like if anyone were to leave a client connected overnight, the entire server is hosed. Given that MySQL is the "official" DB on which we're releasing Enterprise Edition, I think this should be reconsidered as a bug, and a showstopper for GA. We shouldn't put out an EE release on which a user would need to implement a workaround right away! I also see a different error dialog and different backtrace now. Including those as attachments shortly...
Jared Pshedesky made changes - 06/Oct/08 8:46 AM
Resolution Won't Fix [ 2 ]
Assignee Ravi Hasija [ rhasija ] Triage [ project admin ]
Status Closed [ 6 ] Reopened [ 4 ]
Hide
Permalink
Jared Pshedesky added a comment - 06/Oct/08 8:47 AM
Attaching new backtrace.
Show
Jared Pshedesky added a comment - 06/Oct/08 8:47 AM Attaching new backtrace.
Jared Pshedesky made changes - 06/Oct/08 8:47 AM
Attachment backtrace.txt [ 14523 ]
Hide
Permalink
Jared Pshedesky added a comment - 06/Oct/08 8:48 AM
New backtrace is attached as backtrace.txt. Unfortunately, I restarted the server, and will need to leave it for at least 8 hours before I can repro the new message on the error dialog.
Show
Jared Pshedesky added a comment - 06/Oct/08 8:48 AM New backtrace is attached as backtrace.txt. Unfortunately, I restarted the server, and will need to leave it for at least 8 hours before I can repro the new message on the error dialog.
Hide
Permalink
Kurtis Cruzada added a comment - 06/Oct/08 10:31 PM
Ravi... this is fixed right... with 3CP0
Show
Kurtis Cruzada added a comment - 06/Oct/08 10:31 PM Ravi... this is fixed right... with 3CP0
Kurtis Cruzada made changes - 06/Oct/08 10:31 PM
Fix Version/s 2.0.GA [ 10500 ]
Status Reopened [ 4 ] Open [ 1 ]
Fix Version/s 2.0.RC1 [ 10501 ]
Assignee Triage [ project admin ] Ravi Hasija [ rhasija ]
Hide
Permalink
Ravi Hasija added a comment - 07/Oct/08 9:10 AM
Yes C3P0 fixes it. I am not sure what the decision is regarding this. But, if we can build in the solution mentioned above while building for mysql, then the user does not need to make any changes on his side for this particular problem.
Show
Ravi Hasija added a comment - 07/Oct/08 9:10 AM Yes C3P0 fixes it. I am not sure what the decision is regarding this. But, if we can build in the solution mentioned above while building for mysql, then the user does not need to make any changes on his side for this particular problem.
Ravi Hasija made changes - 07/Oct/08 9:32 AM
Status Open [ 1 ] In Progress [ 3 ]
Hide
Permalink
Ravi Hasija added a comment - 07/Oct/08 3:55 PM
Tested with "jdbc:mysql://localhost:3306/hibernate?autoReconnect=true" but the problem still exists. Researched and found another solution but it is JNDI based, and we are moving away from that.

So we are back to C3P0. Need to build that into our build process now. Working on this now.
Show
Ravi Hasija added a comment - 07/Oct/08 3:55 PM Tested with "jdbc:mysql://localhost:3306/hibernate?autoReconnect=true" but the problem still exists. Researched and found another solution but it is JNDI based, and we are moving away from that. So we are back to C3P0. Need to build that into our build process now. Working on this now.
Ravi Hasija made changes - 09/Oct/08 9:26 AM
Status In Progress [ 3 ] Open [ 1 ]
Hide
Permalink
Ravi Hasija added a comment - 09/Oct/08 9:27 AM
c3p0 is ivy'zed and also appropriate changes in mysql.hibernate.cfg.xml have been introduced.
Show
Ravi Hasija added a comment - 09/Oct/08 9:27 AM c3p0 is ivy'zed and also appropriate changes in mysql.hibernate.cfg.xml have been introduced.
Ravi Hasija made changes - 09/Oct/08 9:27 AM
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Gangadhara Boranna made changes - 22/Oct/08 6:24 AM
Customer Case 4285
Hide
Permalink
Jens Bleuel added a comment - 22/Oct/08 8:31 AM
For MySQL there is a chapter with the nice title "MySQL server has gone away" over here:
http://dev.mysql.com/doc/refman/5.1/en/gone-away.html
Show
Jens Bleuel added a comment - 22/Oct/08 8:31 AM For MySQL there is a chapter with the nice title "MySQL server has gone away" over here: http://dev.mysql.com/doc/refman/5.1/en/gone-away.html
Hide
Permalink
Jared Pshedesky added a comment - 12/Nov/08 2:35 PM
Validated on 2.0.0.GA.
Show
Jared Pshedesky added a comment - 12/Nov/08 2:35 PM Validated on 2.0.0.GA.
Jared Pshedesky made changes - 12/Nov/08 2:35 PM
Status Resolved [ 5 ] Closed [ 6 ]
QA Validation Status Validated by QA
Assignee Ravi Hasija [ rhasija ] Triage [ project admin ]
Gretchen Moran made changes - 04/Aug/09 6:44 PM
Workflow Pentaho Engineering 3.0 Workflow [ 214076 ] Pentaho Engineering 4.0 Workflow [ 378046 ]
Marc Batchelor made changes - 04/Aug/09 8:38 PM
Workflow Pentaho Engineering 4.0 Workflow [ 378046 ] Pentaho Engineering 5.0 Workflow [ 410894 ]
Marc Batchelor made changes - 17/Aug/09 10:28 AM
Workflow Pentaho Engineering 5.0 Workflow [ 410894 ] Pentaho Engineering 6.0 Workflow [ 440160 ]
Marc Batchelor made changes - 06/May/10 7:11 PM
Workflow Pentaho Engineering 6.0 Workflow [ 440160 ] Pentaho Engineering 7.0 Workflow [ 510930 ]
Marc Batchelor made changes - 04/Aug/10 10:32 PM
Workflow Pentaho Engineering 7.0 Workflow [ 510930 ] Pentaho Engineering 8.0 Workflow [ 533559 ]
Transition Status Change Time Execution Times Last Executer Last Execution Date
Open Open Open Open
1d 3h 44m
1
Jake Cornelius
18/Sep/08 12:02 PM
Open Open In Progress In Progress
6d 21h 14m
1
Ravi Hasija
25/Sep/08 9:17 AM
In Progress In Progress Open Open
5h 16m
1
Ravi Hasija
25/Sep/08 2:33 PM
Open Open Resolved Resolved
3m 38s
1
Ravi Hasija
25/Sep/08 2:37 PM
Resolved Resolved Reopened Reopened
10d 17h 55m
1
Jared Pshedesky
06/Oct/08 8:32 AM
Reopened Reopened Closed Closed
39s
1
Jared Pshedesky
06/Oct/08 8:33 AM
Closed Closed Reopened Reopened
12m 41s
1
Jared Pshedesky
06/Oct/08 8:46 AM
Reopened Reopened Open Open
13h 45m
1
Kurtis Cruzada
06/Oct/08 10:31 PM
Open Open In Progress In Progress
11h
1
Ravi Hasija
07/Oct/08 9:32 AM
In Progress In Progress Open Open
1d 23h 54m
1
Ravi Hasija
09/Oct/08 9:26 AM
Open Open Resolved Resolved
1m 4s
1
Ravi Hasija
09/Oct/08 9:27 AM
Resolved Resolved Closed Closed
34d 6h 8m
1
Jared Pshedesky
12/Nov/08 2:35 PM

People

Dates

  • Created:
    17/Sep/08 8:17 AM
    Updated:
    12/Nov/08 2:35 PM
    Resolved:
    09/Oct/08 9:27 AM
Atlassian JIRA (v4.1.2#531) | Report a problem | Request a feature | Contact administrators
Powered by a free Atlassian JIRA open source license for Pentaho.org. Try JIRA - bug tracking software for your team.