6/07/2013

CentralCache Management(Memcache) Issue 1 -- net.spy.memcached.internal.CheckedOperationTimeoutException


One main aspect to evaluate one web app is its scalability, whether it can sustain heavy load.
So when we decide to build one web app, each technology that we plan to use must go through enough tests, not only functional test, but also load test. Otherwise, we may lose at the start point.

When I test the performance of tomcat-memcache central session management, I save&get small session data and large session data with both light load and heavy load.

When load is light, everything goes fine. But when load increase, and session data is around 10K, exception will be thrown:

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: localhost/127.0.0.1:11212
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160)
        at de.javakaffee.web.msm.LockingStrategy.onAfterBackupSession(LockingStrategy.java:294)
        at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:1062)
        at de.javakaffee.web.msm.RequestTrackingHostValve.backupSession(RequestTrackingHostValve.java:243)
        at de.javakaffee.web.msm.RequestTrackingHostValve.invoke(RequestTrackingHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1770)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Regarding this issue, I found one discussion page:
https://code.google.com/p/spymemcached/issues/detail?id=136
But I did not see effect solution. Maybe switch to other memcache client lib will be a better solution.
Somebody says that increase the spymemcache's timeout may help, but I haven't got time to test it.
Compared to memcache, now I prefer mongodb as the backup of session. At least it's client lib is more stable and under enough test.

So If we need to setup the central cache for web app, we need to customize and test the lib well based on our requirement. I think no one wants to face this issue in production.

No comments:

Post a Comment