Central Session Management (Tomcat, Memcache/MongoDB, Non-Sticky Session)

For all application server clusters, there are two elementary issues that need to be resolved:

1. How to dispatch requests to the cluster nodes.
2. Use one session management strategy, to ensure that if one node fails, other nodes can still obtained the session data, to achieve fault-tolerant cluster (failover)

For the first issue, the easiest way is to use uniform hash function to dispatch the request to each node. But for application server, session is used. To make sure one session’s request is properly handled, normally we dispatch the requests related to one session to same node to process, this is “Sticky Session” load balance. And the way randomly dispatch request is called “non-sticky session” load balance.

The “sticky session” sometimes restricts the load balance and even the web app’s design and development. To avoid “sticky session”, we need to find appropriate session management strategy.

There are several options for session share.

Session Share Options

Tomcat Clustering

It simply copied session to all the tomcat instances. If there are many instances, this may cause a lot network traffic. 

Save session related data to database

Actually some apps choose this way to make session management under control.

There are two ways to do this,

1.  Customize 'session' in the app. In this case, "session" is not HTTP session any more, it just means data that used as session data. 
   
   Each Tomcat instances can access the central db to get the session related data and load the data into its own cache/session. Also each instances can write data into db for session share purposes.
   
   Each request need to call the db for twice, one get session, one update session.
   Good point: data is fully under control.
   Bad point: database needs to be periodically clean up.
2. Extends Tomcat Manager class and customize your own session manager, backup your tomcat session data into MongoDB.
   
   Reference implementation:
   https://github.com/dawsonsystems/Mongo-Tomcat-Sessions
   https://github.com/dwelch2344/mongo-tomcat-session-manager
   
   Personally I like this way, because MongoDB is a mature product and in additional, its client lib(driver) is also robust. And it's fast. Although the speed is not that fast compared to backup method like memcache which backup session in memory, but it's still fast enough and the data is more safe. But still, old topic, all the decision should be made based on our own requirement.
   
   Little Note: In this way, we do not need extra periodical job to clear the session in db. We can override the backgroundProcess() method inheritted from Tomcat's Manager class.

Session in Cookie

This way is to save session data in the client side instead of server side. Each time client make the request, send the session back to server. In this way, developers can make control on session consume. But known issues is security concern. 

Central Cache (External Cache)

External Cache is a separate service that runs on the machine in the LAN. Use external cache can realize central session management. 

It stores the session in memory and all the tomcat instances can access the central cache to read session and set session. There is only one copy of for each session and all the tomcat instances share the same one.

Normally, there should be at least one central cache node and one backup cache node.

There are some implementations for central cache,  memcached (open-source) and Terracotta (open-source, commercial). Central cache has been proved to be fast and reliable, and many large web sites use central cache for session management.

But to set up central cache, may need professional system engineer to setup environment, maintain and monitor.

Summary

The core idea of using Central Cache and MongoDB is actually same, just backup session into one central place for reference. The only difference is cache saves data in memory and db saves data in hard drive.

We can pick one way based on requirement. Both ways have reference implementation on tomcat session manager.

Central Session Mangement - MongoDB Test Case

To test the central cache, I setup the environment and build one test web app.

central-session-management-mongo-test.zip

Structure:

Environment:

Tomcat 7,

Java Default Serializer,

mongodb-win32-x86_64-2.0.7 (Windows 64 bits version)

#Start mongodb

1. mongobin \>mongod.exe --dbpath=”DATA_PATH”

2. In another console start MongoDB client console by executing mongo.exe then execute the following commands.

   use tomcatsessions

   db.addUser("sessionadmin", "sessionadmin234")

3. Stop the MongoDB server (i.e. press Ctrl+C)

4. When the prompt shows, restart the MongoDB server with the --auth parameter:

   mongobin\>mongod.exe  --auth  --dbpath=”DATA_PATH”

 

#Start two tomcat instances

1. Run tomcat-instance\apache-tomcat-7.0.29-1\bin\startup.bat

   Port: 8080

1. Run tomcat-instance\apache-tomcat-7.0.29-2\bin\startup.bat

   Port: 9080

   

Test Case:

#Put String in the session

Access:

 http://localhost:8080/centralcache-test/SessionPage1

 1. Get the session, if not exist, create a new one.

 2. Add attribute "currentUser": "marym" into session

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

Then Access:

 http://localhost:9080/centralcache-test/SessionPage2

 1. Get the session, if not exist, create a new one.

 2. Print attribute "currentUser" in the session.

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

 

#Put User Object in the session

User{username,password}

Access:

 http://localhost:8080/centralcache-test/login

 http://localhost:9080/centralcache-test/login

 

 If user has not logged in, it will direct to login page. 

 If user has logged in (session has valid attribute "currentUser"), it will display the logged in user's username.  

 If user logout, the "currentUser" in session will be removed. (Not destroy the whole session)

 

Two tomcats share the session. If shut down the first memcache node at port 11211, 

the backup node at port 11212 will be used. Session will not be lost.

Note: Different client side apps do not share session. E.g. IE, Chrome, FireFox do not share session.

#Test Put 10K around session data into session

 http://localhost:8080/testlargeusersession

 http://localhost:9080/testlargeusersession

#Test Put 100bytes around session data into session

 http://localhost:8080/testsmallusersession

 http://localhost:9080/testsmallusersession

 

#Test Multi-thread Support

Open Jmeter Test Case.

Run.

I test 10000 concurrent threads with session data around 10K. Works pretty good.

Central Cache Test Case

To test the central cache, I setup the environment and build one test web app.

central-session-test.zip

Structure:

Environment:

Tomcat 7,
memcached-session-manager-1.6.4

memcached-amd64 (Windows 64 bits version)

#Start two memcache nodes

1. Run memcached-amd64\run-instance1.bat

   Port: 11211

2. Run memcached-amd64\run-instance2.bat

   Port: 11212

#Start two tomcat instances

1. Run tomcat-instance\apache-tomcat-7.0.29-1\bin\startup.bat

   Port: 8080

1. Run tomcat-instance\apache-tomcat-7.0.29-2\bin\startup.bat

   Port: 9080

   

Test Case:

#Put String in the session

Access:

 http://localhost:8080/centralcache-test/SessionPage1

 1. Get the session, if not exist, create a new one.

 2. Add attribute "currentUser": "marym" into session

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

Then Access:

 http://localhost:9080/centralcache-test/SessionPage2

 1. Get the session, if not exist, create a new one.

 2. Print attribute "currentUser" in the session.

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

#Put User Object in the session

User{username,password}

Access:

 http://localhost:8080/centralcache-test/login

 http://localhost:9080/centralcache-test/login

 If user has not logged in, it will direct to login page. 

 If user has logged in (session has valid attribute "currentUser"), it will display the logged in user's username.  

 If user logout, the "currentUser" in session will be removed. (Not destroy the whole session)

Two tomcats share the session. If shut down the first memcache node at port 11211, 

the backup node at port 11212 will be used. Session will not be lost.

Note: Different client side apps do not share session. E.g. IE, Chrome, FireFox do not share session.

Sample configuration:

1. Add dependencies (jars that should be placed under tomcat\lib):

    

    For Tomcat 7

• memcached-session-manager-1.6.4.jar
• memcached-session-manager-tc7-1.6.4.jar
• spymemcached-2.8.12.jar
• couchbase-client-1.1.2.jar

    For Tomcat 6

• memcached-session-manager-1.6.4.jar
• memcached-session-manager-tc6-1.6.4.jar
• spymemcached-2.8.12.jar
• couchbase-client-1.1.2.jar

    
    I've put all these dependencies in the zip file central-session-test.zip

2. Tomcat /conf/context.xml  (All instances use the same conf, non-sticky session)

    #Use Javolution Serializer

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="n1:localhost:11212,n2:localhost:11213"
   sticky="false"
   sessionBackupAsync="false"
   lockingMode="uriPattern:/path1|/path2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
   transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
/>

      Extra Dependencies(jars that should be placed under tomcat\lib):

• javolution-5.4.3.1.jar
• msm-javolution-serializer-1.6.4.jar

 
 #Use Kryo Serializer

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="n1:localhost:11212,n2:localhost:11213"
   sticky="false"
   sessionBackupAsync="false"
   lockingMode="uriPattern:/path1|/path2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
   transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>

      Extra Dependencies(jars that should be placed under tomcat\lib):

• msm-kryo-serializer-1.6.4.jar
• kryo-1.04.jar
• kryo-serializers-0.10.jar
• asm-3.2.jar
• reflectasm-1.01.jar
• minlog-1.2.jar

     Configure Kryo's buffer size.
      By default the initial buffer size is 100K, and maxium is 2M.
      If the session data is big, then we need to update these two values by adding one system property.

      In Tomcat's startup.bat file:

      set JAVA_OPTS=%JAVA_OPTS% -Dmsm.kryo.buffersize.initial=10444800 -Dmsm.kryo.buffersize.max=104448000

3. Start MemCache

Unzip memcached-amd64.zip, Go to memcached-amd64 directory.

memcached -p 11211 -u memcached -m 64 -M -vv

memcached -p 11212 -u memcached -m 64 -M -vv

-vv tells memcached to write lots of stuff to console, so you'll see when a session is requested or stored in the output of memcached.

4. Start Tomcat
    ->Tomcat Base/bin/start.bat

Reference:
---memcached-session-manager WIKI ---SetupAndConfiguration

---Various versions of Memcached for Windows

---memcached-session-manager Source Code

---Memcached on Windows and telnet interface

Friendly Reminder:

Please take care on the version number of extra lib that added into Tomcat's library. There are many difference versions of jar. If use wrong version, it may cause jar conflict or class missing.

In my above example. I use memcached-session-manager-1.6.4, then the extra dependency jar files that I listed is based on version 1.6.4 of  memcached-session-manager. If use other different version of memcached-session-manager, please check the dependencies' version and adjust if needed.


Little Notes:

=====Issue 1=====

If you wanna to install the Memcached for Windows and the follow instruction in Memcached on Windows and telnet interface, it will throw exception. "Failed to ignore SIGHUP: Result too large". To resolve this, can follow below steps:

1) run CMD as an administrator
2) type SC create memcached binpath= "c:\memcached\145\memcached.exe -m 512
-d"
3) type NET START memcahed

Although you get an error, the program started (check task manager or connect to it using telnet).

=====Issue 2=====

If you configure like this:

Tomcat context.xml:

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="n1:localhost:11212"
   sticky="false"
   sessionBackupAsync="false"
   lockingMode="uriPattern:/path1|/path2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
   transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
/>

Error:

java.lang.NoSuchMethodError: net.spy.memcached.MemcachedClient.set(Ljava/lang/String;ILjava/lang/Object;)Lnet/spy/memcached/internal/OperationFuture;

at de.javakaffee.web.msm.BackupSessionTask.storeSessionInMemcached(BackupSessionTask.java:227)

at de.javakaffee.web.msm.BackupSessionTask.doBackupSession(BackupSessionTask.java:194)

at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:119)

at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:50)

at de.javakaffee.web.msm.BackupSessionService$SynchronousExecutorService.submit(BackupSessionService.java:346)

at de.javakaffee.web.msm.BackupSessionService.backupSession(BackupSessionService.java:205)

at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:1059)

at de.javakaffee.web.msm.RequestTrackingHostValve.backupSession(RequestTrackingHostValve.java:229)

at de.javakaffee.web.msm.RequestTrackingHostValve.invoke(RequestTrackingHostValve.java:154)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

at java.lang.Thread.run(Thread.java:619)

Solution:

Update memcached jar to spymemcached-2.8.12.jar or spymemcached-2.8.4.jar

=====Issue 3=====

memcached-session-manager provides several serializers. The testing of this part will be processed later, applicability, efficiency, performance, etc.