How to lookup port usage and release the port. (Resolve Issue: Port in-use by unknown process)

When we develop web applications, a very common problem is that sometimes we don't know what is using the current port that we need. And we cann't start our server while getting this kind of exception:

First we could look at the Task Manager to see whether we can get the process which is using this port. But what if we got nothing? Kill every suspects? It's dangerous and sometimes doesn't help at all.
Restart the computer? Ohh that'll work, that's also what I did previously but it's so out!!! It's so painful to restart everything and it's really unneccessary.

Now I get one solution, which is very easy and helpful.

For example, I want to user port "8080" but it's in use by unknown process.

First open the cmd, execute => netstat -aon|findstr "8080"
Then you'll get result like this:
  TCP    0.0.0.0:8080           0.0.0.0:0              LISTENING       87460
  TCP    [::]:8080                 [::]:0                    LISTENING       87460

Now we know who is using the port, the process with PID "87460".

Then in the cmd, we execute => taskkill /pid 87460 -f
Then you'll get result like this:
SUCCESS: The process with PID 87460 has been terminated.

That's it! Then you retart the server, everything will be fine.

Issue: WARNING: Exception processing loader WebappLoader[/HIDDEN_NAME] background process

After we deployed web app in Tomcat and start the Tomcat. In some case we get exception like below:

org.apache.catalina.core.ContainerBase backgroundProcess

WARNING: Exception processing loader WebappLoader[/HIDDEN_NAME] background process java.lang.StringIndexOutOfBoundsException: String index out of range: 137

        at java.lang.String.substring(String.java:1934)

        at org.apache.catalina.util.RequestUtil.normalize(RequestUtil.java:133)

        at org.apache.naming.resources.FileDirContext.normalize(FileDirContext.java:784)

        at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:823)

        at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:430)

        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1089)

        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1042)

        at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:880)

        at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:974)

        at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:499)

        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:419)

        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1214)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1410)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1410)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1389)

        at java.lang.Thread.run(Thread.java:619)

It is basically because the <context> block placed in the server.xml, which set "reloadable" to "true". (For quick fix, just change the reloadable to "false", but for long time sustain, we better follow the official manual to config our Tomcat.)

In the official manual, we got suggestion like this:

It is NOT recommended to place <Context> elements directly in the server.xml file. This is because it makes modifying the Context configuration more invasive since the main conf/server.xml file cannot be reloaded without restarting Tomcat.

Individual Context elements may be explicitly defined:

• In an individual file at /META-INF/context.xml inside the application files. Optionally (based on the Host's copyXML attribute) this may be copied to$CATALINA_BASE/conf/[enginename]/[hostname]/ and renamed to application's base file name plus a ".xml" extension.
• In individual files (with a ".xml" extension) in the $CATALINA_BASE/conf/[enginename]/[hostname]/ directory. The context path and version will be derived from the base name of the file (the file name less the .xml extension). This file will always take precedence over any context.xml file packaged in the web application's META-INF directory.
• Inside a Host element in the main conf/server.xml.

Default Context elements may be defined that apply to multiple web applications. Configuration for an individual web application will override anything configured in one of these defaults. Any nested elements, e.g. <Resource> elements, that are defined in a default Context will be created once for each Context to which the default applies. They will not be shared between Context elements.

• In the $CATALINA_BASE/conf/context.xml file: the Context element information will be loaded by all web applications.
• In the $CATALINA_BASE/conf/[enginename]/[hostname]/context.xml.default file: the Context element information will be loaded by all web applications of that host.

With the exception of server.xml, files that define Context elements may only define a single Context element.

In addition to explicitly specified Context elements, there are several techniques by which Context elements can be created automatically for you. See Automatic Application Deployment and User Web Applications for more information.

To define multiple contexts that use a single WAR file or directory, use one of the options described in the Naming section above for creating a Context that has a path that is not related to the base file name.

Reference: http://tomcat.apache.org/tomcat-7.0-doc/config/context.html

Also I got a note from one professional: 

"

This is my "dissection" of problematic stacktrace using my limited Java knowledge and Tomcat sources. 
Stacktrace written in "reverse" way to follow code flow more easily. 

        at java.lang.Thread.run(Thread.java:619) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) 

Background thread which periodically calls "backgroundProcess" on container and its children, I believe it is 
used to determine wether app needs to be reloaded (make sense to me that something periodically needs to check 
wether app needs to be reloaded). 

        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1309) 
        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:398) 
        at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:477) 
        at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:822) 
        
"modified" is called in order to find if "one or more classes or resources been modified so that a reload is appropriate?", 
at least that is what the JavaDoc comment says. 

        at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:840) 
        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:747) 
        at org.apache.naming.resources.FileDirContext.getAttributes(FileDirContext.java:429) 

Just forwards call to more appropriate classes up to here. 

        at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:811) 

Here a File object of "Return a File object representing the specified normalized 
context-relative path if it exists and is readable" is created, however name given to it 
chokes up "normalize", it seems. 

        at org.apache.naming.resources.FileDirContext.normalize(FileDirContext.java:771) 
        at org.apache.catalina.util.RequestUtil.normalize(RequestUtil.java:131) 

This is problematic code (part of normalize method): 

        // Resolve occurrences of "//" in the normalized path 
        while (true) { 
                int index = normalized.indexOf("//"); 
                if (index < 0) 
                        break; 
                normalized = normalized.substring(0, index) + 
                        normalized.substring(index + 1); 
        } 

"

Suggestion: 

0. Stop Tomcat. 

1. Create a file (in each app) myappname/META-INF/context.xml. 

2. Copy the entire Context definition & it's sub-elements into the file 

3. Remove the 3 attributes I named (debug is deprecated, path & docBase 
aren't used here) 

4. Completely delete the Context definitions from server.xml. 

5. Start Tomcat. 

Reference: http://tomcat.10.x6.nabble.com/Webapp-reloading-issue-and-intermittent-404-errors-td2128184.html

CentralCache Management(Memcache) Issue 1 -- net.spy.memcached.internal.CheckedOperationTimeoutException

One main aspect to evaluate one web app is its scalability, whether it can sustain heavy load.
So when we decide to build one web app, each technology that we plan to use must go through enough tests, not only functional test, but also load test. Otherwise, we may lose at the start point.

When I test the performance of tomcat-memcache central session management, I save&get small session data and large session data with both light load and heavy load.

When load is light, everything goes fine. But when load increase, and session data is around 10K, exception will be thrown:

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: localhost/127.0.0.1:11212
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160)
        at de.javakaffee.web.msm.LockingStrategy.onAfterBackupSession(LockingStrategy.java:294)
        at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:1062)
        at de.javakaffee.web.msm.RequestTrackingHostValve.backupSession(RequestTrackingHostValve.java:243)
        at de.javakaffee.web.msm.RequestTrackingHostValve.invoke(RequestTrackingHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1770)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Regarding this issue, I found one discussion page:
https://code.google.com/p/spymemcached/issues/detail?id=136
But I did not see effect solution. Maybe switch to other memcache client lib will be a better solution.
Somebody says that increase the spymemcache's timeout may help, but I haven't got time to test it.
Compared to memcache, now I prefer mongodb as the backup of session. At least it's client lib is more stable and under enough test.

So If we need to setup the central cache for web app, we need to customize and test the lib well based on our requirement. I think no one wants to face this issue in production.

Central Session Management (Tomcat, Memcache/MongoDB, Non-Sticky Session)

For all application server clusters, there are two elementary issues that need to be resolved:

1. How to dispatch requests to the cluster nodes.
2. Use one session management strategy, to ensure that if one node fails, other nodes can still obtained the session data, to achieve fault-tolerant cluster (failover)

For the first issue, the easiest way is to use uniform hash function to dispatch the request to each node. But for application server, session is used. To make sure one session’s request is properly handled, normally we dispatch the requests related to one session to same node to process, this is “Sticky Session” load balance. And the way randomly dispatch request is called “non-sticky session” load balance.

The “sticky session” sometimes restricts the load balance and even the web app’s design and development. To avoid “sticky session”, we need to find appropriate session management strategy.

There are several options for session share.

Session Share Options

Tomcat Clustering

It simply copied session to all the tomcat instances. If there are many instances, this may cause a lot network traffic. 

Save session related data to database

Actually some apps choose this way to make session management under control.

There are two ways to do this,

1.  Customize 'session' in the app. In this case, "session" is not HTTP session any more, it just means data that used as session data. 
   
   Each Tomcat instances can access the central db to get the session related data and load the data into its own cache/session. Also each instances can write data into db for session share purposes.
   
   Each request need to call the db for twice, one get session, one update session.
   Good point: data is fully under control.
   Bad point: database needs to be periodically clean up.
2. Extends Tomcat Manager class and customize your own session manager, backup your tomcat session data into MongoDB.
   
   Reference implementation:
   https://github.com/dawsonsystems/Mongo-Tomcat-Sessions
   https://github.com/dwelch2344/mongo-tomcat-session-manager
   
   Personally I like this way, because MongoDB is a mature product and in additional, its client lib(driver) is also robust. And it's fast. Although the speed is not that fast compared to backup method like memcache which backup session in memory, but it's still fast enough and the data is more safe. But still, old topic, all the decision should be made based on our own requirement.
   
   Little Note: In this way, we do not need extra periodical job to clear the session in db. We can override the backgroundProcess() method inheritted from Tomcat's Manager class.

Session in Cookie

This way is to save session data in the client side instead of server side. Each time client make the request, send the session back to server. In this way, developers can make control on session consume. But known issues is security concern. 

Central Cache (External Cache)

External Cache is a separate service that runs on the machine in the LAN. Use external cache can realize central session management. 

It stores the session in memory and all the tomcat instances can access the central cache to read session and set session. There is only one copy of for each session and all the tomcat instances share the same one.

Normally, there should be at least one central cache node and one backup cache node.

There are some implementations for central cache,  memcached (open-source) and Terracotta (open-source, commercial). Central cache has been proved to be fast and reliable, and many large web sites use central cache for session management.

But to set up central cache, may need professional system engineer to setup environment, maintain and monitor.

Summary

The core idea of using Central Cache and MongoDB is actually same, just backup session into one central place for reference. The only difference is cache saves data in memory and db saves data in hard drive.

We can pick one way based on requirement. Both ways have reference implementation on tomcat session manager.

Central Session Mangement - MongoDB Test Case

To test the central cache, I setup the environment and build one test web app.

central-session-management-mongo-test.zip

Structure:

Environment:

Tomcat 7,

Java Default Serializer,

mongodb-win32-x86_64-2.0.7 (Windows 64 bits version)

#Start mongodb

1. mongobin \>mongod.exe --dbpath=”DATA_PATH”

2. In another console start MongoDB client console by executing mongo.exe then execute the following commands.

   use tomcatsessions

   db.addUser("sessionadmin", "sessionadmin234")

3. Stop the MongoDB server (i.e. press Ctrl+C)

4. When the prompt shows, restart the MongoDB server with the --auth parameter:

   mongobin\>mongod.exe  --auth  --dbpath=”DATA_PATH”

 

#Start two tomcat instances

1. Run tomcat-instance\apache-tomcat-7.0.29-1\bin\startup.bat

   Port: 8080

1. Run tomcat-instance\apache-tomcat-7.0.29-2\bin\startup.bat

   Port: 9080

   

Test Case:

#Put String in the session

Access:

 http://localhost:8080/centralcache-test/SessionPage1

 1. Get the session, if not exist, create a new one.

 2. Add attribute "currentUser": "marym" into session

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

Then Access:

 http://localhost:9080/centralcache-test/SessionPage2

 1. Get the session, if not exist, create a new one.

 2. Print attribute "currentUser" in the session.

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

 

#Put User Object in the session

User{username,password}

Access:

 http://localhost:8080/centralcache-test/login

 http://localhost:9080/centralcache-test/login

 

 If user has not logged in, it will direct to login page. 

 If user has logged in (session has valid attribute "currentUser"), it will display the logged in user's username.  

 If user logout, the "currentUser" in session will be removed. (Not destroy the whole session)

 

Two tomcats share the session. If shut down the first memcache node at port 11211, 

the backup node at port 11212 will be used. Session will not be lost.

Note: Different client side apps do not share session. E.g. IE, Chrome, FireFox do not share session.

#Test Put 10K around session data into session

 http://localhost:8080/testlargeusersession

 http://localhost:9080/testlargeusersession

#Test Put 100bytes around session data into session

 http://localhost:8080/testsmallusersession

 http://localhost:9080/testsmallusersession

 

#Test Multi-thread Support

Open Jmeter Test Case.

Run.

I test 10000 concurrent threads with session data around 10K. Works pretty good.

Central Cache Test Case

To test the central cache, I setup the environment and build one test web app.

central-session-test.zip

Structure:

Environment:

Tomcat 7,
memcached-session-manager-1.6.4

memcached-amd64 (Windows 64 bits version)

#Start two memcache nodes

1. Run memcached-amd64\run-instance1.bat

   Port: 11211

2. Run memcached-amd64\run-instance2.bat

   Port: 11212

#Start two tomcat instances

1. Run tomcat-instance\apache-tomcat-7.0.29-1\bin\startup.bat

   Port: 8080

1. Run tomcat-instance\apache-tomcat-7.0.29-2\bin\startup.bat

   Port: 9080

   

Test Case:

#Put String in the session

Access:

 http://localhost:8080/centralcache-test/SessionPage1

 1. Get the session, if not exist, create a new one.

 2. Add attribute "currentUser": "marym" into session

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

Then Access:

 http://localhost:9080/centralcache-test/SessionPage2

 1. Get the session, if not exist, create a new one.

 2. Print attribute "currentUser" in the session.

 Page will show:

 Current session Id: ${session id}

 CurrentUser: marym

#Put User Object in the session

User{username,password}

Access:

 http://localhost:8080/centralcache-test/login

 http://localhost:9080/centralcache-test/login

 If user has not logged in, it will direct to login page. 

 If user has logged in (session has valid attribute "currentUser"), it will display the logged in user's username.  

 If user logout, the "currentUser" in session will be removed. (Not destroy the whole session)

Two tomcats share the session. If shut down the first memcache node at port 11211, 

the backup node at port 11212 will be used. Session will not be lost.

Note: Different client side apps do not share session. E.g. IE, Chrome, FireFox do not share session.

Sample configuration:

1. Add dependencies (jars that should be placed under tomcat\lib):

    

    For Tomcat 7

• memcached-session-manager-1.6.4.jar
• memcached-session-manager-tc7-1.6.4.jar
• spymemcached-2.8.12.jar
• couchbase-client-1.1.2.jar

    For Tomcat 6

• memcached-session-manager-1.6.4.jar
• memcached-session-manager-tc6-1.6.4.jar
• spymemcached-2.8.12.jar
• couchbase-client-1.1.2.jar

    
    I've put all these dependencies in the zip file central-session-test.zip

2. Tomcat /conf/context.xml  (All instances use the same conf, non-sticky session)

    #Use Javolution Serializer

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="n1:localhost:11212,n2:localhost:11213"
   sticky="false"
   sessionBackupAsync="false"
   lockingMode="uriPattern:/path1|/path2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
   transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
/>

      Extra Dependencies(jars that should be placed under tomcat\lib):

• javolution-5.4.3.1.jar
• msm-javolution-serializer-1.6.4.jar

 
 #Use Kryo Serializer

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="n1:localhost:11212,n2:localhost:11213"
   sticky="false"
   sessionBackupAsync="false"
   lockingMode="uriPattern:/path1|/path2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
   transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"
/>

      Extra Dependencies(jars that should be placed under tomcat\lib):

• msm-kryo-serializer-1.6.4.jar
• kryo-1.04.jar
• kryo-serializers-0.10.jar
• asm-3.2.jar
• reflectasm-1.01.jar
• minlog-1.2.jar

     Configure Kryo's buffer size.
      By default the initial buffer size is 100K, and maxium is 2M.
      If the session data is big, then we need to update these two values by adding one system property.

      In Tomcat's startup.bat file:

      set JAVA_OPTS=%JAVA_OPTS% -Dmsm.kryo.buffersize.initial=10444800 -Dmsm.kryo.buffersize.max=104448000

3. Start MemCache

Unzip memcached-amd64.zip, Go to memcached-amd64 directory.

memcached -p 11211 -u memcached -m 64 -M -vv

memcached -p 11212 -u memcached -m 64 -M -vv

-vv tells memcached to write lots of stuff to console, so you'll see when a session is requested or stored in the output of memcached.

4. Start Tomcat
    ->Tomcat Base/bin/start.bat

Reference:
---memcached-session-manager WIKI ---SetupAndConfiguration

---Various versions of Memcached for Windows

---memcached-session-manager Source Code

---Memcached on Windows and telnet interface

Friendly Reminder:

Please take care on the version number of extra lib that added into Tomcat's library. There are many difference versions of jar. If use wrong version, it may cause jar conflict or class missing.

In my above example. I use memcached-session-manager-1.6.4, then the extra dependency jar files that I listed is based on version 1.6.4 of  memcached-session-manager. If use other different version of memcached-session-manager, please check the dependencies' version and adjust if needed.


Little Notes:

=====Issue 1=====

If you wanna to install the Memcached for Windows and the follow instruction in Memcached on Windows and telnet interface, it will throw exception. "Failed to ignore SIGHUP: Result too large". To resolve this, can follow below steps:

1) run CMD as an administrator
2) type SC create memcached binpath= "c:\memcached\145\memcached.exe -m 512
-d"
3) type NET START memcahed

Although you get an error, the program started (check task manager or connect to it using telnet).

=====Issue 2=====

If you configure like this:

Tomcat context.xml:

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
   memcachedNodes="n1:localhost:11212"
   sticky="false"
   sessionBackupAsync="false"
   lockingMode="uriPattern:/path1|/path2"
   requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
   transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
/>

Error:

java.lang.NoSuchMethodError: net.spy.memcached.MemcachedClient.set(Ljava/lang/String;ILjava/lang/Object;)Lnet/spy/memcached/internal/OperationFuture;

at de.javakaffee.web.msm.BackupSessionTask.storeSessionInMemcached(BackupSessionTask.java:227)

at de.javakaffee.web.msm.BackupSessionTask.doBackupSession(BackupSessionTask.java:194)

at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:119)

at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:50)

at de.javakaffee.web.msm.BackupSessionService$SynchronousExecutorService.submit(BackupSessionService.java:346)

at de.javakaffee.web.msm.BackupSessionService.backupSession(BackupSessionService.java:205)

at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:1059)

at de.javakaffee.web.msm.RequestTrackingHostValve.backupSession(RequestTrackingHostValve.java:229)

at de.javakaffee.web.msm.RequestTrackingHostValve.invoke(RequestTrackingHostValve.java:154)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:261)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:581)

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

at java.lang.Thread.run(Thread.java:619)

Solution:

Update memcached jar to spymemcached-2.8.12.jar or spymemcached-2.8.4.jar

=====Issue 3=====

memcached-session-manager provides several serializers. The testing of this part will be processed later, applicability, efficiency, performance, etc.