Issue: WARNING: Exception processing loader WebappLoader[/HIDDEN_NAME] background process

After we deployed web app in Tomcat and start the Tomcat. In some case we get exception like below:

org.apache.catalina.core.ContainerBase backgroundProcess

WARNING: Exception processing loader WebappLoader[/HIDDEN_NAME] background process java.lang.StringIndexOutOfBoundsException: String index out of range: 137

        at java.lang.String.substring(String.java:1934)

        at org.apache.catalina.util.RequestUtil.normalize(RequestUtil.java:133)

        at org.apache.naming.resources.FileDirContext.normalize(FileDirContext.java:784)

        at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:823)

        at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:430)

        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1089)

        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1042)

        at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:880)

        at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:974)

        at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:499)

        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:419)

        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1214)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1410)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1410)

        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1389)

        at java.lang.Thread.run(Thread.java:619)

It is basically because the <context> block placed in the server.xml, which set "reloadable" to "true". (For quick fix, just change the reloadable to "false", but for long time sustain, we better follow the official manual to config our Tomcat.)

In the official manual, we got suggestion like this:

It is NOT recommended to place <Context> elements directly in the server.xml file. This is because it makes modifying the Context configuration more invasive since the main conf/server.xml file cannot be reloaded without restarting Tomcat.

Individual Context elements may be explicitly defined:

• In an individual file at /META-INF/context.xml inside the application files. Optionally (based on the Host's copyXML attribute) this may be copied to$CATALINA_BASE/conf/[enginename]/[hostname]/ and renamed to application's base file name plus a ".xml" extension.
• In individual files (with a ".xml" extension) in the $CATALINA_BASE/conf/[enginename]/[hostname]/ directory. The context path and version will be derived from the base name of the file (the file name less the .xml extension). This file will always take precedence over any context.xml file packaged in the web application's META-INF directory.
• Inside a Host element in the main conf/server.xml.

Default Context elements may be defined that apply to multiple web applications. Configuration for an individual web application will override anything configured in one of these defaults. Any nested elements, e.g. <Resource> elements, that are defined in a default Context will be created once for each Context to which the default applies. They will not be shared between Context elements.

• In the $CATALINA_BASE/conf/context.xml file: the Context element information will be loaded by all web applications.
• In the $CATALINA_BASE/conf/[enginename]/[hostname]/context.xml.default file: the Context element information will be loaded by all web applications of that host.

With the exception of server.xml, files that define Context elements may only define a single Context element.

In addition to explicitly specified Context elements, there are several techniques by which Context elements can be created automatically for you. See Automatic Application Deployment and User Web Applications for more information.

To define multiple contexts that use a single WAR file or directory, use one of the options described in the Naming section above for creating a Context that has a path that is not related to the base file name.

Reference: http://tomcat.apache.org/tomcat-7.0-doc/config/context.html

Also I got a note from one professional: 

"

This is my "dissection" of problematic stacktrace using my limited Java knowledge and Tomcat sources. 
Stacktrace written in "reverse" way to follow code flow more easily. 

        at java.lang.Thread.run(Thread.java:619) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) 

Background thread which periodically calls "backgroundProcess" on container and its children, I believe it is 
used to determine wether app needs to be reloaded (make sense to me that something periodically needs to check 
wether app needs to be reloaded). 

        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1309) 
        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:398) 
        at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:477) 
        at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:822) 
        
"modified" is called in order to find if "one or more classes or resources been modified so that a reload is appropriate?", 
at least that is what the JavaDoc comment says. 

        at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:840) 
        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:747) 
        at org.apache.naming.resources.FileDirContext.getAttributes(FileDirContext.java:429) 

Just forwards call to more appropriate classes up to here. 

        at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:811) 

Here a File object of "Return a File object representing the specified normalized 
context-relative path if it exists and is readable" is created, however name given to it 
chokes up "normalize", it seems. 

        at org.apache.naming.resources.FileDirContext.normalize(FileDirContext.java:771) 
        at org.apache.catalina.util.RequestUtil.normalize(RequestUtil.java:131) 

This is problematic code (part of normalize method): 

        // Resolve occurrences of "//" in the normalized path 
        while (true) { 
                int index = normalized.indexOf("//"); 
                if (index < 0) 
                        break; 
                normalized = normalized.substring(0, index) + 
                        normalized.substring(index + 1); 
        } 

"

Suggestion: 

0. Stop Tomcat. 

1. Create a file (in each app) myappname/META-INF/context.xml. 

2. Copy the entire Context definition & it's sub-elements into the file 

3. Remove the 3 attributes I named (debug is deprecated, path & docBase 
aren't used here) 

4. Completely delete the Context definitions from server.xml. 

5. Start Tomcat. 

Reference: http://tomcat.10.x6.nabble.com/Webapp-reloading-issue-and-intermittent-404-errors-td2128184.html