8/26/2013

Maven assemble package and distribution

For example we have one project called "performance-test". And it has several modules, "shared", "user-creation", "performance" like below:


The parent pom.xml has modules:
<groupId>com.test</groupId>
<artifactId>performance-test</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>

<name>performance-test</name>

<modules>
    <module>shared</module>
    <module>user-creation</module>
    <module>performance</module>
</modules>
While the modules is like:
<parent>
    <groupId>com.test</groupId>
    <artifactId>performance-test</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <relativePath>../pom.xml</relativePath>
</parent>

<artifactId>shared</artifactId>
<name>shared</name>


And the user-creation and performance module has several main classes that used to be separately executed to do the performance test.

First, for module "user-creation" and "performance", we need to package all dependencies into one executable jar file.(Simple way. We can also pick another way by copying all the dependencies to later lib folder and configure the classpath.)

Simply by configuring this plugin in pom.xml of "user-creation" and "performance":

<plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-assembly-plugin</artifactId>
 <version>2.4</version>
 <configuration>
  <appendAssemblyId>true</appendAssemblyId>
 </configuration>
 <executions>
  <execution>
   <id>assemblyJar</id>
   <phase>package</phase>
   <goals>
    <goal>single</goal>
   </goals>
   <configuration>
    <descriptorRefs>
     <descriptorRef>jar-with-dependencies</descriptorRef>
    </descriptorRefs>
   </configuration>
  </execution>
 </executions>
</plugin>

In this way, we can get the executable jar files.

Extension:

When we have multiple modules and each module may have its own resources.We may want to put all the resources into one place that could make the property management easier and clearer.

For example, we may put the central resource folder under the parent project:


To support structure like this, we need one environment variable, for example, "resource.path" to tell the program where to find the central resource folder.

When we make the deployment, we can put the resources folder anywhere we want, as long as we define the variable.

We can define the variable in command line before we run the code.

-Dresource.path="C:/SOME_PATH"

Also, we can add it to system variable.

Extension:

To make the execution of jar easy, we can also write some bat files and put them into folder "bin" and put the folder under parent project, like below:
With this structure, after we make the build, we may want to get one instance folder which has structure like:

"bin": holds all the bat file.

"lib": holds all the jars.

"config": holds all the resources.

"log": holds all the log files, runtime generated.

"records": holds all the records generated by the code.
...

If all the code are put under one project instead of modules, we can easily get this by using the maven-assembly-plugin, configure it in the pom file of the project.

But when we have multiple modules and in the mean time, need to assemble libraries from different modules, we need to create another module "distribution" to access all the other sub-modules and assemble the instance folder.

The new structure of the project will be like:

The parent pom.xml will have modules:
<groupId>com.test</groupId>
<artifactId>performance-test</artifactId>
<version>1.0.0-SNAPSHOT</version>
<packaging>pom</packaging>

<name>performance-test</name>

<modules>
    <module>shared</module>
    <module>user-creation</module>
    <module>performance</module>
    <module>distribution</module>
</modules>

Because the parent pom is executed before all the modules' pom and the modules are built in sequence. So we have to define another module to make the final assemble.

The pom of the distribution will be like:

<parent>
 <groupId>com.test</groupId>
 <artifactId>performance-test</artifactId>
 <version>1.0.0-SNAPSHOT</version>
 <relativePath>../pom.xml</relativePath>
</parent>

<artifactId>distribution</artifactId>
<packaging>pom</packaging>
<name>distribution</name>

<url>http://maven.apache.org</url>
<properties>
 <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
 <dependency>
  <groupId>com.test</groupId>
  <artifactId>user-creation</artifactId>
  <version>${project.version}</version>
 </dependency>
 <dependency>
  <groupId>com.test</groupId>
  <artifactId>performance</artifactId>
  <version>${project.version}</version>
 </dependency>
</dependencies>
<build>
 <directory>${basedir}/../instance</directory>
 <plugins>
  <plugin>
   <artifactId>maven-assembly-plugin</artifactId>
   <version>2.4</version>
   <executions>
    <execution>
     <id>distri-assembly</id>
     <phase>package</phase>
     <goals>
      <goal>single</goal>
     </goals>
     <configuration>
      <descriptors>
       <descriptor>${basedir}/../assemble/distribution-jar.xml</descriptor>
      </descriptors>
     </configuration>
    </execution>
   </executions>
  </plugin>
 </plugins>
</build>
We can see that we change the output directory of "distribution" to "%PARENT_PROJECT%/instance" by specify:
<directory>${basedir}/../instance</directory>

In this way, we will get the instance folder under the root of the parent project. This does not matter actually, we can also just leave it generated under target.

The distribution-jar.xml should be like:
<assembly
 xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
 <id>bin</id>
 <formats>
  <format>dir</format>
  <format>zip</format>
 </formats>
 <includeBaseDirectory>false</includeBaseDirectory>
 <moduleSets>
  <moduleSet>

   <!-- Enable access to all projects in the current multimodule build! -->
   <useAllReactorProjects>true</useAllReactorProjects>

   <!-- Now, select which projects to include in this module-set. -->
   <includes>
    <include>${project.groupId}:user-creation</include>
    <include>${project.groupId}:performance</include>
   </includes>

   <binaries>
    <attachmentClassifier>jar-with-dependencies</attachmentClassifier>
    <outputDirectory>${basedir}/../lib</outputDirectory>
    <unpack>false</unpack>
   </binaries>

  </moduleSet>
 </moduleSets>
 <fileSets>
  <fileSet>
   <directory>${basedir}/../bin</directory>
   <outputDirectory>bin</outputDirectory>
   <filtered>true</filtered>
  </fileSet>
  <fileSet>
   <directory>${project.build.directory}/${project.artifactId}-${project.version}-bin/lib</directory>
   <outputDirectory>lib</outputDirectory>
  </fileSet>
  <fileSet>
   <directory>${basedir}/../resources</directory>
   <excludes>
    <exclude>keystore/*</exclude>
   </excludes>
   <outputDirectory>config</outputDirectory>
   <filtered>true</filtered>
  </fileSet>
  <fileSet>
   <directory>${basedir}/../resources/keystore</directory>
   <outputDirectory>config/keystore</outputDirectory>
   <filtered>false</filtered>
   <lineEnding>keep</lineEnding>
  </fileSet>
 </fileSets>
</assembly>

We can also add some small bat help file to project, so the final structure will be like:


QuickStart.bat:
mvn clean install -Dmaven.test.skip=true
pause
upgradeVersion.bat:
mvn versions:set -DnewVersion=%1

Caution: Maven binary file as resource should disable filter function.

In maven, sometimes when we use binary files as resources or assemble package from files that include binary files, like certificates, if we treat the binary file same as other resources, it may destroy the binary file.

First, the filter function of maven may destroy the binary file.

The description of filter function is in Maven Resource Document.
Basically,

Variables can be included in your resources. These variables, denoted by the ${...} delimiters, can come from the system properties, your project properties, from your filter resources and from the command line.
For example, if we have a resource src/main/resources/hello.txt containing
Hello ${name}
And a POM like this
<project>
  ...
  <name>My Resources Plugin Practice Project</name>
  ...
  <build>
    ...
    <resources>
      <resource>
        <directory>src/main/resources</directory>
      </resource>
      ...
    </resources>
    ...
  </build>
  ...
</project>
Upon calling
mvn resources:resources
This will create a resource output in target/classes/hello.txt which contains exactly the same text.
Hello ${name}
However, if we add a <filtering> tag to our POM and set it to true like this:
      ...
      <resource>
        <directory>src/main/resources</directory>
        <filtering>true</filtering>
      </resource>
      ...
Our target/classes/hello.txt after calling
mvn resources:resources
would be
Hello My Resources Plugin Practice Project
That's because the name variable was replaced by the value of the project's name (which was specified in the POM).
Moreover, we can also assign values through the command line using the "-D" option. For example, to change the value for the variable name to "world", we can simply invoke this command:
mvn resources:resources -Dname="world"
And the output in target/classes/hello.txt would be
Hello world

So we should understand that if we treat the binary file same as other resources, like configuration as below:
      ...
      <resource>
        <directory>src/main/resources</directory>
        <filtering>true</filtering>
      </resource>
      ...

Then when we run the program from the build, it'll throw exceptions when parse the binary file.

Instead, we should configure like this:
<resource>
    <directory>src/main/resources</directory>
    <excludes>
        <exclude>keystore/*</exclude>
    </excludes>
    <filtering>true</filtering>
 </resource>
 <resource>
    <directory>src/main/resources/keystore</directory>
    <filtering>false</filtering>
 </resource>


I see some questions are like "Generated Certificate Stops Working When Moved To Resources Folder"

Basically this is the reason.

Have fun.

Another place that we may face this issue is when we assemble the package.
In the assemble xml file, we need to include the binary file along with other resources into package.

For example:
<fileSets>
 <fileSet>
  <directory>${basedir}/../bin</directory>
  <outputDirectory>bin</outputDirectory>
  <filtered>true</filtered>
 </fileSet>
 <fileSet>
  <directory>${project.build.directory}/${project.artifactId}-${project.version}-bin/lib</directory>
  <outputDirectory>lib</outputDirectory>
 </fileSet>
 <fileSet>
  <directory>${basedir}/../resources</directory>
  <excludes>
   <exclude>keystore/*</exclude>
  </excludes>
  <outputDirectory>resources</outputDirectory>
  <filtered>true</filtered>
 </fileSet>
 <fileSet>
  <directory>${basedir}/../resources/keystore</directory>
  <outputDirectory>resources/keystore</outputDirectory>
  <filtered>false</filtered>
  <lineEnding>keep</lineEnding>
 </fileSet>
</fileSets>

In additional to the filter function, we can also set the lineEnding, directoryMode and fileMode to make sure the file will not be damaged when we make the assembling.








8/07/2013

Query MongoDB with DBRef (xxx: {"$ref":"XXX", "$id": ObjectId("XXX") })

When we have database reference in Mongo Collection. How can we query the record based on the reference id?

Reference:
http://docs.mongodb.org/manual/reference/database-references/

Basically we can query in this way:

For example we have record like this:

Collection : ChannelCfg has this kind of records:

{
  "_id" : ObjectId("51e71075956c1989446b9333"),
  "_class" : "com.test.channel.ConfigEntity",
  "eventTypeId" : "51e71066956c1989446b9213",
  "templateId" : "51e71067956c1989446b922e",
  "createdTime" : ISODate("2013-07-17T21:45:25.073Z"),
  "channelType" : {
    "$ref" : "ChannelType",
    "$id" : ObjectId("51e71064956c1989446b9210")
  }
}

If we want to query the record with specific eventTypeId e.g. "51e71066956c1989446b9213" and specific channelType with id, e.g. "51e71064956c1989446b9210" and assume that this result is unique.

We can make the query:

var event = db.EventType.findOne({"eventType":"TEST_TYPE"});
var emailChannel = db.ChannelType.findOne({"channelType":"SMTP"});
var channelConfig = db.ChannelCfg.findOne( { $and : [{"eventTypeId": event._id.toString()} , {"channelType.$id": emailChannel._id}] });


Note:

In this case the _id in all the collections are all ObjectId type. We can see that in this case, because the 'eventTypeId' in ChannelCfg collection is one String and the event._id is ObjectId type, so we need to make convert "event._id.toString()"(For version 2.0.X)(If it's new version 2.4.x, should use valueOf() instead). And the channelType's "$id" is ObjectId type and we do not need to convert the emailChannel._id.


Issue: WARNING: Exception processing loader WebappLoader[/HIDDEN_NAME] background process

After we deployed web app in Tomcat and start the Tomcat. In some case we get exception like below:

org.apache.catalina.core.ContainerBase backgroundProcess

WARNING: Exception processing loader WebappLoader[/HIDDEN_NAME] background process java.lang.StringIndexOutOfBoundsException: String index out of range: 137
        at java.lang.String.substring(String.java:1934)
        at org.apache.catalina.util.RequestUtil.normalize(RequestUtil.java:133)
        at org.apache.naming.resources.FileDirContext.normalize(FileDirContext.java:784)
        at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:823)
        at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:430)
        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1089)
        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1042)
        at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:880)
        at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:974)
        at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:499)
        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:419)
        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1214)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1400)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1410)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1410)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1389)

        at java.lang.Thread.run(Thread.java:619)

It is basically because the <context> block placed in the server.xml, which set "reloadable" to "true". (For quick fix, just change the reloadable to "false", but for long time sustain, we better follow the official manual to config our Tomcat.)

In the official manual, we got suggestion like this:

It is NOT recommended to place <Context> elements directly in the server.xml file. This is because it makes modifying the Context configuration more invasive since the main conf/server.xml file cannot be reloaded without restarting Tomcat.
Individual Context elements may be explicitly defined:
  • In an individual file at /META-INF/context.xml inside the application files. Optionally (based on the Host's copyXML attribute) this may be copied to$CATALINA_BASE/conf/[enginename]/[hostname]/ and renamed to application's base file name plus a ".xml" extension.
  • In individual files (with a ".xml" extension) in the $CATALINA_BASE/conf/[enginename]/[hostname]/ directory. The context path and version will be derived from the base name of the file (the file name less the .xml extension). This file will always take precedence over any context.xml file packaged in the web application's META-INF directory.
  • Inside a Host element in the main conf/server.xml.
Default Context elements may be defined that apply to multiple web applications. Configuration for an individual web application will override anything configured in one of these defaults. Any nested elements, e.g. <Resource> elements, that are defined in a default Context will be created once for each Context to which the default applies. They will not be shared between Context elements.
  • In the $CATALINA_BASE/conf/context.xml file: the Context element information will be loaded by all web applications.
  • In the $CATALINA_BASE/conf/[enginename]/[hostname]/context.xml.default file: the Context element information will be loaded by all web applications of that host.
With the exception of server.xml, files that define Context elements may only define a single Context element.
In addition to explicitly specified Context elements, there are several techniques by which Context elements can be created automatically for you. See Automatic Application Deployment and User Web Applications for more information.
To define multiple contexts that use a single WAR file or directory, use one of the options described in the Naming section above for creating a Context that has a path that is not related to the base file name.


Also I got a note from one professional: 

"
This is my "dissection" of problematic stacktrace using my limited Java knowledge and Tomcat sources. 
Stacktrace written in "reverse" way to follow code flow more easily. 

        at java.lang.Thread.run(Thread.java:619) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1610) 
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) 

Background thread which periodically calls "backgroundProcess" on container and its children, I believe it is 
used to determine wether app needs to be reloaded (make sense to me that something periodically needs to check 
wether app needs to be reloaded). 

        at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1309) 
        at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:398) 
        at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:477) 
        at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:822) 
        
"modified" is called in order to find if "one or more classes or resources been modified so that a reload is appropriate?", 
at least that is what the JavaDoc comment says. 

        at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:840) 
        at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:747) 
        at org.apache.naming.resources.FileDirContext.getAttributes(FileDirContext.java:429) 

Just forwards call to more appropriate classes up to here. 

        at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:811) 

Here a File object of "Return a File object representing the specified normalized 
context-relative path if it exists and is readable" is created, however name given to it 
chokes up "normalize", it seems. 

        at org.apache.naming.resources.FileDirContext.normalize(FileDirContext.java:771) 
        at org.apache.catalina.util.RequestUtil.normalize(RequestUtil.java:131) 

This is problematic code (part of normalize method): 

        // Resolve occurrences of "//" in the normalized path 
        while (true) { 
                int index = normalized.indexOf("//"); 
                if (index < 0) 
                        break; 
                normalized = normalized.substring(0, index) + 
                        normalized.substring(index + 1); 
        } 

"

Suggestion: 

0. Stop Tomcat. 

1. Create a file (in each app) myappname/META-INF/context.xml. 

2. Copy the entire Context definition & it's sub-elements into the file 

3. Remove the 3 attributes I named (debug is deprecated, path & docBase 
aren't used here) 

4. Completely delete the Context definitions from server.xml. 

5. Start Tomcat. 



8/06/2013

JAXB Marshall and UnMarshall Example (Include post request to REST API)

Pom.xml:
<dependency>
 <groupId>javax.xml.bind</groupId>
 <artifactId>jaxb-api</artifactId>
 <version>2.1</version>
</dependency>
<dependency>
 <groupId>com.sun.xml.bind</groupId>
 <artifactId>jaxb-impl</artifactId>
 <version>2.1</version>
</dependency>

Code:

public final String MESSAGE_URL = "http://test.com/messageservice/send";

public boolean sendMessage(Message message) {
 
       URL url = null;
       try {
           url = new URL(SS_URL);
       } catch (MalformedURLException e1) {
           e1.printStackTrace();
       }

       try {

           System.out.println("Start converting message to XML.");

           // Marshall
       
           JAXBContext jaxbContext = JAXBContext.newInstance(Message.class);
           Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
 
           // output pretty printed
           jaxbMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUTtrue);
 
           ByteOutputStream out = new ByteOutputStream();
           jaxbMarshaller.marshal(message, out);

           String msg = new String(out.getBytes(),"utf-8");

           System.out.println("Finish converting message to XML.");

           System.out.println(msg);
         
           HttpURLConnection connect = (HttpURLConnection) url.openConnection();
           connect.setRequestMethod("POST");
           connect.setDoOutput(true);
           connect.setRequestProperty("Content-Type""text/xml");
           connect.setAllowUserInteraction(false);
         
           // send query

           OutputStream os = connect.getOutputStream();
           jaxbMarshaller.marshal(message, os);
           os.flush();
           os.close();

           if (connect.getResponseCode() != 200) {
               System.out.println("Message sent failed.");
               return false;
           }

           System.out.println("Message sent.");

           System.out.println("Result:");
         
           // Unmarshall
       
             Unmarshaller jaxbUnmarshaller = jaxbContext.createUnmarshaller();
           Message result = (N2NMessage) jaxbUnmarshaller.unmarshal(connect.getInputStream());
           System.out.println("Sent Message Id: " + result.getMessageID());
         
         
           return true;
         
       } catch (Exception e) {
           e.printStackTrace();
           return false;
       }
}