6/18/2013

Apache HTTP Server Setup as Load Balancer

Environment:

Win 7 64 bits, Apache Http Server (httpd) 2.4.3, Apache Tomcat 7.0.29.

1.       Download Apache Http Server:
In this test case, I use the XAMPP(1.8.1) with Apache Server version 2.4.3.
2.       Setup the XAMPP:
Run %XAMPP_HOME%/setup_xampp.bat
3.       Configure the Apache Server:
=====“httpd.conf”=====
#Add below modules under the module list:
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so

#Uncomment below modules:

#Update the listen port:
#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 0.0.0.0:80
#Listen [::]:80
Listen 127.0.0.1:90

NOTE: Multiple ‘ip:port’ can be listed down below, but do not put port only. Also, do not put ‘0.0.0.0:port’. Otherwise, exception will be thrown:

“C:\xampp-win32-1.8.1-VC9\xampp>apache_start.bat
Diese Eingabeforderung nicht waehrend des Running beenden
Bitte erst bei einem gewollten Shutdown schliessen
Please close this command only for Shutdown
Apache 2 is starting ...
(OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted.  : AH00072: make_sock: could not bind to address 0.0.0.0:80
AH00451: no listening sockets available, shutting down
AH00015: Unable to open logs”

If want to check which port is in use by other process, type in cmd:
netstat -a –n

#Update the Server Name:
#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
ServerName localhost:90

=====“httpd-ssl.conf”=====
#
# When we also provide SSL we have to listen to the
# standard HTTP port (see above) and to the HTTPS port
#
# Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
#       Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443"
#
#Listen 0.0.0.0:443
#Listen [::]:443
Listen 127.0.0.1:443

=====“httpd-proxy.conf”=====
#
# Implements a proxy/gateway for Apache.
# # Required modules: mod_proxy, mod_proxy_http
#

<IfModule proxy_module>
<IfModule proxy_http_module>

#
# Reverse Proxy
#
ProxyRequests Off

<Proxy balancer://mycluster>
                  BalancerMember http://127.0.0.1:8080
                  BalancerMember http://127.0.0.1:9080
                  ProxySet lbmethod=byrequests
</Proxy>

#Optional  start
<Location /balancer-manager>
   SetHandler balancer-manager
</Location>
ProxyPass /balancer-manager !   #Not proxy balancer-manager
#Optional  end

ProxyPass / balancer://mycluster/
</IfModule>
</IfModule>

NOTEI: The ‘mycluster’ is the identifier of the load balancer, can pick any value as long as starts with ‘balancer://’

NOTEII: The ‘balancer-manager is one optional configuration, this is one web UI of the load balancer manager. You can choose to use it, or not. URL is http://localhost:90/balancer-manager

NOTEIII: IMPORTANT! ProxyPass /balancer-manager !   , this is to not proxy balancer-manager to tomcat.

NOTEIV: IMPORTANT! ProxyPass / balancer://mycluster/, this is to proxy url from base ‘/’, and the trailing slash is very important. If not have, you’ll have 500 http error code. If you check the error log, below warning will be shown:

“[Tue Jun 18 11:32:39.296798 2013] [proxy:warn] [pid 36488:tid 1732] [client 127.0.0.1:59079] AH01144: No protocol handler was valid for the URL /error/HTTP_BAD_GATEWAY.html.var. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.”

Configuration Summary
You can also group things together by using ‘VirtualHost’.

=====”httpd-vhosts.conf”=====

<VirtualHost *:90>
        ProxyRequests off
        ServerName localhost:90

        <Proxy balancer://mycluster>
                BalancerMember http://127.0.0.1:8080
                BalancerMember http://127.0.0.1:9080

                # Security "technically we aren't blocking
                # anyone but this the place to make those
                # chages
                Order Deny,Allow
                Deny from none
                Allow from all

                # Load Balancer Settings
                # We will be configuring a simple Round
                # Robin style load balancer.  This means
                # that all webheads take an equal share of
                # of the load.
                ProxySet lbmethod=byrequests
        </Proxy>

        # balancer-manager
        # This tool is built into the mod_proxy_balancer
        # module and will allow you to do some simple
        # modifications to the balanced group via a gui
        # web interface.
        <Location /balancer-manager>
                SetHandler balancer-manager

                # I recommend locking this one down to your
                # your office
                Order deny,allow
                Allow from all
        </Location>

        # Point of Balance
        # This setting will allow to explicitly name the
        # the location in the site that we want to be
        # balanced, in this example we will balance "/"
        # or everything in the site.
        ProxyPass /balancer-manager !
        ProxyPass / balancer://mycluster/
</VirtualHost>

4.       Start two Tomcat instances
5.       Start Apache Http server (MUST run as administrator)
Either run the bat file %XAMPP_HOME%/apache_start.bat
Or start XAMPP’s control panel and start the server.
If any error happens, the error log is a good place that could help debug.

6.       Test load balance
For example, I run the web app that test the central session.

Bingo. Happy ending.
This is a very very basic configuration of apache http server as load balancer. Main goal is to make it work. For further usage, please refer to the official web site.

6/07/2013

CentralCache Management(Memcache) Issue 1 -- net.spy.memcached.internal.CheckedOperationTimeoutException


One main aspect to evaluate one web app is its scalability, whether it can sustain heavy load.
So when we decide to build one web app, each technology that we plan to use must go through enough tests, not only functional test, but also load test. Otherwise, we may lose at the start point.

When I test the performance of tomcat-memcache central session management, I save&get small session data and large session data with both light load and heavy load.

When load is light, everything goes fine. But when load increase, and session data is around 10K, exception will be thrown:

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: localhost/127.0.0.1:11212
        at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:160)
        at de.javakaffee.web.msm.LockingStrategy.onAfterBackupSession(LockingStrategy.java:294)
        at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:1062)
        at de.javakaffee.web.msm.RequestTrackingHostValve.backupSession(RequestTrackingHostValve.java:243)
        at de.javakaffee.web.msm.RequestTrackingHostValve.invoke(RequestTrackingHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1770)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Regarding this issue, I found one discussion page:
https://code.google.com/p/spymemcached/issues/detail?id=136
But I did not see effect solution. Maybe switch to other memcache client lib will be a better solution.
Somebody says that increase the spymemcache's timeout may help, but I haven't got time to test it.
Compared to memcache, now I prefer mongodb as the backup of session. At least it's client lib is more stable and under enough test.

So If we need to setup the central cache for web app, we need to customize and test the lib well based on our requirement. I think no one wants to face this issue in production.