Load Balancing with PEN

How many times have you had a very expensive load balancing pair in front of your network and some service behind them that needed to use another load balanced service in the back end? Now you can always throw caution to the wind and allow the big boys up front to do the load balancing for your service and create that horrible path of data movement that nobody really wants. It will work, but it immediately doubles up the load on your front end and it just isn’t linear and pretty. If you’re one of the privileged few, you could have a second set of expensive load balancers in the private space of your network to use precisely for this purpose. If you’re not, let me introduce PEN.

As described by the creators:

This is pen, a load balancer for “simple” tcp based protocols such as http or smtp. It allows several servers to appear as one to the outside and automatically detects servers that are down and distributes clients among the available servers. This gives high availability and scalable performance.

PEN Site

So in the case above, we want our Web Servers to hit the SOLR servers in a round robin fashion as we don’t care if anything maintains state.  Just get the info and stick it in the site.  The big difference between PEN and some other Load Balancers you may be used to, is that it is just a single command line process.  Everything you need to start your pool happens in a single line.  You can download PEN here, or you can use the following repo on RHEL systems:

Once installed you need to identify the following variables to create your pool:

1.) Local source port

2.) Number of servers in your pool

3.) Destination ports on servers in your pool

4.) Details of your control host (this can be used to issue commands to PEN on the fly)

In the example above, we want to create a local port on our web servers that when hit, will request services from the 2 SOLR Servers.  Our command line will look like this:

We are telling pen that (-S) we have 2 servers, (-C) our control server is at port 10081 on the localhost, (-r) we want to perform round robin load balancing on local port 8983 and our 2 servers 10.1.1.10 and 10.1.1.11 will be accepting requests for this pool on port 8983.

I typically run these processes from startup with the script below. You can easily take this script and modify it to meet your needs. There are also parameters in the script to set the number of maximum concurrent connections, the log file to use and where to store the PID. Here’s the output from the PEN manpage with a full list of options.

startup script:

There’s also some cute tricks you can do with the control port. Some of the more useful ones are:

 
penctl manpage