Monit: The Cure for Crappy Software.

I currently manage more than 100 instances of Apache.  Mostly LAMP with some Ruby and Passenger thrown in here and there.  Of all these instances, I’ve never really had to concern myself with leaky apps that soak up memory and don’t give it back until restart.  The first  time I ran into this was with Mongrel, what a terrible app.  At any rate, the cure for Mongrel was Monit.  It’s been at least 3 years since I’ve installed and used Monit, so I thought I’d write it up this time for my own reference and to maybe help somebody else.  I believe the culprit this time has something to do with MSSQL.  I knew I felt dirty when I first installed UNIXodbc and FreeTDS with the intent to allow PHP to connect to an MSSQL server.  Now I think I have my first piece of evidence that I have tainted the stack with the Microsoft evil.  These boxes were fine for more than a year until we introduced the MS evil last week.  Today, one of the boxes started randomly dumping curl calls to an upstream API.  One mind you, not both.  So an hour into testing gateways, and running stream checks in loops with Nagios monitors I thought, what the hell, I’ll HUP apache and pray that it doesn’t fix anything.  Sure enough, it did.  I didn’t check the memory usage before hand, as nothing said “You’re out of RAM” but I started watching it afterward and saw it trickling away.  So the short term solution is to use Monit to restart these Apache instances when the memory usage by Apache gets to high.  So lets get on with it.

You can download the Monit source from here

I chose to take the sissy route with my CentOS 5.5 loads and add a repo where I knew a good package lives.  So I ran this:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-5.noarch.rpm

Once you load up that repo, just run a yum to install Monit:

yum -y install monit

Monit will install an init script, and a monit.conf file that is packed full of instructions for getting started. Some quick config options to start with:

set daemon 120 # Poll at 2 minute intervals
set mailserver mail.yourserver.com #ensure you get alerts when events occur
set mail-format { from: monit@foo.bar } # Change the from user from root (gmail)
set alert sysadm@foo.bar # set the email for all alerts
set httpd port 2812 and allow admin:monit # Setup the Admin web interface with user and pass

Next we need to configure a service. First I want to check Apache memory usage after a fresh restart to get a baseline. Here’s a handy little one liner to do that:

ps -o rss -C httpd | tail -n +2 | (sed ‘s/^/x+=/’; echo x) | bc

In my case, I’m using about 130MB. I’m going to set my threshold at 150MB. There will be plenty of time after setup to tweak these numbers. Next we’ll configure a basic process in Monit that will tell it to restart Apache if we cross the 150MB mark. Here’s what it will look like:

check process apache with pidfile /usr/local/apache/logs/httpd.pid
start program = “/etc/init.d/httpd start”
stop program = “/etc/init.d/httpd stop”
if totalmem > 150.0 MB for 5 cycles then restart

This is a very very simple configuration for Monit, it does much more, but this is where we’ll start. Basically I’ve given monit the pid file for my Apache process, my start and stop commands, and I’ve told it that if Apache’s total memory consumption is greater than 150MB for 10 minutes, that it needs a restart.

Once this is set, fire up Monit from init.d:

/etc/init.d/monit start

Once it’s up, you can start watching from the web interface: http://yourserverip:2812.
Login with the credentials you set in the configuration file. Mine looks like this:

(click to enlarge)

 

 

 

 

 

So you can see, Monit is doing exactly what we told it to do. I’m going to stop here for now, but again, there is a whole lot more you can do with this app. From CPU utilization per process, to overall load, to alerts on file permissions. It’s really more functional than I remembered it being. The sad thing is, I’m only using it right now to compensate for other issues, duct tape if you will. Oh well, tomorrow I’ll be looking at linux mssql driver solutions. MS Evil.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>