Configuring god to monitor Sphinx's searchd

God is a server monitoring tool helpful for starting your server processes and keeping them running, all the while making sure they don't misbehave. In the Rails world, it appears to be used most commonly with packs of mongrels or thins. However, this is not to say it can't be used to monitor other software. For one of my recent work projects, we've been using God to monitor the searchd process that Sphinx uses to serve results to search queries.

The configuration required for this can be based mostly on what you see around the place for thins or mongrels:

require 'yaml'

app_config = YAML.load(File.open(File.dirname(__FILE__) + "/config.yml"))['production']

God.watch do |w|
  w.group = "#{app_config['app_name']}-sphinx"
  w.name  = w.group + "-1"

  w.interval = 30.seconds

  w.uid = app_config['user']
  w.gid = app_config['group']

  w.start         = "searchd --config #{app_config['app_root']}/config/sphinx.conf"
  w.start_grace   = 10.seconds  
  w.stop          = "searchd --config #{app_config['app_root']}/config/sphinx.conf --stop"
  w.stop_grace    = 10.seconds  
  w.restart       = w.stop + " && " + w.start
  w.restart_grace = 15.seconds

  w.pid_file = File.join(app_config['app_root'], "tmp/pids/sphinx.pid")

  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval  = 5.seconds
      c.running   = false
    end
  end

  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = 100.megabytes
      c.times = [3, 5] # 3 out of 5 intervals
    end
  end

  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state      = [:start, :restart]
      c.times         = 5
      c.within        = 5.minutes
      c.transition    = :unmonitored
      c.retry_in      = 10.minutes
      c.retry_times   = 5
      c.retry_within  = 2.hours
    end
  end
end

There are a number of other changes required for this configuration to work properly. Firstly, you will require a config/config.yml file, containing something like the following:

development: &production_settings
  app_name: myapp
  user: deployer
  group: deployer
  app_root: "/home/deployer/deployments/myapp.amc.org.au/current"
test:
  << *production_settings
production:
  << *production_settings

This config file sets the app's short name, the user and group to run the searchd process, and the path to the rails application root. We keep these settings in a yaml file because they are also used in a number of other places besides here. If you don't have this requirement, you can put all of these settings directly into your god config file.

The other requirement is that we make Sphinx use a configuration file that has a predictable name. The default Sphinx config file, when using Pat Allen's awesome Thinking Sphinx plugin, has a file name contains the name of the current Rails environment. God can't use this to launch searchd, because it is does not run within the context of the Rails environment. To set location and name of the sphinx configuration file, set up a config/sphinx.yml file that contains something like this:

development: &production_settings
  config_file: /home/deployer/deployments/myapp.amc.org.au/current/config/sphinx.conf
  pid_file: /home/deployer/deployments/myapp.amc.org.au/current/tmp/pids/sphinx.pid
test:
  << *production_settings
production:
  << *production_settings

Thinking Sphinx respects these settings and uses them when it generates the Sphinx config file.

So that's about it. Make sure you've run rake thinking_sphinx:index at least once in your Rails app, to generate the Sphinx configuration and indexes, and then you are ready to start god and have your searchd automatically started and monitored!