CP's Programming Adventures

Random little things I've learned

Keeping Passenger Under Control

The mysterious “memory leak”

Recently at my workplace, we started to migrate one of our projects from Rails 2 to Rails 3.1. Everything seemed to go pretty well. Then one day, I started noticing my development machine crawling to a halt for no apparent reason. I could see that the memory was full and swapping. My initial reaction was that some sort of memory leak was introduced during the Rails 3 migration, as I also discovered that another colleague was experiencing the same issue. After going through a quick process of kill -9 elimination, the culprit turned out to be Passenger Standalone.

Checking Passenger’s memory profile

Some quick googling turned up the command passenger-memory-stats. The tool shows details regarding the Apache/Nginx processes’ memory profiles, as well as the Passenger processes’ memory profiles. I quickly realized that passenger uses 6 worker processes by default.

One thing that my team quickly realized during the Rails 3 migration was that Rails 3 has a much larger overhead in development mode than Rails 2. This quickly lead to a substantial amount of the dev machine’s memory being utilized when passenger was run and a few requests were made. In my case, I had also started using gnome-shell, which took up a good chunk of memory, and that compounded with 6 memory-intensive workers meant a machine running out of memory.

I ended up spending some time to find out documentations on how to configure the number of worker instances. Much to my surprise, I was not able to find any information on how to do this for standalone Passenger.

It was time to start digging into Passenger’s internal’s.

Discovering how to configure nginx for standalone Passenger

A quick grep for passenger_max_pool_size under the passenger gem root directory turned up this file: phusion_passenger/templates/standalone/config.erb. In the file, I quickly saw these two hopeful looking lines:

phusion_passenger/templates/standalone/config.erb
1
2
passenger_max_pool_size <%= @options[:max_pool_size] %>;
passenger_min_instances <%= @options[:min_instances] %>;

I proceeded to replace the two erb tags with 2 and 1 respectively. After restarting passenger and running passenger-memory-stats, I saw only two passenger workers. Success!!

Looking for a better solution

Modifying the nginx config template worked. However, modifying the gem directly was not an ideal solution. One option that I had was to fork the gem and modify the template, but that also did not seem like a clean solution for this problem. So I started digging a bit further into passenger. A grep for ‘config.erb’ turned up this line: lib/phusion_passenger/standalone/command.rb:179: template_filename = File.join(TEMPLATES_DIR, "standalone", "config.erb")

A quick scan through the file revealed this bit of code:

lib/phusion_passenger/standalone/command.rb
1
2
3
4
5
6
7
8
9
10
def parse_options!(command_name, description = nil)
# cut
global_config_file = File.join(ENV['HOME'], LOCAL_DIR, "standalone", "config")
if File.exist?(global_config_file)
require 'phusion_passenger/standalone/config_file' unless defined?(ConfigFile)
ConfigFile.new(:global_config, global_config_file).options
@options.merge!(global_options)
end
# cut
end

This showed that the template was configurable, by looking at ~/.passenger/standalone/config

A quick peek into the ConfigFile class showed this initializer:

lib/phusion_passenger/standalone/config_file.rb
1
2
3
4
5
6
7
8
9
def initialize(context, filename)
options = {}
@context = context
@filename = filename
File.open(filename, 'r') do |f|
f.flock(File::LOCK_SH)
instance_eval(f.read, filename)
end
end

It looks like Passenger reads in the local config file, and evals the content. Looking at the content of config_file.rb, it contains these two methods:

config_file.rb
1
2
3
4
5
6
7
8
def max_pool_size(number)
allowed_contexts(:max_pool_size, :global_config)
@options[:max_pool_size] = number
end
def min_instances(number)
@options[:min_instances] = number
end

Therefore, inside my local config file, I added these two lines:

~/.passenger/standalone/config
1
2
max_pool_size 2
min_instances 1

Running passenger-memory-stats after restarting Passenger showed that there were infact 2 workers up and running. Success for real!!

I definitely appreciated Passenger giving the option to configure nginx unintrusively, and it was also nice to realize that there were no memory leaks after all.

Comments