When I woke, I was excited to see what little bits of interesting information my application had found. But instead, I saw that it had dumped stack. And as a Ruby newb, here starts the long and arduous journey to understanding how to profile a ruby application. Yikes!

Definition of “Memory Leak” in Ruby

From what I have researched and my own personal experience. You (the developer) can’t really create a traditional memory leak.

Traditional Leak

1. Allocate Memory (malloc, alloc)
2. Assign Value (Object, string, int, etc…) To Memory
3. No longer needing the value, you forget to release (dealloc) it
4. The data stays in memory during the life of your application
5. Particularly problematic when iterating or looping and assigning new values to memory

Ruby Memory Leak

1. You assign a value to a variable
2. You expect that the value has fallen out of scope
3. Instead, it persists in memory as some Object you are unaware of is retaining it
4. Not really noticeable in small applications or page requests where memory is freed at the end of the run cycle

Thats the basic difference (from my understanding). Please correct me if I am off on something.

Tips To Identify That You Have A Memory Leak

So a couple quick ways that you can tell you could have a memory leak.

Total Object Count

Try printing the total Object count to screen while your application is running. Should the count increase over time, this would indicate that you are retaining objects. Please note that you need to force Garbage Collection regularly to make sure that the total Object count is not just the logical growth cycle of your application.

Monitor Memory Usage

Monitor the Process I.D. (PID) of your application and look to see if memory usage increases over a given period of time. Should it continue to go up, this would strengthen the argument that you have a leak.

Again, these are two quick methods I used to come to the following point of action.

“Crap, it looks like something is going on with memory usage, I better take a closer look.”

Try forcing your Objects to release

I learned rather quickly, that by design, Ruby does not allow you to release or unset your Objects manually. So there ya go.

Set your Objects to nil to identify they are ready to be Garbage Collected

I tried this while grasping at straws, it was a dumb idea. I was taking shots in the dark and I wouldn’t advise going this route. What I needed was a really deep look at what Objects I was storing in memory so I could develop a story on why my application was leaking.

Memprof & its visualization partner Memprof.com

This is when I happened across the git repo Memprof by Joe Damato (@joedamato http://timetobleed.com). It was exactly what I needed.

Using Ruby 1.8.7, Install Memprof

$ gem install memprof

Signup for memprof.com

memprof.com (be aware, that when using this tool, it publishes your code to the web)

Include Memprof (at the VERY beginning of your application, just in case you want to profile your included gems too)

require `gem which memprof/signal`.chomp

Wrap Run Loop in Memprof.trace

def run_loop
  while next_link?(@browser.page) do
    Memprof.trace{
      iterate_search_page(@browser.page)
      click_next_link(@browser.page)
    }
  end
end

Run Your Application

$ ruby web_crawler.rb

Find Your Process ID of application

$ ps -ef | grep web_crawler.rb

After 5 minutes, Dump Trace To Memprof.com

$ memprof --pid 27890 --name elvis_dump2 --key YOUR_API_KEY

Analyze the heap visually with Memprof.com

Click On (objects with most outbound references)

On the far right, you will see an Object with the “Mechanize” class, and its length is 1081. This piece of information was critical in identifying that a Mechanize object I had instantiated was retaining a crap pile of objects.

What was the issue?

So it turns out that by default, Mechanize does NOT set a max_history=() value. What does this mean? Basically, my web crawler was storing every visited page, in memory, in entirety. BEWARE!

Hat tip ( @bradhe, @tim_linquist, @joedamato, @elazar)

Note: My “Ruby Memory Leaks” definition is not entirely correct, but for the newbs out there reading this, I think trying to break it down further could potentially confuse you. If you want to understand Ruby memory allocation at some point, go watch this video by Joe Damato. (link)

Since Javascript is completely visible to the client, it cannot be used to prevent a scraper from following links. But it can make life difficult, and until someone writes a Javascript interpreter for Perl or a Mechanize clone to control Firefox, there will be no general solution. But if you want to scrape specific pages, then a solution is always possible.

One typical use of Javascript is to perform argument checking before posting to the server. The URL you want is probably just buried in the Javascript function. Do a regular expression match on $mech->content() to find the link that you want and $mech->get it directly (this assumes that you know what you are looking for in advance).

In more difficult cases, the Javascript is used for URL mangling to satisfy the needs of some middleware. In this case you need to figure out what the Javascript is doing (why are these URLs always really long?). There is probably some function with one or more arguments which calculates the new URL. Step one: using your favorite browser, get the before and after URLs and save them to files. Edit each file, converting the the argument separators (‘?’, ‘&’ or ‘;’) into newlines. Now it is easy to use diff or comm to find out what Javascript did to the URL. Step 2 – find the function call which created the URL – you will need to parse and interpret its argument list. Using the Javascript Debugger Extension for Firefox may help with the analysis. At this point, it is fairly trivial to write your own function which emulates the Javascript for the pages you want to process.

Here’s annother approach that answers the question, “It works in Firefox, but why not Mech?” Everything the web server knows about the client is present in the HTTP request. If two requests are identical, the results should be identical. So the real question is “What is different between the mech request and the Firefox request?”

The Firefox extension “Tamper Data” is an effective tool for examining the headers of the requests to the server. Compare that with what LWP is sending. Once the two are identical, the action of the server should be the same as well.

I say “should”, because this is an oversimplification – some values are naturally unique, e.g. a SessionID, but if a SessionID is present, that is probably sufficient, even though the value will be different between the LWP request and the Firefox request. The server could use the session to store information which is troublesome, but that’s not the first place to look (and highly unlikely to be relevant when you are requesting the login page of your site).

Generally the problem is to be found in missing or incorrect POSTDATA arguments, Cookies, User-Agents, Accepts, etc. If you are using mech, then redirects and cookies should not be a problem, but are listed here for completeness. If you are missing headers, $mech->add_header can be used to add the headers that you need.

LINK