Since Javascript is completely visible to the client, it cannot be used to prevent a scraper from following links. But it can make life difficult, and until someone writes a Javascript interpreter for Perl or a Mechanize clone to control Firefox, there will be no general solution. But if you want to scrape specific pages, then a solution is always possible.

One typical use of Javascript is to perform argument checking before posting to the server. The URL you want is probably just buried in the Javascript function. Do a regular expression match on $mech->content() to find the link that you want and $mech->get it directly (this assumes that you know what you are looking for in advance).

In more difficult cases, the Javascript is used for URL mangling to satisfy the needs of some middleware. In this case you need to figure out what the Javascript is doing (why are these URLs always really long?). There is probably some function with one or more arguments which calculates the new URL. Step one: using your favorite browser, get the before and after URLs and save them to files. Edit each file, converting the the argument separators (‘?’, ‘&’ or ‘;’) into newlines. Now it is easy to use diff or comm to find out what Javascript did to the URL. Step 2 – find the function call which created the URL – you will need to parse and interpret its argument list. Using the Javascript Debugger Extension for Firefox may help with the analysis. At this point, it is fairly trivial to write your own function which emulates the Javascript for the pages you want to process.

Here’s annother approach that answers the question, “It works in Firefox, but why not Mech?” Everything the web server knows about the client is present in the HTTP request. If two requests are identical, the results should be identical. So the real question is “What is different between the mech request and the Firefox request?”

The Firefox extension “Tamper Data” is an effective tool for examining the headers of the requests to the server. Compare that with what LWP is sending. Once the two are identical, the action of the server should be the same as well.

I say “should”, because this is an oversimplification – some values are naturally unique, e.g. a SessionID, but if a SessionID is present, that is probably sufficient, even though the value will be different between the LWP request and the Firefox request. The server could use the session to store information which is troublesome, but that’s not the first place to look (and highly unlikely to be relevant when you are requesting the login page of your site).

Generally the problem is to be found in missing or incorrect POSTDATA arguments, Cookies, User-Agents, Accepts, etc. If you are using mech, then redirects and cookies should not be a problem, but are listed here for completeness. If you are missing headers, $mech->add_header can be used to add the headers that you need.

LINK

“Jared” he started “I was speaking with my wife-“ stroking his thick beard recalling as he went. “-and you know how you told me your annual wage? Is that just your wage?”

I nodded.

“Well, why don’t you go and negotiate yourself into a home? I know you’re smart. But you can’t put your life on hold just to get the lowest home price.” Our conversation moved itself forward from there. But it got me thinking.

Am I putting my life on hold?

And being that I like to think and self reflect, I spent the next two days doing so. And here is what I came up with.

My friends purchased a home in 2006 for $400k. Homes currently sell fro $190-$220k in the area. And the prices are still falling. This couple would love to start having children but they literally cannot afford them. They need both incomes and the hit they would take from paying for childcare would tip their sinking ship. Not to mention his hours were recently cut.

That’s putting your life on hold.

A single guy I know, purchased a home for $300k. After he purchased the home, his pay was significantly cut. He is extremely worried about layoffs. So being proactive, he found a better job near Seattle but doesn’t know if he should take it. Being unable to sell his home.

That’s putting your life on hold.

Another family was struggling to make ends meet. They had heard that I had written a program to collect the County’s data from the county website. They had me over for dinner and I showed them the data I had collected and the trends that the data was showing. The next day they dropped the selling price on their rental by $50k, and miraculously were able to sell the home. A month later Lehman brothers collapsed. After that, the man lost his job. He ended up finding one but they had to move for it and the unfortunate thing is that they were unable to sell their private residence. He currently is trying to rent it but cannot find anyone to cover the mortgage. So the family goes with state funded health care for their children, and no health care for themselves, while dropping $1000 bucks a month on a home they don’t use.

That’s putting your life on hold.

A couple moved here in 2005. They wanted to buy a home, but asked why my wife and I were waiting. I went on to show them the data I had collected and encouraged them to wait things out. She recently lost her job with the school district up North. And his job was on the ropes. Being a teacher but unable to find work, guess what they did. They moved to freaking China.

That’s putting your life on hold.

One of my wife’s childhood friends noticed that prices in their area had declined by 50%. So in April of this year, what do her and her husband do? They bought into a home. The day after they sign, her husband loses his job. This month they have missed their first payment and are worried that they will have to move back into her parent’s house.

That’s putting your life on hold.

My wife and I look around us and feel incredibly blessed.  In October we will hopefully have a beautiful baby girl come bouncing into our world. Things have actually have been very busy in preparation for the baby. There are baby showers, shopping, parenting classes, and more showers. Putting together the crib and picking out baby names. The anticipation grows from these events, making me long to meet my little girl.

And this is when I realize that our life is moving too fast for anything or anyone to slow it down. And should I never own a home so that we can afford my daughter the best possible upbringing; or that my family won’t be debt burdened and have to put their lives or their dreams on hold; well then, I am totally cool with that.

This video is quite timely for me. For I have thought long and hard about putting together a series of posts on data analysis. Particularly on the work I have done in looking at the Deschutes County Clerks Data.

For those of you who don’t know. In the fall of 2007 I created a program that went to the county website and would pull public records and sales data concerning housing. It would throw this data into a usable database.  And I would use this data to educate family and friends on when our housing bubble was started, and how bad it really was.

I built a small website with this data, but stopped my work on it because of several complex reasons.

Anyway, if people are interested in how to analyze data, or would appreciate an ongoing conversation about it, let me know.