There are way too many job listings sites out there today. I’ve come across at least 40 of them personally already and I’m sure there are many, many more. Trying to track postings across all these sites can drive you a bit batty.

The job aggregator sites usually have a way to get email alerts for new job postings sent to you.  You have to make a tough choice though:  search for very specific terms and risk missing out the right job named differently OR be over-inclusive and weed through tons of rocks to find a diamond listing.   Both answers are kind of terrible.  Worse yet, you get one email per site (sometimes per site per search!) so watching 20+ sites means 20+ emails every day with new listings.

I needed something that would do this for me:

I didn’t find anything off the shelf that could do what I needed.  After suffering for a little while through the multi-email blasts each day, I quickly had the strong urge to just fix it.  So I did.

Job Scooper Is Born

Job Scooper downloads the recent job postings from all the sites and for all the searches that I want.  It bundles them up into a single list, removes duplicate posts and then fires off one email each day with the resulting new job list.  So much better!

Just Download, Set Your Searches and Go.

Since this ended up being a lot of fun, but also a lot of code, I thought I should make it as easy as possible to let anyone use it for their own searches.  That way other folks can benefit from the work for their own searches.

I’ve released Job Scooper on Github for anyone to grab and use:jobs_scooper_github

Even if you aren’t a developer, you can download the package to your Mac, set your search parameters in a configuration file and run it.  That really should be all you need to do.

More detailed instructions and examples are available Github that walk through the different features of Job Scooper and how to set them up for your searches.


10,000 Job Listings and Counting

Jobs-to-XLSWith Job Scooper running, I’ve been able to quickly go through posts from over 20+ sites each day.  It’s already processed over 10,000 job postings for me so far.  I quickly found I needed to add a couple other features to make that large of list manageable.

Some sites like Craigslist post the same job with a different ID and no company name.  That meant the same job kept popping up as a new post even though it really was the same one I’d looked at 10 times already.  So one of the first things I added was duplicate post detection.  It’s smart enough to detect and filter out the same job being posted to not only the same site multiple times, as with Craigslist, but also across sites as well.  For aggregator sites like Indeed and SimplyHired, that helped to reduce the noise in the result list big time.

I’ve added a few other features for auto-filtering results to remove certain job titles or companies I wasn’t interested in.  Using these more power features helped to keep my new jobs list to under a dozen or two new listings each day, instead of the hundreds it started at!


20+ Sites Supported So Far; Add a New One Quickly & Easily

I found I was often adding sites for companies I was following in addition to the job search sites, so I needed a way to do that quickly.  I added a plugin model that lets me add a new jobs site in just a few hours.  Any site can be added regardless of whether the jobs are downloaded server-side or client-side through AJAX.

Right now there’s about 20ish sites supported such as CareerBuilder, Craigslist, Facebook, Glassdoor, Indeed, LinkUp, Monster, etc.  Anyone who knows a bit of PHP and HTML should be able to add a new site to the list in under a day as well.  I’d love to see some others get added!

Give me a shout if you need help adding one or have any issues getting Job Scooper up and running!

Note:  I’ve only really tested it on Mac OS/X 10.9.3 with PHP 5.4.24. Your mileage will probably vary on other versions.  Mac OSX is currently required to support client-side job listing downloads; it leverages Safari through Applescript to make those happen. 



About the Author -

Hi, I'm Bryan. Thanks for dropping in my little corner of the Internet. I am currently Head of Consumer Products at PayScale, responsible for the design of the company's consumer-facing web and mobile experiences. Before that, I was Redfin's first (and only) Vice President of Products, responsible for the design of Redfin's website, mobile applications, agent tools and information technology systems. Prior to joining Redfin, I worked for nearly ten years at Microsoft as both an engineering manager and a lead program manager. He was responsible for integrating major partners' services with Microsoft's mobile technologies and, prior to that, building I graduated from the University of Wisconsin-Madison (go Badgers!) with a degree in computer science and communication arts (tv, radio & film.)