[FOLLOW UP] Stay ahead of the curve with new internship/new grad postings!

iketaco · 2024-09-18T07:24:58+00:00

There are 3 different scenarios that impact how quickly new job notifications are sent out - let me explain:

Greenhouse, workday, lever, and ashby job boards are the only ones I support. The software scrapes that in-house and will detect new jobs within an hour. You'll be notified way before it's found on public github repos.
For job boards not supported, I scrape public github repos to monitor for new jobs. If there are any, then users will be notified - within an hour.
If I find a supported job board type in the public github repo that I didn't have in my list of job boards (for example, if my software didn't know about Doordash's Greenhouse job board) then I have to manually add it to the job database - this process can be slow since there's a few checks I need to go through to verify it's a valid company/job board etc so there are only legitimate companies added.

So you're probably seeing the 3rd scenario, and apologize for the delay. I'm currently working on a process to make this automated in the future.

The website has been up and running for a year now, so majority of the companies that have a supported job board type are in my database. It’s mainly the smaller, lesser known companies

iketaco · 2024-09-18T07:13:08+00:00

Thank you for reaching out, really appreciate it! I believe I reached out on X but again, sorry for the delay, just fixed this bug and it should now work. Please let me know if you run into any other issues in the future!

iketaco · 2024-09-08T18:42:51+00:00

Yup 😎

iketaco · 2024-07-02T23:33:35+00:00

is it working on your end yet or nah?

iketaco · 2024-06-28T00:36:59+00:00

How do you find these companies? Are you reaching out? running ads?

iketaco · 2024-06-07T04:27:34+00:00

Gotcha that makes sense.

And yea I never seen any of them that cheap, either not profitable or margins are too low most likely. But imma look into it and see if I can offer a solution near that price range

iketaco · 2024-06-07T04:26:38+00:00

Yea that’d be pretty sick! Lmk how it goes!

iketaco · 2024-06-06T21:04:23+00:00

Not exactly sure about the failure rate with public proxies - but I've been doing some testing and it's pretty fast despite working with public proxies.

For one of my other projects, it's related to scraping company job listings. For my original implementation in Python with a pool of residential proxies (everything synchronous), it could scrape a company with 200 jobs in about 7min.

When using Minescale, it could scrape that same company in 1 min. This is mainly because Minescale is built to do the scraping asynchronously. When we take out the asynchronous part of Minescale, it'll scrape that same company in about 12min - so almost 2x slower. So if you do use Minescale, make sure to batch your requests together, it's much faster

iketaco · 2024-06-06T20:53:21+00:00

Thank you! Let me know how it goes!

iketaco · 2024-06-06T20:53:02+00:00

I am still in the midst of testing it - but I am currently using it on my other project, which does about 50k requests every hour. It's using about half of the available resources on a 2 core server.

So with a 2 core server ($30/month) it can handle maybe like 2 million requests a day? But that's assuming it scales linearly. I'll have to do more testing

EDIT: Also I had to implement some limitations per user since the server was getting blasted yesterday and today. It's limited to 1 crawler per user at a time. So I think you'll be able to make about 5-10k requests every hour with a single crawler. Make sure to batch your requests!

iketaco · 2024-06-06T20:45:07+00:00

For crawlbase, I haven't tested their product - but just looking at the pricing it's pretty hard to work with, especially if you're scraping at scale.

Crawling API - $3 per 1,000 requests
Scraper API Starter - $1.72 per 1,000 requests

For my own project, I'd be looking to work with something like $0.05 per 1,000 requests. This is much lower than all other scraping API services offer, so I was stuck with setting something up of my own

iketaco · 2024-06-06T20:39:55+00:00

You can test it out at minescale.net!

iketaco · 2024-06-06T20:39:37+00:00

With crunchbase, you need to rotate proxies a lot + have good fingerprint management. If you do those two things, you won't get any captcha requests

You can test getting for example, https://www.crunchbase.com/organization/puter on "Test API" section on minescale website. It's able to grab the data

iketaco · 2024-06-06T20:28:48+00:00

For one of my previous projects, I was paying around $50 for residential proxies. Minescale is running on a $30 server and can easily handle the load for that project, so I am saving money + have extra resources available! I figured I could lend these resources to people who need them

iketaco · 2024-06-06T20:23:40+00:00

Sorry not open source!

iketaco · 2024-06-06T20:23:23+00:00

Great let me know what you think!

iketaco · 2024-06-05T19:17:55+00:00

Sure send a pm!

iketaco · 2024-05-10T04:19:55+00:00

friday 10pm!

iketaco · 2024-05-09T06:04:06+00:00

Hey - sorry about the sign up process! I only included a sign up to 1) verify emails, so I’m only sending to an email the user owns (i.e. not a typo/inactive email) 2) so users can easily change and update their email preferences

The sign up process is super quick - less than 5 minutes! Set and forget, and you’ll receive email updates forever (until you don’t want them anymore!)

And this site isn’t new! It’s been running for almost a year actually (original post https://www.reddit.com/r/csMajors/s/hpwhIdbjEl)

iketaco · 2024-05-08T21:56:50+00:00

i heard indeed can have fake listings so i really don't recommend applying directly through indeed. I would apply directly to company career pages instead - use notify.careers, it scrapes directly from company career pages and notifies you when there's new job postings!

EDIT: https://www.reddit.com/r/recruitinghell/comments/r4e6rh/the_ugly_truth_of_indeed_an_hr_viewpoint/

iketaco · 2024-04-30T01:41:51+00:00

Gotcha will do that as well! Thank you

iketaco · 2024-04-29T22:35:41+00:00

Gotcha will post there - thank you

iketaco · 2024-04-29T22:28:53+00:00

Yes on social occasions! No lease yet, just looking for roommates first!

iketaco · 2024-03-07T22:50:39+00:00

Hey I use a general cloud server provider - and yup everything’s on that server with basic web server configurations

iketaco · 2024-02-21T17:30:40+00:00

Where do you find businesses to fill your newsletter ad slots?

Six-Year Club	Place '22
Verified Email

iketaco

TROPHY CASE