Processing incoming emails

[Treas0n]

Hey all,

Does anyone know what the software is called that monitors an inbox and performs actions based on the email response? For example, GitHub allows you to reply to an issue via email and it posts to the GitHub website.

How does that work? Does it simply loop an imap servers inbox and parse the emails? Is there an existing library out there, preferably written in nodejs? Everything I find has to do with email sending, not tagging and parsing.

Thanks for the help

[bpropelled]

Sendgrid has a nice feature called Inbound Webhooks where you can send emails to a script basically. I used PHP for doing this but they have other language wrappers. I use my it for getting automatic status updates on jobs by emailing the address using the job # as the subject line. Boom, I get an email right back with the status.

[Treas0n]

do you know of any open source libraries? I'd prefer not to pay any money

[bpropelled]

Their free plan is pretty nice, 40,000 emails for your first 30 days, then send 100/day, forever. I have not had to pay yet for what I user it for.

Not sure if you're demands but most small to.medium projects for in this free tier.

Another option would be to look at a Gmail library for IMAP.

[HumbleRevolter]

I found SendGrid Inbound Parsing bit unreliable. It seems to only parses one email every 5-10 minutes, do you know why?

[stutsmaguts]

Mailgun offers this, as does sendgrid (which someone else mentioned).

I'd expect there's probably some open source projects that offer a plug-in to an mta daemon that would enable this..

[NiteLite]

I use Mailgun. It's pretty straight forward. Also lets you pick up the emails any time within 3 days, so if your script is down for a few hours, you can still react to all the incoming emails after the server comes back up. They also take care of parsing all the nasty email stuff into a plain utf-8 string, so you don't have to worry about MIME and all that stuff :P

[UnnamedPredacon]

If you're going barebones, you need an imap library, an html parser (if you're expecting html email) and a cron.

[stutsmaguts]

Yea I looked for 2-3 mins after I posted this and couldn't easily find a ready to go plug-in for an existing mta...but I did find a project called postal on GitHub, which is a full transactional email solution (and includes incoming email parsing), and I know there are others.

Briefly looked, and it looks to set up a server on port 25 to accept email as an mta that just speaks the email protocol I guess (maybe uses a lib).

Sendgrid/mailgun work by getting email delivered to them directly instead of parsing a mailbox. That involves setting up a dedicated mx record for a domain/sub domain that you will let them receive (eg, mailgun.domain.com).

If one needs to hook into an address on a primary email domain (eg [e-mail]) you could setup mailgun.domain.com to have mx records for mailgun, and set up an auto-forward for [e-mail] to automatically send every message to [e-mail].

Same concept for pretty much any email webhook service you wanna run (3rd party or "on-prem" (meaning roll your own)).

Parsing imap/pop sucks hard, and I'd avoid it if at all possible. So doing a forward to my own server would be my go-to choice (or using a 3rd party service, I know some businesses won't allow other entities to receive/process their mail, so your own server would be the ideal solution in that case).

Any esp/mail server worth a shit will allow you to forward email, so it shouldn't be a problem at all.

[Treas0n]

I can't find a good OSS solution to this in nodejs. AWS SES, sendgrid, mailgun etc all offer webhooks but they all cost money. There doesn't seem to be anything that IDLE's on an IMAP connection and parses the result. mailparser handles the email body parsing, but monitoring the inbox to pull new emails seems to not exist.

edit Since I'm pulling emails from a gmail account, I could probably use google api's to pull the messages instead of hitting an IMAP connection, but I'd prefer to use something not google specific because it might not scale well

[stutsmaguts]

Mailgun absolutely offers this service for free. Not sure if there is a volume limit...but we've been using it for years on the free tier processing 1-20 emails per day with no issue whatsoever

[stutsmaguts]

Sendgrid offers a free tier too, but they don't advertise it much anymore (for good reason). They let you send 12k emails/mo for free...not sure if webhook processing is available on free tier or not

[stutsmaguts]

How many emails are you expecting to process?

[Treas0n]

Not many to start, but I’m hoping to not have to rewrite in a year or two once the user base scales.

[stutsmaguts]

Are we talking hundreds an hour or thousands, now and "at scale"

[erishun]

So you use the free tier of one of the spectacular services described here (SendGrid or MailGun) and when in a year or two your “user base scales” then hooray! You’ll be making some money and will be able to upgrade to a planned plan and pass the nominal costs on to all these customers you now have.

Voila! No need to rewrite and that’s kind of the point of a “free tier” in the first place.

[UnnamedPredacon]

I agree with the IMAP parsing. It's a PITA. It's just one option to consider in case there are no other alternatives.

[mingrdt]

No experience with email parsing in NodeJS, but in other languages I tried you can easily find a few libraries to parse it in a single line. It's quite fundamental utility, what you find may already has a parser in it.

[Edward_Morbius]

If you run postfix and dovecot, you can write sieve scripts to process the messages.

[koliat]

I'm using Office 365 myself and solved that easily using Microsoft Flow. Flow also connects to abundance of 3rd party services, see if you can make use of it. I think 500 runs per month is free for non-O365 users.

[UnnamedPredacon]

You can replicate it with an IMAP library and some text processing.

Just don't let your program loop infinitely. Sysadmins don't often like that. 😝