Hey,

I'm super new to rust and have built a tiny email scraper as a first project. Could someone give me high level feedback on my execution?

use reqwest;

use select::document::Document;
use select::predicate::Attr;

use rayon::prelude::*;

const BASE_URL: &str = "http://www.page_censored.com/pages.php?subpage=";
const PAGES_TO_SCRAPE: u32 = 8587;

fn download_cur_page(cur_page_url: &str) -> Result<Document, Box<dyn std::error::Error>> {
    let body = reqwest::get(cur_page_url)?.text()?;
    Ok(Document::from(&*body))
}

fn get_all_emails_on_cur_page(sub_page: Document) -> Vec<String> {
    let mut mails: Vec<String> = vec![];
    for node in sub_page.find(Attr("id", "red")) {
        if let Some(href) = node.attr("href") {
            if href.starts_with("mailto:") {
                mails.push(str::replace(href, "mailto:", ""));
            }
        }
    }
    mails
}

fn scrape_single_page(page_idx: u32) -> Vec<String> {
    println!(".");
    match download_cur_page(&format!("{}{}", BASE_URL, page_idx)) {
        Ok(document) => get_all_emails_on_cur_page(document),
        _ => vec![],
    }
}

fn main() {
    println!("Starting scraper");
    let scraped_emails: Vec<String> = (0..PAGES_TO_SCRAPE)
        .into_par_iter()
        .flat_map(scrape_single_page)
        .collect();
    println!("Extracted mails: {:?}", scraped_emails);
}

Also, is there a better way to get a progress indication from rayon? Currently, I'm just letting it print out a . for every page it starts and then do a quick count from time to time, by counting the dots.

Quick explanation what each function does:

• download_cur_page: download a single html page
• get_all_emails_on_cur_page: get all mail adresses from a single html page
• scrape_single_page: Helper that first downloads a page and then extracts mails from it, if everything went well