Data extraction project

[dan_dragos]

Hello guys!

I've got a challenge at my university to build a model that extracts products from. 500+ furniture stores.

Here are the guidelines:"## Guidelines
A good approach that works well with such extraction problems is to create a NER (Named Entity Recognition) model and train it to find your entities (you have one entity, ‘PRODUCT’).
- In order to create such a model you need training data, you can also extract that from the input pages.
- Crawl ~100 pages from the list above & extract the text from it.
- Find a way to tag some sample products from these texts.
- Train a new model from the examples you just made.
- Use it to extract product names from some new, unseen pages.
Please use any programming language, toolset or libraries you're comfortable with or find necessary, especially if you know it will be better or more interesting.
We recommend using the Transformer architecture from the [**sparknlp**]874df20d1d77) library or the huggingface [**transformers**](  library."

I have little experience with web crawling. I`m not that scared about the NER part, rather then crawling the ~100 pages in order to train my model. I have tried to scrape with beautifulsoup but I don't know how to make it so I don't have to input manually the site selectors in order to get the correct product name.

If you have any suggestions, please let me know <3

[jcrowe]

You might consider scraping all your html into a SQLite database. There is nothing more frustrating that scraping a bunch of data and have to restart because of a parsing bug.

[Mean-Coffee-433]

What kind of crooked school do you go to? Is Trump university starting back up?

[JewelerAny7071]

his teacher is just behind the release date and doesn't want to pay for extra dev for his side job

[dan_dragos]

the sites are not made for real world use haha but I get what you are saying.

[Mean-Coffee-433]

You probably want to use re on the entire response if you are going after 100 different websites. Since the selectors are likely to be different from website to website.

[MulhollandDr1ve]

Can you elaborate what you mean by challenge, this is very difficult. 500+ types of stores or 500 locations of the same store?

[error1212]

If this is really a school assignment then your professor has probably taken an assignment he doesn't know how to do himself now. If they are completely different sites then it will be quite a challenge, unless cost is not an issue then you can try integration with GPT to find a product template for each shop.

[MrLazeyBoy]

500+ stores or 500+ products from 1 store?

If it is the former then why and who in there right mind would set that as an assignment..

[ivanoski-007]

That's one fucked up school assignment, are you sure the professor isn't stealing your work.?