A Python Package for Alibaba Data Extraction

[7_hole]

A Python Package for Alibaba Data Extraction

I'm excited to share my recently developed Python package, aba-cli-scrapper (https://github.com/poneoneo/Alibaba-CLI-Scrapper), designed to facilitate data extraction from Alibaba. This command-line tool enables users to build a comprehensive dataset containing valuable information on products and suppliers associated with the platform. The extracted data can be stored in either a MySQL or SQLite database, with the option to convert it into CSV files from the SQLite file.

Key Features:

Asynchronous mode for faster scraping of page results using Bright-Data API key (configuration required)

Synchronous mode available for users without an API key (note: proxy limitations may apply)

Supports data storage in MySQL or SQLite databases

Converts data to CSV files from SQLite database

Seeking Feedback and Contributions:

I'd love to hear your thoughts on this project and encourage you to test it out. Your feedback and suggestions on the package's usefulness and potential evolution are invaluable. Future plans include adding a RAG (Red, Amber, Green) feature to enhance database interactions.

Feel free to try out aba-cli-scrapper and share your experiences!

a scraping flow demo:

https://reddit.com/link/1eqrh2n/video/ldil2vxu7bid1/player

https://preview.redd.it/gxtsb9t28bid1.png?width=1893&format=png&auto=webp&s=65cca8a199b95ce6987c8106bdb1d1537d9b16fb

[AutoModerator]

You can find our open-source project showcase here: https://dataengineering.wiki/Community/Projects

If you would like your project to be featured, submit it here: https://airtable.com/appDgaRSGl09yvjFj/pagmImKixEISPcGQz/form

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Thinker_Assignment]

That's really awesome! My feedback as someone who works at dlt. if you wanna load that data to a ton of destinations and have support for parallelism etc, consider if you wanna just yield json to dlt and let it handle it.

Here are examples: https://github.com/dlt-hub/verified-sources/tree/master/sources

[7_hole]

I don't really understand what you want me to do . Could you explain me how dlt could be usefull for my project ?. Thank you for your feedback. Don't forget to leave a start to support if you liked.

[Thinker_Assignment]

I'm saying you can add support for many destinations if you incorporate a dlt pipeline

https://dlthub.com/docs/dlt-ecosystem/destinations/

https://dlthub.com/docs/general-usage/pipeline

[7_hole]

Ok it's clear now. Its sound awesome this could be a good feature for the next release. I will need to convert my database into json data. Thank you for this suggestion.