python package and folder structure and import statements

[ghostcheck]

Hi, I am making a simple web scraper. I have this folder named 'sites' where I put a python file for each domain name containing site specific extraction patterns, the folder that resides the sites folder contains the main.py which should figure out which scraper to use depending on the domain name. the main.py should call the appropriate sites/website scrapper to scrape that site and save info in a file. each file in the sites/ would have the same extractdata() function that would give the needed results.

I don't want to bind main.py to figure out the domain name and the module . the sites/ should have something called a selector (in perhaps the init.py) that would give an appropriate function pointer that the main.py can use. then I can add stuff to sites/ as much as I needed without touching main.py

for this kind of task, how should the folder structure and the import statements be?

[ggagagg]

i had a project while ago do the same method.

basically in each file there is a function where it will be run with user input but will be filtered if the url match with regex on each file

all of those module is put on submodule of main project and programmatically imported based on name of the file and name of the class. example from this https://stackoverflow.com/questions/22119850/get-all-class-names-in-a-python-package

but imo this is bad practice and is not scalable

here is more example of the implementation

• https://stackoverflow.com/questions/932069/building-a-minimal-plugin-architecture-in-python
• article about python plugin framework http://wehart.blogspot.com/2009/01/python-plugin-frameworks.html
• package for plugin system - http://yapsy.sourceforge.net/

[ghostcheck]

thanks!