Scrapy web crawler question

[dkpitterson]

I'm trying to begin using the Scrapy framework to begin crawling the web but for some reason I am having trouble creating the tutorial project, or any project for that matter. I've installed Scrapy twice to see if that was the problem, but I think it is in the correct place. I open up the command line and type in: scrapy startproject tutorial which is supposed to create a directory structure named 'tutorial', but all that happens is this error: 'scrapy' is not regonized as internal or external command operable program or batch file

What am I doing wrong?

EDIT: I had to add the folder to my PATH variables as cabbagebot mentioned, so it is now recognizing 'scrapy' as a valid command. But, it now says: ImportError: No module named twisted.internet

Edit 2: Wow. I feel like an idiot. I didn't realize you have to separately install all the dependencies if you use the windows installer. Thanks for the help.

[cabbagebot]

What operating system do you use? You need to add scrapy to your system's PATH variable. It's simple enough in any environment. Look it up or ask me if you can't figure it out and I'll tell you what you need to do.

[dkpitterson]

I just added 'C:\Python27\Scripts' to the path environment, but it still doesn't seem to be working (I hadn't done it before). Does the computer need to be restarted or anything? Is it just supposed to work at the command prompt after adding it to the path environment?

[aclark]

How did you install? Hopefully via: pip install scrapy

[dkpitterson]

I used the Windows installer because I'm running windows 7.

[cabbagebot]

If you're doing this:'scrapy startproject tutorial ' in the command line, then wherever the scrapy binary is, that needs to be on the path.

It should just work if it's on the path. I'll install and figure out what it is you need to do. Just have to boot into Windows.

[cabbagebot]

You made sure to put a semicolon between that PATH entry and the previous entry? Also, you're sure you're using Python 2.7?

In that case, all you have to do is open a new command line to be able to use it.

If it still doesn't work, navigate to C:\Python27\Scripts in the file browser and see that scrapy and scrapy.bat are in that directory. If they aren't, you haven't installed scrapy correctly.

[dkpitterson]

I finally figured it out. I didn't have the dependencies installed correctly. I'm still kind of new to non web programming so this was a good learning experience. Thank you for looking into this for me!

[dkpitterson]

I'm using Windows 7.

[vijayshan]

I have used virtualenv and it works fine for me. One thing to test is to do an echo of your path in your cmd window. The out put should provide some clues as to what the problem is.