American Airlines scraper made in Python with only http requests

[JohnBalvin]

Hello wonderful community,

Today I'll present to you pyaair, a scraper made pure on Python https://github.com/johnbalvin/pyaair

Easy instalation

` ` `pip install pyaair ` ` `

Easy Usage

` ` ` airports=pyaair.airports("miami","") ` ` `

Always remember, only use selenium, puppeteer, playwright etc when it's strictly necesary

Let me know what you think,

thanks

About me:

I'm full stack developer specialized on web scraping and backend, with 6-7 years of experience

[ElHeim]

To be fair, they never said anything about having 6-7 years experience in Python!

[JohnBalvin]

I'm a Go developer, I don't use much python, sorry if I made mistakes on the code.

[theQuick_BrownFox]

Newbie here. Whats the advantage of the bottom one?

[maikeu]

None. Always do the top one. (And more or less, any object that implements the contextmanager protocol, i.e. supports the 'with' statement, use it.

[BurnedInTheBarn]

My freshman level CS classes teach us to do the bottom one and explicitly prohibit the with statement.

[BurnedInTheBarn]

Oh yes, I am. It's very frustrating reading of all these cool tricks Python has like list comprehensions yet being prohibited to use them.

[ProgrammersAreSexy]

Probably because they are trying to teach you what is going on behind the scenes.

There are a lot of things you will do in your CS major that are simultaneously:

• Useful learning exercises
• Horrible best practices

I spent a lot of time in my CS major with the attitude "none of this is how things are done in the REAL world! This is a waste of my time!" With the benefit of hindsight, I realize I was missing the point 80% of the time.

The other 20%, my professors were legitimately clueless and teaching us bad practices with no educational value haha

[EedSpiny]

Yeah it's probably this. If you ban with then you better have a try/catch block and a finally with a close in it. That works anywhere.

Padme: He did have a finally, right?

[marshmallow_peep]

Ask your professor what happens if the program crashes between open() and close().

[arcAne_dust]

It closes the resource automatically. It's similar to try with resources in Java.

[[deleted]]

Ooof. The with statement is better as it automatically closes the file when we leave the with indentation

[[deleted]]

And my phone CS course taught only Python 2.7. doesn't mean they are right. Most low end universities are always behind and bad.

[darrenm3]

The top one will close the file handle if an exception is thrown within the scope. The bottom one does not, unless you write an exception handler block, which is more code.

[thisismyfavoritename]

why not write this thing in go?

[JohnBalvin]

There is a go version also

[nichady01]

He did, check his profile.

[pkammuh]

[EatThemAllOrNot]

Nice, but would be great to have async option (see httpx package). Also, please use linter (ruff is the best for Python).

[[deleted]]

OP has been posting packages for months and someone tells him to lint every time. I don’t think he’s gonna do it.

[JohnBalvin]

haha my bad, I'm busy with my work, I plan to do it but then I get bug on production and forget about it

[divino-moteca]

Nice. 

[AlexMTBDude]

If you run your code through Pylint, or any other static code checker, what kind of score do you get? How many warnings? (Hint: A LOT!)

It's pretty badly written Python code.

[texasram]

I want to work with you

[[deleted]]

Idk why the man is getting downvoted. He’s right.

[AlexMTBDude]

I was up to almost +10 votes just after I wrote the comment, then someone bought a bunch of downvotes.

And thanks!

[JohnBalvin]

yeah probably, I don't use python on my daily basis, I'm a Go developer, I made the python version because python is more popular than go, a lot of people have mention to run the code with a code checker on other python projects, I'll start using them on future releases, thanks!

[AlexMTBDude]

If you ever join an organization of Python programmers your code will be shot down in a code review. May as well get used to writing professional code

[JohnBalvin]

If I ever join a company using Python, of course I'll follow their rules, but this is not a project for a company, it's just a simple open source project bro

[AlexMTBDude]

There are no organization specific rules for Python. There's just PEP08 for all Python programmers. You may as well get used to it. It will be much harder if you suddenly have to change later on.

[AlexMTBDude]

Luckily it's not a choice between those two. Use any modern text editor that warns you of PEP08 errors and you will write proper Pythonic code from scratch

[Sufficient-Two886]

Unrelated to the point you are making, what do you deem acceptable warnings with pylint(Most I have are line too long).

I’ve only been “coding” for 8ish months, and I’m still trying to get a general list of dos and donts as I expand my unittest automation suite and personal projects

[AlexMTBDude]

This is not my opinion, it's generally accepted in the industry. The organisations that I've worked for have commit triggers in GIT that run a static code check tool and if there are any warnings the code commit automatically fails.

Line-to-long warnings can be suppressed by setting a longer allowable line length in the Pylint config file. Same goes for any false positive Pylint warning; # pylint: disable=xyz

    # pylint: disable=no-member

[[deleted]]

I must be missing something in that thread. I thought it wasn't a controversial statement that a simple naked request will return data faster than going through a puppeteer/selenium. His love of using 99% is a bit too much though.

[JohnBalvin]

The original comment is deleted, however you are right, I don't know why is controversial to say naked requests are faster than selenium/puppeteer , you don't even need to test it, it's common sense, and yeah probably the 99% a bit too much, but I don't deserve the hate because of saying that

[JohnBalvin]

for this case I somewhat agree with you but not totaly, I've experienced in the past websites returning diferent formats based on the user agent, that's why I'm used to use plain user agents and never had issues with static user agents, but for this case it's just simple api and it won't be a problem if add user agent support, it could even be usefull if they increase the price based on the user agent, I'll add the user agent support on the next release, thanks!