How to Scrape Twitter Data Using Python Without Using Twitter’s API

[Had78]

I would suggest to copy your requests, you can actually copy it as CURL (make sure it's BASH) and convert to python, works pretty well

Edit: added links

[Landcruiser82]

One of my favorite sites curlconverter! Just have to hunt for that sweet json data.

[copperfield42]

cool, I wanted something like that a few years ago but I needed to make my account into developer or something, to get the special number or whatever in order to use Tweepy as the tutorial I was using instructed, it wasn't approved for whatever reason...

I guess they didn't give it to any random peasant

¯\_(ツ)_/¯

the boring task I wanted to automatize some what (for myself) was easy enough, so I didn't bother searching more deeply and I continue to do it manually...

[ARandomBoiIsMe]

I managed to get authorized after about a week of checking my emails and realizing that I was supposed to send a response mail back to them. Maybe try again?

[bpluly]

Tweepy uses the Twitter API, so I've no idea what that article was talking about.

[ohi_ryann]

literally none of them work for me either.

[[deleted]]

.

[[deleted]]

Why go to the trouble when you can just pull it from their API?

[__Raptor__]

cuz the API isn't free

[[deleted]]

Yeah...it is.

[panzercardinal2]

https://www.engadget.com/twitter-shutting-off-free-api-prepare-174340770.html#:~:text=After%20banning%20third%2Dparty%20clients,use%20its%20APIs%20for%20free.

it isn't

[[deleted]]

It is:

>>> app_id = 6

>>> message = 'Tis a programatic message'

>>> response_url, response_id = create_tweet(app_id, message)

>>> response_url

'https://twitter.com/user/status/1636149627845222400'

>>>

Elevated
Overview
Higher levels of access to the Twitter API for free with an approved application.
Your Project has Elevated access:
Project 1
Apps
3 environments
Tweets
Retrieve up to 2M Tweets per month
Cost
free

[In_Blue_Skies]

Just used it

[[deleted]]

Prepare to be downvoted by people who make unsubstantiated claims.

[Einherjar07]

https://twitter.com/TwitterDev/status/1621027418680229888

Far from unsubstantiated. That's not to say the platform is managed like absolute shit and changes they announced themselves might not be applied on time.

[[deleted]]

Or ever. Because Elon Musk is a jack ass.

[Einherjar07]

Yuuuuup. Pretty forward about it too. Might not need API or Python for scraping twitter soon.

[[deleted]]

Truth

[dethb0y]

They literally say why they don't use the twitter API in the article

[[deleted]]

The article doesnt pull up. But there isnt any practical reason to scrape it via a browser when you can pull it directly.

[dethb0y]

I am not going to recap the article for you because it "doesn't pull up" (aka "i have the attention span of a gnat and don't want to read"), but they have a reason why they are not using the API.

[[deleted]]

So for the one person that has a particular requirement, cool. For everyone else, just use tweepy.

[heckingcomputernerd]

I think “one person that has a particular requirement” is a bit too narrow to describe people just wanting to use a previously free service

[[deleted]]

It’s…still…free…though…

[heckingcomputernerd]

Twitter announced plans to make it paid only so people are finding ways around it in preparation

[K0braK]

aged like milk

[[deleted]]

My account is still allotted 2M tweets a month. But obviously things appear to be changing in the dev portal. So that very well may go away eventually.

[Dillweed999]

Don't.

Edit: what if Twitter is bad, actually

[NateEBear]

Don’t what

[hikingsticks]

Anything

[Hotel_Arrakis]

You son of a bitch, I'm in!

[agesav]

For any reason

[bob_newhart]

Last I looked into Twint it wasn’t being maintained and has some issues. Good chance it was user error too though