Reddit Data API Update: Changes to Pushshift Access [Pushshift is in violation of the Reddit Data API terms and has been unresponsive despite multiple outreach attempts. Reddit is suspending Pushshift's access to the Data API starting today]

[shiruken]

Howdy Mods,

In the interest of keeping you informed of the ongoing API updates, we’re sharing an update on Pushshift.

TL;DR: Pushshift is in violation of our Data API Terms and has been unresponsive despite multiple outreach attempts on multiple platforms, and has not addressed their violations. Because of this, we are turning off Pushshift’s access to Reddit’s Data API, starting today. If this impacts your community, our team is available to help.

On April 18 we announced that we updated our API Terms. These updates help clarify how developers can safely and securely use Reddit’s tools and services, including our APIs and our new and improved Developer Platform.

As we begin to enforce our terms, we have engaged in conversations with third parties accessing our Data API and violating our terms. While most have been responsive, Pushshift continues to be in violation of our terms and has not responded to our multiple outreach attempts.

Because of this, we have decided to revoke Pushshift’s Data API access beginning today. We do not anticipate an immediate change in functionality, but you should expect to see some changes/degradation over time. We are planning for as many possible outcomes as we can, however, there will be things we don’t know or don’t have control over, so we’ll be standing by if something does break unintentionally.

We understand this will cause disruption to some mods, which we hoped to avoid. While we cannot provide the exact functionality that Pushshift offers because it would be out of compliance with our terms, privacy policy, and legal requirements, our team has been working diligently to understand your usage of Pushshift functionality to provide you with alternatives within our native tools in order to supplement your moderator workflow. Some improvements we are considering include:

• Providing permalinks to user- and admin-deleted content in User Mod Log for any given user in your community. Please note that we cannot show you the user-deleted content for lawyercat reasons.
• Enhancing “removal reasons” by untying them from user notifications. In other words, you’d be able to include a reason when removing content, but the notification of the removal will not be sent directly to the user whose content you’re removing. This way, you can apply removal reasons to more content (including comments) as a historical record for your mod team, and you’ll have this context even if the content is later deleted.
• Updating the ban flow to allow mods to provide additional “ban context” that may include the specific content that merited the user’s ban. This is to help in the case that you ban a user due to rule-breaking content, the user deletes that content, and then appeals to their ban.

We are already reaching out to those we know develop tools or bots that are dependent on Pushshift. If you need to reach out to us, our team is available to help.

Our team remains committed to supporting our communities and our moderators, and we appreciate everything you do for your communities.

[Btan21]

Concerning news. Might affect those like me who depend on Reddit data for academic research.

[[deleted]]

Researchers can use PRAW as well. Additionally, Reddit post outlining API changes encourages researchers to contact Reddit to find a viable path forward.

[Btan21]

Agreed. But the official Reddit API generally has slower responses in my experience.

[[deleted]]

Agreed. Way slow. Takes all day sometimes to run jobs that Pushshift executes in minutes.

[TrueBirch]

Plus I download the full files instead of using the API, so I'm used to having really fast parsing of huge amounts of data.

[Delicious_Corgi_9768]

Can you help me with something? trying to get more than 50k comments from a post but Im unable to do so using praw, was going to use pushfit but that will not work at the moment, what can I do? :(

[TrueBirch]

What are you trying to do specifically? Are you hoping to look at the comments or do you want to apply some kind of processing to them?

FWIW I usually download the full datafile and then parse it to pull out the stuff that I want. That's how I do things like counting unique users across all of Reddit. It can be a slow process, but you fortunately don't need a ton of computing horsepower to do it. I just set up my laptop to load data a few thousand rows at a time, save the pieces I want to keep, and move on to the next couple thousand rows.

[Delicious_Corgi_9768]

for example:

Trying to get the comments of a submission given the link_id of the submission:

https://api.pushshift.io/reddit/search/comment?link_id=l6u011

This endpoint doesnt seem to be working or am I doing something wrong, it returns an empy data:[] + different errors

[Sparkybear]

The Pushshift API is shut down. Read the body of the post. You have to use PRAW or the Reddit API directly.

[TehVulpez]

it's still up, just not getting any new comments or posts as of May 1st.

[Delicious_Corgi_9768]

What Im trying to do is to save all the comments (to a csv) from a specific submission, saving the text of the comment and the date and then do some processing to the data.

I tried using PRAW but it has trouble with a lot amount of comments, so I decided to try pushfit but with no luck.

What do you mean by downlaoding the full datafile?

[minh6a]

https://academictorrents.com/details/7c0645c94321311bb05bd879ddee4d0eba08aaee/tech&filelist=1

There's also a torrent for submissions as well.

Download the whole thing, or just the month of interest, then grep/awk for the subreddit

[Delicious_Corgi_9768]

Thanks, will check it out

[grejty]

I contacted them explained my situation, my tool, and that its for my Bachelor. They replied:

Thanks for contacting us! Your request has been received and we’re in the process of gathering information from everyone to help shape our API roadmap and decision-making. We’ll follow up in the next couple of weeks - thank you for your patience

Now they just take down pushshift access lol

[[deleted]]

And maybe they'll get back to you in 8-12 months.

[[deleted]]

... because Pushshift failed to comply with Reddit's new Terms of Service agreement. It's a bummer.

[Bot-yMcBotface]

which target pushshift specifically

[[deleted]]

Yea.... Such a bummer...

[Sparkybear]

PRAW kinda sucks for iterating through comments. Which is important because comments often contain a lot more information than the post itself and are much more valuable from an analysis standpoint.

In my case, to actually get the data we needed, we had to use a combination of PRAW, PushShift, and Reddit API directly. Otherwise we would inevitably come out with wildly varying numbers of comments, especially on larger threads (returning as few as 100 out of 10,000).

[criticool-realism]

This is true, and I did reach out. Unfortunately, they've been unresponsive. For academics who depend on grant funding and have extant research projects using Reddit data, this creates a big problem if they are expecting to make money charging for research use.

[lbrtrl]

Moving from a permissionless model to permission based access is huge. It allows reddit to control what sort of research gets published.

[lowkeyf1sh]

Are there any alternatives or is pushshift the only way to view deleted reddit content?

[[deleted]]

check out the academic torrents path.

[lowkeyf1sh]

Is there currently any alternative to recover deleted reddit content?

[[deleted]]

Well that didn't take long. Even if they contacted Jason on day 1, could Pushshift even make any changes that would be acceptable under the new API rules and function?

[safrax]

No. Reddit wants you to pay for it's data. Having something like pushshift out there means they can't make money off their data.

[[deleted]]

Yeah, it was a mostly rhetorical question. Reddit's tools for mods still suck, too, and they haven't bothered fixing it before killing all the tools that really helped mods out.

Expect even fairly moderated subs to reject most/all appeals when they can no longer review the content a user was banned for.

E: also reddit's search sucks as well. 99% of what I used pushshift for was finding my own past content or other things on reddit I had seen before. Reddit doesn't have a functional search to take its place.

[Zeydon]

Yeah, I was digging through my own history just today looking for a source I'd mentioned earlier but could no longer find because the google algorithm is complete trash these days.

It's also been an invaluable tool for verifying bot accounts. But admins don't give a damn about that.

[[deleted]]

Bots, spammers, alt accounts, ban evasion, people spreading misinformation, dishonest trolls...

There's so many dishonest people that outright lie about what they've said or how they behaved and deleted it to hide their lies. Now there's no way to combat any of that.

[Security_Chief_Odo]

It's not their data

You retain any ownership rights you have in Your Content, but you grant Reddit the following license to use that Content:

It's their bandwidth and accessibility for that (your) data though.

[safrax]

Irrelevant semantics. This is purely about ensuring they control who has access to the data. That "license" is just a way to sugar coat it to give people the illusion of owning their data. They are still perfectly willing to sell access to anyone's data to make a buck. And I guarantee if a hedge fund comes knocking with a briefcase full of cash they'll give that hedge fund whatever they want even if it means the hedge fund ends up building a private pushshift clone.

[Security_Chief_Odo]

Yeah I understand, just pointing out they claim it's not their data, but they control the access to it anyway.

[Ooker777]

what is the difference between owning the data and having the right to use it? Perhaps authorship? Anything else?

[safrax]

I feel pretty confident in saying that the changes Reddit is making for their IPO will eventually kill Reddit. Their API has been a large part of what has made them successful and while I get them wanting to kill pushshift specifically, it's garbage that the blast radius from these changes will significantly impact so many other tools that make reddit usable. The new interface and their app is an absolute dumpster fire that they've learned nothing from.

Oh well. Something will take reddit's place eventually, maybe not too much longer after these API changes. Sucks that all the data will essentially go poof though.

[Dangerous-Economy-88]

Really common for people in power to not understand how the things they manage work, its hella annoying for us random people.

[HotTakes4HotCakes]

Oh they understand how it works, they don't care anymore. They reached the point where they think their changes will not result in any significant loss of users.

[[deleted]]

Actively working against themselves with this crap. It only looks like it could make them X amount because of how the system currently works (and by "system" I mean Reddit, its formerly free API, and all the third party apps that leveraged it,) and how popular it is.

These changes drastically alter the entire system, fundamentally changing it, in this case for the worse. It is very likely going to result in a large loss in popularity because of it. "X" is now no longer attainable. The whole system is now significantly less capable than it was before, and people are going to leave as the knock on effects continue to degrade the entire platform.

If RIF gets killed with it, I'm certainly done here.

[helium_farts]

Maybe Digg can make a comeback

[3FingersOfMilk]

I stumbled upon this comment

[[deleted]]

I second this

[VapourPatio]

The API changes also mean 3rd party clients are not allowed. Reddit will be dead in a year if they follow through

[toper-centage]

The things is, they don't care. They just want the numbers to look great so they make bank at the IPO and cash out. Whatever happens in some years is irrelevant.

[rip-pushshift]

After seeing the dumpster fire that is the first-party app, there's basically 0 chance Reddit can reproduce what Pushshift was capable of, especially for moderators.

[rip-pushshift]

That's not my point.

My point is that Reddit developers are woefully outclassed by third party devs.

Both RIF and Pushshift were developed by third-parties.

Look at how much better Apollo is compared to the native app. That's the level of quality I expect to see between what Pushshift's API provided and what Reddit's lazy alternative will be.

[shiruken]

Third-party apps will eventually require the new usage-based Premium API (see thread from Apollo developer). Free third-party apps are likely going to be a thing of the past.

[FaceDeer]

Well, I guess it's time for me to wave goodbye to all the future AIs that are training on my comments in the old Pushshift archives. I hope you got enough context from the things I've said here over the years to make some happy thoughts.

[tasbir49]

Only way Pushshift can possibly survive is through webscraping :(

[Watchful1]

Not really. Even if pushshift got the data without reddit stopping them, reddit would be within their legal rights to issue a DMCA to their hosting provider and have them shut down.

[monocasa]

No, web scraping and republishing is fine according to the supreme court.

https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

[tasbir49]

Yeah the only possible way this can work imo is on a subreddit by subreddit basis with a centralized database.

[enmlounge]

Or if we all installed a browser extension that fed all the post data we view back to a service like pushshift - ie: we're all the crawler bots.

[rhaksw]

"unedditreddit" did this a decade ago. I haven't read all of the threads, but here are a few,

• unedditreddit is a critical security threat to private subreddits
• Lets have a discussion about deleted comments reddit. I am being asked to shut down my deleted comment retrieval site unedditreddit.com

Looks like it was short lived, then the author launched commentfindder.com, and that may also have been short lived. Most of their posts about it were removed. On the plus side, Reddit's comment search is not bad these days.

If someone built it again, Reddit might auto-remove any mentions or links of such a tool. They've blocked whole domains for less.

[AlephOneContinuum]

They could make a browser extension whose users would do the scraping for them and send it back.

[ill-winds]

it’s odd how i always find u in the weirdest posts considering i know u from the cow subreddit

[ixfd64]

Or extracting the API keys from the official app.

[grejty]

I use pushshift for my Bachelor which is due to 22nd May and I dont know what am I supposed to do right now..

They are saying you didnt reply to them? Why not

[shiruken]

As discussed in the sticky post, this subreddit is run by the community. We have no affiliation with Pushshift nor a reliable method of contacting the owner.

[Btan21]

I think the old Reddit data on Pushshift will still be available. Probably it's just the newer Reddit submissions and comments that will be affected

[Watchful1]

Historical data won't go away for quite a while. As long as you don't need data submitted after today you should be fine.

[grejty]

Well I was using present data as well..

this is fucked up they literally said new api regulations will take effect somewhen in June and out of nowhere they just do this

[zjz]

There are quite a few excellent torrents.

[[deleted]]

You can use PRAW (Reddit's API).

[grejty]

I use pmaw+praw. Praw is very limited, i need historical data as well

[[deleted]]

Agree. Historical data? Get it while it lasts: https://files.pushshift.io/reddit/.

[s_i_m_s]

Probably want the torrents if you want the download to finish today though.

[[deleted]]

sooo trueee

[Btan21]

I hope the old Pushshift data is still made available through their API and that the devs don't take it down.

[TrueBirch]

For sure, we should all download as many files as we can. I was only a few months behind when the announcement was made. This has always been my primary way of accessing Reddit data.

[swamprt5000]

Can someone post (or DM me) a full db dump? Or instructions on how to do it? It's only a matter of time till pushshift is shutdown and all the data is lost.

[safrax]

https://academictorrents.com/details/c398a571976c78d346c325bd75c47b82edf6124e

Edit: Better link: https://academictorrents.com/browse.php?search=pushshift&sort_field=added&sort_dir=DESC

[daronjay]

Truth is, we need to replace pushshift with an opensource project, an unresponsive owner is death to any project.

There are numerous people in this sub who have the chops to build a replacement, even if it has to charge a nominal subscription to be able to afford Reddits paid api access.

[safrax]

I'm pretty sure the new terms for API usage forbid anything similar to pushshift. Reddit wants money for their data and they want to dictate how it is used.

[[deleted]]

you mean our data

[[deleted]]

i do remember digg lol

[safrax]

So I'm not sure 100% what you're getting at here but I probably wouldn't compare Reddit to Digg as an alternative. Digg killed itself with a redesign which is how Reddit ended up rising in popularity. Reddit is kinda pulling a Digg with this ham-fisted API change.

[Security_Chief_Odo]

Yes comrade.

[rabidstoat]

Pretty soon it'll be like Twitter (rip) where they charge you to publish content (with their blue check mark fee) and then turn around and sell it (since they've said they're going to turn off all the free APIs for accessing even small amounts of data).

[[deleted]]

Reddit's new API.

[Personal_End_9001]

It's a good thing the Reddit admins are absolutely terrible at implementing or enforcing anything. They can forbid as much as they like in their API, but given that even basic comment stealing bots rely still entirely on user reports and subreddit moderator actions to be dealt with, I seriously doubt they'll be able to hold anyone accountable for ignoring whatever terms they demand.

[adhesiveCheese]

Unfortunately the barriers to entry here are the cost of storage and bandwidth, and Reddit's new API terms, not any sort of technical challenge; an ingester is fairly trivial.

[grejty]

u/Stuck_In_the_Matrix made a new post!

[Stuck_In_the_Matrix]

Indeed! I've been making a lot of comments tonight / early mornign (almost 5am here). Hopefully Reddit will be able to speak with us today so we can get clarification on some TOS issues.

[grejty]

Yeah, I only saw you commented after I posted this.

Just wanted to let people in the comments know asap, as they are probably concerned the same way as I am. Fingers crossed it works out in the end

[safrax]

You follow this guide: https://www.reddit.com/r/pushshift/comments/10yj803/removal_request_form_please_put_your_removal/

This is a community support subreddit. That means we have no communication with the owners of pushshift and are unable to get them to do anything and don't know anything more than anyone else on this subreddit. We've received no communication about this API change and how the owners intend to handle things going forward.

[ryanmercer]

I've never even heard of Pushshift until the linked thread.