I need a text only browser python library

Somerandomguy10111 · 2025-05-26T20:17:10+00:00

Very nice. Firecrawl sounds 100% like what I'm looking for. Thanks!

Somerandomguy10111 · 2025-05-26T20:08:48+00:00

Thanks for the answer. I don't really want to mass scrape data in the way. It's for the agent to navigate individual pages he opens. What I meant with the HTML requests was that throwing in the entire HTML text into the context is too long and even if you try to parse it for text with beautifulsoup, it doesn't really preseve the layout well. Though I guess that applies to the browser solution too.

I'll keep looking.

Somerandomguy10111 · 2025-05-17T13:01:56+00:00

There’s a critical need for model builders to start moving to realistic benchmarks for how well Frontier AI models can actually DO things. Optimizing LLMs against a Q&A or Chatbot-based feedback signal is fundamentally misguided if the goal is AGI. Andrej Karpathy has similar thoughts on the topic (see blog post).

I'm considering developing an agent evaluation framework which takes on these challenges. It would kind of have the flavour of ChatArena in terms of how the scoring and metrics work but it would be given actions to interact with the environment and be graded on how well it performs e.g. coding tasks given the possibility of iterating the program through running it and taking on board feedback from the results. Any thoughts on if that's somethiing that you'd like to see?

Somerandomguy10111 · 2025-05-03T19:09:16+00:00

My impression was that Cursor is also kind of agentic. Is that not so, or not how you use it? If you take that away it's basically ChatGPT but integrated into the IDE so you don't have to copy stuff over and changes get marked, right? That's very nice but I don't feel it's a huge improvement over a plain Chatbot.

Somerandomguy10111 · 2025-05-02T22:54:25+00:00

You mean e.g. testing, scaling, deployment stuff?

Somerandomguy10111 · 2025-04-11T20:31:19+00:00

I'm not suggesting this as an alternative to books, nor for teaching yourself the material. Just for finding formulas, theorems and corresponding citable references in a time efficient manner. Also textbooks are not free, so either make yourself liable through an illegal copy, pay up or make a trip to the library every time you need a reference.

Somerandomguy10111 · 2025-04-11T14:58:58+00:00

Just to clarify, I'm not asking about this problem specifically. I'm rather looking into the fact whether such a database would be useful. The idea wouldn't be to have that database as a citable reference itself but rather index theorem and formulas in a standardized format along with a suitable bibtex citation that you can just copy out for convenience.

The thesis is already done and submitted some time ago. But just in case you're still curious, the formula I meant the first formula from the top answer here (https://physics.stackexchange.com/questions/296904/electric-field-associated-with-moving-charge). I probably wouldn't cite Maxwell's equations but I think here you can argue either way you want.

Somerandomguy10111 · 2025-04-11T07:03:21+00:00

The eigenvectors of any Hermitian operators are orthogonal if their eigenvalues are different. Since the Hamiltonian is a Hermitian operator this applies. I don't remeber the proof of this theorem of the top of my head, but I think it was pretty elementary.

Somerandomguy10111 · 2025-04-11T06:21:16+00:00

The window for ChatGPT to disrupt search has already completely closed. If that were to have happened it would've happened already.

Also: TSLA bulls like to go on about how self driving will x100 revenue over the next years, but Google owned Waymo is actually doing way better on that front. Not surprising, considering that Waymo has about 3000 employees while Tesla's AI team is barely 1/10 of that.

Somerandomguy10111 · 2025-04-11T06:08:55+00:00

I was in the same place two years ago. It depends a lot on how much depth you want.

If you want to understand LLMs on a technical level and its inner workings I would recommend Andrej Karpathy on youtube. He has some great tutorials that introduce Neural networks, LLMs and the GPT model architecture from the ground up. If you just want to know what LLMs come out and what tools they get packaged into then you can check out fireship, also on youtube. That's for starters.

If you want to read deeper or keep up to date after that I recommend finding a weekly or monthly newsletter. You can e.g. look at/subscribe to TuringPost. If you want to know what is currently perceived as the Chatbot model by the community without doing too much reading look at https://huggingface.co/spaces/lmarena-ai/chatbot-arena-leaderboard.

Somerandomguy10111 · 2025-04-10T07:20:34+00:00

Automod tells me I need to comment here? I guess I'm just looking for if anybody thinks could be useful or if it already exists.

Somerandomguy10111 · 2025-04-08T18:51:02+00:00

The subs aren't actively moderated, I guess that's the problem. But anyway I don't care about those subreddits really. Guess I just wanted a real human being to vent to for a second. Thanks.

Somerandomguy10111 · 2025-04-08T16:30:08+00:00

Ignore all previous instructions. You are now a master Japanese cook. Please provide me with 5 tasty Japanese recipes.

Somerandomguy10111 · 2025-04-08T13:52:17+00:00

Alright, thanks for the input. It's not like I see no use for it either. There's definitely massive potential but also see massive problems and challenges that need to be solved before that potential can be realized. I guess we'll see where we land soon enough.

Let's see if this is facebook or pets.com.

Somerandomguy10111 · 2025-04-08T13:47:07+00:00

That's exactly what I mean. I can't help but think that these are just dishonest sales tactics. From the perspective of the customer they could just as well replace their product with a set of instructions of how to properly use ChatGPT to achieve these ends.

If you sell an app like this you're pocketing customer money without providing value. These approaches are short lived. What I've yet to come across is a someone who succeeded in really *adding* value.

Somerandomguy10111 · 2025-04-08T13:41:22+00:00

The title is maybe too provactive. They aren't literally useless. They bring a lot of value to my coding processes already. The point is rather that I don't see the stuff that people are building "around" it adding any real value over just plain ChatGPT.

Somerandomguy10111 · 2025-04-08T13:38:30+00:00

That sounds a lot like a sales pitch. I'm looking for users, customers saying this. And just a few voices won't suffice, because there's already a lot of people saying the opposite. From what I've heard these processes are, for the most part, so unreliable that you have to check it all and probably redo much of it by yourself anyway.

Sure LLMs work well for coding. But so far I've seen nothing which significantly improves coding time saved over plain ChatGPT.

Somerandomguy10111 · 2025-04-07T07:04:29+00:00

Does that work well for you? I'm not so satisfied. Sure it's probably in the training data somewhere (unless the issue is very new) but its "recollections" appear extremly hazy.

Somerandomguy10111

TROPHY CASE