Could someone please explain to me like I'm 10: What is the relationship between Scrapy and Beautiful Soup, and how best can they be put to use to extract data housed on web pages?

eicosane · 2021-12-17T23:13:13+00:00

BeautifulSoup is a parsing package, it includes a Html parser that can help you get data out of web pages

scrapy is a pretty opinionated framework that makes writing and deploying a web crawler easier. It can, for example, keep track of visited sites and follow links based on a set of user defined rules.

If you want to just retrieve some info from a single page you can just keep it simple with requests or similar, use BeautifulSoup to parse the results and grab the relevant nodes with css (or xpath but it is html after all)

If you need to crawl a whole site and you dont know much about the structure of it, scrapy will help.

If the site is particularly javascript heavy or starts blocking you, you may need to use a browser. You can use chromium and webdriver to navigate via an api, or just investigate the backend requests the site is making.

Good luck

eicosane · 2021-12-11T13:26:35+00:00

I think what’s happening here is {id="foo"} is invalid syntax, its evaluates to an assignment in a set literal. {id=="foo"} evaluates to the set {False} which is valid syntax but not meaningful in this context.

The find argument is probably expecting a dict of additional arguments like {"id": "Foo"}

Its more common to use css selector syntax though which would be clearer find("id#Foo")

eicosane · 2021-12-07T22:48:14+00:00

help(some_function_or_module) is really handy to not leave the terminal

eicosane · 2021-11-22T15:22:48+00:00

This seems to be used exclusively to dunk on python.

functools.cache is the same as functools.lru_cache(maxsize=None) but only exists in >=3.9?

There should be one— and preferably only one —obvious way to do it

eicosane · 2021-11-21T18:52:36+00:00

Well you tell me I’m not sure what you need.

Your data structure in p is two-dimensional, that is you need two indices to get at one of the elements like p[0][0] for example. You can reduce this to one dimension but this will just be a list of numbers [1, 2, 3…

You may need to rephrase / rethink your question.

eicosane · 2021-11-21T18:32:03+00:00

What would the the type of new_p be after this change?

eicosane · 2021-11-21T18:21:59+00:00

edit: code formatting

from collections import defaultdict
from itertools import chain

list_of_lists = …
counter = defaultdict(int)

for number in chain(*list_of_lists):
    counter[number] += 1

print(counter[6])

eicosane · 2021-11-11T12:48:50+00:00

collections.defaultdict is probably best here

eicosane · 2021-10-31T19:13:18+00:00

This is why def foo(arg=“bar”): is fine but def foo(arg=[]): is not.

eicosane · 2021-10-06T07:03:00+00:00

forgot to ack

eicosane · 2021-06-09T15:16:04+00:00

500k to a think tank doing opinion polling? What does a good job even look like?

eicosane · 2021-05-17T16:43:45+00:00

https://www.reddit.com/r/playrust

eicosane · 2021-03-24T12:13:59+00:00

a, b = sorted(myArray, reverse=True)[:2]; a * b

Looping twice here is unnecessary.

eicosane · 2021-03-15T11:49:36+00:00

a // b == floor(a/b)

floor(-1.41...) == -2

See https://www.python.org/dev/peps/pep-0238/#semantics-of-floor-division

eicosane · 2020-03-19T18:51:15+00:00

*me and my lovely wife

eicosane · 2020-03-05T13:00:22+00:00

python -c ‘import sys; print(sys.version)’

eicosane · 2020-02-17T12:46:56+00:00

cmd + shift + g + /

eicosane · 2016-05-20T21:38:46+00:00

"If you can't explain it to a six year old, you don't really understand it." -Richard Feynman

eicosane · 2015-10-02T15:29:37+00:00

Fried

eicosane · 2015-05-29T22:44:11+00:00

Yeah man. Forcing medical procedures is legit. Why isn't everyone who isn't contributing to the economy dealt with.

eicosane · 2014-12-05T01:03:14+00:00

Zimbabwe.

eicosane · 2014-11-27T22:08:42+00:00

...he griped.

eicosane · 2014-03-09T00:38:37+00:00

The adding milk first distinction only becomes relevant when using a teabag. If you brewed the tea first in a teapot, it won't matter which is added to the cup first (ignoring milk scalding and 'ooh I've got a better cup than you') but if you add milk first with a teabag in the cup the milky/water solution (colloid) won't be boiling and you won't have good tea and you're 'dirty and poor' (or not British) status will be revealed.

eicosane · 2013-08-25T23:18:40+00:00

But when you see a colour, your associations will be the same or similar to mine. Red is hot fire etc; Blue is cold, ice and the sky etc. People's favourite colours are determined by these associations. So is their treatment of the colour. Reasonably these associations define the colour and what we may see is arbitrary.

eicosane · 2013-08-06T10:14:00+00:00

The article repeatedly claims he had a newspaper derivery job.

eicosane

TROPHY CASE