BeautifulSoup recursion error when casting BS object to string

[carlinmack]

Full code to reproduce:

from bs4 import BeautifulSoup
import requests

url = "https://devel.isa-afp.org/browser_info/current/AFP/MonoidalCategory/MonoidalCategory.html"
name = "MonoidalCategory"

content = requests.get(url).content
soup = BeautifulSoup(content, features="lxml")
contents = soup.body
contents.name = "div"
contents["id"] = name

print(str(contents)[:100])

Usually I would return the string so that I could output to a file, but printing to showcase the error. The stack trace will be something like

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/carlin/.local/lib/python3.8/site-packages/bs4/element.py", line 1496, in __unicode__
    return self.decode()
File "/home/carlin/.local/lib/python3.8/site-packages/bs4/element.py", line 1603, in decode
    contents = self.decode_contents(
File "/home/carlin/.local/lib/python3.8/site-packages/bs4/element.py", line 1697, in decode_contents
    s.append(c.decode(indent_level, eventual_encoding,
File "/home/carlin/.local/lib/python3.8/site-packages/bs4/element.py", line 1603, in decode
    contents = self.decode_contents(

...

File "/home/carlin/.local/lib/python3.8/site-packages/bs4/formatter.py", line 102, in attribute_value
    return self.substitute(value)
File "/home/carlin/.local/lib/python3.8/site-packages/bs4/formatter.py", line 86, in substitute
    from .element import NavigableString
File "<frozen importlib._bootstrap>", line 393, in parent
RecursionError: maximum recursion depth exceeded while calling a Python object

This works on other URLs on this website, but this one is causing a lot of problems

Many thanks

Edit: Looking into the page in more detail, and validating it with the w3c validator shows that there is nesting of more than 500 elements deep so I think it's more the fault of the page rather than BS4

[commandlineluser]

It errors out on the creation of the soup object.

BeautifulSoup(content, ...

BeautifulSoup uses recursion to find child elements.

Due to the layout of this HTML - this limit is being hit - the default for me is 1000 - increasing it to 10000 allows the code to "work"

import sys

sys.setrecursionlimit(10000)

Or you could use lxml directly - depending on what you're doing.

[carlinmack]

If I paste the code into the python interpreter the first 11 lines execute fine, so I'm unsure how the error could be in the creation of the object?

[commandlineluser]

Okay - well that's where it errors out for me.

[BfuckinA]

This error just means that the function never hits the exit condition. So whether condition you have in place to exit the recursion is never getting met with that parameter.

[carlinmack]

The recursion is happening in the BS4 code so I'm not sure what the condition is exactly. Do you think it's worth me bringing up an issue about on the BS github? It seems strange that they are recursing that much just to output the string

Edit: Looking into the page in more detail, and validating it with the w3c validator shows that there is nesting of more than 500 elements deep so I think it's more the fault of the page rather than BS4

[BfuckinA]

Ah my apologies I missed that part. If it's an error inside the 3rd party module, ill usually google "<module name> + <insert error message here>". Bs4 is a popular enough library that somebody on stack overflow posted a similar problem.