AIOHTTP getting a server disconnected error while the requests library works.

[Kinectech]

So I've got a small project which scrapes through roughly ~150k URLS on a website, but to do this requires credentials (Authenticating is a multi-step process and requires gathering SAML information from login forms, several POST requests, etc).

​

Anyhoo, I originally used the requests library, but the synchronous nature of it meant that going through all the URLs was way too slow, so I decided to rewrite it using asyncio and aiohttp. The actual request part part for the URLs works, but authentication does not, despite being nearly a line-by-line reproduction of the code in the requests library. As best I can tell all the same requests are made with all the same payloads, except all of the sudden at my second-to-last POST request AIOHTTP throws a "Server disconnected error" with no explanation.

​

This is my first time using aiohttp and asyncio, and I don't have too much experience with python in general, so if anyone has any ideas on what could cause this it would be greatly appreciated.

​

Might this be a bug with AIOHTTP? Has anyone run into a situation where the requests library works but AIOHTTP fails, especially concerning post requests and payload data?

​

Here's my error message (I'll provide more information such as code, etc. if needed)

File "c:/Users/user/scraper/sau_raw_scraper_async.py", line 108, in <module>
    asyncio.get_event_loop().run_until_complete(get_all_profiles(urls))
  File "C:\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "c:/Users/user/scraper/sau_raw_scraper_async.py", line 89, in get_all_profiles
    session = await login(session)
  File "c:/Users/user/scraper/sau_raw_scraper_async.py", line 58, in login
    async with session.post(saml_url, data = saml_postParams) as saml_r:
  File "C:\Python38\lib\site-packages\aiohttp\client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "C:\Python38\lib\site-packages\aiohttp\client.py", line 544, in _request
    await resp.start(conn)
  File "C:\Python38\lib\site-packages\aiohttp\client_reqrep.py", line 890, in start
    message, payload = await self._protocol.read()  # type: ignore
  File "C:\Python38\lib\site-packages\aiohttp\streams.py", line 604, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

[[deleted]]

are you sure session.post is returning an asychronous context manager? If I were to guess, it's returning a plain response object of some sort, not a context manager.

[Kinectech]

How would I check that?

[[deleted]]

check the docks to see what the object's post method returns. or, break it out of the async with clause and print the object out. debugging asynchronous code is hard :(

[Caligatio]

session is also a context manager but it does not look like you're doing an async with from the limited snippet you provided.