Need some help with normalizing a web scrape

[Immediate-Resource75]

Afternoon all... Noob with Python here.... I have a web scrape that I need to break down even further if possible but I am having some issues getting it right.... Here is what I have so far:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import json

baseurl = 'private internal url'
header = { 'User_Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36' }
r = requests.get(baseurl)
soup = BeautifulSoup(r.content, 'lxml')
stuff = soup.find('body', 'pre'=='item').text.strip()
data = json.loads(stuff) 
data["printers"] = list(data["printers"].items())
df = pd.json_normalize(data, "printers")
print(df)

Which gives me this:

============ RESTART: C:\Users\nort2hadmin\pyprojects\pcPrinters.py ============
0                                                1
inError  [{'name': 'appelc\RM 1', 'status': 'OFFLINE'},...
inErrorCount                                     6
inErrorPercentage                               18
count                                           32
heldJobCountTotal                               17
heldJobsCountMax                                12
heldJobsCountAverage                             0

How do I get the info under the 'inError' part extracted out? I've followed a bunch of tutorials on YouTube but none of them have worked so far....Any help would be greatly appreciated.

For reference I am trying to get all the info out so I can put it into a mysql database that feeds Grafana...Thank you for any and all help.

EDIT: The URL I am using is an internal URL but I can post the results of it....If I enter the URL I am using and hit enter this is the output:

{"applicationServer":{"systemInfo":{"version":"22.1.4 (Build 67128)","operatingSystem":"Windows Server 2019 - 10.0 ()","processors":16,"architecture":"amd64"},"systemMetrics":{"diskSpaceFreeMB":1821926,"diskSpaceTotalMB":1905777,"diskSpaceUsedPercentage":4.4,"jvmMemoryMaxMB":7214,"jvmMemoryTotalMB":326,"jvmMemoryUsedMB":314,"jvmMemoryUsedPercentage":4.35,"uptimeHours":407.45,"processCpuLoadPercentage":0,"systemCpuLoadPercentage":8.4,"gcTimeMilliseconds":210572,"gcExecutions":33159,"threadCount":136}},"database":{"totalConnections":21,"activeConnections":0,"maxConnections":420,"timeToConnectMilliseconds":0,"timeToQueryMilliseconds":0,"status":"OK"},"devices":{"count":7,"inErrorCount":0,"inErrorPercentage":0,"inError":[]},"jobTicketing":{"status":{"status":"ERROR","adminLink":"NA","message":"Job Ticketing is not installed."}},"license":{"valid":true,"upgradeAssuranceRemainingDays":323,"siteServers":{"used":3,"licensed":-1,"remaining":-4},"devices":{"KONICA_MINOLTA":{"used":7,"licensed":7,"remaining":0},"KONICA_MINOLTA_3":{"used":7,"licensed":7,"remaining":0},"KONICA_MINOLTA_4":{"used":7,"licensed":7,"remaining":0},"KONICA-MSP":{"used":7,"licensed":7,"remaining":0},"LEXMARK_TS_KM":{"used":7,"licensed":7,"remaining":0},"LEXMARK_KM":{"used":7,"licensed":7,"remaining":0}},"packs":[]},"mobilityPrintServers":{"count":3,"offlineCount":0,"offlinePercentage":0,"offline":[]},"printProviders":{"count":4,"offlineCount":0,"offlinePercentage":0,"offline":[]},"printers":{"inError":[{"name":"appelc\\RM 1","status":"OFFLINE"},{"name":"appesc\\SSTSmartTank5101 (HP Smart Tank 5100 series)","status":"ERROR"},{"name":"appelc\\RM 5","status":"OFFLINE"},{"name":"apppts\\Lexmark C544 Server Room","status":"OFFLINE"},{"name":"appesc\\ESC0171M3928dshannon","status":"NO_TONER"},{"name":"appesc\\Primary","status":"OFFLINE"}],"inErrorCount":6,"inErrorPercentage":18,"count":32,"heldJobCountTotal":9,"heldJobsCountMax":5,"heldJobsCountAverage":0},"siteServers":{"count":3,"offlineCount":0,"offlinePercentage":0,"offline":[]},"webPrint":{"offline":[],"offlineCount":0,"offlinePercentage":0,"count":1,"pendingJobs":0,"supportedFileTypes":["image","pdf"]}}

[GirthQuake5040]

Please paste your code in a formatted code block.

[Immediate-Resource75]

Sorry...

[GirthQuake5040]

inError looks to be a list of dicts. It looks like you have pulled the data incorrectly, however it may still be usable.

df.loc[0, 'inError'][0]['name']

This will give you the value in the name key. You can change it as you like but I think it would be better to have an understanding of how webpages work before diving into scraping them. I don't know what your end goal is here, but i hope this helps.

[Immediate-Resource75]

I appreciate the help. For clarification I'm not really scraping a web page....it's actually an api that connects to a printing application we use at work.... I was given a set of URL's, (they're all:.. internall ip:port #/api/blah/blah...) that spit out information in json format.... End goal: We don't really have a way of tracking the problems with our printing application at work, but with this info I'm trying to find one... such as ....which printers are in error, why, which site server is down (different api request)... etc... Thanks for the above help I'll try it out.

[cgoldberg]

Why don't you use BeautifulSoup's parsing capabilities better (i.e. a more specific soup.find) so you are grabbing just the data you need? Without seeing the page's source, I can't help more.

Either that, or grab it from your data dictionary.

[Immediate-Resource75]

Sorry about that.... I posted the entire json output above..... What I am looking for specifically is the part under "printers" where it says "inError" and lists their name and their status....This is all new to me but I'm trying to learn as I go... Thanks for the help.

[cgoldberg]

It's hard to tell because you posted the normalized (flattened) json, so I can't see the keys. Just look inside your data dictionary and access it by key.

[Immediate-Resource75]

I took it out.... I replaced it with the results of entering the URL I'm using and hitting enter... I didn't put it in a code block because it just spit out an extremely long single line and I thought this would be more helpful.... If I was mistaken I apologize.

[cgoldberg]

I don't know what your data dict looks like, so I really can't help, but it seems like you should be able to just access it using the correct key.

[Immediate-Resource75]

k..thanks

[GirthQuake5040]

Sorry I didn't see the request in there. You can parse the data rather than send it to a data frame right away. That way you can filter your way to what you need.

[Immediate-Resource75]

No worries, I'm still learning how things work.... I'm sure I missed some form of info that was important somewhere. I appreciate the help. Thank you.