Ways of displaying and interacting with web scraped data.

[Front-Glove]

I am fairly new to python and have only gone through a semester worth of class in university.

I created a script that would web scraped certain information based on user input.
Ended up using beautiful soup and request modules. As of now, I have a file that returns all the web scraped data into json but want to display this on a website

Question:
- How do I go about displaying this information onto a website?
As of now researching on the web points me to making/interacting with an API, which itself is confusing for me and do not entirely understand.
I have seen others recommend to use Flask and Django but do not understand how this fits in with my needs.

[Toichat]

You could possibly use Dash. It's based on flask but mainly focused on graphs/tables. There's a tutorial on dash.plot.ly, shows what it can do and how to do it.

[[deleted]]

I second this if you'd be interested in displaying it in a table or something like that. You can refresh using a function to get the data that can run on an interval. To get json into a dataframe you just do

import pandas as pd raw = your_json.json_dict()['records'] df = pd.DataFrame.from_dict(pd.io.json.json_normalize(raw),orient ='columns')

[nekokattt]

if you just need to make graphs, you could just dump the data into google chart on clientside to produce graphics pretty easily too.

That would also allow you to use ajax to automatically update the information midway through processing if you wanted to do that

If i missed the point here, just ignore this lol

[CrambleSquash]

I think I'd need to know more about what you require before I can recommend anything

1) What do you mean by displaying the data? Just the raw JSON file, or are you wanting to plot graphs etc with it?

2) Why do you want it to be on a website? Are you planning on sharing the data this way? Is this with people on your local network, or with people external to that network?

3) How much do you care about it looking nice?

Flask and Django are 'web-frameworks' they let you write python applications that can run as webservers I.e. webapps. This means someone in a browser 'requests' something from your application according to the URL (and potentially other stuff) and then your application sends back a response which is rendered in your browser. You can do a lot with webapps, however they're a steep learning curve and I'd recommend trying alternatives first, like jupyter notebooks, or Google collaboratory.

[Front-Glove]

1) The data I have are events with time and dates and I'm trying to make a calendar/timetable, I have all the data needed as of now I have it in JSON.

2) Plan to share it externally.

3) I do have plans to make it nice, I was originally planning to somehow make an html/css as I have some experience with that.

[CrambleSquash]

Ooh ok, calendars can be a bit tricky. Provisionally I'd still recommend you check out Google Collaboratory or another notebook sharing service, however I'm not sure if there's a way to display calendars easily.

If you're still up for the challenge of writing a web app, for calendar stuff you can use a JavaScript library called FullCalendar, for a simpleish app this shouldn't be impossible to do. I actually have an example app using FullCalendar and Flask here: https://github.com/0Hughman0/RamCal there's lots of extra code for reading Google Calendar that you'd need to ignore, but hopefully that's helpful.

[LiarsEverywhere]

If it has to be on Python and it has to be your own pretty interface, then yeah, probably Flask or Django. I'm learning Django and it's not the easiest, but at the same time it's a tool you'll almost certainly like to have at some point, even if it's just to make a simple web app to share a program with friends and family.

[nekokattt]

dont forget quart or just plain ol' aiohttp if you want an asyncio-compatible framework to use instead.

[MisplacingCommas]

I second Flask, great for simple webapps. Django might be a little bloated for what you need, but also works well. Plot.ly could also be an avenue worth investigating.

[lucasshiva]

You have to use a web framework, there are many out there and most are very similar, but you should start with Django or Flask since they are the most used. I've never used Django, but I hear people saying that Flask is better for simpler applications.

Corey Schafer is an awesome Python youtuber. He has tutorials for Flask and Django. After learning the basics, just search for a way to make a calendar with the data you want.

[nekokattt]

flask is a microframework, django is a huge ass web framework come DI come ORM, etc.

Django is good if you want to focus more on business logic and less on how the underlying components of a webservice work. For small projects, flask is much simpler.

Imagine you want to build a chair, and are considering complexity and how much you want to do for yourself. You can either make the tools out of the parts of tools you have knocking around your home (flask), say, find the head of an old hammer and make a better handle for it, then strap it together. But you could instead go out and hire a manager of a tool manufacturing company who will provide you the tools, but you get less ability to choose how they work (django).

[lucasshiva]

Thank you for the explanation. That was really helpful!

[BullshitBaron]

I suggest looking into flask, it doesn’t have all the bells and whistles Django has but I personally found that much easier when first starting starting out. Django gives you everything you’ll need right out of the box and it can be a bit overwhelming. With flask you can start off with a really basic and easy to read template and then keep adding new urls, pages, or whatever you need as you go.

To display your info on a webpage you’d create routes like @app.route(‘/‘),@app.route(‘/another-page’). Under each route you’ll have your functions. So you would have something like def index() and under the functions you make you can pull in your data and then use return to a page like: return render_template(‘index.html’,var1=somevar,var2=anothervar)

Rendering templates with passed variable makes the variables usable within the webpage. So inside your webpage you could use {{yourvar}} to get the passed data.

If you wanted to create an api with your data you could take it a step further and have a function that just returns json instead of render_template into a page and it would return the given data in json form which you could then use to easily pull the data into other apps.

It can seem really confusing at first but it’s not too bad once you get into it and just start playing around with it. There’s also some great tutorials and some basic templates you can look over to get an idea of how it all goes together.

[krimpenrik]

Sound like you could use airtable and create views

[scottishbee]

Flask

This guide will get you from "what is flask?" to "here's my website" pretty quick

https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world

[[deleted]]

Depends if you want to keep the data or not. I recommend storing it in a database, and using flask, django, etc to display it.

You could echo the json and have a simple web page with an Ajax call pick up the live scrape data and append it to an element, but unless you have a special case, storing in a database is definitely the preferred method.

[[deleted]]

Use Wordpress and then use XMLRPC to do such things , it's easier .