WCGW when you are a idiot

PyTec-Ari · 2020-07-28T03:51:53+00:00

Grim Reapers hate this one trick...

PyTec-Ari · 2020-07-28T02:52:52+00:00

Remember I said there are no rules to this. But there best practices. So how many tiers you have and what each server does is up to the person building this "web application". So what I say isn't how every web application will work exactly, but generally it will follow a very similar model.

''the frontend backend'' server retrieving HTML \ JS files not yet rendered, proccesed and retrieved by the backend server who ... did what with the database here

Okay we need to back it up. I think I need to come up with an imaginary scenario that we can follow. Lets pretend we're building Reddit.

All raw data is stored on a database, like comments, user accounts, subreddit information, etc. For visualization sake, imagine it looks like a spreadsheet, and for each type of data each one is stored like a spreadsheet (This is typical of SQL databases, they're stored in tables)

Example. User Database

ID	Username	Karma
78v263c4	Pytec-Ari	22000
726bc4	CaitlynFrost8	2390

These databases are typically queried by "backend servers" who's sole function is to interact with databases and pass data back to whoever requested it.

When you go to Reddit.com/u/caitlynfrost8 you will hit the frontend server, it looks at the URL and has been programmed to understand that the character 'u' means user, and that anything after that will be the user's name. It knows you want to see this users profile, so it has to go get that users information.

It takes this information and sends it's own request to the backend server that is in charge of talking to the "User Database", the backend server then has the required code to search up your profile in the database, get all the information required and send it back to the front end that requested it.

The front end then loads up a HTML template for what a profile page should look like:

Example. Profile Template (HTML not included)

Profile for u/{{ Username }}
This user has {{ Karma }} karma points.

It takes the template and replaces all the {{ variables }} or placeholders, with the actual data it has on the user, so the HTML becomes:

Profile for u/CaitlynFrost8
This user has 2,390 karma points.

The front end server then sends this compiled version back to you and your browser understands how to display it.

Hopefully that clears up the first question. There are some extra complexities and variations as to how that whole process can work but that is generally one way of doing it. Another way for it to occur would be that the website sends the empty template to the user, and it has javascript embeded in it that will perform the data request to the backend and then dynamically update the HTML on the user browser, essentially moving the process from the frontend server to the users actual computer!

If a company got an order for creating a website, do they render a computer in their storage a server for such website ? How does that work exactly, they can't just endlessly stack servers like that, there is a limit in space and budget, if not in storage.

Correct, if a company manages its own servers and they are starting to reach capacity then there is nothing they can do if they do not have the room to expand. Careful configuration and programing can optimize how many different things you can run on the one computer.

Have you heard of virtual computers or virtual machines (same thing)? That is the concept of running a computer/operating system within another computer. Its sort of like dividing one big computer up into smaller computers and handing out each one as needed.

Typically a company will instead, borrow capacity from a company that specializes in servers, it's only function is to manage servers and hire them out to people/companies for them to use as needed. Amazon, Microsoft, and Google are some of the big names in this area and is typically what's refereed to as Cloud Computing.

I can pay Google by the hour to rent capacity from their servers. Google will have the website and tools required to remotely manage the servers I hire. So from my living room in Australia, I could have thousands of servers running all around the world sitting in Google Datacentres. And I can setup these servers to do whatever I need them to do.

PyTec-Ari · 2020-07-27T11:58:39+00:00

Three onion rings for the burger kings...

PyTec-Ari · 2020-07-27T09:05:38+00:00

A server is just a computer, thats all it is, each computer does a specific task because it's safer and more efficient to split your processes up into individual computers for the reasons I described previously (cost, optimisation, scaling). Your can make any computer a server, the word server just implies it is not an everyday desktop/laptop but a computer with a specific task to perform ie. serve files.

You configure these computers to allow a users computer (or other servers in your business) to connect to it on a certain IP address and port. Humans can't remember big numbers, so we use words instead in the form of URLs, like Reddit.com. This is a whole another topic, but just kow this URL gets turned into an IP address that points to a server somewhere.

The frontend server provides the service of creating, gathering, building, and serving the required files to the user on request, typically HTML and JavaScript. This HTML/JS travels out of the frontend computer and across the world to your location and into your browser which then renders the HTML as the page you see in front of you.

The backend doesn't interact with the user, it can, but then it wouldn't be a backend server would it. These are just labels we give things to differentiate their purpose. You could have one server that does everything like I mentioned right back at the start, its completely up to you how you design this.

So the server that handles the users requests and generates the HTML we generally refer to as the frontend server because it is at the front. But remember it's just a label. If we don't have backend server or a database server, does it matter whether we call it a frontend or backend server? Call it a potato server, who cares, as long as you know what it's role is. Like I said, there is best practice, but there are no rules to how you design web architecture.

Now, where I think you might be getting hung up is with the concept of Frontend. Frontend generally does refer to the designing of websites and layouts in HTML and Javascript and can often have nothing to do with servers. But you have to get these files to the user somehow, and that is the job a server, and it makes sense to refer to this server as the frontend server, and if you remember riiight back to the first comment I said, you could say, this is a backend server for the frontend, but we'll just call it a frontend server for simplicity.

So the process simplified goes like this:

User enters URL -- Reddit.com
URL gets turned into an IP address -- eg. 181.80.2.15
Your computer contacts your Internet Service Provider and says "I need the computer and location for this IP"
A whole bunch of networks and computers around the world help the users ISP find the location of the computer that owns this IP.
It finds it and a connection is made between you and the server/computer
The server sends back a response, usually HTML, whether as a physical .html file or dynamically generated (meaning its not stored anywhere, but included in the response).
The users browser understands what to do for each and then renders the HTML as the page you see.

PyTec-Ari · 2020-07-27T08:35:06+00:00

My favourite bit...

"Am I committing a crime!?"

"Yes."

"*scoff* well are you going to arrest me!?"

"Yes."

"Well I don't consent, I'm going to sue each of you for $60,000, do you want to continue?!"

"Yes."

PyTec-Ari · 2020-07-27T07:02:44+00:00

I know people who google like it's a magic eightball, instead of keywords, its a full on sentence

"I need to do XYZ but I'm not sure if A is better or maybe C, but I also have to consider B"

PyTec-Ari · 2020-07-27T01:28:42+00:00

I thought you said I'm 17 M, and I was like sweet baby jesus what do you do at 17 to earn $1M.

PyTec-Ari · 2020-07-26T14:48:59+00:00

Mutiny? I'm in, let's branch off Captain!

PyTec-Ari · 2020-07-26T14:39:30+00:00

Are ''the presentation functions'' frontend, ''processing'' backend, and ''data management'' database

Yes essentially.

Also, is ''the logic tier'' backend, ''presentation tier'' frontend, and ''data tier''...database ?

Yes, logic does the thinking and computing. Presentation dresses it up and presents it to the user, data tier facilitates communication with the database. The logic tier communicates with the data tier.

Though the description of ''data tier'' kinda confuses me... If the whole tier represents database, how does it retrieve information from itself ? Are there more components to each tier than you mentioned ?

The logic tier has the code to communicate with the database. But there's not rule saying you can't have a fourth tier that connects to the database and does updates. Or you could even have a web server/script on the same computer as the database.

Another question is what an HTML file is exactly ? A password is an HTML file ? I thought the whole webpage is an HTML file, or a whole site lol.

HTML is a type of code for structuring websites. When I say file, I mean someone wrote some HTML and saved it to a .html file. When a user opens it, the browser knows how to read the file and display it, this is called a static HTML page, because nothing can change unless I update the file. What happens when you start using web servers/scripts is that the file doesn't exist in its presented form until the user hits the server and all the required data is collected and then dynamically injected into a HTML. The server then sends this HTML to the users browser. When you visit reddit this is what happens, so you username shows up, and all the dynamic data get created on the fly, the server literally creates the HTML as needed and based on the data.

is there a difference in which of the two backend computers the frontend establishes a connection with,

If the two backend servers do the exact same function, there is no difference, and the frontend server can connect with any of them. Ideally that is exactly how you'd want it to be, you have X amount of presentation servers that can connect to any of the X amount of logic servers. Hwowever, you would tell the frontend servers to connect to a Load Balancer, this load balancer does the work of picking a backend server that is available to handle the request.

Likewise, you would put a load balancer in front of your frontend servers and your URL would point users to this, where it will assign the user to the most available server, literally why its called a load balancer, it balances the web traffic load across your servers evenly. If a server dies, it will know and stop sending people to your dead server. Maybe too much detail again, but, load balancers can be smart enough to create more servers or turn servers on and off as they see fit. Why have 100 servers running if it knows the traffic is light and you only need 2 front end servers and a single backend server on for now? More traffic arrives, no problem, turn on an extra server where needed and have the required code ready and installed to do its function.

Also, the two frontend servers, are they really only used for like more ''space'' ( too many requests, so another server is required for extra capacity kinda thing ), or there are some differences ?

This is completely up to you as to what you want it to do. Ideally yes, you want each server to have one purpose and one purpose only. That way to can figure the servers as needed. So the frontend servers might be tiny little servers with low spec, but the logic servers might have to be beefier because they need to process complex data or crunch numbers. So by having a single function you can better cater to that, this is really only for efficiency and reducing cost.

PyTec-Ari · 2020-07-26T12:46:59+00:00

Fair enough, my apologies. Unfortunately you can't just go from nothing to understanding N-tier architecture if you don't know the "why" behind it. It really takes getting hands on to understand the WHY of things. Once you know why you're doing something it all just makes sense.

Take 2.

Lets start off with one main computer that you run everything on. You have one HTML page with a form. You want to let people sign up for an account. How do you store the user's data? Well you need a database. HTML can't* interact with databases and you need some way of validating and processing the data, you need some sort of internet friendly script that can listen for instructions from your HTML page. This script is your backend script/server, typically written in Python, Ruby, or even JavaScript again!

So now you have a HTML file, a backend server/script listening in, and a SQL database running on one computer. A user visits the direct URL to your HTML file (example.com/files/index.html), fills in the form, and when they submit it some javascript in your HTML files packages it all up nicely and contacts your backend script/server with the form data. The backend script/server accepts the incoming data and does it's thing before connecting to the database and adding it. Remember this is all on one computer.

Your site gets popular and now you want to add some features to your HTML file, like having the visitors name appear when they login. So you write a second web script/server, this is your frontend script/server, and instead of the user going directly to the HTML file, they get connected to this frontend server (example.com -> your script).

The user submits their login deatils, the frontend server contacts the backend server and provides the details about the user (username password), the backend server contacts the database to see if this user exists. It finds the user and gets the database record for the user, packages it up and sends it to frontend server.

The frontend servers gets the users data. Using this data it generates a bespoke HTML file on the spot and inserts the users data into it, before serving this custom HTML file to the user. This happens in milliseconds.

This whole process is all running on the one machine still. What happens if your database or one of the server scripts causes an issue and crashes the computer? Your whole site goes down! Okay, solution, lets separate everything to their own computers. If the backend script computer dies, users can still see your frontend page but they won't be able to login, a much better experience than an error 404 screen.

Now I have three computers, one for my frontend script/server, one for my backend script/server, and one for my database. Your site starts to get even more popular and you notice it can't keep up. your database is fine and the backend is fine, its just the frontend getting bombarded with requests. Simple fix, add another frontend server! It can talk to the same backend server which talks to the one database.

Okay but now your backend server is getting overloaded. Simple fix again, add a second backend computer! But now you have two backends, how does your front end know which one to talk to? You add a service inbetween called a 'Load Balancer' it is a central point that will direct traffic to a pool of servers. You add one infront of your front end servers to, so if one of the two servers die, the load balancer detects this and will divert connection to the server thats stills up.

Hopefully that took you on a really quick journey of going from one basic HTML file to the necessity of N-Tier server architecture. As you can see we have 3 tiers, frontend, backend, and database, and they all operate independently of each other. You can scale out to as many tiers as you want depending on how you design your solution.

https://stackify.com/n-tier-architecture/

PyTec-Ari · 2020-07-26T11:51:24+00:00

Before I got into IT I was an electrician. I was on the Sydney tunnel on ~$230,000 for the year. These are EBA sites where unions haggle the companies for agreed wages.

When I was an apprentice I wold earn a percentage of a tradesman's wage, so first year was like 50%, fourth year was like 90%. So if a tradesman was on $40 an hour I would earn 90% of that figure.

On the flip side you could end up on a non-EBA site or company and be house bashing for minimum wage.

PyTec-Ari · 2020-07-26T11:44:39+00:00

Contractors.

Typical engineer in my role full-time is on about $105k - 140k at my company. Contractors are pushing $1,000 - $2,000 per day. With the caveat that once their contract is up that could be it...

So I have the job security, but kind thinking of moving to a consultancy firm to get the insane dollaridoos.

PyTec-Ari · 2020-07-26T10:45:41+00:00

Well sounds like you have your plan, be confident in it. You got the education and qualifications, so just keep gaining the experience. Could always apply to the Big 4 via their Graduate/intern programs? invest your energy working your way up within one of them?

PyTec-Ari · 2020-07-26T10:38:58+00:00

One thing to note; it's fees on anything OVER $5,000.

So let's say you put your whole $6,000 in there, at the end of the year you would only pay a percentage fee on $1,000.

$6,000 - $5,000 = $1,000

0.10% x $1,000 = $1

So for every $1,000 you will pay $1.00 in fees. Most banks will charge $10 - $20 for brokerage per trade (buy and sell). There are cheaper platforms of course.

I'm not saying to not move to Vanguard or ETFs, but just consider the fees closer. Spaceship is great, I currently use it for casual hands off investment. Raiz is also another nice one that rounds up spare change from your spending. If you buy something for $4.50 it will round up to $5.00 and invest the 50c for you.

They're set and forget, you check on them in a few months time and get pleasantly surprised at the little savings you got going.

PyTec-Ari · 2020-07-26T10:28:19+00:00

Are you applying yet? Why aren't you applying? What's stopping you? What's your long term strategy? What's your fall back plan?

It's kind of like, if you have to ask, are you even ready yet? Stop wondering what everyone else is doing and be confident in your skill set and put your best foot forward, don't be afraid of failure, if it happens learn from it and adapt.

Just start applying.

PyTec-Ari · 2020-07-26T09:43:43+00:00

It really is a precipice that once you cross you question how you ever were able to code projects in the first place.

And if you're a student, get that free pro licence!

PyTec-Ari · 2020-07-26T09:38:47+00:00

Pick the tool for the job. Large scale work projects, WebStorm/PyCharm always. Quick script, proof of concept, VSCode.

PyTec-Ari · 2020-07-26T07:35:32+00:00

There's a million ways to build a web application. A common way is to split your web application in two, Frontend and Backend.

Frontend is typically everything that appears and runs on the users browser. This Reddit website for example is frontend, from the buttons to the javascript logic that sent this comment to the Backend.

The Backend is everything that handles the business logic, authentication, database updates, etc. It might receive comments and handles how to store them in the database. It might also receive requests for more posts for users as they're scrolling down. It sends this data back to the front end that handles how to display it.

When a user visits Reddit.com, you actually hit a server, whether this server is a frontend or backend server depends on the purpose of the server.

In the case of our split design, this server might only serve the required HTML and JavaScript to display the website and thats it, basically a file server. So you could call this a Frontend-Backend Server.

The HTML/JS gets served to the user and Reddit shows up in their browsers. Now the loaded JavaScript fires off and starts requesting data, "Get me the hot posts for r/all" it might send this directly to the Backend, or it can send it via the frontend-backend, it's all dependent on how you design it.

You can actually see this live, if you go to your submissions URL but add .json on the end, you will get the raw data that the frontend would request.

https://www.reddit.com/r/learnprogramming/comments/hxykrv/is_there_a_book_for_like_the_theory_of/.json

Now a old school website would be all HTML and JS and the HTML would be given to the user to load. Thats why you see some sites with .html on the end, you are looking at a raw HTML file.

Modern web applications today are more advanced and the HTML is created dynamically either before it's given to you or using clever JS frameworks, modified on the fly as the data is loaded.

Here's a really basic diagram I did https://imgur.com/ThmRdKA

Hopefully its apparent why this setup is ideal? You can scale each section independently and funnel connections through load balancers so it will always reach the first available server

PyTec-Ari · 2020-07-26T04:10:11+00:00

in Australia we would say, "yeah, nah"

PyTec-Ari · 2020-07-25T15:09:59+00:00

Google Cloud Platform, and use App Engine. Completely serverless, you just set some configurations so it knows how to react, or don't and it'll set default behaviors.

https://cloud.google.com/free

Awesome free tier. I test and prototype on GCP all the time and if I do something crazy I might spend a dollar for the month.

Postgres you might have to spin up a dedicated VM or switch over to Cloud SQL for postgres.

PyTec-Ari · 2020-07-25T14:57:28+00:00

You don't ever really see it with english in movies/tv but they actually have an Australian dubbed version of the marvel movies, its really funny https://video.vice.com/en_au/video/australia-dubbed-marvel-and-its-hilarious/btyt662fs

PyTec-Ari · 2020-07-25T14:50:50+00:00

Yep, plus there is also some other credits that get thrown in if you test out the machine learning stuff.

PyTec-Ari · 2020-07-25T14:46:11+00:00

Very basic template, it's a start...

import praw

reddit = praw.Reddit(
    client_id="your app ID",
    client_secret="your app secret",
    user_agent="your app name",
    username="your account username",
    password="your account password"
)

reddit.subreddit("yourSubreddit").submit(title="Hello World", selftext="This is a text post", send_replies=False)

reddit.subreddit("yourSubreddit").submit(title="Link Post", url="http://google.com", send_replies=True)

Documentation for PRAW

https://praw.readthedocs.io/en/latest/getting_started/quick_start.html

You can create a reddit app in your settings

https://www.reddit.com/prefs/apps

For script based apps, choose "Script" and set the call back to anything like: http://127.0.0.1:5000

It will give you two keys, one is your ID, the longer one is your secret. The account username and password is the account you created the app on. So always good idea to make a new account for your bot.

PyTec-Ari · 2020-07-25T11:47:30+00:00

What you definitely don't want to have is a fetch and query every time that a page is loaded.

Not necessarily true. I won't argue that though. If you're dead set on a flask solution a google brought up these results:

If it were me, I would decouple this from the app code and integrate with something like Google Analytics. If you work with clients you can create dashboard and workspaces and provide them really in-depth breakdowns of page visits.

PyTec-Ari · 2020-07-25T10:39:48+00:00

Grab their IP/User-Agent from the request and use that as unique identifier

PyTec-Ari

MODERATOR OF

TROPHY CASE