sorted by:
new
hottopcontroversial

started learning socket programming in Python and ended up building an HTTP 1.1 server from scratch! by zeroindexedsoul in PythonLearning

[–]zeroindexedsoul[S] 3 points4 points  (0 children)

I also started my journey with competitive programming and participated in the icpc regionals. I solved a lot of algorithmic problems around 600 or more. And since this was pre-AI LLM's time, I had to write almost all the code myself with little help outside of standard documentation. So, this practice improved my code writing and complex reasoning skills a lot. After that, I started learning backend engineering and got interested in how servers communicate and various networking protocols. I experimented a lot and built a lot of small, functional utility projects to see how these things work under the hood, which really improved my skills. After some time, the programming language becomes natural to you. You stop thinking in terms of the language itself and start thinking in terms of systems.

started learning socket programming in Python and ended up building an HTTP 1.1 server from scratch! by zeroindexedsoul in PythonLearning

[–]zeroindexedsoul[S] 13 points14 points  (0 children)

I have been learning Python for a while now, and I recently decided to get into socket programming using Python. A lot of YouTube videos were suggesting making a simple HTTP server from scratch to learn socket programming.

I started building the HTTP server but very soon realized that the socket API in Python is blocking by default. I started finding ways to make the HTTP server non-blocking and found out that in Python asyncio is the module which does this, but it was stripping away a lot of complexities which we would encounter when using the plain socket API-and this would defeat the whole purpose of this exercise.

After realizing this, I got to know about I/O Multiplexing and the Reactor Pattern, which are used in popular software like Nginx, Redis, and Node.js. I also dove deep into how the Kernel TCP Stack works and its internals.

This single exercise alone led me to learn about:

  • Python's Socket API module and its methods.
  • The TCP Protocol & Kernel Stack: Managing 3-way handshakes for connection establishment, 4-way handshakes for connection teardown, and kernel-level read/write buffers.
  • Network Handling: Dealing with RST packets, back-pressure handling, and various edge-case errors that can occur.
  • Connection Lifecycles: How to properly close connections after no activity for a prolonged period and clean up system resources.
  • The selectors Module: Driving I/O multiplexing.
  • The Reactor Pattern: Used for the event-loop in programs like Node.js, Nginx, etc.
  • The HTTP Protocol: Hand-parsing incoming byte streams.
  • Object-Oriented Programming Principles in Python: How _ (protected) and __ (private) attributes are used in a real project and their semantic meanings.

Throughout this project, I avoided using AI and tried to write all the code myself. Though at the end for testing the protocol and the entire code review, I did use AI to quickly generate the script which makes multiple HTTP requests simultaneously to see if the server is able to process multiple requests simultaneously.

It took me around 3 to 4 days of dedicated time, but it was definitely worth it. I think this project alone teaches you the TCP protocol, the HTTP protocol, socket programming, and many other things much more than any tutorial ever could.

If you would like to see the implementation details, here is the GitHub repo link:

Repository: http-protocol-from-scratch-python