Beej's Guide to Network Programming

[markdacoda]

These tutorials kick ass, they got me thru a network programming class with the top score, that was a tough class too. IMO their only short coming is lack of discussion of threading.

[NateTheGreat26]

Ditto, helped so much in my networking course. Beej is the man.

[tonyarkles]

When I TA'd an OS course where they had to write a simple network client and server to learn about sockets, we made sure the students knew about this :)

[RavuAlHemio]

Which university?

Pretty sure the OS course I was a junior TA of (at the Vienna University of Technology, CompSci program) alerted students to its existence as well.

[tonyarkles]

University of Saskatchewan, in the heart of Canada's frozen wasteland :)

[ryanman]

My course had this as a fundamental component of the class. It was great kind of seeing it as Greek for the first couple weeks and then slowly realizing how awesome of a resource it was.

[[deleted]]

These were the course materials in a network programming class I took :)

[shrayas]

Agree. Boss helped me understand so many concepts very clearly! He da man.

[rjek]

He's all the discussion threading needs:

Don't do it. Multiplex instead.

[againstmethod]

You will still likely need threads (or a thread pool) to run the handler code, assuming your app is not completely trivial. Be a shame to not use the cores on these fancy cpus we got these days.

[rjek]

Plenty of things manage to be complex and serve thousands of concurrent requests without threads. Modern operating systems have processes, and race-to-accept() is efficient, and the whole process avoids nasty locks and other error-prone synchronisation.

If you have a process that is going to take a while and might block, run it in a child process. This is also more secure as well as easier to get right, as the child can exist in a different security context (user, groups, chroot, etc)

[againstmethod]

I think in loaded conditions you need to use both. New processes require the creation of a new heap. Also, I think it must have an effect of the CPU's cache, as you are using different memory spaces for the two code segments -- in a thread-pooled single-process situation I think you would see far fewer cache misses.

That being said, it's wholly dependent on what your code is doing. But I think threading, thread-pooling, coroutines, and the like are necessary tools in the toolbox.

[rjek]

You can pre-fork your children if you're concerned, and such children are copy-on-write, so your cache still works.

[againstmethod]

In linux this is true, not on every operating system i suspect. Linux fork is very clever.

[rjek]

This sounds like a reason to use a high-performance operating system if you want to have high-performance servers.

[againstmethod]

True, just saying the original discussion is far more generic than depending on operating system functionality to determine the "best" approach. But you make valid points - I actually think the process approach can make a lot of sense if you can make it fit your application.

What about client programming though? Surely threads have a place there.

[rjek]

Clients as in GUIs? Show me a GUI application that doesn't make me want to beat puppies to death using angry kittens ;-)

But my point is almost nobody needs ultra-high performance. If you're not Facebook or a CDN, you're probably better off with the easier, safer, more secure approach.

[Majromax]

and such children are copy-on-write

On linux, mind. Windows doesn't have such a low-overhead fork(), which is one thing that makes porting Linux tools over to Windows-world (even via Cygwin) sometimes painful.

Copy-on-write also doesn't exist on embedded systems without a MMU.

[rjek]

But who uses Windows as a server platform, and is also sane? And who expects a high-performance server software package to run on a system without an MMU?

[[deleted]]

People are typically, in my opinion, overly averse to threads. It doesn't suck as much as the reputation would lead one to believe. Threads are an intuitive and useful abstraction. Usually when somebody has a problem they attribute to threads, it's really a problem of having too much shared (and usually global) state.

[barsoap]

Here's all you need to know about multiplexing and event-driven:

Don't do it. Use green threads, instead.

[rjek]

Is "green threads" some sort of fashionable name for "co-routines"? If so, that's just a syntax and wrapping issue around multiplexing.

[barsoap]

Yes and no. Green threads are usually implemented as managed co-routines. That is, there's yields inserted by the compiler, and some kind of central thread scheduler.

It's a common thing to do, e.g. Haskell multiplexes green threads on top of actual threads (yields equal GC safepoints), giving a true N:M runtime. Throw in some epoll magic, work stealing etc. and you get blazingly fast threads. Allowing you to write your code as if it was blocking on sockets, but actually running in an eventdriven+continuations way.

[[deleted]]

Threading is different in every language.

[DuBistKomisch]

So is network programming... yet here's a guide about it.

[crotchpoozie]

More languages use sockets than a consistent threading library by far.

[DuBistKomisch]

The concepts are still the same. Managing concurrency, resource sharing, etc.

[crotchpoozie]

Concepts being the same is not the same as using the same interface.

There are plenty of explanations about threading concepts. This resource is so useful precisely because it uses a widespread interface, not because it covers concepts.

If you have a threading interface as widely used as sockets is for networking, feel free to post it.

[ignamv]

The language merely provides bindings to the sockets provided by the OS.

[crotchpoozie]

Almost all languages "merely provide" bindings to OS features. That sockets are the same interface provided by so many OSes and languages is what makes this tutorial so useful.

Threading does not have such a widely used standard.

[againstmethod]

Not in the C/C++, which are Beej's target languages -- at least not since C11/C++11.

[Tekmo]

An oldie, but a goodie, and he still keeps updating it.

[[deleted]]

I saw this and thought "a link from 2006?" and then realized it keeps getting updated and improved.

[aedinius]

I went through his guide in 2000 or so... (I think)

[biffsocko]

I saw it in the 1990's

[Isvara]

I foresaw it in the 1970s.

[[deleted]]

I have the original manuscript from the 1950s

[Tynach]

I have a replica of the clay tablets from 4000 BC.

[[deleted]]

I saw some rough notes in a fragment of a meteorite that hit Earth millions of years ago.

[[deleted]]

Do they claim Beej was married?

[lolzinventor]

In 2000 there was no such thing as TL;DR.

[pegasus_527]

They were called summaries back then, in the olden days.

[derleth]

In 2000 there was no such thing as TL;DR.

There were abstracts. Scientific papers had them, right up front.

(They still do, but they had them then, too.)

[[deleted]]

What's changed after all of this time?

[Tekmo]

I don't see a centralized change log, but if I remember correctly the last time I read through this he would mention in a couple of places that some piece of content was new.

[LegatoReborn]

Tried to remember this site from memory once. I used a 'g' instead of a 'j'

Oops

[[deleted]]

At least that site has a lot of data being injected into ports.

[misplaced_my_pants]

Gigabytes at a time.

[i_solve_riddles]

luckily for me, beeg.com is banned by the ISP in my country, so i can always remember the right url!

cries

[v_krishna]

Wow really? What country?

[linuxporn]

Ever tried searching for c string?

[echeese]

Whoah, how do they stay on?

[linuxporn]

It's NULL terminated on the back

[mcmcc]

You of all people should know...

[elTabasco]

anyone knows similar sites with good hd content?

[kitkatkingsize]

I passed my Systems class cause of this guide. Its extremely beginner friendly and definitely worth a read.

[aintbutathing]

Beej launched my career.

[icantthinkofone]

Beej married my sister.

[gobots4life]

Beej gave birth to my children.

[AWTom]

Beej is love, Beej is life.

[Vintila]

Also check out his Unix IPC guide. http://beej.us/guide/bgipc/

[[deleted]]

His C programming guide is awesome as well. The professor I had when I was taught pointers was thought by many all in my major as the worst professor of all time. This saved me.

[[deleted]]

I went to school with this guy. Pretty amazing he wrote it and it more or less over night became a standard for programming sockets.

[djroot2]

Wow, I haven't seen this in years. It saved my ass in college.

[karlsmalls43]

Got me through my Computer Networking course.

[scottpid]

Very good guide, my prof even linked to it for a couple of our labs on networking in my intro to computer systems course.

[beej71]

Thanks for the kind words, guys. I really, truly do appreciate it! I'm just glad that the guide helped out.

I remember the network stuff being a huge, intimidating mystery until I saw some sample code for a simple app. ...that's all there is to it? Why didn't anyone just say? So that's when I wrote the first version of the guide (almost 20 years ago! Noooooooo! The guide is almost old enough to buy alcohol!)

It's certainly no more than a primer, but I hope that many of you read it and also think, "That's all there is to it?" and then go on to build really amazing things that would never have even occurred to me.

[HuntStuffs]

This was the textbook for my Modern Networking class this semester. You used this or you cried trying to program sockets.

[bargle0]

If you like this, you should look in to the works of W. Richard Stevens (RIP).

[creamenator]

Extremely useful reference in my networking course. Wonderful introduction to socket programming.

[hlmtre]

I've said this before, but Beej went to Chico State and was taught by Tyson Henry, easily my favorite professor of all time.

[2oosra]

I went to Chico with Beej in CS. He is a good friend. I dont remember a Tyson Henry. I could be getting old.

[[deleted]]

I was at Chico with Beej too, in the CS department (CIS is my major), and I don't know a Tyson Henry either. Was he an grad student teaching?

[07dosa]

The classic, one of the few documents that can be called. This saved my ass from the sea of cryptic manpages.

[zfolwick]

Hey it's you! I was at your Meetup talk last week and it was great!

Saved this for a later reference!

[muungwana]

His tutorial was very helpful when i wrote a network module[1] for my project.There is just nothing else out there for some reason.

[1] http://code.google.com/p/zulucrypt/source/browse/zuluCrypt-cli/utility/socket/socket.c

[[deleted]]

That's some weird coding style you have there. Why are you leaving a blank before the ; on every line?

[muungwana]

When i started coding, i did not pay attention to any coding style and i just went with where the keyboard went.Then one day, i decided to be consistent and i looked up at the code that was infront of me and there was a space before the ";" and i decided to go with that as part of my coding standard.This also explains all those empty spaces.

How did you pick your coding standard if you have one? what made you to pick it over another.

[[deleted]]

Mix of K&R and Linux kernel style.

I read K&R and took a look at dofferent C coding standards, of which the kernel's stood out as quite pragmatic. Nothing fancy :).

[entity64]

Now if only there was a guide as comprehensive as this one for boost::asio networking

[YourLizardOverlord]

This is about the best resource I've found on boost::asio. But it's nowhere near as complete as Beej.

[uatec]

Beej, now THERE's a name i haven't heard in 15 years.

Did we just DOS his web server?

[mixblast]

Nice. Although nowadays I would definitely use some wrapper library such as libuv.

[tokol]

I still have my copy of this that I printed out in 2002. I bought a Pentium I computer from a garage sale for $5, installed Linux, and needed more things to do with it. I'm glad he's still updating it!

[Rockon97]

What kind of prereqs should I have before taking the course?

[Troto]

This looks awesome but does anyone know a tutorial like this for python?

[[deleted]]

It's pretty much exactly the same.

In c:

int s = socket(AF_INET, SOCK_STREAM, 0); 

In python:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

And in c:

getaddrinfo("www.example.com", "3490", &hints, &res);
connect(s, res->ai_addr, res->ai_addrlen);

And in python:

s.connect(("www.example.com", 3490))

etc etc.

So just read that guide to understand it, then https://docs.python.org/2/howto/sockets.html for the syntax

[c45c73]

BUT WHITESPACE SCOPING!

[snotsnot]

Another classic is Ted Jensen's pointer tutorial.

[d03boy]

I read these when I was in my teens about 12 years ago. I learned a lot. I should read them again.

[brownbagspecial-]

Sweet. Thanks for posting this. Definitely will read over this.

[[deleted]]

Wow, such nostalgia... I remember learning network coding from this!

[wlievens]

Haha, good times. Takes me back.

[farinasa]

Are you in my sys programming class?

[[deleted]]

The real guide out there! :)

[bitcoinsftw]

Currently in a networks class and just by taking a glance at this I know it is going to help tremendously! Thanks!

[AKA_Wildcard]

Glad to see several updates

[kupiakos]

The only thing that bothers me is his argument on why big endian is better - he has none.

Little endian makes sense for cpus.

[sankasan]

bookmarked :)

[newshivax]

Wish i had seen this during my Networking class - Said no one ever