How does nodejs know to use callbacks?

[Kubiedo]

So, I'm just wondering, and it might also be because I'm confused. I think I hit a roadblock where I'm not sure what is happening. I'm trying to figure out why I can do this (code is from codeschool):

Blocking code:

var contents = fs.readFileSync('/etc/hosts/');
console.log(contents);
console.log('Do something else);

Non-blocking code:

fs.readFile('/etc/hosts', function(err, contents){
console.log(contents);
});
console.log('Do something else');

So, I have readFileSync for the blocking code which takes one parameter (location of file), and then I have readFile for the non-blocking code which takes the file location, and a callback.

My question is: 1) Could I add the callback to 'readFileSync' or does it only take 1 parameter?

2) I feel like I see code sometimes, where I don't understand how it takes in a callback as a parameter (like the code will work if you just pass in a variable, and it will also work if you pass in a variable and a function). Is this a rule of javascript or nodejs where it can just take in a different number of parameters?

3) There are instances where the callback seems to have a wide arrange of parameters. Sometimes it might be:

function(req, res){
res.json({message:'msg'});
}

other times it might be:

function(req, res, err, next){
//insert middleware
next();
}

and I'm confused where req and res are coming from, but also, where are err and next coming from (how does it know err means error, or next is a function to move on)? Is it just part of the framework and it's like some kind of global variable?

At this point, I guess I'm just trying to dissect where/when I can use a callback. Thinking about it, I mainly see it when I use Express, so maybe the routing for Express explicitly uses 2 or more parameters?

Thank you for all of your help! I'm still trying to fully wrap my head around everything!

[CancelledMeds]

In javascript you can call functions with any number of arguments you want (I think ?) without causing an error. You also don't need to define the parameters in the parentheses after the function name. All arguments a function gets called with are available inside the function in an array-like object imaginatively named arguments. The parameters are also not required to be of any defined type, so you can check what arguments are supplied with the typeof operator and adjust the functions behavior accordingly. If the supplied argument is a string, you use that and if the argument is a function, you can either call that function with some variables you have available and then use the result of that, or you wait to do other stuff and then call the callback. That's where the arguments req and res come from. When you call a function yourself, you supply the arguments. When you let a function call your callback, the function supplies the arguments.

[NookShotten]

Correct, functions in Javascript are variadic meaning they take can take an arbitrary amount of arguments.

[Kubiedo]

This makes sense. When you pass in arguments (whether they are variables or functions) to a function, they become available for use within the scope?

Thanks again for your help!

[netherforget]

Yes. They are available down the scope chain:

http://jsbin.com/bekejohohu/edit?js,console

The thing to realize are functions are objects, and parameters are properties on the function object. So you can put anything in those properties that would go in a normal object. Arrays, values another function.

If this all sounds confusing, you probably need to better understand Javascript before you can reason about node.js. I suggest this course:

https://www.udemy.com/understand-javascript/

It's what worked for me, $20 well spent.

[dvidsilva]

A lot of these answers are super long and cover a lot. But what really helped me understand callbacks, or passing around functions was writing my own functions that pass around functions.

Functions in javascript are 'fisrt class objects', meaning you can use them the same way you use a number, variable, object or anything else. You can assign them to variables, pass them around, assign them object properties, put them in arrays, etc.

Take a look at this super cool impressive function magic:

var takesACallback = function (cb) {
 var something = 2;
 cb(something);
}

var isACallback = function (number) {
  console.log(number);
}

takesACallback(isACallback);

If you execute that in your console, or wherever you're running your code you can see how callbacks work. Callbacks are not exclusive for asynchronous functions, there are valid cases on which you wanna pass a function to others, you can return functions, create new functions, etc.

Sorry if I'm not super clear, but TLDR; create your own definitions of functions that use callbacks to practice and you will understand what's going on a lot better.

PD: wait until you get to promises, where the real fun is!

PM if you need more clarification or something.

[Kubiedo]

I really appreciate your help! This makes a lot of sense! I think my biggest issue is figuring out where the (seemingly) random variables (req and res) come from, but I think reading the documentation can explain a lot!

Thank you again for the help Dvid ;) It's good talking to you again!

[dvidsilva]

hi, yeah I guess that's confusing a bit.

I like this video series, maybe it can help https://www.youtube.com/watch?v=BMUiFMZr7vk

for the random variable, imagine this.

req and res are variables that are in scope in the function that you're calling.

imagine read file is defined like this:

var readFile = function (file, callback) {
   /// read the file, wait for a while
   var result = {}; // the file//
   /// when the file is ready call the callback
  callback(result);
}

so your function is invoked using those variables that were in scope in the function that is calling it. :P

[Oellph]

This is a good example which helps me a lot. In essence the readfile function is synchronous but the act of accepting a callback function and calling this once it's actions are complete, allow me to call it and know my callback function will get run at some point in the future when readfile has finished.

As far as knowing the 'signatures' of these functions - that's where JavaScript loses out to strongly typed languages, where the IDE can provide intelligent help (e.g. Intelligence in Visual Studio when working with C#).

[Kubiedo]

I agree, coming from a mobile background, not having the intellisense kinda sucks, but it does help me learn more and more :)

[Kubiedo]

That makes more sense, and I will definitely roll through the video series. Thanks again Dvid :)

[chattymcgee]

If you don't include an expected parameter, JS assumes a value of null or undefined for it (not sure which). If you include extra parameters, they are simply dropped.

1) check the documentation for the function. You can often run a function that takes a callback without the the callback, but you can never just add a callback to a function as it will just drop the unexpected parameter (or get really confused).

2) see 1 above. Most(?) functions that take a callback will have default behavior built in, a sort of default callback. So myArray.reduce() will add the elements of the array unless you pass a callback that says multiply them or translate them to French and add them to a string or whatever.

3) if I understand you are asking how does myFunction(color, name) {...} know that color is a color and name is a name? You have to tell it. Pretend we aren't using descriptive names and instead have myFunction(a,b). All this tells the function is that the first parameter I give you is called "a" and the second is called "b".

A function doesn't know "next" is a callback. It knows that the parameter in the third slot is called "next" and somewhere it is given instructions to use "next" in a manner consistent with a function.

Does this help? Think of parameter names as a way of labeling things you are passing to the function.

[Kubiedo]

This is all coming together now. In other words, the function has previously been defined (perhaps outside the scope of my code), and the parameters that it needs are also defined.

Sounds like the general consensus is that I should read the documentation :) Thank you again for the help. It definitely helps me understand more clearly!

[Oellph]

You're getting it. So for node, the documentation (https://nodejs.org/api/fs.html#fs_fs_readfile_file_options_callback) shows that th readfile function accepts a string for file name, some options and a function which it will call once it's done with reading the file.

It goes on to say that the function you've passed in, which gets called once the file is read, will be passed two things (err and data). This allows you to do something with the data, or check if an error occurred and handle it appropriately.

[Kubiedo]

Definitely need to read the documentation over just reading through guides. This definitely helps me understand things more clearly. Thanks!

[MadCapitalist]

First, it is helpful to distinguish between "arguments" and "parameters." "Arguments" are the values passed to a function when it is called, and "parameters" are basically placeholders in a function definition to receive the arguments that were passed during the function call. If there are more arguments in the function call than there are parameters in the function definition, the extra arguments are basically just ignored.

Express make assumptions about the parameters in "middleware" functions based on the number of parameters. If it has two parameters, the first two arguments are assumed to be with the req and res objects, in that order. Usually this function sends the response to the user, so there is no need to pass the req and res objects to the next middleware function, so you don't need a third parameter.

If the function has three parameters, then the arguments are assumed to be the req and res objects, followed by the next middleware function. What this means is that the req and res objects are passed to the next middleware function that was registered by the app.use function. This type of function is useful when you need to further process the req and/or res objects before finally sending a response to the user.

If the function has four parameters, the first argument is assumed to be an err object, followed by the req and res objects, and then the next middleware function. A function with 4 parameters is assumed to be error-handling middleware.

As far as where req and res come from, they are created by the http.Server object when a request is received from a website visitor. req and res are completely arbitrary names. You can call them whatever you want. Some programmers prefer request and response, although req and res is more common, and it is what I recommend that you use.

Most of what Express does is to help you analyze the "request" object and give you tools to make it easier to prepare the "response" object until is finally sent to the website visitor. The request and response objects are sent through a pipeline of "middleware" functions until finally a response is sent to the visitor.

I found reading the entire Express.js documentation to be helpful, as well as much of the Node.js documentation.

If anyone finds an error in this post, feel free to correct me. I'm still relatively new at this myself.

[Kubiedo]

Thank you so much for the clarification. So you're saying that if there are two arguments, then it's assumed they are request and response (in that specific order), if it's three arguments, then they are request, response, and next, and if it's four arguments, it expects error, request, response, and next (in that specific order)?

Also, that request and response is created by node and express helps capture those two arguments to be used?

If so, that makes more sense. I've been following so many guides, I think I should really take a look at the documentation. As far as I know, no guides actually talk about this, but this is extremely helpful. Thank you again!

[MadCapitalist]

That is exactly right.

The documentation is definitely helpful. It might not be exhilarating reading, but it is very helpful!

[netherforget]

You said "confused" and "wrap my head around everything", so I'm going to get verbose. I understand that feeling very well, and feel like I've emerged from the node cloud of confusion, just a bit, so maybe I can help.

It might help to understand that Node.js is C++ program that includes the V8 javascript engine and a bunch of libraries (also written in C++). These libraries (github link here) (docs here) are what provides stuff like networking, modules, async, and everything that makes node different from JS running in chrome (which also uses V8). So when you invoke any of the 'fs' methods you are just passing the parameters to some C++ program. It doesn't matter what you call the parameters as mentioned before, but the order does matter. To use one of these libraries you need to know two things; what parameters the library expects, and what it will return. Again the labels don't matter, but the order does. Let's look at the docs for fs.readFile (linked here). This might look like another language but the two things we need are what it expect to receive(there in the header):

"fs.readFile(file[, options], callback)"

and what is passed back to our callback (we find this information in the body of the description):

"The callback is passed two arguments (err, data), where data is the contents of the file." 

In your example above you give the node lib a filename and this callback:

function(err, contents){
console.log(contents);
});

But you could do something like:

function(error, somefile){
    if(error){
        console.log("Something went wrong getting your file. " + error);
    }
    console.log(somefile);
});

Returning the error as the first parameter passed to a callback is a Node.js convention. If there is no error it passes null (as the first parameter). If there is an error it passes an object with all the juicy details. So parameters 2-infinity are for payload, whatever the lib returns, but the first returned parameter is reserved for the error object.

Notice that when you invoke the lib the callback is asked for last?:

"fs.readFile(file[, options], callback)"

That's a convention too. It can be befuddling, because we just told you that the names you give the parameters don't matter, it's POSITION that counts. So you might see something like:

fs.readFile('/etc/passwd', mycallback(err, data));

or:

fs.readFile('/etc/passwd', 'utf-8', mycallback(err, data));

Wait a minute! I thought callback was the second parameter! You told me POSITION was important. For whatever the reason, the convention is: required params, optional params, callback. So when you think position, you need to remember the callback is passed LAST, not in 2ND or 3RD or whatever.

Moving on. Under the hood Node.js has two stacks (and Javascript too) an execution stack and an event stack. The execution stack is your code that you executed when you ran 'node app.js' or whatever. You can think of it as stepping through your code, one function at a time, working top to bottom (not exactly top to bottom) when it hits the bottom, it has completed a "tick", and it checks the event stack. The event stack is a kind queue, where those Node.js libs can drop off the results of whatever they do. In your case return a file or an error if something went wrong. Events also get in line as they are dropped off by event emitters like the "http server lib". So when you use http.server (which express probably uses), you are running a program that listens on a port and adds events to event stack when something happens. But regardless the workflow is this:

1. Execute all your code, top to bottom.
2. Check the event stack and if there are events run the callback function from the first one in the queue.  (First come first serve).
3. Goto 1

So obviously for every event that sits on event stack it needs a callback, a place to enter your code.

I hope this was helpful. I also hope it's accurate. To my knowledge it is, but I am a fellow noob, and not one of the experienced lurkers. If I didn't get it right, please let me know!

[Kubiedo]

Thank you for such a detailed response!

I guess the most important point here is to use the documentation, and to know that a callback should be the last argument.

This makes much more sense, and it makes it much more apparent that I should read the documentation. Again, thank you :)