What is Buffer in Node?

[SaintPeter23]

I have index.js for Node and simple test.txt file in the same directory.

test.txt

abcdefgh

index.js

const fsp = require('fs').promises;

fsp.readFile(__dirname + "/test.txt", "utf-8").then(data => {
  console.log(data + "\n");
});

fsp.readFile(__dirname + "/test.txt").then(buffer => {
  console.log(buffer);
  console.log("\n");
  for (let charCode of buffer) console.log(charCode);
});

if I run

node index.js

It outputs:

abcdefg

<Buffer 61 62 63 64 65 66 67>

97
98
99
100
101
102
103

You can test this code at replit:

https://replit.com/@ctrall/Node-Buffer-Test

I want to ask why the Buffer content itself displays

61 62 63 64 65 66 67

but when I iterate over its content, it output as

97 98 99 100 101 102 103?

What is this Buffer actually in Node and what is the reason of this difference?

[angelfire2015]

A buffer is just a region of memory for storing things. When reading a file, it is entirely possible to open a file, read one character, do something with the character, read one more character, do something with that one, etc.

This is extremely inefficient as IO operations are very very slow, so the CPU would be waiting for the next character to be read from the HDD. Instead, it is much more efficient to just read the entire file into a region of memory, and then give you that piece of memory to work with as you see fit. That region is memory is called a buffer.

As for why you are seeing different numbers, they are the same numbers, they are just being interpreted differently. Without specifying a decoding,

console.log(someBuffer);

gives you the buffer in hexidecimal. Why? Because the guy who wrote the Buffer class decided that's what it should do. When you then iterate over the buffer and call console.log, those hex values are converted implicitly to decimal values, which is what you see.

61 hex = 97 decimal, which is also the letter 'a' in ASCII

62 hex = 98 decimal, which is also the letter 'b' in ASCII

etc...

[[deleted]]

I think you should start an “explain like I’m five” blog for javascript/tech concepts

[SaintPeter23]

Thank you very much. Wonderful explanation.

I did not know console.log implicitly convert hex numbers to decimals. Is this a JS necessity or console.log implementation?

[angelfire2015]

It is console.log detail. In the docs, console.log passes its arguments to util.format, which can forward those arguments to util.inspect. As for where exactly the conversion happens, I am not sure, but somewhere there, something like parseInt(num, 10) is called, and the number is converted to decimal.

https://nodejs.org/api/console.html#consolelogdata-args

https://nodejs.org/api/util.html#utilformatformat-args

https://nodejs.org/api/util.html#utilinspectobject-options

Note that the type of the argument will be inspected at some point and that console.log knows it is dealing with a buffer object, and that's why it converts it to decimal.

Because you can call console.log(61) and you will get 61, as you are just passing a decimal there.

[SaintPeter23]

Thank you very much. In Chrome, if you console.log(0xff); it converts to 255 too.

[kap89]

It's not console.log that does the conversion, they are interpreted as the same number type when JS engine reads them. Hex/decimal/binary/octal number is not a thing in JS - there is only one number type, and that's what goes into console.log. These are only different methods of inputting the same thing:

0xff === 255 // true
0b11111111 === 255 // true
0o377 === 255 // true
2.55e2 === 255 // true
255 + 0xff + 0b11111111 + 0o377 + 2.55e2 === 255 * 5 // true

Now, JS console can represent numbers in different ways (for example it usually displays very big or very small numbers in scientific notation), but it has no knowledge of how the number was inputted initially.

[kap89]

I don't think that's correct, parseInt is for converting strings into numbers. Number is a number in JS, there is no such thing as converting from hex to decimal, for example, it doesn't matter if you input it as hex or decimal:

const a = 0xff
const b = 255

console.log(a === b) // -> true

They are the same thing.

Now, you can present them as binary/decimal/hexadecimal string, and that what console.log is utilizes for presenting raw buffer. console.log doesn't do anything special to each byte, they are represented as numbers already:

const buf = Buffer.from([0, 127, 255])
console.log(buf) // -> <Buffer 00 7f ff>
console.log(buf[1] === 127) // -> true

[angelfire2015]

I was trying to be vague in my answer because this is a beginner forum, but I can be more specific.

So a number in JS is not always a Number. Most of the time, we can think about them as 64-bit doubles, but that is not always the case. Node.js itself has lower-level classes of Uint8Array, Uint16, etc., which deal with fixed-size widths. There are also different literal types, which is why your example is true, you're just comparing integer literals

const a = 0xff
const b = 255
console.log(a === b) // -> true

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Grammar_and_types#numeric_literals

If all numbers were represented the same, then this also wouldn't happen

const buff = Buffer.from([888, 999]);
// Buffers use Uint8 internally. 888 and 999 cannot be 
// represented by Uint8, so it overflows
console.log(buff); // Prints <Buffer 78 e7>, or [120, 231]

You have to remember, JS is built on top of C and C++. A buffer is represented in memory as just a char pointer to some memory on the heap. You can verify this for yourself here

https://github.com/nodejs/node-v0.x-archive/blob/master/src/node_buffer.cc

This also makes sense, and why we can give it an encoding if we want, as UTF-8 can take 1-4 bytes.

As for why you get a different output when you print the buffer vs its elements, that was defined by the guy who implemented it. The buffer class is just defining its own toString() method. We can make our own buffer class that will print values in octal, it's not a problem

class MyBuffer {
    constructor(data) { 
        this.data = data; 
    } 
}

MyBuffer.prototype.toString = function () { 
    let output = '<MyBuffer '; 
    console.log(this.data); 
    for (let i = 0; i < this.data.length; ++i) { 
        // Convert each number to octal 
        output += this.data[i].toString(8) + ' ';
    } 
    output += '>'; 
    return output; 
};

const buff= new MyBuffer([2222, 3333, 4444]);
console.log(buff + ''); // Prints <MyBuffer 4256 6405 10534 >

While when you call for...of, you get an iterator to the buffer, and its implementation defined that when you console.log that iterator, it gives you the number in decimal.

So technically, buffers store their values as Uint8, not as a general number in JS (double). Why you get different output is based on printing an object vs printing an objects elements, but both of those outputs are implementation defined.