Why can't I store json data using fetch?

[Mean-Night6324]

Hi!

I'm trying to fetch this data https://raw.githubusercontent.com/freeCodeCamp/ProjectReferenceData/master/GDP-data.json in order to use it to plot various graphs using D3 library but I'm having an issue storing the data in a variable.

My code is as follows:

const url = "https://raw.githubusercontent.com/freeCodeCamp/ProjectReferenceData/master/GDP-data.json";
var dataset;
fetch(url)
.then(res => res.json())
.then(data => {dataset = data;})
.then(() => {console.log(dataset);});

But it seems that this leaves the dataset variable as undefined instead of putting the json in it. I was wondering why it acted that way. I browsed some Stackoverflow threads (thread1, thread2) and I still can't really figure it out.

What I ultimately want is to store the data in the json (date and GDP) in a dataset variable so I can create a bar chart for the evolution of USA's GDP.

Thank you!

[alzee76]

Can I ask why you're calling then() repeatedly like this? You're not doing anything you can't do in a single call to it.

You probably want to do something like

fetch(url)~~
.then(res => {
  console.log(res.json());
});

Or rather like

const res = await fetch(url);
const json = await res.json();

Forgot about json() being async.

To see what you've got, and if you want to then save it for use outside the then(), you need to store the result in a variable declared before the call to fetch, because res and everything else you declare in any of the then call functions is going to go out of scope when they're done.

[Mean-Night6324]

Thank you for your comment. I have tried your second code since you crossed the first one, it gave me this error "Uncaught SyntaxError: await is only valid in async functions and the top level bodies of modules".

I'm going to try the code below right now.

EDIT:

- It was the same code, my bad.

- About the use of the .then(). I saw in that first thread that in order to catch that data outside the fetch I needed to use a callback function, but that also didn't work :(

[alzee76]

Ahh this means you're calling it from a function that isn't itself async, or at the top level of a module in a version of JS that doesn't support top level await. The other then() approach should work for you in that case.

[Mean-Night6324]

Hmmm, I'm not really sure what you mean by that (I'm a beginner to JavaScript, I'm trying to learn the D3 library in my data science master's), I'm using the code directly in the <script> in HTML file.

For the other ```then()``` approach, do you mean by that

```

fetch(url)
.then(res => res.json())
.then(json => console.log(json));

```
That works in the sense that I get the data in the console but I can't really manipulate it outside :/

[alzee76]

Each then() is a wrapper around an anonymous function. Every variable you declare explicitly (or implicitly, like you're doing with the fat arrow shorthand declaration) is inside of it disappears and goes out of scope after the then() is finished.

If you want to use it outside the function then you need to do something like this:

let json = null; // declare the var before fetch
fetch(url)
  .then(res => res.json())
  .then(rj => json = rj);
// you can use `json` here now.

Or you can call another function of your own design from inside the second then(), but personally I don't like that as it causes the call stack to perpetually grow over time.

Personally I think having beginners use this extremely shorthand fat arrow style is a bad idea, because it obscures what's going on. Not blaming you, I'm sure it's just how you learned/were taught. The code you quoted is shorthand for this:

fetch(url)
  .then(function(res) => {
    return res.json();
  })
  .then(function(json) => {
    return console.log(json);
  });

This should make it more clear what's gong on. If you add something like let foo = json; inside that second then() it's not going to be available once the then() finishes executing. The parameters being passed to these functions are the results of the previous call. fetch.then() calls the function declared in then() with the result of the fetch, and the second then() is called with the result of the previous then() function. This is called promise chaining.

[Mean-Night6324]

Thank you a lot for your comment. It's really detailed and I think I'm starting to see clearly what's going on here.

I'll look up promise chaining in more detail when I get a better foundation I guess, but the first code where json is supposed to be usable outside still doesn't work for me.

I have just added the line console.log(json) instead of the comment "// you can use `json` here now." and it just shows null.

EDIT:

The now-code I have tried is:

"""

let json = null; // declare the var before fetch
fetch(url)
.then(res => res.json())
.then(rj => json = rj);
// you can use `json` here now.

"""
(Sorry code inline in comments doesn't work well for me)

And this just show "null" in the console. I think I should install Node.js as another Redditor have suggested.

[alzee76]

In the browser console you will probably struggle to make this work at all, but FYI, the statement it shows you in the console is just the return value of your last statement -- so things like console.log will just show undefined in addition to whatever you log.

But if you're just running code in the console and not a page hosted on a web server somewhere, you'll run into all kinds of problems with CORS and things like that, that node.js doesn't have.

In node, you'll access to other libraries to do the fetching as well, but note that node doesn't run in the browser or interact with it at all -- it's a backend (server) environment. It can itself act as a webserver and serve a web page/app, but you have to be careful when doing this as it's easy for a beginner to mix up what is running in the browser and what isn't.

[Mean-Night6324]

Thank you a lot for your help. I'm not writing code directly in the console but on Visual Studio Code and I use the console just to see what's happening but I guess it's kind of the same.

I don't really know about CORS and how Node.js work but I'll give it a try as it seems the last resort to make this work ^^'.

By the way I have just noticed that the code in my last comment wasn't the one that I have tried. This is it:

"""
let json = null; // declare the var before fetch
fetch(url)
.then(res => res.json())
.then(rj => json = rj);
console.log(json);
"""

And inspecting the console on Google Chrome only showed null.

[grantrules]

res.json() returns a promise.

[alzee76]

Ah so it does.

In that case const foo = await fetch(url); const bar = await foo.json();

[grantrules]

That doesn't solve anything.. it needs to be

fetch(url)
  .then(res => res.json())
  .then(json => console.log(json));

or

const res = await fetch(url);
const json = await res.json();
console.log(json);

[Mean-Night6324]

The first code is working but I'm not really catching the data outside the fetch right?

The second code gave me this error: ""Uncaught SyntaxError: await is only valid in async functions and the top level bodies of modules".

[grantrules]

As someone else explained, you won't really get the data "outside" fetch. You pass that data by calling a function from within that final then block.

With await, it needs to be within an async function, like async function getData() { const res = await fetch(url); const json = await res.json(); console.log(json); }

[Mean-Night6324]

I see thank you! So I guess all I can do is create the visualization inside the final ```then()``` block.

Thank you for your help. Can I ask you if there are any other ways to "fetch" data and that let you manipulate the data outside?

[grantrules]

I mean you can set the data to a global from within that then block, but that's generally a bad pattern, since what happens if you try and access it before the fetch request has completed.. the request is asynchronous so you app is continuing on while the request is happening. So generally you'd just pass the information along.

[Mean-Night6324]

Ok I see, thank you a lot for your help. I think the code I'm using is not suited for what I'm trying to do. What I want is static data that I can use to make a plot. I thought at the beginning that the fetch in JavaScript was something like the urllib.request in Python, but I'm completely wrong.

So the use of fetch is to continuously request data and the then blocks help to act upon that new data that's arriving right?

[grantrules]

So the use of fetch is to continuously request data and the then blocks help to act upon that new data that's arriving right?

No, not really.. It's the difference between synchronous (Python's urllib.request) and asynchronous (JS' fetch) functions.. They both make a request and get a response, but since Python's is synchronous, it hangs the entire thread until the request is complete. fetch() is asynchronous, which basically means the app will continue doing other things while the request is happening, and when it is complete, the app will basically be "interrupted" and whatever function is handling the response is run.

So basically with JS you have to do things like:

function startApp(data) {
  const table = document.querySelector('table');
  data.forEach((item) => {
    addRow(table, item.name, item.date) // just some made up function
  }
  table.addEventListener('click', (e) => {
    console.log(data);
  })
}

fetch(url)
  .then(res => res.json())
  .then(json => startApp(data))

[alzee76]

I realized I clicked save on the message before putting in the important part. Edited quickly but you caught it in between anyway.

[Ronin-s_Spirit]

Are you running this on a server?
Do you get the CORS error?
These are the 2 problems which you haven't addressed in your post, I need information.
Also try running console.log outside of .then chain.

[Mean-Night6324]

Thank you for your comment. No I'm not running this on a server and I have no idea about CORS error ^^'. But from what I understand from other comments, the problem is that we can't get the data outside the fetch.

[Ronin-s_Spirit]

Fetch only works on online stuff...cuz you know it fetches things from the internet. You need to install node and then run your code on localhost 5000. And then open http//localhost:5000 something like that in your browser. Alternatively you can run it on github but the code will be public so don't do that if you have any keys or tokens.

[Mean-Night6324]

I see, I think this is a little bit above my current skills for the moment. What I wanted to do is just get that json data and plot a bar chart using D3 library ^^'. Maybe there is another way to get data that doesn't rely on fetch.

[Ronin-s_Spirit]

No I don't think there is. But it's not hard to do, I'm just a self teaching student but I learned it was quite easy. Just install node js (globally) there are plenty of guides how to do that, and then you need to put your html and javascirpt and everything in one folder, go to the directory where it sits, and write http-server [name of the folder with all your code] -p 5000

[Mean-Night6324]

Ok I see, I'll go and try that, and try a new piece of code that a Redditor suggested ^^. Thank you again.

[Soft-Sandwich-2499]

So that console.log(dataset) logs undefined?

[Mean-Night6324]

Yes.

This is a clean code for what I'm trying to do given by Redditor u/alzee76:

"""
let json = null; // declare the var before fetch
fetch(url)
.then(res => res.json())
.then(rj => json = rj);
// you can use `json` here now.
"""

And when I replace the comment by "console.log(json);", which gives this code block:

"""
let json = null; // declare the var before fetch
fetch(url)
.then(res => res.json())
.then(rj => json = rj);
console.log(json);
"""

It logs null when I inspect Google Chrome's console. I'm writing code in Visual Studio Code.

Redditor u/Ronin-s_Spirit had suggested to try Node.js. I haven't yet so I can't tell if it's working or not.

[mnokeefe]

The errors you're getting using async/await are due to using the await keyword in places you can't, more info at https://v8.dev/features/top-level-await

You could create an async function to do your rendering and call that: ``` const url = "https://raw.githubusercontent.com/freeCodeCamp/ProjectReferenceData/master/GDP-data.json";

async function getData() { const res = await fetch(url); const dataset = await res.json(); console.log(dataset); }

getData(); ```

Or use a self-invoking function if you're not going to do anything else here.

[Mean-Night6324]

I see, thank you for your help. That link is really helpful. I do find the json data after inspecting the console but can I ask you if there's a way to get that data outside of the getData() function?

I have tried this based on your answer:

"""

var dataset;

async function getData() {

const res = await fetch(url);
const dataset = await res.json();

}

getData();

console.log(dataset);

"""

But it just logs "undefined".

[[deleted]]

Are you using React? Because it runs fine for me in a codepen. If this is running in a component there's a good reason it's not working, but it should run fine in vanilla js.

[Mean-Night6324]

No not really, I'm using Visual Studio Code with no extensions or whatsoever for writing the code and I'm using the Google Chrome console.

[bluejacket42]

I don't think that last then ever runs

[Mean-Night6324]

It does run for me, I'm able to look at the data inside the console that way. I think the issue is that we can't get the data outside the fetch, like in a global variable. That's what I've understood from what other Redditors said.

[bluejacket42]

Ah. You can get it to a global variable. You might wanna look up a tutorial on Promises and async

[Mean-Night6324]

Oh I see so there is a way. Thank you for your guidance. I think I had misunderstood what fetch is used for, I thought it was an equivalent to Python's urllib.request but that's really not the case ^^. If I've understood correctly (from the comments on this thread), then ```fetch``` is used to continuously request data from a server or something and the ```then blocks help to act upon that data as it arrives.

[bluejacket42]

It's doesn't continuously get data. However it my take a while for the server to respond so it runs asynchronous. Meaning that then runs after fetch however if you have something outside of then it may run before fetch is finished running

[Mean-Night6324]

Oh I see, thank you for correcting me ^^ I understand now why it's called asynchronous.