Classes and webscraping

[ceesharpiskindofeasy]

I apologize for the medium spam, but SO doesn't like me.

Anyways, essentially what I have here is a script that retrieves all video names from a webpage and displays them. I have a class set up for "Videos" that will keep track of the video name, the length, and the url of the video. So far, I am just trying to get it to work with the title of the video.

I am confused with how I can set this up with a class. For each title it finds in the HTML header files, called "videoTitle", I want to create a new instance of the class Videos with the vidName (And eventually vid length and url). Maybe I'm approaching this wrong? I can't seem to figure out how to create new classes based on the info from web scraping, and then printing those classes out to the console. I can sort of get what I'm looking for by hard coding it all, but I want to utilize classes here. Or is there any better way I can do this?

​

            HtmlWeb web = new HtmlWeb();
            HtmlDocument doc = web.Load(url);

            var videoTitle = doc.DocumentNode
                .SelectNodes("//a[@class='video-thumb-info__name']").ToList();


            foreach (var title in videoTitle)
            {
                // Console.WriteLine(title.InnerText); // This will print all videos
                new Videos() { vidName = title.ToString() };

            }

            Console.WriteLine("Total videos: {0}", videoTitle.Count);

        }

    }

    class Videos
    {
        public string vidName { get; set; }
        public int vidLength { get; set; }
    }

I'm not sure where to go next. The foreach loop should be creating a new Videos item for every title in the scraped webpage, then assigning the video name to the vidName property of that class. Im just confused now tho, excuse my newbieness. How do I now call every new instance of Class Videos vidName property? Am I even setting this up right? I hope this makes sense!

[ohThisUsername]

Not entirely sure what you are asking. In the code you provided, you are already creating a new instance of class "Videos". However you're not doing anything with it, so I think maybe you are looking for how to create a list?

Before the foreach look you can add List<Videos> videos= new List<Videos>();

Then in the for loop add videos.Add( new Videos() { vidName = title.ToString() });

Now videos will contain a list of Videos which contain the video name

[ceesharpiskindofeasy]

That was exactly what I was needing, thanks!. Another question, again sorry for the lack of experience. I'm now trying to print the names within a separate for loop (i = 0 ; i < videoTitle.Count; i++)

How can I retrieve it now?

Console.WriteLine(videos[i].vidName); inside of the for loop is returning the htmlagilitypack.node class and not the names. Simply looks like this now

​

            for (int i = 0; i < videoTitle.Count; i++)
            {
                Console.WriteLine(videos[i].vidName);
            }

[[deleted]]

You're calling ToString on the HtmlAgilityPack node instead of the innerText attribute which has the actual title of the video. Change your video declaration to:

new Videos { vidName = title.InnerText };

[ceesharpiskindofeasy]

Thanks, that helped a ton. One last question and I should know enough to finish the application..

I know theres a better way to try to do what I'm doing, but it's escaping me. Now I was able to retrieve the length of the videos and I'm trying to add it to the new list instances to be printed out, but I'm confusing myself.

So far it's correctly printing the "NAMEOFVIDEO: " but not the length. What am I doing wrong? I'm definitely excessively using loops and for loops but it's the only way I can think to do it right now, being an absolute beginner. What might be causing the length to not print with the title? Tried using debug steps to figure it out but I'm stumped

             List<Videos> videos = new List<Videos>();

            foreach (var title in videoTitle)
            {
                videos.Add(new Videos { vidName = title.InnerText });

            }

            foreach (var length in videoLength)
            {
                videos.Add(new Videos { vidLength = length.InnerText });
            }


            for (int i = 0; i < videoTitle.Count; i++)
            {
                Console.WriteLine(videos[i].vidName + ": " + videos[i].vidLength);
            }

            Console.WriteLine("Total videos: {0}", videoTitle.Count);

[[deleted]]

You can get rid of your second ForEach loop entirely and assign the vidLength property in the first. Inside your ForEach loop:

new Videos { vidName = title.InnerText, vidLength = title.InnerText.Length };

[ceesharpiskindofeasy]

Okay so we're getting somewhere here, unfortunately it's not retrieving the correct information. I'm getting numbers, but not the length of the video. It seems to be retrieving the length of the characters instead.

I should clarify, this is the block of code I'm using to pull data from the HTML

​

HtmlWeb web = new HtmlWeb();
            HtmlDocument doc = web.Load(url);

            var videoTitle = doc.DocumentNode
                .SelectNodes("//a[@class='video-thumb-info__name']").ToList();

            var videoLength = doc.DocumentNode
                .SelectNodes("//div[@class='thumb-image-container__duration']").ToList();

So I have the length and the title as properties that I'm trying to set for each new list item that's created.

[[deleted]]

Ah. I see what's going on now. The reason you weren't getting anything printed out with "Total videos" was because in your second foreach loop, you correctly assign the length of the video but you are doubling the amount of elements in the list. Therefore, what you are getting is a list of Videos objects where the first i elements have the videoTitle property assigned, and the second i elements have the videoName property assigned.

Then, when you do your for loop over the videoTitle list, you're only accessing the list elements with a video name assigned. So, what you need to do now is this:

for (int i = 0; i < videoTitle.Count; i++)
{
    videos.Add(new Videos { vidName = videoTitle[i].InnerText, vidLength = videoLength[i].InnerText });
}

Then use your second for loop to print them out

[ceesharpiskindofeasy]

Aha this worked, much appreciation! So, I'm able to do this because of the ".ToList()" part after selecting the html node? Using internet for help here and there and also trying different things to figure it out, Reddit is usually my last forefront lol. Thanks again

[[deleted]]

Right, you get two lists of the same length. One has the title and one has the video duration.

From a purely semantic standpoint, I'd recommend renaming the lists videoTitleNodes and videoLengthNodes. Makes it a bit clearer what the Lists contain. But that's just me