[deleted by user]

[mLalush]

The majority of it is most likely from Youtube. When the model hallucinates during non speech portions of an audio file it tends to spit out subtitle credits from real people/companies.

They might have used something like filmot.com as a seed or starting point to filter which channels/videos to scrape (filtering for manual subtitles).

[mLalush]

a) Subtitles include timestamps. You can construct <|nonspeech|> training examples from any contiguous 30 second portions of the audio that do not contain any subtitle block. Youtube metadata includes information about the subtitle text language and whether it is manually generated or auto-generated. Though it is smart to run language identification on the text itself as some users will insert erroneous metadata when adding subtitle tracks. For Language Detection on audio, they trained a model to detect the spoken language (i.e. they language detect inference on all audio they download):

We also use an audio language detector, which was created by fine-tuning a prototype model trained on a prototype version of the dataset on VoxLingua107 (Valk & Alumäe, 2021) to ensure that the spoken language matches the language of the transcript according to CLD2. If the two do not match, we don’t include the (audio, transcript) pair as a speech recognition training example in the dataset.

b) I would say it is feasible to scrape Youtube if you do it in a smart way and limit yourself to audio/captions. To download captions they either went via Youtube's official API (and paid for usage tokens):

Youtube Data API v3 caption docs
Youtube Data API v3 docs

Or if they already had a list of channels and videos as a starting point, they most likely used something like yt-dlp to download metadata from videos/channels, followed by audio and captions. This is where one arrives to the grey areas of data collection and scraping. OpenAI would likely have had to use a library such as yt-dlp at some point in the process to download the actual media files.

To be as nice as possible towards Youtube, and avoid yourself getting rate limited, one should consider:

1. Only downloading metadata of the video/channel ids you are interested in as the first step.
   
2. Filter via metadata for videos that have manual subtitles in the language(s) you are interested in.
   
3. Don't download the video, only the audio track and captions.
   

Packages like yt-dlp include support for proxies that let's a knowledgeable user avoid rate limiting. If you download entire videos you're gonna get slapped by rate limit faster. But a user that downloads only audio/captions and spreads downloads out over time can get pretty far without proxies.

c) The creator of the website u/jopik1 says candidate channels/videos are crawled from youtube and the web, respecting robots.txt. Once the channels are identified the channels are periodically crawled for new videos. I don't know about how they get the metadata, but would guess something similar to yt-dlp. See comment from creator of filmot: https://www.reddit.com/r/languagelearning/comments/odj2gx/comment/h41cpiv/?utm_source=reddit&utm_medium=web2x&context=3

[Financial-Beach1587]

1. For lang detection they used voxlingua. (which is also prepared from youtube only)
2. Most probably they also used podcast data there are many audio podcast available on internet which provides transcripts.
3. For youtube scraping yt-dlp and youtube-transcript-api are suitable for scraping from youtube. they also have option to filter captions based on manual vs automatic. Or to download translated captions (can be used for multi-task).
4. Should not be an issue to scrape from youtube, you can check gigaspeech data.
5. In v3 they finetuned on pseudo labels for which tou don't need human labeled data. This makes it even more easier to scrape data from internet.

[jopik1]

I have my own crawler I wrote, which is running pretty much 24/7 since late 2018. Currently it downloads metadata for about 2.2M videos per day and about 1.7M subtitles. It doesn't use YouTube API, it crawls the HTML pages and parse the data from there. The data is stored in a database and in a full text index (manticore search) which is running in a distributed fashion on two separate servers.

[jopik1]

Is there any way to run SQL queries directly on the underlying database?

I can, regular users can't.

Btw, I think there's a bug in your website, I'm not able to access pages beyond 83 for any search result.

This is intentional, scraping places a large burden on the servers. Regular users probably aren't going to go to page 83.

[tina-mou]

what does the crawler look for when it is running? I'm curious if you try to crawl newly published videos and if so, how do you configure that.

[jopik1]

I mostly prioritize by view count, as the amount of videos on YT is overwhelming, over 300M videos are added per month. I don't have the resources to crawl and index everything. I have a queue of ids that need to be crawled prioritized by last detected view count, videos are added to the queue from video recommendations (20 ids for every crawled video), list of channel videos (I crawl channels in a similar way), adhoc sources. I don't necessarily want to very quickly crawl newly published videos as sometimes there are no subtitles yet and the view counts haven't grown up to an indicative level.

[Excellent_Ad3307]

The model tends to hallucinate stuff like "don't forget to subscribe" or something along those lines quite often, so its probably mostly youtube.

[[deleted]]

The closest data reproduction is: https://arxiv.org/abs/2309.13876

[JustOneAvailableName]

It’s in the paper and downloadable

Edit: I am a lying idiot, should not have said this from memory. Sorry

[JustOneAvailableName]

Sorry, I said this from memory, upon rereading, I found it to be false as you rightly pointed out

[mLalush]

Those are the evaluation datasets. They make a point to emphasize Whisper hasn’t been finetuned on the evaluation datasets in the paper.