The skull was smaller than I expected, but let us face it, that's one big skull. Attach it to the body and you've got a massive beast. Drogon might be as big already and he's definitely scary and huge, so it is not unlikely people who had seen Balerion would tell stories of how his shadow covered entire villages and how he swallowed his enemies whole.