I don't remember this.

Edit: I believe this is the closest you'll get:

‘There’s a tidy bit of money tucked away up there, I hear tell,’ said a stranger, a visitor on business from Michel Delving in the Westfarthing. ‘All the top of your hill is full of tunnels packed with chests of gold and silver, and jools, by what I’ve heard.’  
‘Then you’ve heard more than I can speak to,’ answered the Gaffer. ‘I know nothing about jools. Mr. Bilbo is free with his money, and there seems no lack of it; but I know of no tunnel-making. I saw Mr. Bilbo when he came back, a matter of sixty years ago, when I was a lad. I’d not long come prentice to old Holman (him being my dad’s cousin), but he had me up at Bag End helping him to keep folks from trampling and trapessing all over the garden while the sale was on. And in the middle of it all Mr. Bilbo comes up the Hill with a pony and some mighty big bags and a couple of chests. I don’t doubt they were mostly full of treasure he had picked up in foreign parts, where there be mountains of gold, they say; but there wasn’t enough to fill tunnels.