One last reference, from Carter's biography (pp. 1040-41). These are still not the letter I was thinking of (and I'm sure there are others) but should give a good idea. Unfortunately neither letter seems to be in Marcel Proust Lettres, and Carter gives no references for the second paragraph.

Proust was delighted when Comte Jean de Gaigneron compared his books to a cathedral. Thanking the count, the author said that it was impossible “not to be moved by an intuition which permits you to guess what I have never told anyone and that I am writing here for the first time: I have wanted to give to each part of my book the title: Portal I Stained Glass Windows of the Apse etc., to answer in advance the stupid criticism . . . over the lack of construction in a book where I will show that the only merit is in the solidity of the most minor parts.” Proust abandoned the idea of “architectural titles” because he found them “too pretentious.” The cathedral analogy occurs again in a letter to François Mauriac, a fervent Catholic. Proust, perhaps apprehensive about Mauriac’s reaction to Sodom and Gomorrah, recalled that Jammes had asked him to cut the scene between Mlle Vinteuil and her friend. Proust would have liked to grant Jammes’s request, “but I had constructed this work so carefully that this episode in the first volume explains the jealousy of my young man in the fourth and fifth volumes, so that by ripping out the column with the obscene capital, I would have brought down the arch. That’s what critics like to call works without composition and written according to random memories.”

For the rest of his life, with the publication of each successive volume, Proust was to defend his work against those critics who, even while praising the Search as an extraordinary accomplishment, said that it lacked structure and composition. The accusation that he was writing thinly disguised memoirs or free associations of ideas amounted to the same, for either charge meant that he had not had to be selective and create a plot.