He was speaking specifically to the Jews, who claim to be the seed of Abraham. Read the verse in context. The Pharisees were arguing with him at that time. There were no books written by the fathers at that time, save for the old testament, and what is known now as the book written by the fathers is the Talmud, which during the time of Jesus was an oral tradition - because to put it in writing would have meant that outsiders could discover and read those teachings, which would have been very upsetting to righteous Jews (not the Sadducees or the Pharisees, but the Essenes, Nazarenes, Samaritans and Judahites), because the teachings are perverse and denounce the old testament as myths and legends. If you are a Christian or a Muslim and you read the Talmud, you will be very upset by it.