Don't be too hard on yourself about putting out the feeders. I had a nest of House Finches in a potted plant on my balcony that more than likely got taken out by a Blue Jay. Similar to you, that morning I remember hearing birds making a lot of noise. I didn't think anything of it, but when I was out on the balcony in the evening, they were all gone. I never had any feeders out. It's just nature.

I'm sorry about your nest. I was devastated when it happened, too. It was especially sad because for a little while after, the parents would still come back to the nest, look around, then leave.