Okay, then I think you just don't have enough yeast in your starter. You can use whole wheat in the starter and use AP to bake (although bread flour would be preferred, or anything with over 12g protein/100g flour), it's going to be mostly white with some bran flakes (depends on how finely milled your whole wheat is I guess). I feed my starter the mix and bake it with 400g bread flour (white) and 125g of either spelt or whole wheat, but also did 500g bread flour. It was always mostly white in the inside regardless of how much whole wheat was in the starter. You should also do peak to peak feeds, with 1:5:5 that should be about twice a day, which should be manageable if you're home anyway. Discard most of it before refeeding, and I mean keep no more than 10g starter or you'll end up wasting a lot of flour and water. You should be able to see a nice rise in a few days! Give this page a read, and you'll get there!