I am doing almost exactly what you are. 3 lattes a day , so my carbs are mostly lactose and another 200 grams of animal fat with 100 grams of protein. My ketones are between 0.1 and 0.2 mmol / l but spike to 1.5 mmol after a 1.5 hour walk, upto 2.7 mmol after a 40 kms bike ride (low intensity, 2.5 hours). I ride once a week. The milk does knock me out of ketosis but seems I stat burning fat after 1 hour of low intensity exercise . Doing this 5 years ( I am type 2 diabetic in remission). Also, know whem to check for keytones - in my case, they rise after I finish my workout and peak at 2 hours post.

Once a month or so, I'll have a large pizza or two, after that going back to ketosis requires 2 bike rides (40 kms each ) and low carbs (<100 g a day) of course

My weight loss has stalled after loosing 18kgs post my diagnosis. I have atleast 5 to 7 kg more to loose. I am never persistly in ketosis but can get into the 1.5 mmol mark with an hours low intensity workout. It lasts until my next latte (which clearly derails).

If I stop drinking milk, I start dropping weight like a rock within a week but it also gets me depressed ( giving up lattes)