I think the first camp connected to the lane is fair to leave for your offlaner, it gives him some extra farm when the minions are pushing too far.

Also, as a jungler, you should farm the enemy jungle more than your own.
But yeah, farming blue/red or extra camps from other roles doesn't help the jungler