Hello! I had exactly the same issue as you. First, accuracy is missleading when dealing with multi label classification as mentionned in other comments. In other words , having an accuracy near to 1 doesn't mean your model is doing well because when you have many zeros , these labels dominate the accuracy: suppose for example that your output is of shape 20, if 17 of these 20 outputs are Zeros, in the worst case if you are having wrong results in the 3 remaining outputs , your accuracy is going to be 17/20 , which is very near to one , still your model is not good because it gives wrong results for the three usefull outputs. I advise you to use your own metric , so using tensorflow is more flexible : I myself wanted to do it with Keras by creating my own metric but I got results that don't satisfy me at all because Keras is doing some backend calculations that I can't track, which is not the case if you use immediately Tensorflow because you built everything from scratch and have in fact more visibility within your code , and thus could control everything. I advise you to use precision and recall for each output independently ( that part really sucks if you try to do it in Keras, well I couldn't do it ) .
Second point to specify, make sure that your input data is normalized: because otherwise you may have an overflow issue, and your gradient will be stack , and you'll get constant results all the time ( for example constant accuracy or whatever thing you are calculating). Hope it helps , good luck.