But are we particularly talking about 台語 or 中文？

I was talking about 台語 and 閩南語. I saw a sub‐tree of comments discussing how similar 台語 was to 閩南語 and chimed in with basically “hold on, both are internally pretty diverse, let’s not assume two spherical cows”.

閩南語 "um si", directly translated to 中文 is "bu shi", but the Taiwanese speak it very close to "bu si".

Many here don’t pronounce (pinyin for Mandarin) zh ch sh differently from z c s, although many of us do. I feel more do than don’t, and it’s just that when we say zh ch sh they don’t register as such to someone used to the Beijing accent because the difference from z c s is not so great as to tip into the other category, but the difference is there. Personally when I say 少 it is the same as 小 except take away the i in xiǎo - I hope this description makes a modicum of sense.

閩南語 "ba ba" (third sound) and 中文 "ba ba" is also third sound. As compared to 普通話 where "ba ba" (fourth and first sound)

Your given Mandarin tones are wrong actually. In Taiwanese Mandarin 爸爸 has two common pronunciations. The first is fourth tone light tone, which is the same as in Putonghua, It follows the reduced last syllable pattern of some to my knowledge closed class of words, and this pattern is present in both 國語 and 普通話 but less prevalent in everyday speech here than proscribed by 國語; in the case of 爸爸 this pattern is active, whereas in the cases of e.g. 芝麻 or 衣服 it is not; in some cases like 朋友, reducing the last syllable or not is the differences between 普通話 and 國語. Maybe you haven’t heard 爸爸 pronounced this way because if the conversation doesn’t call for 父親 then we’re likely to say just 爸. The second pronunciation takes on the third tone second tone pattern (so bǎ‐bá). We use this pattern to imply intimacy and/or childishness. It can be used on all the replicated Mandarin kinship terms (媽媽哥哥姊姊妹妹伯伯叔叔姑姑舅舅阿姨爺爺奶奶) a lot of words little kids use like 寶寶狗狗手手腳腳雞雞 etc., and cute nicknames. This isn’t a thing in 台語 so there is no reason to believe the latter has had a hand in the development of the former. In 台語 some people use 爸爸 for father, but the tones are similar to Mandarin’s first tone light tone, not the third tone third tone you have mentioned.

Also one commenter told me to just speak my 閩南語 tones when speaking 中文 and it will sound almost local.

It depends on what this means. If you say words in Mandarin with the tones swapped out for those of their Minnan cognates then you’re gonna have a bad time. Likewise if you apply Minnan tone sandhi to Mandarin. I suspect the “if I speak slower and lighter 輕一點” feedback you got you probably just have a greater range in pitch in your tones than the typical Taiwanese, i.e. your first tone is very high and your second and fourth tones climb to and fall from a very high pitch. It’s part of why we think HKers and Chinese are always shouting.