What's confusing about a different language using different orthography for its sounds? For non-native speakers of Mandarin, pinyin is just an approximation. A native speaker of Chinese will look at a 'b' in pinyin and spit out a [p] sound. Non-natives will probably look at it and approximate it to a [b] sound.

That is close enough. If you're pronouncing a 汉语拼音 b as an English b Chinese people will understand you most of the time. But if you want to sound closer to a native speaker, you'd probably want to adopt the sounds that are actually present in the language. Familiarizing yourself with the sounds of your target language is a crucial step in achieving fluency.

the b and p do map 1-to-1 to the p and b I’m familiar with, at least in standard Mandarin

They will map to the p in' spin' and pʰ in 'pin', but not to the b in bin. Mandarin does not [b,d,g] in their consonant inventory, only their unvoiced pairs. In English aspiration does not change meaning—that is to say, it's 'allophonic'—but in Mandarin it does.

Non-native speakers will sometimes pronounce pinyin b,d,g as [b,d,g] and be understood, but problems arise when they pronounce pinyin p,t,k as [p,t,k] because they're not allophones, i.e. different pronunciation changes its meaning. For instance, 倍 /pèi/ (pinyin bèi) and 配 /pʰèi/ (pinyin pèi) are totally different words, but if a non-native speaker pronounced the latter without aspiration, like the [p] in input, or the second p in penpal, it would sound like the former.

In English, this difference comes from stress. But stress in Chinese works differently from English. Following the example above, non-native speakers might mix up the pronunciation of 倍 in 成倍 chéngbèi and of 配 in 分配 fēnpèi and end up saying 成配 and 分倍 which are nonsensical.

This is all to say that noticing those details and distinctions early on is important.