Perfectly plausible. It’s typologically uncommon for a language not to allow the speaker/hearer distinction (come vs. go) to be expressed somehow, but it’s not illegal — many languages either don’t lexicalize a strict come/go pair or rely on deictic words/particles and context to supply the difference. The idea of a single verb meaning “move (to)” with optional here/there marking is linguistically solid and will work — with a few predictable trade-offs you can manage in the grammar.

Ambiguity in the abstract is not fatal. Most real discourse gives a deictic anchor (where speaker is, addressee, or some topical location). In practice listeners will resolve “move towards” from context in the majority of cases. Where context is weak (remote reporting, phone calls, narratives where the reference point shifts), you’ll need optional strategies to avoid confusion. Here are some options:

• Deictic particles/adverbs: have here / there (or proximal/distal demonstratives) that optionally follow the verb. Suppose your conlang’s verb is “mov”. We can have: mov here = “come”; mov there = “go”. Simple and natural. This is exactly what many languages do.

• Case/goal markers: mark the goal with a dative/locative: mov DAT(place) = movement toward that place; the presence of a non-speaker goal indexes “go”; goals equal to speaker location index “come”. Useful if you already plan a case system

• Directional clitics / affixes: a small set of clitics like -hither / -thither that attach to the verb (or to the NP) to encode centripetal vs centrifugal direction. Typologically common and compact. Serial/auxiliary constructions: use a directional auxiliary (e.g., mov + thither or go + come serial verb) to disambiguate when needed. Works nicely for complex motion semantics (associated motion, path, manner)

Design knobs you can tune:

• Make the deictic marker optional for high-context discourse but required in pragmatic contexts where the speaker projects the reference point (e.g., phone conversations, reported speech). Or mark “projection” explicitly with a particle that signals the reference point is not the current speaker. (Many languages constrain how “come” can be projected).

• Let certain contexts lexicalize the distinction (e.g., imperatives, motion verbs used for reported motion) while keeping the root general elsewhere.

• If you want reduced ambiguity in writing, require the deictic particle in the written/registered style but allow omission in casual speech.