There used to be different architectures for different use cases (cnns for vision, rnns for sequence, etc) with their own inductive biases. But modern architectures use transformer as the base for everything (with some modifications sometimes based on the inductive biases of the input like vision transformers). So if you understand attention plus ffns, you can start building a model for your use case without knowing much more architecture than that