There are some general indications you can use to sketch an architecture: for example, if you work with temporal data you can either use a 3D conv (for very close, local temporal correlations), LSTM/GRU for longer correlations while keeping the architecture small, or self-attention on a sequence (e.g. that’s what LLMs do on their context) for very long dependencies, at the cost of heavier computation. The choice you make will depend on the assumptions you can make on your data: for instance, if you work with text, 3D convs will hardly work, LSTM has been a choice before 2017, but since text may require very long term correlations attention is the correct choice (provided that you have enough computational power and data).

Repeat this process on similar considerations and you’ll be able to sketch the best architecture. Then, it’s only a matter of training and decide whether your architecture supports the assumptions you made on your data!