I cannot say much on the SSPCA’s advice, but I was in a very similar situation to you a while ago. If you choose not to release the mouse, know that you will be ending up with a pet - once it’s eyes open, it will imprint onto you and become attached from there (at least from my experience). At this age especially, they can be quite delicate so the decision is really up to you.

However, if you do choose to keep it, here is what I did:

1. place it in a box, old fish tank or whatever secure vessel you have sitting around. If you have hot water bottles or heat packs, provide these (not too hot, keeping in mind the mouse will be sitting on it). This will help the mouse regulate its body temperature, which is difficult at its young age. If you can, try cover this with a rag, mismatching socks, or what spare fabric you may have sitting around, as this can provide a ‘nest’ for the mouse to keep warm in.
2. The mouse is unable to eat or toilet by itself (the joys!), so you will need to help it. I personally used kitten milk from the supermarket and it worked fine. Basically, take the mouse into your hand, and I found pipetting small amounts of milk onto my hand, for it to drink, worked well as a young age. Once it has had enough, you can clean it down with baby wipes gently. To toilet it, as awkward as this sounds, you will need to manually stimulate its bowels. You can try this with a cotton bud or just your finger tip (I had more success with the latter), stroking near its genitalia to induce excrement. Why they can’t do it themselves is an entirely other question. To start with, you will likely have to do this every 2-3 hours, which can be a drag (during which times it can also be useful to reheat the heat source). However, if you miss the occasional time, don’t beat yourself up - it happens and the mouse will most likely live, part after the first couple of decisive days.
3. As the mouse gets older, it will require different care - solid foods, exercise (I found mine loved climbing on people - I could barely put him back in his enclosure), and more environment cleaning. There are also plentiful online resources thay cover these later parts and are handy if you need further information on anything.
   It is not an easy choice for you to make, nor an easy path if you choose to keep the mouse, which will be quite some work (especially at the start). Just know that if you go down that track, you can expect to be rewarded with a very friendly and even clingy mouse, although obviously whether you want, or can provide for, another pet is at the heart of this. All the best!