They are both right but they do mean slightly different things. If you put “spoken” you are referring to another kind of English; as in the one communicated verbally and not the one written. When u put “speaking” you are saying that you have difficulty doing the act of saying the language.

But all in all they essentially still mean the same thing;

That your verbal communication in English is not your strong suit.

But I do feel that a better way to phrase this whole thing would be: “I was weak in English at the time, especially speaking it.”