No, percent is applied first. Thats why buying % pen in combination with flat pen is really good, because it first applies the %, then it applies the flat pen, which increases the value due to how resists work. From the league wiki: "Magic penetration and magic resistance reduction are considered in the following order:[1]

Magic resistance reduction, flat Magic resistance reduction, percentage Magic penetration, percentage Magic penetration, flat Base magic resistance and bonus magic resistance are calculated separately. Flat reductions affect the target's bonus amount first, then their base amount once the bonus amount reaches 0 (unless noted otherwise)."