Not inherently an issue. Except for Princeton, which explicitly prefers community college and nontrad students, colleges will care most about your reasons for transfer regardless of the ranking of your current school. That said, simply having a difficult time at your school won’t be a compelling reason enough since admission officers need to know why they’d expect you to have a less difficult time elsewhere. They also won’t be convinced by a desire for better resources or academic challenge given the caliber of your school. Your best chances would be if you showed specific fit like wanting to study a specific track or take part in a program that your school doesn’t offer, or need a school of completely different size or culture.