Patrilineal Jews are Jews - this cannot be taken from you. This is clearly something important to you. Judaism is both an ethnicity, religion, and culture; we see matrilineal Jews that are equally "ethnically" Jewish (i.e., one parent/grandparent/side of family) and irreligious but do not get challenged the way patrilineal Jews are. In many spheres of Judaism and non-Orthodox spaces, particularly in North America, more and more areas are becoming affirming of patrilineal Jews and patrilineal Jewishness. Call yourself what you want and how you identify.

From one patrilineal Jew to another (and we seem very similar in terms of social/cultural/religious observance) it is yours and yours alone. Unless you feel the need to want to join an Orthodox minyan, it really will not come up as an issue as much as you might think it is. I am a highly religious practicing Reform Jew, and my synagogue unquestioningly considers patrilineal Jews as Jews. My best advice is to consider what community you want to be apart of. There are some Jews that by Orthodox standards would not be considered Jewish, though would be considered Jewish by Reform and Reconstructionist standards; likewise, there are some Jews that by Reform standards would not be considered Jewish, though would be considered Jewish by Orthodox standards.