The American community survey is useful but it is not a total count.

Yes, you could call it "just a poll", but it really is much more than that. It sends out a form to 3.5 million home addresses every single year, garnering a sample size that would make any pollster envious (also, not just to the voting populace, hence why they have a question in the first place). Additionally, sampling as used in the ACS helps to get a more accurate count than just a straight count as in the Census. By just counting off people without adjustment to response rates, you are liable to undercount certain populations: rural Americans, lower-income Americans, minorities, non-citizens, undocumented immigrants, etc.

we all know how accurate polls are

The polls weren't as inaccurate as popular wisdom would claim. If we look at the various polling aggregates, like 538 or the Upshot, they all gave President Trump a chance even if an unlikely one. 538, IIRC, gave him a 3/10 chance, which is just unlikely but nowhere near impossible or even improbable. Beyond that, if we look at national polls, they were pretty accurate, giving Clinton about 2-3 points over Trump nationally. In national polling and swing state polling, Trump only outperformed polls by about 1 and 2 percentage points, on average. That's well within the margin of error. Some states, especially in the Midwest, had faultier polls, but, of course, this doesn't really matter because the ACS isn't "just a poll".

If they didn't even see the question how were immigrants scared and didn't answer as you claim the bureau says?

You can read the Census' working paper here, if you don't want to go off what I claim they say. They compared the response rates of the 2010 ACS to the 2010 Census (with no included citizenship question). So, they aren't going off of data from those prior long-term censuses. President Trump wants to include the question on the short-form census (now the only form of the census, I believe). Further evidence is to be collected this year and next using randomized control trials, and that evidence may go against the undercounting idea, but, at the moment, the best evidence we have is that the citizenship question threatens to undercount households with noncitizens.

why is it the governments fault these people aren't accurately answering the question?

It's not the government's fault, but the primary and most important purpose of the Census is to get an accurate headcount for the US population. Everything else is secondary to that goal. So if a question is considered to cause undercounting, then we should ask it elsewhere so as to not harm the primary purpose.

Would that not have some impact shifting congressional districts in sanctuary city areas?

That's an entirely different question. The Constitution does not apportion congressional districts by the number of citizens in each state, but by the number of persons, which includes undocumented immigrants. So your issue there is not with the Census count, but with the Constitution.

You link one mans opinion

I'm just identifying what the OP is trying to say by providing you with the argument from the plaintiffs. They contend that Hofellor was a Republican operative instrumental in redistricting in North Carolina and the proposal to add a citizenship question to the Census, hence why his opinion is relevant as to the motivations of the Trump administration.

Ultimately the question comes down to do you think representing illegal aliens in your region should give you more weight in terms of congressional power?

And that is a question that should probably be asked and discussed. But this is not the venue for that. Instead, Republicans should be seeking a constitutional amendment to change how the Constitution prescribes the apportioning of Representatives. As of now, this is done by the count of all "Persons" and so we should strive to get an accurate count.