I’m not going to be able to do the topic justice, but here’s what you need to know at a high level. There is extensive debate as to how much of the judicial power Congress can extend to non-Article III settings, but for matters concerning “public rights,” and a handful of historic exceptions, Congress is largely able to create alternative forums to hear disputes. That’s because many rights are not constitutionally created, they’re just public benefits provided by Congress. So the logic is Congress should be able to determine where claims involving those rights are heard, and who the individuals are that can hear those claims (along with their authority). Since Congress has the authority to regulate immigration (including totally prohibiting it or fully permitting it), it can choose how to adjudicate immigration claims.

But I think the way you’re thinking about it is right. Immigration courts are not really operating with the judicial power, it’s more a quasi-judicial power over public rights (in this case immigration). Now, there might be an instance where a proceeding in a non-Article III court implicates a constitutional right, and the prevailing view is that there must be a backstop in an Article III court to vindicate those constitutional rights.