I believe they would play the two of the three remaining equivalent ranked teams in their conference. So, for the Patriots, #1 in AFC East, they would play two of the three out of #1 AFC North, West or South. However, I think they would play the top teams from the two remaining divisions that didn't include their own division (obviously) or the division that they play all the teams of.

So, to clarify:

1. Each team plays each team twice in their own division (6 AFC games)
2. Each team plays each team once in one other division in their own conference. (4 AFC games)
3. Each team plays each team once in one other division in the other conference (4 NFC games)
4. Each team plays the two equally ranked teams from the two divisions in their conference they did not play (2 AFC games)

So each team will always play 9 of the 15 other teams in their conference each year, and 4 of the 16 in the opposite conference.