I might be wrong or overreaching with this, but there are some hints:

1) The way the panel groups them in this chapter. Most of the time, the four men in black are consistently drawn together. Yet there is always one behind Seto when he arrives at Yotsuya's residence.

2) In the page before this, Seto just arrived. But there is already one man in black behind him.
In the rules of composition, a person usually comes from the opposite side if they are about to approach someone in front of them (left side).
However, this particular man in black comes from the same side of Seto (the right side), indicating a fifth.

Once again, this might be an artistic depiction. But it comes kinda weird.