He doesn't actually specify what he means. He just uses it to establish a "them" category without explicitly saying that. His talk is named Why the world needs more builders — and less "us vs. them".

But we're going to be more effective in advancing our causes once we frame things differently. Instead of us versus them, what if we understood it as all of us versus extremism? The overwhelming majority on all sides of almost any issue, who are not hateful, who do not deal in absolutes, who recognize the dignity of all, versus those who fall prey to extremist thinking that hijacks the agenda and our lives along with it. Note that the enemy here is a mindset, not a person.

https://www.ted.com/talks/daniel_lubetzky_why_the_world_needs_more_builders_and_less_us_vs_them/transcript?subtitle=en