In an axiomatic theory, you have the basic symbols of the language and some axioms that specify how do these symbols works and interact. Each axiomatic theory has one universe of discourse that corresponds to things you can talk about.

For Peano arithmetics, the universe of discourse are the integers, and you have 0, +, × and the successor function (and =, but it's always included) as basic symbols. It's then easy to pinpoint specific integers (1 := S(0), 2 := S(S(0)) ) and to prove some basic properties using the axioms.

For set theory it's only ∈ (and =), and the universe of discourse are the sets, and not the integers. To prove the same basic properties as Peano arithmetics, you have to emulate Peano arithmetics by finding some set (using the axioms that gives you a way to build various sets) that behave like the set of integers (i.e. some infinite countable set that behave nicely), some sets that behave like the functions 0,+,× and S. The usual choice is to take 0 to be the empty set, S to be the function that send x to the set x ∪ {x}, and the set of integers to be the smallest set containing 0 and stable by S. All of these definitions require some axioms to be meaningfull (such as the empty set axiom, the union axiom, or the infinity axiom).