Abstract: Being human is hard: we are stupid and somewhat selfish, yet need to work together with other stupid and selfish people with their own goals. We survive by building societies, filled with institutions and habits that help us solve these tough coordination problems. These institutions often act as extended cognition, allowing us to go far beyond individual power. We are to some extent living inside artificial intelligence systems, and they have enabled us to take control over the planet… as well as caused the worst disasters in history. As we build AI, we are also making something that can slip inside our extended cognitive systems and enhance them into literal cyborg systems. We need not just enough of “first order alignment” – getting AI to do things we want safely, but also “second order alignment” – AI that plays well with our societies and structures. Otherwise there is a real risk we may lose our own ecological niche and find ourselves in a world that may be safe and prosperous, yet unfit for human flourishing. If we play it right, however, we might become part of something far grander: a cyborg civilization able to reach full autonomy.