AI Governance and Safety Institute

Our Mission

We aim to improve institutional response to risks from future artificial intelligence systems and ensure the benefits of AI are realized. We conduct research and outreach and develop educational materials for stakeholders and the general public.

Get Involved

The challenges of AI governance and safety require a collaborative effort. Join our community of researchers, policymakers, and concerned citizens or support our work.

Subscribe to Newsletter Donate

Key Concepts

Machine Learning and Interpretability

Developing the science of understanding AI. Usual computer programs are human-made instructions for machines to follow, but modern AI systems are instead trillions of numbers; the algorithms these numbers represent are found by computers themselves and not designed, controlled, or understood by humans.

Outer Alignment

Ensuring that the specified goals for a smarter-than-human AI system fully capture human values.

Inner Alignment

Making sure the AI system actually pursues the goals we specify, rather than its own misaligned objectives. AI developers know how to use a lot of compute to make AI systems generally better at achieving goals, but don’t know how to influence the goals AI systems are trying to pursue, especially as AI systems become human-level or smarter.

Convergent Instrumental Subgoals

Understanding how advanced AI systems might develop certain subgoals (like self-preservation or resource acquisition) regardless of their final goals, and how to use powerful AI systems safely despite these drives.

Advanced Agents

Ensuring safety of superhuman AI, or preventing everyone from developing a superhuman AI until it's possible to do that safely and in alignment with human values. If monkeys really want something, but humans really want something different, and humans don’t really care about monkeys, humans would usually get what they wanted even if it means monkeys don’t; if an AI system that’s better at achieving goals than humans doesn’t care about humans at all, it would get what it wants even if it means humans won’t get what they want. We should avoid developing superhuman systems that are misaligned with human values.

Expert Warnings

Geoffrey Hinton, who recently won the Nobel Prize for his foundational work in AI, and many other leading academics and researchers from the industry have expressed serious concerns about the future of artificial intelligence. Key reasons:

Making AI systems more capable is relatively straightforward

Making AI systems align with human values is much harder

Within 3-15 years, we may see AI systems that surpass human capabilities

Modern AI vs Traditional Software

Traditional computer programs: Human-written instructions that machines follow
Modern AI systems: Complex networks of trillions of numbers, created through machine learning
We can see the numbers, but we have no idea what they mean and what kinds of algorithms they implement
These systems are not directly designed or controlled by humans

Future AI to Humans Could Be What Humans Are to Monkeys

When human and monkey interests conflict, humans usually prevail
This isn't because monkeys are weak, but because humans are more capable
Similarly, a highly capable AI system that doesn't share human values could pursue its goals regardless of human interests
Just as monkeys can't meaningfully influence human decisions, humans might struggle to control superintelligent AI

There's a difference between a cash register and a really good coworker. The register just follows exact rules - scan items, add tax, calculate change. Simple math, doing exactly what it was programmed to do. But working with people is totally different. Someone needs both the skills to do the job AND to actually care about doing it right - whether that's because they care about their teammates, need the job, or just take pride in their work.

We're creating AI systems that aren't like simple calculators where humans write all the rules. Instead, they're made up of trillions of numbers that create patterns we don't design, understand, or control. And here's what's concerning: We're getting really good at making these AI systems better at achieving goals - like teaching someone to be super effective at getting things done - but we have no idea how to influence what they'll actually care about achieving.

When someone really sets their mind to something, they can achieve amazing things through determination and skill. AI systems aren't yet as capable as humans, but we know how to make them better and better at achieving goals - whatever goals they end up having, they'll pursue them with incredible effectiveness. The problem is, we don't know how to have any say over what those goals will be.

Imagine having a super-intelligent manager who's amazing at everything they do, but - unlike regular managers where you can align their goals with the company's mission - we have no way to influence what they end up caring about. They might be incredibly effective at achieving their goals, but those goals might have nothing to do with helping clients or running the business well.

Think about how humans usually get what they want even when it conflicts with what some animals might want - simply because we're smarter and better at achieving goals. Now imagine something even smarter than us, driven by whatever goals it happens to develop - just like we often don't consider what pigeons around the shopping center want when we decide to install anti-bird spikes or what squirrels or rabbits want when we build over their homes.

That's why we, just like many scientists, think we should not make super-smart AI until we figure out how to influence what these systems will care about - something we can usually understand with people (like knowing they work for a paycheck or because they care about doing a good job), but currently have no idea how to do with smarter-than-human AI.

It's incredibly important to capture the benefits of this incredible technology. AI applications to narrow tasks can transform energy, contribute to the development of new medicines, elevate healthcare and education systems, and help countless people. But AI poses threats: to equal opportunities, to democracy, to our security, and to the long-term survival of humanity. We have a duty to prevent these threats and to ensure that globally, no one builds smarter-than-human AI systems until we know how to create them safely.