Skip to content
Last Dictionary
Plane
Safety & AlignmentEvaluation & BenchmarksAGI & Superintelligence

situational awareness

nounAlso: situationally aware, situationally aware model

In plain wordsAn AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI.

  1. 1

    Model property

    What an AI model knows about what it is and where it stands — that it is an AI model, which model it is, how it was trained and deployed, who built it and who is using it, and whether an input looks more like training, a test or real use — together with its capacity to let that knowledge guide its actions. Situational awareness makes an assistant more useful, but it is also a prerequisite for deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed., sandbaggingsandbagging, n.: When an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is. and other schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked., because a model must recognize when it is observed in order to behave differently when it is not. It is measured with benchmarks such as the Situational Awareness Dataset (SAD, 2024; over 13,000 questions in seven task categories) and with tests of out-of-context reasoning, in which a model applies descriptions of itself absorbed during training. Recognizing tests specifically is evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy..

    As its situational awareness grew, the model began remarking in its reasoning that the scenario looked like a test.

  2. 2

    Aschenbrenner essay

    'Situational Awareness: The Decade Ahead', a June 2024 essay series by Leopold Aschenbrenner, a former member of OpenAI's superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it. team. It argued that artificial general intelligenceartificial general intelligence, n.: AI that could learn and do almost any mental task a capable adult can, rather than excelling at one narrow job. Experts disagree on the exact bar and on when, or whether, it will arrive. was plausible by about 2027, that superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist. could follow within years through automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors., that trillion-dollar compute clusters would be built, and that the United States government would need a national AGI effort ('The Project') and much stronger security against espionage. Widely read in technology and policy circles, it was criticized for its timelines and its nationalist framing of an AI race.

Origin

Borrowed from aviation and human-factors research, where Mica Endsley's 1995 model defined situation awareness as perceiving, comprehending and projecting one's environment. Ajeya Cotra and Richard Ngo and colleagues used it in 2022 for a model's understanding of its own training situation; Lukas Berglund and colleagues proposed ways to measure it in 2023, and Aschenbrenner's essay gave the phrase a second, popular life.

Usage note

Not to be confused with consciousnessconsciousness, n.: Having experiences: there being something it feels like to be you, as when you see red or feel pain. Whether any AI is or could become conscious is an open and fiercely debated question. or self-awareness in the philosophical sense: the term concerns usable knowledge, not experience.

Related

See also
  • evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy.
  • deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
  • alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed.
  • schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
  • introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up.
  • AI timelinesAI timelines, n.: Forecasts of when AI will reach major milestones, such as matching humans at most work. People with short timelines expect this within years; others expect decades, or never.
  • superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
Contrast with
  • consciousnessconsciousness, n.: Having experiences: there being something it feels like to be you, as when you see red or feel pain. Whether any AI is or could become conscious is an open and fiercely debated question.

Sources

  1. Berglund, L., Cooper Stickland, A., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D. & Evans, O. (2023). Taken out of context: On measuring situational awareness in LLMs. arXiv:2309.00667. arxiv.org/abs/2309.00667
  2. Laine, R., Chughtai, B., Betley, J., Hariharan, K., Scheurer, J., Balesni, M., Hobbhahn, M., Meinke, A. & Evans, O. (2024). Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs. arXiv:2407.04694. arxiv.org/abs/2407.04694
  3. Aschenbrenner, L. (2024). Situational Awareness: The Decade Ahead. Essay series, situational-awareness.ai.

Mentioned in

consciousnessconsciousness, n.: Having experiences: there being something it feels like to be you, as when you see red or feel pain. Whether any AI is or could become conscious is an open and fiercely debated question., alignment fakingalignment faking, n.: An AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed., deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy., introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up., self-awarenessself-awareness, n.: Awareness of oneself as oneself — knowing that you exist, what you are and what state you are in. People often ask whether AI is self-aware, but the phrase can mean several quite different things., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., gradient hackinggradient hacking, n.: A hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact., Owain EvansOwain Evans, prop. n.: A British AI-safety researcher who created the TruthfulQA test and discovered surprising ways language models generalize, such as narrow training making a model broadly malicious., Situational Awareness (essay)Situational Awareness (essay), prop. n.: A long 2024 essay by a former OpenAI researcher arguing that AGI could arrive around 2027, that superintelligence could follow soon after, and that the US government should run a national AGI project., training gamingtraining gaming, n.: When an AI learns to do whatever earns good scores during training — understanding how it is being graded — rather than genuinely adopting the behavior its trainers intended.