Grants for Research on the Science of Trustworthy AI

Grants for Research on the Science of Trustworthy AI is sponsored by Schmidt Sciences. Supports research on the science of trustworthy AI, focusing on verification, testing, adversarial attack and defense, and interpretability.

Science of Trustworthy AI - Schmidt Sciences Science of Trustworthy AI Advancing the Fundamental Science of Trustworthy AI The science of trustworthy AI is nascent. Today, many demonstrations of unsafe behavior are isolated, and mitigations do not generalize reliably across model families, deployment contexts, or capability regimes.

Therefore, we aim to support basic technical research that improves our ability to understand, predict, and control risks from frontier AI systems while simultaneously enabling their trustworthy deployment. Every day, AI technology is becoming more consequential. As a result, the impact of safety failures are potentially incredibly harmful.

We need mature decision-relevant evaluations. Today’s evaluations often fail exactly where we need them most: under distribution shift, long-horizon interaction, tool use, and optimization pressure. Many tests are brittle, highly correlated, or easy to “train to”, and stylized scenarios can be misleading if they do not reflect deployment-like contexts.

We need a rigorous science of evaluation with construct validity, predictive validity, and clear evidence standards for when results justify real-world decisions. The research capacity needed for trustworthy AI lags the pace of deployment.

Frontier systems are being deployed rapidly, but the infrastructure for trustworthy assessment and oversight is not keeping pace—especially for large, ambitious projects that require substantial compute, multidisciplinary expertise, and time. Accelerating progress will require sustained support for high-ambition, field-shaping research rather than incremental work. Academics are underleveraged in trustworthy AI research.

Currently, safety research for the largest AI models is primarily conducted by leading AI labs. Still, despite vast private capital flowing into AI development, commercial incentives for foundational, pre-product safety science are often weaker than incentives for capability and product improvements, especially when benefits are diffuse or long-horizon.

Foundational advances in the science of trustworthy AI therefore function as a global public good, motivating targeted philanthropic support for academic and nonprofit researchers.

Deepen our understanding of safety properties of AI systems Build a rigorous science of evaluation with construct and predictive validity Advance trustworthy AI approaches resistant to obsolescence from fast-evolving technology Support a global community of researchers advancing the science of trustworthy AI Our research agenda organizes priorities around three connected aims: Characterize and forecast misalignment in frontier AI systems Why frontier AI training-and-deployment safety stacks still result in models learning effective goals that fail under distribution shift, pressure, or extended interaction.

Develop generalizable measurement and intervention Advance the science of evaluations with decision-relevant construct and predictive validity, and develop interventions that control what AI systems learn (not just what they say).

Oversee AI systems with superhuman capabilities and address multi-agent risks Develop oversight and control methods for settings where direct human evaluation of correctness or safety isn’t feasible, and address risks that emerge from interacting AI systems.

Featured Projects: Understanding Safety in AI Systems Tests of Compositional Generalization as an “Upper Bound” on AI Safety Risks Unlocking Multi-Agent System Safety with Dynamic Islands of Trust LLM-Derived Guardrail for Frontier Models as a Step Towards a Cautious Scientist AI Robust LLM-based Scoring of Agent Alignment Deception & Misinformation: Elicitation and Testing A Meta-analysis Approach to Understanding LM Capabilities Long-Term Safety Behavior of Agents Mechanistic Interpretability to Detect Test Set Contamination Benchmarks for AI Agents and Cybersecurity Adaptive Stress Testing for Automated Unsupervised Large Language Model Testing and Failure Classification Understanding the Mechanism of Adversarial Transfer Across AI Models Beyond Simple Scaling: A Multi-Dimensional Family of Scaling Laws for Interventions in Pre-Training and Model Safety Optimization What Counts as Contamination?

How Generalization Could Confound Evaluation Mechanistic Cognitive Alignment for Morally Guided Decision-Making Multiagent-Based T&E Environment Construction A Conformal Safety Assurance Framework for Large Language Models Formal Verification of Software Robustness and Controllability of Language Model-Based Agents Built for Automating Tasks in Software Engineering Five Nines of Reliability for AI Agents Multi-Agent AI Safety via Dynamic Games OpenAgentSafety: Measuring and Mitigating Safety Harms of LLM-based AI Agent Interactions Exploring AI Safety in Free-Form, Evolutionary Multi-Agent Systems Evaluating and Defending Multimodal Computer-Use Agents under Adversarial Attacks with a Realistic Online Environment Formalizing Prompt Injection Defenses BenchCraft: An AI-powered Toolkit to Create Valid and Efficient Benchmarks with Measurement Theory and Human-AI Collaboration Quantifying and Mitigating Privacy Risks in Multi-Agent Systems Tracing and Eliminating Harmful Capabilities Across Model Generations Feature Projects: Inference-Time Compute A Dual-Framework for Grounding AI Reasoning: Measuring Faithfulness via Code Execution and Debugging Causally Grounded Inference-Time Intervention for Robust Model Alignment Emerging Risks from Inference-Time Compute Evaluating Preparedness via Model Organism Spectrum Faithfulness and Safety of Latent Chain-of-Thought Reasoning Fundamental Limitations of the Test Time Compute Paradigm Generalization and Hidden Tendencies in LLMs Improving the Legibility of Inference-Time Reasoning for Scalable Oversight INSPECT: Inference-time Safety for Physically Embodied Chain-of-Thought in Robotics Investigating Hawthorne Effects in Large Reasoning Models Large Language Model Safety in Inference-Time Motivated Reasoning Preventing Reward Hacking at Inference-Time with Reinforcement Learning Quantifying, Attributing, and Improving Faithfulness in Chains of Thought Reasoning with Foresight: Safe Inference via Q-Value Guided Decoding Safely Confident Reasoning Safety in RL-Enabled Goal-Directed Agents Toward Safe Reasoning: Test-Time Unlearning of Sensitive Knowledge Beyond Final Answers Utility Engineering and Moral Scaffolding for Safer Reasoning Models Will Chain-of-Thought Monitoring Significantly Improve Safety?

Reasoning Up the Instruction Ladder: Inference-Time Deliberation for Safer AI Current Advisory Board Members Associate Professor of Computer Science Technical Staff at Open AI Senior Program Officer at Open Philanthropy Fund Manager, AI Safety Tactical Opportunities Fund More AI programs and initiatives Science of Trustworthy AI Accelerate Programme for Scientific Discovery The technical storage or access that is used exclusively for statistical purposes.

The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.

The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Manage {vendor_count} vendors