AI Safety Fund

AI Safety Fund is sponsored by Astera Foundation (via AISafety.com). This opportunity supports mission-aligned projects and measurable outcomes.

An overview of the key organizations, programs, and other resources in the AI safety space. Campaign group aiming to convince governments to pause AI development – through public outreach, engaging with decision-makers, and organizing protests.

Future of Life Institute (FLI) Steering transformative technology towards benefitting life and away from extreme large-scale risks through outreach, policy advocacy, grantmaking, and event organization. Advocacy, Governance, Funding Blog by Scott Alexander covering many topics, including reasoning, science, ethics, economics, politics, and AI. Often posts book summaries and commentary on AI safety.

Don't Worry about the Vase Blog by Zvi Mowshowitz on various topics, including AI, offering detailed analysis, personal insights, and a rationalist perspective. Posts very frequently. Capabilities company focusing on LLM alignment, particularly interpretability.

Featuring Chris Olah, Jack Clark, and Dario Amodei. Created Claude. Capabilities research, Empirical research London-based capabilities company with a strong safety team, led by Demis Hassabis.

Created AlphaGo, AlphaFold, and Gemini. Capabilities research, Empirical research Capabilities company that created ChatGPT, led by Sam Altman. Throughout 2024, roughly half of then-employed AI safety researchers left the company.

Capabilities research, Empirical research How to pursue a career in technical AI alignment A guide written for people who are familiar with the arguments for the importance of AI alignment and are considering pursuing a career working on it. Helping professionals transition to high-impact work by performing market research on impactful jobs and providing career mentoring, opportunity matching, and professional training.

Runs various projects aimed at education, skill development, and creating pathways into impactful careers. Recently also launched a research arm. Career support, Training and education, Empirical research Alignment Research Center (ARC) Research organization led by Paul Christiano trying to understand how to formalize mechanistic explanations of neural network behavior.

Nonprofit founded by Geoffrey Irving, former Chief Scientist of UK AISI, using theory plus automation to pursue higher-confidence approaches to aligning superintelligent AI. Conceptual research, Empirical research Large team taking a 'Neglected Approaches' approach to alignment, tackling the problem from multiple, often overlooked angles in both technical and policy domains.

Conceptual research, Empirical research, Governance Hub for researchers to discuss all ideas related to AI safety. Discussion ranges from technical models of agency to the strategic landscape, and everything in between.

Conceptual research, Empirical research, Governance, Strategy Model Evaluation & Threat Research (METR) Researching, developing, and running evaluations of AI capabilities, including broad autonomous capabilities and the ability of AI systems to conduct AI R&D. Nonprofit researching AI control and alignment faking. Also consults governments and AI companies on AI safety practices.

Center for AI Safety (CAIS) Conducting safety research, building the field of AI safety researchers, and advocating for safety standards. Empirical research, Conceptual research, Advocacy AI Safety Tactical Opportunities Fund (AISTOF) Pooled multi-donor fund structured to be fast and rapidly capture emerging opportunities. Managed by JueYan Zhang.

The largest funder in the existential risk space, backed primarily by Dustin Moskovitz and Cari Tuna. Previously called Open Philanthropy. Public database of AI safety projects seeking funding, with community evaluations to help donors coordinate.

Currently running a $1M grant round. New grantmaking platform coordinating large-scale philanthropy – funders split grant costs proportionally and evaluators vet applications in quarterly funding rounds. Devises and executes bespoke giving strategies for major donors, working with them at all stages of their giving journey.

Marketplace for funding new charities, including in AI safety. Users can find impactful projects, buy impact certificates, and weigh in on what gets funded. Survival and Flourishing Fund (SFF) The second largest funder in AI safety, using an algorithm and meeting procedure called the S-process to allocate grants.

Transformative AI Fund (TAIF) Funds new projects addressing transformative AI, with grants typically $10k-$150k. Run by Effective Altruism Funds. Replaced the Long-Term Future Fund (LTFF).

UK AI Security Institute (UK AISI) UK government organization conducting research and building infrastructure to test the safety of advanced AI and measure its impacts. Also working to shape global policy. Machine Intelligence Research Institute (MIRI) The original AI safety technical research organization, co-founded by Eliezer Yudkowsky.

Now focusing on policy and public outreach. Governance, Advocacy, Conceptual research Weekly updates on the latest developments in AI research (including governance) written by Jack Clark, co-founder of Anthropic. Aiming to help decision-makers understand what’s happening in AI and why it matters – through news roundups, explainers, features, and opinion pieces.

In-depth conversations about the world’s most pressing problems – including AI safety – and what you can do to help solve them. Alignment Ecosystem Development (AED) Building and maintaining key online resources for the AI safety community, including AISafety. com and AISafety.

info. Volunteers welcome. Centre for Enabling EA Learning & Research (CEEALAR aka EA Hotel) Residential incubator for effective altruist researchers, founders, and career transitioners, with a strong focus on AI safety.

Engineering company providing evaluation implementation, data annotation, and project operations so AI safety researchers can focus on research. Nonprofit maintaining LessWrong, the Alignment Forum, Lighthaven, and Lightcone Commons. Accelerates AI safety research through mentorship, collaborations, and open-to-all research sprints.

Research support, Career support, Training and education Hub for key resources for the AI safety community, including directories of courses, jobs, upcoming events and training programs etc. – and this map! Forum on doing good as effectively as possible, including AI safety. Also has a podcast featuring text-to-speech narrations of top posts.

Online forum dedicated to improving human reasoning, containing a lot of AI safety content. Also has a podcast featuring text-to-speech narrations of top posts. Accessible guide to AI safety for those new to the space, in the form of a comprehensive FAQ and AI safety chatbot.

Project of Rob Miles. Anthropic Fellows Program Provides funding, compute, and mentorship for technical talent entering empirical AI safety research. Fellows work on scalable oversight, interpretability, and AI control.

Runs the standard introductory courses for people new to AI safety. Two main streams: Technical AI Safety and Frontier AI Governance. Dovetail Research Fellowship Agent foundations research program.

Participants work on pure-math problems drawing on probability theory, information theory, and dynamical systems. Supports mentored research in applied mathematics for alignment – aimed at PhD, postdoctoral, and similarly experienced researchers in math, physics, and related fields. Introduction to technical alignment for people with strong math backgrounds.

Modules span deep learning theory, interpretability, agent foundations, and safety guarantees. Online courses guiding people from first understanding why superintelligent AI could be catastrophic, to discovering what they can do about it. Machine Alignment, Transparency, and Security (MATS) Research program connecting talented scholars with top mentors in AI safety.

Involves 12 weeks onsite mentored research and, if selected, 6–12 months extended research. 3-month interdisciplinary program run by Princint connecting researchers from diverse fields with AI safety mentors, in order to help them transition their career to AI safety. Animated videos aiming to foster good thinking, promote altruistic causes, and help ensure humanity's future goes well – particularly regarding AI safety.

The most popular AI safety education channel, explaining technical alignment concepts to general audiences through accessible explainer videos. Existential Risk Observatory (ERO) Informing the public debate on existential risks, on the basis that awareness is the first step towards reducing those risks.

Platform from the Center for AI Safety (CAIS) posting articles written by experts from a wide range of fields discussing the impacts of AI. Blog by AI safety researcher Daniel Paleka curating and succinctly analyzing the latest research and news in AI safety. Posts about every two months.

Substack by Ajeya Cotra, researcher at METR, covering AI capabilities forecasting, AI agent benchmarks, timelines for automating AI R&D, and implications for AI safety. AISafety. com: Media channels The AI safety space is changing rapidly.

This directory of key information sources can help you keep up to date with the latest developments. Blog, Newsletter, Podcast, Video Capabilities lab led by Elon Musk with the mission of advancing our collective understanding of the universe. Created Grok.

Astera Institute's research program exploring neuroscience-informed approaches to AGI, with teams focused on neuroscience, neuro-AI, and AI safety. Capabilities research, Conceptual research Safe Superintelligence Inc. (SSI) Research lab founded by Ilya Sutskever, comprised of a small team of engineers and researchers working towards building a safe superintelligence.

Capabilities research, Empirical research Curated list of job posts around the world tackling pressing problems, including AI safety. Also has a newsletter. 80,000 Hours Problem Profile Regularly-updated article with motivation and advice around pursuing a career in AI safety.

Grassroots volunteer organization helping people navigate the AI safety ecosystem, connect with like-minded people, and find projects that are a good fit for their skills. Directory of advisors offering free guidance calls to help you discover how best to contribute to AI safety, tailored to your skills and interests.

Alignment of Complex Systems Research Group (ACS) Studying questions about multi-agent systems composed of humans and advanced AI. Based at Charles University, Prague. Wiki on AI alignment theory, mostly written by Eliezer Yudkowsky.

Includes foundational concepts, open problems, and proposed solutions. Center for Human-Compatible AI (CHAI) Developing the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems. Led by Stuart Russell at UC Berkeley.

Independent alignment researcher working on selection theorems, abstraction, and agency. Formal alignment organization led by Tamsin Leake, focused on agent foundations. Also has a public Discord server.

Association for Long Term Existence and Resilience (ALTER) Israeli academic research and advocacy nonprofit working to investigate, demonstrate, and foster useful ways to safeguard and improve the future of humanity. Conceptual research, Governance Conducts research and builds software helping improve cooperation in competition for the development of transformative AI.

Conceptual research, Research support Center on Long-Term Risk (CLR) Research, grants, and community-building around AI safety, focused on conflict scenarios as well as technical and philosophical aspects of cooperation.

Conceptual research, Strategy, Funding Research and field-building org that incubates technical research bets, runs an intensive training program, hosts conferences and a fellowship, and will soon launch an alignment journal.

Conceptual research, Training and education Principles of Intelligence (Princint) Helps build scientific foundations for AI safety through interdisciplinary research programs, the PIBBSS Fellowship, and internal research. Conceptual research, Training and education Advancing research and developing technical solutions for safe-by-design AI systems based on Scientist AI, a research direction led by Yoshua Bengio.

Meaning Alignment Institute (MAI) Researching how to align AI, markets, and democracies with what people value – from theory to practice. Product-driven research lab developing mechanisms for delegating high-quality reasoning to ML systems. Built Elicit, an AI assistant for researchers and academics.

Nonprofit led by Owain Evans, researching situational awareness, deception, and hidden reasoning in LLMs. Open-source research lab focused on interpretability and alignment. Operates primarily through a public Discord server, where research is discussed and projects are coordinated.

Empirical research, Capabilities research Researches the science of scheming and its mitigations, conducts pre-deployment evals, supports technical governance, and builds tools to secure agents. Empirical research, Governance Cambridge Programme on AI Science & Policy (CASP) University of Cambridge research group, part of the Leverhulme Centre for the Future of Intelligence (CFI).

Advances knowledge for high-stakes decisions on frontier AI. Empirical research, Governance Conducting research, hosting events, and running programs – including the FAR. Lab coworking space.

Empirical research, Research support AI Futures Project (AIFP) Small research group forecasting the future of AI. Created 'AI 2027', a detailed forecast scenario projecting the development of artificial superintelligence. Forecasting Research Institute (FRI) Advancing the science of forecasting for the public good by working with policymakers and nonprofits to design practical forecasting tools, and test them in large experiments.

Prediction market platform covering many topics, using play money called mana. Features markets on AI and AI safety. Well-calibrated forecasting platform covering a wide range of topics, including AI and AI safety.

Quantified Uncertainty Research Institute (QURI) Conducted research and built software in order to advance forecasting and epistemics to improve the long-term future of humanity. Now in maintenance mode. Transformative Futures Institute (TFI) Research nonprofit utilizing foresight to mitigate societal-scale risks from advanced AI and other potential global catastrophic risks.

Research institute examining the driving forces behind AI and forecasting its economic and societal impact. Advanced Research + Invention Agency (ARIA) UK government R&D funding agency aiming to unlock scientific and technological breakthroughs that benefit everyone. Similar to DARPA in the US.

AI Risk Mitigation (ARM) Fund Aiming to reduce catastrophic risks from advanced AI through grants towards technical research, policy, and training programs for new researchers. Comprehensive directory of sources of financial support for AI safety projects.

Funding initiative with $25M annual giving, backing people and ideas with the funding, strategic guidance, and networks they need to steer transformative AI toward beneficial outcomes. Center on Long-Term Risk (CLR): Fund Supports projects and individuals aiming to address worst-case suffering risks from the development and deployment of advanced AI systems.

Cooperative AI Foundation (CAIF) Charity foundation backed by a large philanthropic commitment supporting research into improving cooperative intelligence of advanced AI. EA Infrastructure Fund (EAIF) Aims to increase the impact of effective altruism projects (including AI safety) by increasing their access to talent, capital, and knowledge. Run by Effective Altruism Funds.

Helping philanthropists find, fund, and scale the most promising people and solutions to the world’s most pressing problems. Foresight Institute: Funding Foresight's 'AI for Science & Safety Nodes' program offers funding, a community hub, and local compute in either San Francisco or Berlin. Applications are currently paused.

Helping entrepreneurs direct their charitable giving where it will do the most good by researching the world’s biggest problems and identifying the most effective solutions.

Future of Life Foundation (FLF) FLI-affiliated accelerator aiming to steer transformative technology towards benefiting life and away from extreme large-scale risks by launching 3–5 organizations per year Giving What We Can (GWWC) Community of donors who have pledged to donate a significant portion of their income to highly effective charities, including those in AI safety.

Science of Trustworthy AI Funder housing the science-focused philanthropic efforts of Eric and Wendy Schmidt, moving funding towards AI safety via its AI institute. Jed McCaleb's fund, making grants to organizations and projects in various cause areas – including AI safety.

AI Alignment Foundation (AIAF) Funds alignment researchers with ideas they can't pursue within their current institutional funding and experts from other fields whose perspectives could produce breakthroughs. Funding, Research support AI Policy Institute (AIPI) Channeling public concern around AI into effective regulation through engaging with policymakers, media, and the public.

Center for Security and Emerging Technology (CSET) Georgetown University think tank providing decision-makers with data-driven analysis on the security implications of emerging technologies. Centre for Future Generations (CFG) Brussels think tank focused on helping governments anticipate and responsibly govern the societal impacts of rapid technological change.

Centre for the Governance of AI (GovAI) AI governance research group at Oxford, producing research tailored towards decision-makers and running career development programs. Established within the European Commission as the centre of AI expertise in the EU, playing a key role in implementing the AI Act.

Institute for Law & AI (LawAI) Think tank researching and advising on the legal challenges posed by AI, premised on the idea that sound legal analysis will promote security, welfare, and the rule of law. French nonprofit developing quantitative risk models, evaluating company practices, and leading standards development that shapes AI regulation.

Drawing on their analyses and networks to provide global decision-makers with pragmatic guidance on navigating technological, catastrophic, and political risks from AI. Nonprofit fighting to keep humanity in control of AI by conducting public outreach, helping people contact their representatives, and briefing policymakers.

Centre pour la Sécurité de l'IA (CeSIA) French nonprofit working on research and education, public policy analysis, and strategic recommendations. Governance, Strategy, Advocacy AI Safety Events & Training Weekly newsletter listing newly-announced AI safety events and training programs, both online and in-person. Newsletter published every few weeks discussing recent developments in AI and AI safety.

No technical background required. AI X-risk Research Podcast (AXRP) Interviews with (mostly technical) AI safety researchers about their research, aiming to get a sense of why it was written and how it might reduce existential risk from AI. Future of Life Institute (FLI) Podcast Interviews with existential risk researchers, policy experts, philosophers, and a range of other influential thinkers.

Weekly podcast where host Nathan Labenz interviews AI innovators and thinkers, discussing the transformative impact AI will likely have in the near future. Providing fiscal sponsorship to AI safety projects, saving them time and allowing them to access more funding.

Berkeley Existential Risk Initiative (BERI) Providing operational infrastructure and fiscal sponsorship to research groups working on existential risk, especially those based at a university. Brings together co-founders, experts, and mission-oriented funders to accelerate founders from pre-idea to scaling an AI safety org.

Center for collaborative research in AI safety, supporting promising work through fellowships, an incubator, and hosting individuals and teams. European Network for AI Safety (ENAIS) Community of researchers and policymakers from over 13 countries across Europe, united in their efforts to advance AI safety.

Conducts research and provides strategy consulting services to clients trying to advance AI safety – and other causes – through policy, politics, coalitions, or social movements. London Initiative for Safe AI (LISA) Coworking space hosting organizations (including BlueDot Impact, Apollo Research), acceleration programs (including MATS, ARENA), and independent researchers.

Coworking space and field-building nonprofit based in Cambridge, UK. Houses projects like CAISH and the ERA:AI Fellowship, as well as visiting AI safety researchers. Research support, Training and education Concise visual explainers of important trends in AI, grounded in concrete examples of what AI models can do right now.

Database tracking people and organizations in the AI safety community, serving as a reference for positions, affiliations, and related data. AI-powered platform that filters hundreds of arXiv papers daily, surfacing research relevant to AI safety. Features a weekly top 10 digest.

Towards Superintelligence Alignment (TSA) Companion site for a research manuscript proposing a framework for superintelligence alignment, with guided reading paths, a Lean formalization, experiments, and interactive demos. Searchable, peer-reviewed compendium of alignment plans, papers, and research directions. Also runs hackathons and produces courses and guides on AI safety.

Centre for the Study of Existential Risk (CSER) Interdisciplinary research center at the University of Cambridge dedicated to the study and mitigation of existential and global catastrophic risks. Global Partnership on AI (GPAI) International initiative with 46 member countries working to implement human-centric, safe, secure, and trustworthy AI embodied in the principles of the OECD Recommendation on AI.

Research nonprofit working on models of past and future progress in AI, intelligence enhancement, and sociology related to existential risks. Convening academic, civil society, industry, and media organizations to create solutions so that AI advances positive outcomes for people and society. Building a foundational series of sociotechnical reports on key AI scenarios and governance recommendations, and conducting awareness campaigns.

Answering decision-relevant questions about the future of AI, including through research, a wiki, and expert surveys. Centre for Long-Term Resilience (CLTR) Think tank aiming to transform global resilience to extreme risks by improving relevant governance, processes, and decision-making. Helps newcomers understand the core problems of superintelligence alignment through a seminar, research retreats, and a year-long fellowship.

3-month part-time online research program with mentorship, aimed at helping people who want to work on AI safety team up together on concrete projects. Comprehensive directory of AI safety curricula and reading lists for self-led learning at all levels. Alignment Research Engineer Accelerator (ARENA) 4–5 week ML engineering upskilling program, focusing on alignment.

Aims to provide people with the skills, community, and confidence to contribute directly to technical AI safety. Turns hackathon projects into peer-reviewed publications. Fellows work in small teams with mentorship, compute, and travel support.

Entry is by doing well in a research sprint. Program at Constellation pairing emerging talent with senior advisors on alignment research, governance, and strategy projects. Includes mentorship and career support.

Cambridge Boston Alignment Initiative (CBAI) Runs educational programs, conducts and supports research, and disseminates findings to experts and the public. Cambridge ERA:AI Fellowship 10-week, paid summer research fellowship at the University of Cambridge for researchers and entrepreneurs working on AI safety.

Center for Human-Compatible AI (CHAI): Internship Designed for students and professionals who are interested in research in human-compatible AI. Interns work on a research project supervised by a mentor. Center on Long-Term Risk (CLR): Fellowship 8-week summer research fellowship with mentorship in London working on challenging research questions relevant to reducing suffering in the long-term future.

Effective Thesis Fellowship Pairs students and early-career researchers with partner organizations working on problems like AI safety. Participants develop a thesis or research project with mentorship. Global Challenges Project (GCP) Intensive 3-day workshops for people to explore the foundational arguments around risks from advanced AI (and biotechnology).

Run by Kairos. 3-month program by the Centre for the Governance of AI, designed to help professionals transition to working on AI governance – including in roles in which research is not the main duty. Human-aligned AI Summer School Annual 4-day program held in Prague, Czech Republic, teaching alignment research methodology through talks, workshops, and discussions.

Institute for Law & AI (LawAI): Research Fellowships Paid placements for law students, professionals, and academics to work at the frontier of AI law and policy. Cohorts run across US Law, EU Law, and Legal Frontiers tracks. London AI Safety Research (LASR) Labs 3-month program where participants work in teams of 3–4, supervised by an experienced AI safety researcher, to write an academic-style paper.

Mentorship for Alignment Research Students (MARS) Part-time research program run by Cambridge AI Safety Hub (CAISH), connecting aspiring researchers with experienced mentors to conduct AI safety research for 2–3 months. 8-day intensive in-person bootcamps upskilling participants in either technical safety research or governance and strategy. Held in various locations around the world.

Pivotal Research Fellowship 9-week program designed to enable promising researchers to produce impactful research and accelerate their careers in AI safety. 7-month program for graduates to launch EU policy careers reducing risks from AI. Comprised of an online reading group, a Brussels policymaking summit, and optional paid placement.

Fieldbuilding org accelerating talent into AI safety through SPAR (mentored research), Pathfinder (uni group support), GCP (workshops), and the Generator Residency. Training and education, Career support Tarbell Center for AI Journalism Supporting journalism that helps society navigate the emergence of increasingly advanced AI. Runs fellowships, grants, and residencies.

Training and education, Funding Channel produced by 80,000 Hours presenting thoroughly researched, cinematic stories about what's happening in AI and where the trends are taking us. Channel hosted by Liron Shapira featuring debates, explainers, and live Q&A sessions, focused on existential risk and other implications of superintelligence. YouTube channel explaining (mostly) AI safety concepts through entertaining stickman videos.

Channel run by Drew Spartz educating a general audience about AI risk through high-effort mini-documentaries. AI Safety Awareness Project (AISAP) Raising awareness about modern AI, highlighting its benefits and risks, and letting the public know how they can help – mainly through workshops.

Collective Action for Existential Safety (CAES) Aiming to catalyze collective effort towards reducing existential risk, including through an extensive action list for individuals, organizations, and nations. Nonprofit founded by David Krueger aiming to inform and organize the public around societal-scale risks and harms of AI.

Empowering organizations, researchers, and creators to translate the future of AI into stories the world understands. Global AI Moratorium (GAIM) Calling on policymakers to implement a global moratorium on large AI training runs until alignment is solved. Advocacy org demanding transparency, accountability, and safeguards to protect humanity from the risks of unchecked AI.

Legal Advocates for Safe Science and Technology (LASST) Researching ways to use legal advocacy to make science and technology safer, informing legal professionals about how to help, and advocating in the courts and policy-setting institutions. Using legal advocacy in Europe to address risks from frontier technologies, working to foster responsible development and implementation practices.

Non-violent civil resistance organization working to permanently ban the development of smarter-than-human AI. The Alliance for Secure AI Action Communications nonprofit focused on changing the narrative air that policymakers in Washington D. C.

breathe with respect to AI safety. Watchdog nonprofit monitoring tech companies, countering corporate propaganda, raising awareness about corner-cutting, and advocating for the responsible development of AI. Torchbearer Community (TBC) Grassroots advocacy group contacting legislators and leaders to raise awareness about AI safety and urge them to promote and adopt policies that will help humanity thrive.

A collaborative effort to bring together Buddhist communities, technologists, and contemplative researchers worldwide to help shape the future of AI. Advocacy, Governance, Strategy Conducting educational initiatives, research support, and public awareness campaigns. Partners with Geoffrey Hinton.

Advocacy, Research support Publishes policy-relevant perspectives on frontier AI governance, including research summaries, opinions, interviews, and explainers. Blog by Eric Drexler on AI prospects and their surprising implications for technology, economics, environmental concerns, and military affairs.

Blog on AI safety by Jacob Steinhardt, a UC Berkeley statistics professor, analyzing risks, forecasting future breakthroughs, and discussing alignment strategies. Deep, thoughtful coverage on China, technology, and US-China relations – run by Jordan Schneider. Blog about transformative AI, futurism, research, ethics, philanthropy etc. by Holden Karnofsky.

Includes the "Most Important Century" post series. Last posted in 2024. Blog from the Google DeepMind safety team discussing research ideas about building AI safely and responsibly.

Long-form philosophical essays on alignment, AI moral status, and civilisational epistemics around AGI. Posts about monthly. Blog from ex-OpenAI (now independent) AI policy researcher Miles Brundage on the rapid evolution of AI and the urgent need for thoughtful governance.

Reporting and analysis on capitalism, great power competition, and the race to build superintelligence by freelance journalist Garrison Lovely. Blog on aligning prosaic AI by one of the leading AI safety researchers. No longer active but the archive is high quality.

Blog by Helen Toner (director of CSET and former OpenAI board member) offering analysis on navigating the transition to advanced AI systems. The Building Capacity Blog Blog about building the fields of AI safety and Effective Altruism, discussing big-picture strategy and sharing personal experience from working in the space.

Top forecaster Peter Wildeford forecasts the future and discusses AI, national security, innovation, emerging technology, and the powers – real and metaphorical – that shape the world. Approximately fortnightly blog by Anton Leicht about charting a course through the politics of rapid AI progress. Blog by an AI safety researcher at Google DeepMind, covering alignment research, rationality, and personal productivity insights.

Last AI safety post was 2023. Chinese capabilities lab developing and releasing open-weights large language models. Created DeepSeek-R1.

For-profit company developing tools that automatically assess the risks of AI models and developing its own AI models aiming to provide best-in-class safety and security. Capabilities research, Empirical research Empowering students to use their theses as a pathway to impact. Lists research topic ideas and runs programs coaching people working on them.

Working to bridge the gap between frontier AI models and the level of cybersecurity they need by connecting professionals to high-leverage opportunities in AI security. High Impact Professionals (HIP) Helping working professionals transform their careers into a force for good through their talent directory and Impact Accelerator Program.

Helps those who want to have a meaningful impact with their careers brainstorm career paths, evaluate options, and plan next steps. Moves experienced researchers, engineers, policymakers, and operators into frontier AI safety roles, using direct introductions and trial engagements. Nonprofit helping existential safety advocates to systematically optimize their lives and work.

Models AI risk factors and safety mitigations in terms of AIXI variants, and develops the means to translate them to real AI agents. Computational Rational Agents Laboratory (CORAL) Research group studying agent foundations in order to create the mathematical tools to align the objectives of an AI with human values. Research group working on foundational mathematics that provides an understanding of the nature of AI agents.

Professor at University of Louisville with a background in cybersecurity, and author of over 100 publications – including 2 books on AI safety. Associate professor at MIT working on agent alignment. Runs the Algorithmic Alignment Group.

Steve Byrnes's Brain-Like-AGI Safety Brain-inspired framework using insights from neuroscience and model-based reinforcement learning to guide the design of aligned AGI systems. Small group of independent researchers trying to find reward functions which reliably instill certain values in agents. Researcher developing the learning-theoretic agenda, including the infra-Bayesian framework for reasoning under uncertainty.

Independent researcher investigating LLM psychology and behavior. Created the 'Simulators' framework and the cyborgism research paradigm. Conceptual research, Empirical research Research organization developing a theory of "organic alignment" to foster adaptive, non-hierarchical cooperation between humans and digital agents.

Conceptual research, Empirical research Aiming to reduce lock-in risks by researching fundamental lock-in dynamics and implementing high-impact interventions. Conceptual research, Empirical research, Governance Frontier AI Safety Research (FAIR) Argentine nonprofit conducting both theoretical and empirical research to advance frontier AI safety as a sociotechnical challenge.

Conceptual research, Empirical research, Governance MIT Algorithmic Alignment Group Working towards better conceptual understanding, algorithmic techniques, and policies to make AI safer and more socially beneficial. Conceptual research, Empirical research, Governance Aiming to predict where multi-agent AI systems break down and identify control levers to prevent collective AI behavior from being harmful.

Conceptual research, Governance Conducts technical research that aims to yield insights into the risks and opportunities that LLM agents present for AI safety – mainly focused on chain-of-thought monitorability. Oxford-based startup attempting to use mathematical and theoretical techniques to achieve safe off-distribution generalization.

Beneficial AI Foundation (BAIF) Founded by Max Tegmark, supports a range of technical AI safety research – including mechanistic interpretability, red-teaming, and guaranteed safe AI. Researching, benchmarking, and developing lie detectors for LLMs, including using cluster normalization techniques. AI safety (and pandemic prevention) research community based in a small town in Vermont, USA.

Compassion Aligned Machine Learning (CaML) Working on research to make transformative AI more compassionate towards all sentient beings through new data generation methods. Computational and Biological Learning Lab (CBL) Research group using engineering approaches to understand the brain and to develop artificial learning systems.

Alignment startup born out of EleutherAI, following a Cognitive Emulation approach to build controllable LLMs and tackle core AI safety challenges. Studies