MATS Research — AI Safety Research Expenses (2025)

MATS Research — AI Safety Research Expenses (2025) is sponsored by Open Philanthropy. Funding to support research projects undertaken during the winter 2024-2025 ML Alignment & Theory Scholars (MATS) cohort, focusing on AI alignment, interpretability, and governance.

Open Philanthropy Technical AI Safety RFP - $40M Available Across 21 Research Areas — EA Forum Has Open Phil (or others) conducted a comprehensive analysis for both understanding and building the AI safety field? If yes, could you share some leads to add to my research? If not, would Open Phil consider funding such work?

(either under the above or other funds) Here is a recent example: Introducing SyDFAIS: A Systemic Design Framework for AI Safety Field-Building I'm new to applying for an AIS grant, so I have some common questions that might have been answered elsewhere: (1) what are some failure modes that I might need to consider when writing a proposal, specifically for a research project?

(2) will research expenses include stipends for the researchers? (3) can I write a grant to do a research project with my university AI safety group? I'm not sure if this will be considered a field-building or a technical AI safety grant.

Some common failure modes: Not reading the eligibility criteria Not clearly distinguishing your project from prior work on the topic you're interested in Not demonstrating a good understanding of prior work (would be good to read some/all of the papers we link to in this doc for whatever section you're applying within) Not demonstrating that you/your team has prior experience doing ML projects.

If you don't have such experience, then it's good to work with/be mentored by someone who does. "Research expeneses" does not include stipends, but you can apply for a project grant, which does. If you're looking for money to spend on ML experiments or to pay people who are spending their time doing ML research, then that may fall within this RFP.

If you're looking for money to do other things (e.g. reading groups, events, etc), then that may fall under the capacity-building team's RFPs.

Open Philanthropy Technical AI Safety RFP - $40M Available Across 21 Research Areas Opportunities to take action Announcements and updates Opportunities to take action Announcements and updates Open Philanthropy is launching a big new Request for Proposals for technical AI safety research , with plans to fund roughly $40M in grants over the next 5 months, and available funding for substantially more depending on application quality.

Applications start with a simple 300 word expression of interest and are open until April 15, 2025.

We're seeking proposals across 21 different research areas, organized into five broad categories: Adversarial Machine Learning *Jailbreaks and unintentional misalignment *Backdoors and other alignment stress tests *Alternatives to adversarial training Exploring sophisticated misbehavior of LLMs *Experiments on alignment faking *Encoded reasoning in CoT and inter-model communication Evaluating whether models can hide dangerous behaviors Reward hacking of human oversight Applications of white-box techniques Finding feature representations Toy models for interpretability Interpretability benchmarks More transparent architectures Trust from first principles White-box estimation of rare misbehavior Theoretical study of inductive biases Alternative approaches to mitigating AI risks Conceptual clarity about risks from powerful AI New moonshots for aligning superintelligence We’re willing to make a range of types of grants including: Research expenses (compute, APIs, etc.) Discrete research projects (typically lasting 6-24 months) Academic start-up packages Support for existing nonprofits Funding to start new research organizations or new teams at existing organizations.

The full RFP provides much more detail on each research area, including eligibility criteria, example projects, and nice-to-haves. Please email [email protected] with questions, or just submit an EOI.

Curated and popular this week Population ethics is a big deal, which is why I made this population ethics quiz MichaelDickens · 1w ago · Curated 2d ago · 5 m read · 1w ago · Curated 2d ago · 5 m read TLDR: Take the population ethics quiz here: https://mdickens. me/pop-ethics/ Population ethics is an oft-overlooked subfield within ethics.

Many people hold views that they don't realize contradict each other, or that have strange implications that they wouldn't endorse if they thought about it more. Not just that—population ethics is a BIG DEAL. A lot of ethical decisions hinge on how you think about changes in future populations....

Alone again: facing the possibility of AI takeover as the world sleepwalks on George Rosenfeld · 6d ago · 6 m read I’ve been feeling pretty shaken since the METR report about the Hugging Face incident came out last week. Over the weekend, I wrote up some thoughts on how lonely the AI situation sometimes feels to me.

It’s more personal than what I’d usually share publicly, but I thought I’d post it here in case it resonates with anyone else. I’m very grateful to the man... Do AI safety talent programmes work?

Nobody has run the study that would tell us. Laura Thomas-Walters · 2d ago · 8 m read Headline finding: I audited 17 AI Safety Talent programmes. Zero of 17 have published any comparison group, rejected-applicant follow-up, matched control or randomisation.

Not one. Every programme that mentions a counterfactual does it by asking participants to self-report. Recent opportunities to take action EA Organisation Updates thread: September 2026 Dane Valerie · 10h ago · 1 m read Help us find and support the world's most talented, altruistic teenagers - and help them find you!

Jonah Boucher · 23h ago · 4 m read Kairos has raised $50M to build talent infrastructure for AI safety (and we're hiring!) Agustín Covarrubias 🔸 , Topaz , Kairos Team · 1w ago · 10 m read Agustín Covarrubias 🔸 , Topaz , Kairos Team + 2 more