Technical AI Safety Research is sponsored by Coefficient Giving (formerly Open Philanthropy). Funds technical AI safety research, with a focus on areas offering high leverage for improving understanding and control of AI, including robust unlearning, exploring sophisticated misbehavior in large language models (LLMs), and alternatives to adversarial training.

Three positive updates I made about technical grantmaking at Coefficient Giving (fka Open Phil) — LessWrong World Optimization Personal Blog Three positive updates I made about technical grantmaking at Coefficient Giving (fka Open Phil) Open Philanthropy’s Coefficient Giving’s Technical AI Safety team is hiring grantmakers .

I thought this would be a good moment to share some positive updates about the role that I’ve made since I joined the team a year ago. tl;dr: I think this role is more impactful and more enjoyable than I anticipated when I started, and I think more people should consider applying.

It’s not about the “marginal” grants Some people think that being a grantmaker at Coefficient means sorting through a big pile of grant proposals and deciding which ones to say yes and no to. As a result, they think that the only impact at stake is how good our decisions are about marginal grants, since all the excellent grants are no-brainers. But grantmakers don’t just evaluate proposals; we elicit them.

I spend the majority of my time trying to figure out how to get better proposals into our pipeline: writing RFPs that describe the research projects we want to fund, or pitching promising researchers on AI safety research agendas, or steering applicants to better-targeted or more ambitious proposals.

Maybe more importantly, cG’s technical AI safety grantmaking strategy is currently underdeveloped, and even junior grantmakers can help develop it. If there's something you wish we were doing, there's a good chance that the reason we're not doing it is that we don't have enough capacity to think about it much, or lack the right expertise to tell good proposals from bad.

If you join cG and want to prioritize that work, there's a good chance you'll be able to make a lot of work happen in that area. How this cashes out is: as our team has tripled headcount in the past year, we’ve also ~tripled the amount of grants we’re making, and we think the distribution of impact per dollar of our grantmaking has stayed about the same.

That is, we’ve about tripled the amount of grant money we’ve moved towards the top end of the impact distribution as well as at the marginal end. To be even more concrete, here’s one anecdote I can share. About a year ago, Jason Gross asked me for $10k for compute for an experiment he was running.

I spoke to him a few times and encouraged him to make grander plans. The resulting conversations between him, me, and Rajashree Agrawal led to me giving them a $1M grant to try to found something ambitious in the formal software verification space (I’m reasonably excited about FSV as a def/acc + mitigating reward hacking play.)

They eventually founded Theorem , a startup focussed on formal software verification, which went on to be the first FSV startup accepted to YC, and they subsequently raised at one of the largest valuations in their cohort. Jason and Rajashree say that they would have been very unlikely to set their goals that big without my initial grant. Nothing about that seems marginal to me, yet it wouldn’t have happened had I not been here.

There is no counterfactual grantmaker When I was offered the job a little over a year ago, I was told that I was the only candidate still being considered for the role, and that there was no one left to make offers to if I didn’t accept. In our current hiring round , we’d like to hire 3-4 technical AI safety grantmakers, but once again it’s far from obvious that we’ll find enough candidates that meet our bar.

If you get an offer and don’t take it, the likeliest result is that we hire one fewer person. Why is this? I think the main reason is that fewer people apply to our roles than you might expect (if you’ve already applied, thank you!)

We are looking for people who could succeed in a research career, and most such people don’t want to leave research. It also helps for this role if you are well networked and have a lot of context on technical AI safety. Most people with a lot of context are settled in their roles and unlikely to apply.

Separately, the set of skills required to be a good grantmaker includes some things that aren’t as important for being a good researcher, so occasionally strong researchers who apply have disqualifying qualities, even people who on paper seemed like they might be really good. What this all means is that our top candidates end up being extremely counterfactual.

Their acceptance or rejection of the role doesn't just improve outcomes very slightly relative to some other person we could have hired, but counterfactually causes tens of millions of dollars to move out the door to really impactful projects that wouldn't have otherwise been funded. If we're so starved for grantmaker labor, why don't we lower our hiring bar?

I think we’re going to have a slightly lower bar than we’ve had in the past; we really want to fill these roles. But also, we think there are diffuse long-term negative effects of seriously lowering our hiring bar. I acknowledge that perhaps we're making the wrong tradeoffs here.

(If you feel moved to apply by the counterfactual argument, but would drop out if it turns out that we think we have enough other good applicants, please feel free to indicate that in your application. If we get an unexpected windfall of strong applicants, such that we have more qualified candidates than we can hire, we’ll be happy to let you know, and there will be no hard feelings if you drop out.)

Grantmaking is more fun/motivating than I anticipated Before I joined OpenPhil, I was about as “research archetype” as they get. I spent most of my time thinking about wacky theory math ideas . My work style was chaotic-academia: I went to bed at random times and worked at random times and in random places, mostly on whatever interested me at the time.

Now I have a team and a manager, and I have lots of things that need to be done. I am not planning to have any papers with my name on them in the foreseeable future. But, I'm really enjoying it!

So why am I enjoying it more than you might expect, and indeed indeed more than I expected going in? Some factors: I do actually spend a decent fraction of my time thinking about object-level technical stuff. Mostly that looks like talking to top researchers, but it also looks like reading academic papers, and going to conferences to talk to people about their research or argue about AI safety.

That can be a big part of the job if you want it to be. One thing that's nice about the technical roles at cG is that we have a lot of generalist grantmakers but not a lot of technical specialists. That means that people are focused on leveraging the technical expertise of those who have it efficiently, which means that I get to spend a larger than expected fraction of my time on technical stuff.

For example, with large grants that have a significant technical and non-technical component to the investigations, I often pair with a generalist grantmaker who will take the non-technical aspects off my hands. I get to spend more time thinking about which research agendas are promising, and less time worrying about whether an organisation has a healthy board of directors etc, than I expected.

The work I'm doing feels important and tangible. I go to conferences and see people walking around and giving talks who wouldn’t be in the room without my grant. A promising junior person mentions they just got a job at an org I funded to grow its headcount.

Maybe it's cliche, but I actually do think that seeing the effects of my work on the shape of the field, is pretty motivating. I'm significantly more empowered than I expected to be when I joined. I've been given much more trust than I expected, and I've been empowered to make decisions based on my inside view.

My manager is constantly pushing me to take on bigger projects, be more ambitious and creative, and be more agentic. As a result, I think I have become more ambitious and agentic, and noticing that in myself has been very motivating. I think if you think that a more agentic, ambitious version of yourself is someone you'd like to grow into, then this might be a good role for you, even if you're not sure how well that will go yet.

This role can be very sociable. I spend a lot of my time talking to researchers about the research they're doing and why. I don't get as much time to spend on getting into the low-level technicalities, at least not in my day-to-day, but I actually find that high-level strategic thinking which still interfaces with technical details can scratch much of the same itch as doing the research myself.

I also think that thinking through strategic questions about technical AI safety and the future of AI are extremely interesting questions. If all this sounds appealing to you, you can apply here by December 1st ! Our team funds a lot of great research – from scaling up research orgs like Redwood or Apollo , to eliciting projects like those described in our RFP , to proactively seeding new initiatives.

Last year, the Technical AI Safety team made $40 million in grants; this year it’ll be over $140 million. We want to scale further in 2026, but right now we only have three grant investigators on the team, so we’re often bottlenecked by our grantmaker bandwidth. If you think you might be a strong fit, your application could be the difference between us finding the right person or leaving a role unfilled.

If you have more questions, you can dm me on LW or reach out to me at jake [dot] mendel [at] coefficientgiving [dot] org Three positive updates I made about technical grantmaking at Coefficient Giving (fka Open Phil) 20 plex World Optimization Personal Blog 3 comments , sorted by top scoring Click to highlight new comments since: Today at 9:53 PM In my three calls with cG following my post which was fairly critical of them (and almost all the other grantmakers) I've updated to something like: cG is institutionally capable of funding the kinds of things the people who have strong technical models of the hard parts of alignment think might be helpful.

They mostly don't because most of the cG grantmakers don't have those technical models (though some have a fair amount of the picture, including Jake who is doing this hiring round). My guess as to why they don't is partly normal organizational inertia things, but plausibly mostly because the kinds of conversations that would be needed to change that don't happen very easily.

Most of the people who are talking to them are trying to get money for specific things, hence the conversation is not super clean for general purpose information transfer, as one party has an extremely strong interest in the outcome of the object level.

Also, most of the people who have the kinds of models of technical I think are needed to make good calls are not super good at passing the ITT of prosaic empirical stuff, so the cG grantmakers probably feel frustrated and won't rate the incoming models highly enough.

My guess is getting a single cG grantmaker who deeply gets it, has grounded confidence and a type of truth-seeking that will hold up even if people around you disagree, and can engage flexibly and with good humor to convey the models that a bunch of the most experienced people around here hold would not just something like double the amount of really well directed dollars, but also maybe shift other things in cG for the better.

I've sent them the list of my top ~10 picks and reached out to them. Many don't want to drop out of research or other roles entirely, but would be interested in a re-granting program, which seems like a best of both worlds. (I am also a grantmaker at Coefficient/OP) The arguments/evidence in this post seem true and underrated to me, and I think more people should come work with us.

In particular, I also have updated upward on how impactful the job is over the last year. It does really seem to me like each grantmaker enables a ton of good projects. Here’s an attempt to make more concrete how much is enabled by additional grantmakers: If Jake hadn’t joined OP, I think we would our interp/theory grants would have been fewer in number and less impactful, because I don’t know those areas nearly as well as Jake does.

Jake’s superior knowledge improves our grantmaking in these areas in multiple ways: Better sourcing: Jake’s involvement meant that the proposals in these areas that were even available to us to evaluate were much better.

His contributions to the interp/theory sections of the RFP meant the incoming proposals were higher-quality than if I had attempted to write them, and he had good suggestions/steers for grant applicants that I couldn’t have offered. He was also able to proactively ideate and realize projects that I wouldn’t have thought of or wouldn’t have had time for.

More grants, in more varied subareas: because Jake knows those areas better, he can evaluate proposals faster and is more comfortable arguing for/defending these grants than I am. This allows us to make more, and more varied, grants in those areas. [The obvious one] Jake has better discernment among proposals in these areas than I do, which straightforwardly increases the impact of our grantmaking.

I think there are probably more buckets of similar scale/impact grantmaking to interp and theory that we’re currently neglecting. We need to hire more people to open up these new vistas of TAIS grantmaking, each of which will contain not just mediocre/marginal grants, but also some real gems!

I think this dynamic is often underappreciated; additional grantmakers take ownership for new areas, rather than just helping us make better choices on the margin. I also think that Jake obviously had way more impact on theory/interp than if he had done direct work. He funded dozens of projects by capable researchers, many of whom wouldn’t have worked on AI safety otherwise.

I think most TAIS researchers aren’t taking this nearly seriously enough, and I think the case for grantmaking roles looks very strong in light of this. I am so happy to read this post, knowing Jake is working on this makes me glad. For sure I will be nudging more people I know to apply.

I hope Coefficient gets really high quality applicants in this round. In fact whenever I am asking for help to pivot towards contributing to the field I use the OP's technical AIS RFP as a resource to get people to think about what kind of experiments they would want to do or upskill to be able to do. so "But grantmakers don’t just evaluate proposals; we elicit them."

is really powerful and true. If there's something you wish we were doing, there's a good chance that the reason we're not doing it is that we don't have enough capacity to think about it much, or lack the right expertise to tell good proposals from bad. If you join cG and want to prioritize that work, there's a good chance you'll be able to make a lot of work happen in that area.

This definitely makes the role highly impactful and the main update I had from this is that it is more fun/motivating than you would have imagined. Thanks for sharing. Curated and popular this week 3 What it's like to be an AI safety grantmaker (and why we need more of them)