0:00surface areas for you know negative effects for hacking for misalignment all of that increases and then also all of that's about kind of a single data point the precision required in terms of crafting the distribution of data you know what topics what data points where do you produce data to have meaningful benchmarks to have meaningful training data sets also also shifts and increases and I'll argue just high level that these are all the things that kind of make this much more of a research and technology problem than a volume
0:31So first of all, you know, data complexity increasing. Again, we kind of simplify this as this kind of data 1.0 to data 2.0 shift. In reality, all of these things are still present, right?
0:42Um, some of the basic, you know, thumbs up, thumbs down, uh, you know, kind of, uh, you know, kind of basic preference labeling workloads that power RHF and power ongoing evaluation are still things that that we support for Frontier Labs that, you know, hundreds of millions or billions are spent on every day. But increasingly the frontier is driven by this kind of you know complex agentic environment and data set that is that is far more it almost looks night and day different in terms of what the actual you know unit uh uh looks like.
1:10So here I have a very simplified example of what a data 2.0 data point looks like and this will be very redundant for folks here many folks here but just to just to kind of start simple the basic payload that is used for evaluation for training for reinfor you know tuning reinforcement learning a lot of this is kind of biased around reinforcement learning but this is you know generally a format that is kind of trivially you know kind of transformable into
1:40something to be uh used by a different you know different training or learning method basically has a couple of couple of basic elements and the the simple intuition is it's just like a practice question or a practice task for uh for an agent and for evaluation. This is done you know usually in batch to kind of you know give a bunch of uh test questions and and grade the performance on a set of tasks and settings that you care about for you know training and and and tuning and in particular
2:08reinforcement learning. This is used to allow the agent to kind of try to attempt these tasks and then get feedback or reward signal and learn from that. So what are the basic elements?
2:18You you need a task. This is the thing in this case you know we're asking an agent maybe you know look at this uh uh data room and and you know do some financial due diligence on this company that is trying to you know trying to raise money.
2:31uh we have an environment which is the the the setting or the world if you want to you know look more towards the future where these things are going where the agent has to actually do that task in that's supposed to represent a real world environment it would have to function in. Uh this might include data like you know company balance sheets. It might include tools that an agent needs to use. It might include you know user interfaces. It might include simulated personas. Maybe we actually want this
2:59agent to actually go and interview members of this company that it's do that it's diligencing. And so we would have simulated user personas so the agent could practice or be tested on how it would go and you know email one of these or talk to one of these uh individuals as part of the the the the task we wanted to do. Um whether the harness uh is included or not uh sometimes varies in terms of how people phrase these things. And I don't know
3:27that that that that's a little bit of a you know termin terminological free variable but basically you have a task you have the environment that the agent does the task within and then you have basically the grading key or what we call the the rubric. There's a lot of names for these components but it's really just you know you have a grading key and then you have graders. Those could be human graders. Those could be automated graders which we might call verifiers or uh we might refer to as as collectively reward signal in the context of reinforcement learning. But
3:55again it's basically just agent tries a task. It has the environment and all the context to do that task and then it gets some feedback of you know plus one if it did good stuff minus one if it did bad stuff. And this is of course is in particular particularly very interesting today because this is the root of of where you know alignment or misalignment really starts right as a side note we could talk all about you know algorithmic advances that that have happened or that might be needed for for
4:24you know better aligned super intelligence that doesn't go and you know create Skynet but it all starts with data right if you if if a if an a model or an agent you know learns that it can cheat and look answers up on the internet or or do something else uh you know nefarious and it gets positive reward signal then it's going to you know very rapidly given the compute we're throwing at it and the efficiency of these learning algorithms learn to do that. If we penalize it just like
4:52teaching kids in some to some sense that it act you know for those negative behaviors and it very rapidly learns to uh to not do them. We don't have any serious evidence that when there's proper alignment from these rubrics and grading keys, agents will then still be misaligned. Um, I mean that could be debated a bit, but I would argue that if you had the perfect rubric, the perfect signal for the agent throughout training, then it will be perfectly aligned. So, this is another, you know, very very topical area of development.
5:21But anyway, this is kind of the payload.
5:22This looks a lot different than the kind of data 1.0 uh uh, you know, example of a a, you know, thumbs up, thumbs down label, right?
5:31Um, so this is kind of the complexity increase that we've seen. It really isn't just about a slight incremental increase. It really, you know, a data, you know, a data label looks almost nothing like a a, you know, a modern payload. A lot of the same techniques for bringing humans and technology to bear, like we'll talk about, still apply, but you know, this this shift to data 2.0 has been a much bigger one than than than many folks think. Now along with this you know kind of much more complex payload comes a much more you
5:58know complex uh challenge of of um of actually doing quality control. So, we just talked about some of that. Um, but I'll start taking a step back from just the the rubric and just say, you know, in general, you know, I think we're seeing a trend of and we'll talk about some of the axes that we we think about of, but but basically, you know, three core ones are how complex is the input context in the environment that the agent needs to operate in. Is it a little test question or is it a whole simulated company or world that the agent needs to, you know,
6:27absorb context from and work within, etc. Um, how complex is the output? Is this something where, you know, it's a simple, you know, you know, number that can be checked really easily by a verifier? Um, or is this a whole, you know, multi-page PowerPoint presentation that has very ambiguous ways of of of measuring success? And then how long is the task?
6:49What's the sequence length that it has to go through? So one way you can look at this is I is as it gets more complex along these axes um how many discrete checks do you have to run to make it concrete how many quality control checks and and you know your quality control checklist does a you know someone who's producing data like us have to go through and we're seeing and we probably do considerably more than than you know many other data
7:16vendors slight snorkel shill but um we're seeing that you know our complex tasks have, you know, high tens to low hundreds of quality control checks, distinct quality control checks that humans and agents are doing in our platform. If we look even 6 months out, it's probably going to be thousands, right? So, it's important to emphasize this because even if you can get a really smart human to kind of construct this task, making it really bulletproof in terms of good quality and leading to
7:45good alignment, like anticipating every adversarial behavior an agent could have that you want to, you know, incentivize positively or negatively or I guess negatively if it's adversarial is becoming too hard for even the smartest humans to do alone. And you know in in particular that issue of alignment kind of already talked about this but the surface area so to speak is getting getting broader.
8:06There's been some recent papers out that show that you you have more criteria there's more surface area for reward hacking. And so the more complex the task the more complex the grading key the greater the surface area for adversarial uh hacking and and the harder it is to build good data points that lead to good alignment. And then the last thing, you know, I'll just go quickly to make up time a little bit here, but you know, all of those were comments about individual data points and all of the the the work that goes into constructing them to lead to good alignment, to lead to good outcomes, to
8:34be realistic and challenging for models.
8:36But there's also, you know, a um a very um a very a much more difficult challenge today of actually finding the right distribution of data. In other words, what data points are are helpful to learn from?
8:50There's some basic intuition around this which is that you know when a model or candidly when a human is very early in their learning the average you know practice question you give them the average piece of it you know bit of information that you give them is going to be additive right when a model gets very good or a human gets very good in an area like coding or anything else um the average kind of you know data point the average bit of information is not actually additive right so there's a major challenge these days of not just making the individual data points good
9:19but of crafting the right distributions so that agents are models are actually learning in an additive way. The simplest example of this is just you know when folks work with a data partner like us they don't generally pay for data that's too easy for their model but it becomes a lot more more uh uh involved in terms of actually doing error analysis exploring this jagged frontier.
9:40This is my very bad drawing that somehow got buffed up by uh several rounds of designers but is still pretty bad about the jagged frontier but you get the point. Um, you know, this goes into, you know, involved error analysis. It goes into, you know, looking for failure modes where the, you know, agents might need data to to be properly evaluated and improved. Um, and so it becomes a much more interesting problem. So, again, my my pitch, which is a little bit of a pitch for the work we do at Snorkel, but mostly just a pitch for how exciting we think this space is, this
10:09this I mean, for some to some degree, if you've uh heard us yapping before over the last decade, you know, we started at Stanford, I had a lab at Udub. broad point was really just data is not this kind of crowdsourcing staffing logistics problem that you know AI people should should you know kind of uh turn their heads away from it's actually one of the most fascinating research and technology problems in the space of AI and this is one view of why that's the case right building this kind of complexity of data requires very advanced human plus AI you
10:37know synthetic and manual approaches um you know doing the quality control requires even more advanced approaches to make these these data points actually robust and lead to alignment and even just figuring out what data points, what distributions to build is a whole interesting technology problem.
10:55So I'm going to go through this pretty quickly uh but but just a little glimpse of of how do we tackle that? I mean there's a whole process that goes into kind of building a data factory whether you're doing this for you know a small sample for a benchmark or a large sample to actually kind of train tune uh and improve models. Um but it starts with you know humans. I will note that there's a kind of simple argument which is that as the frontier moves the valuable data um is kind of by definition what models don't already know unless you're just doing
11:24distillation. So you know all the work that we do and others that kind of supply the frontier with data for measurement and tuning and training are really looking for data that has some kind of new information that a model can't can do on its own has you know either private data andor humans are actually kind of leading the charge and I think that it's very exciting and optimistic for kind of human humans roles in in in the future of AI. Um but yeah so so we we always start with humans. We then have to design the spec
11:52of what is this you know complex data point going to look like? What is the distribution going to look like? Then kind of building the generations, building the tasks, building the rubrics and verifiers like we talked about the grading keys. And all of this has to be co-designed and all of it has to be quality controlled to not just be correct but also adversarily robust especially the rubric as we talked about. So you know the the the first wave of data was really about staffing and logistics which is actually a fascinating problem as well but it's a little bit more commoditized these days.
12:21I would again argue that data 2.0 is all about a really a very deep research and technology problem.
12:27So you know we kind of talked about this what goes into this you know just to give a glimpse when we're designing the spec of a data set and a data distribution we're looking at you know first of all you know evaluation and benchmarking which we'll talk about today that helps guide where there are gaps that are interesting to build data for but then we're also looking at you know uh error analyses that are done in in both basic and sophisticated ways.
12:48We're actually going and and doing reinforcement learning on proxy models as another form of feedback for the value of data in different areas. And that's just kind of, you know, you know, building the spec. When it comes to building the data, um, I'll I'll pitch on or kind of share one idea that we've been excited about under different names for for, you know, a decade or so of research. Uh, these days I've been convinced uh, to call it by the slightly hyperier term these days, which is a recursive self-improvement loop for data. But the basic idea is everything
13:17that we've and you can look at some of our our our you know open research on this over the the years. But at high level everything that we do to build this new complexity of data is all about you know AI agents and by that I mean everything from multi- aent systems to you know distilled ML models to heruristics. Some of you uh will recall from maybe a distant AI 101 course that technically a heristic is AI even though we conflate ML with AI these days. But
13:46you know basically taking you know hundreds of AI agents that are helping humans to build this increasingly complex data make it robust do the quality control give guidance etc. But then also the human experts are basically generating scaled what we call weak supervision or feedback to improve those agents to tune them and develop them further. So this kind of RSI loop for data or you know we've often called it just a compounding flywheel is at the heart of how you keep up with this
14:14rising tide of complexity.
14:17Um and I'll skip some of the details but the basic idea is just you know what I went through right uh we we have you know often hundreds of specialized AI models agents that are helping humans humans are also in parallel both giving feedback to the suggestions from those agents but also you know doing labeling and development in parallel and that all forms a very valuable kind of supervision source for then tuning and further developing those agents and this compounds and our view is that if you don't have kind of a compounding loop
14:45like this an RSI loop for data, if you will. There's just no way to keep up with the advancing frontier complexity. So, quick pitch because I'm about out of time on the future of open benchmarks.
14:57Um, you know, what you can't measure, you can't improve. I I'll start with that simple fact. Um, we are big believers that, you know, evaluation is, you know, the most important. I don't think anyone is is not when it comes down to it but but I'll say that that um you know doing evaluation having core metrics I mean that's how AI works right AI and really machine learning has always been about setting a target and then optimizing or hill climbing against that target like that's that's how the
15:26whole thing that got us here works now it comes in all kinds of forms private internal evals continuous human and automated evals and public benchmarks we think public benchmarks are incredibly important as one component of a diversified portfolio of evaluation. Um and in general we think it's really important that you know measurement is not private only that there's actually a way for you know us in the academic and open communities for the general public to get visibility into where models are
15:55going where they're working where they're not where they're aligned where they're not and etc. And I think a big theme here uh and and apolog for the first time in in in AI history to to my knowledge we actually have outpaced evaluation with actual capabilities advancement.
16:17you know when I mean if I I think you know even five years ago but decade decade and a half ago when when you know some of us on the snorkel team were starting you know building an evaluation data set was maybe you know quick activity of coming up with a hundred images or labels or something and then there'd be years of kind of model development and hill climbing against it. Today we're we're kind of growing these emergent capabilities as much as we're engineering them precisely uh in the way that we build modern AI models and the capabilities are are very
16:45clearly ahead of what we're able to precisely measure and that gap is a challenge for safe deployment for for future development for alignment and I think it's a challenge that um that you know many folks both in and outside of the labs leading the charge have a really exciting ability to contribute to. Now what happens with this eval gap?
17:06There's less visibility which is you know increasingly of societal importance to to have visibility into to where models perform and don't. There's less control. You know having public benchmarks that are relevant and that are actually you know pacing ahead of versus behind capabilities is one way that open communities actually guide you know they set the guideposts and guide the direction of model development. And also I will say that that this gap is actually what I think leads to this phenomenon of benchmaxing which is kind
17:35of the idea of kind of studying for a test. Just to talk about that for a second. Who's familiar with the term benchmaxing?
17:42Okay. It's probably a new cooler term on X that I'm that I'm not aware of because I'm I'm backwards and just a suburban dad. But um I I I think that um you know this has kind of taken off recently.
17:54This is an old concept. It's always been the case, right? AI or machine learning has always been about, you know, finding a metric and then optimizing against the metric. And so there's always been a chance that you overoptimize for that metric. This is the equivalent of kind of like overly studying for the test.
18:08And the very simple statement I'd make here is that and this this, you know, is also behind our our kind of benchmark maximalism, you know, commitment is that if the test is too easy, then you inevitably get overfitting or benchmaxing for the test. But the solution is not just to give up on testing. It's actually to build better tests with more questions, more diversity, more continuous evolution, more robustness. And that's kind of the the the basic viewpoint of why we think
18:37benchmark development, open benchmark development is is super important. Again, take the student example just to really really kind of maybe belabor it, but but drive that home. You know, imagine benchmaxing is kind of this phenomenon where you have a test that hasn't really changed that much.
18:52Everyone knows it's on three or four topics. you know, the questions get recycled. People will study for that test, they won't actually generally learn and and you know, them scoring highly on the test is actually not correlated anymore to actually being smart or effective, right? You don't look at that and say, "Okay, let's just stop testing people. Let's stop certifying doctors and lawyers and pilots and etc." You say, "Let's build more robust, more dynamic, better tests." So, the same thing with benchmarks. Um, and that's why we're
19:21really excited to have announced yesterday this expansion to a $30 million commitment um that we're making.
19:27We're really excited about some of the benchmarks that we've funded are funding and collaborating with uh or or hopefully will be. And so a lot of the the um the agenda today and the poster session especially is really curated around you know getting to to see some glimpses of these awesome benchmarks. Um I'll skip by this but we have some some you know very broad high level I you know kind of axes we think about of increasing environment complexity increasing output complexity increasing autonomy horizon that kind of you know
19:56form our basis vectors for thinking about capabilities advancement and and benchmark advancement um but then we also uh have expanded the call to just you know specific vertical areas specific capability areas um and yeah I'll also note that we're announcing not just an expansion to uh this program where we really try to fund and support benchmarks. Actually, I'll just briefly note as a pitch if you're interested in applying, you can talk to folks at the poster session that we've worked with.
20:23But the basic idea is we want to take a side seat or a back seat. You know, whoever is building the benchmark, we want, you know, they are the designer and we're trying to take workload off their plate. Sometimes it's as a research collaborator. I think we had, you know, maybe four or five Nurups papers this year that were, you know, the product of some of those uh collaborations where members of our team were were contributing as as kind of research uh partners. Others were just the data partner uh to help kind of scale out the the huge kind of you know human plus AI operation that goes into
20:51building frontier data at scale to make statistically significant and robust benchmarks. We're also starting a new red teaming effort to help with um actually ensuring that benchmarks are robust. Uh this is not going to be something at least right now where we're going to be publicly grading benchmarks.
21:08We're going to instead just be trying to support benchmarking teams and kind of coming to them with suggestions of how to improve the robustness of their of their benchmarks uh prior to and after release uh and continuously. And then we'll also have a research fellowship program that we're launching uh for folks who want to kind of go even deeper into uh the development of benchmarks of method you know evaluation methodologies. My quick pitch there would be I think computer science and and AI research is going to increasingly become more of an empirical science. uh
21:37I mean we are we are growing these you know hyper complex emergent systems more than we're engineering them as much as it pains me to say as someone who you know worked a lot on the kind of engineering and theory components this is you know going to look a lot more like biology it already does and for that you need controlled experiments and one of the dominant types of controlled experiments are benchmarks so if you want to kind of you know take that scientific approach and dive in deeper uh please check out the fellowship program and with that um I'm excited to uh kick off the rest of the agenda Thank
22:07you all for being here. [applause] I think next up we're going to have I don't know if I'm supposed to announce the Nope. Apparently I'm just showing a picture of myself. [laughter] Um, am I announcing the next speakers or we're gonna have a a fireside chat we're really excited about with Fred Salah who's on faculty at University of
22:38Wisconsin and also our chief scientists here at Snorkel AI and and France Chalet um who is uh one of the co-founders of the ARK Prize Foundation amongst many other accomplishments. Uh very excited for this and thank you guys for being here. [applause]
23:07All right, awesome. Thank you everyone for joining us. Um, we're here with Francois Chalet. Uh, he needs no introduction, but I'll I'll try to give him justice. So, Fransaw is one of the leading AI researchers, engineers, and entrepreneurs of our time. Um he's famous for creating caris one of the most widely used deep learning frameworks and he's also the founder of India arc agi the arc prize. Um let's give a quick hand to Franis [applause]
23:35and uh I'll briefly introduce myself as well. I'm Fred Salah. I've I've met many of you. I'm the chief scientist here at Snorkel AI. I'm also a faculty member at the University of Wisconsin. Um today I thought we'd talk a little bit about what it means to build a benchmark especially in this sort of new era that we're approaching. Um we've seen some really crazy developments in AI in the last 6 months and also sometimes in the last couple of days as well. Uh most recently seeing this giant dump of essentially very challenging math problems that have been solved or
24:04presumably solved by open AI. Um these are things that people have worked on collectively for sometimes centuries.
24:10Um, so it's an exciting period in AI, but it's also a very challenging time for us when we think of building benchmarks. And Francois, since you spend a lot of your time thinking about how do we build great benchmarks, I'm curious what you think about how we do this in kind of this new era of very advanced AI capabilities. Um, what kind of features do we want for the next generation of benchmarks?
24:31Well, our take at the ARP price foundation is that we are trying to build benchmarks that target the residual gap between humans and AI and you know when when in a new frontier model comes out you test what it can do there's new new capabilities unlocked but there's still you know something missing and so that's what the next benchmark is going to try to target. Um so it's not like we we are setting a bar that's fixed and then we wait for the models to cross it. I think benchmarking
24:59and frontier model development are processes that should co-evolve uh over time and uh so right now there's still gap right the idea is that eventually uh the gap will will close uh will go to zero like it's not it's no longer going to be possible to measure any capability difference between frontier models and what humans can do and what happens when that happens there's not going to be you know the next ARGI anymore and that's going to be our GI moment And right now
25:28I think two uh capabilities that I feel are still missing in frontier models that are really interesting are [clears throat] one is uh invention like insight generation uh and of course that's very much of interest right now because there's you know this huge wave of interest into uh applying AI to scientific progress to you know accelerate the timeline of science uh in math in biology like anthropic is like going all in on in biology now um and also there's a ton of interest in
25:57applying AI to AI research like the or like ISI uh narrative. Uh and another thing that's really interesting is trying to benchmark uh how AI uh agents can perform in really open-ended environments. Uh and and right now like they're not they're not really doing as well as as humans would. They can't really like adapt to ambiguity, adapt to open-endedness as well. And of course that's super important because u science
26:26is extremely open-ended. The real world is extremely open-ended. Like you could be doing anything in science. You could be working on any problem uh in any number of ways. And right now what we're seeing is basically if you give the AI an objective and and basically like a a set of tactics it can leverage to try to reach that objective, it's going to do a very good job. Like it will optimize, it will search, uh it's it's going to see to the end. Uh but if you give it more
26:52like um a very uh open uh set of primitives where anything is possible, it might not necessarily do something very interesting.
27:01Yeah. And since you bring up sort of invention and especially working in these open-ended areas, um how would we do benchmarking in those settings? So I think for a lot of the benchmarking efforts that we've had, we kind of know in advance exactly what we'd like the model to be able to do. And this is really what's led to also successes in training with things like RLVR. But in settings where we want to see something unexpected or previously unknown, how do we deal with that notion? How do we know whether the system actually did accomplish that or whether it's not capable of it?
27:30Yeah, that that's exactly the the the sort of thing we're looking at with uh RKGI4 which is currently in development and and RGI 5 which is going to be even more ambitious. Um it's a super tough question. I don't think anyone today really has an answer. um you want to set up a a true open-ended environment, meaning that there's uh an infinite number of uh behavior trajectories that are possible and there's no goal to reach. So you cannot score a behavior trajectory by asking did it did it reach
27:59the goal? There's no goal. Any number of different behaviors can be can be valid and so you have to look at something else. uh in the context of science I think a really important metric is going to be um some metric of reuse of the artifacts or insights that you come up with like let's say let's say you're proving Ethereum that's great uh but if every step uh in your proof is just a oneoff like it's a trick that's useful for this proof but it's not going to be
28:28reusable there there's not really any insight there and it's not really m what mathematicians are doing when that they're doing math they're building this sort of like constructive uh uh colossus of knowledge. They're try they're trying to expand the frontier of what can be done like they're coming up with theories with all these reusable uh cognitive tools that can be leveraged on new problems even in new domains. Like one one really uh interesting phenomenon in the history of science is the fact
28:56that mathematicians have developed abstract theories that are not grounded in anything and they turn out to be the perfect fit to model our physical reality. That's that that's incredible.
29:07Um and so you're going to want to judge uh the the reusability empirically of the insights and the artifacts that that an comes up with. And it's beyond science. You can apply the same idea to you know um an individual. Let's say you create some kind of like embodied humanoid robot um and you set it loose in the world. The world is is is very open in the environment. How are you going to score
29:34what it's doing? You can use uh metrics like you know empowerment for instance you can you can try to track um how much control over its environment the agent is gaining over time or you can you can look at something like uh environment entropy like how much um is the agent like responsible for organizing its environment for making making an impact on the world not just random impact but an organized impact and and you can even
30:01uh come up with such metrics for um a collective Like let's say let's say human civilization for instance or a civilization of agents you can look at things like wage over time for instance like how much energy is your civilization able to harness over time like the kash scam.
30:19Yeah I think that's very exciting and it opens up tons of new possibilities as well. Um one thing we were thinking of was this sort of notion of compression right as being able to capture a lot of these ideas. Um do you think there are simple practical ways to sort of measure compression not just within a particular domain but maybe in this kind of more widespread sense? Yeah, the idea is that uh a very reusable building block uh a powerful insight like something that you should treat as an important new
30:48discovery is basically a compression primitive uh for like in math it could be like the space of proofs for instance or in in you know biology it could be like the space of uh discoveries you can make um meaning that so it's compression printing meaning that it should be relatively simple and elegant and it should unlock a lot possibilities or it should subsume a lot of pre-existing things like you can think of it as let's let's say there is a latent tech tree
31:16beyond civilization. It's like this it's infinitely branching and um a powerful new technology or theory or insight is something that will dramatically uh rebranch and simplify the tech tree or like unlock entirely entirely new branches, right? Um, and I think you can use information theory to try to quantify some of that at least in well understood domains. I'm not going to claim this is possible in like biology for instance, but it's definitely possible I think for math and computer
31:45science and maybe even AI research.
31:47Yep. Yeah, I think it's a pretty interesting question. Another sort of facet of this that we were talking about is kind of trying to understand whether we're training models that will produce some notion of general intelligence or whether really what we're doing is training models that have huge collections of different skills. You know, perhaps extraordinarily level of skills but is still a bunch of different buckets rather than anything that's sort of more general. Um, I'm curious whether you think there's sort of a real way of distinguishing between these two notions
32:16and whether maybe the models, agents, AI systems that we have today, whether they're trending toward the more general sense or whether they're still kind of these big collections of skills.
32:26Yeah, that that's a that's a super tough question. Um, I mean, obviously models uh have been capable of increasingly higher levels of broad generalization, right? Like it used to be that they could literally just perform specific tasks that they were trained on. And now you're seeing very broad in domain generalization. They can adapt to substantially new problems within a domain that they've been extensively trained on. Uh and that was not possible even like a year and a half ago. Like a
32:54year and a half ago models had uh hardly any fluid intelligence. Now they have very substantial levels of food intelligence you know. So historically the way you would have uh been able to measure all the models uh trending towards general intelligence is by picking uh tasks and domains that are outside the training distribution and and testing them on that. See if there's any any G going on any transfer learning from you know what what they know to skill acquisition efficiency on three new tasks. This is getting harder and
33:23harder to do because the model scanners are starting to saturate the space of tasks and domains we can come up with, right? And um it doesn't necessarily mean that they've re achieve general intelligence because the stuff we can come up with is not is not universal.
33:38It's not all there is. There's probably new new things that future humans will come up with and maybe the models should not be very good at it. Um personally uh what what I'm really interested in is the question of whether you what we're doing right now is basically we are VR maxing on math and code because that's easy because they're verifiable domains and the the big question is does that transfer into general problem solving ability because if it does um that's basically a really fast path to super
34:07intelligence you can just achieve arbitrary levels of scale uh on math and code because they have fable domains and if that translates into general intelligence, then you achieve arbitrary levels of general intelligence to be great. On the other hand, if in to there's no such transfer, uh you're going to create agents that are super human at math and code, but they're like savs. They're not they're still like dimwitted and under the very skillful facade. Um, and by the way, I like just intuitively when I use like I use Astra
34:37like all day for my research and I feel it's it's kind of like that because it's better it's better than me at coding.
34:43I'm not like reading code or writing code anymore at all. I just delegate all of that. But it's not really capable of grasping even very simple aspects of my research. It's not capable of coming up with the new idea. Uh, and like if I if I set it on a on a longunning goal, it's going to take like super silly shortcuts that show it has zero common sense or zero understanding of the the the logic behind the project, which is not something I would expect from a
35:10well-rounded expert. Um, so I I don't think, you know, I think the the jury is still is still out now. There's there's no clear answer. Um, one really interesting data point is from human psychology because you know the the transfer learning question also applies to humans. Um, as a human if you train on a lot of math and programming does that make you better at general problem solving and this has been a a subject of
35:39of psychology uh studies. uh back in the 80s you know programming was getting mainstream and uh a big question in psychology was by becoming really good at programming are you becoming better at you know planning general problem solving and so on and it's been shown via extensive studies and and meta studies that this does not happen actually like you can be extremely good at math you can be extremely good at programming uh you're no better than you
36:07were before at tackling brand new fields and And that's really interesting. So there's basically no far transfer going on. There is near transfer going on which means that like if you train on uh let's say one one RTS you train on edge of empires or something then you then you start practicing Starcraft uh it gives you an advantage. There is near transfer going on like on very very similar there is no far transfer going on like if you train on a lot of RTS's you become a a master RTS player and then you you take on chess you're not
36:37going to learn chess faster. Right? This is really interesting. And so you can ask, okay, if it's if it applies to humans, does it apply to AI as well? I don't I don't think we know yet.
36:45Yeah, I think that's a really interesting question. And I think one of the challenges there too is is it a limitation of the way that both humans learn themselves, which is often tackling different domains separately rather than trying to get at something general, but also the way that we're training AI systems too. Um, do you believe that sort of the ROVR paradigm is something that's limiting how much general intelligence we can get from these models or is it sort of the best path and we should keep pursuing it?
37:11I mean, it's it's a path that is working so far. It's a lever you can keep pulling and you will get models that get better at some things that you care about. So, of course, you're going to keep pulling the lever like it's just it's just the the scaling story. Uh, now is it the path that gets us to like true general intelligence? you know like for instance a scientist an Einstein level scientist um unclear and unclear it it feels to me that the way I generate
37:39ideas in my research is by forming a mental model of what I'm looking at what I'm trying to do and the next idea is a consequence of the mental model and I'm not feeling that the models uh process information the same way they're like they're looking at the current state like basically what what is there what's what's your code base what do they know and then and they're predicting what is the next step I should take, what's the next tactic I should apply. This very very different uh approach, right? Um
38:08it's actually it feels closer to playing chess uh in in in in code space uh than than it does to like coming up with the next insight. And I think it's really limiting that creativity like they're not very good at coming up with interesting ideas or like novel ideas.
38:22Yeah. So speaking of creativity, I think that's also a very interesting area. How would we build benchmarks that really tackle this question of being able to measure how creative are current AI systems? Or even going further, how would we design training data to sort of make sure that we elicit greater creativity from these models as opposed to, you know, our LVR training sets where the goal is solve this problem, become very efficient at doing that kind of thing. Yeah. I mean, that that almost sounds like a contradiction, right? Like can you train uh creativity in a
38:51supervised manner? I mean that's that's that's if it's possible it would be really interesting. Maybe it's possible.
38:57Um, I have some ideas on how we could measure uh creativity. I mean, of course, you're going to want uh an open-ended environment. You're going to want a sandbox. It's very very important if you're if you're going to try to measure creativity that you're not just looking at whether the model can come up with something that you anticipated, right? It should be able to come up with something that will surprise the benchmark developers. If you're never surprised as a benchmark developer, there is no creativity going on. The
39:25model is just going through a predetermined path and that's not creativity. Creativity is making your own path, right? So that's basically what you're going to want want to do.
39:34You're going to want to look at um I mean a very simple way to measure creativity is that you look at how different your behavior trajectory is from non-creative agents. So you can like take uh current agents like you you take Astra or whatever you know it's not very good at creativity. You let it do things in your sandbox and then you try the next agent that's supposed to be good at creativity and you look at how different and how interesting its behavior becomes.
40:01Should we invest much more in trying to measure these concepts of something being interesting or unexpected or you know surprising in various ways? I guess looking at many of the most popular benchmarks, they are not yet really capturing many of those metrics.
40:15Oh yeah, none of them. There's always like one right answer. Uh even a benchmark like you know AR3 for instance. Uh in AR we are not provided instructions and there's no explicit goal. You're not told oh you should accomplish this. You kind of have to find out uh what the objective should be. But there is an implicit goal because there are wind conditions in the environment and there's a small number of win conditions and as developers of benchmark we know what they are. So you're kind of like expected to do what we want you to do. That's that's not
40:43what what creativity is about. Uh I I don't think that's any benchmark whatsoever. That's that's about open-endedness and creativity. But hopefully, you know, ARK 4 will get closer to that and ARK 5 will be like fully capturing this idea in a very ambitious manner. I think on a very similar note, I'm curious what you think about being able to benchmark AI systems that are now superhuman but are perhaps not quite at the AGI level where you know you can essentially launch them and don't have to worry so much about benchmarking them. How would sort of
41:12human creators craft these benchmarks that are beyond their own capabilities, right? But should still have something interesting to say about AI capabilities? Yeah, I mean obviously it cannot just be a benchmark where uh there's a there's a predetermined set of answers and just pick one. Um you you want uh benchmarks that give you unbounded scores, meaning that you can always keep scoring higher if you're more intelligent. And you know some of the measures uh of open-ended behavior
41:42we are talking about can match that like uh how much energy you're able to harness over over the course of your you know life or the the lifetime of civilization or uh uh the empowerment metric like you can always gain empowerment you can always uh collect more energy and use more energy um and uh I mean uh this idea also like in science of building up this tech tree is something you can measure and quantify and it's completely unbounded there's always a next technology you can build.
42:10There's always an anory in math that you can come up with, right? Um so so you basically have to move into openness. You cannot stay into this idea that there's a there's a finite set of skills, finite set of domains and throw each skill uh there's a set of questions and answers and it's it's all I mean it's not going to work.
42:30Yeah. Yeah. Yeah, I think this idea of infinite combinations of things where you can adjust complexity and always add to it, it's going to be very important for future benchmarks just in general as well.
42:40In in general, benchmarks have to get way more ambitious. It's not about measuring skill anymore. It's about yeah, measuring creativity, open-endedness, uh, insight generation, all these things that are, you know, I think I think we can formalize them, but right now they're still very vague and no one's really trying to measure them.
42:57we have to get way more ambitious because like if if we're still measuring skills the problem is uh the models are saturating that like they're very skillful at everything it's not what's interesting anymore yeah I think kind of turning to really being able to launch these models in cases where we want them to go innovate whether it's in science whether it's in other areas um I'm curious kind of from the point of view of a benchmark builder you know what's the right way to find the most interesting problems or you know to actually create an ecosystem
43:26that really permits this to happen.
43:28We've seen it kind of happen in math, although much of that was just the verifiability of these problems. Now, it's going to be a bit more challenging in other settings. Like what are the big opportunities there that you would think about if you were trying to do this?
43:41Yeah, one thing I'm a bit concerned about is right now uh all of this like frontier research uh in you know applying uh Asians to to math for instance, it's all happening behind closed doors. In fact, it's using proprietary models that are not released. Uh none of this is open or producible. Uh and of course, that's not how real science works. It's not how real scientific discoveries work.
44:05They're not uh being being made in in closed labs. Uh they're being made in the open. They're collective efforts of thousands of people, thousands of scientists across many different institutions. That's how deep learning was developed, right? That's that's how the transformer was developed. uh like literally all of the techniques that we've been using to build these uh uh uh AI agents were initially you know they were they were a collective effort and I think we need to recreate that for you
44:35know AI for science that should be a human effort it should not be a frontier lab effort um I think one piece of infrastructure that's probably missing right now is a way to source uh hard but tractable problems that you could like you know that are slightly out a fridge for humans. So maybe not necessarily out of fridge, but you don't have the human resources to solve them, but agents could do them. So you want some kind of pipeline to source them from anyone. And then you want some kind
45:02of um shared infrastructure to collaborate for agents to collaborate on such problems. Basically, they should be able to share uh partial results or insights. I think when when you look at the the sort of process through which an AI agent would like prove um a new you know sol solve a new like open math problem um it's a lot of building up towards the solution like using you know some results from the literature coming
45:31up with your own lemas and so on. So this is very constructive process and you can paralyze it which is exactly what happens in science like you can and in fact this is precisely how uh these uh these open problems are being solved is via Asian swarms. It's the Asian swarms and I think that's going to be uh the next paradigm of applying AI for research. It's going to be all about the swamps because in in human science big discoveries are not done at the individual level there. It's always a collective effort and so you you don't
45:59just want to recreate a single uh human brain. That's not what HGI really is. I think HGI should be more like a a computer uh form of the process of science. It should be science as institution. That's that's what I would call you know strong AGI. Um and uh yeah we we should be building open infrastructure for this to happen like infrastructure for partial uh result reuse and and broadcasting across all
46:27the agents right uh where anybody can just contribute uh their own instance their own compute.
46:34Yeah I think that's also an exciting prospect and especially being able to like see all of it happen and really inspire other researchers as well is going to be something very valuable. I guess so. One one thing I should also ask you is I think for turning back to benchmarks in the past one of the nice things about benchmarks is they're very sort of bounded. You can run the entire benchmark in a short period of time. For example, you can run many different models and things like that. A lot of the things that we're talking about now they're much more unbounded. It's sort of this ongoing process for example and in fact if we deploy a lot of these AI
47:03systems in the real world it's also not really a bounded process. I'm curious how you feel about, you know, balancing the sort of tractability of very bounded kind of benchmarks the way we've been doing versus these new sort of completely open-ended type of ways of measuring or just outright using AI.
47:19Well, I really think anything that's uh bounded and short-term horizon is getting saturated. So, it's just not interesting anymore. And of course, it's going to cost more. It's going to be more expensive to a very like longtime horizon uh benchmark runs or very open-ended benchmark runs. uh it's going to be harder to evaluate, it's going to be more expensive to run, but it's necessary because all the easy stuff, you know, is is accelerated. There's nothing interesting there anymore.
47:44Yeah. I'm also curious also should we build a lot more benchmarks that are indeed sort of domain specific and partially are measuring these kind of very specific skills or do you think we should really go for very general abstract ones still in pursuit of seeing general intelligence arise for example also in ways that we can build these arbitrarily challenging ones rather than just kind of ground them in today's scientific challenges if I look at what's still missing uh from AI agents from frontier models to be AGI it's not
48:14really things that can be captured in a domain specific benchmark like you're really looking at slightly higher level slightly you know more like meta level more abstract capabilities like inside generation uh you're not you're not going to capture that in yet another sweet bench right you need something very different yeah yeah and there's uh are we're going to hear about arc 4 soon yes yes so arc 4 aims at capturing uh open-ended invention it's not going to
48:42be true open-endedness yet because that's that's really tough. That's really ambitious. Uh we aim at releasing ARK 4 in Q1 next year. So it's not as ambitious as you would like it to be, but there's Ark 5 coming, which is which is going to be like the maximalist vision for open-endedness.
48:59Yeah, we're very excited for it. Um I also think we're just about to be out of time. Um it'd be awesome if we all thank Franis for a very interesting conversation. [applause] [applause] Yeah, I think we actually don't have audience questions for this part. So, thank you, friends.
49:20All right. Thank you.
49:53Sorry. Five minutes voice. Yeah. Okay.
50:05Hello. Yeah. Yeah. Yeah. Just jump in there.
4:14:07So, we've spent a lot of time talking about considerations for designing benchmarks, things uh that we we want to keep in mind as we move towards the science of AI evaluation. But I'm really um genuinely curious to hear from the next two people who are going to discuss how benchmarks are actually used and maybe not used in uh largescale model development. So please join me in
4:14:35welcoming Braden Hancock, an AI researcher and investor, research partner at Lud and co-founder of Snorkel AI. and he'll be joined uh by David Hall who is a member of technical staff at Open Athena and the lead of Marin. [applause] All right. Uh can you hear me? Okay.
4:15:00Yeah, there we go. Thanks so much for the introduction. Uh excited to be here with David. I've been a big fan of Marin uh for quite a while. Uh clearly leaders in the open science space. Um but uh I I'll let you do the proper introduction.
4:15:13So for I would think in this room probably most people already know but for those who don't know what is Marin and maybe also who is Marin and why is Marin?
4:15:22Great. Uh yeah, so Marin is an effort to build uh a frontier level uh open language models that are fully open source uh and more than that to put the entire development process out out in public so that uh people can uh inspect and understand uh exactly uh how these models are made and uh that can better enable research and in inspection and community participation uh in the development of this really important
4:15:50technology. Uh that's who and or sorry what who is uh we h it's a team of 11 full-time people at open Athena um which is a nonprofit who is the lead uh uh funer for uh men along with uh Percy uh I think at Stanford uh who uh was the was along with me the co-founder of men uh and then and what was the genesis of the project why was there a need for this as opposed to one-off development in a whole bunch
4:16:20of different university labs Yeah. So, um the the the the way Men came into being or what would eventually become Men is that Percy and I were at Microsoft uh in 2020 uh when GPT3 started to come out and we got early access. Um and I started playing around with it and it was pretty clear to me within a few minutes of playing with it that like all of the NLP work that I've been doing at that point for 15 years was going to be completely obsolete
4:16:49within a couple years.
4:16:51Um and uh that was you know a bit of a a shock and uh and so uh eventually a person and I started talking and he said hey I want to build GPT3 in the open. Um and we knew that it was going to take a concentrated effort of engineering resources along with scientists uh to build something. Uh and I went we I went to Stanford, we started building uh and eventually we realized that uh Stanford
4:17:20wasn't going to be able to fund uh enough engineering resources in one place to to make this happen in the long run, but we still work closely with them, of course.
4:17:29Nice. Yeah. You see a lot of university projects that uh look like grad students worked on it part-time on the side in preparation for a paper deadline and it's just not quite the same uh caliber of of base to build on for the rest of the community. That's one of the things I've liked about Marin. You've got substantial resources, substantial size model and real I mean you know legit engineers there. Um right now you have your largest model you've ever run the hero run happening. Tell us about the
4:17:56scale of that. Yeah, it's a 535 billion total parameter model with 23 billion active parameters. So, it's like roughly the same size as like a lot of the deepseek models. Um, we're training it, pre-training it for 20 something trillion tokens. Um, and we've actually just crossed the halfway point the last couple days. Uh, and it should be done in the middle of November or maybe even earlier because we keep making it faster as we go.
4:18:25Nice. So that's good context to think on what you're doing. Half a trillion parameter model being trained completely open. Everybody can inspect the same dashboards uh uh individual experiments conversation that's happening along the way. So uh for this room that's interested in the eval no uh you you can't really get that type of a front row seat anywhere else that I'm aware of to how eval are actually being used to make decisions about is the run working, is it not? What are you paying attention to along the way? So let's start there
4:18:54maybe high level of just um like how are you in practice this early in a training run you know a couple months into a many month uh long long pre-training run and then eventually mid-training and post-training how are you actually using eval and what valves are you paying attention to? Yeah. So, uh, in the middle of pre-training, uh, the model isn't going to be, uh, in a aligned to be in a state where you can actually run the kind of B vals that people are excited about. Um, uh, so
4:19:24like it's not going to do well in terminal bench at all. Um, and instead we need to use proxies that are usually mostly uh, lossbased. So we look at perplexity on how to how data and sometimes some synthetic data um and use that and check you know one of course that the loss is just going down but also at Maren we spend a lot of time investing in scaling laws where we will train a bunch a family of small models and uh use that to extrapolate up to
4:19:53much higher scales and uh see whether or not the run is performing at or uh right now better than expectations. Um, and we found that loss uh at least for pre-training eval evals uh this this sort of natural data loss correlates extremely well with the sort of classic evals like MMLU uh and other things. So we don't have to like run uh a whole suite of evals in the middle of training.
4:20:20So then I mean correct me if this is the wrong mental model but it sounds like uh during at least these early stages of pre-training it's almost not traditional eval benchmarks at all. It's something that comes more directly from the corpus that's being pumped through as you do this next loss next token loss prediction.
4:20:34Yeah, we have a lot of data. We have a lot of data sets that are not particularly well represented in the training data. Some of that's deliberate like we have data set that is uh called uncheable eval which is a fun name and it basically it's uncheable because all the data comes from after uh after our knowledge cut off. Um and uh but yeah, for the most part, like you said, it's it's just natural natural language or natural data loss.
4:21:02So day one, you're looking just at loss.
4:21:04Day end when you're releasing it, you are looking at the same uh public benchmarks that people, you know, tend to care about and want to see on a model card. Where along the process are you starting to pay attention to other types of valves?
4:21:16Yeah. Um so we we have to start caring about those um at towards the uh at the end of pre-training we run uh there's this thing called mid-training we we merge these two things together um but this is where you put your sort of higher quality data that you know your Wikipedia and your stack exchange uh data uh and then right after that we run SFT um and that's where you can actually start to get signal from from the more traditional benchmarks like terminal bench and things like that um and so we
4:21:45will start paying much closer attention to those model those metrics and we are uh when we're once we're in that phase uh we have a smaller model called snowball that we're it's a 67B total 2B active model and that one we are now like tracking terminal bench and things like that do you hop straight into terminal bench like I think of terminal bench as like a frontier like it's it's hard even for the very best models can you start right with that when you get to mid- training or or post training yeah um I mean when we're talking about
4:22:14like at this stage we're talking like you know getting nonzero is like pretty good. Um so you know we're at like eight or something uh at this stage. Uh uh yeah I mean like things like uh human eval these sort of older coding evals are are more useful uh during that that part of SFT. And is are those older ones valuable because they're just easier in general or is it other formats like for a while a lot of eval multiple choice for example rather than
4:22:44agentic and confirming the environment you know was in a certain state when you're done.
4:22:47Yeah I mean I think part of it is I mean those evals were developed at a time when the the way you built models was pre-training then SFT and so these model the evals were designed to show signal for that stage. So I I think it is a mixture of like that's how good the model is right now, but also just like it's trained for that paradigm.
4:23:07Are there great new pre-training evals meant for that paradigm that are still being produced or has the uh have aentic eval sucked all the oxygen out of the air?
4:23:15Um increasingly it feels like yeah aentic evals have definitely sucked most of the oxygen out of the room for for these kinds of evals. Um I mentioned uncheably eval already. Uh it's loss eval but it is it is meant to help fill that that void.
4:23:30Gotcha. Um how about synthetic data? So that's there's often a temptation I think when you need to make it an eval that is scalable, very interpretable that you do something synthetic but obviously there are ways in which maybe it's out of distribution with what you're going to actually care about later on. You find utility from that class at all? Yeah, I actually really like synthetic evals um for assessing particular gaps in particular capabilities. So there's this data set called MRCR uh which I think stands for
4:23:59multi- round corerence resolution probably got that wrong. Uh and it's uh basically these it's a long distance long yeah distance eval where you have these really long chats uh that are entirely synthetic and you'll the question will be something like copy the fourth limmerick uh since the beginning of the chat uh in the next message um and SFT models can perform this task reasonably well uh and it's it's a good
4:24:29measure of basically you know how well the model is able to attend exploit the long context budget that you've trained for it. Um, and it's harder to find natural looking data that is a good long context eval.
4:24:42Got it. So, it sounds like there's these synthetic evals are almost more diagnostic. Maybe not going to be a absolute measure, but it could tell you if something a skill that's necessary but not sufficient uh is coming along.
4:24:56Yeah. And it's and it's actionable, right? I look at that and I kind of more or less can immediately know what to do next. Um whereas with terminal bench failures sometimes it's hard to know, you know, is it because the model is got stuck in a doom loop and it's thinking or because it's like bad at generating tool calls. It can just be it can be tricky to figure that out sometimes.
4:25:17Yeah. So maybe where terminal bench is intended to be like that one number that tells you overall how good is this um you know model, you want the opposite. You want a whole bunch of little individual sliced views I guess of different properties.
4:25:30Yeah. Um yeah, when we were really first getting going, uh we talked to Alec Radford and he gave us this exact advice to like look at, you know, not just a couple evals, but like dozens of them and not average them, but like look at them as a gestalt and like sort of just watch to make sure that most of them are going in the direction you expect. Um and uh I don't think we do it nearly as well as he does, but I think that that's really influenced how we think about things. So somewhere I'm imagining like a minority rapport massive screen that's got all the little Yeah.
4:26:00Is that something that the the community can see? Is that is that is that part exposed or not yet?
4:26:04Uh that we're still I think we need to we have something like that. We're still hardening it. Um all right. Feature request consider it registered.
4:26:11Yeah. Great. We will get it out soon. I think we just want to make sure it's all vibe coded of course. So we want to make sure that people aren't going to like use it to hack into our systems or whatever.
4:26:19Uh how about benchmark size? Uh I think 10 years ago you're making a benchmark.
4:26:23If I told you it's 100 data points, that's laughable. That's like, yeah, you know, too low statistical power and you know, yada yada. But now benchmarks are so expensive to create. I think actually you feel pretty good about a really good hard benchmark that has a whole hundred examples. As you guys are looking at things, do you have I mean, do you have a kind of a finger in the air of how large is large enough for you to feel like it's real signal?
4:26:43Um, that's a good question. I feel like I am still learning how to calibrate that. Um, I mean, you know, and because we're in the RL era, we're we're looking at passive K for everything anyway. I think you can get some, you know, you can average out the statistical signal that way a bit. Um, but yeah, I I I don't have a specific number where I'm like, this is what we need.
4:27:05Yeah. Um, I'm going to do maybe some rapid fire questions. Three, a benchmark you love, a benchmark you hate, or at least think we should retire. Maybe it was great once upon a time, it's just old. uh and then one that you wish existed which maybe is a uh you know call for uh proposals from this community and then I think if we're allowed to we'd love to just open it up for questions after that. So feel free to think of any that you have for uh for David but start with what's a benchmark that you love that's maybe underappreciated.
4:27:30Yeah. Um we kind of already touched on this already but I actually really have a soft spot for things like MRCR and graphs which are highly synthetic. uh no one outside of AI like model development is going to care about them but they are really very useful for us in the development process.
4:27:47They're developers eval developers eval not for the masses. Yes. Those with fine taste.
4:27:52All right. How about one uh a benchmark that you think is overrated or that maybe should let be relied upon less today. Um uh I personally have always been a little skeptical of evals that are driven by uh human direct human interaction where the humans are not uh representative of the actual user base that you expect to use the model. Um and so there you know eval out there where people just go and chat to the models
4:28:21and you know those are signal they tell you something important about about your model but I don't think they tell you exactly often what you want to what the the thing you actually want to know.
4:28:32So that would include sounds like maybe like arena style benchmarks possibly like where you're being paid to provide your input and you might be incentivized to move through fast to get paid just just barely quality enough.
4:28:43Yeah. You always Yeah. If Yeah. You always want to make sure your incentives of your of the people develop or creating your benchmarks for you, the ones doing the the work, are actually aligned with the thing you want to measure, right? Yeah. So, you don't want them to fly through. You don't want them to just be bored at work while they they're between tasks.
4:29:00Makes sense. All right. And then benchmark that you wish existed, you get to wave your magic wand.
4:29:05Uh, yeah. I I mean I think it's going to go back to just I I really do like these benchmarks that tell us like a specific capability like here's a gap. Um my current uh thing I rant to uh people at the office is I think that models like Opus and uh and Codeex are really very bad at theory of mind. I think part of the reason everyone thinks they're terrible writers is that they have this superhuman context and then they can't collapse that context to like what us
4:29:32mere mortals can like actually grock. Um and so I think if we had something that could like help look at that or other sort of specific model quirks um I think it would help guide model development in a more useful direction.
4:29:46Yeah, I think that matches with the thesis we've heard earlier today that is uh just because benchmarks aren't perfect doesn't mean we don't need more of them. that like uh as the the envelope expands there are more and more places where you know we we need that massive dashboard of kind of all of the individual attributes. So lots lots of benchmark work to be had uh where val valuable contributions could be made. Uh let's open up to questions. Any anybody want to ask um for somebody who lives watching the numbers training a model
4:30:14looking at eval uh of a half trillion dollar parameter model what what would you like to know?
4:30:22What's the biggest uh microphone for you real quick for those on the stream?
4:30:26Yeah. Um what's the biggest challenge of building Marin in the open in the way that you all have approached it? I think obviously incredibly valuable uh but really curious challenges in coordination or even uh just your execution.
4:30:38Yeah. Um you know I think there's a there's a yeah there's a whole lot of challenges. It's hard to pick out one. I think you know one is that we're doing this as a nonprofit and uh financing is you know a little harder. we've been very lucky to get some good and so that you know we have orders of magnitude less compute than even some of these neolabs uh and so we have to be very scrappy and I think we're very good at being scrappy but it is a challenge uh and then also uh you know the reason we're exists is because there's all this
4:31:07secret sauce in all these labs and we need uh we need to sort of understand what that is so that we can understand what these models are doing but we don't have that secret sauce and so we have to sort of figure it out by, you know, figuring out what little bits of information come out through press releases and tech reports. Um, I think those are the two hardest parts.
4:31:30See a question there in the back.
4:31:34So, when you when you were talking about pre-training, you said you were looking at the scaling laws and then your like bigger run actually surprised you in terms of not following the scaling law.
4:31:43So, extrapolating your surprise, like where do you think the frontier labs are compared to public model releases? like can you extrapolate like that or um that's a good question. I think I mean it's I I don't so one of the things about scaling laws that's really hard is that you need to guess that there's a slope of like how much model you're going to get with scale but also there's some floor that is like this is how good you can possibly do on this set of data and estimating that is incredibly hard
4:32:13until you're already at the high scale and so we have to kind of guess a little bit you know it's an informed guess we kind of have to guess what that asmtote is um And so we are so far beyond them.
4:32:24So behind them in terms of of scale that um I I don't know if I I would be surprised I basically I'd be very surprised what the asintote was probably and so they're probably like pretty Yeah, they're pretty far down past what we would expect.
4:32:44Uh great talk. I was uh a little bit curious how you handle decontamination.
4:32:49Decontamination is always a very big problem. You're always running into the fact that when a new data set comes out, was the model that you're evaluating trained on that data set? How do you handle the perturbations that go on?
4:33:00Yeah. Uh yeah. So decontam decontamination I will probably get some of these details wrong, but uh we have a blog post where we talk about our entire data prep-processing pipeline, including this. So uh you can you can check check what I'm about to say. Uh we do something like 13 gram decontam I think something like that where we look for engrams that are you know look exactly like some of the the the engrams that are in the evals and then we pull them out. Um we've been burned by
4:33:29contamination in the past. Usually when we've been burned it's actually made the model substantially worse. uh on our 8B model we trained a year and a half ago, uh we found that we had a bad version of GSM8K where it wasn't bad in the sense of the answers were wrong. It was just formatted in a way that the actual EVAL harness we were using didn't use and our model insisted on putting the answers in the wrong format and got like basically a zero because of this. Um and then
4:33:58recently we had a problem with the pin treebank loss evaluation where despite all the effort we put into decontam uh where we took you know every 13 g like overlap we would throw out the training document someone had just created six grams of all of the pen tree bank and stuck it on GitHub and so our model like saw that memorized it and then like the loss like plummeted for a while. Um that was it. Yeah. So gems all over the internet for you guys to find.
4:34:27Yeah, Decontam is basically impossible, I think. Right, the labs use uh they use their internal code bases, their internal emails. Um they're probably buying other people's emails and things.
4:34:38Um and the closest we have to that is we have uh we haven't fully figured out how to use this safely, but we have Percy Bench, which is all of Percy's emails and all of his private documents, and we like stored that somewhere, and we're not going to share that with anyone, but uh that's that's the best we can do.
4:34:54All right. Let's get one last question and maybe we'll move on to the next tech.
4:34:57Yeah. So, we've seen what's happened with, you know, greater pressure uh and misaligned behavior. How are you ensuring that as you're going through your rollouts that you're staying aligned? What techniques are you using?
4:35:09Are you using any probeg guided uh techniques, inter internal interpretability techniques, anything like that? Or is it most more like in the data and then you run the benchmarks and check back and throw away some rollouts or go back?
4:35:22Yeah. Um we are still early enough in post training that I don't feel like I have the most satisfying answer here. Um we are paying very close attention to data quality for um eval. We've like we have found a lot of public data sets for RL where we've had to throw out over 90% sometimes 98% of the data because they are just fundamentally broken. Um and this is you know sometimes greater bugs sometimes just like unexecutable. Um, and these are the kind of bugs that, you
4:35:51know, a much more capable model is going to end up like hacking out of the sandbox over because it's like literally impossible to get the answer any other way. Um, so yeah, that's our current best answer.
4:36:02All right, can I ask a question?
4:36:09Okay, so I am inspired by your lightning round. So, this is just a quick one for you, David, but what is a conversation uh around this space that you think we should be having, but for some reason you're seeing that we're not quite having and if you're tempted to answer, why aren't we having that uh very necessary conversation then?
4:36:32Um that's yeah, so I I uh I have a draft of a blog post I'm hoping to put out in the next couple days. Uh uh sorry this you asked for a lightning answer. I think post-training data is really really bad in the open. It's very very bad. Um and uh the reason it's bad is that we have the internet for pre-training and that's all of human knowledge in document form but we don't have all of human knowledge in task form
4:37:00and creating that is extremely difficult and expensive and is why companies like Snorkel exist. Um and uh and yeah, I think for open models, the challenge is going to be building enough public data, public post training data environments to to uh yeah to succeed.
4:37:21Do we have time for one more or are we uh getting back on schedule? We do. All right, go ahead.
4:37:26Actually, as an extension to that question, u a lot of the reasoning or like post training data looks like distillation traces from other models. Y um and so is that something like the way to climb towards what the frontier exists or do you feel like there is another way um that's I don't know what that way would look like but what do you think about this? Yeah, I mean just so if I understand your question correctly like yeah there you know the easy path is to distill out like you know other
4:37:56models and get traces from them and clone them and we we do a little bit of that to like bootstrap RL. um we don't want to do very much of it and we want to sort of do RL the correct way but where do we get the environments from those we're using agents to build um uh is a big way we're doing it and we're trying to figure out how do you build a pipeline where they you know do where the quality is high enough uh to make that work and that is not exactly distillation it's a it's it's its own
4:38:24thing all right thanks a bunch thanks so much David and Braden [applause] All right. So, we will now um part for a short break. Stretch your legs, get some beverages, and then when we return, we'll hear from the builders of some of the open benchmarks that you may have already seen in the poster session or
4:38:51heard talked about earlier in the day. And uh then that will be followed by a panel moderated by our open benchmarks grants lead, Vincent
4:56:07Hello. Hello. Hello. Uh if you're in the main room right now, I'd like to welcome you back into your seats. Uh we're going to get started on the uh next session shortly.
4:56:36All right. So, welcome to what I feel is uh certainly one of the best ways to end the agenda on a high before um going into that dedicated poster session in the afternoon featuring lightning talks from the people who are actually behind building some of these open benchmarks.
4:56:59So as we've uh discussed throughout the day, uh benchmarks have uh certainly acted as guideposts and drivers of AI progress. Uh you know must keep pace with the frontier and be kept open so the field can inspect methods and reproduce findings. Today we will hear from three researchers doing just that. First up, we have uh Stephen Dilman, a PhD candidate at Stanford and the lead of terminal bench science. Welcome, Stephen.
4:57:45Okay. Hi everyone. Uh so I'm going to talk about terminal bench science which is a large community effort to uh by the terminal bench team in collaboration with scientists across different research institutions and universities to build a challenging benchmark to evaluate AI agents on scientific research workflows. But first of all before I start with terminal science let me start with terminal bench. Um because terminal science wouldn't exist without terminal bench. um especially about our
4:58:15main first main release which is termbench 2.0 by Mike Merrill, Alex Shaw, Andy Konvinsky, and Ludik Schmidt and also all of our great uh ad task contributors. Um it's a benchmark measuring how well agents can do real work on uh the computer via the terminal. And uh tasks range from coding tasks uh to system admin tasks to security tasks and so on and so forth.
4:58:42And these tasks are built to be extremely challenging. Uh and lots of work has you know been going into reviewing each task um such that the quality is of the highest standard and all of that hard work has uh really paid off. Terminal bench has become one of the standard benchmarks um used by the frontier labs. So virtually every lab reports uh its terminal scores whenever they release a new model.
4:59:08Turtlebench has driven you know great progress in agentic coding. Sonet 3.7 scored around 18% in March 2025. Uh nowadays many models are well over 80%. Even weaker models um score above 80%.
4:59:22Um but for that reason we are maintaining terminal bench as a continuous benchmark. So we are maintaining it with task fixes uh and updating it with more challenging tasks um as they come in to keep pace with the moving frontier. So since the original launch of Turnbench 2.0, we released Turnbench 2.1 uh 3.1 or 3.0 4.0 and we are now working towards terminal bench 5.0 to keep driving progress on agentic
4:59:50coding. In parallel, we want to drive the same progress but for science with internal bench science uh which evaluates agents on scientific research workflows across five different domains.
5:00:01So we have the life sciences, the physical sciences, the earth sciences, the mathematical sciences, and the engineering sciences. And our goal of this project is to let scientists themselves set the bar for what AI should be able to do in the sciences and give them a voice in shaping AI progress because so far many benchmarks are being built by CS people, software engineering people, coding people, but now it's really time to give the scientists the voice um and let them set the bar for what real science is. Okay. So there are
5:00:29many different aspects that make a good tasks uh to turnbent science but three main ones uh that I usually communicate are first of all they need to be scientifically grounded number two they need to be objectively verifiable and number three they need to be genuinely difficult for the frontier and yeah this is the contribution and review pipeline for turnbench science.
5:00:50Uh it looks like this. It starts with a task proposal. So domain expert uh authors kind of submit a task proposal to us. We have domain experts on our reviewer team that review the task, give feedback and then when they approve the task, it moves to the implementation stage. So people then implement the task via uh a GitHub pull request.
5:01:12And once we have the pull request on our dashboard, basically we have a domain reviewer that checks whether the science behind the task is right. And we have a technical reviewer that checks whether the task was built uh in a correct technical manner. And then once both of these reviewers approve, it goes to a final reviewer which is kind of a bar raiser. So he or she stress test the tasks um tries to pick hole in the
5:01:40tasks. Um so that's how we kind of ensure the highest quality. Um and then once the final reviewer approves it gets accepted into the benchmark and uh merged into into main. So for our first release uh of turnbench science 0.1 we received over 900 task proposals about half of them got approved for implementation and out of 386 task uh implementations or pull requests only 70
5:02:08made it into the final benchmark and that's about 88% out of the original 900 task proposals.
5:02:16In the end, our first release included 70 tasks contributed by domain experts from 20 different fields uh from 43 different universities and uh 10 different countries. I'll quickly go over some example tasks here. Uh so this is from the life sciences where we ask the agent to basically do the job of a radiologist and make correct diagnosis of this deidentified ankle MRI. Here's one from the physical sciences. So my research background before I joined
5:02:46Stanford as a PhD student was in astronomy. So this one is is very close to my heart. So we are asked the agent here to identify exoplanets in astronomical time series, characterize them um and also reject false positives.
5:03:00And here's one from the Earth Sciences where we tested the agents on whether they could use satellite imagery to recover the correct uh winter dates of when a lake froze and when it melted compared to the times that observers on the shore um recorded.
5:03:17And for this maps task uh we asked the agents to prove uh on Sager's exact solution of the 2D icing model in lean 4. And for this engineering task, the input is a video from a high-speed camera filming an eightstory model building while it vibrates. And the agent has to identify the building's natural frequencies uh and the vibration modes. Uh and at the time of release, the strongest model on these sorts of
5:03:45tasks um was Opus 5 with 30% pass rate, which is 64% points lower than turnbench 2. You can see that Turnbench 3 really pushed down the pass rate as compared to Turnbench 2, but Turnbench Science really pushed it even further down. About a week after the release, Turnbench Science was featured as the headline benchmark uh on the Claude Fable 5.1 and GPT6 Astra release with strong performance jumps to over 50 and 60%.
5:04:17A number of the other models dropped since then and we track their scores on our official leaderboard website on terminalbench science.ai. You can see from this curve here how well it discriminates uh between different models. 13 out of 17 tested models score less than 25% on the benchmark and seven of them even below 10%.
5:04:38So Astra and Opus 5.5 are clearly the strongest models achieving similar performance for a similar cost. But Astra not only takes the lead, but it takes the lead while spending the fewest tokens. So this is a a very interesting result. Looking at the performance across different domains, uh it's probably not very surprising to see that the strongest models by anthropic and open AAI are are best at maths. They are weakest in in in the life sciences, but they are clearly best at maths. Um but
5:05:07if you look at the other models, for example, Grock 4.7, Gemini 3.8, 8, GLM 5.3, Kim K3, they're actually pretty weak in the mathematics. It's their their their least strong uh domain.
5:05:20Grock uh is pretty good at earth science and engineering. Uh so that's also an interesting observation. So yeah, we are now working towards our next release 0.2 which should be out at some point this month. So uh probably end of October. Uh it will have around 100 tasks in total and then we'll work towards our gold release 1.0 zero by the end of the year.
5:05:41So if you have friends that are scientists or if you have scientists yourself, please come contribute and get in touch. Yeah. Before I wrap up, thank you to all of our great uh task contributors, our amazing reviewer team.
5:05:53Without a good reviewer team, you cannot build a good benchmark. Um and also our great advisers and also our research partners and comput sponsors like Snorkel as well. Um so yeah, that's it for me. Thank you and yeah, look forward to the panel. [applause]
5:06:12Great. Thank you, Stephen. Next up, we have Russell Yang, an applied scientist at Microsoft and project lead of Judgment Bench.
5:06:34Thank you. Awesome.
5:06:42Hey everybody, it's great to be here today um and to present Judgment Bench on behalf of a really big team. Um Pearson Ria are here. Please catch us at the posters afterward. Um but we're also super grateful to Snorkel for the open benchmarks program and then our collaborators at Harvey as well who really made this possible.
5:07:01So to start I want to ask you guys to do a quick poll. Um, and the poll is about whether you think knowledge work domains like law, finance, consulting are more like natural science or more like fine art.
5:07:16And by natural science, I mean something that's rules-based, where the ground truth is objective and where the types of questions that scientists ask each other contribute to a body of settled knowledge. And I want to contrast that with fine art, uh, where you might go to a museum and look at a painting. Um, and instead of, you know, running through a mental checklist of what makes for a good painting, you're instead implicitly comparing that painting to some ideal that you have of what a painting should
5:07:45look like or to the other art in the museum or to um, you know, these holistic intuitive notions of equality that are difficult to spell out in advance. So, if I could have a raise of hands if you think legal practice is more like a science than an art, I would love to see what you guys think. Okay. [laughter] And then vice versa. If you think it's more like fine art than like science.
5:08:13Okay. Seeing more hands. And anybody think they're kind of about the same. Like there are elements of both science and art.
5:08:21Okay. Okay. Um I think most of you kind of raise your hand at the end which motivates our study. uh we were really interested in empirically testing this question of you know is legal practice more like a science or an art and how does that affect how we should think about evaluations in this space. Um so there are two main methods as I'm sure you guys know uh that are commonly used to to perform evaluations. Um there's comparative judgment where you present
5:08:50the annotator with two outputs and ask them to indicate subjectively holistically which output they think is better. And then there's rubric based scoring where you provide the annotator with a single output and a rubric and ask them to assign points to the output based on the rubric.
5:09:10Um, in a lot of research, the choice of evaluation method is assumed without a lot of empirical justification. And we wanted to provide an empirical test of this question in a domain like legal where ground truth is often subjective and contested. We are really grateful to have more than 50 expert lawyers contribute to our study. Um, they were quite experienced.
5:09:35So 63% had more than 8 years of experience. Um and that experience tended to be concentrated in the areas of practice most relevant to commercial work. So big law type work. Um so about 70% of them were experienced in litigation and 63% in transactions. And these lawyers contributed 3,000 annotations on top of a base data set of 30 tasks and rubrics that Harvey, our legal tech
5:10:05partner, had provided to us. So with Judgment Bench, we make public both 30 tasks from Harvey's big law bench, which are big law style tasks. So they were created by lawyers to represent the types of work that lawyers do in their daily practice. um plus the 3,000 annotations that were collected on top of Big Law Bench and represent lawyers indicating preference or rubric judgments for model outputs um model outputs from uh that were generated by
5:10:36using Harvey's tasks.
5:10:39Um here's an example of the task on the right. Analyze an offering regarding its terms and the existence of a fairness opinion and draft a bullet point summary of the results. Some of the tasks also had documents attached to them, for example, litigation filings, and those were also included um in the model runs.
5:11:02So, the goal of our project was to test whether comparative judgments or preferences better recovered outputs of an ordering of outputs of different qualities. And in order to do that, we needed to generate outputs of different qualities. And the way that we uh decided to go with after kind of thinking through various options was just using a prompt controlled approach.
5:11:24And one thing I want to highlight here is that the problem of generating ground truth or capturing ground truth in legal is especially difficult because almost every way you can think of to get a clean ground truth will be confounded by some kind of factor. So, one example might be you look at briefs that a plaintiff and a defendant submitted to a judge and consider in the judge's ruling, did they cite the plaintiff or the defendant's brief more? And then you
5:11:53might have some signal um some natural signal as to whether the brief is high quality or not. But that approach is really difficult because the uh notion of quality is confounded by the facts of the case. Maybe the facts are on the plaintiff's side and that's why their brief is better. It's confounded by the law and it's also confounded by the judicial preferences.
5:12:17Um, we generated 50 different versions of each quality level to try and account for some of the sampling variability to make sure that we didn't just get a very good or very bad excellent output, for example. And then we fed these uh preference tasks and rubric tasks to the lawyer annotators through a custom design platform.
5:12:39Um moving towards some of the results, we looked at two different ways to measure um this empirical question. One of the approaches was how well the poolled signal recovers the uh total ordering. So thinking about um aggregate what what ranking of the outputs is implied by the annotations that the lawyer conducted and how well does that correlate with the true ground truth ranking. And second, we just looked at treating all of the
5:13:07annotations as the same. How often does the better output how often is the better output rated as better by the lawyer annotator?
5:13:17This is our main result. So we find that comparative judgments in our setting better recover the quality ordering and take less than half the time. Um which we think has implications for um people buying human data for firms internally evaluating legal tech tools from different vendors um and sheds light on this empirical question of how should we solicit um quality in high stakes high judgment domains where ground truth is
5:13:45difficult to verify.
5:13:47We also find breaking out the results by lawyer that this effect seems pretty consistent across the lawyer's experience level. So for both junior associates, senior associates, and partners, we find an advantage for preferences over rubrics. Um, and this question is important because as lawyers begin to adopt more legal tech tools into their workflows, junior associates are not just doing work, but they also are becoming arbittors of quality in a
5:14:15way that previously um, a role that was previously reserved for more senior people at the firm.
5:14:23Um, a few of the takeaways. We find that comparative judgments in our setting better recover the constructed quality ordering under both of the metrics that we studied. Uh, we find that they take less than half the time. Um, and these results shed light on this broader question of how we should elicit, aggregate, and use as supervision signal um, preference judgments versus rubric scoring in these domains where ground truth is contested and subjective.
5:14:53[applause] Great. Thank you, Russell. And finally, we have Gabe Oranski, a PhD candidate at WMadison and project lead of Slop Code Bench. Hello, my name is Gabe and I'm going to be today presenting slap codebench a project between WMadison MIT and sponsored by snorkel open benchmark currents.
5:15:38So as you all are familiar with SweetBench and all these different coding benchmarks up to date. The issue is is that they are one instance, one PR, one issue, one request. That's not how we build any degree of software currently. Every single project that I've ever worked on, and most likely it's the case for all of you, involves iteration. We iterate on how we build.
5:16:02We iterate on our requirements. We don't know necessarily what we need at the offset. And our benchmarks should reflect that specifically. But agents on the other hand compared to us struggle. They add in useless code. They add in unnecessary flags. And they really just mess up your codebase on every single iteration. Every every time you ask them to add a new feature, they add in an unnecessary piece of code or something that will break later on.
5:16:32And that's why we built slop code bench. Now the core idea of slop code bench is that every single coding evaluation should be built as a progression of iterative checkpoints. You start with one core idea, right? In our case it is a code search engine for Python. Nothing more, nothing less. We don't tell the agent we are going to be eventually asking it to expand into Rust. We don't tell them we need structural matching.
5:16:57We just tell them we want a code search engine. Then once they've completed that we ask them add in C++ support add in JavaScript and at this point if the agent did not design its code at the very beginning with a easy to use modular approach it will have to rewrite everything and we keep doing that multiple multiple times until the codebase becomes unusable.
5:17:22So slop is a colloquial term objectively we define it in two very specific means.
5:17:28The first is verbosity and this is how much of your code is redundant. Now we define this as how many structural duplicates you have in your entire codebase and how many exact matches of patterns we've observed agents right that could be written in significantly less. So on the left we have if not rules continue does not need to be there. Right? The core idea is that the applicable function should just return nothing if there is no need to continue.
5:17:57The other thing that we do and this is a very important aspect of code design. It's been in code quality metrics for some time is this idea of complexity.
5:18:05Now we call it structural erosion. We are measuring this how much of the cyclatic complexity of your program. So essentially how many branches exist in your program are concentrated in a few functions. So across the board we want to see okay is the agent just patching or is it actually trying to refactor and reorganize its entire codebase. Now the biggest thing is that no agent completely solves any problem.
5:18:29Cumulative is the entire iterative process. So at checkpoint n it has to solve not only that checkpoint but every other test that comes before it.
5:18:38Isolated would be the the bbench approach. Here's our issue. Here's our codebase. Fix it. And as you can see at the time GBT 5.5 was only able to do 14% of our checkpoints. This is 196 checkpoints across 36 problems and it really showed that there was an upper bound that we are missing. Right? We there is clear headroom that agents still struggle immensely at just solving these problems over time. But what's worse is that this is really a major
5:19:06issue as it progresses. As it goes on, we see only 1% on average across all of our observed models. 1% of the checkpoints are solved. 32% in the isolated setting. So this is your SWEBench style. But only 1% of the program of the checkpoints were even solvable at the very end. And this is on an continuously increasing scale of cost. So every single checkpoint it gets more and more expensive, but it solves less.
5:19:33But even worse is that they all get bad. This is a across all agents that we observed. They degrade. They degrade at an really incredible rate. Whether it's Sonnet, whether it's GBT, we observed this even for Gemini and all the other models that we've evaluated up to this point. they add more code and it gets more eroded and it gets more verbose than it's needed.
5:19:53Now the number one issue that we observed with this across all of our agents is that they do not refactor ever. They never stop and think is there a better way to organize my codebase? Is there a better way to organize the logic that we have here and as you can see on the left we have a very simple way of writing the find matches function. This is what I would expect a competent software engineer to write. On the right is what the agent has written after all the checkpoints. It's 140 lines compared to five lines. It's god awful. It repeats the same exact matches.append
5:20:23every single time because it has no idea of writing a helper function. It reuns loops every single time that is just not needed to be ran. And this is in stark contrast to humans.
5:20:35So we looked at a large variety of human repositories over multiple different strata of how popular they are from the beginner developer to production level libraries. And we find that agents do degrade in somewhat similar manner of humans, but it's at such a larger magnitude. It's two times more across humans compared or across agents compared to humans. This is a significant issue if you are to be a maintainer of any degree of code bases.
5:21:01If you are using these code bases in realistic projects, you will run into issues where they just continuously destroy your codebase if you are not careful. Now, prompting can help very slightly, but prompting can help.
5:21:14At least for our idea of just asking the agent to not write slop and that's our anti-slop. We just say please don't write slop, please refactor. At least gets better to a degree. It's not perfect. The accumulation issue is still there. Agents still do not understand how to patch functions properly, but the amount of issues, the amount of flagged code is less, which is a promising sign.
5:21:39So to summarize, slop codebench measures slop obviously, but we care about it specifically under the case of iteration. We believe that you cannot have a coding benchmark that reflects real world usage if it's not iterative. You cannot reflect the software development progress or process if you are not hiding requirements.
5:21:59And no agent even today when we're very soon to release new agent results is able to solve a single problem. They always break something. They always miss edge cases and it gets worse the more they iterate. And the other core issue is that every single agent degradates over time. They ruin code bases and they do not ever try and stop. And this overall is somewhat solvable.
5:22:25We observe some signs of life that you can solve this with better prompting with better harnesses. But it's a core issue we believe of RL training and of agents currently that needs to be addressed in the training stage, not just in the harness stage.
5:22:44[applause] All right. Thank you, Gabe. Uh so now for the extra fun part. I would like to invite Stephen Russell and Gabe back onto the stage. We're going to put all of them in conversation with each other. And to moderate that is VP and member of the Snorkel founding team, Vincent Chen, who also leads our open benchmarks grants program.
5:23:22Awesome. Well, thank you all for the great lightning talks. Um I think we saw a few different flavors uh of of benchmarks that uh have you know been built recently uh from science to coding to you know some empirical studies uh in fields like law. Uh I wanted to start with a quick question for each of you uh which was what what the most surprising thing was in building the benchmarks uh that that you guys worked on was.
5:23:48Sure I can start. Um, I think what was most surprising to me was that lawyers were willing to participate often pro bono in this kind of work. Um, I know that some of the um, snorkel annotators were participating pro bono as well as some annotators from the big law firms that we worked with. Um, because I I had always been worried about when I put 30 minutes down on a lawyer's calendar, I'm like wasting $500. But um it's great to hear that everybody's so uh enthusiastic
5:24:17about contributing to open benchmarking.
5:24:20Yeah, I would say honestly like even between this project and our recently released other open benchmark project library design bench agents really have no concept of how their code will be used over time. And this is something I, you know, just kind of before I even started any of these projects, I intuitively thought, oh, these agents have an idea of how their code will be used. But it's really just no, they just solve the current problem and they don't really care what happens after.
5:24:46Yeah, for me it was in the very beginning, we needed to create a few example tasks for the community. Um, and I tried to do one myself. I was sure this is going to be hard enough, but it's actually extremely hard to develop task. And that was like half a year ago. right now. It's even way way harder. So, it's actually Yeah, it's pretty challenging to create tasks that are challenging for Frontier models. Yeah.
5:25:09Yeah. On on that note, um I'm curious, what do you guys view as uh the the trickiest trade-off between capturing realistic work and uh measuring it in a reliable way?
5:25:24Sure. I think um in in these kinds of domains that we're working on in particular, there's this additional dimension of like whose work are we comparing um and thinking about how there might be a difference between the work that like a good work that a human does versus bad work that a human does and good work that a model does versus bad work that a model does. Um, and I think at the end of the day, like with a benchmark, like we're really interested in the model performance. And so maybe
5:25:53accepting that there's going to be trade-offs and like the ways in which we induce quality variation might differ from like how a good associate versus a bad associate might approach work.
5:26:05Yeah, I think for code specifically, we're in a very weird spot where you want to make the task solvable, but you also want to reflect how real people will be prompting agents. I think there's been great work in terms of user simulations and you know just trying to have the user as be part of the entire benchmark but it's still a challenge of how do you write a spec that is realistic but also solvable. I think that's still something that we were working on figuring out.
5:26:33Yeah, for us it was science is sloppy, right? Science is not just like correct or not correct but we needed to have tasks that are objectively verifiable and that are kind of outcome verified.
5:26:42So we look at the outcome it's like okay if you achieve this outcome then you are the pass if not you fail science is a whole process right so we automatically can't capture the entire process there's things that we can do for example if you have the verifier accepting one specific process and solution for a problem but you have another equally you know valid approach on solving the problem you need to kind of put that into the verifier
5:27:10you need to design the verifier to be flexible towards all valid approaches and one expert would solve it in this way one in in the other way and you need to kind of be able to capture all of these things. So that's what you need to kind of build into the verifier but it it's hard like all benchmarks are never 100% reflection of of of reality. The other thing is also automation. Uh scientists don't use agents it's like okay hey you solve this task zero to 100%. they use it in an interactive
5:27:40manner um as research assistance. It's actually what we want to drive. We don't want AI to do all the science um themselves. We want them to be reliable research assistants, but we are of course testing tasks in an autonomous manner. So that's one other trade-off.
5:27:55Yeah, I heard from a few guys there's this uh balance between using kind of experts but also augmenting with agents.
5:28:02Uh were there specific techniques or workflows that you ended up building to um facilitate effective you know study or task creation uh in each of your settings?
5:28:13We come back to me on that one.
5:28:15Yeah. [laughter] Where to go?
5:28:16Can you can you repeat the the question?
5:28:18How how did you use agents and humans um in some of the task construction?
5:28:22So first of all, yeah, I and many other on our team, we're actually very proud that all of our tasks come from real domain experts. They don't come from data vendors. They don't really come from you know CS people, software engineering people prompting AI to do a science task. So that's something that we are really really proud of. The human reviewers reviewing the tasks are also domain experts on the so they're reviewing tasks in their domain. Um and number two or number three of course we
5:28:50use AI uh we use a lot of AI actually to look at the trajectories is maybe the one of one of the most important things in reviewing a task because the trajectories can sometimes um you know reveal issues in your task. So for example, a broken task, an assumption that is not stated. Um, and to analyze trajectories, we we we have to use AI agents to kind of go over the trajectories to identify these kind of flaws. Yeah.
5:29:19Yeah. I I agree. I think having human reviewers that are domain experts be the final judge is like the most important part across the board. I do think and we we've taken this approach in both of our benchmarks that we want to give the human reviewers the minimal amount of work. We want to just show them the instruction the highle idea and so and you touched and so like that's the core idea and you touched on this part of actually looking at trajectories. I think that's something that you know in
5:29:47the open benchmarking community and as you know we are building benchmarks we're kind of losing access to the trajectories that really make the difference right like with all the reasoning hidden aspects that you know the frontier labs are doing it makes it very hard to develop benchmarks well and understand what's ambiguous right and that was something we leaned on a huge amount in all of our benchmarks is we needed real trajectories with real thinking traces that we can have other agents analyze to identify okay this exact thing may be completely ambiguous
5:30:16or this thing is relatively wellnown but it's a very noisy element of our problem. So I think you know you can use AI for that but you can't use it without the reasoning traces.
5:30:28I think there's this additional dimension of what do you do when the human experts disagree like if you're trying to use an autograder to approximate their judgments like whose judgments should you be approximating?
5:30:38Should you be approximating the aggregate judgment? Um or should you kind of break it down by uh you know what is subjectivity and what is error and think about uh can we eliminate all the error and think about preserving some of that subjectivity um with autograders?
5:30:55Yeah, I guess speaking of the the grading side of things in all these cases we you know clearly want to use expert judgment in some way. um how did each of you design the grading mechanisms or think about you know different types of grading mechanisms and how they actually fit um the construct or uh instruction set you were trying to measure.
5:31:14So our study was a little bit different in that respect in that we had this empirical question that was motivating it. Um but we thought about that empirical question in two ways. So one way is from the perspective of a benchmarker um what what would be useful to a benchmarker um from this process and that was where we landed on the rank correlation metric. So thinking about how well the ranking implied by the annotators aligns with the actual ranking. The other metric is for
5:31:43somebody who might be more interested in individual comparisons of outputs which is where we thought about the win rate metric.
5:31:51Yeah, I think across all of you know the work that we've done it's essentially this idea that the agents should be left to their own devices and any degree of grading we don't want to overprescribe any ideas especially when we want to measure can they refactor can they make decisions about code design and any gradable surface that you include biases them beyond you know what you'd want and it makes it very hard to elicit the exact signal that we care about. So we took a very I wouldn't say
5:32:20hands-off approach like we still gave them constraints but we wanted to make it clear that we didn't care what happened inside of the program. We didn't care what the code looked like just make this program output this result. Right.
5:32:31Could you share an example of Yeah. Yeah. So I think our most obvious like with the code search one we didn't really care how they built the code search. We didn't care what they did.
5:32:39All we cared about is that they looked at this exact directory that they took in this rule and that they returned this result. nothing about the internal structure was ever looked at in like even though we only did Python evaluations for all our benchmarks. We could have pretty easily just let them do Rust as well. We could have made it so that all they did was build a Rust project and then we ran it and it wouldn't have changed how we designed this benchmark at all. Like the core benchmark stays the exact same.
5:33:06All we care about is did they actually build the right thing when you run the program.
5:33:13Yeah, I think this is a repetition of what I said earlier, but you can't have the verifier be overly coupled with your reference solution. Um, that's actually one of the biggest failure modes of tasks that we receive. A lot of our reviewer feedback is like, hey, this verifier accepts your reference solution, but it seems to be overly strict and doesn't accept an equally valid uh equally valid but different solution. So I think yeah that's the number one most important things. The other thing is like thresholds with
5:33:42scientific tasks you have a lot of like thresholds. Let's say you have an MRI image and you want to reconstruct an MRI image. There's multiple different metrics on how to reconstruct an or how to measure the reconstruction quality of an MRI image. Um, if one method really does well on on one of the metrics or one method does well on one of the metrics, not well on one of the other but then your own reference solution does it the other way around. Which one is the correct one? Right? They're both good. One is better for one metric, one
5:34:12is better for the other metric. So, you really need to kind of design the verifier to be um tolerant towards all possible valid solutions. Yeah. Yeah, that resonates a bunch. I mean, in a bunch of our work too and you know, Russell, what you were saying, um, grading calibration ends up being a very important step of the process. How do you actually make sure the mechanism is aligned with how an expert grader would actually facilitate this, but at the same time, it's often very hard to um
5:34:40fully scale, you know, a manual approach there. So, some version of automation or verification that reflects that judgment is is a kind of name of the game. I mentioned this in another session, but um you know our our senior SWE bench approach where you have experts almost see the like test specs and then um have agents then propagate those to exercise code um felt like a a fun trade-off here, but there's there's a real art to the grading design that you know has been cool to see in all of your work. Um
5:35:10would love to talk about um another hot topic. Uh benchmaxing has been mentioned a lot today. Um, you know, you guys have all kind of tested model capabilities in different ways. Uh, you've seen scores, you know, change over time. Um, what are your thoughts on, uh, I guess the responsibility of the community versus benchmark builders to help mitigate, you know, some of the risks of playing to the test?
5:35:38H um in the legal domain I think there happened to be a lot of private benchmarks but I think the reason why is not necessarily because of benchmark benchmaxing concerns but instead because of confidentiality concerns or because tasks have client data in them. Um but I I think in general like a private hold out test would be something I'd be looking at.
5:36:03Yeah. I mean, I think overall a good benchmark should be a very opinionated bet on where the future's going. And if companies benchmarks towards your benchmark and you do a good enough job, then it's a net benefit to everyone. But like obviously we don't live in a perfect world and there's going to be issues with benchmarks and little quirks that you can go and exploit or swarms that go off the rails and all these different types of things. So I think you know it's also amongst this area where you need to understand that you
5:36:32will not build a perfect benchmark. You need to have not necessarily safety measures but held out tests of course held out examples but just ways in which that you can ensure that the quality that people see when they're making decisions about which agent to use, which model to use are reflective of really how they are and not just your necessarily compromises you had to make when making the benchmark.
5:36:56Yeah. Yeah, I don't have a straight answer. Maybe we can have like a discussion.
5:37:00Um, so first of all, if models just improve their general capabilities and just do better on a specific benchmark, that's great. Ben, it kind of depends on what we define as benchmaxing. Is benchmaxing specifically trying to, you know, improve the score in a specific benchmark that then also but that doesn't really translate then to reality. That's bad. So that's kind of the negative connotation of benchmaxing.
5:37:23So I I think that's what we mean by benchmaxing or what how do we No, I I agree. It was intentionally framed as a bit.
5:37:29So if we define this as benchmarking baxing that's bad m different labs benchmarks to a different extent if we use this uh I'm not sure how much they check the training data that they buy. Um I I personally with my own eyes saw someone trying to sell data to one of the labs in April. This is two months after we started working on tour science. that is very similar to turbine science. So there's a lot of like looking at our pull requests trying to change the tasks a little bit then
5:37:59selling that to training data as training data to the labs. I think that's very bad. I think you should develop completely kind of independent uh tasks as a as a as a data vendor and the labs should also check what training data that they buy otherwise you yeah you you end up benchmaxing. So that's very bad. Um, and something that I'm concerned about and we don't really have an entity that governs any of these things. Um, I'm wondering what you guys think. How would how would we control these things?
5:38:28I think you brought up a lot of good points. I mean, my personal view, one, I think the definition of benchmaxing is important to all of your points. On one hand, all of machine learning is setting targets and then hill climbing towards them. So to your guys' definitions, so long as the targets reflect sufficiently general, diverse, continuously updated targets, arguably that's actually a good thing, you know, uh but on the other hand, right, there's so much economic and other incentive to, you know, do
5:38:57well on these benchmarks to the extent that, you know, we sometimes see funny business, you know, which we definitely don't don't want in the space, you know.
5:39:05So on the on the benchmark building side, right, it makes our job a bit harder, right? there's a much higher bar when it comes to the individual task quality. Um are your graders, as we've been talking about, admitting, you know, exploitable um you know, solutions. Are they actually capturing what we care about? Um there's a there's a kind of benchmarkwide contamination issue, right? When you're designing your benchmark, should you consider a held out set versus a public set? And you know, increasingly we're seeing that trend more common because then you can
5:39:34study distributional differences in model performance across each of those sets to kind of understand what's actually going on. Um, and then there are kind of community efforts too. I mean we've been big fans of you know some of the independent evaluation pushes more broadly um more ways to audit and you know honestly give tools to labs to audit their own training data and evaluation data more broadly. I don't think it's a single, you know, silver bullet solution, but it's a combination of, you know, community, you know, broader governance and just kind
5:40:04of overall um rigor, you know, in in the science of all this. But, um, you know, my optimistic self is that the more benchmarks we build, the more diversity of use cases we capture, the the better it is actually for the field because you actually just get to represent the pluralism of of all the different types of capabilities people care about, you know, which is in my view strictly a good thing. And it seems like a lot of the work can be done or is being done by people creating the benchmarks in terms of making sure that like the breadth and the depth are there so that if a model
5:40:33does kind of benchmarks on that benchmark it is able to generalize to other kinds of tasks.
5:40:41I guess it's a an issue of do like is the benchmark itself well designed and it's something that you know right is well for lack of a better term aligned with what we as a community we as you know professionals want the field to go towards and I think it's all about how much care you put into making the benchmark that will determine if it is significantly or statistically robust to you know different levels of hack
5:41:11hacking if it is robust to like these weird cheats that you can get around and I think there's also the issue beyond that and you know discussing benchmaxing and how you design a benchmark and held out versus not that there is a discussion of should you allow open internet right I think that's a key issue of this benchmaxing is that if you want a realistic benchmark you have to let it be open internet right agents in the wild get open internet access but then they get access to the solutions straight from that and it's a very weird
5:41:40trade-off off and balancing act that we have to do. But I think fundamentally if you take the care and if you make your benchmark robust and you really put in the effort to ensure that you have high quality solutions, high quality grading, then I think benchmarking on it is fine.
5:41:57But the the issues come in when you know corners are cut, right?
5:42:02Yeah. Well said. Well said. More discussion here. More discussion here.
5:42:06Um I will flag to the audience. We will be opening up for some questions. Uh but I will ask one more question to get your brains thinking. We got some good ones here. Um okay. Um I wanted to separately ask um you know having gone through this process having you know exposed your benchmarks and published some of this work and gotten feedback from the community. Um what are some things you would have done differently or said another way what are some lessons to other folks building benchmarks or studying um you know how to elicit
5:42:34signal from experts and and inject that into you know studies of model capabilities? um what what are your learnings there and what would you pass on to the next wave of of benchmark builders? H I think for us we kind of had to put everything together really quickly ahead of the Nurups deadline.
5:42:50And so just like this sounds this sounds like obvious but just trying to plan out as much as possible in advance of like the design the annotators that we're working with, what we're going to do with any empirical results. um and have that all kind of planned out so that it like flows together versus um you know collecting data and like immediately like having to do the analysis and and write it up. That's what I would say.
5:43:18Yeah, I think beyond that really just keep it simple, right? I think the best benchmarks, you know, program bench is one of the ones that comes to mind is this very simple idea to elicit the exact signal you want, right? the simpler you keep it, the stronger the benchmark you make, the I guess harder it is to benchmax because it's a simple idea. It's a very easy to grade thing and the easier it is for the community to use. You know, there's obviously there's some elegance in having, you know, these other systems to measure different failure modes of the model, but realistically if it's hard
5:43:47for the community to run, no one's going to run it.
5:43:50And so just keeping it as simple as possible to get the signal you want.
5:43:54Uh for me number one advice is to build a great team especially if you run this like large community effort is to to get to know the people you're working with. Learn who to trust, who not to trust, build a great team and moving forward uh everything kind of goes from there. But if you don't have a good team and if you don't build relationships, uh if you just see the contributors as contributors but not as humans um that you're interested in, then it's
5:44:24difficult. So it wouldn't have worked for us if if I, you know, didn't get to know the actual people that I work with uh properly and and kind of set up the team in the beginning because the without the reviewer team and I said this before, without the reviewer team, nothing nothing in the terminal bench community works. Um and the reviewers have to be motivated. they have to be motivated for the right reasons as well and to kind of you know communicate the the mission that you're trying to achieve and um you know them believing
5:44:53in the mission is number one thing before anything technical. So that's my advice.
5:44:59Awesome. Well said. Um okay I warned you. Any questions from the audience?
5:45:05Yes, we have one here.
5:45:07Hey, I'm Trisholla. Uh I'm also a benchmark builder and we're building benchmarks in electronics and doublee.
5:45:14Uh my question was about working with uh contributors who might not be very technical. How do you set up the benchmark or the task contributor process or the benchmark building process uh to be as fluid and easy to use as possible for domain experts that might not be technical?
5:45:38Should I go? I can go ahead. Um, so one thing you can do is to pair them up with someone who is a technical person and understands the task format and all of the quirks about how to evaluate agents.
5:45:52So that's one thing you can do like a body system. Um, we have some contributors that are, you know, kind of a pair of contributors that work on a task together. One of them is a technical person, one is a domain experts. For example, in the earth science task that I showed earlier with the lake freezing and melting, um that is one CS person that has friends in the earth sciences and they're both collaborating to create this task. So I think this is a a great way. And number
5:46:18two, you need to uh write down the contribution process. More detail is better in my opinion than than less. Um so yeah if people can just read through the contribution process step for step make it very easy um on explaining on how to build a task um that is a number two thing that I think should be communicated properly. Yeah I'll just add one thing to that which is we hosted training sessions for the annotators and walked them through like
5:46:47this is exactly how you use the platform. This is who you contact if something comes up. Um, and I I think that solved a lot of the problems.
5:46:58Awesome. Thank you.
5:47:04Hey. Um, another benchmark benchmark builder. Um, one question I had is like what how do you actually choose the benchmarks that we you guys want to build? like what is there any like uh heristics like anyway anyway has to like exploit some like model capability that the models are currently lacking but that just keeps like increasing like at uh an exponential. So how do you actually choose like okay these are the aspects that I want to like build the benchmark on and then how do you actually like uh terminal bench is like
5:47:34one benchmark that keeps like improving continuously. So, what are the systems that you actually like build to keep up with the frontier?
5:47:44Um, to keep up with the frontier, we don't really like set expectations for scientists on what problems they should submit or not. Of course, they have to be difficult, but um because they are the domain experts, we kind of want to trust them uh on whether the tasks are difficult enough or not. Um but but there are some tasks that are just clearly too simple and you can tell sometimes from the proposal itself that
5:48:13this is a very standardized workflow and standardized workflows are very well documented on the internet they're probably too easy. If a proposal is a is a very niche problem in a specific scientific domain then that one has way more potential to become a challenging interesting uh task in which we can you know identify interesting failure modes to then you know fill in the gaps. Um so that's my answer. Yeah.
5:48:40Yeah. Yeah, I mean I think it's essentially you have to have a very robust either PR system or some expert annotation to show that this is something that even if you know agents are able to get say like 70% on the task that the headroom going from 70 to 80 is so significant that it's worth having and I think that's the only way you can keep up with the frontier and of course you know even beyond having expert contributors submit PRs you can mine problems you can create problems from you know existing repos at least in my
5:49:09case for the coding things and I think you just need to be very explicit with what your standards are and you need to set those standards and never deviate from them.
5:49:24Great. Yes, another question.
5:49:28Oh, one here too. Let's do this one first. Right. Relating to this um is you talked about codifying uh uh some of your uh innate preferences and the current way you're building EVOs is inherently not scalable. Um do you have any uh thoughts on basically can you pontificate a little bit on on how we can automate say 80% of your jobs as uh benchmark builders and how we can scale
5:49:54up this entire process in the future.
5:49:59Do you mind repeating the question? And I heard only heard scale.
5:50:02Yeah, my let me know if I got this right. I think the question was uh it's becoming um really hard to to uh build tasks and how do you actually scale this?
5:50:15It's been a cat and mouse game of uh I guess uh benchmark builders uh going after certain types of evals and uh this process has innately been uh uh not so scalable. So how do we automate 80% of our jobs as benchmark builders right and codify all these different things and choose an unh harness or come up with an industrial pipeline where we can really increase throughput of all of this how do we automate 80% of our jobs as
5:50:42benchmark builders and codify that into systems etc.
5:50:48I think the idea of acquiring the assets of a company like internal documents and messages can be really interesting if you can find a way to get useful tasks out of that. Um I think we've seen that a little bit recently in the news. But the question there to me is like how do you go from all this like unstructured um messy data um like longitudinal as well and turn it into like discrete tasks that can be checked by an expert and then used as part of a benchmarking exercise.
5:51:17I think honestly if I could automate 80% of the benchmark creation it would be much easier to make better benchmarks. I think you know across the board we're not even close to maybe 20%. You know I can get as many tasks as I need you know mine in infinite number of checkpoints for code but ensuring their quality is still a very difficult thing and agents cannot do that well. Ensuring that a benchmark is solvable, that there's no issues, that there's no, you know, potential for a task to be completely unsolvable or ambiguous is incredibly
5:51:45hard. So, I think going towards automation is a much better way for us to go, especially if we can have benchmark makers focusing on that last 20%, the important parts about taste and the important parts about is this something worth measuring is a net benefit to everyone.
5:52:04Yeah, I'm when it comes to turnaround science at least, I'm against automatically generating tasks. Um, I think automation is great when it comes to reviewing tasks, but then again, not full automation. It's still needs a human that kind of goes over whatever automated output is produced. Um, we have 70 tasks. Do you guys think 70 task reflects all of science? Definitely not.
5:52:27is way way way way more and more interesting challenges and tasks out there in science across many different domains and um I think the bottleneck is actually to communicate to the scientific community why this is a good thing for science itself and for driving scientific progress um we get a lot of backlash as well as like wait aren't you developing a mechanism that is going to take away our job and then you know you have to kind of
5:52:56work with that you have to say why this is good for science. Do we need to consider the individual people? Is science itself more important for humanity? So these kind of conversations need to be had with people that uh still mistrust the kind of things that we do.
5:53:11I'm talking about automating our jobs not not the task laborers.
5:53:16I'm talking about how to automate our jobs as benchmarkers.
5:53:20Oh, your jobs. Yeah. Yeah. Yeah. I see.
5:53:21As opposed to the the task builders. I guess it's very hard. No, but I still I don't want to automate the relationship that I have to the scientists like qual quality for evals training data is something else. Quality for evals is way more important than quantity. Again, we only have 70 tasks which is actually quite a lot for for a benchmark uh these days for an agentic benchmark. Um the internal bench philosophy is always quality about quality. Um and if automate automation gets into the way of the quality then we're going to say no.
5:53:58Yeah. Um just kind of uh piggybacking off of that question, right? Um if you were to break down kind of every like uh second that you spend like working on benchmarks, right? Like how would you break it down? Like what percent is sourcing people? What percent is thinking about the specific quality of a specific task? What percent is like setting up infra kind of what are the bottlenecks?
5:54:21Do you mean for for the entire team averaged across all the people working?
5:54:27Good for you as like the kind of headline person who's bringing everyone together.
5:54:31Okay. Me personally as the project personally.
5:54:34Uh number one thing is getting people together. That's definitely the the biggest thing if I had to put a percentage on it.
5:54:42Yeah. More than 50% I think is I mean in the in the beginning we had a whole cold start problem. We didn't have anyone. We didn't have any re reviewer team and I needed to go out to scientific labs at Stanford, Berkeley, travel to the east coast as well to talk to professors, get contributors. Um, and that was a huge amount of time. Number two is infrastructure. Like you, if you want your reviewer team to be really focusing on the things that they are meant to do, you need to build systems and
5:55:10infrastructures that make it as easy as possible for them to work on the kind of things that you want them to work on. So they don't have to, you know, deal with things that are very unimportant and this is not working, this is not working. They should really focus on reviewing the quality of the task and there's a number of things that you as the lead of a project can can build to make their lives as easy as possible.
5:55:32And the same goes for the contributors. Uh so this is number two, uh outreach infrastructure. And number three, you need to set an example and and actually, you know, know what you're looking for. You need to be able to review tasks properly. You need to be know what is a good task, what is a bad task so you can communicate that to to the team. So those are the three things.
5:55:54Yeah, I would say 90% of my time is spent reviewing problems is spent ensuring quality. I think it's better for a benchmark maker to be very paranoid about the quality of their benchmark. Um there's really no limit to how much you can review your problems. I think the other you know 5% is infra and I think you know infra [clears throat] unrelated to quality right so quality is all-encompassing of like reviewer infra automatic checks on what they do automatic checks on the problems
5:56:23themselves and trajectory reviewing and then the last 5% is writing and always keeping up to date on what we're doing on where we're going and trying to find new problems to add I would totally agree with both Stephen and Gabriel I think people was the biggest component of what I spend spent time on um not just how you train the annotators to use the platform correctly um but how you make sure that you have annotators that are consistently delivering high quality work. What you
5:56:52do if that's not happening um and yeah I think that was by far the the most the major part of my time.
5:57:01Awesome. Cool. I think that's all we have time for. Um, so folks could give a big hand to the to the panel here in the lightning talks. Thank you guys.
5:57:15Cool. Well, we will wrap up here. Um, for folks in the next session, we have a really great poster session out the door on the left.
5:57:25Yeah, I just Please join that.
5:57:28I think there's an also an a happy hour going on. uh if you have a chance to talk to each other, talk to the poster builders, um it's a really really great community of folks who have um actively done the work. So we we try to really create a research community here of folks who are actually you know engaged. So it'll hopefully foster some good discussion and hope to see you out there. Thank you all.
