We speak with engineers and HalachaBench creators Ben Niewood and Sam Weiss about how AI performs on rabbinic ordination exams.
In this episode of the 18Forty Podcast, we speak with engineers and HalachaBench creators Ben Niewood and Sam Weiss about how AI performs on rabbinic ordination exams—and what that reveals about halacha, rabbinic authority, and the future of Jewish life.
In this episode we discuss:
Tune in to hear a conversation about how “the process is as important as the question in lived Judaism.”
Interview begins at 4:16.
Ben Niewood is a mechanical engineer who builds robots, and Sam Weiss is a software engineer with a background in machine learning. Together with Ilan Weitzman and Jesse Bernstein, they created HalachaBench, a benchmark that tests how today’s leading AI models perform on Israeli Rabbanut semicha exams. Their results, first shared on X, drew attention from across the AI world.
For more 18Forty:
NEWSLETTER: 18forty.org/join
CALL: (212) 582-1840
EMAIL: info@18forty.org
WEBSITE: 18forty.org
IG: @18forty
X: @18_forty
WhatsApp: join here
Transcripts are produced by Sofer.ai and lightly edited—please excuse any imperfections.
Dovid Bashevkin: Hi friends, and welcome to 18Forty, where each month we explore a different topic trying to balance modern sensibilities with traditional sensitivities to give you new approaches to Jewish ideas. I’m your host, Dovid Bashevkin, and today I am so excited, because we are exploring the intersection between AI, artificial intelligence, and halacha, Jewish law. This podcast is part of a larger exploration of those big, juicy Jewish ideas, so be sure to check out our website, 18forty.org, that’s 18, F-O-R-T-Y.org, where you can also find videos, articles, recommended readings, and weekly emails. So I’m not sure when it was, but very early on when AI was just emerging, and really even in our original intro video, I realized that a lot of what was going to steer and drive the work of 18Forty was understanding how modernity and how the world that we live in today really shapes and affects our religious lives.
They’re not two separate worlds, but the culture, the ideas, the technological innovation shapes our very sense of self, which in turn shapes, in many ways, our approach to religious life. And the questions that emerge regarding AI have always fascinated me, not because I am all that excited about technological innovation—I actually am not, and I don’t even know that much—but I have long been interested in ideas related to consciousness and self, to really understand what makes us most human, what is our self, what are we experiencing. And those ideas, especially as AI got stronger and stronger, really highlighted, through the inverse, as computers do more, we start to ask, like, so what is our humanity about? And last year, around this time, on 18Forty, we did an entire series with our partners at American Security Foundation, ASF, an incredible, incredible foundation dedicated to the intersection between religion and AI, and we did a series, and you can check out some of those episodes, whether it’s on our YouTube page or previous episodes. But today’s episode is actually something a little bit different, and I have to give a big thank you to my brother-in-law’s brother, the best Jewish introduction to any podcast, for introducing me to two incredible computer scientists—one’s a mechanical engineer—who were doing incredible work in how to evaluate the competence of AI in approaching halachic questions.
And my brother-in-law’s brother, Brett Spier, thank you Brett, showed me that they tweeted on social media under the handle HalachaBench, because benchmarks, as our conversation discusses, is a major, major theme in evaluating AI, and they developed HalachaBench. How does AI perform when it comes to rabbinical, halachic knowledge, knowledge that emerges not just from the Talmud, but from all of our codes of law, the Shulchan Aruch, etc., etc.? And I shared their research on my own Twitter feed, my X feed, whatever you call it, and happens to be I have a little bit of a bigger following than they did at the time, and I saw the conversation emerging. All of the major players were really, really interested in what their work showed and didn’t show about the development of AI. And I invited them to come on to 18Forty and talk a little bit not only about their own work, give a little bit of an introduction, really, just like a primer on what AI is all about, what their work says about the intersection between AI and halacha, why the AI world—and this is absolutely true—is so fascinated by Talmud and Jewish codes of law.
And if you haven’t seen it already, aside from reaching out to religious leaders, so many people in AI are fascinated by the Talmud. And we speculate a little bit as to why, and then finally, we talk a little bit about what they built and really what the future holds in terms of the intersection between halacha and artificial intelligence. I am so excited for this conversation, and I know you will enjoy it. I am so excited, and really, it is such a privilege and pleasure to have Ben Niewood—
Ben Niewood: Yeah.
Dovid Bashevkin: Niewood, and Sam Weiss. And you came onto the scene, and I have to mention him right up top in the beginning, my brother-in-law’s brother—could you think of a more Jewish introduction to a conversation? Not at all. My brother-in-law’s brother, Brett Spier, who I know and I love, is not usually the person who is sharing with me breakneck innovation in the Talmudic world, we got to give him credit, he is the one who connected me to you, and you broke out onto the scene because I shared on Twitter—it wasn’t so popular when I shared it, I don’t think it had—
Ben Niewood: No, I mean, we had just posted it. It was nowhere.
We were just We just put it out into the world and thought, is anyone going to see this? And then Brett said, “Oh, I can send this to my brother’s brother-in-law.”
Dovid Bashevkin: Yeah, brother’s brother-in-law.
And—
Ben Niewood: we were like, “Yeah, sure, go for it.” And, you know, then you posted it, and then it got really picked up, for sure.
Dovid Bashevkin: And it was called HalachaBench, and what you did is you had several different AIs all taking rabbinic ordination exams, what are known as semicha bechinos. Semicha is rabbinic ordination, and bechinos, a test.
And you basically took all the different AI models and had them take different rabbinic ordination exams. And before we kind of get into like what that tells us about AI, what that tells us about rabbinic ordination and halachic progress, and we can have that conversation, I want to start way back with the basics. And let’s talk about even the title, Halacha Benchmark. Explain to me what is a benchmark as it relates to AI?
Ben Niewood: In the simplest sense, a benchmark is checking how good AI models are at certain things.
They came out because originally the first models that came out were only so capable, they could do some things, but not others. So people wanted to see, okay, how good are the models at taking the SATs or taking, you know, the LSAT, or, you know, deciding the outcome of a legal case. And so people started to build these benchmarks. I mean, I’m a hardware engineer, I’m a mechanical engineer, I build robots, and there’s just now the models are getting really good at hardware-type design things.
And so there’s all these new benchmarks coming out now, HardwareBench, that’s how good is the AI at designing a hardware system. So it’s important cuz it helps figure out, as the models progress, can you treat them like a software coworker, like a legal coworker, like a hardware coworker, or how much can you trust them to do well at different tasks?
Dovid Bashevkin: And where we are now in the development of AI, for most benchmarks, like, let’s say, an LSAT or an SAT, are most of them able to score exceptionally well on an SAT, a GRE?
Sam Weiss: Yeah, I think for something like the SAT, where it’s a very structured test, especially, you know, it’s multiple choice, and the way that the test is put together is is very orderly, I don’t know the benchmark numbers offhand, I would assume they do very well on those. On other types of benchmarks where it’s more open-ended, like if there’s other tests that are more essay-based, even though I think they might do less well, but I think even on like the bar exam they’ve been taking, they’re certainly passing the test, and I think probably doing very well.
Dovid Bashevkin: Your background is a little bit more closely connected to the world of AI.
Where is your work? Ben, you work in robotics and mechanical engineering, and your background is where?
Sam Weiss: I’m a software engineer, I’ve done a lot of machine learning, which, you know, is another word for for AI, and I’ve worked at a number of different tech companies doing that, so that—
Dovid Bashevkin: Let’s talk a little bit about why what we call these are all really what we call LLMs, large language models. Why would a large language model be good at some tests and not at others? Like, I remember when I was in school, the kids who do really well on standardized tests, like, they do well on all the standardized tests. Like, they’re like they’re so good at it. And the kids who just like, you know, 1550 on the SATs or however it’s scored nowadays or the ACTs, those are the kids, the MCATs, the LSATs, like, and there were other kids who like you needed the tutors, you needed the books, but there were certain kids they just picked it up.
When we think of an LLM, can you explain why is the structure of how an LLM learns would make it good at some intellectual tasks, but not as good at others?
Sam Weiss: Yeah. So if you think about the way an LLM is trained, right, an LLM is a is a type of neural net, so you essentially have this big web of of weights, right? And the way they train it is those weights start out—the connections between them start out effectively random. It’s a bunch of numbers, right? And they feed data into it. So they feed in books and the entire internet, and as they do that, it sets these weights to to sort of recognize those patterns in the training data that—
Dovid Bashevkin: Is the machine doing setting the weights on its own, or that’s the programmers doing that?
Sam Weiss: So the algorithm that trains it is setting the weights.
So essentially what they do, there’s something called like next-token prediction, right? Where like you you essentially will will give it The task that it’s trying to learn is, given a bunch of input, can it predict what should be the next output? So you would, you know, give it a bunch of text, and maybe you know what the next output should be, and the machine doesn’t, you have it try to produce what should be the next output, and if not— it does well, it gets rewarded, and if—
Dovid Bashevkin: So you might say, “Old MacDonald had a—”
Sam Weiss: Exactly, right.
Dovid Bashevkin: —and like, can you fill that in that you know it’s like a farm, and now it sees that it knows the language of that. Okay. Right.
Sam Weiss: So that that’s roughly like what’s going on there. So they’re they’re learning these statistical patterns. So then your question about why are they good at some things and not others? So some of this honestly is like not entirely well understood, like how these weights, you know, giving it huge amounts of training data actually produces a lot of these capabilities is not entirely clear. A good example is is math, right? You might remember a few years ago, you could ask an LL— LLM, you know, even simple addition problems, and they did not do well on them.
And the reason is because they were trained on text. They’re recognizing statistical patterns in text, and maybe the phrase 1 1 2 comes up a lot in text, so maybe they can do that. But then more, you know, arbitrary combinations of things don’t, and so it can’t even add a few numbers together. But the labs that are working on these models, they’ve recognized those deficiencies, and then they steer the training data and the training process to like account for that.
And that’s actually coming back to the benchmarks, where the benchmarks come in, where people might publish, let’s say, a math benchmark, and then the makers of these models, the labs, can assess how well their model is doing against those different areas, and they can get better and better and better that way.
Dovid Bashevkin: So, you might benchmark might be like the ninth grade We took the Regents when I was growing up, so you might take the ninth-grade Regents. What’s very interesting to me about LLMs, machine learning, and AI is it kind of sits on the nexus between math and language, and this goes back to the original computing. Now, I’m the kind of person who’s like I’m dangerous because I have a lot of general information, but I’m not an expert in anything.
But one of the thinkers who I am so fascinated by, I’m fascinated by his theories and ideas, is Claude Shannon. Claude Shannon is really like the grandfather of all information theory, and his basic idea, which I think must be connected in some ways to these LLMs—I don’t know if you’re familiar with it—but his basic idea, he was involved in codebreaking, and the basic idea that he came up with was that all information can actually be broken down, this is the bit system, how statistically possible it’s going to be. And when you’re transmitting no information, like if I roll a die, and I say, “This die always, all the time, falls out on four,” and then I roll the die, so there’s no new information being disclosed. If I roll a die that could fall in one of two ways, so that’s like I think like a bit of information or two bits of information, and we’re able to process more and more and more based on just how how statistically complex is this.
With LLMs, is the way that they’re processing information kind of like they’re learning the statistical likelihood of the next word or the next piece of information?
Sam Weiss: On a very basic level, that’s right, right. That that’s like the the elemental level of what they’re doing. And and it’s very connected to like Claude Shannon’s ideas, right? So, he had a lot to say about compression, right? Like, what’s the most efficient way you can compress information without losing anything, or with losing things in certain ways. And you can think of an LLM as like, effectively, a method of compressing information, right? You’re taking a huge amount of information, and you’re compressing it into some smaller space, but you’re still retaining a lot of that information in some way.
And you’re also there’s, you know, obviously it’s learning connections between those things, but it’s kind of a fuzzy picture of all of that information that you’ve put in.
Dovid Bashevkin: It’s so fascinating. Now, let me ask you before we get into the actual halacha benchmark, and your work in AI, let me just ask you one more question: What is your relationships in your own lives with, you know, whether it’s ChatGPT or Claude? What do you use them for? Is there anything that you deliberately don’t use them for because you kind of have more information than most?
Ben Niewood: Yeah, I mean I use it for almost everything these days. There’s very little that I won’t go to to Claude or ChatGPT for first.
In in work, I use it all the time. It’s a a thought partner. It helps speed up work. I use it, like I said earlier, they’re getting better and better at even hardware work and mechanical design, and so I’ll kick off tasks and use it as almost coworkers or or people who work for me and, you know, kick off agents who are running different projects that I would have to do myself, and now I can rely on the early steps of those projects to be done by an AI.
On a personal level, I use it to, you know, help manage my inbox, my calendar, order groceries for me, meal plan.
Dovid Bashevkin: Really?
You let AI send emails without checking them first?
Ben Niewood: I usually tell it to give me a draft, but if it’s not something I care about, then, you know, maybe, or if it’s not to a person I care about, then maybe. I definitely have it, like I have it meal plan.
Dovid Bashevkin: To all Ben’s friends, double-check—
Ben Niewood: Yeah, exactly.
No one ever really knows anymore if it’s me or— Yeah, and certainly for work, I I read the emails, but I have it write emails for me for work all the time. You know, “Oh, can you check my inbox and follow up? Tell me what you’re saying, what new information there is.” So I don’t even have to read my email anymore because Claude just, you know, digests everything and tells me, pulls out the important bits and then archives all the other stuff. I use it to meal plan, and now Sam uses my meal planning engine on Claude these days every week. But, but yeah, it’s all, it’s like I use it every day.
Sam Weiss: I use it all the time. Our recipes were too complicated.
Ben Niewood: Yeah, he wanted easier stuff.
Dovid Bashevkin: What’s what’s your relationship, Sam, with with AI?
Sam Weiss: So professionally, and and also as like a hobby programmer as well, right, they they have completely taken over software engineering, right? Like the way that software engineers write code is completely different than it was even like, you know, nine months ago.
They really can do a lot of tasks now end-to-end. You They still make mistakes, and you have to oversee what they’re doing, but they’re really incredibly capable. It’s one of the domains that they’re most advanced at. And then in my personal life, I think maybe a little bit to less of an extent than Ben, but I’m I’m doing very similar things.
I think like I also do think about, and this maybe will connect to some of the halacha discussion, there’s some danger there of like relying on it for certain things, right? Like, should this email that has a like are you losing your voice in some way if you’re letting the the AI write this email for you, right? For some emails, maybe that doesn’t matter; for others, it does, right? And I think that’s something that people need to think carefully about. I think there’s also a danger, and you see this on the programming angle as well, of like losing your understanding of the system if you just kind of like let it let it run, which for some things doesn’t matter, and maybe if I’m building a personal tool for something, I don’t I don’t care, but for other things, I really want to be able to think back in three weeks, be like, “Oh yeah, how did that thing work?” And if if you’re more involved and hands-on and working with it as like a partner than letting it just run with something, I think that’s it’s much easier to still have a handle on what’s going on.
Dovid Bashevkin: And most, generally speaking, I just really want like almost like a thumbs up or thumbs down from each of you. The the angst and anxiety about like the doom coming of AI, mass job displacement, you know, people kind of losing their sense of humanity, major economic upheaval, would you each put yourselves on team, you know, optimism or team doom? I’m curious for each of you.
Ben Niewood: I feel like I go back and forth every day depending on what’s going on that day. I mean, I think I generally am on the team optimist side. I think it’s a great tool right now as people are using it. I think there certainly will be some amount of economic upheaval.
There’s going to be things that change over the next two, five, ten years, no doubt, but I I generally fall on the side there’s a book series, a sci-fi series that I love called The Culture series, where the AIs end up as kind of benevolent overlords. They protect humanity. They let us all like we get we get to just go do art.
Dovid Bashevkin: That’s the best-case scenario?
Ben Niewood: That’s the best Well, I don’t know about best-case scenario, but we we get to go live.
All all humans in this world are It’s a world of abundance. Everybody has everything they need, and everybody can just spend their lives doing art and culture, and that they’re called The Culture because it’s all about the the best people in the world are the ones who, you know, do make plays and music, and I think that that’s a possible outcome that could be, you know, hopefully is the best-case scenario, where it’s always going to be something that just is there to help us.
Dovid Bashevkin: And what about you? Where are you on on this?
Sam Weiss: I also go back and forth. I think, in general, I’m an optimist, but I also think that that optimistic outcome doesn’t happen by itself, right? It takes human beings to actually think about what that looks like and steer it in the right direction.
Dovid Bashevkin: Yeah.
Sam Weiss: So, I don’t think it’s like, “Oh, we can just, you know, see what happens,” right? There’s certainly danger.
Dovid Bashevkin: Yeah, we need—
Ben Niewood: to guide it.
Dovid Bashevkin: Yeah, I appreciate that.
So, let’s shift and start to talk a little bit about what you shared really with the world. I mean, it got more than the 100,000 I’m just looking at just how many impressions, how widely it was shared. It was really, really remarkable, and there was a great interest within the AI world about what you built. Now, explain to me, Mose, basically, what is Halacha Benchmark?
Ben Niewood: Yeah, so we we wanted to see how, like other benchmarks that we discussed, LegalBench, HardwareBench, how good is AI at I don’t know what our first thought was, being a rabbi, understanding halacha.
We, you know, we kind of were throwing a couple ideas around, and we settled on, okay, if we’re going to figure out how good is AI at paskening halacha, or how do we determine that?
Dovid Bashevkin: Paskening is the word that we use to kind of like deciding like what what practically, taking all rabbinic literature, taking all the Talmud, all the Mishnah, I have a question in front of me, is this kosher, is this not kosher, tell me how to proceed, we talk about that decision as a psak.
So, please continue.
Ben Niewood: So, the first way we came up with to kind of understand that, as benchmarking the models in that world, was can they pass a test to become a rabbi? You know, if if that seems like the best way to determine is this going to pass the the mark of, you know, are you good enough to do this, because those are the people who give us those decisions in our world. We go to our rabbi when we have a question about how do you handle a situation, and so we wanted to see, okay, people are going to LLMs for these.
We know that people are going onto ChatGPT and saying, “Can I use my dishwasher for, you know, meat and then for milk, or can I do this on Shabbos,” or whatever. For sure, people are asking these questions. So, we wanted to understand We need to know how good the models are at answering those questions truthfully and faithfully, and, you know, that’s what that’s what we were kind of trying to figure out.
Sam Weiss: I think it started around the time that OpenAI and and Anthropic were dropping these solutions to all these like long-standing open math problems.
Dovid Bashevkin: You mean big math problems. That would be make the front page of The New York Times.
Sam Weiss: Yeah, absolutely.
Dovid Bashevkin: And the people who were the most upset were the mathematicians.
Sam Weiss: Right.
Dovid Bashevkin: We’ll see how the rabbis react.
Sam Weiss: We’ll get there, but— Yeah, we’ll see. Right.
So, it started, we were making some jokes about like, oh, you know, can is AI going to be able to like solve longstanding open problems in halacha? Are we going to be able to keep, you know, one day of Yom Tov in, you know, a bit of a joke, it does obviously doesn’t work that way, but that got us thinking about it. We’re like, wait, maybe there’s something interesting here that like we can we could work on.
Dovid Bashevkin: So, so the next step, you have this idea, we have other benchmarks, we have benchmarks in legal thinking, we have benchmarks in mathematical thinking, there are benchmarks in cybersecurity cybersecurity. Is there benchmarks in like literature?
Sam Weiss: There probably are.
I mean, there’s so many of them now, like there probably are, yeah.
Dovid Bashevkin: Yeah, there’s so many. So so you now have to do halacha benchmark. I would imagine, and maybe this is just because I know so I feel like the hardest part is like how did you get a hold of all the different semicha bechinos, the rabbinical ordination exams? That could be hard to get your hands on.
Ben Niewood: Yeah. There was not Yeah, it was not. Yeah. To start with, like I said, we were kind of throwing ideas around about how do we do this.
And so I went to Claude. It was not it didn’t solve this problem on our own. We used AI to figure out how to benchmark AI, which, you know, we can talk about how how that makes sense.
Dovid Bashevkin: Oh, interesting.
AI was helping you create the benchmark itself.
Okay.
Ben Niewood: Absolutely. Yeah, so I went to Claude and I said, “Hey, I want to do this.
What are some ideas? What are some different things we could do?” It threw out a bunch of different ideas. Obviously, you need a set of questions to be able to benchmark against. I can’t write those questions. I don’t have semicha.
I’m, you know, not a rabbi, so we didn’t feel qualified to do that on our own, but Claude did some research for us and it found that all of the Israeli Rabbanut semicha tests are available online and you can download them. And very critically, there were a number of test questions that were posted after the model training cutoff dates. So the models had not seen these questions before.
Dovid Bashevkin: When you talk about the models, you’re talking about the the big LLM models? The more prominent ones? ChatGPT, Claude?
Ben Niewood: Big LLM models, yeah.
Dovid Bashevkin: Exactly. And when’s that cutoff date?
Ben Niewood: It’s different for each model, yeah. Depends on the model.
Sam Weiss: It’s when the model was published, essentially.
Dovid Bashevkin: Gotcha.
Ben Niewood: So new models are dropping this week, like we might not be able to benchmark those, cuz likely they have seen the questions, cuz they’re too—
Dovid Bashevkin: Cuz they already saw the whole internet.
Ben Niewood: Right. Exactly.
Sam Weiss: And and this one of the ways that a lot of these benchmarks work, which we can get into how we might improve our benchmark to do something like this as well, but a lot of the way that they work is they write a set of questions themselves or they produce a set of questions somehow that they keep completely private. They’ve never been on the internet, no one’s ever seen them, right? And then they might publish some examples so that people that are working with the LLMs can understand what the questions look like, but they’ve never seen them before. And that way they know that there’s no essentially cheating going on where the model might have trained on that data.
Dovid Bashevkin: It just saw the answer.
It doesn’t really understand it, but it’s just regurgitating.
Sam Weiss: Right, right.
Yeah, the semicha exams that we’re using are from October of this year, right? So if we had used ones from a year ago, two years ago, the models you you couldn’t trust those answers, right, because the models almost certainly have crawled the internet and read those exams, right.
Dovid Bashevkin: Interesting.
You know, and it’s probably getting harder and harder as more AI is coming out to find those unique, innovative questions. It reminds me of something, and this is maybe our generational gap, but there used to be something called a Googlewhack. I’m not sure if you know what that is. A Googlewhack, and I’ll tell you exactly why I know what this is, is when you would put in a search query of at least two words that are actual English words in the dictionary and only get one response from Google.
Sam Weiss: I heard of this. Yeah, I remember this, yeah.
Dovid Bashevkin: And I had a close friend, I mean we’re still close, we don’t talk as often, named Chaim Schuss, and you can look it up because on the Googlewhack websites, he was, and this is I can tell you exact this is in 2002, and he hit a Googlewhack with the term “parochial noctambulist.” I forgot what that means, but then and they had a website of like all the Googlewhacks that people found, but it’s impossible to get a Googlewhack now. Any two words aside from the fact that all the Googlewhacks, you’ll get those results, but you’re not going to put in any two words that is only going to get one result on Google anymore.
That’s impossible. And it’s probably getting harder and harder to come up with questions that the AI has not already seen.
Ben Niewood: Yes.
Sam Weiss: It’s a similar principle, yeah.
Ben Niewood: That is one of our questions about if we even can devise a set of questions that the AI will not have seen before, even if we keep them private, cuz there’s only so many questions you can ask for semicha, I imagine, and so maybe they’ve seen all the questions before. So we we don’t know. That’s something we need to look into still.
Dovid Bashevkin: So tell me a little bit about what we learned from the outcome of the different benchmarks.
Tell me which AIs did you test and how did they do?
Ben Niewood: So we tested all of the frontier models, which are models models from labs like ChatGPT, Anthropic, Google, Grok, yeah, from xAI. We we’ve actually recently tested the Muse models from Facebook. We haven’t and LLaMA from Facebook. We haven’t gotten those ones back yet.
We’re still It’s still running, actually, but.
Dovid Bashevkin: Oh, really?
Ben Niewood: Yeah, because we’ve been deciding to to get more more of them going. So we have a few that’ll be coming out later.
Dovid Bashevkin: Do different models take different amounts of time? to complete the tests?
Ben Niewood: Yes. Yes.
They do.
Sam Weiss: And we actually have some interesting stats on that, too.
Dovid Bashevkin: Okay, so I want to hear I want to hear all of this. So you use any AI that our listeners are familiar with in their homes?
Ben Niewood: We probably benchmarked the ones all your listeners will will have used.
Dovid Bashevkin: Yeah. I think my most common AI question is really around grilling.
Sam Weiss: I use it for cooking—
Ben Niewood: All the time.
Dovid Bashevkin: Whenever I’m at the grill, I use it for grilling, and grilling, I don’t know what, like it really is, and they’ll tell me exactly how to grill and move it on to—
Ben Niewood: Yeah.
“Oh, I forgot to do this. What do I do?”
Exactly!
Dovid Bashevkin: They’re really, really good. But what did you find for the results? And it’s so fascinating. I would assume that they would all either, you know, get the same kind of level of score, they you know, I don’t sense so much of a difference with my questions whether I’m asking ChatGPT or Gemini, like they all kind of you know, some voices I find more annoying than others.
I can find ChatGPT sometimes just very like, “Stop being so friendly, and like and like that you know me, like just back up a little bit.” But I’m curious, what were the results on the basic smicha test? What did you find?
Sam Weiss: Roughly that the larger models do better. Larger meaning the more complex models. So like Fable, if you’ve heard of that, from from Anthropic, which is like their largest Claude model, I I think is the leader right now. And then a lot of the larger ChatGPT models are are right up there.
There’s some interesting exceptions to that, like I think one of the Google Flash models, which is a smaller one, is pretty high, and Grok, which is a larger model, is actually down lower, but the broad picture is that the more complex the model is, the better it does. Which makes sense, because a lot of what’s going on in this smicha test is is recall. Like not entirely, obviously there’s there’s reasoning being applied there, but a lot of it is knowing the right sources and applying it to the right question in the right way. So it makes sense that a model that has presumably a lot more training data and is a larger neural network is going to perform better, and and we do pretty much see that with some exceptions.
Ben Niewood: There were a couple notable exceptions to that. Like you said, there was one smaller model from OpenAI that an older version of the model did better than the new one. And when we looked into why, the results from the older model were more verbose, and when the results were more verbose, it included more information in the answers to the questions, and so it hit more of the points that a smicha grader would look for. The newer model, which must have been told by OpenAI to be less verbose, because most people, when they’re asking a question, they want a snappy answer, it was worse because it left a lot of stuff out because it was trying to cut things back to the bare minimum, and it left a lot of things out, which is when you’re answering a test for a smicha bechinah, you want to be verbose.
You want to pack it in. And so, that was interesting.
Sam Weiss: I think there’s an interesting principle there to be learned from that, which is that like the incentives of the way these models are trained to work are not necessarily aligned with the way that you want to use it, right? If you’re asking it for a psak on something, not understanding the way that they’re trained, then you might get a subpar result for whatever domain you’re talking about, right? Because like Ben is saying, they might have gotten feedback, or they did, I remember, like they got a lot of feedback from people that these things are just saying way too much. It’s annoying.
“I don’t want to read these paragraphs of text.” So then in later iterations, they train them to be more concise, right? Because it’s a better conversationalist, and it’s it’s more readable, and it’s more efficient. It might not be good at, you know, giving, you know, rabbinic opinions. So that’s—
Dovid Bashevkin: Sure.
Sam Weiss: —important to understand.
Dovid Bashevkin: But let’s just get to the basic: who passed, who failed, and most importantly, who graded these bechinos?
Sam Weiss: Yeah, that’s that’s an important point.
Dovid Bashevkin: Like who—like again, I love both of you, and you look like wonderful—
Sam Weiss: We did not do it.
Dovid Bashevkin: Yeah, I was like, if you two graded them, then—
Sam Weiss: There would also be—we would have spent months grading them, you know, if we even if we could have.
Ben Niewood: We can talk a little bit about the pipeline.
I think that’s important, like how it was set up. And this is an important point that a lot of people asked about on X when we posted it, because we did have AI grade the AI. So that’s, you know, how does that work? The important point about the way the test is set up is that the models took the tests kind of closed book. They were walled off, they were not able to access the internet and look up sources, they were only told the questions, and then they had to answer it from their own memory and reasoning.
So it’s almost like you or I studied for a test, we walked into a room, the door was closed, the test is in front of you, you have to answer the questions, you can’t ask anybody.
Dovid Bashevkin: When we talk about memory of an LLM, I mean, I assume that that machines have perfect recall. Do we find that they forget things, that they did see things? Like what have they learned? What I don’t know what yeshiva these LLMs go to. So what information are they even drawing upon that you could talk about like walling them off from the active internet, fine, but like if you’ve been through all of Sefaria, which is all of Shulchan Aruch, all like all of Talmud, all of Tanach, meaning explain to me a little bit more clearly what it means to be walled off, and what would they be left with, and what does it mean to have a memory when we’re talking about a machine?
Sam Weiss: I think so if we go back to the discussion about how they’re trained, basically you can imagine that they’re feeding in the entire internet, plus physical books, too, you know, they’re scanning books, so—
Dovid Bashevkin: Yeah.
Sam Weiss: We can assume that they’ve all of Sefaria, right? And other, other sources as well that are similar. Chabad.org, Aish.com. For sure, for sure, yeah. So then the question is, like, if it’s a computer, why can’t it just, like, you know, why can’t it just regurgitate this? It goes back to that question of, like, that this is like of some form of compression, right? That we’re learning statistical patterns about this data, we’re not memorizing the data itself.
If they were just memorizing the data itself, the model itself would be as large as the entire internet, and it wouldn’t be useful, right? It would be just too large.
Dovid Bashevkin: If it was constantly walking around with the entire internet that any basic question would take five hours.
Sam Weiss: It’s not just that they’re learning this information, they’re also, as they do this, they’re learning like statistical relationships between things in in pretty complicated and like unexpected ways, it’s not just that we’re feeding in information and they remember it, so it is more than that, too. But then, you know, they, when we say recall, right, it’s it’s somewhat human-like, right? For example, we see instances in these tests where they will get the answer to the question right, and they will cite a source, and the content of the source they’re citing is correct, it’s they’re citing some Gemara, and maybe the content of the Gemara is right, but they get the daf wrong, right, which is like something that like is very human, right? That’s the type of like recall that we’re we’re dealing with, right?
Dovid Bashevkin: So like they might like get a mix up of masechta, or the exact citation they don’t have at their fingertips necessarily.
Sam Weiss: Right, because they learned some relationship between these words and those words and this masechta and this concept, and and it doesn’t always like, you know, it wasn’t always digested, you know, in the right way.
Dovid Bashevkin: That’s very interesting.
Ben Niewood: Okay. It is very human, it’s funny.
Dovid Bashevkin: Go back one more time and explain the AI grading the AI bechinos.
Ben Niewood: Yes.
When the models took the test, they were walled off. They were in a room, the door was closed, they couldn’t access anything except their own, you know, compressed memory.
Dovid Bashevkin: Is there a way to set that up that AI people do? Do regular people have access to that?
Sam Weiss: Yes, but you wouldn’t want to, right? In most cases, right, because it’s mostly useless. You would like it when it doesn’t know something to go out and read Sefaria, or, you know, go to whatever cooking website and read something, so.
Dovid Bashevkin: Correct, but you can wall off your AI.
Ben Niewood: You can tell it not to call not to use any tools, not to look anywhere else, and keep keep in its own its own process on the computer without talking to anything else.
So then, we had all of the top-level frontier models, so Fable 5.1 from Anthropic, OpenAI’s Astra 6, and Gemini’s Argon 4, three of the biggest, you know, top top-tier models right now. We let them out of the sandbox. We told them to look at the internet and look at Sefaria and look at everything else. And then we said, make us a rubric.
Make us something to grade against.
Sam Weiss: And the questions, they had the information.
Ben Niewood: So they could go look up the information. So now it’s not just recall and reasoning, now they’re using the internet to actually check their sources and confirm the correct answers.
And that’s how we made the rubric to grade against.
Sam Weiss: And and they they collaborated on this rubric essentially.
Ben Niewood: So Fable wrote one, and then Astra 6 said, “No, you have to change this,” and then Gemini said, “No, you have to change this.” And then they all, I mean, they didn’t talk to each other directly. We we told them, “Here’s the answers.
Can you go look at it and confirm?” And then we kind of came up with an average rubric based on that that was with all of them collaborating.
Dovid Bashevkin: Did they know whose test they were grading?
Ben Niewood: No, they didn’t know which models were being graded.
Sam Weiss: That’s a good question. Yeah.
Dovid Bashevkin: Is there any sense, you know, I remember when you were in class, you know, sometimes like you feel like the teacher like—
Ben Niewood: They—
Dovid Bashevkin: gave a higher mark cuz it’s the goody-two-shoes, and you’re stricter on me, or, you know, sometimes you’re always in class, you know, like where the child of the teacher’s in the class, and there’s always biases. Would you expect that type of grading bias?
Sam Weiss: You might, and and we were concerned about that, right, that so the the main judge was Fable, right, one of the Claude models. So we were worried about like maybe just because it writes in a certain way, it is going to think that other text in that way, you know, should score higher.
So what we did is we took a random sample of the of the exams, and we had them also graded by Astra, one of the one of the GPT models, right? And we do see that the scores align pretty well, right? They’re they’re it’s just kind of the difference is noise, right, which gives us some confidence that like it’s not biased towards its own like model family. We did this in another case as well, where we added our first pass did not include Grok, and then we we added it after, and—
Ben Niewood: It was terrible.
Sam Weiss: Unexpectedly, it was it was it did very poorly.
Ben Niewood: Grok is a Grok is a frontier model.
4.7 is supposed to be good at a lot of things. It got like a 22 or something or a 23.
Sam Weiss: It was not good at this. And we thought maybe there’s something about the way it’s being judged, that there’s some problem with the experiment here, and so we decided to regrade it using Grok as the grader, and it got the same score.
Ben Niewood: We also revised the rubric. So it looked at the rubric. It said, “Here’s what I would change about the rubric.” Made a bunch of bad decisions, but we said, “Sure, we’ll use your rubric.” Grok used its own rubric to grade itself, and it did worse.
Dovid Bashevkin: My dear friend Josh Groonbaum, I believe, texted me.
He imagines, and and I I participate, each of the models being like a different type of like yeshiva or student. And Grok to me is like your kind of Beis Sh- One of the bros, probably Mevaseret, like definitely like goes into town to watch the Super Bowl instead of night seder, like he’s like Grok knows how to chill, you know, like. So, before we get more to the results, I just want to come back to that question of how long it took different AIs to take the exam and why did it take different AIs different amounts of time for the exam?
Sam Weiss: First, before we we should probably talk about reasoning, which is something that the models do, which is how we were measuring like essentially how hard they’re thinking. So, this is a a newer thing in the past couple of years that the models do where they they do something called reasoning, where essentially rather than just running the question through the model and giving you the answer, the model essentially has like a scratch pad off to the side, and it can produce an answer or or some output, put it in the scratch pad, and then run the whole thing again.
It’s essentially like somebody ruminating on a problem, and and and you can change the model settings to have this.
Dovid Bashevkin: Which when we took tests we would do that also, sometimes you would have like a separate notepad to check out the problem, and then when you put in the blue book—
Sam Weiss: You got to get it right, yeah.
Dovid Bashevkin: Then it’s official.
Sam Weiss: Exactly.
Yeah. So, you can actually change the settings on the models to allow them to think longer or to think quicker, and you get faster answers if they don’t think as much, but you might get better answers. So, there that it’s a it’s a setting you can you can play with, right? And we had them all pinned to like the same sort of like medium like thinking level, and but then what you can do is you can measure basically how long they thought about it in terms of like tokens essentially that they that they spend on this. And so we and we do see a pretty broad variation in this.
You have to be a little bit careful because the way that the models measure this is only consistent within its own like measurement scheme, right? The the way that Claude is measuring how hard it’s thinking is not the same necessarily as Grok. What you can do is look at like the ratio between how long it’s thinking and how long the answer is.
Dovid Bashevkin: Interesting.
Sam Weiss: That you could you could compare, right? And do you do you remember, Ben, like—
Ben Niewood: some of the I think Grok reasoned for a very long time and then answered super quickly, so it’s almost like it thought really hard about it, but couldn’t figure it out, yeah.
Dovid Bashevkin: Got to love the bro We love the bros.
Ben Niewood: Exactly, yeah. It just like it just scribbled down whatever it could at the end, you know, kind of ran out of steam almost. The other ones were I think more similar, but there were I think we did actually post about it afterwards, like what the differences were between the the reasoning time and the the answer time, but one of the most notable differences for sure.
The other ones were all a little bit more similar. But also the smaller models that we tested, so I mentioned that like 5.6 Soul did a lot better than 6 Soul, both models from OpenAI, 6 Soul is the newer version, 6 Soul spent a lot less time answering than 5.6. So, the the models that spent more time answering often just included more information, which did better on the test, because that’s what the test was looking for.
Dovid Bashevkin: How many of them passed with flying colors? How many of them failed?
Ben Niewood: Most of the newest models, except for Grok, passed with with with flying colors.
Fable 5.1 got almost an 80, which were I mean, again, a part—
Sam Weiss: Passing score I think is 65, although we’re not using human graders.
Ben Niewood: Right, so we’d have to have we’d have to have actual learned, you know, rabbeim look at these and and tell us what their score would be. We don’t We haven’t done that yet. We’d like to.
But Fable got almost an 80, which is, you know, I think an like truly a a gadol score, like an excellent excellent score.
Dovid Bashevkin: Is that—
Ben Niewood: That’s my understanding that if on if a human grades, you know, an Israeli Rabbanut semichah test and you get an 80, that’s like really good. Indeed really good. Yeah, it got almost an 80.
I think it was a 79 point something. Gemini Argon within the confidence bounds got pretty close to there. Astra 6 also got pretty close. 5.6 Soul got fairly close.
So, a lot of the newer models did very, very, very well. You know, it’s interesting that the older models didn’t do as well. Again, we’ve all been asking halachic questions to these models for three years. Three years ago, the answers probably were not that good, and now they’re probably a lot better.
Dovid Bashevkin: So, listeners, go back and just—
Ben Niewood: Reevaluate your discussion.
Dovid Bashevkin: —if you’ve been relying on a leniency from you asked ChatGPT three years ago.
Ben Niewood: Exactly.
Sam Weiss: Even today the model types matter a lot, right? So, Haiku, which is like small model from Anthropic, and the point of Haiku is to give really fast answers, right? It’s an efficient model, it’s cheap, and but it doesn’t do well on a test like this, right? So, if you’re using Haiku to ask this kind of question—
Dovid Bashevkin: And those failed?
I mean, there were some that—
Sam Weiss: Yeah, they were below like a human passing score.
I think we have to be careful, I guess, comparing these to human scores ’cause they’re they’re not, right? I think that’s one of the things though that makes the AI grader acceptable is that we’re not trying to compare this necessarily to a human in that way. We can compare it to relative to itself, right? So that like if we know that Fable got 79 with this AI grader, and, you know, the next Fable 6 comes out and it gets an 82, we can compare those two scores, right? So, that that’s where it becomes useful. Comparing it to a human and saying, “Would it pass this?” that’s where we would need human graders to actually grade the test as they would a human.
Dovid Bashevkin: So, let’s talk a little bit about the areas where the AI struggled the most and where the areas where it found most easily, and then we’ll get a couple of like actual questions and maybe we could hear some of the AI answers from this.
But tell me, generally speaking, where were the areas that it struggled the most?
Ben Niewood: I have the kind of semicha papers here. We posted this on X as well. A lot of the models had a lot of trouble in chuppah vekiddushin and niddah, which is interesting. Astra, which is one of the frontier models, didn’t do that well in those.
Fable still did pretty well. All of the models, it looks like, did better in tefillah uvrachos, mikvaos, and mitzvos hataluyos ba’aretz. So there is a skew in that direction.
Dovid Bashevkin: Let me just translate some of those for our listeners.
Ben Niewood: Yeah, please do.
Dovid Bashevkin: So the areas that it struggled in the most, which is actually not a surprise to me at all, were the areas of marriage and the laws of governing marital intimacy, specifically niddah, which is about menstruation and the separation of husband and wife during menstruation. The areas that it did the best in were the laws of prayer, the laws of mikvah, which is not so surprising to me because there’s actually mikvaos has a very almost like mathematical construction, like it, and finally the laws of mitzvos hataluyos ba’aretz, and those are laws of tithing food, terumah, ma’aser, shemittah, yeah. What doesn’t surprise me, especially with niddah, is probably the area that there’s a great deal of human components in this, meaning every stringency in niddah can actually lead to a dangerous leniency, because anytime you say, “Oh, just be strict, be strict,” you’re actually telling a husband and wife not to be together, which is that’s not good, we don’t want that.
So the one that needs to factor in the most, I think, human components and the human factor of halacha—halacha is deeply, deeply human, we’ll get to that in a moment—it does not surprise me that niddah they struggle the most, and even in chuppah vekiddushin, where there are so many personal factors that go on in there, does not surprise me entirely. Right now we’ve spent a few minutes trying to find the actual questions and the actual—
Ben Niewood: We hid them from ourselves!
Dovid Bashevkin: —and the actual answers. Why did you, like, literally, like, hide these questions?
Ben Niewood: So we mentioned before that we wanted to find questions the models had not been trained on before. We wanted to be able to post the results of the study publicly online so other people could see them, and we used a website called GitHub, which is very common in the software engineering world to post code or or results from experiments like this.
Dovid Bashevkin: Sure.
Ben Niewood: But we didn’t want to post the actual questions and answers, because once they’re public online, the models can see them, and anybody can see them. And so if we had a model that was trying to answer the questions again, it might just find the answer posted on our GitHub and have the answer available to it, which would skew even the grader. So we didn’t want that to happen.
Dovid Bashevkin: I’m looking now at the actual questions that were asked in the laws of niddah, that’s what we’re looking at now. These questions are extraordinarily sophisticated. I have taken semicha exams. These are definitely sophisticated questions.
I don’t want to get into all of the details. There happen to be niddah questions, so they’re like an intimacy to them because talking about, you know, the female anatomy and what they’re going through. But I’m looking at their answers, and their Hebrew is written beautifully, and the way that they kind of reference—I’m looking over here, le-daas ha-Rema, shom, siman yud-aleph—they’re citing everything really correctly and and impressively. Now, let’s talk for a moment what you think these results tell us about halachic discourse.
Tell me a little bit, each of you, about your own Jewish background and how would you describe, if you’re comfortable, your relationship, your own relationship to halacha in your own life?
Ben Niewood: I grew up Modern Orthodox. I went to Maimonides growing up in Boston. I went to a year of yeshiva in Orayta in the Old City in in Yerushalayim—
Dovid Bashevkin: Shout out to your rebbe, Rabbi David Silverstein, a dear friend of mine.
Ben Niewood: Yeah.
I describe myself as very, very Modern Orthodox. I think I love the community part of Judaism, the intellectual part of Judaism. I grew up going to Shaarei Tefillah in Newton, Massachusetts, which is—
Dovid Bashevkin: Beautiful.
Ben Niewood: —probably four or five of the top Jewish scholars in the country go there and would give speeches from the pulpit, a lot of really smart professors in Jewish history from Brandeis and Harvard and BC and all over the place.
And so we would have our Rabbi Samuels, who did, you know, married my wife and I is very close with, and also an extremely learned rabbi. His speeches are very intellectual. So I grew up in kind of like halachic discourse. We’d also have these professors and other people speak from the pulpit.
Anyone, you know, we had times every month when anyone could get up and give a speech or we’d go through a cycle of people speaking, and so it was a very learned community.
Dovid Bashevkin: And you grew up in a like a halachically observant home?
Ben Niewood: Yeah, yeah, absolutely. Yeah, grew up kosher, Shabbos, you know, full, full Modern Orthodox.
Dovid Bashevkin: And what about you, Sam?
Sam Weiss: Yeah, so I grew up in Sacramento, California.
I went to—
Dovid Bashevkin: West Coast.
Sam Weiss: West Coast. I went to public school for high school, and then I went to Eretz HaTzvi for the year after. The community I grew up in is, you know, especially by New York standards— standards, very small, but I think that made it like a very, very hands-on community where because of that it was required that everyone be very participatory, right? If they had, you know, people leining every week and needed enough people at the minyan, so it definitely you couldn’t just like, you know, sit back, right, in that way.
My relationship with halacha today is like still, you know, very Modern Orthodox, so…
Dovid Bashevkin: Did you grow up in a halachic home?
Sam Weiss: Mhm, yeah.
Dovid Bashevkin: Like Shabbos, kosher?
Sam Weiss: Yeah, absolutely.
Dovid Bashevkin: You know, it’s really interesting cuz in our parents’ generation, or maybe your grandparents’ generation depending where you place in the family, but people who grew up in the ’50s and ’60s, which is when my parents were kind of coming of age, there was a sense that halacha and the Orthodox community were kind of like a relic of like European Judaism.
And like post-Holocaust, people thought the future certainly did not think would hold the Orthodox community is not something that people thought was going to exist in 2026. And what I find so fascinating now in this moment is that a lot of the most cutting-edge researchers in AI are fascinated with the Talmud, with halachic authority, with decision-making in this area. And like what we thought a hundred years ago was like Orthodox community, is it going to be like America, there’s freedom of religion, you can’t be forced, like this is going to disappear. And there were serious sociologists who, without any animus even, were just like, look, I just don’t think this community is going to survive.
And we’re at such a different moment now. Why do you think so many AI researchers are so fascinated with the Talmud and Talmudic life, like halachic life?
Sam Weiss: I think part of it, there’s definitely different layers to it. I think on, on somewhat of like a practical layer for what they’re actually doing with training these models, I think especially when it comes to something like Gemara, it’s, it’s a very unique data source, right? These labs have kind of exhausted all of the available, obvious information out there, and especially if you’re interested in learning about how reasoning works and how thinking works and how humans put together arguments, like what could be better, right, than, than that kind of data source?
Ben Niewood: So I think that’s—
Dovid Bashevkin: It’s very different from Shakespeare. It’s a totally different thought pattern.
Sam Weiss: For sure, yeah. And you have it over thousands of years, and you can look at modern interpretations and ancient interpretations, and it’s just there’s not, like, very few things if anything that’s like that.
Dovid Bashevkin: It really is. Like, there’s something in the last 2,000 years of all of the things that have been produced and all of the incredible books and works of art, there is something that AI people are like, the Talmud, like, that’s cool.
And like, there’s something that, like, I find it moving, like, look at us. Like, you know what I’m saying? Who would’ve who would’ve thought, you know, that like this is the area like they’re I think they might be onto something. Any other speculation?
Ben Niewood: Yeah, I think, I mean this is pure theory, but I’m I’m not in touch with any of the people from the frontier labs who are studying Talmud personally, but I do think that there is some sense that AI is growing up almost like a child since we’re training it and teaching it. And I think, especially these days, I mean Judaism has on some level tapped into this real truth about the strength of community.
I mean, Shabbos is just an when you talk to anyone who isn’t Jewish about the idea of Shabbos, you turn your phone off, you turn Instagram off, you turn X off, you know, whatever, no one’s on Facebook.
Dovid Bashevkin: We’re really doing this now.
Ben Niewood: Yeah, we really do it.
Dovid Bashevkin: Yeah.
And there’s not another community, I say this openly, like I think it should be studied by scientists and sociologists, there is not another community in the world that consistently is involved in technology six days a week so that like I’m not talking about Amish, six days a week in the world, and genuinely shuts off, and it’s not just the rabbis. Meaning, I’m not asking what somebody’s doing in the bathroom, I’m not asking if their cholent is set up perfectly exactly right, but communally we are off of technology one day a week.
Ben Niewood: 100%.
Dovid Bashevkin: And shuts it off.
Ben Niewood: Yeah, yeah. The idea, yeah. 100%. It’s really wild.
And I think it’s almost like a universal truth. It’s something that it’s it it’s unbelievable that the world of Judaism 2,000 years ago would apply that way today, and how important it is. Something that I love about Judaism is having that community. I think that you couldn’t leave it.
It’s it’s important for our children. You talk to people who read Professor Jonathan Haidt and his books. He I mean, he’s obsessed with this idea because we’ve found this universal truth about how important community is and how important Shabbos is. And I think, to some degree, you know, we we’ve trained we’ve raised our our communities and our children for years on the Talmud and the discourse and those ideas.
And it A, it trains really smart people. Anyone who reads Talmud, you know, can you people say the best lawyers are Talmudists because they really get the the discourse. But also it trains good people. On the whole, the Jewish community has, you know, good values, our morals are aligned, and I think a lot of AI researchers do care a lot about what Sam said earlier, which is alignment.
We need to steer the models to the right place, otherwise who knows what could happen. They could start growing recursively and just be evil or who knows what. And so Talmud, you know, may be one way that we can we can figure that out.
Dovid Bashevkin: The authority of the Talmud is so organic within the Jewish people, meaning there was never a moment where like we banged on the table, like, “Okay, now everybody has to listen to the Talmud now.” It evolved organically.
There is no single author. It emerged, you know, during a period where we really don’t have any other major works emerging from the Jewish community. But the amazingness of the Talmud is that it’s continually relevant, cuz its authority always remains in the hands of the lived Jewish community. And so—
Ben Niewood: Correct.
Dovid Bashevkin: And every community, so long as they remain tethered to the Talmud, which not every community has done, but halachic communities, and my litmus test is can the community sustain public Shabbos observance? Communities that are doing that are still living with the Talmud. Right now, if you go around and it’s 2026, and your, you know, four-year-old child knows when it’s Shabbos, and he knows that he’s not allowed to do certain, like it’s working, and it’s this perfect intersect between a lived community and like a a dataset, you know, not that the Talmud is a dataset, but it’s a dataset that enhances human life rather than than restricts it. Even though it it does restrict it, and there are things that are no longer relevant because of the Talmud, much like AI, there are going to be things that are no longer relevant, but it really empowers humanity, and the way that its continued authority generation remains in the hands of humanity, I think is really, really, really, really powerful. And when you look at this, and I think that’s the one thing that people may fret about or maybe worried about, when you see your results, and surely, you know, they’ll improve over the years, do you think that this makes rabbis, God forbid, you know, I’m right behind me, I got one right here, am I, do I need to throw this out? This is my rabbinic ordination.
What do you think is the future of the rabbinate from from where you’re sitting?
Sam Weiss: I mean, I don’t think that that this threatens like rabbis, right? Like, cuz I think that especially even if you’re looking for like a psak on something, like you’re looking for an answer of like, “Can I use this in my kitchen? I did this and whatever,” asking that question, you’re not just looking to look up an answer, right? There’s a relationship between the person asking the question and the person answering the question that is unique to that question, right, and that exists in no other question with no other person asking that question.
Dovid Bashevkin: Because the halacha is deliberately addressing you within your own life.
Sam Weiss: Right.
Exactly.
Dovid Bashevkin: You know, when somebody asks a halachic question, but doesn’t say, you know, “My parents bought me this food, can I eat it?” Well, were your parents, were they given a yeshiva education? What type, what’s the prohibition that we’re dealing with? How much money did they spend on it? Are they struggling financially? Like is that like there’s so many human factors that come up. And really to me, and this is kind of the point I was trying to make, the right answer is always in dialogue with the community.
Sam Weiss: Exactly.
Dovid Bashevkin: Meaning nobody would go and say like like publicly do something that they’ve never seen in the community and say, “You’re allowed to do that?” and they be, “Oh no, I asked ChatGPT.” Like Right. No, we’re mediated by a community. There are a lot of things that we don’t do communally that halachically are not so problematic, but it’s just it’s not done. Like an example of this, and again, I’m not judging anybody who does or does not, in a lot of Orthodox communities you will see that even when it’s raining, nobody uses an umbrella.
Now, I’m not saying that there’s no halachic reason not to use an umbrella on Shabbos, it’s one of those things, if you examine closely, you could find a way to be lenient even on a rainy day to shul on Shabbos with an umbrella, though if you ask the rabbi, and say, “Look, I live two miles away from the synagogue, and this is what I’m dealing with,” you might get a different answer. But so there’s something very communally mediated about it.
Sam Weiss: Right. Like Claude is not walking around this world living in our community, understanding people, right? It might have more information and even maybe better reasoning than like, you know, any given human.
Maybe that maybe they’re not there yet in every domain, but maybe they’ll get there, right? That still doesn’t mean that you’re getting what you’re looking for by asking a halachic question. I think there’s a place for it, right? I do this. I’ll ask I’ll ask questions to it, of course. I think everyone does.
But you have to understand what’s going on when you’re asking that question. And I think what this demonstrates as well is like another layer of like, not only what’s going on in with an LLM in general, but these things differ across even these systems, and understanding that as well is like another layer of importance.
Ben Niewood: Yeah. I mean, I think it does go back to what I was talking about before, the this universal truth of Judaism about how important the community is.
We’re always going to have rebbeim, and we need relationships with our rebbeim, and and that’s that’s the important part. The process is as important as the question in lived Judaism. That’s not what you get by talking to an LLM. You get an answer.
And that’s good for some things. It’s great for math, it’s great for coding, it’s great for, you know—
Sam Weiss: Taking a rabbinic test, maybe. Right.
Ben Niewood: Exactly.
It might be really, right, it might be really good when you’re taking a rabbinic test, you want an answer. That’s actually the way to pass the test. But the important part is the relationship and the process.
Dovid Bashevkin: Exactly.
And the real arena of Judaism is Jewish life itself.
Sam Weiss: Yes.
Dovid Bashevkin: Meaning Claude can’t keep Shabbos for you.
Sam Weiss: Right.
Dovid Bashevkin: Claude can’t keep Yontif for you. Will a day come where you could have Claude look and make sure that your esrog is kosher?
Ben Niewood: Right, probably. Yeah, probably.
Sam Weiss: Yeah, probably could today, yeah, to some extent.
You could probably already do that.
Dovid Bashevkin: But the lived experience is what Jewish life is all about. So for me, I actually think that like AI is like a clarifier where it almost allows you to focus on the experience itself rather than like all the information that you have to carry around. And even our relationship to the information, it’s not about knowing it, it’s about your involvement, engaging with it.
Like Torah study is such a different pursuit than other forms of knowledge because, everyone would say this, the very term of Torah lishmah—
—is very almost anti the way the rest of the world looks at information and content, where it’s like about having the content for most of the world, and in the Jewish world, Torah learning lishmah, which means for its own sake, it’s the joy of learning. Yeah. The same reason like You’re not looking for a result. Right.
Sam Weiss: Exactly. I think it can actually enhance that too though, right? Like if you if you ask a question of an LLM, a halachic question, you might end up learning about sources that you didn’t know about, right, or like, and it leads you down paths you wouldn’t have gone if you had just Googled something, right? Which is maybe that’s like another comparison you can make, it’s just like looking up something online, right? It’s probably a more rich experience than that in terms of what you’re going to learn and and and be exposed to. And it’s also conversational, right? You could say like, “Well, I want to know a little bit more about that angle that you mentioned. Let’s go down that path,” right? That’s obviously an improvement over just being able to look something up online.
Dovid Bashevkin: Where can people find and follow your work?
Ben Niewood: We’re We have our our X account, HalachaBench. We have a website or you can email us at hello@halachabench.ai. And yeah, we’re open to chatting about collaborating or—
Dovid Bashevkin: We’re writing some tests.
Ben Niewood: Yeah, we have some more ideas for things we want to do and we’ll be posting more.
Dovid Bashevkin: Then I’m going to get you a better grader for your—
Ben Niewood: Yes, we need some graders.
Dovid Bashevkin: —for your—
Ben Niewood: We’d like to develop our own repository of questions that are private also to help make this more of a, you know, more of a legitimate benchmark, and, you know, we we all have our day jobs, but we want to keep working on this and figuring out where it’s going from here.
Dovid Bashevkin: Absolutely fantastic, and again, thank you both to Sam and Ben, and thank you to Bret. We love you.
Ben Niewood: Thank you for making And shout out Ilan Weitzman and Jesse Bernstein, who are the other two people in our in our group who are working on this with.
Dovid Bashevkin: And they’re all all from the same neighborhood.
Ben Niewood: We all live in Scarsdale, yeah.
Dovid Bashevkin: That’s really, really, really, really sweet.
And your incredible rabbi, Rabbi Morgenstern, who I love to pieces. I always wrap up my interviews with more rapid-fire questions.
Ben Niewood: Okay.
Dovid Bashevkin: I always love great book recommendations.
Is there a book that you would recommend that’s either a great introduction to understanding what AI is all about, not not for an expert, but just a a basic understanding of this revolution that’s taking place, or a great introduction to halacha that you’ve enjoyed? You could pick either one.
Sam Weiss: Have to think about that.
Ben Niewood: Yeah, I’d have to think about it. I haven’t read any AI books.
I feel like I read about it more online.
Dovid Bashevkin: You only read technical stuff. You’re not the You’re you’re you’re not you’re not the right guys for this. But have you read any, do you have favorite Jewish books, either of you?
Ben Niewood: Yes.
Off the top of my head, I haven’t read any Jewish books recently. It’s been a minute. It’s It’s been a minute.
Dovid Bashevkin: Yeah.
Sam Weiss: I mean this isn’t this isn’t necessarily halacha.
I I just read Letter in the Scroll by Jonathan Sacks, which I really enjoyed.
Dovid Bashevkin: We love it.
I think it’s up on the shelf somewhere.
Sam Weiss: Right.
Which was really about like what we were talking about earlier of the nature of like Jewish community and what it means to be like one member of this larger community that’s trying to accomplish something over time as well as just like right now.
Dovid Bashevkin: I really mean this. The ultimate data set of Judaism is not the Talmud and it’s not Tanach, it’s the Jewish people themselves.
Sam Weiss: It’s a very embodied—
Dovid Bashevkin: Correct.
It’s the embodiment of the tradition. It’s not traditions up here, the ultimate tradition is is the lived experience of the Jewish people. My next question, I’m always curious if somebody gave you a great deal of money and allowed you to take a sabbatical, go back to school wherever you need to study any other subject or area of your choice, what do you think you’d go back and study?
Ben Niewood: Such a hard question because I really love what I studied. I do.
I I really love mechanical engineering. I think these days if I were to go back in school, I would probably want to study more philosophy. I did very technical classes in college. I didn’t do a lot of humanities and didn’t have a lot of time for like thought process—
Dovid Bashevkin: Sure.
Ben Niewood: —kind of classes, so I think that’s important these days to think about the way the world works and the way that people think about things, so that’s probably what I would take time to do.
Sam Weiss: Yeah, if I were to go back and and do like a, you know, a PhD, I think it would probably be something in biology, which is again like different than than what I studied in school. You know, there’s something that the the AI people like to say that like these these models, unlike computer programs, they’re they’re grown, not not programmed, right? So there’s some some overlap there with like the the way that computer science is starting to look like biology in some ways,.
Dovid Bashevkin: Fascinating.
Sam Weiss: And vice versa, where you now have like techniques in biology where you can can like program cells to do things, right? So these things are like converging in an interesting way.
Dovid Bashevkin: Wow, really fascinating. My final question, I’m always curious about people’s sleep schedules. What time do you go to sleep at night, and what time do you wake up in the morning?
Ben Niewood: Way too late and way too early. I’m usually not in bed until midnight, sometimes later.
The kids wake me up at, you know, 6:30 to 7:00 most days. I try to get more. I’ve got my Whoop on and it always tells me how bad a job I’m doing, but so every once in a while I’ll crash early. But yeah, I spend way too much time up and and working or doing things, or yeah.
Sam Weiss: I can’t function past like 10:00 p.m. Ideally, with young children, that’s hard sometimes, but like I would say like 10:00 to 6:30 would be ideal.
Ben Niewood: Halacha benchmarks kept Sam up late a couple nights.
Sam Weiss: Last night I was I was I was waiting for one of these evaluation runs to finish until, you know, midnight.
Dovid Bashevkin: Ben and Sam, thank you so much for joining 18Forty today.
Ben Niewood: Thank you for having us.
Dovid Bashevkin: Ben and Sam, what I love about them is they’re kind of like your regular run-of-the-mill Jews. They live in a regular Scarsdale, a regular kind of Jewish community. They are living extraordinarily Jewish lives.
And we were talking on the steps after the interview, we were talking about what’s going to really preserve Jewish life going forward as AI kind of advances more and more. And what we both agreed on is the centrality of communal Shabbos observance, and how of all of the differences that we have communally in the Jewish world, I think the difference that will be the most essential and fundamental going forward is whether Jewish communities are able to sustain public Shabbos observance. Now, I’m not asking and talking about what we’re doing behind closed doors, in our bathroom, that that’s between you and your God, and maybe your rabbi. But in terms of the model of a Jewish community that is able to sustain Jewish life effectively, I think that the denominational framework, that’s going to be in the past.
The only framework I think going forward that will essentially tell a Jewish community, “Are we capable of sustaining a vision and a model of Jewish life going forward?” is going to be communal Shabbos observance. And I say this with both great optimism, joy, and not an ounce of judgment. I realize not every Jewish community is going to be able to sustain that, but it’s more of an invitation. Think now about what it’s going to be like growing up Jewishly in the world that we have now.
Think about what childhood is going to feel like. If you’re a child right now, you’re two years old, what’s it going to feel like being seven years old? And I believe the vehicle that will preserve our humanity more than anything else is Shabbos observance, and specifically on a communal level. Shabbos is not something that you can really do by yourself, surrounded by those who don’t. It’s been done, it’s very, very difficult, it’s very isolating.
The beauty of Shabbos emerges communally. And really as an invitation, communities should start thinking about this. How can we encourage? Federations should start thinking about this. We should start talking about this much more publicly.
How do we encourage communities to develop an ethic of Shabbos observance? Because I think in the world that we have now, what is going to really preserve, sustain, and really not just sustain, but create the growth and the catalyst for the Jewish future that we all want, a lot is going to come down to Shabbos, that weekly communal unwinding and disentangling from technology. Shabbos in a lot of ways has gotten harder and harder to observe as technology weaves its way more and more and encroaches more and more in our lives. However, I think the idea of Shabbos has become so much more compelling, and we should think about this. Think about hold your your, you know, look at a three-year-old, a toddler.
Look at a first grader and ask, without Shabbos, what is their experience of childhood even going to be like? And I think that of all of the questions that AI raises, it asks a lot of questions of things that are going to change, but one thing that is going to remain and get only stronger, in my opinion, and this is the conversation that we had after the interview, was the lived, communal experience of Shabbos. So thank you so much for listening to 18Forty. This episode, like so many of our episodes, was edited by our incredible team. Thank you, Dinah.
Thank you, Sam. We cannot do this work without you. And speaking of cannot do this work without you, we cannot do this work without our supporters. So please, if you enjoyed this episode or any of our episodes, please subscribe, rate, review, click down, subscribe on YouTube.
All of this helps us build and reach a wider audience to sustain substantive Jewish conversation across borders. This is what we’re trying to achieve, and we want you and invite you to be a part of it. And of course, you can also support us the old-fashioned way and donate at 18forty.org/donate. That’s 18forty.org/donate.
All of your efforts really help us reach new listeners and continue putting out great Jewish content. And of course, you can leave us a voicemail with feedback that we may play on a future episode, and that episode is coming up very soon. And that number is 212-582-1840. Once again, that number is 212-582-1840.
If you’d like to learn more about this topic or some of the other great ones we’ve covered in the past, be sure to check out 18forty.org, 18forty.org, where you can also find videos, articles, recommended readings, and weekly emails. Thank you so much for listening, and stay curious, my friends.
Shulchan Arukh by Rabbi Yosef Karo
“A Mathematical Theory of Communication” by Claude Shannon
Consider Phlebas by Iain M. Banks
HalachaBench results on GitHub
HalachaBench on X
The Anxious Generation by Jonathan Haidt
A Letter in the Scroll by Rabbi Jonathan Sacks
18Forty Podcast:“Will AI Replace Rabbis?”
In this episode of the 18Forty Podcast, we talk to Judah, Naomi, and Aharon Akiva Dardik—an olim family whose son went to military jail for refusing to follow to IDF orders and has since become a ceasefire activist at Columbia University—about sticking together as a family despite their fundamental differences.
In this episode of the 18Forty Podcast, we talk to Rabbi Pini Dunner and Rav Moshe Weinberger about the Yabloner Rebbe and his astounding story of teshuva.
In this episode of the 18Forty Podcast, we talk to Talia Khan—a Jewish MIT graduate student and Israel activist—and her father, an Afghan Muslim immigrant, about their close father-daughter relationship despite their ideological disagreements.
In this episode of the 18Forty Podcast, we talk to Rabbi Eric and Dr. Adina Yoffie about their journey as a family through ideological differences.
In this episode of the 18Forty Podcast, we sit down with Joey Rosenfeld, social worker and kabbalist, to talk about the differences between mysticism and rationalism and the roles they should play in our lives.
The true enemy in Israel’s current war, Einat Wilf says, is what she calls “Palestinianism.”
We talk to Berel Leib Epstein about rational reasons to choose to believe in Judaism and live a Jewish life.
Efraim Palvanov answers questions on misinformation, prayer as self-reflection, and every Jew’s role in bringing Mashiach.
Rabbi Yaakov Klein joins us to discuss the soul of God, Jewish particularism, and personal identity.
In this episode of the 18Forty Podcast, we talk to Aliza and Ephraim Bulow, a married couple whose religious paths diverged over the course of their shared life.
We mark the yahrtzeit of Rav Ezra Dovid Halevi Neuberger with remembrances from Dovid Bashevkin and David Fohrman.
We talk to Michael Eisenberg about the state of the Jewish People in Israel and the diaspora.
In this episode of the 18Forty Podcast, we talk to Noam Weissman, executive vice president of OpenDor Media, about Israel education.
In this episode of the 18Forty Podcast, we sit down with Rabbi Daniel Grama—rabbi of Westside Shul and Valley Torah High School—and his daughter Aliza—a former Bais Yaakov student and recovered addict—about navigating their religious and other differences.
In this episode of the 18Forty Podcast, we sit down for a special podcast with our host, David Bashevkin, to discuss the podcast’s namesake, the year 1840.
In this episode of the 18Forty Podcast, we sit down with Rav Moshe Weinberger, rabbi and educator, to discuss the role of mysticism in modern-day Judaism.
We talk to Rabbi YY Jacobson about how each of us can experience redemptive consciousness by feeling Hashem’s infinite love and presence in this moment.
In this episode of the 18Forty Podcast, we sit down with Dr. Ora Wiskind, professor and author, to discuss her life journey, both as a Jew and as an academic, and her attitude towards mysticism.
In this episode of the 18Forty Podcast, David Bashevkin opens up about his mental health journey.
In this episode, we talk to a father and daughter who were estranged and then reunited.
We speak with Rabbi Shalom Rosner, who delivered a widely shared eulogy for Eliyahu Moshe Zimbalist—and with Eli Mo’s mother, Sara Zimbalist.
Rabbi Ari Ze’ev Schwartz joins us to discuss faith, the purpose of Torah study, and the dangers of mystical study.
In this episode of the 18Forty Podcast, we talk to Liel Leibovitz and Lisa Ann Sandell about what happens when one partner wants to increase their religious practice.
In this episode of the 18Forty Podcast, we talk to author and historian Dr. Marc B. Shapiro about the blurry borders that distinguish Reform, Conservative, yeshivish, Modern Orthodox, and the many other Jewish sects from one another.
After hundreds of days in Gaza, I lost my sense of self. Reading Jung and Manitou helped me understand how embracing our…
Perhaps the most fundamental question any religious believer can ask is: “Does God exist?” It’s time we find good answers.
The Jewish tradition of sanctifying the new moon reminds us that in our loneliest moments, we are never truly alone.
Joy and meaning can be found not only despite the brokenness, but even because of the brokenness.
In the age of social media, every experience needs a narrative arc. Judaism offers a different way to make meaning.
My family made aliyah over a decade ago. Navigating our lives as American immigrants in Israel is a day-to-day balance.
To talk about the history of Jewish mysticism is in many ways to talk about the history of the mystical community.
Israeli minister Itamar Ben-Gvir wears the mantle of Kahane in Israel. Many Orthodox Jews welcomed him with open arms.
I consider the Rebbe to be my personal teacher, and I find this teaching particularly relevant for us now.
Rav Froman was a complicated character in Israel and in his own home city of Tekoa, as people from both the right…
A 94-year-old Holocaust survivor, a lone soldier, and more. Here are seven olim sharing their stories of aliyah.
If You’re Reading These Words is a book in which all the heroes have died, yet it overflows with life.
A personal return to the poems that taught me how to imagine America
Children don’t come with guarantees. Washing machines come with guarantees.
From verses in Parshat Bo to desert caves, tefillin emerge as one of Judaism’s earliest embodied practices.
Christianity’s focus on the afterlife historically discouraged Jews from discussing it—but Jews very much believe in it.
In Klara and the Sun, Kazuo Ishiguro asks us to reconsider what, if anything, separates humans from machines.
Parshat Toldot makes it clear that the Torah’s family narratives aren’t idealized models but honest invitations to examine our own stories.
A Hezbollah missile killed Rabbi Dr. Tamir Granot’s son, Amitai Tzvi, on Oct. 15. Here, he pleas for Haredim to enlist into…
As podcasts become more popular, the interview has changed from educational to artistic and demands its own appreciation.
Children cannot truly avoid the consequences of estrangement. Their parents’ shadow will always follow.
Religious Zionism is a spectrum—and I would place my Hardal community on the right of that spectrum.
Kosher phones make calls and send texts. No Instagram, no TikTok, and no distractions. Maybe it’s time the world embraces them.
Today, we’re sharing an episode of our program Talmud Daily featuring Michelle Margolis, the Norman E. Alexander Librarian for Jewish Studies at…
In this episode of the 18Forty Podcast, we speak with Shais Taub, the rabbi behind the organization SoulWords, about shame, selfhood, and…
In this episode, Liel Leibovitz reflects on falling in love with the Talmud precisely because it is so difficult—and on how its…
In this episode of the 18Forty Podcast, we talk to Rabbi YY Jacobson, one of the foremost online teachers of Torah, about…
In this episode of the 18Forty Podcast, we talk to Rav Judah Mischel, executive director of Camp HASC and author of the…
In this episode of the 18Forty Podcast, we speak with engineers and HalachaBench creators Ben Niewood and Sam Weiss about how AI…
In this episode of the 18Forty Podcast, we speak with Diana Fersko, senior rabbi of the Village Temple Reform synagogue, about denominations…
On this 18Forty panel, we speak with Alex Jakubowski of Lightning Studios, Sara Wolkenfeld of Sefaria, and Ari Lamm of BZ Media…
In this episode of the 18Forty Podcast, we talk to Lahav Harkov, senior political correspondent for Jewish Insider, for a basic primer…
In this episode of the 18Forty Podcast, we talk to Rabbi Dr. Ari Lamm, co-founder and president of SoulShop Studios, about doing…
In this episode of the 18Forty Podcast, David Bashevkin answers questions from Diana Fersko, senior rabbi of the Village Temple Reform synagogue,…
Talking about the “Haredi community” is a misnomer, Jonathan Rosenblum says, and simplifies its diversity of thought and perspectives. A Yale-trained lawyer…
We talk to author and theoretical physicist Berel Leib Epstein about rational reasons to choose to believe in Judaism and live a…
In this episode of the 18Forty Podcast, we talk to Ben and Yael de Toledo, founders of Pnima Jerusalem, a soulful community…
A free 40-week journey through the Talmud. No Hebrew or background required. 7,000+ people have already signed up — join us.
