Hamel Husain

“And then finally, the AI responds, "Hey, we have several one-bedroom apartments available, but none specifically listed with a study. Here are a few options."”

“And then it says, "Can you let me know when one with a study is available?"”

“And then it says, "I currently don't have specific information on the availability of a one-bedroom apartment."”

“So you write this note, and then we can go on to the next trace. So this is the next trace. I just pushed a hot key on my keyboard. Let me go back to looking at it.”

“Yeah, so this is a different kind of error.”

“This is more of, "Hey, we're not handling this interaction correctly. This is more of a technical problem," rather than, "Hey, the AI is not doing exactly what we want." So we would write that down to...”

“Yeah, you might not know this is happening, right? And so you would just say, "Okay." You would write a note like, "Oh, conversation flow is janky because of text message."”

“So here's another one, and let's skip the system prompt again. And the user asks, "Hey, I'm looking for a two- to three-bedroom with either one or two baths. Do you provide virtual tours?"”

“So it is hallucinating something that doesn't exist. Then you kind of have to bring your context as an engineer, or even product content, and say, "Hey, this is kind of weird. We shouldn't be telling ...”

“So you would say, "Okay, offered virtual tour," and you just write the note. So you can see there's a diversity of different kinds of errors that we're seeing, and we're actually learning a lot about ...”

“No problem. And we'll actually show the LLM automation in this example, because we're going to take this example, we're going to go all the way through.”

“Yeah. It's going to be okay. It's not perfection. You're just trying to make progress and get signal quickly so you have an idea of what to work on because it can become infinitely expensive if you're...”