SoftTalker 7 minutes ago
ryanisnan 29 minutes ago
Looking at the questions they asked, they are quite broad. In my experience, part of the value of LLMs comes from its randomness, but you must understand it as a characteristic virtue of these tools, and must adapt accordingly.
Asking questions with tight scopes, clear constraints, and specific outcomes, always yields better results, in my opinion.
seizethecheese 10 minutes ago
(For what it's worth, I asked an agent to compute p-vals based on available information for all the stats in this blog post. More than half are significant but a lot are noise. Not going to post the results because I haven't validated, and the agent made a lot of assumptions. The author should update the post.)
hnlmorg 18 minutes ago
What separates one product from another is going to be constraints specific to the project you’re integrating this into. And none of that is defined in your prompts. So you’re effectively just asking an LLM to roll the dice and pick something at random and then acting surprised means a different answer each time.
Edit: their conclusions at the end of the article miss the most important point. They talk about asking multiple times over multiple days, but that’s just repeating the “garbage in -> garbage out” cycle without addressing the root cause of the problem: not defining your requirements properly.
And if you don’t have any requirements beyond “something cheap” (for example) then who cares what answer you get when any result is equally valid for your use case.
seabre 10 minutes ago
mentalgear 26 minutes ago
> If you're not a leader, this is the part that matters. Your visibility depends on which pages the engine reads that day. The way to make it stable is to be on more of those pages.
Ultimately a true facts knowledge graph independent of mentions is needed.
luisfominaya 30 minutes ago
ppierre 2 minutes ago
22 minutes ago
Comment deletedkylehotchkiss 30 minutes ago
avatar99 an hour ago
84% of answers named a different set of brands the next day. On Gemini it was 95%.
The cause is easy to see in the data. Each answer is a summary of a few web pages, and the engine picks those pages fresh each time. Fewer than half of the pages cited on day one were cited again on day two. When the pages change, the brands change.
One example: asked for the “easiest CRM for a team that has never used one”, Google AI Mode read Zapier’s list and put Less Annoying CRM first. The next day it read a Slack blog post instead, and Less Annoying CRM wasn’t mentioned at all.
Who survives depends on how many pages you’re on. A brand ranked #1 was named again the next day 90% of the time. A brand ranked sixth or lower: 61%. Big brands are on most of the pages, so whichever ones the engine reads, they’re there. Smaller brands are on a few, so they come and go.
josefritzishere 13 minutes ago
Comment deletedfwip 28 minutes ago
If you asked me what my top six favorite shows were, and then again 2 days later, I'm sure I would have one or two series that were new to the list or changed spots.
Edit: If this is something you're actually interested in measuring, there are several statistical tools meant for comparing the similarity of rank-ordered lists. I'd probably reach for a rank-biased overlap measure first.