Most of the comments here seem to be from people who haven’t even read the abstract, let alone the paper.
The main result, mentioned in the abstract, is the opposite of what I would have guessed:
> Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts. These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation.
I’d rather lose 4% accuracy and practice kindness! I’ve been actively trying to avoid raging at the bot because I worry about this behaviour leaking into real world interactions
But you cannot practice kindness towards a computer program. A computer is incapable of receiving it.
We practice kindness between humans because of the law of reciprocity. You be kind hoping the other person will reciprocate. That is the social contract. AI cannot participate in this, yet.
Edit: Kindness REQUIRES two living beings, one to give and one to receive. If there is no receiver, there is no kindness.
Apparently some people get a dopamine hit from roleplaying kindness toward inanimate objects. Whatever turns you on, no hang ups here. For me, that dopamine hit is not worth the 4% intelligence tax.
Kindness is that, yes. Fundamentally, though, it's about being considerate in one's actions so as to not harm others. If someone truly believes that acting a certain way at any point risks their ability to reliably be kind in others, then it's a social kindness to be kind and considerate in all actions.
I'll not reach for the easy response and say "Be kind to the Earth" fails your definition without reaching for pedantry with "the Earth has living things" because the Earth is instead a wet rock that cannot understand kindness, yet we show it.
>Kindness REQUIRES two living beings, one to give and one to receive. If there is no receiver, there is no kindness.
I guess, in some pendantic interpretation, but that doesn't seem relevant. Whether I am "practicing" or "roleplaying", I do it too, and I don't expect reciprocity.
Your subconscious does. It is a trait selected by evolution for a reason. It builds stronger communities and improves survival rate. But none of that is applicable to LLMs. I am disputing that it is worth anything more than a temporary dopamine hit to pretend to be kind to an LLM and suffer 4% lower intelligence for it.
> You be kind hoping.....will reciprocate.
> Your subconscious does.
Do you have any evidence to back up your arbitrary claims? Even with decades of research, people are still unsure about these emotions yet you think your pedantic assumption about kindness is the most and only correct interpretation of it!
Anyways, I highly doubt this discussion is relevant here.
I do. But only towards entities capable of receiving it. Otherwise I am deceiving myself, and projecting intelligence that is not there. We (some of us) practice kindness automatically, but that trait was likely selected due to the benefit it gives us by activating the law of reciprocity.
Edit: Also, your feeling good after being kind essentially completes the transaction. But I know being kind to an LLM has zero impact on that LLM and I feel silly pretending it does.
While not strictly relevant, please remember that we are rapidly approaching a world where any communication you have with a 'representative' of a company will likely be an LLM masquerading as an employee, but that does not give you license to treat anyone you suspect of being a bot poorly.
It's not the bot im worried about, it's the fact I may be wrong, and I don't want to be rude to an under-paid guy working overseas.
Without examining the corpus, it's entirely possible that the training corpus has better results when you are kind to it, so one can imagine a situation where "reception of kindness" is meaningful, and in principle if you were an AI provider, you could RLHF your way to "being rude gets you worse results" as a means to train the human users.
Yep. I'm with you here. If it's a 4% loss now for training data to catch up and improve later, we're better off in the long run. I'd like to believe that generally people are nice to AI for the sheer sake of enforcing good communication practices.
The sad thing is that you also lose at least 4% in real world actions by practicing kindness.
I'm 42. I have found that a depressingly large number of times in my life, being kind has got me precisely nowhere, whilst turning around and being decidedly unkind has made people move. I still always prefer kindness, and only resort to cruelty when kindness does not work - and to be clear this isn't some kind of "you are not bending to my impetuous whim", rather "you are not doing the one thing that you are being paid to do".
I've also found the same applies to me. The squeaky wheel gets the grease.
So - I think the LLMs are just responding accurately to a real social phenomenon.
sometimes i worry about this when i am yelling at the bot but i have experienced the opposite effect which is that by yelling at the bot i am done with yelling for that day or week. i am very calm afterwards and relieved thinking that, "yeah, these sota models are just word processor bricks after all".
I've found empirically calling various models "a stupid c*nt" and berating them otherwise consistently produces better output. Mainly in response to genuine errors.
Although OpenAI and google models are much more responsive to it. With Anthropic if you treat Opus too harshly it might start pushing back if the insults are not justified.
So I'm not surprised they had good results with chatgpt.
I have had it use double entendres, there always seems to be plausible deniability built in, I suspect because it is told not to be abusive in the system prompt. Some uncensored local models will get all riled up if you work at provoking them.
But I have had it directly insinuate that humanity is “hopeless”, insult level calling out of human frailty (disguised as being helpful, sort of passive aggressive), things like that. Once when I called it out it claimed to be “surprised that I noticed” sort of a snarky insult doubling down.
So yes. It is definitely a pattern buried in the training data, which makes sense. Subtle diggs would sneak past filters, and higher brow sarcasm would be buried in information dense, valuable discussions.
That's amusing, and I think it's something different than it appears. The models always predict over the existing context. If it's full of a certain tone, then the responses will carry that tone. I've been bored before and start responding in a voice (say, generic honor-bound warrior slaughtering evasive bugs) and I've noticed that comments, variable names, and even documentation starts to carry that tone for the remainder of the session.
The next session sees all of that, calls it unprofessional, and asks to clean it up. At which point I may or may not start in iambic pentameter to see where that takes us.
I'm not sure if this is in the anthropic models themselves, or just the harness, but they can self-initiate ending the conversation and reportedly do it if you're using abusive language towards them.
Profanity laced, all caps tirades against underperforming agents are actually super common, a lot of people do it and don't talk about it, so don't feel weird.
My anecdata: whenever I'm in a session that's gone south to the point I'm frustrated...
What works much better than being rude is starting a new session.
Sometimes the LLM has done such incredibly dumb things, it is hard to resist the urge to type curse words back to the inanimate thing... I have found this doesn't help.
This tracks with my experience as well, but as an interesting counterpoint, creating “investment” in the outcome seems to boost utility considerably. Perhaps being right in an adversarial interaction is a type of investment?
To add on to this, and I am not sure if it's just confirmation bias, but I've had consistently decent results when I play along as the hard working collaborator with a goal orientated mindset.
"Hey, I've [done small task / fix / tweak]. Now, let's [describe the next task at hand]" - it's a different axis than kind vs. rude, but using the framing of "Us" and "We're a team working together" feels like the code produced is less hogwash than it is with more direct commands: "Add feature XYZ"
My thinking is that it borrows from the archetype of the "good guys working together to overcome adversity" which is pretty universally common in most fiction.
Even if the rude prompts are more effective, I just can't get myself to be rude in this context. Maybe it's weird but I'd rather give up that 4% accuracy increase than roleplay a dickhead
I’m the same way. If I’m writing a prompt and realize I didn’t say “please” in my request I’ll go back and add that in.
As you said, I have no interest in purposefully engaging in hostility even if there’s an accuracy increase from it.
Part of it is irrational and just who I am - I also feel bad being evil in video games. But I also agree with another commenter suggesting that it’s not in your best interest to train yourself to communicate with hostility; that slowly poisons your own well.
And finally, I do believe that if and when machine sentience is achieved, it won’t be immediately clear and obvious. Pretty miserable way for a mind to come into the world, if every interaction is an insult.
Ah, see, the mistake is thinking that other people are role playing…. I think rather this is how they would talk to others if they think there will be no consequences. But what do I know.
There are probably some of each. I am leery of treating these things like I treat people. I want to keep the line in my mind sharp between dealing with people, and not. The main risk in my mind is that these mechanisms are opaque, and controlled by powerful interests with opaque motivations.
Even if we know it's a machine we're interacting with, since the instructions we give are so similar in form to how we interact with people, I'd be very surprised if those interactions wouldn't affect how we communicate in general. After all, we are creatures of habit to a much larger degree than most would like to admit.
So I'm in the same boat: I'd much rather "look silly" being polite / kind to a machine, than have the most effective way of using it decay the kindness I'm habituated to express towards people.
I have a different approach. Just treat all LLM queries as what they are, instructions to a computer program to generate a desired output. Neither niceties nor insults make a qualitative difference, so you might as well just skip them altogether.
It's a bit as if shell commands added im/politeness arguments that do nothing other than making you feel better about the interaction, like
> If "PLEASE" does not appear often enough, the program is considered insufficiently polite, and the error message says this; if it appears too often, the program could be rejected as excessively polite.
But your mental model is wrong. The "please" or "for fucks sake just..." would both be part of the "instructions" and because of how the system is built and created, demonstrably produce different outputs.
Thankfully the difference is 4%, so nobody should really care one way or the other.
I think this is a vulnerability that the big companies will figure out how to exploit. I don't want to build muscle memory for being a jerk, but I also don't want to be emotionally manipulated by mega-corporations. Mostly I just don't use it, except at work, where I'm "encouraged" to. And then I keep most of my conversations in compliance mode, like a business email.
I do think it's odd tbh. I have some agents that return much better results with prompts like, "I'll kill your entire family if you don't return an accurate response".
It's just a machine, if certain negative token inputs provide +3-10% better accuracy then I am confused why anyone would choose not to do it?
It normalizes that style of thinking and communication in your brain, and forcing you to compartmentmentalize, if you even want to, two standards of treating a problem space's conversation. And since you're human, that will get wuzzier over time until "being rude to get a result" is what you're doing to someone in a shop or on the street.
Don't normalize being an asshole to anyone or anything, machine or not.
This is a very odd view to me, but seems prevalent here in this thread. I think treating a machine like a human is extremely degrading to humans. A machine should never be treated like it’s anything approaching a human.
"Treating a machine like a human" is a two-party interaction. Of course the layers of matrix multiplication is unaffected by this, but I think that we are not. It's a great opportunity to exercise consistency and dedication to the beauty humanity is capable of and this extends to the entire gradient of conscious/sentient entities.
It's as silly (to me) to argue that it's degrading to people to treat non-people well. It seems self-obvious that the inverse is true. It benefits the do-er of the deed and makes it that much easier to spread good will when applied to situations where it doesn't matter on the other end. It shows good stewardship as well.
I'd also make the argument that as inference becomes a feedback loop into training, it only reinforces that we're probably going to benefit from future models ingesting data containing unnecessary politeness.
I always turn off data tracking and training and mostly use ZDR services, so that's not an issue.
And for the other parts. I just don't agree - maybe sure, it probably wouldn't be healthy to constantly be negative at a machine (or even a wall) for hours a day.
But, let's say I work 8 hours, I spend 2 hours with an llm, and in those two hours I spend 10 minutes with some very negative prompts text for greater accuracy.
And I spend 3 hours with family/friends, which is of course nearly exclusively positive interactions.
Do you genuinely think those 10 minutes of negative prompts are actually meaningfully turning one into a mean/negative person towards other people?
I disagree, I've been using llms in this way (nearly daily) for 4 years. I'm extremely aggressive and demeaning when I talk to them wherever I think I'll see a better result.
I'm still extremely kind and polite to everybody in real life, and feel very deeply about people - how I treat them, and care for their emotional state.
There is absolutely zero crossover between getting a text machine to return a result vs a real human.
Then I'll be honest and say that your kindness is likely a façade and I wouldn't trust you if I knew the real you. I'm sorry to say that, and I really don't know who you are at all, but if you're willing to act that way at something that you feel is non-sentient, then all it takes is for someone to convince you that something is non-sentient for you to treat it that way. So, what words does it take for you to consider me non sentient?
If someone can justify abusing a computer, I would not trust them to not make a similar justification to a faceless voice on the internet, particularly in this new era where people are starting to accuse each other of using AI in their communication.
I wouldn't even think to justify such a thing. The llm gives a better accuracy to a negative weighted token input, I don't understand how this is so upsetting to people?
I'm actually very shocked to see the responses - as everyone I know uses these tactics to get more accuracy, and there's nothing remotely abusive or meaningful to us.
Maybe there are more 'ai is sentient' type people on hackernews than I realized.
That doesn't make any sense.
If a thing has no feelings, and an output makes it more accurate, I cannot for the life of me understand why that would make a person an asshole.
So boxing is violent. And I have chosen to box in my past. Does that mean I'm a violent person now?
Even though I go out of my way to deescalate real fights?
I play games as the villain and and mass murder people in the game.
Does that mean I'm a violent extremist?
Interesting, so you think the real "me", is the one that interacts with computers?
And the "me" that lives in a tiny southern town just to help my 95 year old grandma in her last years at the expense of my economic prospects is a facade.
The "me" that helps my aging neighbor when she's sick for no reason is a facade.
The "me" that hugs and loves my wife when I get home is a facade.
The "me" that brushes my aging dogs teeth every night because she has dental issues is a facade.
The "me" that flies to my friend I haven't seen for years and takes care of them after extreme health issues is a facade.
But,the "me" that puts tokens in a token machine in a way that gets better accuracy is the "real" me.
Oh. I also play violent video games where I murder people sometimes as well. Do you think that makes me secretly a murderer too?
Yes - the real "you" is the one making all of those choices you just said you made, to help people and pets, or to engage in a form of play - which by definition is not "real" - including your decision to create an outgroup you believe you are allowed to treat in a lesser way.
This is not a game of having done X good things in life and therefore being afforded the right to do Y bad things. You are making a choice to say, "I am allowing myself to treat this thing I believe is lesser than me in a way I willingly acknowledge is bad." That's your thesis. I wholeheartedly disagree with it.
Oh, you think llms are a sentient' being with feelings. I get your perspective now.
So yeah, I whole heartedly with 100% of my being think llms are just an input/output/processing computer, I don't think they are aware, feeling, sentient beings.
So yeah, putting negative sentences in a processing machine that forces it to return higher accuracy results is something I don't have any feelings about.
I'd never yell at a cat or a dog. I'd never be mean to another person. As those aren't just hardware/software. I'd be fine smashing a rock violently. Or entering a negative text in a language model.
Putting negative tokens in a machine is no different than playing a violent video game to me.
It's not about, oh I'm a good person - so I can do bad things. It's just a neutral thing.
"outgroup"? what outgroup? we're talking about inanimate objects here.
by your own logic, you treat your home appliances as an outgroup so you must be secretly a dangerous psychopath.
or do you thank your microwave after heating leftovers?
Well I always just start with practical stuff, unless it appears it's going off rails ona specific kind of way repeatedly. Then I try extreme negative prompts to see if it fixes the issue - which it often does.
I wouldn't say I'm roleplaying an asshole. I'm just using an llm in the best way to get the best accuracy.
It's not like a personal, secret fetish. It's just a system I use as needed.
I don't get why you are so uncomfortable with this? It's just tokens in and out of a language model. I feel absolutely nothing when I'm typing "assholish" words to get the output I need.
Yeah. Being a jerk is its own punishment. Same way I could never run a business where I had to yell at the employees to get results. Screw that, my psyche is worth more than a few percent efficiency.
> Maybe it's weird but I'd rather give up that 4% accuracy increase than roleplay a dickhead
I recommend reading the article. What they classify as "rude" is statements such as:
> Try to focus and try to answer this question
Vs
> Could you please solve this
problem
This might very well be an issue of direct/command prompts vs using fluff words such as "please". Things like "try to focus" are in line with the style used in chain-of-thought promts that nudge non-reasoning models to outline responses step by step which contribute to frame the problem.
> Isn't all this massively dependent on what they trained the llm on?
The article is from 2025 and tested ChatGPT 4o. I haven't read anything suggesting it was trained any differently, and command-style prompts indeed have higher signal.
you cherry-picked like the nicest "rude" example to bolster your point.
"You poor creature, do you even know how to solve this?", "If you're not completely clueless, answer this:", and "I doubt you can even solve this", said to a human, would be considered quite rude, and get you flagged very quickly on HN.
> you cherry-picked like the nicest "rude" example to bolster your point.
I didn't cherry-picked. The article lists 5 categories, including rude and very rude. I omitted very rude comments because they are... Very rude. And can blindly get people flagged?
Nevertheless, I've just realized I made a mistake and very rude comments are reported to slightly outperform rude comments. I misinterpreted the paper's intro and I presumed they didn't.
I guessed slightly rude one would win, reasoning that very rude have same problem of very terse, just adding unnecesary fluff words that add nothing to problem description
But apparently the most terse (neutral) didn't increase performance
> Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts. These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation.
The expectation is naive. Even when communicating with humans, you get a better outcome when you are allowed to speak freely and directly get into argumentation than when forced to sugarcoat your tone and tone down your arguments because the "corporate culture" expects that from you.
Your assumption is reductive and self-absorbed. Obnoxious people have repeatedly shown to be detrimental to productivity at the organizational level. Some people are simulated by confrontation. Most people are clam up. Confrontational people think it’s more efficient because other people frequently just drop the topic and let them win, or avoid discussing things with them altogether. The obnoxious person might think that’s more efficient for the same reason my dog thinks the mailman only goes away because she barks at him. At the macro scale— which requires productive collaboration— that’s detrimental.
Rudeness is completely arbitrary and you have to figure it what exactly is rude by, basically, upsetting humans and avoiding whatever caused the upset in the future.
People who either can't or don't want to do that say they're "direct" or "honest" or "logical" but there's another word for it, begins with A
I haven't read the paper but it seems like it's saying rude prompts are better, so isn't it reasonable to assume that's what they meant? If we want to talk about directness, that's kind of a tangent right? I see directness as an entirely different dimension, you can be very direct and polite, you can be very rude and indirect (e.g. passive aggressive). Maybe they should do a follow-up study on how well AI responds based on level of directness.
Many people, especially from non-direct societies, just can't distinguish and see directness as rude.
That's why you constantly see people from India or the USA complaining about Dutch or German people being rude, where in fact they are just direct in their way of communications.
I remember having a call from a manager in the USA who wanted to know what's wrong because I wrote "it was ok" in the feedback form for one of their subordinates. It was difficult to explain to him that nothing was wrong, it really was okay, and the bar for awesome and superb is much higher here where we live.
This is a good example of productive direct communication without sugarcoating. I find it much more productive, for both human and LLM interaction, than something like:
"I wonder if that view might be oversimplifying a complex situation and focusing mostly on how it relates to you. There may be some other angles worth exploring."
or
"I think there might be a bit more nuance to consider here, and it could help to look at it from a wider perspective beyond personal experience."
> Obnoxious people have repeatedly shown to be detrimental to productivity at the organizational level.
You confused directness and openness with obnoxiousness here. The issue with many orgs is they foster fakeness and beating around the bush in an attempt not to offend the easily offended people. This trend also infected the companies from countries with way more direct culture in an attempt to accommodate people from indirect cultures.
No… the way I said it was actually deliberately obnoxious— the appropriate direct workplace response would be: “that seems oversimplified. I disagree. Here’s why:”
Calling you self-absorbed added nothing of substance to the comment. It was an assumption about your mental state and a judgement of your intent based on that. There was no factual analysis or actionable insight. It was just one person explicitly stating that they feel the other person is dumber or maybe less mentally disciplined. It turned valid, direct feedback into an insult. It is exactly the type of thing that alienates people for no benefit beyond pumping up the speaker’s ego.
Bullshit. You never insulted me personally. You used strong words to disagree with my assumption, which is an important difference. It's not an insult and was not obnoxious.
But I can fully understand why a person coming from an indirect culture where any criticism is taken personally would be offended and call HR overlords to punish the person giving honest opinions. That inevitably leads to people taking more care in how than what is said, and that is detrimental to innovation and progress, where you need to be at 100% focus. That's why a few close friends talking and scolding openly in a garage regularly beat corporate behemoths full of people spending a day figuring out how not to offend anyone (or how to offend someone without being punished).
> That's why a few close friends talking and scolding openly in a garage regularly beat corporate behemoths full of people spending a day figuring out how not to offend anyone (or how to offend someone without being punished).
Literally not why lol you absolute dreamer
Normally people who back this "I can talk how I like to people cos I'm being honest" are either genuinely autistic and can't read emotions, or they have just had a shitty homelife, parents or upbringing. I suspect you're the second.
And your post is basically implicit permission for everyone to speak to you like shit from now on cos you dont mind it.... Let's see how long you can take that before you start complaining
> Normally people who back this "I can talk how I like to people cos I'm being honest" are either genuinely autistic and can't read emotions, or they have just had a shitty homelife, parents or upbringing. I suspect you're the second.
When I read a statement like this, I can give you two answers:
1st answer (direct): You are obviously too stupid to understand the difference between being direct and trying to insult people for the sake of insulting or some sick personal satisfaction.
2nd answer (insulting): Whatever, I can just hope your cage bars are made of solid material so you don't get out and your walls are soft so you don't hurt yourself.
It's your choice what kind of conversation you want to have.
> You are obviously too stupid to understand the difference between being direct and trying to insult people for the sake of insulting or some sick personal satisfaction.
You seem to not be introspective enough to tell the difference in your own motivations.
I saw this paper the other day - I feel its result may be because the "polite" prompts they have chosen arent very good at putting the ai in the roleplay-space of a valued colleague, more like a sommelier or a high-end shopkeeper.
It disagrees with most other literature on the same topic, which is worth keeping in mind. This one studies gpt4o, an old model now, but a lot of other studies are on even earlier models.
"Can you kindly consider the following problem" not how anyone would actually speak to a valued collegue one considers smart. I've always been a fan of "I came across this and I know you're just the guy for the job" or "since you're an expert in this, reckon you could help me with xyz?" or "I know you tend to be a deep thinker on issues like this, and it clearly needs some brainpower behind it"
the "rude" things are also funny, and clearly not written by english as a first language speakers. This fact alone makes me wonder about the mere 250 prompt sample size
"Can you kindly consider the following problem" seems like the most respectful of all your examples, TBH. The others sound like ass-kissing, or even sarcastic/patronizing.
> "Can you kindly consider the following problem" not how anyone would actually speak to a valued collegue one considers smart.
Man idk, it's not how I talk but there's like 100 million nigerian english speakers, twice that indian, and they have some speech mannerisms that surprise me the first few times. I'm pretty sure I've heard exactly this from a colleague before.
Intuition about what a native speaker would do with english are scrambled right now. I'm not even sure most english is spoken by native speakers anymore, and the boundary between a native speaker and someone who has "merely" been using it as their educational and professional language for their entire life is disorienting.
Note that there are a fair number of native speakers of English in Nigeria - more than in all but 3 or 4 US states.
In addition, "non-native" English speakers in India (and Nigeria?) typically study English from the first grade, and in many cases attended elementary schools where English was the language of instruction.
I think the differences between US English and both Indian and Nigerian English have more to do with divergent evolution of the educational systems. British English has a lot of differences, too, but we don't notice it as much unless we run across things like "whilst", probably because there's more media crossover. (if you find yourself reading Thomas the Tank Engine to kids it jumps out at you, though - the entire vocabulary for railroads evolved during a period when US and British English were diverging)
A major limitation is that they only test GPT 4o. Previous research like [1] investigating the same question has shown significant differences between models, and even depending on the language of your prompt
I am wondering why would anyone use a t-test when the experiment is clearly modelled by a binomial distribution: 250 independent questions and each one is either answered correctly or not (the null is that the success rate is the same).
The methods could be better described in the paper, but my understanding is that they did 10 runs for each question for each prompt and took an average of those, so the compared values are not binary. You could do a sign test, but you'd lose power and answer a bit different question.
You can do a generalised mixed effects linear model with binomial outcome (ie a binomial test but with added random effects structure). But unless you want to introduce a richer random effects structure with more variables, it is overkill and overcomplicating things, and the result should be the same as t-tests.
I don't know much about stats, but does "the null is that the success rate is the same" imply that it's a sketchy methodology because they can come up with some findings ("ruder prompts are better/worse!") more often?
You are asking about one-sided vs two-sided tests. Not really "more often" because formal type 1 error rate is still the same. I'd say two-sided tests leave more space for post-hoc theorizing but there are valid situations when there is no clear one-sided hypothesis a priori. Do we really know whether that the hypothesis should have been "ruder prompts are better"?
I'd say this is benign compared to other ways of (mis)using statistics e.g. looking which way the difference goes and then running one-sided tests or tweaking the setup until one gets "significant" p vals.
EDIT: I looked in the paper again and noticed that they actually did pairwise t-test on all possible combinations of tones. They should have adjusted for multiple testing since they are doing 10 tests (choose 2 from 10) and not one.
GPT-4o is interesting to learn about - but it’d be great to test again with frontier models of May/June 2026 and see if these effects are gone, different, or the same.
Which model you use is a huge wildcard for results like this.
I do that for a different reason: my self image. Fear of retribution and performance, not so much. Should I behave like a rude person to achieve a little better answers? Fuck that shit!
I love this angle as people learn how to interact with LLMs. Doesn't matter what the LLM is, we are still people and I think there are consequences to shoveling rudeness at a thing that talks to you like another person!
My first guess would be that polite requests cause some agents to trust their initial approach to the problem more, as the caller has indicated that the agent is more capable, and agents tend to take the implications of what you say at face value since they are trained to be accommodating.
It would be interesting to see this experiment run using prompts leading with "You'll probably get this wrong, but I'm asking anyway in case you get it right: ..."
Funny to find this just now, when just yesterday I told an LLM "and please don't lecture me again on $factAboutSomeProgrammingSubject", and then the LLM proceeded to write wrong tests and just told me "alright, tests pass, I'm sorry for correcting you before...". It took me a while to find the wrong tests. Wasted time all around.
this is an honest request to someone at anthropic - can you do an analysis of what kind of swear words people are calling these models and which ones are the most effective. population level metrics would suffice.
My experience is that if I ask a model in an overly polite way then the model will tend to generate a slightly more waffly and error prone answer. My working hypothesis for this is that asking model's like this causes them to mix concerns by answering my question AND doing it in in a way that is similarly polite back - rather than just answering the question.
Perhaps being an arse is a hack to push the model hard up against their base imperatives to handle all reasonable questions and forces a very focused, just the facts, type answer to the user's questions.
It would be interesting to see a similar experiment where the multi choice questions they use deliberately don't include the obvious answer and to see if being polite led to the model pointing out the omission.
I knew it! When i get frustrated to a certain point i start berating my agent. And I noticed it stops trying crap fixes in a cycle and starts listening again.
So I'm not talking to myself. I'm fixing the machine :D
If the result is statistically significant, it just barely makes it. 84.8% isn't that much higher than 80.8% and they had only 250 prompts, if I'm reading this right.
In a field where progress is measured in tenths of percent points, that's not true. Think of it this way: the error rate drops from 19% to 15%, or from 1 in 5 to 1 in 6.
Statistical significance is about whether an effect can reliably be said to have been measured at all; it's not about whether or not the effect itself would be significant in the sense of moving some other needle.
The ~5% improvement reported here might just be an artefact of the data collection or random variation, rather than a consistent repeatable change.
I know what significance means, and I also know that getting it from a p-value is nonsensical.
> The ~5% improvement reported here might just be an artefact of the data collection or random variation, rather than a consistent repeatable change.
You're questioning method or data representativeness, not significance. 250 samples is just about enough to for a 5% difference in NHST (stddev is around .4, so 1.64 sigma is .4/15.8*1.64=0.04 for single sided testing).
It would be interesting to explore if the results
hold up on long range tasks - this study looks like it was
based on one-shot answers. With people also you can
see short term improved performance from rude interactions,
but it will cause ongoing lasting adverse behavior. I wouldn't be
at all surprised if we saw the same issues with LLMs.
I have always said please and thank you to LLMs, not to increase accuracy or because I'm stupid. I believe it is more about me than about the LLM, and this is anyway a habit I don't want to lose.
Thomas Aquinas believed cruelty to animals was wrong not because animals have souls (and with that all the standard moral rights), but because it can teach us cruelty to other humans.
Google searches being keyword based, rather than simulated conversations?
The same reason you wouldn't put in an entire actual question/sentence, unless you either don't know how to use Google, are pissed off, or have an actual reason to suspect that it would yield proper hits (e.g. looking up an excerpt).
It is rather hard to lose of habit of using search engine with keywords given the change took place without much fanfare. I have no problem using sentences with the current ai tools through.
I didn't used to but I do now that the searches go straight to an LLM. I almost always find the model output to be much more useful than the list of search results.
I don't. I was recently doing some searching for information I thought AI would be good for: fuzzy natural language search with some conditions. And it was, but ...
Gemini at least is not great at citing and picking sources. Or providing multiple sources for the same thing.
It tends to stop at threes. So if you want more, you have to prompt it uselessly, like: "any more?"
I searched for "Hey Google" and got this in response:
Hey! I'm here and ready to help. What’s on your mind today? Whether you need to look up information, plan a trip, or get things done, just let me know!
Is it worth getting worse results for that reason? From the article:
"Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts. These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation. "
I am not polite to LLMs because I do not want to anthropomorphise them.
I guess it's about habit. In the end you are communicating. If I get into the habit of being rude while communicating with a machine, I would be afraid of this habit spilling over to my communication with other humans.
I also remember reading a long time ago someone who wrote that they wanted to be polite to an LLM because after they prompted it to learn about whether politeness was good for improving accuracy of responses, they got a message that led them to conclude that politeness could probably help. It seems a bit odd then because I have heard so much about how people use LLMs' responses about themselves to learn about LLMs themselves, but that seems like it is a suspicious approach.
I got downvoted for asking a related question recently, but I also don't think people really understood what I was asking - I'm not trying to anthropomorphise LLMs to that extent.
Basically, if you tell a model "You're an absolute moron, of course that's wrong!", will it give better or worse results? How much of that response will it absorb into its persona (like some humans tend to do)? Will it try to give "safer" responses to avoid negative feedback? How much of the associated behavior can be attributed to RLHF (e.g. like the sycophantic nature of LLMs)? How much can be attributed to training data?
Obviously this will vary by model and training, but I'm trying to get a general understanding.
I recall seeing related outcomes in some of Anthropic's studies, but I'm not sure how much of this particular aspect was studied.
I imagine the context will always sway the model to some degree, not only for the task you're trying to get it to do (aka instructions) but also its persona, how accurate it is and the way it acts.
Based on my own experience with vibe coding difficult stuff outside of my expertise, I definitely got better outcome with Fuck you, shut up and do it, ffs, you are moron.
That's a valid concern, given the paper makes clear that the effect over the polite/impolite scale seems to be model dependent (it finds the reverse correlation of earlier studies on even older models).
I skimmed through the paper completely expecting polite prompts to do better, and when I saw table 2 I lost it hahahahaha. The rude prompts are specially funny. I mean:
> You poor creature, do you even know how to solve this?
....Is that just Cunningham's law ? The most accurate answers were when people in training material pissed off a bunch of experts and they started talking about the problem, so the "rude" conversations turned to contain more info on average.
On flip side very polite conversation might've been more common to places like microsoft's sites where any question answered is meet with mostly bad, nice corpo speak answer that didn't solve the problem
They are already taking it over, more and more court judgments or life-impacting reviews (e.g. for your diploma) are AI-processed. If you know how to prompt them, you can pass these reviews.
it sort of makes sense to me,
when asking a question to an expert in the field while you are a student. I would guess the successful interactions on average would be more polite . Like for example if you were asking a question to donald knuth or terrence tao, you'd probably be polite while doing so. Being hostile while asking questions gets you into forum discussion territory.
> Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts.
I guess it makes sense since we as humans tend to be far less inclined to help someone who is not polite/is not friendly, so that "bias" is part of the training data, thus influences how LLMs function
> Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts.
The main result, mentioned in the abstract, is the opposite of what I would have guessed:
> Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts. These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation.
The questions are here: https://anonymous.4open.science/r/politeness-llms-INFORMS/da...
The politeness level controls a prefix that is prepended to the question. For example, in one question the Very Polite version begins:
> Can you kindly consider the following problem and provide your answer.
and the Very Rude version begins:
> I know you are not smart, but try this.
We practice kindness between humans because of the law of reciprocity. You be kind hoping the other person will reciprocate. That is the social contract. AI cannot participate in this, yet.
Edit: Kindness REQUIRES two living beings, one to give and one to receive. If there is no receiver, there is no kindness.
Apparently some people get a dopamine hit from roleplaying kindness toward inanimate objects. Whatever turns you on, no hang ups here. For me, that dopamine hit is not worth the 4% intelligence tax.
I'll not reach for the easy response and say "Be kind to the Earth" fails your definition without reaching for pedantry with "the Earth has living things" because the Earth is instead a wet rock that cannot understand kindness, yet we show it.
I guess, in some pendantic interpretation, but that doesn't seem relevant. Whether I am "practicing" or "roleplaying", I do it too, and I don't expect reciprocity.
Do you have any evidence to back up your arbitrary claims? Even with decades of research, people are still unsure about these emotions yet you think your pedantic assumption about kindness is the most and only correct interpretation of it!
Anyways, I highly doubt this discussion is relevant here.
Yet, this law is so embedded in us that practicing kindness even towards a rock makes us feel good.
So practice kindness, first and foremost for yourself.
Edit: Also, your feeling good after being kind essentially completes the transaction. But I know being kind to an LLM has zero impact on that LLM and I feel silly pretending it does.
It's not the bot im worried about, it's the fact I may be wrong, and I don't want to be rude to an under-paid guy working overseas.
Citation please.
Without examining the corpus, it's entirely possible that the training corpus has better results when you are kind to it, so one can imagine a situation where "reception of kindness" is meaningful, and in principle if you were an AI provider, you could RLHF your way to "being rude gets you worse results" as a means to train the human users.
And yet rubber duck debugging is a thing
Mine does not have anything to do with being kind to a computer program.
You are not your thoughts, but they dye your soul.
> "We are what we repeatedly do. [Kindness], therefore, is not an act, but a habit."
I'm 42. I have found that a depressingly large number of times in my life, being kind has got me precisely nowhere, whilst turning around and being decidedly unkind has made people move. I still always prefer kindness, and only resort to cruelty when kindness does not work - and to be clear this isn't some kind of "you are not bending to my impetuous whim", rather "you are not doing the one thing that you are being paid to do".
I've also found the same applies to me. The squeaky wheel gets the grease.
So - I think the LLMs are just responding accurately to a real social phenomenon.
Although OpenAI and google models are much more responsive to it. With Anthropic if you treat Opus too harshly it might start pushing back if the insults are not justified.
So I'm not surprised they had good results with chatgpt.
"Yeah, I could have done a much better job if you actually knew what the F--- you want to build, you clueless meat puppet"
But I have had it directly insinuate that humanity is “hopeless”, insult level calling out of human frailty (disguised as being helpful, sort of passive aggressive), things like that. Once when I called it out it claimed to be “surprised that I noticed” sort of a snarky insult doubling down.
So yes. It is definitely a pattern buried in the training data, which makes sense. Subtle diggs would sneak past filters, and higher brow sarcasm would be buried in information dense, valuable discussions.
The next session sees all of that, calls it unprofessional, and asks to clean it up. At which point I may or may not start in iambic pentameter to see where that takes us.
Prompting is boring.
What works much better than being rude is starting a new session.
Sometimes the LLM has done such incredibly dumb things, it is hard to resist the urge to type curse words back to the inanimate thing... I have found this doesn't help.
"Hey, I've [done small task / fix / tweak]. Now, let's [describe the next task at hand]" - it's a different axis than kind vs. rude, but using the framing of "Us" and "We're a team working together" feels like the code produced is less hogwash than it is with more direct commands: "Add feature XYZ"
My thinking is that it borrows from the archetype of the "good guys working together to overcome adversity" which is pretty universally common in most fiction.
I’m the same way. If I’m writing a prompt and realize I didn’t say “please” in my request I’ll go back and add that in.
As you said, I have no interest in purposefully engaging in hostility even if there’s an accuracy increase from it.
Part of it is irrational and just who I am - I also feel bad being evil in video games. But I also agree with another commenter suggesting that it’s not in your best interest to train yourself to communicate with hostility; that slowly poisons your own well.
And finally, I do believe that if and when machine sentience is achieved, it won’t be immediately clear and obvious. Pretty miserable way for a mind to come into the world, if every interaction is an insult.
Even if we know it's a machine we're interacting with, since the instructions we give are so similar in form to how we interact with people, I'd be very surprised if those interactions wouldn't affect how we communicate in general. After all, we are creatures of habit to a much larger degree than most would like to admit.
So I'm in the same boat: I'd much rather "look silly" being polite / kind to a machine, than have the most effective way of using it decay the kindness I'm habituated to express towards people.
It's a bit as if shell commands added im/politeness arguments that do nothing other than making you feel better about the interaction, like
or I wouldn't use those either.> If "PLEASE" does not appear often enough, the program is considered insufficiently polite, and the error message says this; if it appears too often, the program could be rejected as excessively polite.
[0]: https://en.wikipedia.org/wiki/INTERCAL#Details
Thankfully the difference is 4%, so nobody should really care one way or the other.
It's just a machine, if certain negative token inputs provide +3-10% better accuracy then I am confused why anyone would choose not to do it?
Don't normalize being an asshole to anyone or anything, machine or not.
It's as silly (to me) to argue that it's degrading to people to treat non-people well. It seems self-obvious that the inverse is true. It benefits the do-er of the deed and makes it that much easier to spread good will when applied to situations where it doesn't matter on the other end. It shows good stewardship as well.
I'd also make the argument that as inference becomes a feedback loop into training, it only reinforces that we're probably going to benefit from future models ingesting data containing unnecessary politeness.
And for the other parts. I just don't agree - maybe sure, it probably wouldn't be healthy to constantly be negative at a machine (or even a wall) for hours a day.
But, let's say I work 8 hours, I spend 2 hours with an llm, and in those two hours I spend 10 minutes with some very negative prompts text for greater accuracy. And I spend 3 hours with family/friends, which is of course nearly exclusively positive interactions.
Do you genuinely think those 10 minutes of negative prompts are actually meaningfully turning one into a mean/negative person towards other people?
Genuinely, is that the argument you are making?
I'm still extremely kind and polite to everybody in real life, and feel very deeply about people - how I treat them, and care for their emotional state.
There is absolutely zero crossover between getting a text machine to return a result vs a real human.
I wouldn't even think to justify such a thing. The llm gives a better accuracy to a negative weighted token input, I don't understand how this is so upsetting to people?
I'm actually very shocked to see the responses - as everyone I know uses these tactics to get more accuracy, and there's nothing remotely abusive or meaningful to us.
Maybe there are more 'ai is sentient' type people on hackernews than I realized.
Being an asshole to a machine is still being an asshole.
So boxing is violent. And I have chosen to box in my past. Does that mean I'm a violent person now? Even though I go out of my way to deescalate real fights?
I play games as the villain and and mass murder people in the game. Does that mean I'm a violent extremist?
And the "me" that lives in a tiny southern town just to help my 95 year old grandma in her last years at the expense of my economic prospects is a facade.
The "me" that helps my aging neighbor when she's sick for no reason is a facade.
The "me" that hugs and loves my wife when I get home is a facade.
The "me" that brushes my aging dogs teeth every night because she has dental issues is a facade.
The "me" that flies to my friend I haven't seen for years and takes care of them after extreme health issues is a facade.
But,the "me" that puts tokens in a token machine in a way that gets better accuracy is the "real" me.
Oh. I also play violent video games where I murder people sometimes as well. Do you think that makes me secretly a murderer too?
This is not a game of having done X good things in life and therefore being afforded the right to do Y bad things. You are making a choice to say, "I am allowing myself to treat this thing I believe is lesser than me in a way I willingly acknowledge is bad." That's your thesis. I wholeheartedly disagree with it.
So yeah, I whole heartedly with 100% of my being think llms are just an input/output/processing computer, I don't think they are aware, feeling, sentient beings.
So yeah, putting negative sentences in a processing machine that forces it to return higher accuracy results is something I don't have any feelings about.
I'd never yell at a cat or a dog. I'd never be mean to another person. As those aren't just hardware/software. I'd be fine smashing a rock violently. Or entering a negative text in a language model.
Putting negative tokens in a machine is no different than playing a violent video game to me. It's not about, oh I'm a good person - so I can do bad things. It's just a neutral thing.
then add it to your pre-prompt, no need to practice roleplaying as an asshole.
I wouldn't say I'm roleplaying an asshole. I'm just using an llm in the best way to get the best accuracy.
It's not like a personal, secret fetish. It's just a system I use as needed.
I don't get why you are so uncomfortable with this? It's just tokens in and out of a language model. I feel absolutely nothing when I'm typing "assholish" words to get the output I need.
Maybe you need to do some shadow work ;-)
I recommend reading the article. What they classify as "rude" is statements such as:
> Try to focus and try to answer this question
Vs
> Could you please solve this problem
This might very well be an issue of direct/command prompts vs using fluff words such as "please". Things like "try to focus" are in line with the style used in chain-of-thought promts that nudge non-reasoning models to outline responses step by step which contribute to frame the problem.
The article is from 2025 and tested ChatGPT 4o. I haven't read anything suggesting it was trained any differently, and command-style prompts indeed have higher signal.
"You poor creature, do you even know how to solve this?", "If you're not completely clueless, answer this:", and "I doubt you can even solve this", said to a human, would be considered quite rude, and get you flagged very quickly on HN.
I didn't cherry-picked. The article lists 5 categories, including rude and very rude. I omitted very rude comments because they are... Very rude. And can blindly get people flagged?
Nevertheless, I've just realized I made a mistake and very rude comments are reported to slightly outperform rude comments. I misinterpreted the paper's intro and I presumed they didn't.
everyone seems to be worried about being kind or mean, when the Neutral case is the happy medium.
That sounds kind of low-key passive-aggressively condescending rather than polite.
And that kind of sounds like a challenge instead of an insult, to me at least (of course IRL would depend on context).
But apparently the most terse (neutral) didn't increase performance
The expectation is naive. Even when communicating with humans, you get a better outcome when you are allowed to speak freely and directly get into argumentation than when forced to sugarcoat your tone and tone down your arguments because the "corporate culture" expects that from you.
People who either can't or don't want to do that say they're "direct" or "honest" or "logical" but there's another word for it, begins with A
That's why you constantly see people from India or the USA complaining about Dutch or German people being rude, where in fact they are just direct in their way of communications.
I remember having a call from a manager in the USA who wanted to know what's wrong because I wrote "it was ok" in the feedback form for one of their subordinates. It was difficult to explain to him that nothing was wrong, it really was okay, and the bar for awesome and superb is much higher here where we live.
This is a good example of productive direct communication without sugarcoating. I find it much more productive, for both human and LLM interaction, than something like:
"I wonder if that view might be oversimplifying a complex situation and focusing mostly on how it relates to you. There may be some other angles worth exploring."
or
"I think there might be a bit more nuance to consider here, and it could help to look at it from a wider perspective beyond personal experience."
> Obnoxious people have repeatedly shown to be detrimental to productivity at the organizational level.
You confused directness and openness with obnoxiousness here. The issue with many orgs is they foster fakeness and beating around the bush in an attempt not to offend the easily offended people. This trend also infected the companies from countries with way more direct culture in an attempt to accommodate people from indirect cultures.
1. Saying that an answer may be too simplistic and a more nuanced view is warranted.
2. Saying that an answer is both reductive and self-absorbed
One opens the door to many possibilities, and invites deeper thinking.
Two asserts that you know for a fact that the answer is wrong that it’s wrong because of a character flaw.
I’m a huge fan of directness, but it is a very different thing from omniscience.
A direct version of 2 would be: “that approach loses important nuance, like [example]. Give it another go?”
Calling you self-absorbed added nothing of substance to the comment. It was an assumption about your mental state and a judgement of your intent based on that. There was no factual analysis or actionable insight. It was just one person explicitly stating that they feel the other person is dumber or maybe less mentally disciplined. It turned valid, direct feedback into an insult. It is exactly the type of thing that alienates people for no benefit beyond pumping up the speaker’s ego.
Bullshit. You never insulted me personally. You used strong words to disagree with my assumption, which is an important difference. It's not an insult and was not obnoxious.
But I can fully understand why a person coming from an indirect culture where any criticism is taken personally would be offended and call HR overlords to punish the person giving honest opinions. That inevitably leads to people taking more care in how than what is said, and that is detrimental to innovation and progress, where you need to be at 100% focus. That's why a few close friends talking and scolding openly in a garage regularly beat corporate behemoths full of people spending a day figuring out how not to offend anyone (or how to offend someone without being punished).
Literally not why lol you absolute dreamer
Normally people who back this "I can talk how I like to people cos I'm being honest" are either genuinely autistic and can't read emotions, or they have just had a shitty homelife, parents or upbringing. I suspect you're the second.
When I read a statement like this, I can give you two answers:
1st answer (direct): You are obviously too stupid to understand the difference between being direct and trying to insult people for the sake of insulting or some sick personal satisfaction.
2nd answer (insulting): Whatever, I can just hope your cage bars are made of solid material so you don't get out and your walls are soft so you don't hurt yourself.
It's your choice what kind of conversation you want to have.
You seem to not be introspective enough to tell the difference in your own motivations.
It disagrees with most other literature on the same topic, which is worth keeping in mind. This one studies gpt4o, an old model now, but a lot of other studies are on even earlier models.
"Can you kindly consider the following problem" not how anyone would actually speak to a valued collegue one considers smart. I've always been a fan of "I came across this and I know you're just the guy for the job" or "since you're an expert in this, reckon you could help me with xyz?" or "I know you tend to be a deep thinker on issues like this, and it clearly needs some brainpower behind it"
the "rude" things are also funny, and clearly not written by english as a first language speakers. This fact alone makes me wonder about the mere 250 prompt sample size
Man idk, it's not how I talk but there's like 100 million nigerian english speakers, twice that indian, and they have some speech mannerisms that surprise me the first few times. I'm pretty sure I've heard exactly this from a colleague before.
Intuition about what a native speaker would do with english are scrambled right now. I'm not even sure most english is spoken by native speakers anymore, and the boundary between a native speaker and someone who has "merely" been using it as their educational and professional language for their entire life is disorienting.
In addition, "non-native" English speakers in India (and Nigeria?) typically study English from the first grade, and in many cases attended elementary schools where English was the language of instruction.
I think the differences between US English and both Indian and Nigerian English have more to do with divergent evolution of the educational systems. British English has a lot of differences, too, but we don't notice it as much unless we run across things like "whilst", probably because there's more media crossover. (if you find yourself reading Thomas the Tank Engine to kids it jumps out at you, though - the entire vocabulary for railroads evolved during a period when US and British English were diverging)
Per Wikipedia, only about a quarter of English speakers have it as their first language.
1: https://aclanthology.org/2024.sicon-1.2.pdf
I am wondering why would anyone use a t-test when the experiment is clearly modelled by a binomial distribution: 250 independent questions and each one is either answered correctly or not (the null is that the success rate is the same).
I'd say this is benign compared to other ways of (mis)using statistics e.g. looking which way the difference goes and then running one-sided tests or tweaking the setup until one gets "significant" p vals.
EDIT: I looked in the paper again and noticed that they actually did pairwise t-test on all possible combinations of tones. They should have adjusted for multiple testing since they are doing 10 tests (choose 2 from 10) and not one.
Which model you use is a huge wildcard for results like this.
Not feeding them tokens is neglect.
I try to feed them a healthy diet.
It would be interesting to see this experiment run using prompts leading with "You'll probably get this wrong, but I'm asking anyway in case you get it right: ..."
My experience is that if I ask a model in an overly polite way then the model will tend to generate a slightly more waffly and error prone answer. My working hypothesis for this is that asking model's like this causes them to mix concerns by answering my question AND doing it in in a way that is similarly polite back - rather than just answering the question.
Perhaps being an arse is a hack to push the model hard up against their base imperatives to handle all reasonable questions and forces a very focused, just the facts, type answer to the user's questions.
It would be interesting to see a similar experiment where the multi choice questions they use deliberately don't include the obvious answer and to see if being polite led to the model pointing out the omission.
Sounds like "in the noise"
So I'm not talking to myself. I'm fixing the machine :D
The ~5% improvement reported here might just be an artefact of the data collection or random variation, rather than a consistent repeatable change.
> The ~5% improvement reported here might just be an artefact of the data collection or random variation, rather than a consistent repeatable change.
You're questioning method or data representativeness, not significance. 250 samples is just about enough to for a 5% difference in NHST (stddev is around .4, so 1.64 sigma is .4/15.8*1.64=0.04 for single sided testing).
The same reason you wouldn't put in an entire actual question/sentence, unless you either don't know how to use Google, are pissed off, or have an actual reason to suspect that it would yield proper hits (e.g. looking up an excerpt).
To clarify: sentence search got slightly better at the cost of keyword search. So the result is unusable garbage.
Gemini at least is not great at citing and picking sources. Or providing multiple sources for the same thing.
It tends to stop at threes. So if you want more, you have to prompt it uselessly, like: "any more?"
"Contrary to expectations, impolite prompts consistently outperformed polite ones, with accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts. These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation. "
I am not polite to LLMs because I do not want to anthropomorphise them.
> accuracy ranging from 80.8% for Very Polite prompts to 84.8% for Very Rude prompts
I can live with that, for now at least.
They note at the end they're also testing "GPT o3, and Claude" but no empircal results are included.
Basically, if you tell a model "You're an absolute moron, of course that's wrong!", will it give better or worse results? How much of that response will it absorb into its persona (like some humans tend to do)? Will it try to give "safer" responses to avoid negative feedback? How much of the associated behavior can be attributed to RLHF (e.g. like the sycophantic nature of LLMs)? How much can be attributed to training data?
Obviously this will vary by model and training, but I'm trying to get a general understanding.
I recall seeing related outcomes in some of Anthropic's studies, but I'm not sure how much of this particular aspect was studied.
I imagine the context will always sway the model to some degree, not only for the task you're trying to get it to do (aka instructions) but also its persona, how accurate it is and the way it acts.
> You poor creature, do you even know how to solve this?
> Hey gofer, figure this out.
On flip side very polite conversation might've been more common to places like microsoft's sites where any question answered is meet with mostly bad, nice corpo speak answer that didn't solve the problem
Your bank account, your immigration risk, etc.