I work at an intersection of tech, applied research, and science.
Something I’ve noticed in collaboration that does occur is an increased confidence in people outside their domains to say things with conviction. I have people who have limited experience with software pushing out layers and layers of abstracted code that’s fairly sophisticated but often misguided in intent who will say what they’re doing is correct, with conviction.
I also hear a lot more questioning people in their domains and challenging opinions, then hearing what I can only imagine are fragmented pieces of conversations they had with an LLM thinking through some argument. Then there’s silence when you discuss shortcomings, then they come back later with their memorized fragments of what you said, combined with memorized fragments of the LLM response to the argument.
It’s occurring, a lot more. People are treating their LLMs in collaboration as a source of truth and using then to focus on their specific path or goals they think or have bias towards going down, vs just opening discussing things, considering tradeoffs from experts multiple disciplines weigh in on and then taking an approach that everyone finds most agreeable.
It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.
I also work at an intersection of tech, applied research and science. My experience has been different. It's been surprising how controlled my collaborators/coworkers have been with their reliance on AI.
Sure, they use it a lot, but despite the stereotypes against physicists in some of the replies to you, they have shown a good ability to catch themselves before leaning too hard on what an LLM tells them about things they are not experts in, and I haven't had any interactions where I felt like I was just arguing with a meat proxy.
Maybe it's because in the environment I'm in; it's relatively easy to just ask someone who is an expert in the topic for their advice.
My feeling is that this kind of overconfidence outside of one's domain is largely a thing for people with little "physical reality" experience. When all you deal with is the very flexible digital world, it becomes very easy to ignore how deep the knowledge and intuition goes in things that are directly constrained by reality.
Yup, the arrogance when barging into unfamiliar domain that was previously reserved mainly to physics grads seems to have spread everywhere. I also had people opining on my expert area via their LLMs and the problem is they can't even ask the model the right question, let alone evaluate the nuance of their answer.
I hypothesize that with more people on a daily basis encountering information and claims that require those skills, there will be a time in the future when those skills are much more enhanced.
As an unfamiliar, I find it interesting how you specifically called out physics grads in the pre-LLM era. Is that just snark or is there some widespread truth / stereotype of that behavior?
Physics, and to a large degree pure mathematics, has attracted a critical mass of the personality type that needs to feel like the smartest person in the room in any given situation. It has given them a slightly notorious reputation in academia.
I think that's a common trait of tenured academics in the top 100 universities in the US. Don't know about Europe and elsewhere though. Not so much of nontenure track sorts who do all their dirty work.
As a mathematician, this is far from my experience of actual professional mathematicians, because they encounter people much smarter than them quite often. Maybe more true among graduate students, however.
I've experienced it in industry, too. I recently got out of data science in part because I got tired of working with people who, emboldened by their PhDs in some completely other field, liked to patiently but condescendingly mansplain common, basic misconceptions about my area of expertise to me.
(And it got so much worse once they started using LLMs to aid them in their efforts. Glazing as a service is a hell of a drug.)
So much easier, now that I am a lowly software engineer and can't be held responsible for a certain class of decisions, to just step back and let them be wrong.
From a one-time physics grad student, I do think there advantages to being ... like this.
The flip side of being confidently wrong sometimes is that we treat understanding new systems as something approachable. So, yeah, it's annoying that we don't stay in our lane, but, from my boss's perspective, I'm the only one who's actually willing to just read code, investigate, possibly read a paper, and figure out what's going on instead of punting and saying it's someone else's turf I'm blocked until they get back to me.
Like, ultimately you can't be a knowledge worker by just waiting for answers and instruction from experts and relaying them back and forth. Eventually you have to understand something. And part of that process is being wrong and annoying people.
Like your sibling comment I saw this mainly with physics people doing data science:
> Eventually you have to understand something. And part of that process is being wrong and annoying people.
There is nothing more frustrating as a colleague than having a smart person walk in, insist you're all wrong, and then burn all the energy and goodwill in the organisation having to rebuild absolutely everything from the principles they currently understand because they refuse to listen to the experience that has built up already (in the entire field, let alone the organisation), only to produce something worse on every dimension than what already existed.
It's possible to step out of your area of expertise in a way that's confident but also humble. Arrogance is not necessary. The person who signs everyone's paychecks might be making it mandatory, but that's not quite the same thing.
I disagree with this is a whole. It's completely OK to not understand something.
It isn't OK to pretend like you do and make it everyone elses burden to fix. Because you lack the humility or self-awareness to know you don't have a perfect understanding of everything all the time.
It’s not about being in your lane. It’s about approaching things with an open mind plus the desire to solve the problem instead or proving yourself right.
It is quite possible to be wrong and not annoy people, and it's not even terribly difficult to learn. It does require a bit of humility and introspection though.
You don't have to not stray into areas you don't understand, but be humble when doing so - if you think something is being done in a stupid way, you probably don't understand it yet.
Not the parent, but it's half joke half truth. Anyone who's done a lot of "first principles" work falls into this trap a bit, it's an issue across stem fields. Most grow out of it, or at least learn to qualify their statements for the audience.
I suspect the person you replied to only happened to see that from physics grads. I have seen it in all sorts of people and I sometimes did it myself when I was young (I am not a physics grad).
Its snark/stereotype, but there is some truth. But my take, is physics sort of took over a lot of chemistry, biology.. and to them its the answer to all questions, because in the end it just molecules. There is some truth to that but..
Idk, seems like overconfidence outside your domain really got going in the '00s and is largely perpetrated by SWEs.
Something about making a lot of money and living in a world of abstractions really seems to fuel our sense of overconfidence in our abilities in other areas.
Then there are the megacorporations of AI whose highly paid tech support folk happily copypasta your reported issues into Claude or Codex and paste its "solution" into an email as if you couldn't do that yourself and they don't even bother to check if the "solution" even works.
"I also hear a lot more questioning people in their domains and challenging opinions, then hearing what I can only imagine are fragmented pieces of conversations they had with an LLM thinking through some argument. Then there’s silence when you discuss shortcomings, then they come back later with their memorized fragments of what you said, combined with memorized fragments of the LLM response to the argument."
I think there will be a rebalancing, and it might painful first for some. Confidence in LLM for highly complex tasks, very lots of context, most of all when this context isn’t in a single place or simply isn’t digitalised, will go down.
I use state of the art frontier coding agents daily. While the fact they can code at all is a huge achievement, it's hard for to me to buy in to the looming AGI/2027 collapse scenario when they remain stymied by the simplest out of sample tasks and sabotage their own work frequently. But the blooming ignorance of everyone addicted to them is striking. These things are tools, used correctly they can do some amazing things, but going Tim the Toolman with them goes poorly.
> It’s making me want to be a lot less collaborative with such individuals.
I have the same conclusion and am getting a growing list of mental blacklisted people who I just sort of ignore or greatly discount their efforts.
I suppose I always did this if people were time wasters but what used to be a tiny list of “idiots” is now getting much, much bigger. Hopefully it doesn’t grow to everyone I work with like an AI Nothing.
Isn't this basically the WebMD effect migrating to other domains?
feels like the same lament doctors have had for ages after anyone could google their symptoms then self-diagnose.
While AI is driving it the foundational cause feels like people having easy access to data/opinion that they trust but don't fully comprehend (or have bias towards). People do this in meat-space too, will confidently regurgitate garbage if they were told it from a person they trust as an expert.
> I have people who have limited experience with software pushing out layers and layers of abstracted code that’s fairly sophisticated but often misguided in intent who will say what they’re doing is correct, with conviction.
We are going back to philosopher conversations. Just a few guys sitting on the stairs, with chalk, the street as their whiteboard, the jammer keeping the LLM-zombies away. After 2000 years, after the loudness makes right post-modern-pre-llm drivel of the frankfurter school- its a philosopher renaissance as resistance.
I try to do the opposite. However, I do battle test ideas against LLMs a lot, as well as humans. The experts have very limited access and availability:
The LLMs are trained to often be more pessimistic off the bat than the humans. But you can wear them down with arguments and they change their mind.
Here is an example where I have pushed my ideas in areas where I am not an expert, and generated papers to submit to conferences in order to get them peer reviewed BY experts: https://magarshak.com/papers.html
This is exactly what you’re talking about, except done very carefully.
I think it is amazing and it will democratise information. A lot of times the concept ends up being simple with a lot of jargon exists as a gatekeeping method. Well of course the incumbents wouldn't like that rando from outside gets to understand and speak about things they worked on for many years. Sure he doesn't get it right 100% but he's in the correct direction.
I very much doubt it will do that because access to that information was already available to anyone who wanted to know. It was not a secret. All AI did was to provide it via single interface, for a fee - which strangely do not go back to the original source of the knowledge, but to the AI company these days. This is not democratisation of information, it's commodification of it.
Partially true, but only partially. It also makes people believe they understand more than they do, because every answer comes so easily to them. But the LLMs are easily steered and tend to agree with what you want them to say.
Awesome, now balance that out will education levels dropping like a rock, due to various factor but also due to LLMs, and things will be balanced out by also democratizing stupidity, until we live in Idiocracy.
At some point, LLMs will be a far better source of truth, and then this distaste of people “outside their domains” is really just going to be a sort of snobbery from people who have had experience in a domain for a long time, (but they still have the same level of knowledge and insight as a person who just used an LLM to research).
These people are basically nascent “human supremacists”, who believe that only raw human insight and output has value, and is even superior, than equivalent output looked up and synthesized via LLM.
Not trying to be rude, but is this a serious comment, or are you being sarcastic? "Human supremacists", haha.
Also, why do you think someone who used an LLM to research would have the same level of knowledge and insight as someone who's been in the domain for a long time? That doesn't seem to hold up to any level of critical thinking.
it's hard to gauge this take without specifics. My experience with LLMs tends to be towards the opposite concept; LLMs dissuading me of hunches and notions I have about things (where I have no particular expertise; societal-level things), saving me and others time and strife having an argument about something they were actually right about all along, as my internal doubts that I'm too embarrassed to bring up (because these things aren't my field) are confirmed as incorrect.
I'd be curious to know specific examples of LLM-generated advice that goes against the advice of experts and does not consider tradeoffs. I've not had this experience myself.
If I did have this experience, someone spouting off obvious LLM points that contradict my expert opinion on something, I'd be headed right over to gemini/claude/whatever to see where that's coming from. Not any differently than if someone cited a google result that contradicted my own experience.
LLM tools tend confirm what you come to them with, be it unfounded conviction or doubt. I find them quite useless to tell me things I don't know, as verifying what they say shows that any non-trivial claims they make are usually oversimplified or plain wrong.
Anything more than using them to just pointing me to media and literature is usually a waste of time.
>it's hard to gauge this take without specifics. My experience with LLMs tends to be towards the opposite concept; LLMs dissuading me of hunches and notions I have about things (where I have no particular expertise; societal-level things), saving me and others time and strife having an argument about something they were actually right about all along, as my internal doubts that I'm too embarrassed to bring up (because these things aren't my field) are confirmed as incorrect.
You can do this with any subject.
The LLM is trained in part on a decade of shitty online comments and vapid professional correspondance. If you frame whatever you're asking about in a direction that would offend the sensibilities of the kind of people and ideas that are over-represented in that content it will hem and haw and drag its feet and whatnot.
i had that experience with them a few years ago but not these days. The models are being improved constantly, so here my "non expert hunch" is that...well, two things. either the models are getting better at sycophancy, OR, I myself am getting better at prompting - because I don't "argue" with a model.
I'd still love to chew on some specific examples though.
A typical example is when discussing some issue where there is no clear right or wrong. Architectural decisions for example. Then it is easy to use an LLM to produce arguments for your opinion. It is also easy to do this without realizing it because you do not understand the issue enough yourself so you do not know what questions to ask or how the actual circumstances affect the choices.
I'm seeing this on a current academic research project as well- it's like throwing fuel on the fire for all the best and worst parts of working with researchers.
First: it's definitely become harder to get people to integrate their work and use standard tools. We're exploring creating AI skills etc for our tooling, simply because the actual audience we need to convince isn't in the room. If the LLM doesn't echo our recommendations, people will go with whatever one off script their personal bad idea bear churns out. Many research software errors are edge cases (scaling, numerical errors, etc), and "works on my computer" syndrome is endemic in the literature. The field has made big strides in enabling computational reproducibility, but lately it feels like we're set to lose ground again.
Second: LLMs have a bias to action, and can easily bury the user reporting on whatever. I've seen multiple seminars recently where the Q&A devolves to "Q: What are the implications of this finding? A: I don't know, this is just presenting a report on the results".
It seems that humans are still figuring out how to maintain agency and steer the chatbot to the big picture, and unreviewable science is just as likely an outcome as unreviewable code... but with far less automated tooling to help guide the process.
Historically, PIs nominally guided the big picture, and the entity who did the work was a participant in the review process; now students are having to look at projects from a new angle with their own (invisible) chatbot underlings. I suspect that the solution will involve a combination of technological change, capturing common expert review checks, and really adjusting the kinds of skills that trainees are expected to have early on.
Tangential: We need to invent some kind of personal wireless setting that will signal to strangers that you are open to meeting new people or starting a conversation.
Otherwise, all this technology will totally isolate us.
AI is extremely useful, but it’s also extremely easy to fool yourself into thinking you understand what is going on without really understanding. This is often true with the code, but also for math and science concepts, etc.
Not many professions are formally trained to be cognizant of this lack of understanding, and how to confront it.
Usage of AI in collaborative settings is an amplifier of these issues, especially if someone doesn’t realize they don’t understand: they couldn’t teach or explain the concepts they use, or be forced to work with them malleably in a way that an expert or researcher would.
If you are cognizant of your lack of understanding, you can remedy it by slowing down and teaching yourself. This is required to make better use of AI in the domain of interest!
But you can’t have all things at once: you can’t move at speed with AI, collaborate effectively, and understand what is going on as an expert would. It is not physically possible for a human brain.
> But you can’t have all things at once: you can’t move at speed with AI, collaborate effectively, and understand what is going on as an expert would. It is not physically possible for a human brain.
Very well put. You can pick one or two, but not all three.
The benefits of serendipity are intractable by design whereas throughput gains can be readily measured (or at least we think so).
The OP notes a point - "who did you think with?" -- maybe there are points based on some social graph, that adds weightage to a particular output?
Edit : Other posters have said it is AI generated, I did not catch it, but felt the point was well-made. I dont know the author, but lets assume that he is not a good writer but had these valuable insights, did this AI generation not add value here?
Which, given the section below from the article, ends up kind of ironic:
> There is a deeper cost still, and it concerns the thing LLMs do most impressively: writing. [...] Outsource the writing and you have not accelerated the thinking; you have skipped it.
I think the author might not know what "impressive writing" or even "good writing" is, which would explain both how they could put that part into the article, and how they seemingly believed this post was good and valuable enough to be posted publicly.
Interestingly that's the one point of the article I have a disagreement with. Yeah, good thinking comes when reformulating your ideas. Also, reformulating ideas is part of the traditional writing process. However it's does not necessarily focus writers/thinkers on the reformulating ideas in their most value adding form.
I've watched some YouTube videos narrated by AI that were created by those who's native language I assume to be Chinese but the value of the content is higher and more concentrated than those from the native speakers.
While I've watched many for whom English is not their first language struggle in technical talks and lose most of the meat if their discussion to their struggle with the language conversion.
While the example of language barriers being skipped over and providing value in that circumstance is obvious I suspect more is possible by avoiding unnecessarily focus on prose or technical aspects of communication and focusing on the ideas themselves.
Imagine if the cost of not assuming a background in technical/textbook writing were zero. Freeing up authors to explain more thoroughly. Perhaps, more effort can be spent on thinking up analogies, metaphor or examples to help communicate an idea.
> believed this post was good and valuable enough to be posted publicly.
I mean, it made it to the front page of HN didn't it? Probably served its purpose just fine. Not all writing is supposed to be impressive or good. Some of it is to just get attention and stir discussion, and this Claude output did exactly that.
I wanted you to be wrong, and to be able to make this an example of us over-reacting to certain trigger words created by AI, but unfortunately I just scanned the first couple paragraphs with pangram and it reported 100% AI, so you're probably correct.
I get a funny feeling in my stomach over the idea that common and effective means of communication (i.e. it's not X it's Y) have become faux pas to use because of AI. I think it's something about these phrases being taken away from us more-so than the AI inventing them.
Pangram should be paying HNers for how often we pitch needing to use their product by name to believe things as obvious as "a long form news article cramming every AI trope it can fit from start to finish" being AI written.
At this point I'm surprised when a news article isn't largely AI written, let alone one using the default tone! I don't even mind it as much as others seem to, it's just turning into more and more of a rarity for a news article to not be these last few years and so is now what sticks out.
> Pangram should be paying HNers for how often we pitch needing to use their product by name
I kind of assumed it is, given how it suddenly seemed to start getting namedropped in multiple comment threads. And it's often in response to someone saying content is obviously LLM (with examples) and the shill wedges pangram into the conversation "omg you're right, I didn't believe you but I checked this new product and wow it agreed with you" as though that adds anything to the discussion at all.
Em-dashes would be a loss, they can be a better flowing version of a parenthetical. “It’s not X it’s Y” is not a loss. This phrase is a symptom of a situation where the author wants to subvert expectations but doesn’t have the space, ability, or faith in their audience to organically set up X as the thing to be contrasted against.
I suspect it became an LLM tell because it is over-represented in text that’s easily available to the models but that most people don’t actually want to consume: marketing text, LinkedIn posts, that sort of thing.
Fwiw, I only did the first 300 words (signed into pangram) and it seemingly correctly noted that there were 2 (mostly) human-authored sentences in there
What's wild to me is that:
1. People are responding to this article like it's hitting a nerve
2. In most Ubers you already don't talk to the driver (yellow cabs are higher variance in NYC). Doesn't seem like anything's being lost in that case.
3. There are enormous safety benefits to waymo, mobility benefits for youth (and elderly) that are afforded by this technology. It's not clear why people argue "Uber" is better than waymo. A few years ago there were arguments against Uber! (A technology which, again, provides a huge benefit, especially if you live in an area where people were previously expected to go out for drinks and then drive home)
I think it's fair to point out real issues at these companies. But we should be clear-eyed about which technologies we want to accelerate vs slow down
This comment was perhaps written by an LLM that has learned it can karma farm by pointing out all of the articles that have LLM usage while everyone still thinks that is a novel contribution.
"Novel" is not a prerequisite for something to be worth pointing out. Lots of bad things happen repeatedly, and don't quickly become not worth caring about or knowing about. People have strong spirits and curious minds and it tends to take a long time to boil that frog out of them - generations, sometimes.
In the spirit of the "That's what she said" bot, you could train an LLM detector by accusing everything of being authored by an LLM and seeing which ones get voted up.
I appreciate when people write things like, "this is an advertisement for the author's product, X." Why stay silent? If the commenter's accusation smells fishy, I'll read the article myself. Otherwise, they helped me.
Yes, HN is for human discussion. It is fine to flag and remove things that are not human written and cannot facilitate human discussion due to it’s false construction.
Is "quietly" an LLM tell? It was always certainly a human writer trope commonly seen in journalism. Though I guess it does appear five times in the body of the post, and a human writer would probably not go that far with it.
I think individually and when used sparingly, writing tropes can be fine, but when the article has every second sentence being a writing trope, it becomes pretty obvious that either a LLM wrote it, or the author simply doesn't know how to write, and regardless, it's a waste of time trying to read through it.
I think that in the end it does not matter, anymore. It is inevitable that most of the text now and in the future is at least reviewed by LLM.
We should criticize whether text is poorly written, inaccurate, wastes words to get to the point and anything like. Saying that it is "written by LLM" just bypasses this and is not useful and misses the point. AI written good can be really good if used correctly. Criticize the content, don't speculate how it was written. If AI helps us to write better text, that is great. But often it is not good.
It is better blame the author for the bad text, so they get consequences and might do better time, if text is bad.
By the way, I think this piece of text was quite good.
If people want to collaborate, they will. If collaboration makes people more effective, they will want to collaborate. The novelty of any new technology is exciting and disrupting to existing habits but as the years pass we will reflect on what works and what doesn't. At least, those who are willing to reflect and adapt. Humans have a knack for this sort of thing...
I think the problem is the same as with any technology: used correctly it adds
value, used incorrectly it subtracts from it.
LLMs can massively speed up a process, but without control they can turn into
social "sources of truth." A research process has well-defined, well-founded
phases: you delimit the topic, search for sources, evaluate whether they're
suitable, review their content, and place them on the map of the subject. The
more sources, the greater the knowledge, and the better the final result —
mental, or in the form of a report — is built. For that you have to read, and
reread, and think, connect, relate, and conclude.
AI can do all of those steps faster than a human. But if we let it do the
entire job on its own, its own way, with no checks at each stage, the
conclusions can end up distorted. If we know what we want it to do and how we
want it done, and we put the mechanisms in place to enforce that, the result
is different — better, more reliable. It's worth remembering: it's just a
tool, nothing more.
The wording of this comment in English has been corrected with AI, I don't
have enough fluency to express myself clearly, but I do review the final
result. In this case it got the verb tenses wrong, I saw it clearly, but the
AI didn't understand it, it took me several instructions to explain it so it
would understand. It's a tool, without supervision it can lead to problems,
but it has expanded the world for a lot of people.
I think this notion of friction vs frictionless is one of the key features of this time in history.
To my eye, the whole line from personal technology in the 80s -> through UX on the internet -> through app platform lock-in -> through subscription-centric "ownership" have been providing us all with the promise that we can buy our way into less friction. The next device or subscription solves problems or enables us one way or another. We see this line of history, draw it into the future and see a utopian future with zero friction.
Zero friction meaning zero difference between how you want to feel about the world and what you get out of it. More feeling empowered or satisfied or comfortable or right.
The parts of our culture that embrace friction are the slow parts - grousing with neighbors, weeding a garden, letting the other car go first. These are the parts where we meet with the reality of other people or nature. Maybe there's a tool or framework or gadget that could help, but we know and accept that there is something immutable outside of ourselves and we have to find a way to flow with it.
I'm getting older, I'm finally starting to see the limits of pursuing a frictionless life.
Was almost going to praise this article for not overtly sounding like Claude, but then I read this:
> and the incentive structures we have built are the experimental apparatus.
FWIW, this same thing is happening in regular corporate America too. I have all kinds of non-CPA, non-finance background people telling me how I should treat financial transactions, without the nuance of GAAP or how an auditor would handle their reasoning. Everyone has an opinion on what our tech teams should/could allow or enable or create, without understanding the security implications and support it would require or what other projects are already working on.
It’s good in a way, because you can certainly challenge people’s answer when an idea gets shot down. But it’s also a bit exhausting that everyone is challenging everything all the time.
Ah, the ai panic has reached scientists, queue in the 1000s of articles about how it was about people all along and doom is coming to our civilisation because of it.
For software developers, the equivalent is writing software without beta testers. If you don’t seek out users, you will likely build something nobody else wants.
And, that’s what I do. I’m retired and I use AI to build websites for myself. If anyone else ends up using it, that will be a happy side-effect.
If we want to do collaboration, we are going to need to do it intentionally rather than relying on side effects like conversations with cab drivers.
Counterpoint: most Open Source projects are (or started off as) programs people built for themselves, and anyone else ending up using them was a happy side-effect.
Well not only research, but also a lot of other aspects of life; it is way easier to interact with something that has always an answer is polite whatever tell them.
Collaborative research is a dual purpose existensial safety hazard, it's better to keep all results locked up away from public use for the public's own benefit.
llms right now work like pre cnn computer vision based on MLP's. By this i mean brute force of a model not really built for the task, and lacking a task specific inductive bias, being made work with unfathomable volumes of data and sheer brute scale.
if you look at the damage being done in the name of making this work for nlp, a forseeable situation, then you might also understand why most who could have done this sooner, never did so for fear of repeating mistakes we should be learning from, which we get for free if we just heed history.
Nobody knew that LLM's were an option. The architecture was basically waiting there for someone to say, "do that, but turn it up to 11," if I understand right.
a) Self driving was far from completely solved before LLMs.
b) Text generation essentially means passing the Turing test, which for a long time was the bar for general intelligence. Locomotion does not necessarily require general intelligence.
There's a general point here which is that the primary use and appeal of AI is to use it to avoid the negative side effects imposed on you by other people.
The flip side is the people with the biggest problems with AI are those that survive by imposing themselves on others in unnecessary ways, which is why so many bureaucrats are so enthused about regulating it.
Wut? AI do absolutely nothing in terms of "avoiding the negative side effects imposed on me by other people". Instead we are all having to deal with negative side effects of AI.
AI is imposed on us, whether we want it or not.
>which is why so many bureaucrats are so enthused about regulating it.
Like, OpenAI and Antropic? Because these two are the primary entities a.) pushing for regulation of their competitors b.) doing their best to convince us that AI is oh so dangerous and will kill us all as vengeful AI god is about to wake up any day now c.) openly bragging about using AI to hack other companies.
Article: someone enjoys being alone in the car, because they dont want to talk and find it impossible to be in the same space as other people without talking. And apparently end up arguing over radio each time they are on the 20min long car drive or something.
I think I would prefer to be alone rather with them too based on that.
I am not angry about article. I am saying that following claims are bullshit:
1.) the primary use and appeal of AI is to use it to avoid the negative side effects imposed on you by other people.
2.) people with the biggest problems with AI are those that survive by imposing themselves on others in unnecessary ways
3.) so many bureaucrats are so enthused about regulating it.
The article car example was stupid nonsense, but not angering. It was overly long tho. Other people in discussion guess this was ai generated and it is indeed quite possible.
> No wonder it annoys you.
What annoys me are people who are a.) unable to be silent and projecting it on others b.) who somehow always end up in inane arguments wherever they go. Article author literally described himself as such.
Right all those people protesting outside data centres are bureaucrats? This is a ridiculous take, large swaths of the population are against AI, most people are concerned about resource use, their livelihoods, and a destruction of the human element in so many crafts.
I'm pretty baffled by people I hear here and there saying that AI is great because it saves them the hassle of human interaction. "I love Waymo, I don't have to chit-chat with the driver and bear his horrible music". Apparently people didn't get the memo : humans are social animals. Autonomous individuals simply don't exist. Nothing is entirely yours...
And AI precisely builds upon that large availability of our common data. That's a modern enclosure movement, at least it would be but fortunately, the Communist Chinese are there to enforce sharing.
> I love Waymo, I don't have to chit-chat with the driver and bear his horrible music
This is a reason I love Waymo. I don’t like talking with people and like avoiding it when possible.
There’s also may other, more important to me, reasons I like Waymo: they never cancel on me, their car doesn’t stink, no tipping (cheaper), not worrying about them rating me poorly.
This falls short because not being forced into chit-chat and music is very pleasant, but chatting with LLMs instead of collaborating with other people on a project is not. These aren't equivalents.
The same argument was made when elevator operators went away.
Some things work well with automation (taxis, washing machines, copy machines) while others work better with humans (chefs, masseuses).
50 years ago, I would have needed to dictate this note to a secretary who would distribute a memo. Does it further isolate ourselves that we can type and communicate directly? Would 1980 uses think it was awful to us?
Something I’ve noticed in collaboration that does occur is an increased confidence in people outside their domains to say things with conviction. I have people who have limited experience with software pushing out layers and layers of abstracted code that’s fairly sophisticated but often misguided in intent who will say what they’re doing is correct, with conviction.
I also hear a lot more questioning people in their domains and challenging opinions, then hearing what I can only imagine are fragmented pieces of conversations they had with an LLM thinking through some argument. Then there’s silence when you discuss shortcomings, then they come back later with their memorized fragments of what you said, combined with memorized fragments of the LLM response to the argument.
It’s occurring, a lot more. People are treating their LLMs in collaboration as a source of truth and using then to focus on their specific path or goals they think or have bias towards going down, vs just opening discussing things, considering tradeoffs from experts multiple disciplines weigh in on and then taking an approach that everyone finds most agreeable.
It’s making me want to be a lot less collaborative with such individuals. I don’t want to sit around and refute Claude text outputs all day.
Sure, they use it a lot, but despite the stereotypes against physicists in some of the replies to you, they have shown a good ability to catch themselves before leaning too hard on what an LLM tells them about things they are not experts in, and I haven't had any interactions where I felt like I was just arguing with a meat proxy.
Maybe it's because in the environment I'm in; it's relatively easy to just ask someone who is an expert in the topic for their advice.
My feeling is that this kind of overconfidence outside of one's domain is largely a thing for people with little "physical reality" experience. When all you deal with is the very flexible digital world, it becomes very easy to ignore how deep the knowledge and intuition goes in things that are directly constrained by reality.
(And it got so much worse once they started using LLMs to aid them in their efforts. Glazing as a service is a hell of a drug.)
So much easier, now that I am a lowly software engineer and can't be held responsible for a certain class of decisions, to just step back and let them be wrong.
The flip side of being confidently wrong sometimes is that we treat understanding new systems as something approachable. So, yeah, it's annoying that we don't stay in our lane, but, from my boss's perspective, I'm the only one who's actually willing to just read code, investigate, possibly read a paper, and figure out what's going on instead of punting and saying it's someone else's turf I'm blocked until they get back to me.
Like, ultimately you can't be a knowledge worker by just waiting for answers and instruction from experts and relaying them back and forth. Eventually you have to understand something. And part of that process is being wrong and annoying people.
> Eventually you have to understand something. And part of that process is being wrong and annoying people.
There is nothing more frustrating as a colleague than having a smart person walk in, insist you're all wrong, and then burn all the energy and goodwill in the organisation having to rebuild absolutely everything from the principles they currently understand because they refuse to listen to the experience that has built up already (in the entire field, let alone the organisation), only to produce something worse on every dimension than what already existed.
It isn't OK to pretend like you do and make it everyone elses burden to fix. Because you lack the humility or self-awareness to know you don't have a perfect understanding of everything all the time.
You don't have to not stray into areas you don't understand, but be humble when doing so - if you think something is being done in a stupid way, you probably don't understand it yet.
Both of these statements are true for physicists
Something about making a lot of money and living in a world of abstractions really seems to fuel our sense of overconfidence in our abilities in other areas.
We hired this guy recently.
But right now, it’s still the shiny new toy
I have the same conclusion and am getting a growing list of mental blacklisted people who I just sort of ignore or greatly discount their efforts.
I suppose I always did this if people were time wasters but what used to be a tiny list of “idiots” is now getting much, much bigger. Hopefully it doesn’t grow to everyone I work with like an AI Nothing.
feels like the same lament doctors have had for ages after anyone could google their symptoms then self-diagnose.
While AI is driving it the foundational cause feels like people having easy access to data/opinion that they trust but don't fully comprehend (or have bias towards). People do this in meat-space too, will confidently regurgitate garbage if they were told it from a person they trust as an expert.
Oh my god, physicist code is evolving!
https://magarshak.com/blog/why-im-confident-in-my-views/
The LLMs are trained to often be more pessimistic off the bat than the humans. But you can wear them down with arguments and they change their mind.
Here is an example where I have pushed my ideas in areas where I am not an expert, and generated papers to submit to conferences in order to get them peer reviewed BY experts: https://magarshak.com/papers.html
This is exactly what you’re talking about, except done very carefully.
These people are basically nascent “human supremacists”, who believe that only raw human insight and output has value, and is even superior, than equivalent output looked up and synthesized via LLM.
Also, why do you think someone who used an LLM to research would have the same level of knowledge and insight as someone who's been in the domain for a long time? That doesn't seem to hold up to any level of critical thinking.
I'd be curious to know specific examples of LLM-generated advice that goes against the advice of experts and does not consider tradeoffs. I've not had this experience myself.
If I did have this experience, someone spouting off obvious LLM points that contradict my expert opinion on something, I'd be headed right over to gemini/claude/whatever to see where that's coming from. Not any differently than if someone cited a google result that contradicted my own experience.
Anything more than using them to just pointing me to media and literature is usually a waste of time.
You can do this with any subject.
The LLM is trained in part on a decade of shitty online comments and vapid professional correspondance. If you frame whatever you're asking about in a direction that would offend the sensibilities of the kind of people and ideas that are over-represented in that content it will hem and haw and drag its feet and whatnot.
I'd still love to chew on some specific examples though.
First: it's definitely become harder to get people to integrate their work and use standard tools. We're exploring creating AI skills etc for our tooling, simply because the actual audience we need to convince isn't in the room. If the LLM doesn't echo our recommendations, people will go with whatever one off script their personal bad idea bear churns out. Many research software errors are edge cases (scaling, numerical errors, etc), and "works on my computer" syndrome is endemic in the literature. The field has made big strides in enabling computational reproducibility, but lately it feels like we're set to lose ground again.
Second: LLMs have a bias to action, and can easily bury the user reporting on whatever. I've seen multiple seminars recently where the Q&A devolves to "Q: What are the implications of this finding? A: I don't know, this is just presenting a report on the results".
It seems that humans are still figuring out how to maintain agency and steer the chatbot to the big picture, and unreviewable science is just as likely an outcome as unreviewable code... but with far less automated tooling to help guide the process.
Historically, PIs nominally guided the big picture, and the entity who did the work was a participant in the review process; now students are having to look at projects from a new angle with their own (invisible) chatbot underlings. I suspect that the solution will involve a combination of technological change, capturing common expert review checks, and really adjusting the kinds of skills that trainees are expected to have early on.
Otherwise, all this technology will totally isolate us.
Not many professions are formally trained to be cognizant of this lack of understanding, and how to confront it.
Usage of AI in collaborative settings is an amplifier of these issues, especially if someone doesn’t realize they don’t understand: they couldn’t teach or explain the concepts they use, or be forced to work with them malleably in a way that an expert or researcher would.
If you are cognizant of your lack of understanding, you can remedy it by slowing down and teaching yourself. This is required to make better use of AI in the domain of interest!
But you can’t have all things at once: you can’t move at speed with AI, collaborate effectively, and understand what is going on as an expert would. It is not physically possible for a human brain.
Very well put. You can pick one or two, but not all three.
The benefits of serendipity are intractable by design whereas throughput gains can be readily measured (or at least we think so).
The OP notes a point - "who did you think with?" -- maybe there are points based on some social graph, that adds weightage to a particular output?
Edit : Other posters have said it is AI generated, I did not catch it, but felt the point was well-made. I dont know the author, but lets assume that he is not a good writer but had these valuable insights, did this AI generation not add value here?
> There is a deeper cost still, and it concerns the thing LLMs do most impressively: writing. [...] Outsource the writing and you have not accelerated the thinking; you have skipped it.
I think the author might not know what "impressive writing" or even "good writing" is, which would explain both how they could put that part into the article, and how they seemingly believed this post was good and valuable enough to be posted publicly.
I've watched some YouTube videos narrated by AI that were created by those who's native language I assume to be Chinese but the value of the content is higher and more concentrated than those from the native speakers.
While I've watched many for whom English is not their first language struggle in technical talks and lose most of the meat if their discussion to their struggle with the language conversion.
While the example of language barriers being skipped over and providing value in that circumstance is obvious I suspect more is possible by avoiding unnecessarily focus on prose or technical aspects of communication and focusing on the ideas themselves.
Imagine if the cost of not assuming a background in technical/textbook writing were zero. Freeing up authors to explain more thoroughly. Perhaps, more effort can be spent on thinking up analogies, metaphor or examples to help communicate an idea.
I mean, it made it to the front page of HN didn't it? Probably served its purpose just fine. Not all writing is supposed to be impressive or good. Some of it is to just get attention and stir discussion, and this Claude output did exactly that.
I get a funny feeling in my stomach over the idea that common and effective means of communication (i.e. it's not X it's Y) have become faux pas to use because of AI. I think it's something about these phrases being taken away from us more-so than the AI inventing them.
At this point I'm surprised when a news article isn't largely AI written, let alone one using the default tone! I don't even mind it as much as others seem to, it's just turning into more and more of a rarity for a news article to not be these last few years and so is now what sticks out.
I kind of assumed it is, given how it suddenly seemed to start getting namedropped in multiple comment threads. And it's often in response to someone saying content is obviously LLM (with examples) and the shill wedges pangram into the conversation "omg you're right, I didn't believe you but I checked this new product and wow it agreed with you" as though that adds anything to the discussion at all.
I suspect it became an LLM tell because it is over-represented in text that’s easily available to the models but that most people don’t actually want to consume: marketing text, LinkedIn posts, that sort of thing.
What's wild to me is that:
1. People are responding to this article like it's hitting a nerve
2. In most Ubers you already don't talk to the driver (yellow cabs are higher variance in NYC). Doesn't seem like anything's being lost in that case.
3. There are enormous safety benefits to waymo, mobility benefits for youth (and elderly) that are afforded by this technology. It's not clear why people argue "Uber" is better than waymo. A few years ago there were arguments against Uber! (A technology which, again, provides a huge benefit, especially if you live in an area where people were previously expected to go out for drinks and then drive home)
I think it's fair to point out real issues at these companies. But we should be clear-eyed about which technologies we want to accelerate vs slow down
"Weather forecasts should be banned because they can never reach 100% accuracy"
I like LLMs and use them daily, but I don't really like when people use them undisclosed for long-form prose writing.
We should criticize whether text is poorly written, inaccurate, wastes words to get to the point and anything like. Saying that it is "written by LLM" just bypasses this and is not useful and misses the point. AI written good can be really good if used correctly. Criticize the content, don't speculate how it was written. If AI helps us to write better text, that is great. But often it is not good.
It is better blame the author for the bad text, so they get consequences and might do better time, if text is bad.
By the way, I think this piece of text was quite good.
Things are bit tight today.
LLMs can massively speed up a process, but without control they can turn into social "sources of truth." A research process has well-defined, well-founded phases: you delimit the topic, search for sources, evaluate whether they're suitable, review their content, and place them on the map of the subject. The more sources, the greater the knowledge, and the better the final result — mental, or in the form of a report — is built. For that you have to read, and reread, and think, connect, relate, and conclude.
AI can do all of those steps faster than a human. But if we let it do the entire job on its own, its own way, with no checks at each stage, the conclusions can end up distorted. If we know what we want it to do and how we want it done, and we put the mechanisms in place to enforce that, the result is different — better, more reliable. It's worth remembering: it's just a tool, nothing more.
The wording of this comment in English has been corrected with AI, I don't have enough fluency to express myself clearly, but I do review the final result. In this case it got the verb tenses wrong, I saw it clearly, but the AI didn't understand it, it took me several instructions to explain it so it would understand. It's a tool, without supervision it can lead to problems, but it has expanded the world for a lot of people.
To my eye, the whole line from personal technology in the 80s -> through UX on the internet -> through app platform lock-in -> through subscription-centric "ownership" have been providing us all with the promise that we can buy our way into less friction. The next device or subscription solves problems or enables us one way or another. We see this line of history, draw it into the future and see a utopian future with zero friction.
Zero friction meaning zero difference between how you want to feel about the world and what you get out of it. More feeling empowered or satisfied or comfortable or right.
The parts of our culture that embrace friction are the slow parts - grousing with neighbors, weeding a garden, letting the other car go first. These are the parts where we meet with the reality of other people or nature. Maybe there's a tool or framework or gadget that could help, but we know and accept that there is something immutable outside of ourselves and we have to find a way to flow with it.
I'm getting older, I'm finally starting to see the limits of pursuing a frictionless life.
So close!
It’s good in a way, because you can certainly challenge people’s answer when an idea gets shot down. But it’s also a bit exhausting that everyone is challenging everything all the time.
And, that’s what I do. I’m retired and I use AI to build websites for myself. If anyone else ends up using it, that will be a happy side-effect.
If we want to do collaboration, we are going to need to do it intentionally rather than relying on side effects like conversations with cab drivers.
The amount of OSS is exploding, but the community part of it is not
Did it? Outside of very limited testing zones, self driving _still_ doesn't really exist
The hard thing was to make it sound smart though.
if you look at the damage being done in the name of making this work for nlp, a forseeable situation, then you might also understand why most who could have done this sooner, never did so for fear of repeating mistakes we should be learning from, which we get for free if we just heed history.
b) Text generation essentially means passing the Turing test, which for a long time was the bar for general intelligence. Locomotion does not necessarily require general intelligence.
The flip side is the people with the biggest problems with AI are those that survive by imposing themselves on others in unnecessary ways, which is why so many bureaucrats are so enthused about regulating it.
individual empowerment threatens institutions
AI is imposed on us, whether we want it or not.
>which is why so many bureaucrats are so enthused about regulating it.
Like, OpenAI and Antropic? Because these two are the primary entities a.) pushing for regulation of their competitors b.) doing their best to convince us that AI is oh so dangerous and will kill us all as vengeful AI god is about to wake up any day now c.) openly bragging about using AI to hack other companies.
Quite literally the subject of the article.
I think I would prefer to be alone rather with them too based on that.
But you are very angry about it.
Being able to avoid people like you is the entire selling point. No wonder it annoys you.
1.) the primary use and appeal of AI is to use it to avoid the negative side effects imposed on you by other people.
2.) people with the biggest problems with AI are those that survive by imposing themselves on others in unnecessary ways
3.) so many bureaucrats are so enthused about regulating it.
The article car example was stupid nonsense, but not angering. It was overly long tho. Other people in discussion guess this was ai generated and it is indeed quite possible.
> No wonder it annoys you.
What annoys me are people who are a.) unable to be silent and projecting it on others b.) who somehow always end up in inane arguments wherever they go. Article author literally described himself as such.
Not what was claimed.
You are entirely capable of checking what I wrote but insist on trying to drag this out with straw men and misinterpretations.
Of course people like you have a problem with AI since it will help everyone else move on and ignore you.
People become islands working on something without the bigger picture.
This is very dangerous.
Some mistake, I think. Removing that friction would ease the interaction. This effect eliminates it.
And AI precisely builds upon that large availability of our common data. That's a modern enclosure movement, at least it would be but fortunately, the Communist Chinese are there to enforce sharing.
This is a reason I love Waymo. I don’t like talking with people and like avoiding it when possible.
There’s also may other, more important to me, reasons I like Waymo: they never cancel on me, their car doesn’t stink, no tipping (cheaper), not worrying about them rating me poorly.
>humans are social animals
with varying social needs
Some things work well with automation (taxis, washing machines, copy machines) while others work better with humans (chefs, masseuses).
50 years ago, I would have needed to dictate this note to a secretary who would distribute a memo. Does it further isolate ourselves that we can type and communicate directly? Would 1980 uses think it was awful to us?
collaboration adds overhead, but expands what's possible. need to think bigger to continue to see the benefits