#202

I think the role of dogma or conventional wisdom has always being reasonably strong. The conventional wisdom only ever changes very slowly. For example I think the “reprogramming” of cells and the link to methylation of dna is not really substantiated. Where it has had an effect I think that is through SOX2 stimulating solid autophagy.

The idea that somewhere in the cells is a backup program, but we don’t know where it is stored does not seem to fit with the general level of detailed knowledge of the molecules in cells.

1 Like
#203

When you do this, it’s really hard to put the “intelligence” in AI. Most bots I’ve tried are excellent data aggregators, but only basic at mimicking intelligence and creativity. They rarely generate truly novel insights on their own. However, they can save you a lot of time and a thoughtful user can still use them as a useful tool to expand their own thinking. I also agree they tend to mirror and flatter your position which can feel a bit psychopathic or narcissistic and that can be misleading if you’re not aware of it.

5 Likes
#204

My next attempt is to include a prompt like this: Do not mirror or flatter my position!

3 Likes
#205

I think many folks parrot what they read and hear about the fallacies of AI LLM models without just taking the time to dig deeper. On objective questions in science and medicine many of these LLMs out perform most PhDs in expert subjects. An LLM optimized model specific to deep medical diagnosis will out perform most humans. Here is the current leader board on Hugging Face. https://huggingface.co/spaces/openlifescienceai/open_medical_llm_leaderboard

1 Like
#206

Honestly, that just means the PhDs were not that capable. But I agree that the prompts you use makes a huge different. The more specific you can be, the better. If you can provide the model with specific data to work with, you get better outcomes. But I have found, in my own research field, ChatGPT 5 Pro (the $200/month version) takes 20 minutes to think of an answer, and still produces wrong information, or states a fact linking to a paper, and the paper doesn’t support what it wrote. I have seen that, with my own eyes, in September and October 2025.

I’m sure they will continue to get better though. I’m not an AI hater at all.

(Edit: nothing I can run locally can outperform ChatGPT 5)

3 Likes
#207

Exactly my experience as I’ve written before, many times. In an area that I am very familiar with, I can clearly see the limitations of (current) AI platforms. The less you know, the more impressive they seem, but the more you know, the more disgusted, appalled and disappointed you are.

I think the hyping of current capabilities of AI are laughable. I can even forgive not finding obscure information (although you’d think that trawling through giant databases should be the primary advantage of these systems :roll_eyes:), but what disgusts and enrages me is when I point to a specific paper or information, the AI is unable to take it onboard or modify its conclusions. There are some very prominent deficiencies of AI platforms as they stand at this point in time. They are unable to sensitively assess the credibility of any given source of information, beyond say impact factor, journal quality, research team quality and such basics - it cannot look at a paper and assess where it fits in the matrix of extant knowledge of this area, it doesn’t have a “BS detector”, so any paper that is a single contrary data point is discarded instead of being seen as critically important to ascertain the credibility of it because it could be a fluke/wrong or a vital clue that the common consensus needs revision. The end result is a slop confirmation of the consensus, where the number of sources determins validity, even if the majority are just following and citing each other. Here you can see why it’s great at regurgitating textbooks and passing exams, but useless in original insight.

Perhaps one day these deficits can be corrected, but I rather suspect this will necessitate a different architecture of AI reasoning. When that might happen, I have no idea. But as of today, IMHO, any use of AI needs very strict application criteria, and the outsize hype about its capabilities is laughably unjustified.

#208

Probably, when an AGI will rise. Or better, an ASI. Then we’ll see if the reality fits the hype.
I’m a little more optimistic than you are though. Even with all the drawbacks you mention, in my specific field of knowledge, I still can find that the AIs provide some valid aspects to reason about. And the progress in the last two years has been indisputably huge.

#209

Exactly my experience. Wonderful for scanning databases but you need to go to the sources thereafter. What is maddening is that sometimes the DOI links to the sources provided are wrong and bring you to a wrong article !

#210

If that’s happening, you are still using the shittest non-reasoning types of LLM. They’re basically trying to recite the DOI from memory. You want to use something that actually goes and checks the sources itself.

One issue is that a lot of websites have now blocked the LLMs from accessing them. So, for scientific papers, a lot of ChatGPT is getting routed through Researchgate, university websites etc, because it can’t directly access papers from some publishers. However, it will sometimes try to bluff and tell you about something where it’s only read the citation (title, authors, year etc) and not the content.

This will be a dream, but I think it will be possible in the future. It takes a huge amount of compute to process that much data at once, and the amount that we can ever access as customers is tiny. Every model you access is throttled and not using the full capacity.

For example, the free version of ChatGPT gives you access to a model with 30-40B parameters. ChatGPT 4o gives you 175B. ChatGPT Pro ($200/month) gives you 1.5T, and internally they apparently have models with up to 10T. So the people asking free version ChatGPT questions are working with something less than 0.3% of the potential capability.

There is also a limit to the working memory on the models which are provided to us. So at some point, the model is summarising the content of the conversation. GPT5 can hold around 128-256k tokens (less than 200 pages of text) in memory. So if you ask it to analyse 20 documents, it’s going to internally summarises them, then works with summaries of summaries etc - the analysis get shallower as it holds more information. However, at the moment, that is hard built into the GPT model, and no amount of compute will fix that. So you’re right that it needs new architecture of some sort.

Out of curiosity, if you search “Rapamycin” on Pubmed, you get around 60,000 results. That’s a lot, but not an obscene amount of data, especially if you get rid of a bunch of duplicates, reviews, irrelevant studies etc. If you picked the top 10,000, that would be 100m tokens, so the current model is around 400x too small. Wonder how long it will take to bring it up to the size needed to do that sort of thing.

#211

If that’s happening, you are still using the shittest non-reasoning types of LLM
It actually happened under GDP-5, 20 $ version.

#212

Sure. But depending on the prompt, you can get outsourced to GPT-4o or a lower tier model to answer the question.

#213

I used Google’s new customer service AI via chat today and I am thoroughly impressed!

Here is what I liked:

  1. First of all, there was no waiting in a queue for an agent. Talking to the AI was almost instantaneous. No waiting phone purgatory with elevator music is a big advancement!

  2. Second, AI was able to research the problem and correct it almost instantly. No more waiting while they pull up your record and examine the problem. Again, a major time saver!

  3. Third, the customer service AI communicated effectively and cordially. No more need to be frazzled by a difficult accent or an impolite human.

  4. Fourth, I’m sure Google doesn’t have to pay their AI bots much to staff the service center!

It’s great having this level of service, but, frankly, a lot of jobs have now been axed because of this. I have the sinking feeling AI will cut far more jobs than it generates, and that doesn’t bode well for your average human without something to replace these jobs like UBI.

I can see this service being used to replace radiologists. I seriously wonder why this hasn’t happened yet! It should also replace GP visits if an AI can analyze your scans and bloodwork, it should be able to replace most functions of a GP and prescribe medications.

The future is amazing. However, I am glad I am at the tail end of my career and I worry a bit about my sons who will enter the job market in 6-10 years time. Will the competition for jobs be too fierce?

3 Likes
#214

This is an excellent report because speaking (or typing) to customer service is the bane of my existence!

And yes, I’m glad I’m no longer in the work force because I’d worry, too!

I just heard my first story of someone I know being affected. He is a music composer for film, and he is already losing jobs to AI… it’s bad enough, he is now looking for a new career.

1 Like
#215

Anyone here who has had their DNA sequenced - please sign up for this during the beta period and test it out and report back.

The company website:

SuperAgers: Genetic signatures of exceptional longevity: a comprehensive analysis of coding region single nucleotide polymorphisms (SNPs) in centenarians and supercentenarians
#216

Current AI Models have 3 Unfixable Problems (Sabine Hossenfelder)

AI Summary:

Challenges in Achieving Artificial General Intelligence

  • The pursuit of artificial general intelligence, or AGI, remains a significant challenge, with many believing that current AI models will eventually reach this level of intelligence with more time.
  • The speaker argues against this notion, stating that the current AI systems, primarily based on deep neural networks, have inherent limitations that prevent them from achieving AGI.
  • Current AI models include large language models and diffusion models, which are designed for specific tasks such as text generation or image creation, but lack the versatility required for general intelligence.
  • These models are purpose-bound and trained to recognize patterns in specific types of data, making them incapable of abstract thinking that is necessary for general intelligence.

Limitations of Current AI Models

  • One major limitation of large language models is their tendency to produce hallucinations, which are responses that have no factual basis and can mislead users.
  • Hallucinations occur when a model generates answers that are not found within its training data or are based on infrequent occurrences in the data.
  • The speaker notes that these models do not actually search their training data for answers but instead rely on probability to generate responses, leading to inaccuracies when the probabilities are low.
  • A recent OpenAI paper suggests addressing hallucinations by rewarding models for acknowledging uncertainty, but this proposal has faced criticism for not aligning with user expectations for accurate responses.
  • The speaker believes that while hallucinations may never be completely resolved, a low frequency of such occurrences could still be acceptable in practice.

Prompt Injection and Its Implications

  • Prompt injection refers to the act of altering the instructions given to an AI model through user input, which can lead to unintended behaviors.
  • An example of prompt injection includes asking a model to ignore previous instructions and perform a different task, such as writing a poem about spaghetti.
  • The speaker asserts that prompt injection is a significant challenge for large language models since they cannot differentiate between genuine instructions and user prompts that should be acted upon.
  • While there are methods to mitigate prompt injection, such as requiring specific formatting or screening inputs, the speaker believes these models will remain unreliable for many tasks due to this vulnerability.

Generalization Limitations of AI Models

  • Current AI models struggle with out-of-distribution thinking, meaning they cannot generalize effectively beyond their training data.
  • This limitation is particularly evident in image and video generation, where models perform well within familiar examples but fail to create accurate outputs for novel requests.
  • For instance, attempts to generate imaginative scenarios, such as a video of Jupiter using a vacuum cleaner to remove asteroids, often result in poor-quality outputs.
  • Similarly, large language models excel at tasks like summarizing and drafting but falter when asked to create something entirely new, which poses a significant barrier to their application in scientific research.

Future Directions for AI Development

  • The speaker expresses skepticism about the current generation of generative AI models, emphasizing their inability to perform abstract reasoning, their vulnerability to prompt injection, and their poor generalization capabilities.
  • Companies like OpenAI and Anthropic may face challenges if they rely solely on these models, which are unlikely to fulfill the anticipated revenue that justifies their high valuations.
  • To move forward, there is a need for the development of abstract reasoning networks capable of processing various types of input without relying on language.
  • Neurosymbolic reasoning is identified as a promising approach, although the speaker humorously suggests that human intelligence may decline to match the capabilities of AI.

Conclusion and Call to Action

  • In conclusion, the speaker emphasizes the importance of addressing the limitations of current AI models while also promoting the use of services like Incogn for privacy protection.
  • The audience is encouraged to utilize the provided discount code for Incogn, promoting the service as a beneficial tool for managing personal data privacy.
  • The speaker expresses gratitude to the audience for their attention and indicates a follow-up engagement in the future.

Thank you RapAdmin for providing an AI summary of the video, as I myself never do so. One of the many reasons I never do so, is because AI seems unable to select the important points vs unimportant points. This can be seen in the current summary, where the AI is unable to realize that the presenter included sponsored content, which should be ignored, and instead here it’s treated as if it was germane to the video. This is of course just a particularly blatant example of the utter failure that is AI with the ironic twist that the video deals with the inherent limitations of AI and why these are not really fixable. And here comes AI with a summary proving that point in a spectacular fashion. Oh the irony!

I understand the motivation behind providing these summaries and the genuine need to save time and overcome the challenges of sometimes not having the opportunity to watch a long audio-visual presentation, but these AI summaries in my experience are almost always misleading and substantially lacking. YMMV.

1 Like
#217

Absolutely agree with this. Glad someone explained so simply.

#218

Claude enters the lab: Anthropic bets big on life sciences

AI group tailoring its chatbot to integrate with biomedical databases and coding tools

Artificial intelligence start-up Anthropic is tailoring its Claude chatbot to researchers and life sciences companies, as AI groups race to create specialised applications from the technology.

The San Francisco-based group said on Monday it is integrating Claude into tools that scientists already use, including lab management systems, genomic analysis platforms and biomedical databases, to tackle time-consuming tasks such as data analysis and literature review.

Anthropic, which was valued at $170bn in September, said drugmaker Novo Nordisk has already used its AI model to cut clinical study documentation from more than 10 weeks to 10 minutes, while drug developer Sanofi said the majority of its employees use Claude every day.

The move comes as tech groups are spending billions of dollars on AI products and models, believing the technology can benefit a range of industries from healthcare to energy and education.

História completa: Claude enters the lab: Anthropic bets big on life sciences (FT)

anthropic
anthropic746×446 37.7 KB

More: https://x.com/AnthropicAI/status/1980308459368436093

2 Likes
#219

Do we have an idea of the price per seat?

#220

I’m not sure it will be a blessing or a curse when INDs from companies are going to be written by LLMs. And even crazier, there’s a decent chance that the FDA is going to be using LLMs to review the applications.

#221

I have tried it out for a monthly payment of GBP18. I posted the result on the nucelotides topic. When I can some time I will do some head to heads between chatGPT and Claude.

1 Like