The hallucination issue is genuine and worrying. When I asked ChatGPT (free version) a legal question which needed law reports for the answer (it was related to interpretation not statutes) I ended up wasting hours downloading cited reports, reading them, then word searching, and not one of them was relevant.
I suspect part of the problem of your test is the format of online data, particularly pdf documents. If you copy and paste complex pdfs like modern company reports it's often the case that columns and rows don't align in the way you would expect. I think that when LLMs have the ability to visually see and "read" documents in the same way a human does the results may be better. As for the raw RNS data, maybe the stock exchange is blocking LLM bots from scraping some or all of their data, or even deliberately confusing them. If LSE can't sell its proprietary data because LLMs have harvested it then that's going to hit their revenue.
More positively, today I asked ChatGPT (the free version) to answer some questions about Inheritance Tax Taper Relief, using examples, and it produced an excellent report, every bit as good as an accountant would provide, with tables, and when I checked back to the HMRC IHT manual I found it had got it all right.
So far as investment is concerned, one of the more cynical American investor newsletters I sometimes read is convinced AI is leading to a gross misallocation of capital and it will all end in tears.
I wonder if we will get start-ups starting to build niche or use case specific AI models soon. As you rightly say, so much depends on feeding the model good, structured, and comprehensive data. Then training a model to deal with the idiosyncrasies of a specific subject area, and then optimising the output to what's the most effective way.
There's so much that can automated by AI in the process of researching shares!
No-one has been talking about the cost of all the capex required, and whether AI models can deliver unit economics that are profitable. Reminds me of Pets.com; brilliant idea that works now in the 2020s, but back in the 1990s it had such terrible unit economics that of course it wasn't a viable business idea.
There are AI startups but I think what's more important is data owners who are using AI to build LLMs on top of their own proprietary data. LexisNexis, part of Relx (REL:LSE) are big on this - https://www.lexisnexis.co.uk/. In my career and retirement I have seen legal information change from libraries and office bookshelves which Henry II might almost have recognised after he introduced his reform of the law, through loose leaf binders (God I hated the job of keeping them up to date each month), to the very early electronic access to information which came in from 1985, to online search and word processors building documents, to what we have now which is quite soon going to make a big difference to how law is practiced. Just for fun I asked ChatGPT to draw up heads of terms for a Shareholders Agreement and it was very good. Same for accountancy. My best guess is that data owners will become VERY picky about who they allow to use their data, there might be scope for specialists who can sell AI systems which can sit on top of that data, but the mad rush to invest in anything with AI in the title will end in tears.
OpenAI employees testing the new AI model ‘Orion’ have disclosed that while its overall performance is better than OpenAI’s existing models, the rate of improvement is much lower than it’s been in past upgrades—like the jump in improvement from GPT-3 to GPT-4, for example—and it might not be consistently better in specific areas like coding and completing complex reasoning tasks.
🔑 Key Points:
The issue is (across the entire industry) there’s a lack of new, high-quality, diverse training data to expand the AI model's understanding, so to overcome this, OpenAI has formed a foundations team.
The team is trialing using synthetic data—artificial data produced by AI models—alongside real-world data, as it could introduce new layers of variability and nuance, improving the model's ability to handle complex scenarios.
Post-training, they're also planning to use techniques like reinforcement learning and fine-tuning on specific tasks, to address performance gaps that real-world and synthetic data isn’t enough to fill.
🤔 Why you should care: The slowdown in AI model improvement due to a lack of untapped, real-world, quality training data is affecting the entire AI industry, and the limitations of this data shortage are raising concerns over the future of AI advancements and the ability of AI models to reach their maximum potential, so many will be looking to see if OpenAI’s approach of using synthetic data and post-training techniques will work and continue to drive the AI industry forward.
That is interesting! Looks like the industry is hitting a wall? Until they find the next big step-change?
FWIW I am definitely looking forwards to valuable data owners releasing their own AI solutions. I'm also on the lookout for any listed data owners that will be super valuable...
Thanks for the experiment.I must be an anomaly as ive never tried chatgpt myself yet, although i find the 'ai overview' section, that regularly pops up at the top of the search page when asking a question on google now quite useful.I'll have to have a play with Perplexity, even noting its failures.I'm always behind on adopting new tech.But i get there eventually... years later!I do have a smartphone now.
The hallucination issue is genuine and worrying. When I asked ChatGPT (free version) a legal question which needed law reports for the answer (it was related to interpretation not statutes) I ended up wasting hours downloading cited reports, reading them, then word searching, and not one of them was relevant.
I suspect part of the problem of your test is the format of online data, particularly pdf documents. If you copy and paste complex pdfs like modern company reports it's often the case that columns and rows don't align in the way you would expect. I think that when LLMs have the ability to visually see and "read" documents in the same way a human does the results may be better. As for the raw RNS data, maybe the stock exchange is blocking LLM bots from scraping some or all of their data, or even deliberately confusing them. If LSE can't sell its proprietary data because LLMs have harvested it then that's going to hit their revenue.
More positively, today I asked ChatGPT (the free version) to answer some questions about Inheritance Tax Taper Relief, using examples, and it produced an excellent report, every bit as good as an accountant would provide, with tables, and when I checked back to the HMRC IHT manual I found it had got it all right.
So far as investment is concerned, one of the more cynical American investor newsletters I sometimes read is convinced AI is leading to a gross misallocation of capital and it will all end in tears.
I wonder if we will get start-ups starting to build niche or use case specific AI models soon. As you rightly say, so much depends on feeding the model good, structured, and comprehensive data. Then training a model to deal with the idiosyncrasies of a specific subject area, and then optimising the output to what's the most effective way.
There's so much that can automated by AI in the process of researching shares!
No-one has been talking about the cost of all the capex required, and whether AI models can deliver unit economics that are profitable. Reminds me of Pets.com; brilliant idea that works now in the 2020s, but back in the 1990s it had such terrible unit economics that of course it wasn't a viable business idea.
There are AI startups but I think what's more important is data owners who are using AI to build LLMs on top of their own proprietary data. LexisNexis, part of Relx (REL:LSE) are big on this - https://www.lexisnexis.co.uk/. In my career and retirement I have seen legal information change from libraries and office bookshelves which Henry II might almost have recognised after he introduced his reform of the law, through loose leaf binders (God I hated the job of keeping them up to date each month), to the very early electronic access to information which came in from 1985, to online search and word processors building documents, to what we have now which is quite soon going to make a big difference to how law is practiced. Just for fun I asked ChatGPT to draw up heads of terms for a Shareholders Agreement and it was very good. Same for accountancy. My best guess is that data owners will become VERY picky about who they allow to use their data, there might be scope for specialists who can sell AI systems which can sit on top of that data, but the mad rush to invest in anything with AI in the title will end in tears.
Sorry to hog your comments but this might be important. It's certainly relevant.
https://aitoolreport.beehiiv.com/p/openai-in-data-crisis
OpenAI employees testing the new AI model ‘Orion’ have disclosed that while its overall performance is better than OpenAI’s existing models, the rate of improvement is much lower than it’s been in past upgrades—like the jump in improvement from GPT-3 to GPT-4, for example—and it might not be consistently better in specific areas like coding and completing complex reasoning tasks.
🔑 Key Points:
The issue is (across the entire industry) there’s a lack of new, high-quality, diverse training data to expand the AI model's understanding, so to overcome this, OpenAI has formed a foundations team.
The team is trialing using synthetic data—artificial data produced by AI models—alongside real-world data, as it could introduce new layers of variability and nuance, improving the model's ability to handle complex scenarios.
Post-training, they're also planning to use techniques like reinforcement learning and fine-tuning on specific tasks, to address performance gaps that real-world and synthetic data isn’t enough to fill.
🤔 Why you should care: The slowdown in AI model improvement due to a lack of untapped, real-world, quality training data is affecting the entire AI industry, and the limitations of this data shortage are raising concerns over the future of AI advancements and the ability of AI models to reach their maximum potential, so many will be looking to see if OpenAI’s approach of using synthetic data and post-training techniques will work and continue to drive the AI industry forward.
That is interesting! Looks like the industry is hitting a wall? Until they find the next big step-change?
FWIW I am definitely looking forwards to valuable data owners releasing their own AI solutions. I'm also on the lookout for any listed data owners that will be super valuable...
Thanks for the experiment.I must be an anomaly as ive never tried chatgpt myself yet, although i find the 'ai overview' section, that regularly pops up at the top of the search page when asking a question on google now quite useful.I'll have to have a play with Perplexity, even noting its failures.I'm always behind on adopting new tech.But i get there eventually... years later!I do have a smartphone now.
That’s a really interesting experiment. Thank you for sharing.