How does ChatGPT know stuff? And why is it important?
Christopher Martlew • Feb 07, 2023

Why is it important?

Google reportedly issued a “code red” to emphasize the existential threat that ChatGPT (together with Microsoft) may form for their core business of  search.

Microsoft is out to eat Google’s lunch.

This new application of AI is a gamechanger at the magnitude of the PC/Windows, the internet/www and the iPhone/mobile.

GPT is important because there are unlikely to be more than a handful of these massive AI engines on the planet. They are simply too big, too complex and too expensive to build and run.

A future clutch of AI titans may well form the backbone of a global AI infrastructure — in rather the same sense that network protocols, email, the web and social media defined our internet experience of the last 25 years.

So the battle is on to get into this space and bigtech will be spending gazillions to catch up with ChatGPT. Indeed, Google has just announced its own chat engine called Bard. It will be interesting to see whether Bard matches ChatGPT’s range and depth.

Both ChatGPT and Bard are based on the same Transformer (the ‘T’ of GPT) technology. The transformer architecture originated in Google’s labs (making it rather sour for them that OpenAI is now eating their lunch) and was published in a paper Attention is All You Need (Vaswani et al, 2017).

CEO of OpenAI, Sam Altman says (in an interview with Reid Hoffman ) that a lot of value will be created in a new ‘middle layer’ on top of GPT. We’re already seeing a growing ecosystem of applications built on the ChatGPT foundation — some of these will be bandwagon-jumpers, others may be great value creators.

What might the new middle layer comprise? In short, almost anything a human can do: computer programming, medical diagnosis, biotech, financial advice, share price guidance, customer service chatbots (that actually work) and answering exam questions.

Microsoft will be launching numerous features this year based on its cooperation with OpenAI. Like automated meeting minutes and action lists (how awesome is that?), or dynamically generated images for PowerPoint based on OpenAI’s Dall-E (hopefully better than the image below).

The cooperation with Microsoft will almost certainly consolidate ChatGPT’s position as one of the global leaders in the field. Not to mention the involvement of OpenAI’s sponsor Elon Musk — although he has been backing off lately saying that OpenAI was started as open-source & non-profit. “Neither are still true.” Could be a fly in the ointment there — never underestimate Mr. Musk.

The connection with Microsoft is the existential threat for Google — overlay ChatGPT onto Teams, Office365, Bing and Microsoft’s global reach and Google may rightly be a little worried.

Google made its name (and became a verb) by offering a simple, clean web page with a search box in the middle. This was in stark contrast to the cacophony of other search engines with flashing, ad-covered pages. History may be repeating itself.

Image generated by DALL-E by OpenAI — Shutterstock (cover image) doesn’t need to worry just yet — or maybe…?

How does ChatGPT know stuff?

Well…health warning, often it doesn’t, and it can get things spectacularly and hilariously wrong. Notwithstanding its ability to pass MBA and other university-level exam questions.

But it can be a lot easier and more user-friendly than old-fashioned search with ads. Often 90% accurate is good enough as a starter.

GPT stands for Generative Pre-trained Transformer. Generative means it can generate reasonable human-like responses. Pre-trained means it learns only relatively little in real-time — most of what it knows is pre-trained (or pre-learned). The transformer architecture means it is pretty good at translating what it “knows” into human language, and vice versa, including language translation.

ChatGPT is a conversational interface built on an underlying engine called GPT-3. GPT-3 is a Deep Neural Network. As such, it mimics the human brain. It has ‘artificial neurons’ which create ‘artificial intelligence’. The artificial neurons are organized into layers, and each layer processes information in a similar way that neurons process information in our brains.

Neural networks are trained by passing data through learning algorithms. These algorithms switch neurons ‘on’ or ‘off’ and create links between the neurons. As with our brains, the number of neurons is not the driving factor behind intelligence, it’s the links between the neurons.

The way in which a pathway of neurons and links is formed is called a parameter.

A parameter is a weight or bias in the learning model. It’s the parameters that are adjusted (or ‘learn’) by processing large amounts of data. The more data, the more parameters and the better the engine’s performance and accuracy can be.

Reportedly, GPT-3 has 175 billion parameters. Currently, there is no other model close to that number. And the rumours are that the next version (GPT-4 in 2023?) will have up to a trillion parameters. Bard is reputed to have 137 billion parameters.

Most of GPT’s knowledge recipe comes from Common Crawl (basically, that’s almost everything on the internet). But ‘everything on the internet’ is not very high quality, so it is augmented by WebText2 based on Reddit recommendations (so curated to some extent). Then add in two sets (corpuses) of books (rather cryptically known as Books1 and Books2). Leave to simmer for a few weeks and then add in the whole of Wikipedia (in English).

Following that, the model was refined by human beings on typical questions and answers.

Maybe not surprising that the machine reflects the inputs it’s trained on and has a bias towards the English language and US/Anglo-Saxon culture.

Learning takes a while, and the current version was trained on data up to June 2021, so the model is not good on current affairs. Bard is likely to be more real-time based on Google’s underlying search and web crawl technology.

Next possible gamechanger? Look out for new entrants and a successor to the transformer protocol. Watch this space, but don’t hold your breath on the latter.

Having said that, and to paraphrase Bill Gates, we tend to overestimate the short term impact of new technology, and underestimate the long term.

If you’d like to join the conversation please share via the clapping-hands button below.

Also at:

#OnBeingAgile #mindoftheorg


How does ChatGPT know stuff? And why is it important? was originally published in On Being Agile on Medium, where people are continuing the conversation by highlighting and responding to this story.

01 Mar, 2024
OK…sticking my neck out a little, but there’s a lot going on. And there are one or two stand-out items for 2024 on our forecast bingo card: AI and HR. We certainly have a few wildcards in play in the Middle East and Ukraine, plus a plethora of ‘democratic’ (some more than others) elections across the world. Wildcards (and potential black swans) notwithstanding, and steering clear of politics, wars and religion, here’s my clutch of 24 predictions, mini-rants, speculations and contemplations for 2024. 1. HR: Insofar as Agile and Digital Transformation programs didn’t kill hierarchies and silos in organizations, AI (AGI, ML) will. HR teams will face unprecedented pressure as the labour market remains tight and AI adoption forces the pace. 2. HR (again): will adapt by (an even greater) focus on internal marketplaces, AI usage and more ‘make’ not ‘buy’ through internal skills development. 3. HR (again): Will move out of their comfort zones in re-imagining support for IT and other communities who are going 'radical' on Agile and upending traditional management structures. 4. Artificial Intelligence will accelerate towards full human-level AGI by 2028. Regulators will not be able to keep up. 5. Apple will release AI (ML) integrated across all its ecosystem but centered on the iPhone. 6. AI will impact the Legal profession in ways that we’re only just starting to comprehend. There will be some casualties amongst the laggards. 7. Despite the darker sides of homo sapiens on our planet, people will remain broadly optimistic that 2024 will be better than 2023. (Source: Ipsos https://www.ipsos.com/en/ipsos-global-predictions-2024 ). 8. NASA will land multiple vehicles on the moon. 9. India’s space industry is on a roll. Its agile space industry will flourish with dozens of new space start-ups added to the 54 created in 2023. 10. Spending on Cloud will increase by 20% to $670 billion (Gartner). 11. The Chinese population will continue its inevitable decline towards halving its current number by 2099. 12. The New York Times case against OpenAI and Microsoft will drag on. But the first fines will land on the doormats of LLM owners for privacy violations. 13. >90% of software engineers will use AI to generate code thereby increasing productivity. 14. 98% of creative workers will use Generative AI to create reports, emails, slides, images and ideas. 15. GenAI will prove a headache for cybersecurity staff trying to keep corporate data safe, while unemployment in cybersecurity remains zero. 16. AI will increasingly be used to generate legal contracts – and even negotiate them. 17. Machines will order from machines, or at least influence human buying decisions. Potentially impacting trillions of dollars of trade by 2030 and making 20% of human-readable websites obsolete. (Source: Gartner https://www.gartner.com/en/information-technology/insights/top-technology-trends .) 18. The engines of the global economy will include Foshan (China), Surat (India) and Kumasi (China). Most in the West have never heard of these places. (Source: McKinsey at https://www.mckinsey.com/mgi/no-ordinary-disruption .) 19. Elon Musk will hang on to X but raise additional capital as he takes it in the direction of WeChat. 20. The world’s largest producer of oil will be…the US. 21. 90% of all mobile phones and the majority of IoT devices will be powered by architectures developed by UK-based firm ARM. 22. 100% of all high-end chips will be forged on machines built by ASML of The Netherlands. ASML will roll out more of its ‘High NA’ machines – each one larger than a truck and costing $300M a pop. But will ship less machines overall than in 2023 and none of the High NA will go to China. 23. Economic acceleration will be 10X faster than the Industrial Revolution and 300X the scale. (source: McKinsey https://www.mckinsey.com/mgi/no-ordinary-disruption .) 24. The top 10 advertising agencies will spend a combined $50 million to build custom AI solutions that enable their clients to scale personalized marketing campaigns and brand experiences. (Source: Forrester https://www.forrester.com/press-newsroom/forrester-predictions-2024/ .) Stay safe. Have a great 2024! “You can’t connect the dots looking forward; you can only connect them looking backwards” – Steve Jobs. Image (c) Shutterstock. Forecasts or expectations expressed in this piece may contain inaccurate forward-looking statements and are not intended as investment advice. Also at: amazon.com | amazon.co.uk | bol.com | blog #OnBeingAgile wn from a different source.
by Christopher Martlew 11 Mar, 2023
On Strategy, Mission and Purpose (Ithaca — in Greek Mythology the island home of Odysseus).As you set out for IthacaHope your road is a long oneFull of adventure, full of discovery.Fear, Uncertainty and Doubt —Don’t be afraid of themYou’ll not find those on your wayAs long as you keep your thoughts raised highAs long as a rare excitementStirs your spirit and your bodyFear, Uncertainty and Doubt — you’ll not encounter themUnless you bring them along inside your soulUnless your soul puts them in front of youHope your road is a long oneMay there be many summer mornings when,with what pleasure, what joy,you enter harbours you’re seeing for the first time;may you stop at many trading stationsto buy fine things,and may you visit many citiesto learn and go on learning from their scholars.Keep Ithaca always in your mind.Arriving there is not the goalSo don’t hurry the journey at all.Better if it lasts for years,so you’re wealthy with all you’ve gained on the way,not expecting Ithaca to make you rich.Ithaca gave you the marvellous journey.Without her you wouldn’t have set out.Source: Excerpted and (shamelessly) adapted from: C. P. Cavafy, “The City” from C.P. Cavafy: Collected Poems. Translated by Edmund Keeley and Philip Sherrard. Translation Copyright © 1975, 1992 by Edmund Keeley and Philip Sherrard.On Strategy, Mission and Purpose (Ithaca — in Greek Mythology the island home of Odysseus). was originally published in On Being Agile on Medium, where people are continuing the conversation by highlighting and responding to this story.
Share by: