๐ ๐ช๐ต๐ฒ๐ฟ๐ฒ ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐น๐ฎ๐๐ ๐ฎ๐ฟ๐ฒ ๐๐ฎ๐ธ๐ถ๐ป๐ด ๐๐ : ๐ฏ๐ ๐ฎ๐ฌ๐ฎ๐ด, ๐๐ ๐๐น๐๐๐๐ฒ๐ฟ๐ ๐๐ถ๐น๐น ๐ฟ๐ฒ๐ฎ๐ฐ๐ต ๐๐ต๐ฒ ๐ฝ๐ผ๐๐ฒ๐ฟ ๐ฐ๐ผ๐ป๐๐๐บ๐ฝ๐๐ถ๐ผ๐ป ๐ผ๐ณ ๐ฒ๐ป๐๐ถ๐ฟ๐ฒ ๐ฐ๐ผ๐๐ป๐๐ฟ๐ถ๐ฒ๐
Reminder : โScaling lawsโ are empirical laws saying that if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
๐ As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So letโs take this progress a bit further:
- 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
- 2024: today, companies start training on much larger clusters like the โsuper AI clusterโ of Elon Muskโs xAI, 10^27 FLOPS, $1B
- 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
- 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries?
โก๏ธ Three key ingredients of training might be a roadblock to scaling up :
๐ธ Money: but itโs very unlikely, given the potential market size for AGI, that investors lose interest.
โก๏ธ Energy supply at a specific location
๐ Training data: weโre already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
๐ค Iโd be curious to hear your thoughts: do you think weโll race all the way there?
Reminder : โScaling lawsโ are empirical laws saying that if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
๐ As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So letโs take this progress a bit further:
- 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
- 2024: today, companies start training on much larger clusters like the โsuper AI clusterโ of Elon Muskโs xAI, 10^27 FLOPS, $1B
- 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
- 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries?
โก๏ธ Three key ingredients of training might be a roadblock to scaling up :
๐ธ Money: but itโs very unlikely, given the potential market size for AGI, that investors lose interest.
โก๏ธ Energy supply at a specific location
๐ Training data: weโre already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
๐ค Iโd be curious to hear your thoughts: do you think weโll race all the way there?