The next step in the AI big model may be the cheap solution of Google's early years

Source: Wang Jianshuo

Author: Wang Jianshuo

The appearance of Google's early servers has been lingering in my mind for so many years, and it has become a benchmark for my understanding of technology and startups.

Here's one I saw at a computer museum in Silicon Valley in 2007:

In the photo below, the green and white line is the reset line, which is connected to a button, and the other line is the hard disk light of a normal PC. Four such buttons and lights are fixed directly on a cardboard with four screws.

There is nothing inserted in the graphics card position, and the other expansion slots are also empty.

There is only one network card and network cable.

CPU is Ben II.

This is what the whole motherboard looks like.

This HP Switch has a total of 80 incoming lines.

This cabinet has a total of 20 floors. Each shelf is covered with a layer of pine boards, on which four identical main boards are placed, and four hard drives are placed in the middle.

There is also an existing one in Google's visitor center.

In the early days of the Internet, very quickly, search became a useful and promising thing. At that time, the monopolized search engines were Lycos, AltaVista, InfoSeek, etc., which was very similar to the recent large-scale model companies fighting for hegemony.

But similar to today's large model companies, these search companies use Sun Micro's system and high-end servers such as HP, which have first-class stability and amazing cost. Running a search engine was expensive business back then. As the traffic continues to rise, the cost also rises alarmingly. At the same time, due to the limitation of computing power, their search is still a basic search, which is the reverse index table of text, and the search effect is average.

Google originally thought of the PageRank algorithm, which is to calculate the importance of a web page based on the weight of links from other web pages. This is a good idea, but it requires a lot of computing power to realize it. This process is basically similar to the current large model to calculate the vector of text. If I want to know the weight of a webpage, I need to read the whole web to see which other webpages point to this webpage, and the weight of these webpages, and the weight of these webpages needs to be calculated again by such logic, which is almost an endless loop Same computing power requirements.

Google's solution did not buy a high-end server costing tens of thousands of dollars from the only correct host manufacturer at the time, but put four small motherboards on a piece of cork paper, then tied a hard disk, plugged in a network card, and it was over. up.

Obviously, this kind of stability is far different from the manufacturer's mainframe. Therefore, Google uses software to create a file system distributed by Google File, allowing files to be rewritten in multiple places. If any hardware is broken, the data can be rebuilt in other places immediately, so that you rush over and smash a few "Little computers" are not affected. By adding our own MapReduce framework, computing can be distributed (map) on these small computers, and then the results are aggregated (Reduce), so that the computing power of so many computers can be added together without using one or several computers. A very powerful computer.

In short, after such a lot of tossing, because of cheap hardware, sufficient computing power, and cheap storage, Google is enough to support the huge computing power consumption of PageRank, and very quickly defeated the giant at the time from an unknown small station in Stanford. , became the Google of today. Therefore, from a certain point of view, the huge cost advantage of hardware in exchange for software is a factor that cannot be ignored in Google's early success.

Will this history inspire the current AI landscape?

OpenAI's ChatGPT model piled up with Nvdia V100 graphics card certainly helped us complete the first step from nothing to something, from seeing the possibility to proving the possibility, just like the expensive search engine built by Lycos Same service. However, is there a way like Google, the possibility of using software to crazily reduce hardware costs? Of course, we have passed the era of hand-built servers, and soldering the GPU with a soldering iron does not seem to be a reliable way (Google did not do this back then, but directly used the Intel Pentium II CPU), but will there be How about some amazing solutions that can reduce costs on a large scale?

I am not a big model, and I can't think of any solution. But if such a plan exists, it may greatly change the competition pattern of the large-scale model industry.