Lin Junchao: Looking at the investment opportunities of AI from the perspective of the primary market

Source: Titanium Capital Research Institute

After ChatGPT swept the world, artificial intelligence was pushed to a new outlet. As a subversive intelligent productivity tool, it has already become a hot topic. At present, developed countries and regions around the world have raised the development of the artificial intelligence industry to the national strategy, and related emerging technologies and applications have been continuously implemented. With the in-depth development of the new wave of AI, an industrial revolution led by digital intelligence technology is about to emerge, and it has also opened the prelude to the "big model era" in all walks of life.

Why can large language models lead the trend of AI? What are the investment opportunities for domestic AI? Recently, Titanium Capital invited Lin Junchao, the investment director of Rongyi Capital, to share the theme: Looking at the investment opportunities of AI from the perspective of the primary market. Mr. Lin holds a bachelor's degree in electrical and electronic engineering and a first-class honors master's degree from University College London (UCL), and has three years of entrepreneurial experience and six years of equity investment experience. Its investment focuses include basic software development tools, Metaverse and AIGC, including Jingye Intelligence, Smart Open Source China and other projects. The host of this sharing is Wu Kai, Managing Director of Titanium Capital. The following is the sharing record:

Why the big language model can lead the trend of AI‍‍‍

A Brief History of AI Development

Since Shannon proposed computer games in 1950, AI has experienced more than 70 years of development. We can roughly divide it into four stages: early enlightenment + expert system era (50s-70s), machine learning + computer vision era (80s-90s), deep learning era (2006-2017), multimodal + large language model era (2018-present).

The milestone event in the early enlightenment stage occurred in the Dartmouth College Summer Symposium in 1956. The concept of artificial intelligence was first proposed at the meeting, which officially kicked off the prelude to artificial intelligence. During this period, the first chat robot ELIZA developed by MIT in 1966, the first expert system DENDRAL, and David Marr proposed the concepts of computer vision and computational neurology in 1976.

In the era of machine learning, there is a milestone figure, that is, the godfather of AI who left Google some time ago-Jeffrey Hinton. In 1986, he proposed the backpropagation algorithm, which made large-scale neural network training possible and paved the way for later neural networks and deep learning. At the same time, many milestone events and figures also appeared in this stage, such as the backgammon game in 1979, and Deep Blue's victory over the world chess champion Kastrov in 1997. In 2006, Jeffrey Hinton and others formally proposed the concept of deep learning, thus introducing AI into the third stage—the era of deep learning.

In the era of deep learning, Jeffrey Hinton proposed the convolutional neural network AlexNet in 2012 and won the ImageNet image classification competition. The overall error rate was 10.8% lower than the second place, officially opening the first year of deep learning in computer vision. In March 2013, Google acquired Jeffrey Hinton's start-up company DNNResearch for US$44 million. Since then, Google has begun to lead the development of global AI; in 2015, TensorFlow, the world's most influential deep learning framework, was open-sourced; in 2016, AlphaGo defeated the nine-dan Go master Li Sedol 4:1; in 2017, it launched the Transformer model, which has since opened the era of the current big language model. At the same time, along with the wave of AI led by Google, AI unicorns such as Scale AI, HuggingFace, and OpenAI were also established in 2016. Internet giant Meta also open sourced a more useful deep learning framework Pytorch in 2017.

OpenAI, the leader in the era of large language models, first appeared in early April 2019. The OpenAI Five developed by it defeated the Dota2 world champion OG team with an absolute advantage of 2:0, and then released GPT-3 with 175 billion parameters in May 2020, ChatGPT (GPT-3.5) in November 2022, and GPT-4 in March 2023. Since then, it has officially taken over from Google and started to lead the world. Model Development. Internet/software giants such as Google, Meta, and Microsoft have also adjusted their strategies and actively participated in the large-scale model competition. Since ChatGPT at the end of 2022, we have seen that major manufacturers around the world have entered the large-scale model, and domestic universities, technology giants, start-ups, etc. have also continuously launched various large-scale model products and applications.

The history of AI development in China can be traced back to the establishment of Microsoft Asia Research Institute in 1998. Today, the founders and core teams behind the well-known domestic companies such as Innovation Works, Baidu, Kingsoft, Alibaba, SenseTime, and Questyle all come from Microsoft Asia Research Institute. The first batch of domestic AI companies iFlytek, Dahua, and Hikvision were listed on the A-share market around 2008-2010. From 2011 to 2015, many computer vision startups emerged in China, such as Megvii (founded in 2011), YITU (founded in 2012), SenseTime (founded in 2014), and Yuncong (founded in 2015). In 2018, the national team - Beijing Zhiyuan Artificial Intelligence Research Institute was established. The development of large models this time has also benefited from this wave. Since 2002, AI-related companies such as Cambrian, SenseTime, Haitian Ruisheng, and Yuncong have been listed on the Science and Technology Innovation Board and Hong Kong Stock Exchange.

The charm of ChatGPT and GPT-4

Why does ChatGPT and GPT-4 allow us to intuitively feel the difference and power of this wave of AI from the past? It can mainly be viewed from the following aspects:

**First, from the user's point of view, this time is a very human-like interaction method. **When talking with ChatGPT, the content is generated word by word, and the answer is generated while thinking. At the same time, it also has the ability of multiple rounds of dialogue. In addition, it can also play some roles, such as translators, psychological counselors, etc.

**Second, generalization ability, mainly reflected in the ability to quickly understand requirements and give relatively accurate answers without context. **This relies on the support of massive pre-training corpus and hint engineering.

**Third, the chain of thinking. ** Ability to understand context and context, even long texts. For example, the newly released Claude2 has a context processing capacity of 100,000 tokens, which can basically feed an entire book into it for processing. At the same time, there is also a strong logical reasoning ability, which can gradually disassemble and deduce the problem according to the way of human thinking.

In fact, these capabilities emerge with the increase of model parameters and the extension of training time. In addition, we can see that GPT-4 has excellent results in various human examinations, basically reaching the level of prospective college students.

The composition of the large language model

Going back to the big language model itself, I recommend everyone to watch the State of GPT keynote speech shared by OpenAI co-founder Andrej Karpathy at the Microsoft Developers Conference. He disassembled the training process of the large model into four stages:

**In the pre-training stage, a large amount of relatively low-quality training data (including books, Internet crawling, papers, etc.) and 99% of the computing power and time in the entire large model training process are required to train a basic model. **For example, training a 650 billion parameter LLaMA basic model requires 2048 Nvidia A100 GPUs, which takes 21 days, and the entire training cost is about 5 million US dollars. Therefore, training a basic model is not friendly to start-ups, and such resources and costs are generally only available to large manufacturers.

The second step is supervised and fine-tuned on the basic model, so as to train the SFT model, such as Vicuna-13B and other common open source models, are supervised and fine-tuned models. At this stage, only a small amount of high-quality data needs to be provided, and the demand for computing power is relatively small.

**The third and fourth stages are the reward modeling and reinforcement learning stages, that is, RLHF human reinforcement feedback learning. ** Through these two stages, the output of the model will be far better than the supervised fine-tuning model and the basic model, but the entropy of the basic model will be sacrificed.

From the perspective of primary market industry to see AI opportunities

Looking at AI investment opportunities systematically from the perspective of primary market investment, it can be divided into basic layer, deployment layer and application layer. The AI model community runs through these three layers.

**The basic layer can be divided into infrastructure layer, data layer and algorithm layer. **

The bottom layer of the infrastructure layer is computing power, and it is also the place that is currently facing the most direct bottleneck. Now both Nvidia and AMD GPUs are embargoed in China. Huawei Kunpeng, the leader in domestic computing power, is also facing the problem of tape out. In the future, China may face increasing pressure in terms of high-end computing power. At the same time, there is still the problem of GPU utilization. Even for OpenAI, a large number of Nvidia engineers are resident on site for manual tuning, but its GPU utilization is only 50%-60%. Above the computing power is the basic software layer around the large model, such as AI-native data storage, data transmission, data security and other tools and development and maintenance platforms around the data, including vector databases, LLMOps, MLOps, etc.

There may be more opportunities in the data layer, because the basic model may be developed by leading companies in the future, and there will be a large number of open source basic models, so there is no need to spend a lot of money to develop your own basic model. We should focus on the data accumulation of vertical industries and enterprises themselves, and how to form large-scale applications that customers are willing to pay for. At present, the biggest pain point of large models is how to obtain high-quality data sets and vertical industry data sets. Everyone has seen the potential of large models in the fields of industry, science, medicine, law, finance, and education. Future opportunities may lie in how to efficiently obtain high-quality data, how to process multi-modal data, how to identify, how to capitalize data, how to confirm rights, how to trade, how to protect security, etc.

The core of the algorithm layer lies in the deep learning framework and basic model

The deep learning framework can be described as the operating system of AI. It coordinates the deployment of computing resources downwards, undertakes the building capabilities of AI algorithm models upwards, and provides a large number of operator libraries, model libraries, and document tutorials. It is also an ecology in nature and lowers the development threshold. At present, the world's mainstream deep learning frameworks are mainly Google's TensorFlow and Meta's Pytorch. In China, there are mainly Baidu Fei Paddle, Huawei Shengsi and Oneflow, which was previously acquired by Light Years Beyond.

The basic model itself also has diversity. For example, in terms of technical paths, there are CNN, RNN, GAN, Transformer, etc. The Transformer model can be divided into autoencoding model, autoregressive model, encoder-decoder model, etc., and can be divided into closed source and open source in form. This direction is the most likely to give birth to companies with a market value of hundreds of billions or even trillions, but it is also the main battlefield with the most intense competition.

In the era of the 100-model war, model evaluation has become a core tool to measure the capabilities of various models. At present, various evaluation tools for traditional small models (GLUE, SuperGLUE, etc.), large language models (HELM, BIG-Bench, etc.) and Chinese large language models (SuperCLUE, C-, etc.) have appeared on the market. Like SuperCLUE and C-, a large number of questions with Chinese characteristics (Mao Zedong Thought, the basic principles of Muskism, etc.) and Chinese characteristics (idioms, poems, classical Chinese, etc.) and other dimensions have been added to their evaluation sets. Judging from the evaluation results, except for GPT-4, GPT-3.5 and Claude, the domestic large-scale model products are better than other overseas models in the overall evaluation performance, so the necessity of training Chinese large-scale models is very high.

The application layer can be divided into general large models and vertical industry large models. We mainly focus on the new generation of AI-enabled productivity tools in the field of general large models and the application opportunities of large models in various vertical industry fields.

to C—AI empowered productivity tool

In the era of the epidemic, collaboration-themed productivity tools such as Notion, Figma, and Canva have undergone changes. Similarly, under this wave of AI, productivity tools will also usher in a new revolution.

Now we see that large models have penetrated to varying degrees in text, code, image, music, video, 3D, etc. Various new products and new applications are emerging one after another, such as chatbots in the text field and office product copilot, GitHub copilot in the code field, Midjourney and Stable Diffusion in the image field, AI Stefanie Sun, which was popular in the music field before, and runway in the video field, etc. Domestic companies such as Baidu, Kingsoft Office, Evernote, Zhipu Huazhang, etc. have also launched similar AI products. , are changing the form of traditional productivity tools to varying degrees, but currently they are limited to efficiency tools in the industrial production process in various fields, and cannot realize AGI in the true sense.

At the same time, it can be seen that manufacturers such as Microsoft Office, Adobe Photoshop, Unity, and Notion are also actively embracing this wave of AI, embedding AGI capabilities into their own products and tool chains. It was originally thought that the emergence of Midjourney and Stable Diffusion would completely replace Photoshop, but later it was discovered that AGI, due to problems in controllability and other aspects, made Photoshop combined with AI generation capabilities more powerful and easy to use.

The 3D field is currently one of the most difficult fields to implement AI. The core factor is that there are too few high-quality 3D data. At present, AGI for 3D content is mainly explored and led by NVIDIA, Unity, Stability.ai, and scientific research institutes, but at this stage, it is still mainly demo and scatter-shaped tools, and there is still a long way to go before it can be applied to industrial fields such as film and television special effects, games, and metaverse.

to B—vertical industry model

At present, most of the large-scale model products launched by major manufacturers are general-purpose large-scale models. However, when facing vertical industries, B-end customers need high-accuracy, high-consistency, and easy-to-deploy large models that can efficiently solve specific scenario problems with less data and lower computing power. The latest Pangu 3.0 large model released by Huawei is based on the basic large model, adding N L1 industry large models and X L2 scene model capabilities.

The core key points of vertical industry large models are high-quality industry data sets and engineering capabilities for model tuning, compression, and deployment. This is also an investment opportunity point, just like the container opportunity in the cloud-native era, a large number of small and medium-sized B enterprises in traditional industries need to rely on specialized container manufacturers to help them embark on the road of cloud-native.

At present, there have been a lot of exploration and practice overseas in the field of vertical industry large models, such as BloombergGPT in the FinGPT field. Bloomberg has converted the financial data accumulated in the past 40 years into a financial data set of 365 billion tokens, and combined with general data sets to train its own 50 billion parameters. ed-PaLM 2, Microsoft's Nuance (integrated with GPT-4 and released a voice-supported medical record generation application—DAX Express), etc.

Finally, let’s talk about the capital focus of the overseas AGI track: **From the perspective of the amount of investment, the top five are marketing text applications, audio, customer support/customer service robots, images, and MLOps platforms; from the perspective of financing amount, more funds flow to MLOps platforms, whose core value lies in lowering the threshold for developing large models, followed by customer service robots, audio, digital humans, dubbing, and images. **

Q&A

**Q1: Outsourcing service companies that do data annotation and assist AI development seem to be doing very well recently. What is your investment tendency? **

A: We are currently paying attention to these two directions. The field of data labeling mainly focuses on how these companies use the capabilities of large models to improve labeling efficiency, such as using GPT-4 to label text and SAM to label images. Because the current core competition in the field of data labeling is efficiency and gross profit, who can achieve more efficient labeling with the help of large model capabilities. In terms of large-scale model services, it will be similar to the container opportunities in the cloud-native era. Professional vendors are needed to lower the threshold for large-scale model training, development, and deployment, and help each enterprise realize large-scale model freedom.

**Q2: AI now has two categories: TO C and TO B. Do you think the opportunity is greater for TO C or TO B? **

A: We pay more attention to TO B. Because there are too many Internet giants in the TOC field, especially in places with such a strong domestic APP application ecology, it is easier for major manufacturers to embed AI capabilities in their own APPs. Therefore, we pay more attention to their data integration capabilities, commercial understanding and engineering capabilities.

**Q3: Even for large models with more than one billion parameters, more than 80 have been reported in China. How about the investment trend in the big model? How to choose between open source and closed source? **

A: Regarding open source and closed source, it is necessary to think about how to make good use of open source and how to use the open source model for commercialization. For example, LLaMA has commercial restrictions within the open source agreement. Closed source requires its own ecology and support, which may only be maintained by large manufacturers with the ability and financial resources.

**Q4: From the perspective of AI entrepreneurs, they can be divided into three categories. One category comes from big factories and already famous founders. The other category is academicians, academicians and experts from Tsinghua University or other fields. There is also a category of entrepreneurs who tend to be grassroots. Which of these three categories do you prefer? **

A: Many of the wave of large models in China appeared after Open AI released large models such as ChatGPT, LLaMA and Bloom as open sources. We are currently holding a wait-and-see attitude. Of course, there are also many forward-looking large manufacturers and academic start-up companies in China that have been exploring before this wave of large-scale model boom.

For academic teams, how to achieve commercialization is the most challenging. So I don't think it is necessary to do it by yourself in the future. For example, the cooperation between OpenAI and Microsoft can give large-scale model capabilities to large manufacturers. We are now looking for targets around Open Source China, because it has an IDE product line and needs code copilot, so we are looking for the possibility of cooperation with academic teams. I think this approach is more feasible.

**Q5: Which vertical industries on the B-side are most likely to achieve commercial breakthroughs? **

A: Due to the generalization and multi-modality of the ability of large models, such as the legal field is a very common scene, and the demand for text content is very large. Larger models have exactly this capability, although there are still some problems to be solved in terms of accuracy. In addition, personal assistant products are also a scenario that can be imagined, although the development cycle may be longer.