LINE Japan open source its own large language model
LINE's Japanese headquarters announced this week that it would open source its self-developed Japanese Large Language Model (LLM).
LINE will open source its japanese-large-lm model under Apache License 2.0. In addition to research use, it also allows commercial use. It contains 2 versions of 3.6 billion and 1.7 billion parameters. Both projects can be downloaded from HuggingFace Hub .
Since November 2020, LINE has been committed to launching a number of construction and application-related research and development plans for the company's large-scale language model HyperCLOVA. At the developer conference in May 2021, LINE released the Korean version of LLM HyperCLOVA trained with 204 billion parameters for the first time. In November, it announced the Japanese version with 85 billion parameters, and announced that it will develop a series of Natural language processing services.
The japanese-large-lm model released this time is developed in parallel with HyperCLOVA by different departments. The former is the development result of the Massive LM team. The team pointed out that this model is trained on the basis of LINE's own large Japanese web texts, and uses the HojiChar open source library developed by members to filter a large amount of noise such as source code and non-Japanese text, and finally used a 650GB data set to train. In terms of development time, in terms of the 1.7 billion parameter version, it took 4000 GPU time to train on the A100 hardware.
The research team also provided the accuracy and perplexity score (PPL) comparison data of the two models disclosed this time and the Rinna-3.6B and OpenCALM-7B models.