Movable type is an open source and commercially available large-scale pre-trained language model developed by a number of teachers and students from the Institute of Natural Language Processing of Harbin Institute of Technology. The model is based on the 7 billion parameter model of the Bloom structure, supports Chinese and English bilinguals, and has a context window length of 2048. It has achieved excellent results in the same size on standard Chinese and English benchmarks and subjective evaluation.
Limitations: Due to the amount of model parameters, the small amount of Chinese pre-training data and the autoregressive generation paradigm, movable type may still generate misleading replies containing factual errors or harmful content containing prejudice/discrimination, please Carefully identify and use the generated content, and do not spread the generated harmful content to the Internet. If adverse consequences occur, the disseminator shall be responsible for them.
Model Setting The model base uses BLOOM-7B1, which combines the Chinese ability of the BLOOM model itself. Supports single-card inference while ensuring performance. The command fine-tuning dataset is in ChatML format. The total amount of training data is 15B token, including about 20% pre-training corpus and 80% dialogue and instruction data.
Model Features
Movable Type 1.0
Bilingual Chinese and English: Excellent results have been achieved on standard Chinese/English benchmarks and subjective evaluations, and support Multilingual dialogue ability. Refer to manual comprehensive evaluation for details on index scores. Richer instruction fine-tuning data: More instruction fine-tuning templates are artificially constructed, as well as a series of SFT data constructed by self-instruct instructions, making instruction fine-tuning data richer. Achieve better ability to follow instructions Support to generate codes and forms Higher quality security data: Based on multiple rounds of confrontation attacks, manually design security data in the form of SFT to strengthen the security of model replies and compliance. The security index reached 84. 4 ⁄ 100 , even surpassing ChatGPT.
Movable Type 2.0
Better replies: Replies from Movable Type 2.0 have a better pattern and tend to be more detailed and organized. Stable PPO training that integrates multiple tricks: training is more stable and efficient Average Chinese preference data with multi-dimensional annotation: richer answers, stronger ability to follow instructions, and clearer logic Authenticity and harmlessness are scored in three dimensions Comprehensively consider the preference sorting of Instruction category and reply quality
Visit Official Website
