Firefly (流灰) is an open source Chinese language model project. Just like our project name, we hope that this project can emit a faint light like Firefly, and contribute to the Chinese language model community and promote the Chinese language model community. development of. **
Liuying** (another name for firefly) is a symbol of traditional Chinese culture. Although rotten grass is called firefly, it has a sense of compassion, but even though fireflies are small, they can light up the night sky with their faint fluorescence. The name of this project is taken from Du Mu's "Autumn Evening": a cold painting screen with silver candles in autumn light, and a light fan fluttering fireflies .
《咏萤火》
唐.李白
雨打灯难灭，
风吹色更明。
若飞天上去，
定作月边星。
🔔 The main contents of this project are as follows:
- 📗 Support fine-tuning of full parameter commands, QLoRA low-cost and efficient command fine-tuning, LoRA command fine-tuning (will be provided later support).
- 📗 Support most mainstream open source large models, such as baichuan, Ziya, Bloom, LLaMA, etc.
- 📗 Support lora and base model for weight merging, making reasoning more convenient.
- 📗️ Model clipping: through LLMPruner: a large language model clipping tool , open source clipped Bloom model weights . On the premise of retaining the pre-trained Chinese knowledge, it effectively reduces the amount of model parameters, reduces training costs, and improves training efficiency.
- 📗 Organized and open source instruction fine-tuning datasets: firefly-train-1.1M, moss-003-sft-data, ultrachat, WizardLM_evol_instruct_V2_143k, school_math_0.25M.
- 📗 Open source Firefly series instructions to fine-tune model weights .
🔔 The picture below is the generation effect of firefly-bloom-7b1's multi-round dialogue.
Visit Official Website
