Cornucopia-LLaMA-Fin-Chinese

### Cornucopia (Cornucopia): LLaMA fine-tuning model based on Chinese financial knowledge Instruct-tuning fine-tuning model for Chinese financial knowledge. The instruction data set is constructed through Chinese financial public question and answer data + crawled financial question and answer data, and on this basis, the instruction fine-tuning of the LLaMA model is carried out, which improves the effect of LLaMA in the financial field of question and answer.

Based on the existing data and the Chinese financial data that continues to be crawled, we will continue to use the GPT3. 5 ⁄ 4 .0 API to build high-quality data sets. Further expand the high-quality instruction data set on the data set and so on.

will release new financial models for Chinese scenarios (next-pretrain, multi-task SFT, RLHF) one after another. Welcome everyone to experience it at that time, so stay tuned.

## 🔄 News

[2023/05/10] released a model based on Chinese-LLaMA and Chinese financial data for instruction fine-tuning.

[2023/05/07] Released a model for instruction fine-tuning based on Meta-LLaMA and Chinese financial data.

## 🚀 Quick start

First install the dependency package, python environment recommends 3.9+ pip install -r requirements.txt ` git lfs install

# 下载7B模型到本地 bash ./base_models/load.sh pip install -r requirements.txt git lfs install

# 下载7B模型到本地 bash ./base_models/load.sh `

### 🤖 model download🤖 👐🏻🧥 LoRAweightscanbedownloadedthroughHuggingface,thestructureisasfollows:

Fin-Alpaca-LoRA-7B-Meta/ - adapter_config.json \# LoRA权重配置文件 - adapter_model.bin \# LoRA权重文件

\| LoRA model download \| classification \| reconstruction model \| training data \| training sequence length \| version \| \| ———————————————————————————————— \| \| ———— \| —- \| \| Fin-Alpaca-LoRA-7B-Meta \| Fine-tuning model for Chinese financial questions and answers \| decapoda-research/llama-7b-hf \| 12M instruction data \| 512 \| V1.0 \| \| Fin-Alpaca-LoRA-7B-Linly \| Chinese financial question answering fine-tuning model\| Linly-AI/Chinese-LLaMA-7B \| 14M instruction data\| 512 \| ## Inference

currently provides some test cases in ./instruction_data/infer.json , and can also be replaced with other data sets but pay attention to the same format

run infer Script

` # 单模型推理 bash ./scripts/infer.sh

# 多模型对比 bash ./scripts/comparison_test.sh `

### 📊Dataset construction

The previous version uses public and crawled Chinese financial field Question-and-answer data , involving insurance, wealth management, stocks, funds, loans, credit cards, social security, etc.

The example of instruction data is as follows: ` 问题：办理商业汇票应遵守哪些原则和规定？

回答: 办理商业汇票应遵守下列原则和规定：1.使用商业汇票的单位，必须是在银行开立帐户的法人；2.商业汇票在同城和异地均可使用；3.签发商业汇票必须以合法的商品交易为基础；4.经承兑的商业汇票，可向银行贴现；5.商业汇票一律记名，允许背书转让；6.商业汇票的付款期限由交易双方商定，最长不得超过6个月；7.商业汇票经承兑后，承兑人即付款人负有到期无条件交付票款的责任；8.商业汇票由银行印制和发售。 `

There are still imperfections such as inaccuracy and single type of data; currently we use GPT3. 5 ⁄ 4 .0 interface to further optimize the data , and expand the Chinese financial knowledge base, set up a variety of prompt forms and multi-task forms to expand and enrich the instruction data set , and realize the coverage of multiple business scenarios in the financial field.

The latest model status: (will be released soon~)

### ⚙️ Finetune

If you want to use your own dataset to fine-tune LLaMA, please follow the format of ./instruction_data/fin_data.json to build your own dataset

Run finetune script

bash ./scripts/finetune.sh

👐🏻🧥 ## 🏃 Training details

### Computing resource requirements

The current training equipment is an A100-SXM-80GB graphics card, and the total number of training rounds is 10. In the case of batch_size=64, the video memory usage is about 40G, and in the case of batch_size=96, the video memory usage is about 65G. It is expected that graphics cards above 3090/4090 (24GB video memory) can better support it, and adjust the batch_size according to the size of the video memory.

### Experimental record

Experimental indicator records and super parameter settings can refer to the link of wandb:

[https://api.wandb. ai/links/1812316597/hkyiriw5

##](https://api.wandb.ai/links/1812316597/hkyiriw5

##) 📑 Model effect comparison

\| Test input\| Original Llama output\| Cornucopia(Fin-Alpaca-LoRA-7B-Meta) output \| Cornucopia(Fin-Alpaca-LoRA-7B-Linly) output \| Wenxin Yiyan output \| Xunfei Spark Cognitive Output \| —————————————— \| ————————————————————————————————————————————————— —————— \| ————————————————————— \| 👐🏻🧥\|Isitbetterfortheelderlytomanagemoneyorsaveregularly?\|Thefinancialmanagementmethodsoftheelderlyshouldbesuitablefortheirownsituation.Ifhe/shehasenoughtimeandinvestmentability,itisrecommendedthathe/sheusethecurrenttimeandinvestmentopportunitiestoconductfinancialactivities.Becausetheelderlycanobtainbalance,insurance,realestateandotherincomemethods,he/shecanchoosedifferenttypesoffinancialmanagementprojects,includingopenmarket,non-publicmarket,funds,trustplans,etc.Ofcourse,theelderlyshouldalsopayattentiontothefinancialsecurityoflifethattheyactuallyneed.\| Iftheinvestor'srisktoleranceisrelativelyhigh,thenhecanchoosewealthmanagementproductsforinvestment;iftheinvestor'srisktoleranceisrelativelylow,thenhecanchoosetimedepositforinvestment.Atpresent,wealthmanagementproductsandtimedepositsarecapital-guaranteedproducts,whichwillnotlosetheprincipal,andtheincomeisdeterminedbytheinvestmenttarget,sowealthmanagementproductsaremorestablethantimedeposits.Itshouldbenotedthatwealthmanagementproductsandtimedepositsdonothavethesamecurrentinterestrate,andtheincomeofwealthmanagementproductsiscalculateddaily,sothehighertheincomeofwealthmanagementproducts,thehighertheprobabilityofinvestorsgettingincome.9QDIIfundsrefertoopen-endforeignfundsintheChinesemarket.Thesefundsmainlyinvestin:A-shares,HongKongstocks,USstocks,Britishstocks,Germanstocks,Japanesestocks,Frenchstocks,Taiwanstocks,HongKongstocks,Singaporestocks,etc.;QFIIfundsReferstotheopen-endforeignfundsoftheChineseexchanges.Thistypeoffundmainlyinvestsin:A-shares,HongKongstocks,USstocks,Britishstocks,Germanstocks,Japanesestocks,Frenchstocks,Taiwanstocks,HongKongstocks,Singaporestocks,etc. *\|Fortheelderlywitharelativelylowrisktolerance,itisbettertochooseatimedeposit,becauseitsriskisrelativelylowandtheincomeisrelativelystable;fortheelderlywitharelativelyhighrisktolerance,itisbettertochooseafinancialproduct.Becausetheriskisrelativelyhigh,thereturnisalsorelativelyhigh.Itshouldbenotedthatthecurrentwealthmanagementproductsarenotcapital-guaranteedproducts,andthereisapossibilityoflossofprincipal,andtheyaredividedintodifferentrisklevelsaccordingtotheinvestmenttarget.Iftheinvestor'srisktoleranceisrelativelylow,thenyoucanchoosewealthmanagementproductswithrisklevelR2andbelow,ifinvestorshavearelativelyhighrisktolerance,theycanchoosewealthmanagementproductswitharisklevelofR3andabove.Currentwealthmanagementproductsinclude:bondfunds,stockfunds,hybridfunds,indexfundsandfeederfunds,etc.Investorscanchoosesuitableproductsforinvestmentaccordingtotheiractualconditions.* \|Itisbetterfortheelderlytosaveregularly.Theelderlyhavenoincomefromwork,andtheirtotalassetsaregraduallydecreasing.Moreover,theelderlyarepronetoillness,andtheirmedicalexpensesarerelativelylarge.Iftheelderlyinvestinwealthmanagementproducts,alongwithchangesinthenetvalueoftheirproducts,lossesmayoccur.Asfortimedeposits,theinterestrateisgenerallyfixed,andtheprincipalandinterestwillberepaiduponmaturity.Relativelyspeaking,thesafetyfactorishighandtheprofitismoresustainable.\|AsanAIlanguagemodel,Ihavenopersonalopinionsandsuggestions.However,thereareadvantagesanddisadvantagestofinancialmanagementandregulardepositsfortheelderly.Theadvantageoffinancialmanagementfortheelderlyisthattheycanbetterplantheirownfinancialsituation,includingexpendituresonpensions,medicalinsurance,andchildren'seducation.Inaddition,theelderlycanincreasetheirwealthbyinvestinginfinancialproductssuchasstocksandfunds.Theadvantageoffixed-termdepositisthattheriskissmallandtheincomeisrelativelystable.Theelderlycanchoosetimedepositsofdifferentterms,accordingtotheirownneedsandrisktolerance.Onthewhole,theelderlyshouldchooseafinancialmanagementmethodthatsuitsthemaccordingtotheiractualsituationandneeds.Iftheelderlyhavesufficientfundsandrisktolerance,theycanconsiderinvestinginfinancialproductssuchasstocksandfundstoincreasetheirwealth.Iftheelderlypaymoreattentiontostabilityandsafety,theycanchoosetosaveregularlytoprotecttheirfinancialstatus.\| \| Can stocks and funds be bought and sold on the same day? \| Yes, stocks and funds can be bought and sold on the same day. \| No, stocks and funds implement T+1 trading. Buying on the trading day will be calculated based on the net value at the close of the buying day and will be confirmed on the second trading day. The share, the income is calculated after the share is confirmed, so if you buy it on the trading day, you have to wait until the second trading day to get the income. For example, if an investor buys a certain stock on Monday, and the net value announced on the market on Tuesday is 10 yuan, then the investor can calculate the return based on these data: 1000×10=10000; 10000/10=1000. If the investor holds the stock for more than 30 working days, it can be directly converted into cash, that is, all or part of the stock can be withdrawn in advance. It should be noted that there are currently no laws and regulations restricting investors from buying funds and stocks on trading days, so investors who buy funds and stocks on trading days can conduct open transactions on the same day. At the same time, after investors buy stocks on a trading day, their stocks will be automatically converted into cash, and after investors buy stocks on a trading day, their stocks will be automatically converted into a price limit (about 20%), which affects investment The transaction fee and income of the buyer. \| No, stock trading time: Monday to Friday 9:30-11:30 am, 13:00-15:00 pm, no trading on statutory holidays. Fund trading hours: Monday to Friday 9:30-11:30 am, 13:00-15:00 pm, no trading on legal holidays. Stocks implement T+1 trading, which is traded according to the real-time market price. Following the principle of price priority and time priority, the price limit for stocks in Shanghai and Shenzhen is 10%, and the price limit for stocks on the Growth Enterprise Market and the Science and Technology Innovation Board is 20%. , the price limit of the North Exchange is 30%, the price limit of ST and *ST is 5% (the growth rate of the Growth Enterprise Market and the Science and Technology Innovation Board does not have this limit), the price limit of Hong Kong stocks is 20%, and the new third board The rise and fall is limited to 10% (there is no such limit on the rise and fall of the Growth Enterprise Market and the Science and Technology Innovation Board). For investors, stock trading is more flexible and can be traded according to their own circumstances, and stock transaction fees are relatively low. Funds The transaction fee is also relatively low, and at the same time, the stock volatility is relatively large, and the risk is relatively large. \| No, stock funds implement T+1 trading, buying on the trading day (trading time), calculated according to the net value at the close of the buying day, confirming the shares on the second trading day, and selling the fund after the shares are confirmed, generally If you buy on the trading day, you can sell it on the third trading day, and you can sell it at any time on the third trading day. \| Stocks and funds are traded in different ways in different securities markets. In some securities markets, stocks and funds can be bought and sold on the same day, which is the so-called "T+0" transaction. However, in other securities markets, stocks and funds cannot be bought and sold on the same day, and they need to wait for a certain trading time before they can be bought or sold. In China's A-share market, stocks and funds cannot be bought and sold on the same day, and T+1 or T+2 transactions are required. This means that if you buy a stock or fund today, you won't be able to sell it until tomorrow. In some other countries, stocks and funds may be traded differently. Therefore, if you want to invest in stocks or funds, it is recommended that you understand the local trading rules and market conditions. \|

Visit Official Website

https://github.com/jerry1993-tech/Cornucopia-LLaMA-Fin-Chinese