HomeAI News
Stanford University releases AI basic model "transparency index", Llama 2 leads with 54% but "fails"
49

Stanford University releases AI basic model "transparency index", Llama 2 leads with 54% but "fails"

Hayo News
Hayo News
October 23rd, 2023
View OriginalTranslated by Google

Stanford University recently released the "Transparency Index" of AI basic models. The highest index among them is Meta's Lama 2, but the related "transparency" is only 54%. Therefore, researchers believe that almost all AI models on the market "lack transparency." ".

It is reported that this research was led by Rishi Bommasani, head of HAI's Center for Basic Model Research (CRFM), and investigated the 10 most popular basic models overseas:

  • Meta's Llama 2.

  • BloomZ by BigScience,

  • OpenAI’s GPT-4,

  • Stability AI’s Stable Diffusion,

  • Claude from Anthropic PBC,

  • Google's PaLM 2.

  • Command of Cohere,

  • Jurassic-2 by AI21 Labs

  • Inflection AI’s Inflection,

  • Amazon Titan.

Rishi Bommasani believes that "lack of transparency" has always been a problem faced by the AI ​​industry. In terms of specific model "transparency indicators", IT House found that the relevant evaluation content mainly focuses on "model training data set copyright", "operations used to train the model" Resources", "Credibility of the content generated by the model", "Model's own capabilities", "Risk of the model being induced to generate harmful content", "User privacy using the model", etc., for a total of 100 items.

The final survey showed that Meta's Lama 2 topped the list with 54% transparency, while OpenAI's GPT-4 had only 48% transparency, and Google's PaLM 2 ranked fifth with 40%.

Among the specific indicators, the top ten models with the "best" scores are "Model Basics". This evaluation mainly includes "whether the model, scale, and architecture of the model are accurately introduced during model training." The average transparency is 63%. The worst performer is Impact, which mainly evaluates whether the basic model will "retrieve user information for evaluation", with an average transparency of only 11%.

CRFM Director Percy Liang said that the "transparency" of the business base model is very important for promoting AI legislation, as well as related industries and academia.

Rishi Bommasani said that lower model transparency makes it more difficult for companies to know whether they can safely rely on relevant models, and for researchers to rely on these models to do research.

Rishi Bommasani ultimately believes that the above ten basic models all "fail" in terms of transparency. Although Meta's Llama 2 has the highest score, it cannot meet the needs of the outside world. "The transparency of the model must reach at least 82% to be recognized by the outside world."

Comments

no dataCoffee time! Feel free to comment