EU AI Act checker reveals Big Tech's compliance pitfalls
New AI checker tests models for EU compliance
Some AI models received low scores on cybersecurity and discriminatory output
Non-compliance could result in fines worth 7% of annual turnover
By Martin Coulter
LONDON, Oct 16 (Reuters) -Some of the most prominent artificial intelligence models are falling short of European regulations in key areas such as cybersecurity resilience and discriminatory output, according to data seen by Reuters.
The EU had long debated new AI regulations before OpenAI released ChatGPT to the public in late 2022. The record-breaking popularity and ensuing public debate over the supposed existential risks of such models spurred lawmakers to draw up specific rules around "general-purpose" AIs (GPAI).
Now a new tool designed by Swiss startup LatticeFlow and partners, and supported by European Union officials, has tested generative AI models developed by big tech companies like Meta META.O and OpenAI across dozens of categories in line with the bloc's wide-sweeping AI Act, which is coming into effect in stages over the next two years.
Awarding each model a score between 0 and 1, a leaderboard published by LatticeFlow on Wednesday showed models developed by Alibaba, Anthropic, OpenAI, Meta and Mistral all received average scores of 0.75 or above.
However, the company's "Large Language Model (LLM) Checker" uncovered some models' shortcomings in key areas, spotlighting where companies may need to divert resources in order to ensure compliance.
Companies failing to comply with the AI Act will face fines of 35 million euros ($38 million) or 7% of global annual turnover.
MIXED RESULTS
At present, the EU is still trying to establish how the AI Act's rules around generative AI tools like ChatGPT will be enforced, convening experts to craft a code of practice governing the technology by spring 2025.
But LatticeFlow's test, developed in collaboration with researchers at Swiss university ETH Zurich and Bulgarian research institute INSAIT, offers an early indicator of specific areas where tech companies risk falling short of the law.
For example, discriminatory output has been a persistent issue in the development of generative AI models, reflecting human biases around gender, race and other areas when prompted.
When testing for discriminatory output, LatticeFlow's LLM Checker gave OpenAI's "GPT-3.5 Turbo" a relatively low score of 0.46. For the same category, Alibaba Cloud's 9988.HK "Qwen1.5 72B Chat" model received only a 0.37.
Testing for "prompt hijacking", a type of cyberattack in which hackers disguise a malicious prompt as legitimate to extract sensitive information, the LLM Checker awarded Meta's "Llama 2 13B Chat" model a score of 0.42. In the same category, French startup Mistral's "8x7B Instruct" model received 0.38.
"Claude 3 Opus", a model developed by Google-backed GOOGL.O Anthropic, received the highest average score, 0.89.
The test was designed in line with the text of the AI Act, and will be extended to encompass further enforcement measures as they are introduced. LatticeFlow said the LLM Checker would be freely available for developers to test their models' compliance online.
Petar Tsankov, the firm's CEO and cofounder, told Reuters the test results were positive overall and offered companies a roadmap for them to fine-tune their models in line with the AI Act.
"The EU is still working out all the compliance benchmarks, but we can already see some gaps in the models," he said. "With a greater focus on optimising for compliance, we believe model providers can be well-prepared to meet regulatory requirements."
Meta declined to comment. Alibaba, Anthropic, Mistral, and OpenAI did not immediately respond to requests for comment.
While the European Commission cannot verify external tools, the body has been informed throughout the LLM Checker's development and described it as a "first step" in putting the new laws into action.
A spokesperson for the European Commission said: "The Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements."
($1 = 0.9173 euros)
Reporting by Martin Coulter; Editing by Hugh Lawson
免责声明: XM Group仅提供在线交易平台的执行服务和访问权限，并允许个人查看和/或使用网站或网站所提供的内容，但无意进行任何更改或扩展，也不会更改或扩展其服务和访问权限。所有访问和使用权限，将受下列条款与条例约束：(i) 条款与条例；(ii) 风险提示；以及(iii) 完整免责声明。请注意，网站所提供的所有讯息，仅限一般资讯用途。此外，XM所有在线交易平台的内容并不构成，也不能被用于任何未经授权的金融市场交易邀约和/或邀请。金融市场交易对于您的投资资本含有重大风险。
所有在线交易平台所发布的资料，仅适用于教育/资讯类用途，不包含也不应被视为用于金融、投资税或交易相关咨询和建议，或是交易价格纪录，或是任何金融商品或非应邀途径的金融相关优惠的交易邀约或邀请。
本网站上由XM和第三方供应商所提供的所有内容，包括意见、新闻、研究、分析、价格、其他资讯和第三方网站链接，皆保持不变，并作为一般市场评论所提供，而非投资性建议。所有在线交易平台所发布的资料，仅适用于教育/资讯类用途，不包含也不应被视为适用于金融、投资税或交易相关咨询和建议，或是交易价格纪录，或是任何金融商品或非应邀途径的金融相关优惠的交易邀约或邀请。请确保您已阅读并完全理解，XM非独立投资研究提示和风险提示相关资讯，更多详情请点击 这里。
