Coinbase Tests ChatGPT. Will AI Change Crypto Industry?
Coinbase, one of the largest cryptocurrency exchanges in the world, has conducted a test using the artificial intelligence (AI) application, ChatGPT, to assess the security risks of smart contracts. The San Francisco-based exchange tested the AI tool's accuracy by using its ERC-20 token review framework, publishing the results in a blog post on March 20
Coinbase's security review process involves identifying potential risks associated with new assets and ensuring that they meet the exchange's security standards before they are listed. ChatGPT's ability to quickly assess smart contract risks made it an attractive tool for Coinbase to test. However, the exchange noted that ChatGPT did not meet its accuracy requirements for security reviews, citing inconsistent responses and incorrect labeling of high-risk assets as low-risk ones.
To determine ChatGPT's accuracy, Coinbase compared the results of ChatGPT's analysis with those of its internal team. The company compared a total of 20 smart contract risk scores between ChatGPT and manual reviews, with ChatGPT getting the same result 60% of the time. However, in the eight incorrect outputs, the AI labeled a high-risk asset as a low-risk one five out of ten times.
Coinbase's security team taught ChatGPT how to conduct the security analysis using its format. However, the machine still mislabeled risks because it could not recognize "when it lacks context to perform robust security analysis." Besides that, ChatGPT responses were also inconsistent when asked the same question repeatedly. Coinbase said the AI tool was "influenced by comments in the code and seemed to default to comments rather than function logic occasionally."
Despite ChatGPT's inaccuracies, Coinbase believes that AI and automation can significantly improve security assessments. The exchange stated that once automated or intelligent processes can demonstrate the same level of accuracy as their existing processes, these tools will only be used as an aid, not the main tool.
This is not the first time that Coinbase has experimented with ChatGPT. Earlier this month, Coinbase's CEO Conor Grogan revealed that ChatGPT was successful in finding bugs in smart contracts in his Twitter post. Grogan offered ChatGPT a live Ethereum contract, asking it to discover any security vulnerabilities. Not only did it discover exploits, but it also offered a specific means to exploit the contract.
