February 18, 2025, 3:39:17 PM IST (Published) 3 Min Read Tesla's former AI czar has taken Grok 3 out for a test drive and had some interesting things to say about the AI chatbot that xAI Elon Musk claimed was "the smartest AI on earth". Andrej Karpathy, former Director of Artificial Intelligence (AI) at Tesla, got early access to Grok 3 and ran comprehensive tests on it, and compared the results to competing models from companies like OpenAI and China's DeepSeek. Also read: Grok 3 launch: Early users share their experience with Elon Musk's 'smartest' chatbot Additionally, Grok 3’s "DeepSearch" feature impressed Karpathy by producing high-quality responses to research-oriented questions, such as rumours about Apple’s upcoming launch or the filming locations of White Lotus 3. He likened it to Perplexity’s DeepResearch offering, though not yet at the level of OpenAI’s "Deep Research." The Bad Despite its strengths, Grok 3 struggled with certain tasks. Karpathy said it failed to decode a Unicode-based "Emoji mystery" question, even with hints and generated nonsensical results when asked to create tricky tic-tac-toe boards. The chatbot also fell short in creative tasks, such as generating an SVG (an image file format) of a pelican riding a bicycle, a challenge that stresses AI’s ability to visualise 2D layouts. The 'Pelican test'. (Image: Andrew Karpathy/X) Karpathy noted some sharp edges in the "DeepSearch" feature, including hallucinated URLs and factual inaccuracies without proper citations. For instance, it incorrectly claimed that two cast members of Singles Inferno Season 4 were still dating. Additionally, Karpathy noted the chatbot’s humour capabilities remain underwhelming, with jokes like, “Why did the chicken join a band? Because it had the drumsticks and wanted to be a cluck-star!” Karpathy concluded that Grok 3 is "somewhere around the state-of-the-art territory of OpenAI’s strongest models (o1-pro, $200/month)" and slightly ahead of DeepSeek-R1 and Gemini 2.0 Flash Thinking. He praised xAI’s rapid progress, stating, “This timescale to state-of-the-art territory is unprecedented.” While acknowledging the chatbot’s limitations, Karpathy expressed optimism, adding Grok 3 to his personal "LLM council" and looking forward to its development.