Alibaba released a new artificial intelligence (AI) model on Thursday, which is known as Qwen with Questions (QwQ), and serves as the latest open source competitor to OpenAI o1 reasoning model.
Launched in preview, the QwQ-32B large language model (LLM) is said to outperform GPT-o1-preview in several mathematical and logical reasoning-related benchmarks. The new AI model is available to download on Hugging Face, however it is not fully open-sourced.
What is Qwen with Questions (QwQ)?
Like other large reasoning models (LRMs), QwQ uses extra compute cycles during inference to review its answers and correct its mistakes, making it more suitable for tasks that require logical reasoning and planning like math and coding.
In a blog post, Alibaba detailed its new reasoning-focused LLM and highlighted its capabilities and limitations. The QwQ-32B,as the name suggests, it is built on 32 billion parameters and has a context window of 32,000 tokens. The model has completed both pre-training and post-training stages.The model is currently in preview, which means a higher-performing version is likely to follow.
Coming to its architecture, the Chinese tech giant revealed that the AI model is based on transformer technology. For positional encoding, QwQ uses Rotary Position Embeddings (RoPE), along with Switched Gated Linear Unit (SwiGLU) and Root Mean Square Normalization (RMSNorm) functions, as well as Attention Query-Key-Value Bias (Attention QKV) bias.
According to Alibaba’s tests, QwQ beats o1-preview on the AIME and MATH benchmarks, which evaluate mathematical problem-solving abilities. It also outperforms o1-mini on GPQA, a benchmark for scientific reasoning. QwQ is inferior to o1 on the LiveCodeBench coding benchmarks but still outperforms other frontier models such as GPT-4o and Claude 3.5 Sonnet.
QwQ does not come with an accompanying paper that describes the data or the process used to train the model, which makes it difficult to reproduce the model’s results. However, since the model is open, unlike OpenAI o1, its “thinking process” is not hidden and can be used to make sense of how the model reasons when solving problems.
Notably, Alibaba has made the AI model available via a Hugging Face listing and both individuals and enterprises can download it for personal, academic, and commercial purposes under the Apache 2.0 licence.