AI News

GLM-5.5 Leaked: Parameter Scale May Reach 2.5 Trillion–3.5 Trillion; Zhipu's Compensation Notice Spoils New Model in Advance

36Kr · AIUpdated Nov 16 18:59

AI summary

In a "degraded intelligence" compensation notice to GLM Coding Plan users, Zhipu revealed that an additional quota reset will be granted after the new model is released, which is seen as an early spoiler for the next-generation GLM-5.5. Online rumors suggest that GLM-5.5 may have a parameter scale of 2.5 trillion–3.5 trillion, roughly 3.4–4.7 times the 743 billion total parameters of GLM-5.3, and may continue to use the hybrid linear attention and sparse attention architecture of GLM-5.3-Flash along with a recurrent Transformer. None of the above information has been officially confirmed.

Body · AI translation

GLM-5.5, is it really coming right away?!

Well, wouldn't you know it—some GLM Coding Plan users online received an official compensation notice, because some users had previously run into the model's "dumbing down" problem.

But what's more interesting than the compensation itself is a line Zhipu left in the announcement—

After the new model is released, we'll also gift an extra quota reset opportunity!

Great, great—a single compensation notice actually ended up spoiling the next-generation model. (doge)

And what a coincidence—recently, leaks about GLM-5.5 have also started popping up one wave after another.

There's gossip claiming that GLM-5.5's parameter scale could reach 2.5 trillion–3.5 trillion, a fair bit larger than the previously rumored 1.6 trillion!

Some folks also speculate: could the recurrent Transformer architecture that Zhipu preliminarily validated earlier also be used in the next-generation GLM???

This... the parameter count could surge into the trillions, and deep reasoning capability is also expected to climb another level? (Though who knows if it's true or not.jpg)

So looking forward to it, so looking forward to it, so looking forward to it *100000.

The official word dropped "new model" in advance—is GLM-5.5 about to... arrive?

Although the official side hasn't disclosed specific information about the next-generation model, following recent rumors downward, we really can trace some signs of GLM-5.5!

First, let's talk about the "parameter scale" everyone cares about.

You should know that GLM-5.3, released in August this year, has a total parameter count of about 743 billion and adopts a MoE (Mixture of Experts) architecture.

But based on the current online speculation about the next-generation GLM's total parameter scale, the next-generation GLM's parameter scale could directly soar to 2.5 trillion–3.5 trillion.

Emm... I counted on my fingers, and if this is true, wouldn't that mean—

It directly expands to about 3.4x–4.7x that of GLM-5.3??! (Really or not.jpg)

Of course, here's a point that needs underlining—

For MoE models, the total parameter count and the number of parameters actually activated per inference are completely different things.

For example, a MoE model with trillions of parameters usually only calls on a portion of its expert networks to participate in computation for each Token processed.

So a surge in parameter scale does not mean the inference computation volume will grow by the same multiple.

But even so, if the online rumors are true, this still means Zhipu is preparing a fairly next-level expansion in model capacity~

And merely making the model bigger doesn't seem to be the end of it.

I searched around, and another rumor about the next-generation new model that's quite worth watching is—GLM-5.5 may usher in a major architecture upgrade.

This actually has traces to follow.

In August this year, when Zhipu released GLM-5.3-Flash, it had already performed a major surgery on the model's underlying architecture.

Among it, the most noteworthy is that for the first time in the GLM series, it introduced a hybrid architecture combining linear attention and sparse attention, while also adopting mHC technology to improve information transfer and training stability inside the model.

With this combo, GLM-5.3-Flash surpassed GLM-5.2 in multiple evaluations, while inference price dropped to about one-tenth of the latter.

In other words, Zhipu has already validated a new route in the previous-generation model that balances model capability and inference efficiency.

If this architecture can be further extended to the trillion-parameter level, the next-generation GLM will have the opportunity to continue lowering inference costs while expanding model capacity.

Besides parameter scale and underlying architecture, there's another speculation in everyone's discussion about GLM-5.5 that's quite worth watching—

Will the recurrent Transformer (Loop Transformer) also be used in the next-generation GLM?

Coincidentally, Zhipu has indeed publicly mentioned this technology before.

In August this year, Zhipu explicitly introduced the recurrent Transformer in its interim report and revealed that it had validated the relevant idea in preliminary experiments.

Simply put—

The recurrent Transformer can let the model repeatedly call the same set of network layers, completing more rounds of internal computation without significantly increasing the parameter count, thereby enhancing effective reasoning depth.

According to Zhipu's judgment, parameter scale determines how much information the model can store, while computation depth relates to how complex a reasoning the model can perform.

Especially in complex tasks such as programming and cybersecurity, the model often needs to make more than a dozen or even dozens of consecutive judgments before it can get the correct result.

Once some link in the middle fails, subsequent tasks may go completely off track!!

Zhipu even gave a "specific example" in the report—

A cybersecurity task may require the model to continuously reason along a causal chain as long as 20 steps, with each step maintaining logical coherence.

And the direction explored by the recurrent Transformer is exactly to enable the model to continuously reason along complex causal chains and improve its ability to handle multi-step tasks.

This mechanism also allows the model to increase computation depth without significantly increasing the parameter count, equivalent to providing another path for capability expansion for complex tasks.

However, an increase in the number of loops also brings additional computational overhead, and how to balance reasoning depth and operating efficiency remains a problem to be solved.

As for whether this technology will actually be applied to GLM-5.5, there is currently no official confirmation.

But putting these clues together, the next-generation GLM's technical upgrade is somewhat worth looking forward to~

Looking back at Zhipu's model upgrade path this year, we can actually already see a fairly clear capability evolution route—

In February this year, GLM-5 was released.

In April, GLM-5.1 followed.

In June, GLM-5.2 went online, focusing on improving complex tasks and long-duration Agent execution capability.

In August, GLM-5.3 was released, continuing to strengthen programming capability, and then GLM-5.3-Flash appeared, beginning new exploration in model architecture and inference efficiency.

From this iteration route, Zhipu has indeed been continuously upgrading around programming, Agent, deep reasoning, and inference efficiency~

If GLM-5.5 can really continue to break through in these directions, the upgrade magnitude of the next-generation flagship may indeed be worth looking forward to.

Waiting, waiting, waiting!!! (rubbing hands.jpg)

Reference link:

This article comes from the WeChat public account "QuantumBit", author: Mengyao, published by 36Kr with authorization.

Source

36Kr · AI