Open-source project unlocks hidden compute power in NVIDIA RTX graphics cards, RTX 4090 specific Tensor Core throughput doubles

An RTX 4090, with no hardware changes whatsoever, saw its throughput for specific Tensor Core operations soar from 165.6 TFLOPS to 331.6 TFLOPS—a straight-up doubling!

What's even more surprising is that the RTX 4090 is far from the only card whose hidden potential has been uncovered this time. The RTX 3090 Ti, RTX 4070 Ti SUPER, and even the RTX 2080 Ti released in 2018 all achieved significant performance gains in community testing.

All of this stems from a group of developers on GitHub exploring the underlying compute limits of Nvidia GPUs.

They found that the Tensor Cores on some GeForce graphics cards have an instruction issue rate limit when executing certain mixed-precision operations. By modifying the relevant configuration during the GPU's boot phase, the previously limited compute paths can be restored to higher throughput.

No GPU replacement is needed, and no graphics card BIOS flashing is required. In the verified Linux approach, even the stock NVIDIA driver can run as usual.

At present, related projects for the RTX 20, 30, and 40 series graphics cards have all appeared on GitHub, and some developers have even completed experiments in a Windows 11 environment. A hunt for the "hidden compute power" of Nvidia graphics cards is unfolding in the open-source community.

RTX 4090 compute power doubles

The most eye-catching result comes from the RTX 4090. Developer JustAResearcher demonstrated the test results of the RTX 4090 before and after unlocking in the open-source project gpu-unlock-ada. In just 2 days, the project has already accumulated 2.7k stars.

Specifically, in Tensor Core operations with FP16 input and FP32 accumulation, throughput soared from 165.6 TFLOPS to 331.6 TFLOPS, nearly doubling. Tests with BF16 input and FP32 accumulation showed the same change, rising from 165.5 TFLOPS to 330.6 TFLOPS. This means that under specific mixed-precision operations, the same RTX 4090 can reach roughly twice the previous instruction throughput.

To rule out chance factors, the developer also increased the test loop count from 20,000 to 40,000, yielding FP16/FP32 and BF16/FP32 throughput of 330.8 TFLOPS and 330.3 TFLOPS respectively, basically consistent with the earlier results.

More noteworthy is that the entire experiment involved no replacement of graphics card hardware, nor did it achieve the performance gains by raising the power limit, manually overclocking, or modifying NVIDIA's official driver.

According to the technical report, the test card uses the AD102 chip, has 128 SMs (streaming multiprocessors), runs the Linux 7.2.4 kernel, and keeps the power limit at 450W. The developer leveraged the firmware execution flow during the GPU boot phase to open write access to protected registers, and then modified the register configuration that controls compute rates.

After the configuration was completed, the graphics card underwent a PCIe function-level reset (FLR), and was then taken over normally by the unmodified NVIDIA 615.78.08 driver. Register checks after driver initialization and CUDA testing showed that the unlocked configuration was still in effect. The developer successfully removed the rate limit on the RTX 4090 under certain Tensor Core operation modes, enabling the existing compute units to achieve higher throughput when executing FP16/BF16 input and FP32 accumulation instructions.

However, the performance change did not cover all operation types. For example, the throughput for FP16 input and FP16 accumulation had already reached 325.7 TFLOPS before unlocking, and after unlocking it reached 331.6 TFLOPS in a longer loop test, without a similar doubling. INT8 operation throughput remained basically around 661 TOPS, and ordinary FP32 scalar operations also stayed at about 79.1 TFLOPS. This shows that the unlock mainly affects the specific Tensor Core operation paths that were previously rate-limited, and did not double all of the RTX 4090's compute capability at once.

Meanwhile, another RTX 4090 owner completed an independent retest using the reviewed project code. This experiment used the Linux 7.2.11-arch1-1 kernel and the NVIDIA 615.78.08 driver, with the same graphics card, the same test program, and the same 450W power limit before and after unlocking, further verifying the results. The tests showed that the instruction throughput for FP16 input and FP32 accumulation rose from 160.4 TFLOPS to 320.4 TFLOPS, while BF16 input and FP32 accumulation rose from 159.7 TFLOPS to 320.2 TFLOPS, with both results nearly doubling.

But when the test shifts from low-level instruction throughput to actual matrix multiplication, the performance gains are not so dramatic. In matrix multiplication tests using cuBLAS, FP16 throughput improved by about 12% to 34% depending on matrix size. Among these, the throughput of 4096×4096 matrix multiplication rose from 153.2 TFLOPS to 206.0 TFLOPS, an increase of about 34.4%. BF16 showed greater variation. The 1024×1024 and 8192×8192 tests improved by about 19.6% and 16.0% respectively, but in the 4096×4096 test, performance barely changed.

This discrepancy is related to how actual compute tasks are executed. Low-level instruction throughput tests minimize the impact of factors such as memory access, whereas real matrix multiplication also involves data movement, compute scheduling, and computational efficiency at different scales. Therefore, while the doubling of throughput for specific Tensor Core instructions on the RTX 4090 has been supported by independent retests, this does not mean that actual AI training or inference speed will also double.

RTX 30 series also sees a surge in compute

The RTX 30 series has likewise received good news. Developer Zagitta's open-source project gpu-unlock has already succeeded on multiple RTX 30 series graphics cards.

Among them, the RTX 3090 Ti performs quite impressively. At a 450W power limit, BF16 matrix operation throughput rose from 87.5 TFLOPS to 157.4 TFLOPS, an increase of nearly 80%; the TF32 improvement reached 94%, nearly doubling.

Successful unlocks have also been reported one after another for the RTX 3080, 3070, and 3060. Among them, the BF16 throughput of the RTX 3080 10GB rose from 65.4 TFLOPS to 113.4 TFLOPS, an increase of about 73%; the related tests on the RTX 3070 and RTX 3060 also achieved improvements of about 81% and 76%, respectively.

Similar to the previous unlock method for the RTX 4090, gpu-unlock also exploits the firmware execution flow during the GPU boot phase to modify the register configuration that controls the Tensor Core instruction issue rate.

Specifically, NVIDIA GPUs contain internal registers used to control the execution rate of different instructions, including SS0 and SS1. These registers are protected by permissions, and ordinary drivers cannot modify them directly.

Zagitta drew on the V67 technique from the earlier CMP graphics card unlock project, exploiting a vulnerability in the GPU secure boot firmware to modify the privilege level mask (PLM) through return-oriented programming (ROP), thereby gaining write access to the relevant registers. The program then sets SS0 to 0*88888888 and SS1 to 0*8, switching the corresponding compute paths to a higher instruction issue rate.

However, Zagitta did not modify the NVIDIA driver; instead, the entire unlock process was integrated into a Linux kernel module. At system boot, the module runs before the official NVIDIA driver takes over the graphics card, completes the register configuration changes, and then makes the configuration take effect through a PCIe function-level reset (FLR). The unmodified NVIDIA driver can then load normally. This approach allows some Ampere architecture graphics cards to achieve twice the original instruction issue rate when executing FP16/BF16 input, FP32 accumulation instructions.

As this technology spreads, more and more graphics card users have begun submitting independent test results.

A developer using an RTX 3080 10GB conducted further tests at a 320W power limit. In addition to a BF16 throughput increase of about 73%, the matrix operation throughput for FP16 input and FP32 accumulation also rose from 64.88 TFLOPS to 110.14 TFLOPS, an increase of about 70%.

In a 120-second alternating FP16/BF16 matrix operation test, the number of computations completed by the graphics card increased from 6,940 to 11,860, with average throughput improving by about 71%. Power consumption remained at about 318W in both tests, with peak temperatures of 78°C and 79°C, respectively.

The tests also verified the correctness of the computation results. In FP16, BF16, and TF32 matrix operations, the outputs before and after unlocking were completely identical. But not all compute tasks show similar improvements. FP16 input and FP16 accumulation operations, which already ran at a higher rate, basically remained at around 112 TFLOPS; INT8 operations and ordinary FP32 operations also showed almost no change.

In addition to matrix operation tests, some real AI tasks also achieved acceleration. An RTX 3090 user running a 27B BF16 large model with vLLM found that prompt prefill speed improved by about 33%, and the response time for the first token was shortened by about 24%. Another developer using two RTX 3080s with expanded VRAM also achieved performance improvements of about 33% to 36% in large-model prefill tests.

However, the per-token generation speed in both sets of experiments showed no obvious change. This is because the large-model prefill stage requires a large amount of matrix computation, while per-token generation is often more limited by memory bandwidth.

From low-level register modifications to real AI workloads, this unlock has already achieved considerable performance gains on multiple RTX 30 series graphics cards. And the same technical approach is also beginning to be applied by developers to earlier GPU architectures.

The 2080 Ti from 8 years ago has also had its potential tapped

The progress on the RTX 20 series is even more interesting. Released in 2018, the RTX 2080 Ti is now eight years old. The flagship graphics card that cost over a thousand dollars back then has long since exited the front line of high-end GPU performance competition. But with the spread of NVIDIA GPU compute unlocking technology, this old graphics card has been brought out by developers for testing again.

Developer nlzy ported the relevant gpu-unlock technology to the NVIDIA Turing architecture and open-sourced the project turing-unlock on GitHub.

The developer tested a Dell RTX 2080 Ti with a non-standard 22GB VRAM configuration. The original RTX 2080 Ti only has 11GB of VRAM, so the hardware configuration of this test card differs from ordinary retail versions. At a 250W power setting, the developer used cuBLAS to conduct a matrix multiplication test with FP16 input and FP32 accumulation. When the three dimensions M, N, and K of the matrix were all 2048, throughput increased from 55.7 TFLOPS before unlocking to 91.4 TFLOPS, an increase of about 64.1%.

Other matrix sizes also showed improvements of varying degrees:

1024: from 50.3 to 68.3 TFLOPS, an increase of about 35.8%;

4096: from 53.7 to 81.7 TFLOPS, an increase of about 52.1%;

8192: from 52.8 to 77.9 TFLOPS, an increase of about 47.5%.

For comparison, NVIDIA's Turing architecture whitepaper gives the corresponding FP16 Tensor Core peak throughput for the RTX 2080 Ti as 56.9 TFLOPS. Calculated according to this figure, the pre-unlock test result at the 2048 size reached 97.9% of the nominal peak, and after unlocking it reached 160.6%. This shows that the original rate configuration did indeed limit the throughput that this compute path could achieve. After adjusting the configuration, the existing Tensor Cores can execute more computations in the corresponding operation modes.

Similar results also appeared in another developer's experiment.

GitHub user haohervchb published a separate Turing-unlock research project, testing an ASUS RTX 2080 Ti. This graphics card uses the TU102 chip, has 68 SMs, and has a standard 11GB VRAM capacity. Unlike nlzy's Linux kernel module, this project uses a UEFI execution scheme before system startup, modifying the GPU's relevant configuration before the operating system loads.

In the low-level MMA (matrix multiply-add) instruction test with FP16 input and FP32 accumulation, throughput before unlocking was 46.94 TFLOPS, and after unlocking it reached 93.66 TFLOPS, almost doubling. Measured by the number of floating-point operations executed per SM per clock cycle, performance also increased from about 511.7 FLOP/SM/clock to 1022.4 FLOP/SM/clock.

However, after switching to an actual cuBLAS matrix multiplication test, the increase narrowed somewhat. In an 8192×8192 GEMM test, throughput increased from 48.15 TFLOPS to 85.17 TFLOPS, an increase of about 76.9%. Although it did not reach the nearly twofold level in the low-level instruction test, for a graphics card released eight years ago, this is still a considerable improvement.

At present, the verification scope of both studies is still limited. The existing evidence can only show that some TU102 graphics cards can increase the throughput of specific Tensor Core operations by modifying the rate configuration. Whether other RTX 20 series models can use the same method, as well as compatibility under different drivers and firmware versions, still need further verification. How much actual AI training, inference, and gaming performance can improve also cannot be directly estimated from these test data.

Some people have also succeeded on Windows 11

The above explorations mainly focused on the Linux environment. As multiple projects have been open-sourced one after another, experimental progress has also appeared on the Windows platform.

Developer TOTORONG published AD103_win_unlocker on Hugging Face, demonstrating an experiment of modifying GPU compute configuration in a Windows 11 environment.

The test used an RTX 4080 with expanded VRAM reaching 32GB, using an AD103 GPU.

Unlike the previous Linux kernel modules, this scheme uses a UEFI program to execute the relevant configuration modifications before Windows starts, and then returns control to Windows.

The test results published by the project show that in 4096-size matrix multiplication:

FP16/FP32: 91.54 → 182.80 TFLOPS.

BF16/FP32: 91.91 → 184.49 TFLOPS.

The throughput of both tests nearly doubled. This means that the relevant low-level configuration modifications are not only achievable on Linux systems.

However, the current Windows 11 results still come from a specifically modified graphics card and cannot yet serve as a basis for general compatibility with ordinary RTX 4080s or the entire RTX 40 series.

Reference links:

This article is from the WeChat public account "Machine Heart" (ID: almosthuman2014), author: Focus on Open Source, published by 36Kr with authorization.