Skip to content

ggml-cpu: gelu optimizqation using simd#3939

Open
ameya-ms wants to merge 1 commit into
ggml-org:masterfrom
ameya-ms:gelu-opt-simd-am
Open

ggml-cpu: gelu optimizqation using simd#3939
ameya-ms wants to merge 1 commit into
ggml-org:masterfrom
ameya-ms:gelu-opt-simd-am

Conversation

@ameya-ms

@ameya-ms ameya-ms commented Jul 9, 2026

Copy link
Copy Markdown

Implementation:
Copies the same implementation as ggml_vec_silu_f32. Adding for AVX2, AVX512, NEON, and SVE. SSE2 uses the f16 table look-up.
Rewrote GELU function to only use exp instead of tanh.

Testing:

Edge cases vs scalar reference (0, ±inf, nan, saturated tails, tiny values,
and the scalar tail loop) pass on AVX-512, AVX2 and SSE2 builds.
Compiled and run on x86 AVX-512 / AVX2 / SSE4.2 builds. NEON/SVE compile-only
(not run — no ARM hardware).

SSE2 uses old version (everything else uses new version):
Numbers below are strictly comparing old/new version.

=== compiling gelu_sse2.exe ===
gelu_bench.cpp
=== compiling gelu_avx2.exe ===
gelu_bench.cpp
=== compiling gelu_avx512.exe ===
gelu_bench.cpp

#################### SSE2 ####################
N = 16777216 elements, 30 reps
table path: hardware F16C + precomputed f16->f32 table (matches ggml)

impl                   ms/run        Melem/s     max|err|
scalar tanh           265.859           63.1   0 (ref)
f16 table (old)        17.714          947.1   2.02e-03
SSE2 (new)             21.289          788.1   4.77e-07

speedup new vs table = 0.83x,  new vs scalar = 12.49x
checksum (ignore): 33037958.8

edge cases (SSE2 lane path vs scalar reference):
             x             simd           scalar   status
             0                0                0       ok
            -0               -0               -0       ok
             1         0.841192         0.841192       ok
            -1        -0.158808        -0.158808       ok
             5                5                5       ok
            -5     -2.29179e-07     -2.98023e-07       ok
            10               10               10       ok
           -10     -1.20409e-37               -0       ok
            20               20               20       ok
           -20               -0               -0       ok
           100              100              100       ok
          -100               -0               -0       ok
         1e-06            5e-07            5e-07       ok
        -1e-06           -5e-07           -5e-07       ok
           inf              inf              inf       ok
          -inf        -nan(ind)        -nan(ind)       ok
           nan              nan              nan       ok
tail test (n=1007, last 3 elems via scalar tail): max|err| = 4.77e-07   ok
edge-case result: ALL PASS

#################### AVX2 ####################
N = 16777216 elements, 30 reps
table path: hardware F16C + precomputed f16->f32 table (matches ggml)

impl                   ms/run        Melem/s     max|err|
scalar tanh           261.145           64.2   0 (ref)
f16 table (old)        17.836          940.7   2.02e-03
AVX2 (new)              8.384         2001.2   4.77e-07

speedup new vs table = 2.13x,  new vs scalar = 31.15x
checksum (ignore): 33037958.8

edge cases (AVX2 lane path vs scalar reference):
             x             simd           scalar   status
             0                0                0       ok
            -0               -0               -0       ok
             1         0.841192         0.841192       ok
            -1        -0.158808        -0.158808       ok
             5                5                5       ok
            -5      -2.2918e-07     -2.98023e-07       ok
            10               10               10       ok
           -10     -1.20409e-37               -0       ok
            20               20               20       ok
           -20               -0               -0       ok
           100              100              100       ok
          -100               -0               -0       ok
         1e-06            5e-07            5e-07       ok
        -1e-06           -5e-07           -5e-07       ok
           inf              inf              inf       ok
          -inf        -nan(ind)        -nan(ind)       ok
           nan              nan              nan       ok
tail test (n=1007, last 7 elems via scalar tail): max|err| = 4.77e-07   ok
edge-case result: ALL PASS

#################### AVX-512 ####################
N = 16777216 elements, 30 reps
table path: hardware F16C + precomputed f16->f32 table (matches ggml)

impl                   ms/run        Melem/s     max|err|
scalar tanh           264.979           63.3   0 (ref)
f16 table (old)        18.230          920.3   2.02e-03
AVX-512 (new)           6.771         2477.8   4.77e-07

speedup new vs table = 2.69x,  new vs scalar = 39.13x
checksum (ignore): 33037958.8

edge cases (AVX-512 lane path vs scalar reference):
             x             simd           scalar   status
             0                0                0       ok
            -0               -0               -0       ok
             1         0.841192         0.841192       ok
            -1        -0.158808        -0.158808       ok
             5                5                5       ok
            -5      -2.2918e-07     -2.98023e-07       ok
            10               10               10       ok
           -10     -1.20409e-37               -0       ok
            20               20               20       ok
           -20               -0               -0       ok
           100              100              100       ok
          -100               -0               -0       ok
         1e-06            5e-07            5e-07       ok
        -1e-06           -5e-07           -5e-07       ok
           inf              inf              inf       ok
          -inf        -nan(ind)        -nan(ind)       ok
           nan              nan              nan       ok
tail test (n=1007, last 15 elems via scalar tail): max|err| = 4.77e-07   ok
edge-case result: ALL PASS

Notice -inf -> -nan(ind)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant