HF RL Explorer

Feature Request: Fused RMSNorm + RoPE Operation and Hybrid Attention for Gemma4

Feature Request: Fused RMSNorm + RoPE Operation and Hybrid Attention for Gemma4: a task in LegoFlow-SWE (Harbor dataset). In the current Gemma4 implementation, the RMSNorm and rotary positional embedding (RoPE) operations are applied as separate kernels for the query and key projections. This…

The task

In the current Gemma4 implementation, the RMSNorm and rotary positional embedding (RoPE) operations are applied as separate kernels for the query and key projections. This leads to unnecessary intermediate tensor allocations and memory traffic, especially for large models, and adds latency to the forward and backward…

Part of Lego-X/LegoFlow-SWE.