MambaOut: Do We Really Need Mamba for Vision? 在视觉任务上我们需要Mamba吗?

论文地址
代码地址

知乎解读：王牌飞行员申请出战！
知乎解读：Mamba 模型解读 (一)：MambaOut：在视觉任务中，我们真的需要 Mamba 吗？

Abstract

Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism and subsequently applied to vision tasks. Nevertheless, the performance of Mamba for vision is often underwhelming when compared with convolutional and attention-based models. In this paper, we delve into the essence of Mamba, and conceptually conclude that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. For vision tasks, as image classification does not align with either characteristic, we hypothesize that Mamba is not necessary for this task; Detection and segmentation tasks are also not autoregressive, yet they adhere to the long-sequence characteristic, so we believe it is still worthwhile to explore Mamba’s potential for these tasks. To empirically verify our hypotheses, we construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM. Experimental results strongly support our hypotheses. Specifically, our MambaOut model surpasses all visual Mamba models on ImageNet image classification, indicating that Mamba is indeed unnecessary for this task. As for detection and segmentation, MambaOut cannot match the performance of state-of-the-art visual Mamba models, demonstrating the potential of Mamba for long-sequence visual tasks. The code is available at https://github.com/yuweihao/MambaOut

Mamba 是一种具有类似 RNN 的状态空间模型（SSM）令牌混合器的架构，最近被引入以解决注意力机制的二次复杂性问题，随后被应用于视觉任务。

然而，与卷积模型和基于注意力的模型相比，Mamba 在视觉方面的表现往往不尽如人意。

本文深入探讨了 Mamba 的本质，并从概念上得出结论：Mamba 非常适合具有长序列和自回归特征的任务。

• 对于视觉任务，由于图像分类不符合这两种特征，因此我们假设 Mamba 不需要用于该任务；
• 检测和分割任务也不是自回归任务，但它们符合长序列特征，因此我们认为仍然值得探索 Mamba 在这些任务中的潜力。

为了从经验上验证我们的假设，我们通过堆叠 Mamba 块，同时移除其核心标记混合器 SSM，构建了一系列名为 MambaOut 的模型。

实验结果有力地支持了我们的假设。

• 具体来说，我们的 MambaOut 模型在 ImageNet 图像分类上超越了所有视觉 Mamba 模型，这表明 Mamba 在这项任务中确实是不必要的。

• 至于检测和分割，MambaOut 的性能无法与最先进的视觉 Mamba 模型相提并论，这证明了 Mamba 在长序列视觉任务中的潜力。