While Transformer has become the de-facto standard for speech, modeling upon
the fine-grained frame-level features remains an open challenge of capturing
long-distance dependencies and distributing the attention weights. We propose
\textit{Progressive Down-Sampling} (PDS) which gradually compresses the
acoustic features into coarser-grained units containing more complete semantic
information, like text-level representation. In addition, we develop a
representation fusion method to alleviate information loss that occurs
inevitably during high compression. In this way, we compress the acoustic
features into 1/32 of the initial length while achieving better or comparable
performances on the speech recognition task. And as a bonus, it yields
inference speedups ranging from 1.20$\times$ to 1.47$\times$. By reducing the
modeling burden, we also achieve competitive results when training on the more
challenging speech translation task.

为了压缩声学特征并同时保留更丰富的信息，我们提出了逐步向下采样（PDS）的方法，并开发了一种表示融合方法来减少信息损失。可以将声学特征压缩到初始长度的 1/32，同时在语音识别任务上获得更好或相当的性能，并带来 1.20 倍至 1.47 倍的推理加速。我们还在更具挑战性的语音翻译任务训练时取得了竞争结果。