A novel algorithm for producing smooth nonlinearities on digital hardware is
presented. The non-linearities are inherently quadratic and have both
symmetrical and asymmetrical variants. The integer (and fixed point)
implementation is highly amenable for use with digital gates on an ASIC or
FPGA. The implementations are multiplier-less. Scaling of the non-linear
output, as required in an LSTM cell, is integrated into the implementation.
This too does not require a multiplier.
The non-linearities are useful as activation functions in a variety of ANN
architectures. The floating point mappings have been compared with other
non-linearities and have been benchmarked. Results show that these functions
should be considered in the ANN design phase.
The hardware resource usage of the implementations have been thoroughly
investigated. Our results make a strong case for implementions in edge
applications. This document summarizes the findings and serves to give a quick
overview of the outcomes of our research\footnote{The authors peer-reviewed
manuscripts (available at this https URL) offer
more detail and may be better suited for a thorough consideration}.

提出了一种新颖的数字硬件上生成平滑非线性的算法，具有对称和非对称变体，实现为无乘法的整数 (和定点) 实现，适用于多种人工神经网络结构的激活函数，在硬件资源使用方面的结果表明，这些函数应该在 ANN 设计阶段考虑到其使用在边缘应用上。

高效非线性计算器

Efficient Non-linear Calculators

We investigate the effective memory depth of RNN models by using them for
$n$-gram language model (LM) smoothing.
Experiments on a small corpus (UPenn Treebank, one million words of training
data and 10k vocabulary) have found the LSTM cell with dropout to be the best
model for encoding the $n$-gram state when compared with feed-forward and
vanilla RNN models. When preserving the sentence independence assumption the
LSTM $n$-gram matches the LSTM LM performance for $n=9$ and slightly
outperforms it for $n=13$. When allowing dependencies across sentence
boundaries, the LSTM $13$-gram almost matches the perplexity of the unlimited
history LSTM LM.
LSTM $n$-gram smoothing also has the desirable property of improving with
increasing $n$-gram order, unlike the Katz or Kneser-Ney back-off estimators.
Using multinomial distributions as targets in training instead of the usual
one-hot target is only slightly beneficial for low $n$-gram orders.
Experiments on the One Billion Words benchmark show that the results hold at
larger scale: while LSTM smoothing for short $n$-gram contexts does not provide
significant advantages over classic N-gram models, it becomes effective with
long contexts ($n > 5$); depending on the task and amount of data it can match
fully recurrent LSTM models at about $n=13$. This may have implications when
modeling short-format text, e.g. voice search/query LMs.
Building LSTM $n$-gram LMs may be appealing for some practical situations:
the state in a $n$-gram LM can be succinctly represented with $(n-1)*4$ bytes
storing the identity of the words in the context and batches of $n$-gram
contexts can be processed in parallel. On the downside, the $n$-gram context
encoding computed by the LSTM is discarded, making the model more expensive
than a regular recurrent LSTM LM.

通过使用 RNN 模型进行 $n$-gram 语言模型平滑来研究其有效的记忆深度，实验结果表明，在保持句子独立性假设的前提下，使用 dropout 技术的 LSTM cell 在编码 $n$-gram 状态方面的表现最佳，且在 $n=9$ 时，LSTM $n$-gram 与 LSTM LM 表现相当，同时在 $n=13$ 时略优于其，该方法可以提高模型的性能，特别适用于模拟短格式文本如语音搜索 / 查询语言模型。

使用循环神经网络估计 N 元语言模型

N-gram Language Modeling using Recurrent Neural Network Estimation

The use of Recurrent Neural Networks for video captioning has recently gained
a lot of attention, since they can be used both to encode the input video and
to generate the corresponding description. In this paper, we present a
recurrent video encoding scheme which can discover and leverage the
hierarchical structure of the video. Unlike the classical encoder-decoder
approach, in which a video is encoded continuously by a recurrent layer, we
propose a novel LSTM cell, which can identify discontinuity points between
frames or segments and modify the temporal connections of the encoding layer
accordingly. We evaluate our approach on three large-scale datasets: the
Montreal Video Annotation dataset, the MPII Movie Description dataset and the
Microsoft Video Description Corpus. Experiments show that our approach can
discover appropriate hierarchical representations of input videos and improve
the state of the art results on movie description datasets.

本论文提出了一种循环视频编码方案，可以发现和利用视频的分层结构，并且改善了电影描述数据集的现有技术结果。