We present a vision-based navigation system for under-canopy agricultural
robots using semantic keypoints. Autonomous under-canopy navigation is
challenging due to the tight spacing between the crop rows ($\sim 0.75$ m),
degradation in RTK-GPS accuracy due to multipath error, and noise in LiDAR
measurements from the excessive clutter. Our system, CropFollow++, introduces
modular and interpretable perception architecture with a learned semantic
keypoint representation. We deployed CropFollow++ in multiple under-canopy
cover crop planting robots on a large scale (25 km in total) in various field
conditions and we discuss the key lessons learned from this.

基于视觉的农业机器人导航系统，利用语义关键点实现在农作物下生长环境中的自主导航。

CropFollow++ 部署的启示：基于关键点的地下农田导航

Lessons from Deploying CropFollow++: Under-Canopy Agricultural  Navigation with Keypoints

Many image-based perception tasks can be formulated as detecting, associating
and tracking semantic keypoints, e.g., human body pose estimation and tracking.
In this work, we present a general framework that jointly detects and forms
spatio-temporal keypoint associations in a single stage, making this the first
real-time pose detection and tracking algorithm. We present a generic neural
network architecture that uses Composite Fields to detect and construct a
spatio-temporal pose which is a single, connected graph whose nodes are the
semantic keypoints (e.g., a person's body joints) in multiple frames. For the
temporal associations, we introduce the Temporal Composite Association Field
(TCAF) which requires an extended network architecture and training method
beyond previous Composite Fields. Our experiments show competitive accuracy
while being an order of magnitude faster on multiple publicly available
datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We
also show that our method generalizes to any class of semantic keypoints such
as car and animal parts to provide a holistic perception framework that is well
suited for urban mobility such as self-driving cars and delivery robots.

本研究提出了一个通用的神经网络结构，使用 Composite Fields 来检测和构建一个多帧中的单一的连接图。同时，引入了 Temporal Composite Association Field (TCAF)，使得这个算法成为第一个实时姿态检测和跟踪算法，并在多个公开数据集上表现出具有竞争力的准确性并且速度快一个数量级。此外，作者也展示了这个方法可以泛化到汽车和动物等多种语义关键点，特别适合城市移动，例如自动驾驶车和送货机器人。

OpenPifPaf：用于语义关键点检测和时空关联的复合场

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and  Spatio-Temporal Association

Semantic keypoints provide concise abstractions for a variety of visual
understanding tasks. Existing methods define semantic keypoints separately for
each category with a fixed number of semantic labels in fixed indices. As a
result, this keypoint representation is in-feasible when objects have a varying
number of parts, e.g. chairs with varying number of legs. We propose a
category-agnostic keypoint representation, which combines a multi-peak heatmap
(StarMap) for all the keypoints and their corresponding features as 3D
locations in the canonical viewpoint (CanViewFeature) defined for each
instance. Our intuition is that the 3D locations of the keypoints in canonical
object views contain rich semantic and compositional information. Using our
flexible representation, we demonstrate competitive performance in keypoint
detection and localization compared to category-specific state-of-the-art
methods. Moreover, we show that when augmented with an additional depth channel
(DepthMap) to lift the 2D keypoints to 3D, our representation can achieve
state-of-the-art results in viewpoint estimation. Finally, we show that our
category-agnostic keypoint representation can be generalized to novel
categories.

本项研究提出一种类别无关的关键点表示法，将多峰热图、关键点的三维位置和规范视角特征组合在一起，取得与特定类别状态最先进的方法相媲美的关键点检测和定位性能，并在视角估计方面取得最先进的结果。

面向类别无关的关键点和视点估计的星图

StarMap for Category-Agnostic Keypoint and Viewpoint Estimation

This paper presents a novel approach to estimating the continuous six degree
of freedom (6-DoF) pose (3D translation and rotation) of an object from a
single RGB image. The approach combines semantic keypoints predicted by a
convolutional network (convnet) with a deformable shape model. Unlike prior
work, we are agnostic to whether the object is textured or textureless, as the
convnet learns the optimal representation from the available training image
data. Furthermore, the approach can be applied to instance- and class-based
pose recovery. Empirically, we show that the proposed approach can accurately
recover the 6-DoF object pose for both instance- and class-based scenarios with
a cluttered background. For class-based object pose estimation,
state-of-the-art accuracy is shown on the large-scale PASCAL3D+ dataset.

本文提出了一种新的方法，使用卷积网络 (ConvNet) 预测语义关键点结合可变形形状模型，从单张 RGB 图像中估计物体的连续六自由度 (6-DoF) 姿态 (三维平移和旋转)，可以应用于基于实例和类别的姿态恢复，且无论是纹理还是无纹理对象均可。实验结果表明，该方法可在杂乱背景下准确地恢复实例和类别物体的 6-DoF 姿态，并在大规模 PASCAL3D+ 数据集上显示出最先进的精度。