In this paper, we propose a method for calculating the three-dimensional (3D)
position and orientation of a pallet placed on a shelf on the side of a
forklift truck using a 360-degree camera. By using a 360-degree camera mounted
on the forklift truck, it is possible to observe both the pallet at the side of
the forklift and one several meters ahead. However, the pallet on the obtained
image is observed with different distortion depending on its 3D position, so
that it is difficult to extract the pallet from the image. To solve this
problem, a method [1] has been proposed for detecting a pallet by projecting a
360-degree image on a vertical plane that coincides with the front of the shelf
to calculate an image similar to the image seen from the front of the shelf. At
the same time as the detection, the approximate position and orientation of the
detected pallet can be obtained, but the accuracy is not sufficient for
automatic control of the forklift truck. In this paper, we propose a method for
accurately detecting the yaw angle, which is the angle of the front surface of
the pallet in the horizontal plane, by projecting the 360-degree image on a
horizontal plane including the boundary line of the front surface of the
detected pallet. The position of the pallet is also determined by moving the
vertical plane having the detected yaw angle back and forth, and finding the
position at which the degree of coincidence between the projection image on the
vertical plane and the actual size of the front surface of the pallet is
maximized. Experiments using real images taken in a laboratory and an actual
warehouse have confirmed that the proposed method can calculate the position
and orientation of a pallet within a reasonable calculation time and with the
accuracy necessary for inserting the fork into the hole in the front of the
pallet.

提出了一种使用 360 度摄像头计算叉车侧面货架上放置的托盘的三维位置和方向的方法。

利用 360 度图像的水平面投影定位货架上的托盘

Localization of Pallets on Shelves Using Horizontal Plane Projection of  a 360-degree Image

A Colored point cloud, as a simple and efficient 3D representation, has many
advantages in various fields, including robotic navigation and scene
reconstruction. This representation is now commonly used in 3D reconstruction
tasks relying on cameras and LiDARs. However, fusing data from these two types
of sensors is poorly performed in many existing frameworks, leading to
unsatisfactory mapping results, mainly due to inaccurate camera poses. This
paper presents OmniColor, a novel and efficient algorithm to colorize point
clouds using an independent 360-degree camera. Given a LiDAR-based point cloud
and a sequence of panorama images with initial coarse camera poses, our
objective is to jointly optimize the poses of all frames for mapping images
onto geometric reconstructions. Our pipeline works in an off-the-shelf manner
that does not require any feature extraction or matching process. Instead, we
find optimal poses by directly maximizing the photometric consistency of LiDAR
maps. In experiments, we show that our method can overcome the severe visual
distortion of omnidirectional images and greatly benefit from the wide field of
view (FOV) of 360-degree cameras to reconstruct various scenarios with accuracy
and stability. The code will be released at
this https URL

本文介绍了 OmniColor，一种使用独立的 360 度相机对点云进行染色的新颖高效算法，通过在重建几何图像上联合优化所有帧的姿态，克服了全景图像的严重视觉畸变，并且在精确性和稳定性方面从 360 度相机的宽视角中获得了很大的好处。

OmniColor: 一种全局相机位姿优化方法，用于 LiDAR-360 相机融合生成彩色点云

OmniColor: A Global Camera Pose Optimization Approach of LiDAR-360Camera  Fusion for Colorizing Point Clouds

This paper describes a system that generates speaker-annotated transcripts of
meetings by using a microphone array and a 360-degree camera. The hallmark of
the system is its ability to handle overlapped speech, which has been an
unsolved problem in realistic settings for over a decade. We show that this
problem can be addressed by using a continuous speech separation approach. In
addition, we describe an online audio-visual speaker diarization method that
leverages face tracking and identification, sound source localization, speaker
identification, and, if available, prior speaker information for robustness to
various real world challenges. All components are integrated in a meeting
transcription framework called SRD, which stands for "separate, recognize, and
diarize". Experimental results using recordings of natural meetings involving
up to 11 attendees are reported. The continuous speech separation improves a
word error rate (WER) by 16.1% compared with a highly tuned beamformer. When a
complete list of meeting attendees is available, the discrepancy between WER
and speaker-attributed WER is only 1.0%, indicating accurate word-to-speaker
association. This increases marginally to 1.6% when 50% of the attendees are
unknown to the system.

本研究论文介绍了一种使用麦克风阵列和 360 度摄像头生成会议演讲者注释转录的系统，该系统能够处理重叠性语音并使用连续语音分离方法解决该问题。同时，还通过融合脸部跟踪和识别、声源定位、演讲者识别及先前演讲者信息的在线音频视觉演讲者日记化方法，提供了一个名为 SRD 的会议转录框架，并使用 11 名与会者的会议录音的实验结果表明，连续语音分离方法相对于高度调整的波束成形技术可将字词错误率降低 16.1％。当有完整的与会者名单时，字词错误率与演讲者归属字词错误率之间的差异仅为 1.0％，表明字词与演讲者之间的关联准确无误。当 50% 的与会者对系统不知情时，差异略有增加，为 1.6%。

在线音视频会议转录技术进展

Advances in Online Audio-Visual Meeting Transcription

For human drivers, having rear and side-view mirrors is vital for safe
driving. They deliver a more complete view of what is happening around the car.
Human drivers also heavily exploit their mental map for navigation.
Nonetheless, several methods have been published that learn driving models with
only a front-facing camera and without a route planner. This lack of
information renders the self-driving task quite intractable. We investigate the
problem in a more realistic setting, which consists of a surround-view camera
system with eight cameras, a route planner, and a CAN bus reader. In
particular, we develop a sensor setup that provides data for a 360-degree view
of the area surrounding the vehicle, the driving route to the destination, and
low-level driving maneuvers (e.g. steering angle and speed) by human drivers.
With such a sensor setup we collect a new driving dataset, covering diverse
driving scenarios and varying weather/illumination conditions. Finally, we
learn a novel driving model by integrating information from the surround-view
cameras and the route planner. Two route planners are exploited: 1) by
representing the planned routes on OpenStreetMap as a stack of GPS coordinates,
and 2) by rendering the planned routes on TomTom Go Mobile and recording the
progression into a video. Our experiments show that: 1) 360-degree
surround-view cameras help avoid failures made with a single front-view camera,
in particular for city driving and intersection scenarios; and 2) route
planners help the driving task significantly, especially for steering angle
prediction.

通过一个由 8 个摄像头和路线规划器组成的组合实现自动驾驶， Collect driving data on various scenarios and weather conditions and learn a driving model by integrating information from the cameras and the route planner, which can significantly improve steering angle prediction.