In the pursuit of advancing holographic display technology, we face a unique
yet persistent roadblock: the inflexibility of learned holography in adapting
to various hardware configurations.
This is due to the variances in the complex optical components and system
settings in existing holographic displays.
Although the emerging learned approaches have enabled rapid and high-quality
hologram generation, any alteration in display hardware still requires a
retraining of the model.
Our work introduces a configurable learned model that interactively computes
3D holograms from RGB-only 2D images for a variety of holographic displays.
The model can be conditioned to predefined hardware parameters of existing
holographic displays such as working wavelengths, pixel pitch, propagation
distance, and peak brightness without having to retrain.
In addition, our model accommodates various hologram types, including
conventional single-color and emerging multi-color holograms that
simultaneously use multiple color primaries in holographic displays.
Notably, we enabled our hologram computations to rely on identifying the
correlation between depth estimation and 3D hologram synthesis tasks within the
learning domain for the first time in the literature.
We employ knowledge distillation via a student-teacher learning strategy to
streamline our model for interactive performance.
Achieving up to a 2x speed improvement compared to state-of-the-art models
while consistently generating high-quality 3D holograms with different hardware
configurations.

我们的研究引入了一种可配置的学习模型，通过与现有全息显示的预定义硬件参数相匹配，从 RGB 2D 图像交互地计算 3D 全息图，并实现了与不同硬件配置一致且高质量的生成，同时通过学生 - 教师学习策略实现了 2 倍速度的改进。

可配置的学习全息学

Configurable Learned Holography

With the recent growth in demand for large-scale deep neural networks,
compute in-memory (CiM) has come up as a prominent solution to alleviate
bandwidth and on-chip interconnect bottlenecks that constrain Von-Neuman
architectures. However, the construction of CiM hardware poses a challenge as
any specific memory hierarchy in terms of cache sizes and memory bandwidth at
different interfaces may not be ideally matched to any neural network's
attributes such as tensor dimension and arithmetic intensity, thus leading to
suboptimal and under-performing systems. Despite the success of neural
architecture search (NAS) techniques in yielding efficient sub-networks for a
given hardware metric budget (e.g., DNN execution time or latency), it assumes
the hardware configuration to be frozen, often yielding sub-optimal
sub-networks for a given budget. In this paper, we present CiMNet, a framework
that jointly searches for optimal sub-networks and hardware configurations for
CiM architectures creating a Pareto optimal frontier of downstream task
accuracy and execution metrics (e.g., latency). The proposed framework can
comprehend the complex interplay between a sub-network's performance and the
CiM hardware configuration choices including bandwidth, processing element
size, and memory size. Exhaustive experiments on different model architectures
from both CNN and Transformer families demonstrate the efficacy of the CiMNet
in finding co-optimized sub-networks and CiM hardware configurations.
Specifically, for similar ImageNet classification accuracy as baseline ViT-B,
optimizing only the model architecture increases performance (or reduces
workload execution time) by 1.7x while optimizing for both the model
architecture and hardware configuration increases it by 3.1x.

使用 CiMNet 框架，通过同时搜索最优子网络和硬件配置来创建下游任务准确性和执行指标（如延迟）的帕累托最优前沿，该框架对于 CNN 和 Transformer 系列的不同模型结构都展现了显著的效果。