Group activity detection (GAD) is the task of identifying members of each
group and classifying the activity of the group at the same time in a video.
While GAD has been studied recently, there is still much room for improvement
in both dataset and methodology due to their limited capability to address
practical GAD scenarios. To resolve these issues, we first present a new
dataset, dubbed Caf\'e. Unlike existing datasets, Caf\'e is constructed
primarily for GAD and presents more practical evaluation scenarios and metrics,
as well as being large-scale and providing rich annotations. Along with the
dataset, we propose a new GAD model that deals with an unknown number of groups
and latent group members efficiently and effectively. We evaluated our model on
three datasets including Caf\'e, where it outperformed previous work in terms
of both accuracy and inference speed. Both our dataset and code base will be
open to the public to promote future research on GAD.

Group activity detection addresses the identification and classification of group members' activities in videos. This paper introduces a new dataset called Café, designed specifically for GAD, along with a novel GAD model that efficiently handles unknown group sizes and latent group members, achieving superior performance in accuracy and inference speed.