Many G protein-coupled receptors (GPCRs) reside within cilia of mammalian cells and must undergo regulated exit from cilia for the appropriate transduction of signals such as hedgehog morphogens. Lysine 63-linked ubiquitin (UbK63) chains mark GPCRs for regulated removal from cilia, but the molecular basis of UbK63 recognition inside cilia remains elusive. Here, we show that the BBSome-the trafficking complex in charge of retrieving GPCRs from cilia-engages the ancestral endosomal sorting factor target of Myb1-like 2 (TOM1L2) to recognize UbK63 chains within cilia of human and mouse cells. TOM1L2 directly binds to UbK63 chains and the BBSome, and targeted disruption of the TOM1L2/BBSome interaction results in the accumulation of TOM1L2, ubiquitin, and the GPCRs SSTR3, Smoothened, and GPR161 inside cilia. Furthermore, the single-cell alga Chlamydomonas also requires its TOM1L2 ortholog in order to clear ubiquitinated proteins from cilia. We conclude that TOM1L2 broadly enables the retrieval of UbK63-tagged proteins by the ciliary trafficking machinery.