Serotonin is a neurotransmitter regulating numerous physiological processes also modulated by drugs, for example, schizophrenia, depression, migraine, and obesity. However, these drugs typically have adverse effects caused by promiscuous binding across 12 serotonin and more than 20 homologous receptors. Recently, structures of the entire serotonin receptor family uncovered molecular ligand recognition. Here, we present a map of 19 'selectivity hotspots', that is, nonconserved binding site residues governing selectivity via favorable target interactions or repulsive 'off-target' contacts. Furthermore, we review functional rationale from observed ligand-binding affinities and mutagenesis effects. Unifying knowledge underlying specific probes and drugs is critical toward the functional characterization of different receptors and alleviation of adverse effects.