Relaxin 3 is a neuropeptide that plays a crucial role in reproductive functions of mammals, however, its roles in fish reproduction remain to be elusive. To understand the significance of rln3b gene in male fertility, we generated a homozygous mutation line of rln3b gene in Nile tilapia. Our findings indicated that rln3b mutation delayed spermatogenesis and led to abnormal testes structure. Knocking out rln3b gene resulted in a decrease in sperm count, sperm motility and male fish fertility. TUNEL detection revealed a small amount of apoptosis in the testes of rln3b−/− male fish at 390 days after hatching (dah). TUNEL detection revealed a small amount of apoptosis in the testes of rln3b−/− male fish at 390 days after hatching (dah). RT-qPCR analysis demonstrated that mutation of rln3b gene caused a significant downregulation of steroid synthesis-related genes such as Cyp17a1, Cyp11b2, germ cell marker gene, Vasa, and gonadal somatic cell marker genes of Amh and Amhr2. Furthermore, we found a significant down-regulation of hypothalamic-pituitary-gonadal (HPG) axis-related genes, while a significantly up-regulation of the dopamine synthetase gene in the rln3b−/− male fish. Taken together, our data strongly suggested that Rln3b played a crucial role in the fertility of XY tilapia by regulating HPG axis genes.