Modern SEO is all about quality links. As much as possible search engines like Google try determine the highest quality links. Which mean relevance, accuracy, and popularity of the linking domains. Google is getting better at this with every algorithm update.
So it should be clear why irrelevant links can hurt your search engine ranking.
