Affiliate Marketing Secrets

Back links from high-PR, authority sites, related to your niche are of course SEO GOLD, especially if they have the right anchor text.
But does Google punish back links from sites with completely unrelated content to your own; or does Google simply consider them less important?
Are the links worth having at all or are they ruining all of the other good SEO work that you’ve put in?
I’d really appreciate it if somebody could clear this up conclusively for me?