Affiliate Marketing Secrets

It has not been a while since I started by business, and I am in a very niche industry so I have <5 competitors in my region. One of my competitors has somewhere between 3,000-4,000 backlinks. I know that because I googled: linkto:http://www.website.com
and this particular company is always on the top 3 search result on many keywords I search for.
I am gradually building my backlinks (around 400 now) using directory listings, press releases, technology articles, youtube.. and many other legitimate sources.
another competitor of mine doesnt even show on the google search results, but if I go to linkto:http://www.theirwebsite.com I get almost 1.2 million hits. My theory is, they used one of these backlink building services which created a lot of back-links but made them loose credibility with google.. is this in any way true? are those back-link building websites harmful?
Any other advise you can give?
Thanks