Twitter is a lot of fun and I want to keep it that way. So sometimes I unfollow and sometimes I block. Unfollowing means I don't have much in common with you but I'm happy to talk to you when mention me. Blocking usually means you aren't an anipal. This doesn't mean I think you're a bad person, it only means I don't share your enthusiasm for your Twitter raison d'être. There is no need to be offended. You have hundreds of other followers who hang on your every word.
I just used a web page called Blocked By Me to show the users I have blocked on twitter. One does occasionally block someone by accident.
To use Blocked By Me, log in with twitter and click the Show my blocked users button. The results page is a list of everyone you've blocked. Scan the list and unblock anyone who shouldn't be blocked.
Then I went to a site called Twitblock which examined all my followers for common signs of junk accounts. To use Twitblock, log in with twitter. The results page is a list of followers who act like spammers. Scan the list and unfollow or block anyone who you believe is actually a spammer.
I also use TwitterCounter which shows how many users have followed and unfollowed. If you are concerned with sheer number of followers, you can use this information to make yourself more palatable. Pfffft!!!