"If you had a patent that you felt others were infringing, would you just sit back and do nothing?"

Actually, much of the industry does just that, or used to. A good example is Cisco, who only uses its patent portfolio for defensive purposes.