That is also my general feeling towards patents.
However, I would say it depends on the scale of the sector that the patent falls into. How well developed is the market? Is the patent easy to circumvent? Are there already similar patents that yours is essentially just one of the bypasses of?
It seems one of the best identifiers is how much competition there is. For example I know there are a lot of enforced patents in the snowmobiling sports industry which I feel is largely because there are really only 4 companies that have the specialty to develop products for the market. A good example of a patent that was very worthwhile: https://www.google.com/patents/US7213669. It changed the concept of the product and provided a benefit that was noticeable to the consumer. By having the patent they were able to gain market share and prevent competition from being able to replicate it for ~4-5 years.
If there is more competition it will take more enforcement and it is more likely that someone will find a way to by-pass your patent.