Should I worry?

I have a firewall called "Outpost" and the attack detection get's many hits on it. I have it set to the default settings. I downloaded the port reference file from here and checked out the ports that get "attacked". Most of the time it's ports that are said to be known trojan/rats ports. Is this the usual for just browsing the web? Or should I be more cautious about what web sites I visit? Or maybe there are what I've heard called "Script Kiddies" hitting on my PC? Or am I being to paranoid