I usually don't much care about port scanners. Almost everybody does it nowadays. However, where I do draw a line is when someone attempts to actually SSH brute force in, trying every possible password combination (go ahead you, won't ever get in) to SSH service that is running on non-standard port. Brute forcers will get permanently logged,permanently blocked and then end up into my personal "List of ********" like this: https://www.orwell1984.today/ip_list.php
