I am trying to drop all the outgoing RST and incoming RST on all ports. I am using Debian linux. I tried all possible combinations of commands listed on the internet but nothing seems to work.
For example, i tried:
iptables -A OUTPUT -o eth0 -p tcp --tcp-flags RST RST -j DROP
iptables -A OUTPUT -p tcp --tcp-flags RST RST -j DROP
iptables -A INPUT -p tcp --tcp-flags RST RST -m state --state RELATED,ESTABLISHED -j DROP
iptables -A INPUT -p tcp --tcp-flags RST RST -j DROP
Hmm, it's quite possible that this is going through the forward chain rather than the input or output chain, since you are running this on the host machine.
A trick to debug this is to use iptables -L -v, this displays counts of how many packets go to each rule, if you set up a command that send lots of packets like this
watch --interval 0.1 "nc remote.machine CLOSED_PORT"
you can work out which rule is getting hit. You should also be aware that there are orthogonal tables - sets of rule chains used in different situations (e.g for nat). It might be worth looking in the NAT table - since your virtual host might be NAT'ing through your host rather than having it's own IP address
iptables -L -v -t nat
It would be informative to know what IP address the virtual host has - since if this is disjoint from your network's subnet this will probably be being NAT'ed.