What happens if a browser doesn't support a CSS pseudo-class, like
Browsers currently make no distinction between "unrecognized" or "unsupported" selectors, and "invalid" selectors. If a browser recognizes a selector, generally it'll have implemented it to the best of its ability (and any behavior that's not to spec can be classified as a bug on its bug tracker), even if it doesn't implement all other features in the same level of Selectors (as is currently the case with
:dir() for example, and historically IE7/IE8 with level 3 attribute selectors, and IE6 with the universal selector). If it doesn't recognize the selector, it follows CSS2.1 §4.1.7 to the letter and drops the entire ruleset, no questions asked. Notice that it says
When a user agent cannot parse the selector (i.e., it is not valid CSS 2.1), it must ignore the selector and the following declaration block (if any) as well.
which implies that if a user agent cannot parse a selector, it must therefore be invalid CSS2.1 (or invalid in some other level of Selectors). But this assumes the user agent conforms fully to the standard; we all know that in reality, different implementations have different levels of conformance to each standard. So I treat this as "When a user agent cannot parse the selector" without the parenthetical, and I imagine browser vendors do the same. Note also that Selectors itself makes no distinction between a pseudo-class token with an ident or function that doesn't correspond to a valid pseudo-class, and a series of characters that cannot even be parsed as a pseudo-class — they're both equally invalid — see section 12 of css3-selectors and section 3.9 of selectors-4.
I've not heard of any browser that recognizes a selector as valid for the purposes of error handling, but either ignore it, or outright refuse to match any elements against the entire complex selector even when it's supposed to.
WebKit used to have a really bad habit of accepting CSS rules with unrecognized pseudo-elements, allowing things like
::selection, ::-moz-selection, which proved useless anyway because every other layout engine followed the spec more strictly. I believe WebKit no longer does this, however, but you know how WebKit is with these things.
On the standards side, selectors-4 is set to change this with the static and dynamic profiles, and IIRC it was suggested in a CSSWG discussion (I don't remember if it actually was) that a selector that's in the static profile but not the dynamic profile should be treated as valid, but not match any elements, in a dynamic matching context, though my email on the subject went unanswered.