I have a strange case of trying to combine two pseudoclasses:
I assumed css wouuld filter the :not items first before applying the :nth-of-type
Nope. CSS is fully declarative; every selector is a simple condition that is true or false independently of any selector part. It's not a procedural language where you take a set and process it, narrowing it down with each step. A selector language with procedural rules would be immune to many kinds of optimisation and would be slower.
nth-of-type is only about position within an element's parent, and not position in a 'results list so far' because CSS selectors have no such concept. A selector engine could look up the test for
nth-of-type before narrowing it with
not, as the rules do not interfere with each other.