I need to differentiate a 32 bit PE from a 16 bit DOS MZ.
What is the correct way to do it?
I can use heuristics like looking for the PE header, but I feel like it's not necessarily deterministic
All DOS style executables have an 'MZ' as the first two bytes.
To identify an MSDOS executable vs. the multitude of other variants the best bet seems to be to read the position of the relocation table at offset 0x0018 in the file, if this is greater than 0x0040 (into the file) it is not just plain DOS.
To specifically identify the executable as a 'PE' executable there is a pointer at offset 0x003C in the file. This is an offset within the file the will have the bytes 'PE' and two nuls. Other MSDOS 'MZ' variants will use the same location to put other codes eg: 'NE', 'W3', 'LE' etc.
'PE' style executables also come in many forms, I expect you'll be interested in 32bit vs. 64bit at the very least.
Probably the ultimate authority on this sort of thing is the Unix 'file' command, it's designed to reliably identify ANY file type by investigating it's contents. The MSDOS part is listed here. Microsoft is NOT a reliable authority on this because they ignore non-Microsoft information.