I'm part of a team that's updating a website for a bank.
The web pages use an XHTML doctype:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
As you stated in the comments, you are serving your "XHTML" as HTML, and not true XHTML in the first place. I don't recall if this comes under the definition of "tag soup".
The new standard doctype is backwards compatible for older browsers and works as far back as IE6; maybe even IE4. Since you are already serving HTML, and not XHTML, you shouldn't have issues with changing to the new one but the only way to be sure is to test it.
However, HTML is HTML no matter which doctype you use, and that's for validation purposes only.
The reason browsers will work with your HTML5 markup is because they are required to "do the best they can" with invalid markup. Since HTML5 is not invalid, and they know how to handle it, they do their best.
There are no security or performance issues.
It might help to read this article: Activating Browser Modes with Doctype