<span style="font-weight: bold">
Creating an editor from scratch is a massive undertaking because of the sheer number of browser quirks and bugs with in-browser editing. I've done it several times for various niche projects. My advice would be to start with either TinyMCE or CKEditor as a base and extend them. Both have had unbelievable amounts of development time poured into them over several iterations to get them as good as they are now. Try taking a look at their bug trackers to get an idea of what they have to contend with. Both have decent options for extension, so you could write your own formatting commands to replace
document.execCommand() and in both you can add buttons/tools to the toolbar and context menus.
Self-promotion alert: Another option for the future is to use the commands module I'm working on for my Rangy library. It's some way off completion but will initially have replacements for inline formatting commands offered by
document.execCommand(), and will allow control of the tags/CSS it produces. Rough early demo here: http://rangy.googlecode.com/svn/branches/1point2/demos/commands.html