Hi Wing,
FCKeditor is designed to edit HTML/XHTML. I'm unfamiliar with MathML, but the w3c page on it here:
http://www.w3.org/Math/
says its intended for machine to machine communication, though I do see info about Putting Math on the Web with MathML.
I think in this case FCKeditor is enforcing XHTML and it doesnt know anything about MathML so anything that looks like tags it is html encoding.
The link above metions an editor called amaya which can apparently handle MathML. I had not heard of this before but perhaps we can implement an editor provider for it at some point but I'm not really sure. It looks like its designed to run on a desktop and upload html files. Maybe it could be configured to talk to a web service and could be made to work with a db driven content management system that way.
Its not a high priority for me personally at the moment unless I get a customer who wants to sponsor support for it but you have at least put it on my radar for the future.
You might also try experimenting with TinyMCE or XStandard editors which are already available in mojoPortal and see if they can handle MathML.
Best,
Joe