In your custom feature you would implement a page to serve as the dialog page and make it not inherit from mojoBasePage and not use the normal master page and styling of the regular CMS pages.
If you are looking for example code that does something similar, take a look at the WebStore feature, if you try it on the demo site you will see if you go to Store Adminsitration there is a link for Order Entry that allows backend creation of orders. Add something to the cart then click checkout. On that page there is a link for user lookup that opens a dialog page to search for existing site users.
Then study the corresponding code in WebStore to see how it works and model your own after it. Making th page open in colorbox is easy, just put a css class of cblink on the link to the dialog page and make sure ScriptLoader has IncludeColorbox="true"
We do have a mojoDialogBasePage that may or may not be useful for you to use as a base class for your dialog page.
Hope that helps,
Joe