If you make a copy and modify it you should be able to figure it out by looking at the existing code which has 3 different callbacks based on the 3 different editors which each have their own needs. So each editor is passing in a querystring param to tell the page which editor it is so it knows how to setup the correct javascript for the callback.
Basically you have to have a javascript method in the main page to receive the value and the dialog just calls window.opener.methodname(url) and then closes itself. methodname is the name of the javascript method in the parent page. The javascript method in your main page would store the url in a hidden field and or apply it to an <img src= in your page. Then on postback you get the value from the hidden field and persist it.
To open your custom dialog you might look at using the /Controls/GreyboxHyperlink to open it in a modal window. Then just style it as you would any other link by using a css class.
Hope it helps,
Joe