FCKeditor is a legacy product no longer actively developed, the newer CKeditor is actively developed. I've found that using FCKeditor in IE 9 the browser server dialog does not work at all and throws errors in the IE 9 developer tools console. I don't know if they will ever release a new version of FCKeditor with a fix for this. It works fine in CKeditor.
On the demo site the editor file system is configured to only allow browsing the /Data/Sites/[SiteID]/media folder by this setting:
<add key="ForceAdminsToUseMediaFolder" value="true" />
possibly you have the same setting in your installation.
Hope it helps,
Joe