Hi Kerry,
There is a property on FileBrowserTextBoxExtender for BrowserType="file"
there are 3 valid values for it file, image, and media, file is the default.
this is really just a parameter that gets passed to the FileDialog.aspx page to indicate what kinds of files to support. But that dialog doesn't really support any more specific control other than those 3 types.
You could use a RegularExpressionValidator on the textbox to further limit what the user can choose but it won't prevent him from choosing other file types in the dialog, just from saving them in your feature.
So if those don't satisfy your needs you would have to implement your own modified versions of FileBrowserTextBoxAppender and of FileDialog.aspx. You could copy those into your custom feature, change the names and namespaces then modify them as you see fit. You may also need to implement your own version of /Services/jQueryFileTreeMediaBrowser.ashx
Hope that helps,
Joe