If you configure it for a different port than it should not be trying to use port 25, therefore it seems it is not using the settings you think it is using.
1. Did you check user.config as I mentioned?
2. Possibly you have this setting somewhere:
<add key="EnableSiteSettingsSmtpSettings" value="true"/>
in which case it will not use settings from web.config, but instead will use settings that are managed from Administration > Site Settings
Hope that helps,
Joe