Tom,
The file for setting text attributes including size is style-text.css
If you view the source of the page in the browser you will see that links module renders the list of links inside a ul with li elements using the css classes linkitem and linkaltitem
So look in your style-text.css file for:
li.linkitem a,
li.linkaltitem a { ... }
You can add font-size: 11px; or whatever size you like.
Hope it helps,
Joe