Hi Ben,
Yes, after installation you could delete that folder, the only problem would be that if you later bought one of my add on products they rely on the /Setup/Default.aspx page to install them, so maybe delete all but that one or have a backup in case. Otherwise on upgrades you will get the /Setup folder again anyway, so its not a big issue.
Also if disk space is a concern, there are additional ways to slim down the disk usage listed on our FAQ page.
Hope it helps,
Joe