Hi Johan,
If it fails to load your master page it will log an error and fall back to /App_MasterPages/layout.master
If you aren't seeing anything in the log most likely it cannot write to the log file. I would delete the currentlog.config file, if permissions are correct it will come back automatically, if it does not, then permissions are not correct on /Data folder.
Errors may not be logged instantly when they happen, log4net can buffer them in memory and flush them to the log file in batches.
It could be a file permission issue on your layout.master, if it cannot read the file it will not be able to use it
/Data/systemfiles is only used by some features, specifically the Folder Gallery stores its generated web size and thumbnails there.
Problem 2 sounds like it had cached an empty site map and when you added a page it touched the dependency file and cleared the cache and this time it was able to load the sitemap correctly.
I don't see any reason to believe that a change in db platform from SQL 2005 to SQL 2008 has any relation to the problems you describe.
Hope it helps,
Joe