That should work fine. As soon as you restore your 2.3.5.4 database to the 2.3.5.5 site, just run setup.aspx in the target site to update its schema to 2.3.5.5.
I definitely wouldn't recommend restoring a newer database on top of an older one, however. I'd first make sure the target site code was at least up to the version of the source before proceeding.