There are several inconsistencies:

• The MC is reborn as an Elf, a species whose status has dropped to slaves during his absence, and he couldn't give a f**k about it (other than when others look at him as a slave), but later on he remarks how they're all his wife's children when he discovers a different species has advanced.
• The floors appear to be static (changes don't reset), so why do any threats still remain? They could've easily cleared out the imps/goblins/other hostile living creatures from entire floors, especially with those return stones. Just send some experienced people to the first floor and mop up from there.
• The whole lower floor = lower tech level thing. It would work if travel only went one way, but it doesn't, so it can't.

the fact that the MC's quest to bury his (in his perception) just-deceased wife starts with picking up a new woman. Seriously, just days after coming back.

If the author ever rewrites this, he needs to remove Kazu. The MC picking up his daughters and going to bury/say goodbye to his wife/their mother would be a much better story than this.