This was a big mistake. I should have finished the walls and trim after they were attached so I could have used much stronger Titebond III glue instead of Silicone II caulk. I somewhat compensated for this with lots of screws and making two cleats for each wall