First piece of advice. I'm an idiot that is on the Internet, don't listen to me. Second, If I tell you something, experiment and try it for yourself before you apply it to your build. Now, on to your question.
I think it's possible, however, I would make a slanted roof for rain to run off. This may require the purchase of additional foam.
As for the pinning system to put the walls together, I will tell you my secret idea that I've been playing with.

So don't tell anybody.
Take the panels and frame them with wood that is glued to the edge of the foam. Then get door hinges and screw them to the frames in the corner where the two walls meet. For an eight foot tall wall, I'd use 3 hinges per corner. To quickly disconnect the walls, pull the pins out of the hinge. When assembling, put the pins back in the hinges. Simple and strong. For an added bit of strength, use a type of tongue and grove for the frames to fit together.
As for how to keep the structure attached to the trailer, for windy conditions and such. There are lag screws that are threaded differently on both ends. One end is for screwing into wood, the other end accepts nuts. On the bottom of the walls, maybe 4 per wall, screw these lag bolts into the wall frame. Drill holes in the wood deck of your trailer that line up with your lag screws on the bottom of the wall.
So to assemble your quickie camper, first put the pins in the hinges and assemble the walls together. Then line up the lags next to the holes and push them into the holes, then put the nuts on from under the trailer.
As for the roof, I'd frame it with an apron that slide over the outside walls. Then I'd attach the roof in a similar way.
Good luck with the build and with your racing. Be sure to post pictures of your build.
Never argue with an idiot. They will bring you down to their level and beat you with experience.