You're OK! It shouldn't cause any problems unless your siding is a different animal than the standard anodized sheets somehow. I use odorless mineral spirits to clean up butyl tape residue and road tar from my anodized aluminum on my teardrop without any noticeable issues. It's one of the mildest solvents out there. I use the OMS with a regular cheep paper towel or a microfiber cloth. You don't want to use blue shop towels! They will scratch your finish!!!

After you use the OMS then wash the trailer with a very mild soapy water and immediately rinse then dry with a microfiber towel.
Cheers,
Dave