The first time I annealed some aluminum trim, I cleaned off the soot residue with mineral spirits, acetone, and paper towels. It was kind of messy, but worked OK.
Today after I was done I tried plain old SOS scouring pads. It worked fantastic. Not only did it take off the soot, but it polished the aluminum to a nice, not to bright finish. After rinsing it well, the water even beaded up. It does a nice job even if you aren't annealing it.
Al.