With everything I have read, a 2006 V6 Highlander OEM Tow Rating without factory tow package is rated at 2000lbs. The addition of the OE Tow Package increases the tow rating to 3500lbs. It is unlikely a TD will exceed 3500lbs.
Regardless the weight of your TD, I suggest insuring you have either the OE Tow Package which should include a Trans Cooler, or an aftermarket Trans Cooler of equal or higher cooling capabilities. One of the most abused drive-train component on tow vehicles is the transmission. Adding a cooler is one of the most significant improvements one can make to any TV to aid in preventing pre-mature drive-train failures.