I'm a bit surprised that if you've known about this issue from when the bike was new, that you haven't investigated it by now. Try the simple solutions first. As Scott123007 suggested, a common complaint of this type is simply dirt or lack of lubrication on the shifter pivot. Some people have even added a zirc fitting to make it simple to grease. The shaft has a groove there from the factory.
Key west is a fun spot. Just sucks getting there and getting out of there.
I hope one of your bikes is a scooter! Perfect spot for scooter riding.
What mileage do you have on the bike at this time? Others have offered up good ideas, but having a physical problem with the clutch is quite rare. More common are simple things like the lubing of the pivot or a poor bleed job on the clutch hydraulic system. Further complications such as a leaking hydraulic system or contaminated clutch plates due to incorrect oil could also play a part too. And operator error is not ruled out quite yet, as you did not specify the conditions under which you have the issue other than when the bike is hot.
I try not to assume either way, so is your normal downshifting process when this occurs when you are slowing down or otherwise moving? Or after you have come to a stop?
Have you ever bled the clutch system? Or ever changed the fluid? If you changed the fluid, what did you use?
What engine oil are you using? How old is it? (miles & time) It's common to notice an increase in shifting effort with oil that's due for a change. Oils with "energy conserving" marked on the bottle have friction modifiers that don't play well with wet clutches.
After these simple issues, then it gets into physical problems that have occurred. There is a shift return spring that has broken inside on a few Super Teneres, but it's not common. There have been some sticking clutch friction disks too, usually on bikes that sat for very extended periods before being sold. And as Twinrider notes, there have been a couple of damaged clutch units too, but these are odd manufacturing faults, not common issues.
Last, if you have the extended YES warranty, it might be useful to have the dealer look at it for you as well. I have no idea off the top of my head where your nearest Yamaha dealer is, but FL has a few.