Well, in all honesty, between riding on rollers and a trainer, all you are training your body to do is ride on trainers and rollers. There is no trainer or roller on the market today that can accurately simulate riding on the road, or accurately simulate the intensities felt during a race.
I know you said "workouts seems harder" but that's probably because 1) riding on rollers and trainers is boring as shit and 2) you dehydrate MUCH faster when not moving and, as we all know, dehydration = loss in performance. And yes, your body will adapt to the hot, humid conditions of cycling indoors. That's not to say you won't sweat less. In fact, you will probably end up sweating more (increase likelihood of cramping), but the sweat produced will be more dilute. But even when it comes to the conditions you're racing in, you would IDEALLY want your body to adapt to what it's actually like outside, not in your living room. We're not racing in Florida.
And while rollers CAN help improve your bike handling skills, they come nowhere close to the bike handling skills you would need in a peleton of 50+ people.
Now, I'm not totally sure why you ride bike (fitness or performance or whatever), but the bottom line is this; if your goal is performance, the road is the best place to train (preferably in large groups).