The fact is, he has no way of knowing if you put more calories or less into your bike per week than he does. So really his statement was ignorance, but yet understandable. And maybe he just wanted to strike up a conversation about it. I don't think his comment was all that hostile.
If you had some time, you could explain that you also have a car and motorcycle and that when you take your eBike, you are not doing it to have your easy option.
I would have said "No. It depends how you use it. You can pedal just as hard as without it if you want, and the bike then adds the electric power on top of whatever you decide to do. And then you can just go faster but not have less work. If I wanted no work I could just go into throttle mode and not pedal." I then would have offered to let him try it.
Merlin Titanium XTR, Mac 10T, 48v 820Wh, CA V3.