Clothes should be about right for working outside, you've got too much on if they keep you warm when standing still. (unless you pedal symbolicly)
If you're cold, close/add a layer.
If you're hot, open/remove a layer.
Close before going down a long-ish hill, open before climbing a long-ish hill.
Otherwise it's normal to feel cold while starting off, but you should be warming up by 500m.
If your clothes choice for a given day fails miserably, you can drag a brake and push against it to make heat with less wind.
If your feet get cold/numb you may want to get off and jog every now & then.
Theese days I wear normal indoor clothes (two pairs of socks, shirt in the jeans) + parka + gloves + a balaclava under my helmet. Also got a neck thing to prevent air going there.