Look, filling with oil works well with an outrunner when the oil removes heat from the stator, transfers it to the rotor and then to the external environment, in this case the temperature of the stator (windings) and magnets is equalized, which is also not positive, since the rotor with magnets is an intermediary for transferring heat from the stator, but it works. In the case of an inrunner, God himself ordered the heat from the stator to be transferred directly to the finned and cooled motor housing, so it is better to fill the stator with compound and this is done in industrial machines. This will require a vacuum unit and equipment for a specific motor. This is not a new solution. Winding varnish is used for other purposes - to glue the wires so that the insulation of the cores does not wear out during vibration.I don't think it will really work like this, first, the idea is to fling the oil off the rotor so the rotor will only get as warm as the oil that's picked up, heat won't conduct from the windings to the rotor. Second the rotor is already as hot as that anyway since it's inside with no way to conduct heat other than through the air (ok technically some small amount though the shaft I guess) which is doing the same thing as the oil but worse at moving heat to the water cooled case. And 3rd the rotor is also being heated by magnetic forces and you can overheat the magnets just from that so in some cases that oil may be cooling the rotor. And forth of course removing heat more efficiently in total will keep everything cooler since inside of the motor is more or less one thermal system.
Now will the potting compound also work well, yes and maybe better or worse than oil, no idea. So far I think I haven't seen any EV motors that are potted, some are exterior cooled like the QS we are talking about here and some have oil sprayer systems. Now why is that I don't know, maybe the oil spray works better since they can remove and cool the oil directly, many also use square hairpin windings so the copper is really packed in there and has a lot of contact with everything. Probably also since they can spray the end turns which have a lot of surface area they can suck a lot of heat out. With a wire wound motor though I bet the oil also helps to fill those wire gaps inside the stator better when compared to a hairpin motor.
I think though if you could ensure everything in the motor is compatible with silicone oil, which is probably the most compatible oil anyway and can be found in extremely low viscosities and is highly temperature stable (don't want it getting thick at low temps) may be fairly safe to try. If there is like too much viscous losses or something you could remove the oil, like even if you just got most of it out it would solve the viscous drag issue and the left over would only help conduct heat or you could probably wash it out if you absolutely needed.
Potting on the other hand seems much much higher risk, you mess it up and the motor is done. Really heavily varnishing the windings is the middle ground, a tried and true idea, I've never seen it go wrong and you probably get a fair amount of the gains from the potting by basically gluing all the windings into a block so they can conduct heat to each other and to the stator with no air getting in the way. I did this on my Leaf motor with just a million coats, the stuff is so thin is just soaks right into the windings but you have to repeat the application about a 100 times to get any real buildup.


