You can still sell IP and open source it, you just release it under a GPL license which requires the source code be distributed for the license to be valid, commercial customers typically won't want to release source codes for their products so you negotiate a non-GPL license with whatever terms you want. You're basically using the open source version as free marketing in that case.
Patents are separate but a huge headache, you can try and patent it if it's a novel method. The problem with patents is they're expensive, something like $10k USD+ per patent depending on the region, then even after you pay all the money the patent examiners they don't check for prior art, so you can have a registered patent that is not worth anything. Then even if you have a patent the government won't enforce it for you, if you find a large company is using it you have to sue them via a civil lawsuit, you will then waste about 2 to 5 years worth of legal expenses trying to enforce it. Eventually after all that if the opposing corporate lawyers fail to find any prior art or invalidate another way the patent it will become quite valuable but to get to that stage it would have cost 10 or 20 million.
Neither of them are mutually exclusive, it's just a matter of working out if patents are worth the investment since you have to do them first.