Yah, baking soda and oatmeal are not going to do a whole lot. If you want a natural remedy that actually works, try using tea tree oil or a cream with 3%+ green tea extract (Camellia Sinensis)
Studies on both of these remedies have shown them to be as effective as benzoyl peroxide, but with much less irritation.