In "itchy pants syndrome" you get rash if you wear new "pants" without washing them before the first use. The problem here is not in fabric itself, but probably in some substances used in final phases of clothe production.
Wool can often cause or aggravate eczema.
Rough flax may physically iritate skin, I've tried with one shirt, oh my.