Adverse means unfavourable or harmful, and describes conditions or effects — adverse weather, adverse reaction. Averse describes a person's attitude: unwilling or strongly disinclined, almost always followed by "to". If a person feels it, use averse; if it happens to them, use adverse.
The core difference
One describes circumstances; the other describes a disposition.
- adverse — unfavourable, harmful: adverse conditions, an adverse effect.
- averse — reluctant, disinclined: averse to risk, not averse to a drink.
The common error
"Adverse to change" is the frequent slip; it should be averse to change, because a person is doing the feeling. The tell is the preposition: averse almost always takes to, while adverse normally sits directly in front of a noun.
The roots
Both are from Latin vertere, to turn. Ad- is towards — turned against you. A- is away — turned away from. Adverse conditions turn against you; an averse person turns away.