Coming from enduro I wear a lot of protection, as much as I can without going completely over the top. You usually find that after a wee crash you will manage to hurt something. Living with pain is one thing but spending hours on end in A&E waiting for an X-ray is worse in my opinion, followed by trips back for fracture clinics.
Recently started riding trials and I wear the same stuff as I am used to it now and would feel naked without it. The all in one upper body armour is probably the best purchase I have made, even for simple falls that would hurt, you generally don't feel wearing armour. I also wear knee pads and padded shorts. I have had bursitis in my knee after a heavy landing on it and that was with wearing knee gaurds so I don't know what would have happened without it.
If you haven't had any offs that involve injuries that put you out of action for a month or two then you probably wouldn't see the benefit of body armour. Slowly but surely my body armour inventory has increased over the years.... I don't know if this says something more about my riding style right enough.
The only downside with wearing armour is the length of time it takes to get on and off and the fact that you might be a bit over-confident, oh and the cost. I have however seen the other side where I think that it was money well spent. My armour has paid for itself several times over.