I wonder whether it could simply be that it just hurts too much.

Heath Ledger's immediate family - father, mother and older sister - find it difficult enough - who wouldn't.