Prince Harry has declared he didn't want his "job" as a member of the royal family because "it killed my mum".