The Duke of Sussex also revealed his brother and sister-in-law were perhaps not as close to Megan as they should be