Councillors have backed plans for new homes on the former Warren House site within the conservation area.