"We just got too busy, our schedules didn't match up, and the distance between us didn't help either," the season 2 finalist tells PEOPLE
