The Congress party has faced a significant setback in Assam as its state chief and deputy leader in the Lok Sabha Gaurav Gogoi lost from Jorhat constituency.