Revenue Secretary Arvind Shrivastava said that the speculative trade in F&O was resulting in losses to small and retail investors.