Number of Canadians travelling to U.S. plummets amid threats