Travelers who owe more than $62,000 in federal taxes risk having their passports revoked. The IRS can inform the State Department of delinquent taxpayers, leading to passport restrictions.