Authority of Internal Revenue Enforcement Officers - Title 26, United States Code
The United States Code provides the authority for everything an IRS employee does with regard to you. Click here to view Section 7608. It spells out the specific subject matters of taxation with which Internal Revenue Officers are charged. Strangely, income tax is not mentioned.
So, the next time you tired of being harassed by an IRS agent about your alleged "income tax liability", point out this section to him/her and inquire whether he has any evidence that you have been involved in revenue taxable activities having to do with alcohol, tobacco or firearms. Suggest to him that if he doesn't have any such evidence, you would appreciate being left in peace.
Or, in the alternative, if he/she insists that you are the subject of an income tax investigation, ask if he/she is the criminal investigator mentioned in Section 7608(b), as that is the only place "income tax" is mentioned. Then, of course, according to the Internal Revenue Manual, as a criminal investigator he/she would only be conducting an investigation if you were a U.S. citizen living abroad or a non-resident alien having a filing requirement.
Section 7214 enumerates fairly severe penalties (hefty fines and prison sentences) for IRS employees who operate outside their authority.
Somehow, in my experience, that section does not seem to slow their often overzealous, extra-legal activities. This might be explained by the document linked here.