To quote the docs:
tox will take care of environment isolation for you: it will strip away all operating system environment variables not specified via passenv.
That is, you just do not mention the variables you don't want passed.
For passenv, docs say:
You can override these variables with the setenv option.
Since a shell variable which is unset looks like an empty string, you can try to setenv to '' all the variables you don't want to pass. It may or may not work, depending on how these variables are used by the code, because there is a difference between an empty env variable and an unset one.
I don't see any way to selectively unset variables in the tox docs. I would suggest to use a few wildcards weaker than ANSIBLE_* so that they don't let in the variables you want to stay unset.
Another solution would be to unset the variables in the shell that runs tox, so that the tox process was unable to access and pass them:
#!/bin/sh
unset ANSIBLE_KEYS_TO_KINGDOM
tox "$@"