What about a challenge-response mechanism?
List members passed automatically.
Non-members get an auto-reply with an embedded code.
If they re-reply, their address is added to a second pass list
other than the "member" list.
Filters out the forged from addresses as well as the mass-mailers.
For the few (if any) who abuse the second list, there could be
a manually administered "NEVER" list.
There are list programs that can do this.