If you try to use an application-specific password in the call to Api.new, you get the error:
GoogleVoice::Api::InvalidLoginException: Cannot login to Google Voice with foo@bar.com.
If you use your actual password, you get the same error, but your mobile device will receive a verification code as an SMS. You can't really use this programmatically. (Also, the verification would expire after 30 days, anyway.)