Docker container destroy false fails, leading to kitchen reporting failure when tests pass

[NathanZook]

Macintosh:wp_docker_deploy-cookbook nzook$ boot2docker version
Client version: v1.2.0
Git commit: a551732
Macintosh:wp_docker_deploy-cookbook nzook$ docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.3.1
Git commit (client): fa7b24f
OS/Arch (client): darwin/amd64
Server version: 1.2.0
Server API version: 1.14
Go version (server): go1.3.1
Git commit (server): fa7b24f

...

   Finished in 0.31451 seconds
   12 examples, 0 failures
   Finished verifying <default-ubuntu-1404> (0m2.22s).
-----> Destroying <default-ubuntu-1404>...
>>>>>> ------Exception------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #destroy action: [Expected(200..204) <=> Actual(500 InternalServerError)]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration


Macintosh:wp_docker_deploy-cookbook nzook$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
Macintosh:wp_docker_deploy-cookbook nzook$ docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             VIRTUAL SIZE

f4f5ca0cc370 6 minutes ago 563.4 MB

[adnichols]

Can you show me the kitchen.yml for this? Or the output of kitchen diagnose? The subject references "destroy false" so I'm wondering if you are trying to tell kitchen to not destroy the container on success & that's triggering a failure? Or am I misunderstanding?

[NathanZook]

It is attempting to destroy on success, the deamon reports a fail, but the container in fact goes away.

---
driver:
  name: docker

provisioner:
  name: chef_zero
  log_level: debug
  require_chef_omnibus: 11.10.0

platforms:
  - name: ubuntu-14.04
    driver_config:
      socket:  <%= ENV['DOCKER_HOST'] %>
      image: 'docker.util.pages/wp_ops/base-ubuntu:trusty'
      platform: ubuntu
      provision_command:
        - apt-get install -y git
        - export DOCKER_HOST=<%= ENV['DOCKER_HOST'] %>
      dns:
        - 172.25.0.20
        - 172.25.0.21
      privileged: true

suites:
  - name: default
    run_list:
      - recipe[test_wrapper::default]
      - recipe[wp_docker_deploy::default]
    attributes:
      wp:
        base:
          upgrade_packages: []
          install_packages: []
        docker:
          cleanup_interval: 0

[lfjewett]

I just thought it might be helpful to add to this report we are also seeing the same problem. The output is listed below. Out of the last 15 instances only 2 have failed and I have not been able to determine a way to reproduce it. The Docker image 'is' being removed successfully even though the error occurs. The image is 'not' gone prior to running the destroy that fails.

       Finished converging <default-centos> (1m36.42s).
-----> Kitchen is finished. (1m36.89s)
+ kitchen destroy
-----> Starting Kitchen (v1.2.2.dev)
-----> Destroying <default-centos>...
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #destroy action: [Expected(200..204) <=> Actual(500 InternalServerError)
]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

[NathanZook]

The obvious solution is to rescue the exception, check that the fail message matches the pattern, then sleep a second & retry. Of course, we expect that to fail because the image is already deleted, but we can rescue that.... Bleh.

[tknerr]

Same here, reliably reproducible:

• failed build: https://circleci.com/gh/tknerr/sample-toplevel-cookbook/20
• .kitchen.yml: https://github.com/tknerr/sample-toplevel-cookbook/blob/circleci/.kitchen.docker.yml
• circle.yml: https://github.com/tknerr/sample-toplevel-cookbook/blob/circleci/circle.yml

After logging in via ssh I could successfully execute bundle exec kitchen destroy though

[tknerr]

This is likely related to test-kitchen/kitchen-docker#98 (same error behaviour, just different stacktrace)