Here’s a few things I learned and did when I encountered the very verbose “Resource not found” error from Terraform.
Debug your Infrastructure as Code
More logs?
This is my obvious choice or go-to. Terraform comes with different log levels though it will say itself that every level but TRACE is not to be trusted?
2021/03/02 09:21:33 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility. Use TF_LOG=TRACE to see Terraform's internal logs.
FWIW, DEBUG and ERROR seem to produce okay output to narrow down problems and TRACE seems overwhelming, which is not very helpful.
Refresh, plan?
To narrow down a problem I can run terraform refresh (or import, plan) and hope for the best, but what I found incredibly valuable was adding a -target to either. This allows me to test resources one by one.
To retrieve a list of what is currently known to Terraform’s state:
$ terraform state list
data.openstack_images_image_v2.centos
data.openstack_networking_network_v2.public_network
openstack_compute_instance_v2.jump_host
openstack_compute_keypair_v2.ssh_key
openstack_networking_network_v2.network
openstack_networking_secgroup_rule_v2.jump_host_rule
openstack_networking_secgroup_rule_v2.monitoring_rule
openstack_networking_secgroup_v2.jump_group
openstack_networking_subnet_v2.monitoring
Which seems accurate in my case.
Then I proceeded to go through each of them to find out what I may or may not know:
$ terraform plan -target openstack_compute_keypair_v2.ssh_key
...
Of course, it only failed on the one using literally everything else:
$ terraform plan -target openstack_compute_instance_v2.jump_host
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.openstack_networking_network_v2.public_network: Refreshing state... [id=foo]
data.openstack_images_image_v2.centos: Refreshing state... [id=foo]
openstack_compute_keypair_v2.ssh_key: Refreshing state... [id=foo]
openstack_networking_network_v2.network: Refreshing state... [id=foo]
openstack_networking_subnet_v2.monitoring: Refreshing state... [id=foo]
openstack_compute_instance_v2.jump_host: Refreshing state... [id=foo]
Error: Resource not found
Releasing state lock. This may take a few moments...
Provider
If you’ve read this far, you probably feel my pain. Let’s take a look at the provider and that is in my case the OpenStack provider for Terraform. And this is where I wish I looked yesterday.
The OpenStack provider comes with its own log level: OS_DEBUG=1. This only works with the appropriate Terraform TF_LOG=
statement (spoiler: not TF_LOG=TRACE).
This is what I started out with:
$ TF_LOG=ERROR OS_DEBUG=1 terraform plan -target openstack_compute_instance_v2.jump_host
... [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
Use TF_LOG=TRACE to see Terraform's internal logs.
----
<...snip...>
openstack_networking_subnet_v2.monitoring: Refreshing state... [id=foo]
openstack_compute_instance_v2.jump_host: Refreshing state... [id=foo]
... [ERROR] eval: *terraform.EvalRefresh, err: Resource not found
... [ERROR] eval: *terraform.EvalSequence, err: Resource not found
Error: Resource not found
Releasing state lock. This may take a few moments...
Slightly more helpful (well, not really).
Now re-run the command with TF_LOG=DEBUG and the output will contain API calls made to OpenStack:
... [DEBUG] ..._v1.32.0: Vary: OpenStack-API-Version X-OpenStack-Nova-API-Version
... [DEBUG] ..._v1.32.0: X-Compute-Request-Id: bar
... [DEBUG] ..._v1.32.0: X-Openstack-Nova-Api-Version: 2.1
... [DEBUG] ..._v1.32.0: X-Openstack-Request-Id: bar
... [DEBUG] ..._v1.32.0: 2021/03/02 11:46:21 [DEBUG] OpenStack Response Body: {
... [DEBUG] ..._v1.32.0: "itemNotFound": {
... [DEBUG] ..._v1.32.0: "code": 404,
... [DEBUG] ..._v1.32.0: "message": "Flavor foobar could not be found."
... [DEBUG] ..._v1.32.0: }
... [DEBUG] ..._v1.32.0: }
And this concludes why my terraform plan fails: the flavour I used four months ago is no longer available.
Fin
If I ever get to it, I have to figure out why those error messages are not bubbled up. Or why TF_LOG=DEBUG doesn’t invoke OS_DEBUG=1.
Thank you for reading. Have a great day!