debugging chef recipes on opsworks instances - access to custom json / data bag attributes

The Custom JSON and the Stack's state are pushed to the instance in a JSON struct when an OpsWorks event (setup, configure, deploy, undeploy, shutdown) occurs. If you want your recipe to see the up-to-the-minute state of the OpsWorks Stack then you need to run your recipe from the OpsWorks UI via the Deploy -> Run Command -> Execute Recipes form.

The JSON sent by OpsWorks is stored on the instance. If you're willing to use potentially-stale Stack state info which is only as fresh as the last time this instance ran an OpsWorks event, you can look for the most recent *.json file on the instance in /var/lib/aws/opsworks/chef, and parse it via Ruby code.

You can also use the opsworks-agent-cli utility on the instance to (re-)run the recipes from an OpsWorks event directly from the command-line on the instance. This utility will re-run OpsWorks events - it won't initiate new events, and it does not pull in a fresh copy of the stack state or the Custom JSON, rather it reuses the .json file that OpsWorks sent to the instance when that event was run originally. For example, in order to rerun the setup event on your instance (because the setup event was definitely run already):

sudo opsworks-agent-cli run_command setup

In order to rerun the same set of recipes you executed last time you ran Execute Recipes from the UI:

sudo opsworks-agent-cli run_command execute_recipes

This kind of sucks, because you need to run the event via the UI first. So if you want to run a custom recipe, or if you want to update custom cookbooks, you first need to run that event from the UI. But, the second, third, and subsequent times you can rerun those events via opsworks-agent-cli.

See here for more about the opsworks-agent-cli.

Using advice from @schlomoswidler about the location of the custom json on the ec2 instance file in his answer above I ran the following to get an interactive chef-shell that includes the custom opsworks attributes I was looking for:

root@mongodb1:/opt/aws/opsworks/current/bin# /opt/aws/opsworks/current/bin/chef-shell -j /var/lib/aws/opsworks/chef/2014-10-27-13-46-53-01.json
loading configuration: none (standalone session)
Session type: standalone
Loading.....done.

This is the chef-shell.
 Chef Version: 11.10.4
 http://www.opscode.com/chef
 http://docs.opscode.com/

run `help' for help, `exit' or ^D to quit.

Ohai2u [email protected]!
chef > node['opsworks']['instance']['layers']
 => ["mongodb"]
chef >

You obviously need to replace the json in /var/lib/aws/opsworks/chef folder with an appropriate file on your system.

Update for Chef 12 Linux OpsWorks Stacks (2016)

Chef 12 Linux based OpsWorks stacks work differently than Chef 11 stacks. One difference is that chef search is now the proper way to access data provided by OpsWorks within a recipe. On the instance, attribute data is now exposed through data bags (stack migration & reference). You can get an overview of available data bags by inspecting the directory for one of your Chef runs. Each data bag has its own sub-directory below /var/chef/runs/<ID>/data_bags/.

[root@asd1 ~]# ll /var/chef/runs/c7f67e3e-c15d-4159-bb14-5bde07751543/data_bags/
total 36
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_app
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_command
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_ecs_cluster
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_elastic_load_balancer
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_instance
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_layer
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_rds_db_instance
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_stack
drwxr-xr-x 2 root root 4096 Nov 23 21:19 aws_opsworks_user
[root@asd1 ~]#

You can't use the same chef-shell technique as I used above for Chef 11 stacks.

The best way that I know of to experiment with search is by using a Pry session to get access to the runtime environment of the second chef run which is dedicated to clients.

Examply Pry Session

In the following example I first trigger the "Execute Recipes" lifecycle event from the UI and use "opsworks_cookbook_demo::foo" as the recipe to run. Then I SSH into my instance and edit /var/chef/cookbooks/opsworks_cookbook_demo/recipes/foo.rb, adding the following two lines.

require "pry"
binding.pry

Then I run opsworks-agent-cli run to repeat the latest run. Unless the latest run was of type "Update Custom Cookbooks", this will leave the local changes in place.

The recipe will be run again, but now we have an interactive shell to experiment. Here's how you can perform two searches:

[root@asd1 ~]# opsworks-agent-cli run
[2015-11-23 21:46:35]  INFO [opsworks-agent(3396)]: About to re-run 'execute_recipes' from 2015-11-23T21:43:15
... lots more output ...
From: /var/chef/runs/76ff2d58-ab8f-4cf6-8744-9562025321fd/local-mode-cache/cache/cookbooks/opsworks_cookbook_demo/recipes/foo.rb @ line 4 Chef::Mixin::FromFile#from_file:

    1: Chef::Log.info "foo"
    2:
    3: require "pry"
 => 4: binding.pry

search(:aws_opsworks_stack)
=> [{"data_bag_item('aws_opsworks_stack', 'f24bd5ea-3ff2-4a1a-a4e4-9298495ae263')"=>
   {"arn"=>"arn:aws:opsworks:us-west-2:153700967203:stack/f24bd5ea-3ff2-4a1a-a4e4-9298495ae263/",
    "custom_cookbooks_source"=>{"type"=>"s3", "url"=>"redacted", "username"=>nil, "password"=>nil, "ssh_key"=>nil, "revision"=>nil},
    "name"=>"susan",
    "region"=>"us-west-2",
    "stack_id"=>"f24bd5ea-3ff2-4a1a-a4e4-9298495ae263",
    "use_custom_cookbooks"=>true,
    "vpc_id"=>nil,
    "id"=>"f24bd5ea-3ff2-4a1a-a4e4-9298495ae263",
    "chef_type"=>"data_bag_item",
    "data_bag"=>"aws_opsworks_stack"}}]

search(:aws_opsworks_instance, "self:true")
=> [{"data_bag_item('aws_opsworks_instance', 'asd1')"=>
   {"ami_id"=>"ami-d93622b8",
    "architecture"=>"x86_64",
    "auto_scaling_type"=>nil,
    "availability_zone"=>"us-west-2a",
    "created_at"=>"2015-11-20T12:48:29+00:00",
    "ebs_optimized"=>false,
    "ec2_instance_id"=>"i-be823867",
    "elastic_ip"=>nil,
    "hostname"=>"asd1",
    "instance_id"=>"42d28e39-29a8-4fdf-a327-afdc23668ff1",
    "instance_type"=>"c3.large",
    "layer_ids"=>["f08fb7e2-9278-498a-8c0d-7d1c1bae22aa"],
… lots more data …

The aws blog post Quickly Explore the Chef Environment in AWS OpsWorks has other examples of using pry on an OpsWorks instance.