Cognitect’s aws-api
tl;dr aws-api is a fun way to interact with AWS services.
I got an email from Amazon warning me about the impending EOL of Node 8 Lambdas. Alas, this wasn’t the first such email they’ve sent. But the last time, I acted! I went region by region, in the AWS console, like an animal, and updated any outdated Lambdas with outdated runtimes I found. This was tedious, but not overwhelming, since I’ve only got a handful of Lambdas in the account. I was dismayed to discover my efforts were insufficient.
Clearly, I wasn’t going to go through the AWS console again. Switching regions in the console is slow, which is fine. It’s usually an infrequent task. Initially, I reached for the AWS CLI and my ol’ pal jq. Then I remembered I’d been wanting to mess around with Cognitect’s aws-api library for a while, and this seemed like a golden opportunity.
I created a new folder to house the code,
and set up the following deps.edn:
deps.edn
{:deps {com.cognitect.aws/api {:mvn/version "0.8.408"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.689"}
com.cognitect.aws/lambda {:mvn/version "780.2.584.0"}}}
The first two
(api and endpoints)
are required;
then you pull in the specific services you need à la carte
(lambda, in this case).
Cognitect publishes the version numbers for each service in the repo.
From here, getting started is pretty easy. I fired up a REPL, and ran the following:
(def lambda (aws/client :api :lambda))And now I can start programatically exploring the API:
(aws/ops lambda)
aws/ops returns a big nested map of all the things you can do with that service,
as well as the shapes of the request and response data—even the documentation.
Since it’s all plain data, it’s easy to start working with it with the usual Clojure tools to get exactly what you need.
Maybe I just want to see a list of just the names of the operations.
(->> (aws/ops lambda) vals (map :name))
(I could also have just grabbed the keys,
but they’re keywords and I’m going to want strings in the next step. 😀)
I’m still getting a big list—there’s a lot you can do with Lambdas!—so howsabout I filter it to just the List operations?
(require '[clojure.string :as str])
(->> (aws/ops lambda)
vals
(map :name)
(filter #(str/starts-with? % "List")))("ListAliases"
"ListVersionsByFunction"
"ListProvisionedConcurrencyConfigs"
"ListLayers"
"ListFunctionEventInvokeConfigs"
"ListLayerVersions"
"ListEventSourceMappings"
"ListFunctions" ⬅ 👀
"ListTags")
ListFunctions sounds good.
Let’s find out a little more about it.
(-> (aws/ops lambda) :ListFunctions){:name "ListFunctions",
:documentation "<p>Returns a list of Lambda functions, with the version-specific configuration of each.</p> <p>Set <code>FunctionVersion</code> to <code>ALL</code> to include all published versions of each function in addition to the unpublished version. To get more information about a function or version, use <a>GetFunction</a>.</p>",
:request {:MasterRegion string, :FunctionVersion [:one-of ["ALL"]], :Marker string, :MaxItems integer},
:required nil,
:response {:NextMarker string,
:Functions [:seq-of
{:Environment {:Variables [:map-of string string], :Error {:ErrorCode string, :Message string}},
:LastModified string,
:CodeSha256 string,
:Timeout integer,
:KMSKeyArn string,
:VpcConfig {:SubnetIds [:seq-of string], :SecurityGroupIds [:seq-of string], :VpcId string},
:LastUpdateStatus [:one-of ["Successful" "Failed" "InProgress"]],
:DeadLetterConfig {:TargetArn string},
:FunctionName string,
:LastUpdateStatusReason string,
:LastUpdateStatusReasonCode [:one-of
["EniLimitExceeded"
"InsufficientRolePermissions"
"InvalidConfiguration"
"InternalError"]],
:TracingConfig {:Mode [:one-of ["Active" "PassThrough"]]},
:FunctionArn string,
:StateReason string,
:MasterArn string,
:RevisionId string,
:Description string,
:CodeSize long,
:Layers [:seq-of {:Arn string, :CodeSize long}],
:State [:one-of ["Pending" "Active" "Inactive" "Failed"]],
:Runtime [:one-of
["nodejs"
"nodejs4.3"
"nodejs6.10"
"nodejs8.10"
"nodejs10.x"
"nodejs12.x"
"java8"
"java11"
"python2.7"
"python3.6"
"python3.7"
"python3.8"
"dotnetcore1.0"
"dotnetcore2.0"
"dotnetcore2.1"
"nodejs4.3-edge"
"go1.x"
"ruby2.5"
"provided"]],
:Role string,
:MemorySize integer,
:Version string,
:Handler string,
:StateReasonCode [:one-of
["Idle"
"Creating"
"Restoring"
"EniLimitExceeded"
"InsufficientRolePermissions"
"InvalidConfiguration"
"InternalError"
"SubnetOutOfIPAddresses"]]}]}}
Jeepers, that’s a lot of information. I’m really only interested in what I need to send a request, and what I’ll get back in the response.
(-> (aws/ops lambda)
:ListFunctions
(select-keys [:request :required])){:request {:MasterRegion string, :FunctionVersion [:one-of ["ALL"]], :Marker string, :MaxItems integer},
:required nil}
:required nil?
That’s good to hear:
I can just start off sending a “bare” request,
and punt on thinking about parameters for now.
Next, what does the response give me?
(-> (aws/ops lambda)
:ListFunctions
:response){:NextMarker string,
:Functions [:seq-of
{:Environment {:Variables [:map-of string string], :Error {:ErrorCode string, :Message string}},
:LastModified string,
:CodeSha256 string,
:Timeout integer,
:KMSKeyArn string,
:VpcConfig {:SubnetIds [:seq-of string], :SecurityGroupIds [:seq-of string], :VpcId string},
:LastUpdateStatus [:one-of ["Successful" "Failed" "InProgress"]],
:DeadLetterConfig {:TargetArn string},
:FunctionName string,
:LastUpdateStatusReason string,
:LastUpdateStatusReasonCode [:one-of
["EniLimitExceeded"
"InsufficientRolePermissions"
"InvalidConfiguration"
"InternalError"]],
:TracingConfig {:Mode [:one-of ["Active" "PassThrough"]]},
:FunctionArn string,
:StateReason string,
:MasterArn string,
:RevisionId string,
:Description string,
:CodeSize long,
:Layers [:seq-of {:Arn string, :CodeSize long}],
:State [:one-of ["Pending" "Active" "Inactive" "Failed"]],
:Runtime [:one-of
["nodejs"
"nodejs4.3"
"nodejs6.10"
"nodejs8.10"
"nodejs10.x"
"nodejs12.x"
"java8"
"java11"
"python2.7"
"python3.6"
"python3.7"
"python3.8"
"dotnetcore1.0"
"dotnetcore2.0"
"dotnetcore2.1"
"nodejs4.3-edge"
"go1.x"
"ruby2.5"
"provided"]],
:Role string,
:MemorySize integer,
:Version string,
:Handler string,
:StateReasonCode [:one-of
["Idle"
"Creating"
"Restoring"
"EniLimitExceeded"
"InsufficientRolePermissions"
"InvalidConfiguration"
"InternalError"
"SubnetOutOfIPAddresses"]]}]}
That’s still a lot of information,
but at a glance I see there’s a :Functions entry which is a :seq-of maps describing the Lambda functions.
This is readable stuff!
What’s in those maps?
Specifically,
is :Runtime in there?
I could scroll through this blog entry,
or I could let Clojure do the work.
(-> (aws/ops lambda)
:ListFunctions
:response
:Functions
second ; first is :seq-of; second is the entry spec
:Runtime)[:one-of ["nodejs" "nodejs4.3" "nodejs6.10" "nodejs8.10" ⬅ 👀 "nodejs10.x" "nodejs12.x" "java8" "java11" "python2.7" "python3.6" "python3.7" "python3.8" "dotnetcore1.0" "dotnetcore2.0" "dotnetcore2.1" "nodejs4.3-edge" "go1.x" "ruby2.5" "provided"]]
Jackpot!
Putting it together, now I can
(finally!)
make a request with aws/invoke:
(map (juxt :FunctionName :Runtime)
(:Functions (aws/invoke lambda {:op :ListFunctions})))(["function-1" "nodejs10.x"] ["function-2" "nodejs10.x"] ["function-3" "nodejs10.x"] ["function-4" "nodejs10.x"])
Well, it looks like this region is okay. But I want to know about all the regions, on the off chance that I drunkenly set up a rogue Lambda in a region on the other side of the planet. Sure would be cool if there were a way to programatically get a list of regions…
deps.edn, part 2
{:deps {com.cognitect.aws/api {:mvn/version "0.8.408"}
com.cognitect.aws/endpoints {:mvn/version "1.1.11.689"}
com.cognitect.aws/ec2 {:mvn/version "780.2.53.0"}
com.cognitect.aws/lambda {:mvn/version "780.2.584.0"}}}(def ec2 (aws/client {:api :ec2}))
(def regions (->> (aws/invoke ec2 {:op :DescribeRegions})
:Regions
(map :RegionName)
(into #{})))
(first regions)"ap-northeast-1"
I elided a lot of the discovery step and just jumped to punchline,
but the same aws/ops discovery process works on an EC2 client just as well as a Lambda client.
And aws/invoke is called the same way,
just with an :op for EC2.
Now that I have my list of regions, time to look for offending Lambdas:
(mapcat
(fn [region]
(let [lambda (aws/client {:api :lambda :region region ❶})
fs (:Functions (aws/invoke lambda {:op :ListFunctions}))]
(map (juxt (constantly region) :FunctionName :Runtime) fs)))
regions)
(["us-west-2" "function-1" "nodejs10.x"] ["us-west-2" "function-2" "nodejs10.x"] ["us-west-2" "function-3" "nodejs10.x"] ["us-west-2" "function-4" "nodejs10.x"] ["us-east-1" "function-5" "nodejs10.x"])
(Note ❶ in the above code;
I set up a new client for each regoin,
passing :region as another argument to aws/client.)
Cool!
So I've got all my Lambdas,
from all across the world.
But, wait a minute …
they’re all already running on the nodejs10.x runtime.
What gives?
Turns out, the problem was not with the current versions of the Lambdas. One of the Lambdas was a “Lambda@Edge” function leveraged by CloudFront, and merely updating the runtime of the Lambda is insufficient; you need to configure the CloudFront trigger again. This tells me I need to be more disciplined about moving my Lambda setup into a managed system like Terraform. (Though I dislike the ergonomics of working with Lambdas in Terraform.)
So while my aws-api adventures didn’t yield much fruit in terms of directly solving my problem, it gave me a hint and I was able to move on to solve the next problem. Also, I had a blast using aws-api to explore the AWS API. Being able to drill down into requests and responses with Clojure was great, and as always, the experience is magnified with the tight feedback loop afforded by REPL-based development. If I feel like this code will come in handy again some day, it wouldn’t take much to clean it up into a real program.
Questions? Comments? Contact me!
Tools Used
- Clojure
- 1.10.1
- com.cognitect.aws/api
- 0.8.408
- com.cognitect.aws/ec2
- 780.2.583.0
- com.cognitect.aws/endpoints
- 1.1.11.689
- com.cognitect.aws/lambda
- 780.2.584.0