Steampipe Brings SQL to AWS Cloud APIs

A number of database administrators from SQL, Oracle and other relational database services have transitioned into cloud engineering in AWS over the last few years. Most of the still yearn and miss their SQL queries to pull out the data. 

I have been using Boto3 API calls in my Python and NodeJS script for sometime to pull out data out of AWS services. This data then gets pushed into some of the MySQL RDS instance or an Oracle express edition in the tables, so that I could run some SQL queries to slice and dice this data as per my requirement. Selecting specific columns, filtering with where clause, grouping with Group by and various other SQL native features bring this AWS data to life. But as you can already imagine, this is lot of work and extra overhead not to mention the cost.

This is where Steampipe enters into the picture. As per Steampipe, Steampipe exposes APIs and services as a high-performance relational database, giving you the ability to write SQL-based queries to explore dynamic data. Mods extend Steampipe's capabilities with dashboards, reports, and controls built with simple HCL.

It's sort of live database for which you don't have to create any database infrastructure. It uses your native credentials for AWS and their plugin does the magic. One thing I have encountered while using it is that it starts lagging when there is multi-account scenario is in play and you would face few issues especially at the data retrieved volume increases. But for starters and single account, it works like the charm and you can rely on good old Structured query language.