One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue.

Running AWS Minimal Linux instance got this error in the logs on start up:

Starting cloud-init: Cloud-init v. 0.7.6 running 'modules:config' at Tue, 06 Jun 2017 09:10:02 +0000. Up 18.86 seconds.
Loaded plugins: priorities, update-motd, upgrade-helper

 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Disable the repository, so yum won't use it by default. Yum will then
        just ignore the repository until you permanently enable it again or use
        --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>

     4. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true


There's a lot of information there but if you scroll down a bit farther you'll see something like this:

Could not retrieve mirrorlist http://repo.us-west-2.amazonaws.com/latest/main/mirror.listerror was

12: Timeout on http://repo.us-west-2.amazonaws.com/latest/main/mirror.list: (28, 'Connection timed out after 10001 milliseconds')

This means the network configuration is blocking access the Amazon Yum Repo. 

When I look up the IP address associated with the above repo I get at this moment: 

nslookup repo.us-west-2.amazonaws.com

Name: s3-us-west-2-w.amazonaws.com

Address: 52.218.144.34

OK so apparently the AWS Yum repositories are hosted on S3.

We can look up the latest AWS IP Ranges here:

https://ip-ranges.amazonaws.com/ip-ranges.json

AWS IP Range Updates Are Published As Explained Here:

http://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

There could be a few things causing this:

1. Security group rules disallow access to S3 IP ranges.

2. NACLs do not allow access to S3 IP ranges

3. No Internet Gateway route in subnet route table

4. The traffic routes from this subnet to another subnet in or outside AWS that can access the Internet, but traffic is blocked by a proxy, NAT or firewall that is disallowing the traffic.

As noted using a NAT on AWS is challenging but can be used to route traffic for updates from instances in a private network to the public Internet.

https://cloudonaut.io/private-subnets-are-broken-on-aws/

It may also be a good idea to internally host software on a repository that does not traverse the Internet to get updates. Consider Nexus, Artifactory or similar solutions. Then reaching out to the Internet would be limited to a few computers used to run the repository. A private repository could also be hosted on S3 with an S3 Endpoint.

Find all the AMI IDs from a specific vendor in the AWS Marketplace

I am trying to find a way to get a list of AMIs from a specific vendor in the AWS Marketplace. I thought I figured out a way to do this but turns out the owner is just the "AWS Marketplace" not a specific vendor.

First of all I know the description from the vendor (WatchGuard in this case) of the particular AMI I am seeking (for a WatchGuard Firebox Cloud). The AMI starts with "firebox*". I can query for "firebox*" however the issue would be if someone else published an AMI with a similar name and I accidentally used an AMI that was not actually from the vendor whose AMI I was trying to use. In the past this issue existed on Amazon where people published AMIs with names similar to reputable AMIs to confuse new users as to which AMI they should actually use.

To ensure I am using an AMI that is actually from WatchGuard I can query the AMIs I know are from WatchGuard for the "Owner ID" like this:

aws ec2 describe-images --filters "Name=description,Values=firebox*" | grep Owner

I will get back a list something like this:

            "ImageOwnerAlias": "aws-marketplace", 
            "OwnerId": "679593333241", 
            "ImageOwnerAlias": "aws-marketplace", 
            "OwnerId": "679593333241", 
            "ImageOwnerAlias": "aws-marketplace", 
            "OwnerId": "679593333241", 
            "ImageOwnerAlias": "aws-marketplace", 
            "OwnerId": "679593333241", 
            "ImageOwnerAlias": "aws-marketplace", 
            "OwnerId": "679593333241", 

From here I can see that the owner ID for the AMI from WatchGuard is 679593333241. Now I can use that in my query to get all the AMIs from WatchGuard:

aws ec2 describe-images --filters "Name=description,Values=firebox*" --owners 679593333241

Oh but wait....The ImageOwnerAlias is "aws-marketplace". Does that ID truly only relate to WatchGuard AMIs? Let's query without the description. This query takes quite a while to execute:

aws ec2 describe-images --filters "Name=description,Values=firebox*" --owners 679593333241

Unfortunately this pulls back everything in the AWS Marketplace, not just my WatchGuard AMIs. Will put in a feature request to see if this can be fixed somehow. In the end, I think this is the best I can do to get the latest Firebox AMI but asking AWS support to see if they have a better answer.

amidesc=$(aws ec2 describe-images --filters "Name=description,Values=firebox*" --owners 679593333241  | grep "Description" | grep "pay" | sort -r | grep -m1 -v "rc" | cut -d ":" -f 2 | sed -e 's/^[[:space:]]*//')

imageid=$(aws ec2 describe-images --owners 679593333241 --filters "Name=description,Values=$amidesc" | grep ImageId | cut -d ":" -f 2 | sed -e 's/^[[:space:]]*//')

echo $imageid


Using the above script I can get a list of the available WatchGuard AMIs in my account and region and ask the user to select one, which will be used in the subsequent script:

You can see this code in action in the following file:

https://github.com/tradichel/FireboxCloudAutomation/blob/master/code/run.sh

'capabilities' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy enum value set: [CAPABILITY_NAMED_IAM, CAPABILITY_IAM]]

When running cloud formation templates certain templates require IAM permissions. You will need to add this to your IAM call:

--capabilities CAPABILITY_NAMED_IAM 

The full command may look something like this:

aws cloudformation create-stack --stack-name firebox-nat-vpc --template-body file://resources/firebox-nat/vpc.yaml --capabilities CAPABILITY_NAMED_IAM --parameters ParameterKey=ParamStackName,ParameterValue=packet-capture-vpc

If you get this error:

'capabilities' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy enum value set: [CAPABILITY_NAMED_IAM, CAPABILITY_IAM]]

check to see what is following the capabilities switch to make sure it is correct. Although the error is related to the capabilities switch, the error may be due to something after that flag which is malformed. For example I left out the --parameters switch when dynamically piecing together the CloudFormation call and produced the following by accident and got the above error - noticed there is no --parameters switch:

aws cloudformation create-stack --stack-name firebox-nat-vpc --template-body file://resources/firebox-nat/vpc.yaml --capabilities CAPABILITY_NAMED_IAM  ParameterKey=ParamStackName,ParameterValue=packet-capture-vpc

To see this code in action check out this GitHub repo:

https://github.com/tradichel/FireboxCloudAutomation

Here's the file that is generating the CloudFormation calls with parameters and capabilities:

https://github.com/tradichel/FireboxCloudAutomation/blob/master/code/execute/action.sh

IP Spoofing

IP Spoofing means that someone likely manually crafted a network packet and put in a bogus IP address as the source address.

A network packet consists of a number of fields following a certain protocol. For more information see this post which contains a diagram and description of the fields in an IP packet header. One of these fields in the IP portion of the packet header is the source IP address. The source IP address should be the IP address of the device that sent the packet, and the IP address to which any response packets should return. If someone puts in a bogus IP address as the source IP, the sender won't be able to receive the responses to the network request. The response will go to the bogus IP address.

Why would someone put in a false return IP address? One reason would be to DDOS another computer. Many packets with the incorrect source IP address could be redirected to someone that malicious packet crafter wants to inundate with packets to take down their network equipment or host machines that cannot handle the load. There could be other malicious reasons for this.

An IP packet captured with tcpdump will have more or less information depending on what flags you set when you run the command, but basically it will look something like this:

 0x0000:  0001 0800 0604 0001 02a5 c63c 2226 0a00
 0x0010:  0001 0000 0000 0000 0a00 006d 0000 0000
 0x0020:  0000 0000 0000 0000 0000

That hex data can be converted to a human readable form. If you really want to nerd out you can break down the IP packet header as explained here to determine the source IP in your packets:

http://websitenotebook.blogspot.com/2014/05/decoding-ip-header-example.html

Another option to learn about decoding packets would be to take classes at SANS Institute, starting with the SANS Bootcamp:

https://www.sans.org/course/security-essentials-bootcamp-style

If a packet is truly spoofed, the IP address in the source address field will be incorrect, not matching the host that truly sent the packet. In some cases however, certain networking equipment has to try to figure out if an IP is spoofed by seeing if it can return a packet to the host. It may not be able to and assume the IP is spoofed, when the actual problem is that for some reason it received a packet but cannot send a response back to the host due to network routes or firewall rules.

If you know the return IP address in your packets is good, check the route tables and network configuration to make sure the packet is able to return to the host that sent the packet.

If the source IP is actually not valid, hopefully your network security services will detect that before it has a negative impact on your network or applications and block the traffic. Some external providers offer DDOS protection which will handle any extra load produced by traffic like this. AWS has some built in DDOS protection and a higher end service for larger customers called Shield. This SANS white paper has more information about defending against DDOS Attacks.

Error: The Device Administrator admin from x.x.x.x has selected Edit Mode. You cannot select Edit Mode until admin has selected View Mode.

If you see this error trying while working on the command line of WatchGuard Firebox:

WG#configure %Error: The Device Administrator admin from x.x.x.x has selected Edit Mode. You cannot select Edit Mode until admin has selected View Mode.

This could mean that you are trying to run a command after you typed configure to enter the configuration mode and the command needs to be run in main mode.

If you are in configure mode you can type "exit" to return to the main, read only mode.

For example, when trying to run tcpdump on a Firebox in configure mode you would get this error because tcpdump should be run in the main, read only mode.

For more information refer to the latest version of the WatchGuard Firebox CLI Documentation

0.0.0.0/0 in AWS Route Tables and Network Rules

Public Safety Announcement:

0.0.0.0/0 should be used sparingly. It means any host on any IP address (or any IPv4 address to be precise) on the Internet can use this route to connect to things on the other side. That means any host in this subnet can contact any host on the Internet and vice versa.

Here's a random sampling of traffic that hit my Firebox Cloud as soon as I set it up on the Internet, and why you might want to open up this type of traffic sparingly. As soon as you do this various nefarious (and accidental) traffic will start hitting any host with access from the Internet.

If you do open up to the Internet you might want security appliances inspecting traffic to and from your hosts prevent malicious traffic from accessing them. For example a WatchGuard Firebox would have prevented anyone using it from being infected by the WannaCry virus if configured properly. You can also use NACLs and Security Groups to limit access as will be noted in upcoming blog posts.

Route Tables: Protecting Your Network

When you set up your network it is very important to understand how route tables work and how they open up access to your network in unintended ways on AWS. Route tables define where traffic can flow on your network. They provide the routes or "roads" traffic can take to get from point A to point B. The following example architecture for a Firebox Cloud Architecture with a management network will be used to explain route tables. These routes are also explained in the AWS NAT documentation.

Let's say you want the resources in a subnet to have Internet access. In this case you create a PUBLIC subnet and put an Internet Gateway in the route table. The Internet Gateway allows resources in that subnet can get to the Internet and hosts on the Internet can connect to resources in this subnet (unless blocked by other security controls such as security groups, NACLs, or a Firebox Cloud):

If we look at the route table we can see that there are two routes.

The first route is added by default for the local VPC IP Range or CIDR. Since I created my VPC with the CIDR 10.0.0.0/16 the local route will allow any host in the associated subnet(s) to send data to or receive data from any host with an IP of in the IP range of 10.0.0.0/16 or 10.0.0.0 to 10.0.255.255. Again this can be further restricted by other controls.

The second route allows any host in the network to send data to or from the Internet via the AWS Internet Gateway (igw-xxxxxxxx) to anywhere: 0.0.0.0/0.

If you want to keep a subnet PRIVATE (meaning the hosts in the subnet cannot directly access the Internet) you need to ensure the subnet does not have a route for an Internet Gateway. In addition you need to ensure that any routes in that subnet to do not in turn route to something that ultimately can route or proxy that traffic to the Internet.

Here's a private subnet:

If we click on the Route Table tab we can see the following:

The local VPC route is again added by default.

In addition we allow any host in this subnet to send traffic to the PRIVATE (trusted) ENI of a WatchGuard Firebox Cloud. That means nothing can get out of our Private Subnet to the Internet without going through our Firebox Cloud if these are the only two subnets in our VPC.

There is NO route to the Internet Gateway. Therefore there is no way for hosts in this subnet to get directly to the Internet from this particular subnet (see caveats at the end of this article).

In other words any host in this subnet trying to send traffic and looking for a route to get it from point A to point B has two options - send it to something else in the VPC or send it to the Firebox Cloud.

In addition, we will also add a route to access an S3 Endpoint as explained previously, to ensure all access to our S3 bucket for managing the Firebox is in the PRIVATE network, never on the Internet and only from our management network to protect our Firebox SSH key:

If I go check out my elastic network interfaces I can verify that my Firebox Cloud Private or Trusted ENI is in this private subnet. That means any traffic to this private ENI must come from within the VPC and this private ENI cannot send data to or receive data from the Internet. The traffic can route through the Firebox to the public ENI which is how we inspect all the traffic getting to and coming from the Internet.

I can look at the details of the ENIs to make sure they are configured correctly. According to the WatchGuard Firebox Cloud documentation, the public Interface should be on eth0 and the private should be on eth1.

eth0 - Public

eth1 - Private

By architecting the network this way, we can make sure traffic that is allowed to access the Firebox management key and also the Firebox CLI is only allowed in the private network (if the rest of our network doesn't have extraneous holes in it). Additionally, we can also add security groups to whitelist and further restrict access as will be discussed in an upcoming blog post.

Now imagine you add a route to the Internet Gateway in one of the subnets that are currently private. You have just allowed them to bypass the Firebox and get to the  Internet without inspection. You may also be sending management of the Firebox over the Internet to the public ENI, instead of to the private or trusted ENI.

Additionally as shown in the diagram for the AWS NAT documentation, traffic to hosts placed inside the public subnet can bypass the NAT, which in this case is our Firebox Cloud. There are other possible configurations but this is the one we are considering at the moment, so if you are using this configuration be aware of this fact and don't put hosts in the Firebox public subnet that you want to be protected by the Firebox cloud.

Here's another scenario: you add a route from this VPC to another VPC and another subnet in your account or another account using VPC Peering. If that other VPC subnet has Internet access, you have just allowed traffic to bypass the Firebox potentially and get to the Internet.

You may be thinking that due to the local route, anything in the private subnets can route to the public subnet, then to the Internet. This is true. A host in the private subnet, without any other controls, could connect to something in the local VPC public subnet. The host in the public subnet with Internet access could be used as a proxy to send data to the Internet (including malware stealing data), so it's best to limit what is in the public subnet and use Security Groups and NACLS to further restrict the traffic, as will be explained.

Understand the your network routes. Data on your network is like water. If you open a hole...it will flow there.

Manually Activating WatchGuard Firebox Cloud Marketplace AMI

In order to use the WatchGuard Firebox Cloud AMI in the automated deployment scripts you will first need to activate the AMI in your account. Unfortunately think you will need to take this manual step before you can use the automation. Perhaps it is a good thing to make sure people cannot instantiate things you don't want them using in your account.

To manually instantiate the WatchGuard Firebox Cloud, in your AWS account navigate to the EC2 service and launch instance. This is basically going to create a virtual machine with WatchGuard Firebox Cloud running on it.

Choose Marketplace and search for WatchGuard. Select the desired option. At the time of this writing the options "Pay As You Go" or "BYOL" (Bring Your Own License). If you would like a license reach out to a WatchGuard partner who can also help you set this up if needed. Pay as you go will, as it sounds, bill you based on what you use. BYOL will allow you to use a license for a flat fee. The Pay As You Go option currently supports T2.Micro instances. Both options support C4.Large and up. Pay As You Go does not include Threat Detection and Response (endpoint protection). Check the WatchGuard web site as these things may change over time.

Complete the steps to instantiate the WatchGuard instance same as you would for any other AWS EC2 instance. If you using BYOL you will need to manually activate the license.

Once you have instantiated the instance, the AMI will be registered in your account, at which point you can run the automated scripts to complete the automated deployment of a WatchGuard Firebox Cloud. You can terminate the instance you launched manually.

"ResourceStatusReason": "The image id '[ami-a844d4c8]' does not exist"

If you're getting this error when creating a CloudFormation stack that means your code is trying to create a stack with an AMI that doesn't exist in your account and/or region, or perhaps it is referencing an old AMI that has been deprecated in favor of a new one.

 "ResourceStatusReason": "The image id '[ami-a844d4c8]' does not exist"

In the case of the Firebox WatchGuard Cloud AMI it currently has a different ID in each region. There is also a separate AMI for the BYOL (bring your own license) vs. Pay as you go AMI.

You can find these AMIs by manually creating a WatchGuard Firebox Cloud from the AWS Marketplace and then looking at the details of the instance you launched to get the AMI ID. I am looking for a better way to get this information but at this time that will help you determine which AMI to use.

In the case of the WatchGuard Cloud Automation templates the AMI ID is used in this template:

https://github.com/tradichel/FireboxCloudAutomation/blob/master/code/resources/firebox-nat/firebox.yaml

The script is being revised to let you select the AMI when you run it and instructions in the readme will be updated.

I was trying to create a script to look up the latest AMI for a particular vendor in the AWS Marketplace to programmatically get around this issue but this is the closest I could come. Maybe a feature request?

aws ec2 describe-images --filters "Name=description,Values=firebox*" | grep 'ImageId\|Description' | sed 's/ *\"ImageId\": "//;s/",//' | sed 's/ *\"Description\": "//;s/"//'

"ResourceStatusReason": "No export named [x] found. Rollback requested by user.",

Was having someone help me test FireboxCloud automation script and got this error.

"ResourceStatusReason": "No export named [x] found. Rollback requested by user.",

What does this mean?

At the bottom of the templates that create the resources there are a list of outputs. For example in this template:

https://github.com/tradichel/FireboxCloudAutomation/blob/master/code/resources/firebox-nat/subnets.yaml

There are outputs at the bottom that look like this:

Outputs:
  FireboxPrivateSubnet:
    Value: !Ref FireboxPrivateSubnet
    Export:
      Name: "FireboxPrivateSubnet"
  FireboxPublicSubnet:
    Value: !Ref FireboxPublicSubnet
    Export:
      Name: "FireboxPublicSubnet"
  FireboxPublicRouteTable:
    Value: !Ref FireboxPublicRouteTable
    Export:
      Name: "FireboxPublicRouteTable"
  FireboxPrivateRouteTable:
    Value: !Ref FireboxPrivateRouteTable
    Export:
      Name: "FireboxPrivateRouteTable"
  FireboxPublicNacl:
    Value: !Ref FireboxPublicSubnetNacl
    Export:
      Name: "FireboxPublicNacl"

There is some fairly new (and awesome) functionality in CloudFormation that allows you to easily import the output of one template into another using ImportValue.

When I create my FireboxCloud I want to put the ENIs (one public and one private) into their respective subnets. I can reference the subnets using !ImportValue as shown below:

Resources:
  FireboxPublicNetworkInterface:
    Type: "AWS::EC2::NetworkInterface"
    Properties:
      Description: Firebox Public Network Interface
      GroupSet:
        - !ImportValue FireboxPublicSecurityGroup
      SubnetId: !ImportValue FireboxPublicSubnet
      SourceDestCheck: false

When the person helping me ran my scripts they got this error:

"ResourceStatusReason": "No export named [x] found. Rollback requested by user.",

It's a little odd that I was not also getting this error. Typically this error is caused when the ImportValue has a typo in the name of the OutputValue it is referencing.

What was odd in this case: I wasn't getting the error. That's because I had added a resource after the fact that reference the output value of a CloudFormation stack I had already created. This seems like it shouldn't be possible and is not correct. If referencing the value in the template itself I should have referenced the name of the resource created in the same template using Ref.

The subsequent problem was that I could not delete the CloudFormation stack because it said the stack was creating an output that was referenced by something else (itself!). I got around this by updating the stack and removing the incorrect ImportValue to a resource in the stack and replacing it with Ref. After that was fixed I could delete the stack.

Firebox Cloud Automation

As noted in this blog post, security automation can help prevent errors that lead to security problems.

For this reason I want to completely automate the deployment of security devices in my AWS account. Over the next few blog posts we will be automating the use of a WatchGuard Firebox Cloud on AWS. I also want to only allow configuration from within the private network.

Here's a picture of what we are going to create.

The code can be found in this GitHub repository for automation of a WatchGuard Firebox Cloud on AWS.

The goal of this code is to completely configure our Firebox Cloud without leaving the private network by deploying only with code from source control. 

Note that the Firebox Cloud by default opens up the required route to the subnet it lives in for management access. In order to access the CLI from a Lambda or EC2 instance we will need to put it in this same subnet. It is advisable to lock down management ports and/or create separate network interfaces and subnets for other resources that should not have access to the management interface and port.

Notice that in our diagram above the public ENI is an Internet-accessible subnet connected to the Internet Gateway. The private ENI is in a private subnet with no access outside of our VPC.

Along the way I'll explain some security best practices including those already explained in previous blog posts on secure access from Lambda to an S3 bucket.

For step by step instructions to run the code check out my Secplicity blog post on How to Automate Deployment of a WatchGuard Firebox Cloud on AWS.

Follow me on Twitter @TeriRadichel and at Secplicity for updates!

Creating Paramiko and Cryptography Virtual Environment to Run On AWS Lambda

A prior blog post explained how to obtain the dependencies to successfully build Paramiko and Cryptography on an AWS EC2 instance in a virtual environment. This post will show how to package up those dependencies for a Lambda function using EC2.

I've created some networking in order to automate deployment of a WatchGuard Firebox Cloud which I am using for my EC2 instance below. If you are unsure how to set up the networking that securely allows Internet access you could run the CloudFormation templates in my FireboxCloudAutomation Github repo and use that networking to complete the steps below. Stay tuned for more networking information here and on Secplicity. It is highly recommended that you strictly limit any SSH access to instances in your VPC and ideally remove that access over the network when not in use. You can also create a bastion host.

For now I will manually create an EC2 instance. Might automate this later. I am not deploying production systems here simply testing. I would automate all of this if actually using it in a production environment.

First instantiate the EC2 instance.

Choose the AWS Linux AMI which has the necessary dependencies matching what is on a Lambda function.

Choose your instance type. Smallest is probably fine.

Configure networking - this is where I am using the networking I created for the WatchGuard Firebox Cloud as noted above so I will have SSH access to my new instance without having wide open networking. Choose the Firebox VPC, the Public Subnet which allows Internet access, and auto-assign an IP. Can't connect without that.

Tag your instance with a name which is helpful for finding it in the console.

Create or use a restrictive SSH security group. The ONLY port we need open for this is SSH port 22 and I only need to be able to access it from My IP address as selected below. Then in theory the only way someone could get to this instance would be to get onto my network (which could be done of course, but we are limiting the attack vector as much as possible). Also I haven't thoroughly reviewed these software packages. If for some reason they had some malware that would reach out to a C2 server, it wouldn't be able to reach that server due to my network rules so I feel a bit safer with this configuration.

Select a key that will be used to SSH into this instance. KEYS ARE PASSWORDS. Protect them.

Wait for the EC2 instance status checks to pass and the indicator below to turn green.

Right click on the instance to get the command to connect to the instance. 

Follow the instructions to connect to the instance. If having problems read this blog post on connecting to EC2 instances using SSH.

Once connected to EC2 instance in terminal:

Run the commands from the my post that explains how to build Paramiko and Cryptography.

Note that you will likely want to use Python 2.7 due to inconsistencies between EC2 (Python 3.4 or 3.5) and Lambda function (Python 3.6). You can probably make it work but this will get you up and running faster:

sudo yum update -y

sudo pip install virtualenv --upgrade

cd /tmp

virtualenv -p /usr/bin/python2.7 python27

source python27/bin/activate

sudo yum install gcc -y

#probably don't need these but just in case libs are missing

#sudo yum install python27-devel.x86_64 -y

#sudo yum install openssl-devel.x86_64 -y

#sudo yum install libffi-devel.x86_64 -y

pip install --upgrade pip

pip install paramiko

Commands to create virtual environment also found here with comments:

https://github.com/tradichel/FireboxCloudAutomation/blob/master/code/execute/paramiko.sh

Following these directions I zip up the files in the lib/python2.7/site-packages dir

http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Zip the files as explained on this page:

http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python

#change to the site-packages directory

cd python27/lib/python2.7/site-packages

zip -r9 /tmp/lambda.zip . -x \*__pycache__\*

We also need the files in the lib64 directory:

cd ../../..

cd lib64/python2.7/site-packages

zip -g -r9 /tmp/lambda.zip . -x \*__pycache__\*

Now we should have a lambda.zip file in the tmp directory on the EC2 instance:

Now I can deactivate the virtual environment because I'm done with it.

deactivate

Run this from local machine to copy down the zip we just created to the folder where your python file lives that you want to include in the zip file for the Lambda function.

#scp username@ipaddress:pathtofile localsystempath

#change the key and IP address below

scp -i "[yourkey].pem" ec2-user@[ip-address]:/tmp/lambda.zip lambda.zip

Now I can add my own python files to the zip for use in a lambda function, again as explained on this page to run SSH commands from a lambda function:

http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python

Heres the code I use to add my fireboxconfig.py to the lambda.zip file I downloaded. I actually copy the lambda.zip to a new file, add my python file and upload it to an S3 bucket so the zip file can be used in my Lambda CloudFormation Template.

For more on that check out my next blog post where I'll explain how I use it in the rest of the code on GitHub to automate configuration of a WatchGuard Firebox Cloud.  I'm using the Lambda function with Paramiko to connect to a WatchGuard Firebox Cloud and configure via the Firebox CLI (command line interface). For more on the WatchGuard CLI check out the latest version of the WatchGuard Firebox Cloud CLI Commands.

Questions or Suggestions - DM me on twitter @teriradichel

invalid ELF header - Import Error

If you see this error when running an AWS lambda function:

{
  "errorMessage": "/var/task/cryptography/hazmat/bindings/_constant_time.abi3.so: invalid ELF header",
  "errorType": "ImportError"
}

...then you need to include required libraries used by your Lambda function.

The problem arises when attempting to package up libraries from the OS on which you are developing and the OS to which you are deploying has different dependency requirements.

For example the libraries required on Windows are different than the libraries required by an AWS Linux EC2 instance when dealing with C libraries.

The solution is to do this packaging with virtual env on an EC2 instance, which will then package up compatible libraries for your Lambda function.

---

Trying to set up a Python virtual environment that has the correct libraries and/or works on AWS Lambda? Check out these blog posts which are specific to Paramiko and Cryptography but explain how to determine which libraries to use and package up a virtual environment that has the correct underlying libraries:

http://websitenotebook.blogspot.com/2017/05/installing-paramiko-and-crytography-in.html

http://websitenotebook.blogspot.com/2017/05/creating-paramiko-and-cryptography.html

Installing Paramiko and Crytography in Python Virtual Environment

This blog post describes how to run SSH jobs from an AWS Lambda function:
https://aws.amazon.com/blogs/compute/scheduling-ssh-jobs-using-aws-lambda/

It seemed like that would be the solution for running SSH in an AWS Lambda function for the purposes of automating configuration of an WatchGuard Firebox Cloud.

The only issue was when attempting to run the code I realized that additional libraries are required. I started with Python 3.6 because why not? It's the most up to date version of Python on Lambda and sounds like Paramiko will work with that version. Turns out Paramiko must be packaged up in the Lambda code. That in turn requires the cryptography package which in turn uses some C libraries. Packaging this up on a mac or Windows machine would include OS specific libraries that wouldn't work in a Lambda function, which presumably runs on something like an AWS EC2 Linux instance.

Always looking for a quick fix I reached out to my friend, Google. There I found some recommendations suggesting creation of the virtual environment on an EC2 instance. However that wasn't as straightforward as one might hope. The required libraries were not all installed by default and the names of the libraries are different than the documentation and various blog posts on the topic. Basically in order to create a python virtual environment you'll need to install gcc, as well as specific versions of python-devel and openssl-devel. I describe how to find and install those libraries in a bit more detail in my previous posts.

Here's what I came up with. Looks so simple now...and by the way by the time I wrote this something changed so make sure you check what packages are available to install as noted in my recent blog posts. I also show how to use the list command below to find all the packages with python3 in the name.

#update the ec2 instance
sudo yum update -y

#see what versions of python 3 are available
sudo yum list | grep python3

#install the one we want
sudo yum install python35.x86_64 -y

#switch to temp directory
cd /tmp

#create virtual environment
virtualenv -p /usr/bin/python3.5 python35

#activate virtual environment
source python35/bin/activate

#install dependencies
sudo yum install gcc -y
sudo yum install python35-devel.x86_64 -y
sudo yum install openssl-devel.x86_64 -y
sudo yum install libffi-devel.x86_64 -y

#install cryptography and parmaiko paramiko
pip install cryptography
pip install paramiko

And finally - it works.

Successfully installed asn1crypto-0.22.0 cffi-1.10.0 cryptography-1.8.1 idna-2.5 packaging-16.8 paramiko-2.1.2 pyasn1-0.2.3 six-1.10.0

Great but guess what. Tried running this on Lambda and get missing library errors.

Digging further I figured out how to find out what versions of Python are available on Lambda by using this blog post:

https://www.linkedin.com/pulse/running-python-3-aws-lambda-lyndon-swan

Ran this code in my lambda function:

args = ("whereis","python3")
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
popen.wait()
output = popen.stdout.read()
print(output)

Looks like only Python 3.4 and Python 3.6 are available and all of the above is based on 3.5.

b'python3: /usr/bin/python3 /usr/bin/python3.4m /usr/bin/python3.4 /usr/lib/python3.4 /usr/lib64/python3.4 /usr/local/lib/python3.4 /usr/include/python3.4m /var/lang/bin/python3.6m /var/lang/bin/python3.6-config /var/lang/bin/python3 /var/lang/bin/python3.6m-config /var/lang/bin/python3.6 /usr/share/man/man1/python3.1.gz'

Options would be go back to 2.7 or try to use 3.4 since 3.6 doesn't appear to be available on EC2 instances. *sigh*.  Let's see if we can build a 3.4 virtual environment.

#see what versions of python 3 are available on EC2 instance
sudo yum list | grep python3

#output gives us python34.x86_64

#install the one we want
sudo yum install python34.x86_64 -y

#create virtual environment
virtualenv -p /usr/bin/python3.4 python34

#activate virtual environment
source python34/bin/activate

#install dependencies
sudo yum install gcc -y
sudo yum install python34-devel.x86_64 -y
sudo yum install openssl-devel.x86_64 -y
sudo yum install libffi-devel.x86_64 -y

#install cryptography and parmaiko paramiko
pip install paramiko

Installing collected packages: pyasn1, paramiko

Successfully installed paramiko-2.1.2 pyasn1-0.2.3

Great. But it didn't run on Lambda either.

{ "errorMessage": "No module named '_cffi_backend'", "errorType": "ModuleNotFoundError"}

Presumably I need to set up my lambda function to use 3.4 as noted above but lets roll back to 2.7 and see if that works. Since EC2 instances use 2.7 by default we won't hopefully need all the extra packages.

#update the ec2 isntance
sudo yum update -y

#switch to temp directory
cd /tmp

#create virtual environment
virtualenv -p /usr/bin/python2.7 python27

#activate virtual environment
source python27/bin/activate

#install dependencies
#sudo yum install gcc -y
#sudo yum install openssl-devel.x86_64 -y
#sudo yum install libffi-devel.x86_64 -y

#install cryptography and parmaiko paramiko
pip install paramiko

Successfully installed asn1crypto-0.22.0 cffi-1.10.0 cryptography-1.8.1 enum34-1.1.6 idna-2.5 ipaddress-1.0.18 paramiko-2.1.2 pyasn1-0.2.3 pycparser-2.17

And... testing it on a 2.7 Lambda function, it works. No missing libaries.

Read on if you want to see how the Lambda function is set up to use Paramiko and Cryptography to connect to configure a WatchGuard Firebox Cloud via the Command Line Interface and SSH.