Automated Traffic Mirroring in AWS with Tags

Other Posts in This Series

  1. Automated Traffic Mirroring in AWS
  2. Automated Traffic Mirroring in AWS at Scale
  3. Automated Traffic Mirroring in AWS with Full Coverage
  4. Automated Traffic Mirroring in AWS with Tags (you are here)

To consider this solution properly baked, we need tags. The good news is that we only need one tag to rule them all. Using a tag allows for much more granular control of the automation, as you’ll see shortly.

Summary
In this post, we will update both of our Lambda functions to respect a specific tag. This tag will then be used to scope the automation to EC2 instances with the tag applied, as well as VPC Traffic Mirror Filters and Targets with the same tag. Prior to making this change, the automation would just take the first Traffic Mirror Filter it found in the region. By placing a Mirror tag on an instance with a value of extrahop, the Lambda function can specifically ensure the ENIs of that instance are mirrored to an extrahop target using an extrahop filter, in the appropriate region, az, and subnet, of course.

So here we go, the final post in the series until I add more posts to the series!

Step 1: Add Environment Variable definitions to your Manager function in Lambda
This is super easy. I’m adding these to my rx-vpctm-manager function, right in the AWS Console. We really don’t need to do this, but it’s just good form to keep things like this out of the code so it can be easily changed. You’ll notice I’ve pulled the LAMBDA_CREATE definition out of the code and into an EV as well. So go ahead and create the following Environment Variables:

  1. TAG_MIRROR: EC2 instances with this tag will be considered in-scope. The Value of the tag is used to locate corresponding Filters and Targets for new VPC Traffic Mirror Sessions.
  2. LAMBDA_CREATE: This is just the name/alias of the LAMBDA function to invoke for creating new VPC Traffic Mirror Sessions

image

Step 2: Update the existing Manager Function in Lambda
By now you should definitely have a grasp of what’s going on here, so I’ll just drop the code below that you should use to replace what we have so far. In essence, you can search for TAG_MIRROR in the code and find the places where things changed to include the tag as criteria for querying instances, and how the tag key is now passed to the LAMBDA_CREATE function as well.

Replace your existing function code with the following:

'use strict';

const AWS = require('aws-sdk')
const LAMBDA = new AWS.Lambda({apiVersion:'2015-03-31'})
const EC2 = new AWS.EC2({apiVersion:'2016-11-15'})

// EC2 instances with this TAG will be considered in-scope. The VALUE of TAG is
// used to locate corresponding [filter] and [targets] for new VPC TM Sessions
const TAG_MIRROR = process.env.TAG_MIRROR

// The name/alias of the LAMBDA function to invoke for creating VPC TM Sessions
const LAMBDA_CREATE = process.env.LAMBDA_CREATE

exports.handler = async (event) =>
{
  try
  {
    // Safety Net to ensure the actual request resulted in a successful response
    if (! event.detail.responseElements) {throw `This does not look like a successful API request`}

    // Populate the [instanceIds] Array based on [eventName]
    let instanceIds = []
    switch (event.detail.eventName)
    {
      // Convert [instancesSet] to an Array of [instanceIds]
      case 'RunInstances':
      case 'StartInstances':
        instanceIds = event.detail.responseElements.instancesSet.items.map(i => (i.instanceId))
        break

      // Simply pluck [instanceId] and go (operation does not support multiples)
      case 'AttachNetworkInterface':
        instanceIds.push(event.detail.requestParameters.instanceId)
        break

      // Get [instanceIds] for all NITRO EC2 Instances in this REGION
      case 'DeleteTrafficMirrorSession':
        instanceIds = await getNitroInstances()
        break

      // Safety Net for unsupported Operations
      default: throw `Unsupported Operation: ${event.detail.eventName}`
    }

    // Create a VPC Traffic Mirror Session for all [instanceIds]
    await Promise.all(
      instanceIds.map(async id =>
        LAMBDA.invoke({
          FunctionName: LAMBDA_CREATE,
          Payload: JSON.stringify({instanceId:id, tagMirror:TAG_MIRROR})
        }).promise().then(response =>
        {
          // Send LOG/ERROR to CloudWatch for each [instanceId] individually
          let payload = JSON.parse(response.Payload || '{}')
          if (payload.statusCode === 200)
          {console.log(event.detail.eventName, payload.body)}
          else
          {console.error(event.detail.eventName, payload.body, payload.statusCode, payload.error)}
        })
      )
    )
  }
  catch (err) {console.error(event.detail.eventName, err.message || err)}
}

/* ========================================================================== >>
   WORKER FUNCTIONS
============================================================================= */
async function getNitroInstances ()
{
  let instanceIds = []

  // Get all EC2 Instances with [TAG_MIRROR] as [reservations] in this REGION
  let reservations = [], token = null
  do {
    let data = await EC2.describeInstances({
      MaxResults: 1000,
      NextToken: token,
      Filters: [{Name:'tag-key', Values:[TAG_MIRROR]}]
    }).promise()
    reservations = reservations.concat(data.Reservations)
    token = data.NextToken
  }
  while (token)

  // Extract the [instanceId] and add to list if [isNitro]
  reservations.forEach(r =>
  {
    r.Instances.forEach(i =>
    {
      if (isNitro(i.InstanceType)) {instanceIds.push(i.InstanceId)}
    })
  })

  return instanceIds
}

/* ========================================================================== >>
   HELPER FUNCTIONS
============================================================================= */

// The [hypervisor] property on the [instance] object is not useful for this, so
// it's necessary to check the instance family manually.
// Reference: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances
function isNitro (instanceType)
{
  const nitro = [
    'A1', 'C5', 'C5d', 'C5n', 'G4', 'I3en', 'Inf1', 'M5', 'M5a', 'M5ad', 'M5d',
    'M5dn', 'M5n', 'p3dn.24xlarge', 'R5', 'R5a', 'R5ad', 'R5d', 'R5dn', 'R5n',
    'T3', 'T3a', 'z1d', 'a1.metal', 'c5.metal', 'c5d.metal', 'c5n.metal',
    'i3.metal', 'i3en.metal', 'm5.metal', 'm5d.metal', 'r5.metal', 'r5d.metal',
    'u-6tb1.metal', 'u-9tb1.metal', 'u-12tb1.metal', 'u-18tb1.metal',
    'u-24tb1.metal', 'z1d.metal'
  ]

  return (new RegExp(`^(?:${nitro.join('|')})`, 'i').test(instanceType))
}

Step 3: Update the existing Create Function in Lambda
The changes here are more significant, because the Value of the Mirror tag is used to determine which Targets and Filters to get. So that requires us to grab the full Instance object to get the Value of the Mirror tag before we make our calls to get the Targets and Filters.

Replace your existing function code with the following:

'use strict';

const AWS = require('aws-sdk')
const EC2 = new AWS.EC2({apiVersion:'2016-11-15'})

exports.handler = async (event) =>
{
	try
	{
		// Simple Safety Net
		if (! event.instanceId || ! /^i-/.test(event.instanceId) || ! event.tagMirror)
		{throw `Invalid Request: ${JSON.stringify(event)}`}

		// Get the full [instance] object using [instanceId] and [tagMirror]
		let instance = await getInstance(event.instanceId, event.tagMirror)

		// Initialize [TAG_MIRROR] with the VALUE from this [instance]
		const TAG_MIRROR = {
			Key: event.tagMirror,
			Value: instance.tags[event.tagMirror]
		}

		let [targets, filterId] = await Promise.all(
		[
			// Get available traffic mirror [targets] in this REGION with [TAG_MIRROR]
			getTargets(TAG_MIRROR),

			// Get traffic mirror [filterId] for this REGION with [TAG_MIRROR]
			getFilterId(TAG_MIRROR)
		])

		// Set the best [target] for each ENI attached to the [instance]
		instance = await setTargets(instance, targets)

		// Create a new VPC TM Session for each [eni] attached to [instance]
		await Promise.all(
			instance.nics.map(eni => EC2.createTrafficMirrorSession({
				NetworkInterfaceId: eni.id,
				SessionNumber: 1,
				TrafficMirrorFilterId: filterId,
				TrafficMirrorTargetId: eni.target,
				Description: `Managed (${TAG_MIRROR.Value})`,
				TagSpecifications: [{
					ResourceType:'traffic-mirror-session',
					Tags:[
						TAG_MIRROR,
						{Key:'Name', Value:instance.tags.Name || instance.InstanceId}
					]
				}]
			}).promise().catch(err =>
			{
				// Catch and safely ignore "already in use" errors
				if (/SessionNumber [0-9]+ already in use/i.test(err.message)) {return}
				if (/is in use by target/i.test(err.message)) {return}
				throw err.message
			})
		))

		// All done ...
		return {
			statusCode: 200,
			body: `VPC Traffic Mirroring enabled for EC2 Instance: ${event.instanceId}`
		}
	}

	catch (err)
	{
		console.error(err.message || err)

		return {
			statusCode: (err.message ? 500 : 400),
			body: `Error enabling VPC Traffic Mirroring for EC2 Instance: ${event.instanceId}`,
			error: err.message || err
		}
	}
}

/* ========================================================================== >>
   WORKER FUNCTIONS
============================================================================= */
async function getInstance (instanceId, tagMirror)
{
	// Get the full [instance] object using [instanceId] and [TAG_MIRROR]
	const data = await EC2.describeInstances({
		InstanceIds: [instanceId],
		Filters: [{Name:'tag-key', Values:[tagMirror]}]
	}).promise()

	// Extract the single [instance] from the response [data]
	let instance = data.Reservations[0].Instances[0]

	// Ensure this is a Nitro instance
	if (! isNitro(instance.InstanceType))
	{throw `Instance not eligible for VPC Traffic Mirroring [isNitro]: ${instanceId} (${instance.InstanceType})`}

	// Setup Utility Variables
	instance.az = instance.Placement.AvailabilityZone
	instance.nics = getNics(instance.NetworkInterfaces)
	instance.tags = getTags(instance.Tags)

	return instance
}

async function getTargets (tag_mirror)
{
	// Get all available traffic mirror [targets] in this REGION with [tag]
	const data = await EC2.describeTrafficMirrorTargets({
		Filters: [{Name:`tag:${tag_mirror.Key}`, Values:[tag_mirror.Value]}]
	}).promise()

	// Convert [TrafficMirrorTargets] into a hash table of [targets] by ENI
	let targets = {}
	data.TrafficMirrorTargets.forEach(target => (targets[target.NetworkInterfaceId] = {
		'id': target.TrafficMirrorTargetId,
		'tags': getTags(target.Tags)
	}))

	return targets
}

async function getFilterId (tag_mirror)
{
	// Get traffic mirror [filters] for this REGION with [TAG_MIRROR]
	const data = await EC2.describeTrafficMirrorFilters({
		Filters: [{Name:`tag:${tag_mirror.Key}`, Values:[tag_mirror.Value]}]
	}).promise()

	// There should be only one [filter] in this REGION with [TAG_MIRROR]
	return data.TrafficMirrorFilters[0].TrafficMirrorFilterId
}

async function setTargets (instance, targets)
{
	// Get full detail for each [target] ENI
	const data = await EC2.describeNetworkInterfaces({
		NetworkInterfaceIds: Object.keys(targets)
	}).promise()

	// Add applicable ENI properties to [targets] from [data], and filter out
	// [targets] not in the same [AvailabilityZone] as the [instance]
	for (const eni of data.NetworkInterfaces)
	{
		if (eni.AvailabilityZone !== instance.az)
		{
			delete targets[eni.NetworkInterfaceId]
			continue
		}

		targets[eni.NetworkInterfaceId] = {
			'id': targets[eni.NetworkInterfaceId].id,
			'subnet': eni.SubnetId,
			'vpc': eni.VpcId
		}
	}

	// Convert [targets] hash table to an Array of [targets]
	targets = Object.values(targets)
	if (! targets.length) {throw `No Traffic Mirror Targets available: ${instance.az}`}

	// Set the [target] for each [nic] on the [instance]
	instance.nics = instance.nics.map(nic =>
	{
		nic.target = false

		// Determine if a [target] exists in the same [subnet]
		for (const target of targets)
		{
			// Stop instantly: a local target is always ideal
			if (target.subnet === nic.subnet)
			{
				nic.target = target.id
				break
			}

			// Set the [target] based on VPC until something better is found
			if (target.vpc === nic.vpc) {nic.target = target.id}
		}

		// No [targets] in local [subnet] or [vpc], use first found in this AZ
		if (! nic.target) {nic.target = targets[0].id}

		return nic
	})

	return instance
}

/* ========================================================================== >>
   HELPER FUNCTIONS
============================================================================= */

// The [hypervisor] property on the [instance] object is not useful for this, so
// it's necessary to check the instance family manually.
// Reference: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances
function isNitro (instanceType)
{
	const nitro = [
    'A1', 'C5', 'C5d', 'C5n', 'G4', 'I3en', 'Inf1', 'M5', 'M5a', 'M5ad', 'M5d',
    'M5dn', 'M5n', 'p3dn.24xlarge', 'R5', 'R5a', 'R5ad', 'R5d', 'R5dn', 'R5n',
    'T3', 'T3a', 'z1d', 'a1.metal', 'c5.metal', 'c5d.metal', 'c5n.metal',
    'i3.metal', 'i3en.metal', 'm5.metal', 'm5d.metal', 'r5.metal', 'r5d.metal',
    'u-6tb1.metal', 'u-9tb1.metal', 'u-12tb1.metal', 'u-18tb1.metal',
    'u-24tb1.metal', 'z1d.metal'
  ]
  return (new RegExp(`^(?:${nitro.join('|')})`, 'i').test(instanceType))
}

// Converts a [networkInterfaceSet] into a simple dictionary of [nics]
function getNics (networkInterfaceSet)
{
  if (! networkInterfaceSet || ! networkInterfaceSet.length) {return false}
  return networkInterfaceSet.map(nic => ({
    'id': nic.NetworkInterfaceId,
    'subnet': nic.SubnetId,
    'vpc': nic.VpcId,
    'owner': nic.OwnerId,
    'status': nic.Status
  }))
}

// Collapses a [tagSet] into a more usable hash table of [tags]
function getTags (tagSet)
{
  if (tagSet === undefined || ! tagSet.length) {return null}
	let tags = {}
  tagSet.forEach(tag => tags[tag.Key] = tag.Value)
  return tags
}

Step 4: Go tag stuff!
For this to work, you will need to first add the Mirror tag with the appropriate Value to your VPC Traffic Mirror Targets and Filters. I’m using extrahop as my tag value because I mirror all my stuff to ExtraHop Reveal(x). Choose something intuitive that describes the target you’re sending to.

After you’ve tagged your Targets and Filters you’ll need to add the same tag to any EC2 instances you want to be managed for you. Easy right?

PRO TIP: if you include the Mirror tag in your EC2 Launch Templates, CloudFormation Templates, shell scripts, or otherwise ensure the instance is tagged at the time of creation (ie. ec2 run ...), the VPC Traffic Mirror Session(s) will be created for all attached ENIs before the instance is out of pending status, before it even boots! Think about that, you’ll never miss a packet this way… pretty neat.

I’ve also added the Mirror tag to my console view so I can quickly see where my packets are being mirrored, which looks like this:
image

Conclusion, for real …
So there you have it. If you’ve been following the full series, you now have a fully automated solution for managing your VPC Traffic Mirror Sessions in AWS, providing full coverage for most (all?) relevant operations, it’s scalable, and you can control the beast with a simple tag.

I’ve really enjoyed working through this and writing it up, it’s been a great learning experience, I can only hope someone else finds it useful as well. My next task is to learn about Lambda Applications so I can package this thing up and place it in the AWS Serverless Application Repository for click-n-play deployment… but where’s the fun in that, right? :slight_smile:

1 Like