Developing Serverless HTML Form on AWS Part2

Introduction

In Part1, we have built an application called “Serverless HTML Form” with serverless architecture. In this post, we are going to enhance this application.

The application in Part1 has a problem: the form always shows “Email sent!” even if a user had a typo in email address. In this case, the user can not receive the email but he don’t know what happened. There are 3 situations it may happen:

  1. The email is still being sent.
  2. The email failed to be sent.
  3. The email is sent but sent to the wrong address.

Therefore, we are going to implement “real time email status check” feature. With this feature, the user will know the situation and he can take some actions like:

  1. Sending: Wait more seconds to see the result.

  2. Failed: Depend on the reason. Maybe there are typos in email address, or the email server is down.

  3. Succeeded: Still not received? Maybe the email is in the spam box, or you just filled in your friend’s email address!

The user will be no longer just waiting and retrying submitting the form. It can improve the user experience!

In addition, I’m going to add reCAPTCHA to our form. reCAPTCHA can protect our form from spamming by bots. More info about reCAPTCHA, and also you can get the reCAPTCHA settings from here: https://www.google.com/recaptcha/intro/invisible.html

How to get email status?

The most difficult part of implementing “real time email status check” feature is: How to get email status?

After the SMTP server delivered an email, the status is only known by the SMTP server. Therefore, you need to dig into the SMTP server and find how to get the email status. Maybe you need to use a polling method to get the email status from the SMTP server or you need to monitor the log files. If you use another cloud service, there might be no way to get the email status.

Fortunately, Customers Mail Cloud (CMC) has a feature called “Event Webhook”. You can set an endpoint to Event Webhook, then CMC will send a HTTP POST request with the email status to your endpoint every time CMC delivered an email. In other words, you will be notified by CMC when CMC delivered an email. With the Event Webhook feature, implementing “real time email status check” feature becomes simple and possible.

Implement real time email status check

To implement the “real time email status check” feature, we need to integrate more cloud services:

  • Amazon DynamoDB: The database to store email status.
  • CMC Pro Plan: You need to upgrade your account to Pro so that you can use “Event Webhook” feature.

Because our Lambda function is going to access DynamoDB, we also need to add the AmazonDynamoDBFullAccess policy to the IAM role who executes the Lambda function.

The image below shows the new architecture of Serverless HTML Form.

f:id:hde-cm:20170411112810p:plain

Here are the descriptions of each component:

  • Amazon S3

    • Fileindex.html: The static webpage which contains HTML Form for user to fill in his email, name and inquiry.

  • Amazon API Gateway

    • Resource/submit-form: The endpoint of submitting the form. It triggers the Lambda function submit_form.
    • Resource/cmc-webhook: The endpoint of CMC Event Webhook. It triggers the Lambda function cmc_webhook.
    • Resource/check-status: The endpoint of checking email status. It triggers the Lambda function check_status.

  • AWS Lambda

    • Functionsubmit_form: Verify the data input by the user (reCAPTCHA, email format, …), then send emails using CMC HTTP-API and return message_id.
    • Functioncmc_webhook: Parse the data sent by CMC, and put the email status into DynamoDB.
    • Functioncheck_status: Get the email status from DynamoDB.

  • Amazon DynamoDB

    • Tablecmc_demo: Store the email status. (Primary key = message_id)

The image below shows the message flow:

f:id:hde-cm:20170411142127p:plain

Setup Amazon DynamoDB

  1. Click “Create table” in the AWS DynamoDB console.

  2. Give it a name and set the Primary key to message_id.

    f:id:hde-cm:20170411160208p:plain

  3. Click “Create”, and your table is now created.

Setup AWS Lambda

We need to create 3 Lambda functions for this application. The steps to create Lambda function are described in Part1. The only different things are code and environment variables.

  • Functionsubmit_form:

    • Environment variables:

      • MANAGER_EMAIL, CMC_USER, CMC_KEY, CMC_ENDPOINT, CMC_SENDFROM: Same as Part1.
      • RECAPTCHA_SECRET: Your reCAPTCHA secret.

    • Code:

""" submit_form """
from urllib.parse import urlencode
from urllib.request import urlopen
import json
import os
import re


RECAPTCHA_ENDPOINT = 'https://www.google.com/recaptcha/api/siteverify'
RECAPTCHA_SECRET = os.environ['RECAPTCHA_SECRET']

MANAGER_EMAIL = os.environ['MANAGER_EMAIL']
CMC_USER = os.environ['CMC_USER']
CMC_KEY = os.environ['CMC_KEY']
CMC_ENDPOINT = os.environ['CMC_ENDPOINT']
CMC_SENDFROM = os.environ['CMC_SENDFROM']

CONTENT_TO_USER = '''\
Dear {name}:

We have received your inquiry.
Thank you for your help!

Below is your inquiry:
===================================
{inquiry}
===================================
'''
CONTENT_TO_MANAGER = '''\
A user just send an inquiry.

Name: {name}
Email: {email}

Inquiry:
===================================
{inquiry}
===================================
'''


def cmc_sendmail(email_to, subject, content):
    payload = {
        'api_user': CMC_USER,
        'api_key': CMC_KEY,
        'to': email_to,
        'from': json.dumps({'name': '', 'address': CMC_SENDFROM}),
        'subject': subject,
        'text': content,
    }
    resp = urlopen(CMC_ENDPOINT, data=urlencode(payload).encode())
    body = json.loads(resp.read())

    print('mail to "%s" with subject "%s"' % (email_to, subject))
    print('    status_code =', resp.code)
    print('    text =', body)
    return body['id']


def verify_recaptcha(g_recaptcha_response):
    verify_data = {
        'secret': RECAPTCHA_SECRET,
        'response': g_recaptcha_response,
    }
    resp = urlopen(RECAPTCHA_ENDPOINT, data=urlencode(verify_data).encode())
    body = json.loads(resp.read())
    if body['success'] == True:
        return True
    else:
        print('verification_error =', body['error-codes'])
        return False


def lambda_handler(event, context):
    # Verify the reCAPTCHA.
    if not verify_recaptcha(event['g-recaptcha-response']):
        return {
            'success': False,
            'message': 'reCAPTCHA verification failed.',
        }

    # Verify the information that input by the user.
    email = event['email']
    name = event['name']
    inquiry = event['inquiry']
    if not re.search('^[a-zA-Z0-9_\.+-]+@[a-zA-Z0-9\.-]+\.[a-zA-Z]+$', email):
        return {
            'success': False,
            'message': 'Email Format not correct.',
        }

    # Send emails.
    mid_to_user = cmc_sendmail(
        email,
        'We have received your inquiry',
        CONTENT_TO_USER.format(name=name, inquiry=inquiry),
    )
    mid_to_manager = cmc_sendmail(
        MANAGER_EMAIL,
        'New Inquiry',
        CONTENT_TO_MANAGER.format(name=name, email=email, inquiry=inquiry),
    )

    return {
        'success': True,
        'mid_to_user': mid_to_user,
        'mid_to_manager': mid_to_manager,
    }
  • Functioncmc_webhook:

    • Environment variables:

      • DYNAMODB_TABLE_NAME: The table name you set in the previous section.

    • Code:

""" cmc_webhook  """
from urllib.parse import parse_qs
import json
import os

import boto3
db = boto3.resource('dynamodb').Table(os.environ['DYNAMODB_TABLE_NAME'])


def lambda_handler(event, context):
    # API Gateway doesn't parse the urlencoded. We need to parse by ourselves.
    cmc_payload = parse_qs(event['payload'])
    if cmc_payload['event_type'][0] != 'deliveries':
        return 'skip - event_type is not deliveries.'

    cmc_event = json.loads(cmc_payload['event'][0])

    for status in cmc_event['deliveries']:
        db.put_item(Item={
            'message_id': status['messageId'],
            'status': status['status'],
            'reason': status['reason'] if len(status['reason']) != 0 else ' ',
        })
        print('Put to database')
        print('    message_id =', status['messageId'])
        print('    status =', status['status'])
        print('    reason =', status['reason'])

    return 'OK'    # response to CMC.
  • Functioncheck_status:

    • Environment variables:

      • DYNAMODB_TABLE_NAME: The table name you set in the previous section.

    • Code:

""" check_status """
import os

import boto3
db = boto3.resource('dynamodb').Table(os.environ['DYNAMODB_TABLE_NAME'])

def lambda_handler(event, context):
    item = db.get_item(Key={'message_id': event['message_id']})
    print('Get from database:', item)
    return item.get('Item', None)

If you want to test cmc_webhook, the following Python3 code can help you generate the sample data which CMC will send to the Lambda function. You can copy the output of this code and paste it to “input test event” of Lambda function to test cmc_webhook.

import urllib.parse
import json

cmc_event = {'deliveries': [{
    'created': '2017-03-29 17:46:28',
    'envelopeFrom': 'example@example.com',
    'envelopeTo': 'example@example.com',
    'messageId': '<12345678-aaaa-bbbb-cccc-abcdef012345@mta01.sandbox.smtps.jp>',
    'reason': '',
    'senderIp': '153.149.8.217',
    'sourceIp': '',
    'status': 'succeeded',
    'subject': 'test email',
}]}

payload = {
    'api_key': 'key',
    'event': json.dumps(cmc_event),
    'event_type': 'deliveries',
    'server_composition': 'smtp',
}

print(json.dumps({
    'payload': urllib.parse.urlencode(payload)
}))

If you want to test check_status, you can use the follow text as the “input test event”.

{ "message_id": "<12345678-aaaa-bbbb-cccc-abcdef012345@mta01.sandbox.smtps.jp>" }

You can use print() in the Lambda function and read the stdout in CloudWatch.

Setup Amazon API Gateway

This time, we need to setup 3 resources in API Gateway.

  • Resource/submit-form: This is the same resource that you created in Part1. Here are the steps if you forget how to do.

    1. Create a resource under / and name it submit-form.
    2. Create a method under /submit-form and select POST.
    3. Choose “Lambda Function” as the Integration type, select the Lambda Region, and fill submit_form in Lambda Function.
    4. Enable CORS by clicking the resource /submit-form > Actions > Enable CORS > Enable CORS and replace existing CORS headers.

  • Resource/cmc-webhook: This is the endpoint of CMC Event Webhook. There is an additional setting we need to do because CMC sends the HTTP request in urlencoded format, which is not accepted by API Gateway by default. Note that we don’t need to enable CORS because CMC is not a browser!

    1. Create a resource under / and name it cmc-webhook.
    2. Create a method under /cmc-webhook and select POST.
    3. Choose “Lambda Function” as the Integration type, select the Lambda Region, and fill cmc_webhook in Lambda Function.

    4. Add a Body Mapping Template.

      1. Click the POST method under /cmc-webhook, then click “Integration Request”.

        f:id:hde-cm:20170411183318p:plain

      2. Expand the “Body Mapping Templates” section, and click “Add mapping template”. Fill in application/x-www-form-urlencoded, which is the content type that CMC gives.

        f:id:hde-cm:20170411183324p:plain

      3. After clicking the “v” button, the textarea will expand vertically. Fill the following text in:

        {"payload": $input.json('$')}

        This means that the original payload will be mapped into a JSON object, which has a key-value pair that key=“payload” and value=“{the original payload}”.

        f:id:hde-cm:20170411183327p:plain

      4. Click “Save”.

  • Resource/check-status: This is the endpoint to get email status. Note that this endpoint is relative to neither create nor update action. Therefore, using GET method is more suitable.

    1. Create a resource under / and name it check-status.
    2. Create a method under /check-status and select GET.
    3. Choose “Lambda Function” as the Integration type, select the Lambda Region, and fill check_status in Lambda Function.
    4. Enable CORS by clicking the resource /check-status > Actions > Enable CORS > Enable CORS and replace existing CORS headers.

    5. Add a Body Mapping Template. This time, we need to map the parameter specified in URL to a JSON object. Please follow the steps described above.

      • Content-Type: application/json
      • Template content: {"message_id": "$input.params('message_id')"}

After creating those API resources, your API will look like this:

f:id:hde-cm:20170412121254j:plain

At last, don’t forget to deploy the API by clicking Actions > Deploy API.

Setup Customers Mail Cloud

Add a new endpoint to the Event Webhook. Fill in the endpoint of /cmc-webhook of API Gateway, and make sure that the “deliveries” checkbox is checked. Therefore, every time CMC delivers an email, no matter whether it succeeded or not, CMC will send the result to /cmc-webhook.

//////Add an image of setting webhook in cmc. I can’t access the console.

Setup Amazon S3

In index.html, we need to add the reCAPTCHA div in our form to enable the reCAPTCHA feature. Remember to replace {YOUR_RECAPTCHA_SITEKEY} to your reCAPTCHA sitekey.

<form>
    <div class="form-group">
        <label for="email">Email address</label>
        <input type="email" class="form-control" id="email" name="email"
         placeholder="Email" />
    </div>
    <div class="form-group">
        <label for="name">Name</label>
        <input type="text" class="form-control" id="name" name="name"
         placeholder="Name" />
    </div>
    <div class="form-group">
        <label for="inquiry">Inquiry</label>
        <textarea class="form-control" rows="5" id="inquiry" name="inquiry"
         placeholder="Your Inquiry"></textarea>
    </div>
    <div class="form-group">
        <div class="g-recaptcha" data-sitekey="{YOUR_RECAPTCHA_SITEKEY}"></div>
        <button type="submit" class="btn btn-success">Submit</button>
    </div>
    <div id="message"></div>
</form>

The Javascript part also needs some changes:

  • When submitting the form, send the value of g-recaptcha-response so that the Lambda function submit_form can verify it.

  • After submitting the form, it should send HTTP GET request to /check-status to get the email status every 5 seconds until it succeeded or failed.

var API_GATEWAY_ENDPOINT = '{API_GATEWAY_ENDPOINT}';
var UPDATE_RATE = 5;

function show_message(type, message) {
    $('#message').empty().append(
        $('<div class="alert alert-' + type + '" role="alert">').html(message)
    );
}

function check_status(message_id) {
    $.ajax({
        url: API_GATEWAY_ENDPOINT + '/check-status',
        method: 'get',
        data: {message_id: message_id},
        contentType: 'application/json'
    })
    .done(function(data) {
        console.log('ajax done');

        if(data == null) {
            //still sending, check again next time.
            setTimeout(function(){
                check_status(message_id);
            }, UPDATE_RATE*1000);

        } else if(data.status == 'succeeded') {
            show_message('success', (
                'The confirmation email was sent!<br>' +
                "If you haven't received it, please check your spam box."
            ))

        } else if(data.status == 'failed') {
            show_message('danger', (
                'Failed to send confirmation email to you.<br>' +
                'Reason: ' + data.reaseon
            ))
        }
    })
    .fail(function(data) {
        console.log('ajax failed.', data);
        show_message('danger', (
            'Failed to get email status.<br>' +
            'Please check your network connection.'
        ))
    });
}

$(function() {
    $('form').submit(function(event) {
        event.preventDefault();
        $.blockUI();

        $.ajax({
            url: API_GATEWAY_ENDPOINT + '/submit-form',
            method: 'post',
            data: JSON.stringify({
                email: $('#email').val(),
                name: $('#name').val(),
                inquiry: $('#inquiry').val(),
                'g-recaptcha-response': $('#g-recaptcha-response').val()
            }),
            contentType: 'application/json'
        })
        .done(function(data) {
            if(data.success == true) {
                console.log('mid_to_user =', data.mid_to_user);
                console.log('mid_to_manager =', data.mid_to_manager);

                show_message('info', 'Sending email...')

                //We only care about the email which sent to user.
                setTimeout(function() {
                    check_status(data.mid_to_user);
                }, UPDATE_RATE*1000);

            } else {
                show_message('danger', data.message);
            }
            $.unblockUI();
        })
        .fail(function(data) {
            show_message('danger', 'Failed to send inquiry.')
            console.log(data.responseText);
            $.unblockUI();
        });
    });
});

The upload method and permission are the same as described in Part1.

We are done!

Now, our Serverless HTML Form can show the email status to the user, and have the ability to prevent bots from spamming. The most important thing is, it’s still serverless!

With those various cloud services, we can build the application by integrating them. Setting up servers is no longer a requirement. Let’s start to think with cloud!