SMS Bot

Twilio provides Api’s to help developers integrate SMS into their applications. You can easily send automated text messages with Twilio, and also use webhooks to receive text messages. In this tutorial, we will be creating a bot with a SMS interface that allows us to text links to the bot and have those links saved in a Notion Database.

Other Data Stores
In this tutorial, we are using Notion to store our links, but the tutorial is structured such that you will only need to change the last step to send the links to a different service.

Create Twilio Account

You can sign up for a trial Twilio account that can be used for this tutorial. The trial account will come with a credit balance that is more than sufficient for this project. The account will also have access to all features needed for this tutorial. At the end of the tutorial, you can delete your account or upgrade it to a full Twilio account without losing your work.

When creating your account, you will have to verify both an email and phone number. When you are asked to add some information about how you will be using Twilio, you can use the following settings change_app_name


Now that your trial account is set up, you can transition to creating your Cdev project. Once we have set up the template project, we will go back to the Twilio Console to create our phone number and webhook.


Create Cdev Project

We will be starting this tutorial from the quick-start-twilio template.

cdev init link-bot --template quick-start-twilio
pip install -r requirements.txt

Now we can deploy our project to get a live Webhook

cdev deploy
Deploy Output

This step should output the live url of your webhook and look like

Base API URL -> <your-endpoint>

We can check that the webhook is live and working by running

curl -X POST https://<your-endpoint>/twilio_handler

You should receive output like

<?xml version="1.0" encoding="UTF-8"?><Response><Message>Hi from your backend!</Message></Response>
Custom Responses

In src/ink_bot/handlers.py, you will see that the route event is set up to reply back with application/xml. This will map the return value from the Serverless Function as the Body of the HTTP Response with type application/xml. If you need more customization over the HTTP Response, you can use a more detailed return value

return {
    "isBase64Encoded": False,
    "statusCode": 200,
    "headers": { 
        "Content-Type": "application/xml"
    },
    "body": response.to_xml()
}

Connect the Twilio Number to the Webhook

We can now go back to the Twilio Console to create our phone number and attach our webhook. In the console, create your first phone number. change_app_name


Now that we have a Twilio Phone Number, create a Messaging Service. change_app_name

change_app_name


Then add a sender. By default, it will add your created number. change_app_name

change_app_name


Now add the webhook information. You will need to add the generated url for your webhook from your Project.

Retrieving your url

If you do not remember your url that was outputted by the deployment, you can run the following command to retrieve your url

cdev output link_bot.api.demoapi.endpoint
endpoint -> <your-endpoint>

Make sure to add the /twilio_handler path when adding it to the Twilio Console.

change_app_name


Finally, you can leave the Business Registration information empty and complete your Service

change_app_name
Business Registration
If you decided to upgrade your trial account into a full Twilio account, you will need to add some information about how you use the service. This helps Twilio detect spammers and stay in compliance with their carriers.

You can now send a test text message from your number to yourself. change_app_name

change_app_name


You should receive your demo text message from your Twilio Number! You can then reply to the number, which will trigger your webhook and reply back with "Hi from your backend!".


Congratulations 🎉 You have created a live bot! We will now be adding more logic to our backend to have the bot save links that we send!

Create a Contact
Although not necessary, it is fun to save your number in your contacts and give it a photo to make the bot feel more alive!

Structure Incoming Messages

Looking at the logs of the current handler, we can see that the first step to parsing our message is structuring the data that is passed to our web hook. Twilio passed the data about the incoming message as a query string that is Base64 encoded. We need to decode this data and turn into a structure that we can work with.

See the logs

You can check the logs with the following command.

cdev run function.logs link_bot.twilio_handler

In the logs, you should see a message like

event body -> VG9Db3VudHJ5PVVTJlRvU3RhdGU9TUkmU21zTWVzc2FnZVNpZD1TTTIyMmUxNTNkNTYwZTc4Njg2YTBlMjRmMDkwMmY1YTM4Jk51bU1lZGlhPTAmVG9DaXR5PUdSQU5EK0JMQU5DJkZyb21aaXA9MjgwMjcmU21zU2lkPVNNMjIyZTE1M2Q1NjBlNzg2ODZhMGUyNGYwOTAyZjVhMzgmRnJvbVN0YXRlPU5DJlNtc1N0YXR1cz1yZWNlaXZlZCZGcm9tQ2l0eT1DT05DT1JEJkJvZHk9SGVsbG8rbXkrbmV3K0JvdCtGcmllbmQlMjErJkZyb21Db3VudHJ5PVVTJlRvPSUyQjE4MTA0NDI0NzIyJk1lc3NhZ2luZ1NlcnZpY2VTaWQ9TUc2ZjM2ZDdiMjkzNzhiM2YyMzAyMjMyYzNhNGVhNGU5MyZUb1ppcD00ODQzOSZOdW1TZWdtZW50cz0xJlJlZmVycmFsTnVtTWVkaWE9MCZNZXNzYWdlU2lkPVNNMjIyZTE1M2Q1NjBlNzg2ODZhMGUyNGYwOTAyZjVhMzgmQWNjb3VudFNpZD1BQzcxMDc1MTQ4OGVkZWRlMGRkOWMwYTNkOTkyODBkNmQ2JkZyb209JTJCMTcwNDQ5MDY0NjImQXBpVmVyc2lvbj0yMDEwLTA0LTAx

Create a src/link_bot/serializer.py file and add the following code to the file. The work for deserializing the data from Twilio is done in the TwilioWebhookEvent class initializer (lines 11-13).

    
from aws_lambda_powertools.utilities.data_classes import event_source, APIGatewayProxyEventV2 from aws_lambda_powertools.utilities.data_classes.common import DictWrapper import base64 from urllib.parse import parse_qs from typing import Dict, Any class TwilioWebhookEvent(APIGatewayProxyEventV2): def __init__(self, data: Dict[str, Any]): body_str = base64.b64decode(data.get('body')).decode("utf-8") parsed_data = {k:(v[0] if len(v)==1 else v) for k,v in parse_qs(body_str).items() } data['body'] = TwilioWebhookData(parsed_data) super().__init__(data) @property def body(self) -> 'TwilioWebhookData': """ TwilioWebhookData: Wrapper around the data send from a Twilio Webhook. More details can be found on their [official documentation](https://www.twilio.com/docs/messaging/guides/webhook-request). """ return self.get("body") class TwilioWebhookData(DictWrapper): """ Wrapper around the data send from a Twilio Webhook. More details can be found on their [official documentation](https://www.twilio.com/docs/messaging/guides/webhook-request). """ @property def MessageSid(self) -> str: "A 34 character unique identifier for the message. May be used to later retrieve this message from the REST API." return self.get("MessageSid") @property def SmsSid(self) -> str: "Same value as MessageSid. Deprecated and included for backward compatibility." return self.get("SmsSid") @property def AccountSid(self) -> str: "The 34 character id of the [Account](https://www.twilio.com/docs/iam/api/account) this message is associated with." return self.get("AccountSid") @property def MessagingServiceSid(self) -> str: "The 34 character id of the [Messaging Service](https://www.twilio.com/docs/sms/send-messages#messaging-services) associated with the message." return self.get("MessagingServiceSid") @property def From(self) -> str: "The phone number or [Channel address](https://www.twilio.com/docs/messaging/channels#channel-addresses) that sent this message." return self.get("From") @property def To(self) -> str: "The phone number or [Channel address](https://www.twilio.com/docs/messaging/channels#channel-addresses) address of the recipient." return self.get("To") @property def Body(self) -> str: "The text body of the message. Up to 1600 characters long." return self.get("Body") @property def NumMedia(self) -> str: "The number of media items associated with your message" return self.get("NumMedia") @property def ReferralNumMedia(self) -> str: "The number of media items associated with a 'Click to WhatsApp' advertisement." return self.get("ReferralNumMedia")
Powertools Dataclasses
The TwilioWebhookEvent is derived from a data class from the Lambda Powertools library. This library provides a light weight mechanism to provide additional information about the triggering event for a handler. This allows developers to be given type hints when using the object in the handler.

We can now update our webhook to use our created class to have easier access to the data from Twilio. Update your handlers.py to the following code. The newly created twilio_event object will provided access to all the available data from the Twilio event.

    
from twilio.twiml.messaging_response import MessagingResponse from cdev.aws.lambda_function import ServerlessFunction from .api import twilio_webhook_route from .serializer import TwilioWebhookEvent @ServerlessFunction("twilio_handler", events=[twilio_webhook_route.event("application/xml")]) def twilio_handler(event, context): twilio_event = TwilioWebhookEvent(event) print(f"Received message -> {twilio_event.body.Body}; from -> {twilio_event.body.From}") response = MessagingResponse() response.message("Hi from your backend!") return response.to_xml()

Now that we are able to understand the data provided to our webhook by Twilio, we need to define the structure of the message that we will support. Our bot will receive a link to store, a description of the link, and set of tags.

Create a file called link_service.py, and add the following code. This code provides the business logic for taking a message and returning the url, description, and list of tags. It uses a regular expression to parse the url from the beginning of the message, then parses the remaining part of the message into the description and tags based on the presence of the # character. If there is not a url present at the beginning of the message, it raises an exception.

    
import re from typing import List, Tuple starts_with_url_regex_str = r"^(?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'\".,<>?«»“”‘’])" starts_with_url_regex = re.compile(starts_with_url_regex_str) hash_tag_regex_str = "(#[\S]*)" hash_tag_regex = re.compile(hash_tag_regex_str) class ParsingError(Exception): def __init__(self, message: str): self.message = message super().__init__(self.message) def __str__(self): return f'Parsing Error -> {self.message}' def parse_message(message: str) -> Tuple[str, str, List[str]]: """Service to parse a text message into structured data about a link. Args: message (str): the message Returns: str: The Url str: Description for the Url List[str]: Tags for the Url Raises: ParsingError: An error occurred parsing the message. """ matched_url = starts_with_url_regex.search(message) if matched_url: _, url_end = matched_url.span() url = message[:url_end] description_tags = message[url_end+1:] if description_tags: description, tags = _parse_description_tags(description_tags) else: description = "" tags = [] return url, description, tags raise ParsingError(f"Could not match url in {message}") def _parse_description_tags(body: str) -> Tuple[str, List[str]]: """From a string of the form: <description> [#<tag>] parse the description and list of tags. Args: body (str): String contains the description and tags Returns: str: Description List[str]]: Tags Example: input: 'intersting article #blog #python #learning' output: ('intersting article', ['blog', 'python', 'learning']) """ parts = body.split("#", 1) description = parts[0].strip() if len(parts) == 1: return description, [] raw_hashtags = "#" + parts[1] try: hash_tag_list = hash_tag_regex.findall(raw_hashtags) hash_tags = [x[1:] for x in hash_tag_list] except Exception as e: hash_tags = [] return description, hash_tags

We can now update our web hook to use the link_service. Update your handler.py to the following. Once deployed, you can see the updates by texting your bot and checking your function’s logs.

    
from twilio.twiml.messaging_response import MessagingResponse from cdev.aws.lambda_function import ServerlessFunction from .api import twilio_webhook_route from .serializer import TwilioWebhookEvent from .link_service import parse_message, ParsingError @ServerlessFunction("twilio_handler", events=[twilio_webhook_route.event("application/xml")]) def twilio_handler(event, context): twilio_event = TwilioWebhookEvent(event) print(f"Received message -> {twilio_event.body.Body}; from -> {twilio_event.body.From}") response = MessagingResponse() try: url, description, tags = parse_message(twilio_event.body.Body) print(f"{twilio_event.body.Body} -> ({url}, {description}, {tags})") response.message("Parsed Message!") except ParsingError as e: response.message(e.message) return response.to_xml()

Save Information Into Notion

Now that we have our messages properly structured, we can use the Notion Api to store our links. Our webhook will make a request to the Notion Api, then if successful, reply back to the user that the link and additional data was stored.

You can sign up for a free personal Notion account or use an existing account. We chose to use Notion because we use it for note collection and other tasks, but you could use any other service that has a publicly accessible Api.


Create an Integration

To send data through the Notion Api, we need to create an integration. This integration will have the permissions needed to add data to our Notion Database. To create an integration, go to your accounts Settings and Members Page, then the Integrations Tab. change_app_name

When configuring your integration, you will need to grant it permissions to read, update, and insert content, but you do not need to grant any user information. If you are signed into multiple workspaces, make sure you select the desired workspace in the Associated Workspace dropdown.

change_app_name


After confirming the integration settings, you will be taken to the integration page. Note your integration token because it will be used by our webhook to authenticate with the Notion Api.

change_app_name

Create the Notion Database

Now that the integration is complete, you can create a Database in your workspace that will be used to the store the information. Note that to work with the code provided in this tutorial, you will need to make the properties of the database match our database. The properties should be:

  • Description (Title)
  • link (URL)
  • Tags (Multi-select)
  • Status (Multi-select)

Also note that you will need the Database id. When on the Database page, this can be found by looking at the url of the page. The Database id will be the sequence of characters before the ?. In the example image, our Database id is: 5416acb700044512a5d02e9cea7dfb93. change_app_name


Finally, you will need to grant the integration access to this database. You can do this by using the Share button in the top right of the page.

change_app_name

Save with the Notion Api

Create a new file called notion_service.py and add the following code. This code uses the Notion Api to save the provided data. Like the link_service this code is designed to be independent of the computing platform it runs on.

Different Database Set Up
If your Notion Database has a different set of properties, you will need to adjust the return value of the create_properties function. You can learn more about setting properties from the Notion Api Documentation.
    
import os from notion_client import Client from typing import Dict, List NOTION_TOKEN = "NOTION_TOKEN" NOTION_DB_ID = "NOTION_DB_ID" notion_token = os.environ.get(NOTION_TOKEN) notion_db_id = os.environ.get(NOTION_DB_ID) notion_client = Client(auth=notion_token) def save_info_notion(url: str, description: str, tags: List[str]) -> str: print(f"url {url}; description {description}; tags {tags}") notion_client.pages.create(**{ "parent": { "database_id": notion_db_id }, "properties": create_properties(description, url, tags) } ) print(f"Added info to database {notion_db_id}") return u"Saved to Notion \u2705" def create_properties(description: str, url: str, tags: List[str]) -> Dict: tag_list = [{"name": x} for x in tags] return { "Description": { "title": [ { "text": { "content": description } } ], }, "link": { "url": url, }, "Tags": { "multi_select": tag_list }, "Status": { "multi_select": [ {"name":"Unread"} ] }, }

We need to pass the NOTION_TOKEN and NOTION_DB_ID into the environment of the handler. We will create a custom settings class and use that to pass the values. Create a file called src/link_bot_settings.py and add the following code.

    
from core.constructs.settings import Settings class LinkBotSettings(Settings): NOTION_SECRET: str = "" NOTION_DB_ID: str = "<your-db-id>" # ex: 5416acb700044512a5d02e9cea7dfb93

You then need to add the NOTION_TOKEN value using a file in the settings folder. Create a file called settings/<your-environment-secrets>/cdev_notion_secret then paste your value into the file.

Set this as the settings for your environment using the following command

cdev environment settings_information --key base_class --new-value src.link_bot_settings.LinkBotSettings

Finally, update your handler.py to the following to use the new service and pass the environment variables. Then deploy the changes.

    
from twilio.twiml.messaging_response import MessagingResponse from cdev.aws.lambda_function import ServerlessFunction from cdev import Project as cdev_project from .api import twilio_webhook_route from .serializer import TwilioWebhookEvent from .link_service import parse_message, ParsingError from .notion_service import save_info_notion from .notion_service import NOTION_DB_ID, NOTION_TOKEN from ..link_bot_settings import LinkBotSettings myProject = cdev_project.instance() mySettings: LinkBotSettings = myProject.settings notion_env_vars = { NOTION_DB_ID: mySettings.NOTION_DB_ID, NOTION_TOKEN: mySettings.NOTION_SECRET } twilio_webhook_permissions = [] twilio_webhook_env_vars = {**notion_env_vars} @ServerlessFunction("twilio_handler", events=[twilio_webhook_route.event("application/xml")], environment=twilio_webhook_env_vars, permissions=twilio_webhook_permissions ) def twilio_handler(event, context): twilio_event = TwilioWebhookEvent(event) print(f"Received message -> {twilio_event.body.Body}; from -> {twilio_event.body.From}") response = MessagingResponse() try: url, description, tags = parse_message(twilio_event.body.Body) print(f"{twilio_event.body.Body} -> ({url}, {description}, {tags})") try: return_message = save_info_notion(url, description, tags) response.message(return_message) except Exception as e: response.message("Could not save to Notion") except ParsingError as e: response.message(e.message) return response.to_xml()

You should now be able to text your bot and have the data saved to Notion🎉