Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi 9783895765315, 9783895765322, 3895765317

Table of contents :
Search…
Programming Voice-controlled IoT Applications
All rights reserved.
Contents
About the Author
Introduction
1 • Chapter 1 Alexa History and Devices
1.1 Alexa voice service and AWS Lambda
1.2 Pricing
1.3 Alexa skills
1.4 Supported programming languages
1.5 Terminology – Invocation, Utterances, Intents and Slots
1.5.1 Alexa Wake word
1.5.2 Invocation
1.5.3 Utterances
1.5.4 Intents and requests
1.5.5 Slots
1.5.6 Interaction model
1.5.7 Endpoints
1.5.8 Regions
1.6 Skill Sessions
1.7 Session attributes
1.8 Request and response JSON
1.9 Blueprint skills
1.10 Summary
1.11 References
2 • Creating your Amazon Account
2.1 Introduction
2.2 Create your Amazon account
2.3 Your sk
2.4 Hosting
2.5 Summary
3 • Creating an Alexa Skill
3.1 Introduction
3.2 Your first skill
3.2.1 The interaction model
3.2.2 Choose a method
3.2.3 The Invocation Name
3.2.4 The Intents
3.2.5 The code
3.3 Testing your skill
3.4 Skill I/O
3.4.1 Skill request
3.4.2 Skill response
3.4.3 Speech Synthesis Markup Language (SSML)
3.5 Code editing
3.5.1 Edit the HelloWorldIntentHandler code
3.5.2 Add some debug code
3.6 Test your code
3.7 Utility code
3.8 Debugging
3.9 Node.js differences
3.10 Node.js debugging
3.11 Summary
4 • Slots and Dialogs, Saving Session Data
4.1 Introduction
4.2 Slots in action
4.3 Slot skill
4.3.1 Invocation Name
4.4 Skill flow
4.5 Add the intent to our skill
4.6 Evaluate your model
4.6.1 The JSON editor
4.7 Accessing the slot
4.8 The code
4.8.1 Test your skill
4.9 Session attributes - saving slot values
4.9.1 Remember their name
4.10 Dialog delegation
4.11 The Birthday code
4.12 Handling Yes and No intents
4.13 Multiple Yes / No sources
4.14 AMAZON.SearchQuery
4.15 ASK SDK Utilities
4.16 Intent error logging
4.17 Language understanding NLU and Automatic speech recognition ASR
4.18 Summary
5 • S3 Storage and DynamoDB Database
5.1 Introduction
5.2 Local storage
5.3 Persistent attributes, DynamoDB and S3
5.3.1 Code example
5.3.2 DynamoDB database storage
5.4 Request and response interceptors
5.5 DynamoDB
5.6 S3 storage
5.7 Summary
6 • Certification and Publishing
6.1 Introduction
6.2 Adding further languages
6.3 Distribution
6.4 Availability and Beta Testing
6.5 Beta Tester
6.6 Validation
6.7 Submission
6.8 Post Publication
6.9 Analytics
6.10 Summary
6.11 References
7 • Creating Skills with Lambda and ASK CLI
7.1 Introduction
7.1.1 AWS Lambda skill
7.2 ASK CLI
7.3 Visual Studio code
7.4 Local debugging
7.4.1 Add Alexa debugger configuration
7.4.2 Test your Alexa skill in VS code
7.5 Summary
7.6 References
8 • Alexa Presentation Language – APL
8.1 Introduction
8.2 APLA
8.2.1 APLA components
8.3 Datasources
8.4 APLA datasource example
8.5 Adding an APLA reprompt
8.6 Summary
8.7 References
9 • APL Visual Multimodal Responses
9.1 Introduction
9.2 Creating an APL Visual Response
9.3 Visual Components
9.4. APL component example
9.5 Using the Authoring Tool
9.6 Integrating APL and code
9.6.1 Check for screen support
9.7 APL Commands
9.7.1 Standard Commands
9.7.2 Media Commands
9.7.3 User-defined commands
9.7.4 Execute Commands directive
9.8 Responsive components and Alexa Layouts
9.9 Converting Text to speech – using Transformers
9.9.1 Transformer APL design
9.9.2 Operation
9.10 Summary
9.11 References
10 • Alexa In-skill Purchasing (ISP)
10.1 Introduction
10.2 Create your ISP skill
10.3 Accessing your ISP code
10.4 Retrieve in-skill products, get their information and purchase.
10.5 Produce detail and purchase
10.6 Purchase
10.6.1 Failed Purchase
10.6.2 Refunds
10.7 References
11 • Progressive Response - Accessing the Internet
11.1 Introduction
11.2 Steps to Send a Progressive Response
11.3 Progressive response example
11.3.1 Code response
11.4 asyncio, async and await - awaiting a web response
11.5 References
12 • Creating a Raspberry Pi IoT Thing
12.1 Introduction
12.2 Create a Raspberry Pi IoT
12.2.1 a) Create our ‘Thing’ and its certificates
12.2.1 b) Thing’s endpoint
12.2.1 c) Transfer the certificates to your Pi
12.2.2 Create and run the Python code
12.2.3 Send messages to your Pi
12.2.4 Create an Alexa-Hosted Skill
12.2.5 Test the skill
12.3 Add intents to the Alexa skill
12.4 Control the robot
12.5 Add intent handlers to the skill code
12.6 Modify your code
12.6.1 Modify your Pi code - LED
12.6.2 Modify your Pi code - explorerhat
12.7 Test your robot or LED
12.8 Summary
13 • Smart Home Devices
13.1 Introduction
13.2 Alexa Interfaces
13.3 Login with Amazon (LWA)
13.3.1 Create a security profile
13.4 Create your Smart Home Skill
13.5 Create a Lambda function
13.6 Lambda skill code
13.7 Test your Lambda function
13.8 Link the function to the skill
13.9 Configure account linking
13.10 Enable and Link the skill
13.11 Clean up
13.12 Troubleshooting
13.13 Summary
13.14 References
14 • Controlling a smart home raspberry Pi with SQS
14.1 Introduction
14.2 Create an SQS Queue
14.3 Raspberry Pi SQS code
14.4 Create a Smart Home skill
14.5 Create the function
14.6 Create a security profile
14.7 Configure the smart home skill
14.8 Add the function code
14.9 Test the function
14.10 Discover your device
14.11 Test from an Alexa device
14.12 Clean up
14.13 Summary
14.14 References
15 • IoT, Pi and Node-RED
15.1 Introduction
15.2 Prerequisites
15.3 Installation
15.4 Running node-RED
15.5 Node-RED user interface
15.6 First flow design - Hello world
15.7 Hardware I/O
15.7.1 Add an input
15.8 Using the Sense Hat
15.9 Node-RED dashboard
15.10 Sense Hat output
15.11 IoT - Receiving MQTT messages
15.12 Create a new IoT thing for MQTT communication
15.12.1 Subscribe to a topic
15.13 Node-RED IoT Application
15.14 Receiving MQTT messages
15.15 Summary
16 • Proactive Events – Sending Raspberry Pi Alexa Notifications
16.1 Introduction
16.2 The Lambda function
16.3 Send a notification
16.4 Code to get the access token
16.5 Send the notification
16.6 Summary
16.7 References
17 • Raspberry Pi as a Stand-alone Alexa Device
17.1 Introduction
17.2 Raspberry Pi setup
17.3 Procedure
17.3.1 Register your AVS device with Amazon
17.3.2 Download and install the AVS SDK
17.3.3 Run and authorize the sample app
17.4 Use the sample app
17.5 Summary
17.6 References
18 • Conclusion
Index

Citation preview

books books

books

with Alexa and Raspberry Pi The book is split into two parts: the first part covers creating Alexa skills and the second part, designing Internet of Things and Smart Home devices using a Raspberry Pi. The first chapters describe the process of Alexa communication, opening an Amazon account and creating a skill for free. The operation of an Alexa skill and terminology such as utterances, intents, slots, and conversations are explained. Debugging your code, saving user data between sessions, S3 data storage and Dynamo DB database are discussed. In-skill purchasing, enabling users to buy items for your skill as well as certification and publication is outlined. Creating skills using AWS Lambda and ASK CLI is covered, along with the Visual Studio code editor and local debugging. Also covered is the process of designing skills for visual displays and interactive touch designs using Alexa Presentation Language. The second half of the book starts by creating a Raspberry Pi IoT “thing” to control a robot from your Alexa device. This covers security issues and methods of sending and receiving MQTT messages between an Alexa device and the Raspberry Pi. Creating a smart home device is described including forming a security profile, linking with Amazon, and writing a Lambda function that gets triggered by an Alexa skill. Device discovery and on/off control is demonstrated. Next, readers discover how to control a smart home Raspberry Pi display from an Alexa skill using Simple Queue Service (SQS) messaging to switch the display on and off or change the color. A node-RED design is discussed from the basic user interface right up to configuring MQTT nodes. MQTT messages sent from a user are displayed on a Raspberry Pi. A chapter discusses sending a proactive notification such as a weather alert from a Raspberry Pi to an Alexa device. The book concludes by explaining how to create Raspberry Pi as a stand-alone Alexa device.

SKU20400_COV_Programming Voice-controlled IoT Applications_v02.indd Alle pagina's

John Allwork graduated from Sheffield University where he developed an interest in computers and gained his MSc at UMIST. After two years working for ICL as a design engineer, he returned to UMIST where he graduated with a PhD in ‘Design and Development of Microprocessor Systems’. He worked for several years in technical support and as manager in electronics distribution, working closely with Intel engineers and later designing Inmos Transputer systems. Having taught electronics at Manchester Metropolitan University, he retired in 2011 but retained his interest in electronics and programming. His other occupations consist of traveling, walking, geocaching and spending time on his allotment.

Elektor International Media www.elektor.com

Programming Voice-controlled IoT Applications • Dr John Allwork

Programming Voice-controlled IoT Applications

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

tials import creden

” # constants T%H:%M:%S.00Z d % m % Y % “ = h/o2/token” t u a / m o c UTC_FORMAT . n o z a m https://api.a TOKEN_URI = “ s constants # Token acces CLIENT_ID’] ‘ [ y e k . s l a i t n T’] rede ‘CLIENT_SECRE CLIENT_ID = c [ y e k . s l a i t n e = cred CLIENT_SECRET ken():

s_to def get_acces

= { ”, token_params t_credentials n e i l c “ : ” e p “grant_ty ve_events”, i t c a o r p : : a x e “scope”: “al LIENT_ID, C : ” d i _ t n e i l “c CLIENT_SECRET : ” t e r c e s _ t n “clie }

= { set=UTF-8” r a h c ; n o s j / token_headers n o ”: “applicati “Content-Type } oken_headers t = s r e d a e h , I _UR Dr John Allwork ts.post(TOKEN s e u q e r = e s n respo

16-01-2023 11:08

● This is an Elektor Publication. Elektor is the media brand of Elektor International Media B.V.

PO Box 11, NL-6114-ZG Susteren, The Netherlands Phone: +31 46 4389444

● All rights reserved. No part of this book may be reproduced in any material form, including photocopying, or

storing in any medium by electronic means and whether or not transiently or incidentally to some other use of this publication, without the written permission of the copyright holder except in accordance with the provisions of the Copyright Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licencing Agency Ltd., 90 Tottenham Court Road, London, England W1P 9HE. Applications for the copyright holder's permission to reproduce any part of the publication should be addressed to the publishers.

● Declaration

The Author and Publisher have used their best efforts in ensuring the correctness of the information contained in this book. They do not assume, and hereby disclaim, any liability to any party for any loss or damage caused by errors or omissions in this book, whether such errors or omissions result from negligence, accident, or any other cause. All the programs given in the book are Copyright of the Author and Elektor International Media. These programs may only be used for educational purposes. Written permission from the Author or Elektor must be obtained before any of these programs can be used for commercial purposes.

● British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library

● I SBN 978-3-89576-531-5 Print

ISBN 978-3-89576-532-2 eBook

● F irst edition

© Copyright 2023: Elektor International Media B.V. Editor: Alina Neacsu Prepress Production: Jack Jamar | Graphic Design, Maastricht

Elektor is part of EIM, the world's leading source of essential technical information and electronics products for pro engineers, electronics designers, and the companies seeking to engage them. Each day, our international team develops and delivers high-quality content - via a variety of media channels (including magazines, video, digital media, and social media) in several languages - relating to electronics design and DIY electronics. www.elektormagazine.com

●4

Contents

Contents About the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Chapter 1 Alexa History and Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.1 Alexa voice service and AWS Lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.2 Pricing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3 Alexa skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4 Supported programming languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5 Terminology – Invocation, Utterances, Intents and Slots . . . . . . . . . . . . . . . . . . 16 1.5.1 Alexa Wake word . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5.2 Invocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5.3 Utterances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5.4 Intents and requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.5.5 Slots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.5.6 Interaction model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.5.7 Endpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.5.8 Regions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.6 Skill Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.7 Session attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.8 Request and response JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.9 Blueprint skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.11 References: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Chapter 2 Creating your Amazon Account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2 Create your Amazon account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.3 Your skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.4 Hosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Chapter 3 Creating an Alexa Skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2 Your first skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

●5

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

3.2.1 The interaction model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.2 Choose a method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.3 The Invocation Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.2.4 The Intents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2.5 The code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.3 Testing your skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.4 Skill I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.4.1 Skill request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.4.2 Skill response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.4.3 Speech Synthesis Markup Language (SSML) . . . . . . . . . . . . . . . . . . . . . . . 43 3.5 Code editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.5.1 Edit the HelloWorldIntentHandler code . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.5.2 Add some debug code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.6 Test your code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.7 Utility code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.8 Debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.9 Node.js differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.10 Node.js debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Chapter 4 Slots and Dialogs, Saving Session Data . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.2 Slots in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.3 Slot skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.3.1 Invocation Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.4 Skill flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.5 Add the intent to our skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 4.6 Evaluate your model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.6.1 The JSON editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.7 Accessing the slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.8 The code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 4.8.1 Test your skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.9 Session attributes - saving slot values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.9.1 Remember their name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.10 Dialog delegation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.11 The Birthday code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 4.12 Handling Yes and No intents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 4.13 Multiple Yes / No sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

●6

Contents

4.14 AMAZON.SearchQuery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.15 ASK SDK Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.16 Intent error logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.17 Language understanding NLU and Automatic speech recognition ASR . . . . . . . . 82 4.18 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Chapter 5 S3 Storage and DynamoDB Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2 Local storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.3 Persistent attributes, DynamoDB and S3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.3.1 Code example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 5.3.2 DynamoDB database storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.4 Request and response interceptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.5 DynamoDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.6 S3 storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Chapter 6 Certification and Publishing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.2 Adding further languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 6.3 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.4 Availability and Beta Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 6.5 Beta Tester . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.6 Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 6.7 Submission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 6.8 Post Publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 6.9 Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 6.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 6.11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Chapter 7 Creating Skills with Lambda and ASK CLI . . . . . . . . . . . . . . . . . . . . . . . . 107 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.1.1 AWS Lambda skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.2 ASK CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 7.3 Visual Studio code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 7.4 Local debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 7.4.1 Add Alexa debugger configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 7.4.2 Test your Alexa skill in VS code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

●7

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

7.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 7.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Chapter 8 Alexa Presentation Language – APL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 8.2 APLA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 8.2.1 APLA components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 8.3 Datasources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.4 APLA datasource example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 8.5 Adding an APLA reprompt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 8.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Chapter 9 APL Visual Multimodal Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 9.2 Creating an APL Visual Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 9.3 Visual Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 9.4. APL component example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 9.5 Using the Authoring Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 9.6 Integrating APL and code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 9.6.1 Check for screen support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 9.7 APL Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 9.7.1 Standard Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 9.7.2 Media Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 9.7.3 User-defined commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 9.7.4 Execute Commands directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 9.8 Responsive components and Alexa Layouts . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 9.9 Converting Text to speech – using Transformers . . . . . . . . . . . . . . . . . . . . . . . 151 9.9.1 Transformer APL design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 9.9.2 Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 9.9.3 Using the ExecuteDirective command . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 9.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 9.11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Chapter 10 Alexa In-skill Purchasing (ISP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 10.2 Create your ISP skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

●8

Contents

10.3 Accessing your ISP code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 10.4 Retrieve in-skill products, get their information and purchase. . . . . . . . . . . . . . 164 10.5 Produce detail and purchase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 10.6 Purchase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 10.6.1 Failed Purchase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 10.6.2 Refunds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 10.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 Chapter 11 Progressive Response - Accessing the Internet . . . . . . . . . . . . . . . . . . . 179 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 11.2 Steps to Send a Progressive Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 11.3 Progressive response example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 11.3.1 Code response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 11.4 asyncio, async and await - awaiting a web response . . . . . . . . . . . . . . . . . . . . 183 11.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 Chapter 12 Creating a Raspberry Pi IoT Thing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 12.2. Create a Raspberry Pi IoT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 12.2.1 a) Create our ‘Thing’ and its certificates . . . . . . . . . . . . . . . . . . . . . . . . 184 12.2.1 b) Thing’s endpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 12.2.1 c) Transfer the certificates to your Pi . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 12.2.2 Create and run the Python code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 12.2.3 Send messages to your Pi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 12.2.4 Create an Alexa-Hosted Skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 12.2.5 Test the skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 12.3 Add intents to the Alexa skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 12.4 Control the robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 12.5 Add intent handlers to the skill code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 12.6 Modify your code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 12.6.1 Modify your Pi code - LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 12.6.2 Modify your Pi code - explorerhat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 12.7 Test your robot or LED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 12.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Chapter 13 Smart Home Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201

●9

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

13.2 Alexa Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 13.3 Login with Amazon (LWA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 13.3.1 Create a security profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 13.4 Create your Smart Home Skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 13.5 Create a Lambda function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 13.6 Lambda skill code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 13.7 Test your Lambda function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 13.8 Link the function to the skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 13.9 Configure account linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 13.10 Enable and Link the skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 13.11 Clean up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 13.12 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 13.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 13.14 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Chapter 14 Controlling a smart home raspberry Pi with SQS . . . . . . . . . . . . . . . . . . 214 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 14.2 Create an SQS Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 14.3 Raspberry Pi SQS code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 14.4 Create a Smart Home skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 14.5 Create the function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 14.6 Create a security profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 14.7 Configure the smart home skill . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 14.8 Add the function code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 14.9 Test the function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 14.10 Discover your device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 14.11 Test from an Alexa device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 14.12 Clean up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 14.13 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 14.14 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Chapter 15 IoT, Pi and Node-RED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 15.2 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 15.3 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 15.4 Running node-RED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 15.5 Node-RED user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

● 10

Contents

15.6 First flow design - Hello world . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 15.7 Hardware I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 15.7.1 Add an input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 15.8 Using the Sense Hat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 15.9 Node-RED dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 15.10 Sense Hat output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 15.11 IoT - Receiving MQTT messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 15.12 Create a new IoT thing for MQTT communication. . . . . . . . . . . . . . . . . . . . . . 242 15.12.1 Subscribe to a topic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 15.13 Node-RED IoT Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 15.14 Receiving MQTT messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 15.15 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 Chapter 16 Proactive Events – Sending Raspberry Pi Alexa Notifications . . . . . . . . . 250 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 16.2 The Lambda function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 16.3 Send a notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 16.4 Code to get the access token . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 16.5 Send the notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 16.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 16.7 References: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Chapter 17 Raspberry Pi as a Stand-alone Alexa Device . . . . . . . . . . . . . . . . . . . . . 262 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 17.2 Raspberry Pi setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 17.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 17.3.1 Register your AVS device with Amazon . . . . . . . . . . . . . . . . . . . . . . . . . 263 17.3.2 Download and install the AVS SDK . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 17.3.3 Run and authorize the sample app . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 17.4 Use the sample app . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 17.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 17.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 Chapter 18 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 18.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

● 11

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

About the Author Dr. John Allwork was born in 1950 in Kent, England and became interested in electronics and engineering at school. He went to Sheffield University on their BEng Electrical and Electronic Engineering course. There he developed an interest in computers and continued his education on an MSc course in Digital Electronics and Communication at UMIST. After two years working for ICL as a design, commissioning and test Engineer he returned to UMIST where he graduated with a Ph.D. in ‘Design and Development of Microprocessor Systems’. He worked for several years in technical support and as a manager in electronics distribution, working closely with Intel Application Engineers and followed this with design work using the Inmos Transputer systems. Having taught at Manchester Metropolitan University he retired in 2011 but has kept up his interest in electronics and programming as well as his other occupation of travelling, walking, geocaching and spending time on his allotment.

● 12

Introduction This book is aimed at anyone who wants to learn about programming for Alexa devices and extending that to Smart Home devices and controlling hardware, in particular the Raspberry Pi. It covers Alexa programming concepts from the basic concepts of Alexa Voice service, the interaction model and the skill code which runs on AWS (Amazon Web Services) Lambda. It takes the reader through all stages of creating skills to certification and publishing, including writing skills that involve in-skill purchasing. It discusses different ways of creating skills, then moves on to creating visual skills using APL (Alexa Presentation Language) for screen-based Alexa devices. The book then moves on to cover different ways of controlling hardware including the Internet of Things and Smart Home devices. There are interfaces with the Raspberry Pi using MQTT and SQS communication, displaying on the Pi using Node-RED and Python code. Although mostly based on Python, Node.js examples or links are also provided. The full code is provided in a separate document. Please note that Alexa skill development, the developer console and APL versions have changed since writing this book, so please bear with the author if there are slight differences. I do not pretend to know all there is about Alexa and Raspberry Pi programming – they seem to advance faster than I can follow! I have a background in hardware and software design. I am sure that there are areas where some programmers may be offended by my code and that there may be better ways to write it, but I have written and tried all the examples and know they work. I hope the examples will spur you on to find solutions to your own problems. Should you need more information then please try the online help and the Raspberry Pi or Alexa forums: alexa.design/slack is particularly good. There are plenty of programmers out there willing to help solve your problems, often extremely quickly; certainly faster than I would get back to you! I wish to thank my friends for encouraging me, especially Dr. Hugh Frost, Andy Marsh and Dr. John Nichols; the Alexa staff: in particular Jeff Nunn, Jeff Blankenburg and Ryan J Lowe; helpers on the alexa.design/slack group, including Andy Whitworth; subscribers of my YouTube and GitHub channels who have made encouraging comments; and the many anonymous people on the internet, forums, blogs and websites who have answered many questions, not just my own – keep up the good work. Not least of all I would like to thank my wife Penny, for supporting me throughout. And of course, you for buying the book!

● 13

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 1 • Chapter 1 Alexa History and Devices Alexa is the virtual voice assistant and Echo is the device. The standard Amazon Echo device general release was in 2015. In 2019, newer versions were released, with more rounded designs and better audio. Amazon Echo Dot was released in 2016 with a smaller design than the standard Echo. Various releases and designs, including a kid’s version, have continued to the 5th generation with a clock and improved LED display in 2022. In 2017, Amazon released a combination of the Dot and Show, called the Echo Spot. In the same year, the Echo Show was released and featured a slanted, 7-inch touchscreen, camera and speaker. This later changed to a 10-inch screen (Echo Show 10), and more recently, added a 360-rotating display. The Echo Show 5 came in 2019, (2nd gen in 2021), as well as Echo Show 8 and an Echo Show 15 in 2021 designed for wall mounting. There are other devices too, including the Button, Flex, Input, Look and recently the Astro robot. Here are some of my devices (not including smart devices). From the top: Echo Show 8, Fire TV stick, Echo Auto, my original Echo dot, and the Echo Spot.

● 14

Chapter 1 ● Alexa History and Devices

Even though many devices have a screen, you should always design for ‘voice first’.

1.1 Alexa voice service and AWS Lambda Alexa Voice Service is Amazon’s cloud service that processes the audio, determines the appropriate action (AVS) and returns a response to the device. For the skill to produce the appropriate response, designers need to implement two major parts: the interaction model and the skill code which runs on AWS (Amazon Web Services) Lambda. The interaction model is what your users say and how they communicate with your skill. AWS Lambda is a serverless, event-driven computing service that lets you run your code. Lambda can be triggered by many AWS services and you only pay for what you use. When a user interacts with an Echo device, AVS sends a request to the skill which is running on AWS Lambda. The skill replies with a response that is turned into a speech and/ or visual response back to the user.

1.2 Pricing Although there are charges for AWS Lambda, the AWS Lambda free tier includes one million free requests per month and 400,000 GB-seconds of compute time per month, as well as 500 Mb storage. As you can see, this is more than enough for a beginner. For more information, see https://aws.amazon.com/lambda/pricing/ For developers whose skills use more than this, Amazon provides Promotional Credits which reward those who build cloud-hosted applications, software, or tools for sustainability-related work. For FAQ see reference 1.

1.3 Alexa skills There are a few different types of Alexa skills2. You may already have realized that a skill communicating with an Alexa device is different from one switching on your lights or telling you there’s someone at your front door.

● 15

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

At the moment, there are 15 different types of Alexa skills. The more common ones are: Skill Function

Skill Type

Description

Automotive Custom Flash briefing Games Music Smart Home Video

Pre-built and Custom Custom Pre-built Custom Pre-built Pre-built Pre-built

Automotive applications Voice and visual (APL) applications Provide news and short content information Voice and visual driven game skills Skills to control audio content Skills to control smart home devices Control video devices and content

We’ll be concentrating on Custom skills. Blueprint pre-built skills are also available and easy to develop but have reduced options for user experience. We’ll also study smart home skills, of course.

1.4 Supported programming languages AWS Lambda natively supports Java, Go, PowerShell, Node.js, C#, Python, and Ruby code. This book will mainly use Python, but it also provides code snippets and links for Node.js.

1.5 Terminology – Invocation, Utterances, Intents and Slots As with learning anything new, there is new terminology to be understood. You will soon meet (or may already have met) Invocation, Utterances, Intents and Slots. 1.5.1 Alexa Wake word

This is the word used to start your Alexa device listening to your command. Currently, there are five wake words: ‘Alexa’ (the default), ‘Amazon’, ‘Echo’, ‘Computer’ and ‘Ziggy’. You can change these for your devices, but not invent new ones. 1.5.2 Invocation

The ‘invocation’ is the phrase used to trigger your skill, e.g.: ‘Alexa, open Johns’ weather skill’ or ‘Alexa, launch my cooking skill’. 1.5.3 Utterances

Utterances are the phrases that your user says to make a request. There can be many ways to achieve the same result, e.g.: What’s the time? What’s the time now? What time is it? – you will have to think of as many possible ways that your user can interact with your skill. Nevertheless, Alexa will build your model and try to find similar utterances. All the possible ways to do this can be difficult to describe (considering, for instance, how many different ways and types of pizza someone might order), so Amazon has recently announced Alexa Conversations to help with this. For more information, see reference 3.

● 16

Chapter 1 ● Alexa History and Devices

1.5.4 Intents and requests

The utterances are linked to one intent in the code. For instance, all the ‘time’ utterances would be linked to the same intent, e.g., GetTimeIntent, and this would trigger a GetTimeIntent function in our skill code. There are two types of Intents: • Built-in Intents - Standard built-in intents: These are provided by default by Amazon that every skill must have, e.g.: AMAZON.StopIntent, AMAZON.CancelIntent, AMAZON.FallbackIntent, etc. and include: AMAZON.YesIntent, and AMAZON.NoIntent, intents for screen control (e.g., scroll up/ down/ left / right) and media intents (pause, repeat, resume), and also an AMAZON.SendToPhoneIntent. You can see these when you add an intent, and select “Use an existing intent from Alexa’s built-in library” - The Alexa Skills Kit also provides a library of specific built-in intents and includes intents such as Actor intents, Books, Calendar, LocalBusiness, Music, TV, Series, WeatherForecast, etc. These intend to add functionality to your skill without you having to provide any sample utterances. For example, the WeatherForecast includes a search action (What is), an object (WeatherForecast), location (London) and date (tomorrow) . We won’t cover them in this book, see: https://developer.amazon.com/en-US/docs/alexa/custom-skills/built-in-intent-library.html • Custom Intents These are created as required for the skill (e.g., GetTimeIntent) If you use an Amazon template, the code for the built-in intents is provided for you. There are three types of requests that the skill can send: • A Launch request that runs when our skill is invoked (as a result of the user saying, ‘Alexa open …’ or ‘Alexa, launch ...’). • An Intent request which contains the intent name and variables passed as slot values. • A SessionEnded request, which occurs when the user exits the skill, or there is an unmatched user’s response (although you may be able to trap this out with AMAZON. FallbackIntent). This information is all packaged and sent as a request (and returned as a response) as a JSON file. We’ll look at the JSON code later. 1.5.5 Slots

A slot is a variable that contains information that is passed to an intent. The user might say ‘What’s the time in London’. Here ‘London’ (or Paris or Rome) is passed as a slot variable to the intent code.

● 17

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Amazon provides built-in slot types, such as numbers, dates and times, as well as builtin list types such as actors, colors, first names, etc. In the previous example, we could use AMAZON.GB_CITY which provides recognition of over 15,000 UK and world-wide cities used by UK speakers.

However, some of these slots are being deprecated (including AMAZON.GB_CITY in favour of AMAZON.CITY), so check. The full list is covered at ‘List Slot Types: https://developer.amazon.com/en-US/docs/alexa/custom-skills/slot-type-reference.html#list-slot-types Alexa slot types fall into the following general categories: • Numbers, Dates, and Times • Phrases • Lists of Items Developers can create custom slots for variables that are specific to their skill. When we define our utterances, slots are shown in curly braces: {city}, e.g.:

Example: Intent GetTimeIntent Utterance What is the time in Slot {city} Utterances can have many slots and slot types. The GetTimeIntent will trigger a function in your skill (which you might sensibly call GetTimeIntentFunction). Slots are used to pass data from your VUI (voice user interface) to your program. As an example, we might have an Alexa skill that asks for your name. The VUI might go like this: User: Alexa: User: Alexa:

● 18

“Alexa, Open What’s my name” (Invoke the skill – the launch request executes) “Welcome, please tell me your name” “My name is John” (“John” is passed in a slot to myName intent) “Hello John” (Your intent picks John from the slot passed to it and responds)

Chapter 1 ● Alexa History and Devices

At this point, the slot data is lost unless you save it. You can save data in a temporary folder but more often data is stored in session attributes, you will find out later. 1.5.6 Interaction model

The combination of the utterances and their related intents and slots make up the interaction model. This needs to be built by the developer console, and in doing so Alexa may recognize further utterances similar to those you have defined. 1.5.7 Endpoints

The endpoint is where your code is hosted. You can choose an Amazon-hosted, AWS Lambda ARN (Amazon Resource Name) site or host it yourself on an HTTPS site that you manage. If you choose an AWS site, it will give you an ID beginning arn:aws:lambda and look like: arn:aws:lambda::function:. You skill also has an ID looking something like this: amzn1.ask.skill.a0093469-4a50-4428-82e6-abcde990fgh3. 1.5.8 Regions

If using an AWS-hosted site, you should host your code in a region near to your user. We’ll see that there are lots of regions, but for some skills currently only North Virginia is available.

1.6 Skill Sessions The period that your skill runs for is called a session. A skill session begins when a user invokes your skill and Alexa sends your skill a request. Your skill receives the request and returns a response for Alexa to speak to the user. If the shouldEndSession parameter is ‘true’ the skill terminates, otherwise, the session remains open and expects the user to respond. If no user input occurs, a reprompt is sent if included in the code. If the user still doesn’t respond (after about 8 seconds), the session ends 4. Skill connections and progressive responses may override these rules. For example, if a skill has to get further information from another source, e.g., when your taxi will be available, or pizza delivered.

1.7 Session attributes Session attributes are used to hold data during a session, for example, your user’s name. When the session finally ends, the data is lost. To prevent this from happening, data can be stored permanently in persistent attributes. This can be held in a DynamoDb database which is provided as one of the AWS services and easily accessed using an Alexa-Hosted Skill. With an Alexa-Hosted Skill, you can build your model, edit your code and publish your skill all from within the developer console.

● 19

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

1.8 Request and response JSON We saw in the figure above how the request is sent from the user (Alexa Voice Service) to your code and how the response is returned from your code. This data is passed in a JSON format. JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write and easy for machines to parse and generate” 5 The basic JSON request contains information on the active session, the context, the system information on the application (ID), user, deviceID, and the request itself: {

“version”: “1.0”, “session”: { ( ..session parameters) }

“context”: { ( ..information on the Alexa device) }

“System”: { …

“request”: { “type”: “IntentRequest”, “requestId”: “amzn1.echo-api.request.745d…9a”, “locale”: “en-GB”, “timestamp”: “2022-04-14T09:27:01Z”, “intent”: { “name”: “HelloWorldIntent”, “confirmationStatus”: “NONE” } } The reply JSON contains is the response speech and reprompt, as well as the state of the EndSession and session attributes. {

“body”: { “version”: “1.0”, “response”: { “outputSpeech”: { “type”: “SSML”, “ssml”: “Welcome, you can say Hello or Help.” },

“reprompt”: { “outputSpeech”: { “type”: “SSML”, “ssml”: “Welcome, you can say Hello or Help.” } },

● 20

Chapter 1 ● Alexa History and Devices

“shouldEndSession”: false, “type”: “_DEFAULT_RESPONSE” },

“sessionAttributes”: {}, “userAgent”: “ask-python/1.11.0 Python/3.7.12” } }

The response can get more complicated if there is a dialog session occurring (i.e., if the program hasn’t enough information to complete the intent request and has to ask for more). We’ll look at the information passed in the JSON request and response and how to extract it in a later chapter.

1.9 Blueprint skills Alexa provides blueprint skills, where you can ‘fill in the blanks’ to make your skill. These are worth looking at for some fun and information presenting skills.

The current categories are: At home, Kids recommended, Learning and knowledge, Fun and Games, Storyteller, Greetings and Occasions, Communities and Organizations, and Business 6. We won’t cover them here.

● 21

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

1.10 Summary We’ve seen how the Alexa devices have developed from the original voice-only device to screen-based and robot devices, how the Alexa Voice service works and looked at terminology – Invocation, Utterances, Intents and Slots. Finally, we looked at a skill session and how data is passed and saved during a session and between sessions. In the next chapter, we’ll see how to set up an Alexa account before moving on to our first Alexa skill.

1.11 References: 1. https://aws.amazon.com/lambda/faqs/ 2. https://developer.amazon.com/en-US/docs/alexa/ask-overviews/list-of-skills.html 3. https://www.youtube.com/watch?v=1nYfRvg976E 4. https://developer.amazon.com/en-US/docs/alexa/custom-skills/ manage-skill-session-and-session-attributes.html 5. https://www.json.org/json-en.html 6. https://blueprints.amazon.com

● 22

Chapter 2 ● Creating your Amazon Account

Chapter 2 • Creating your Amazon Account 2.1 Introduction There are three ways to create an Alexa Skill. 1. Using Alexa-Hosted Skill – this is most probably the easiest way. You don’t need an AWS account – you can just get started. 2. Using AWS Lambda and the Developer Console. 3. Using an IDE (such as Visual Studio code) and ASK CLI (Alexa Skills Kit and Command Line Interpreter). In any case, you need an Amazon developer account. We’ll use an Alexa-Hosted skill where we can. It provides most of the requirements we need including extra storage (S3) and a database (DynamoDB). We’ll study these in a later chapter.

2.2 Create your Amazon account You don’t need an AWS account for Alexa-Hosted skills, but you will need one for smart home skills. Go to the Amazon Developer website at: https://developer.amazon.com/

Click ‘Developer Console’ and create your Amazon Developer account.

● 23

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Complete the form and create your account. You may have to authenticate it with an OTP. Answer the rest of the questions in the Amazon Developer Registration.

Agree to the terms and click submit. You can now provide payment information, user roles, etc., or leave until later. If you want to earn money from In Skill Purchases, you will have to complete this. The next screen is the Amazon Developer Dashboard – My Account page: https://developer.amazon.com/settings/console/myaccount However, you can always complete this later.

● 24

Chapter 2 ● Creating your Amazon Account

Eventually, you will get to the developer console.

We will be using the Alexa Skills Kit but note the other items here. • Alexa Voice Service (AVS) is used to integrate Alexa features and functions into a product (see chapter 17). • Amazon Appstore is an app store for Android and provides apps for Fire Tablets. • Amazon Dash console enables you to manage Amazon dash buttons. • Amazon GameOn is for mobile gamers to share their experiences (high scores, tips, tricks, etc.) and to participate in challenges. • Login with Amazon (LWA) is used to authenticate users using their Amazon accounts. You need this if you create any smart home devices. We’ll use LWA and AVS later.

● 25

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

2.3 Your skills Clicking the Alexa Skill Kit takes you to the development console. If this is the first time you’ve been here, you won’t have any skills listed.

2.4 Hosting Click on the Hosting tab to see what you’re allowed to access.

You’ll find the storage and data transfer more than enough for now.

● 26

Chapter 2 ● Creating your Amazon Account

2.5 Summary In this chapter, we saw the different ways of creating and Alexa skill, created our developer account, and saw how to start an Alexa-Hosted skill. In the next chapter we’ll develop our first skill.

● 27

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 3 • Creating an Alexa Skill 3.1 Introduction In the last chapter, we saw how to start an Alexa-Hosted Skill. This provides access to three AWS Lambda endpoints, an S3 bucket for storage and access to a DynamoDB database. We’ll discuss these in later chapters. Let’s create our first skill now.

3.2 Your first skill As we saw, clicking the Alexa Skill Kit takes you to the development console. If this is the first time you’ve been here, you won’t have any skills listed. It’s at: https://developer.amazon.com/alexa/console/ask

Click the Create Skill button. In the next screen, you’re asked for a skill name, the locale (language and location) for your skill, and if you want more than one locale, to sync them. This will link all the utterances. The user interface has changed slightly since originally writing this chapter, but the process is the same.

● 28

Chapter 3 ● Creating an Alexa Skill

There are rules as to what names you can give your skill. It must be at least two characters and less than fifty, and brand names are only allowed if you show proof of rights. When you publish your skill, these will be checked. It’s sensible to give your skill a two- or three-word name. Try ‘ first skill’. I’ve used John’s first demo skill. 3.2.1 The interaction model

The interaction model is how your skill communicates with the user. Some templates are provided for you (Flash briefing, smart home, video, etc.). Leave it as Custom. 3.2.2 Choose a method

The choices are Alexa-hosted – Node.js or Python, or Provision your own. In this book, we’ll mostly be using Python, but there will be Node.js examples or links. Choose Node.js or Python.

● 29

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

In the top right-hand corner, you can select the hosting region. It’s sensible to select one nearest to you.

Click ‘Create skill’. The next screen provides some templates. There are more Node.js templates than Python. Some provide examples of intents, data persistence and personalization. Choose ‘Start from scratch’ and ‘Continue with Template’.

● 30

Chapter 3 ● Creating an Alexa Skill

You can also import a skill, say from GitHub. Your Alexa skill will be created. It takes about a minute and then you’re taken to the developer console build tab. The first thing you notice is the video showing you how to build an Alexa Skill. You can view this now if you want to.

At the top left is your skill name. To the left of this, you can return to your skill list. To the right, you can move between the build tab (currently displayed), the Code and Test tabs which we’ll look at soon and the Distribution, Certification and Analytics tabs which are used when publishing the skill.

● 31

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

This template skill can now be tested and executed – you don’t have to do anything. But before we run our skill, let’s take a look at the invocation and intents. Remember, the invocation is what your user says to invoke (start) your skill, for instance: “Alexa, open John’s first demo skill”; the intents are what your users say when the skill is running during a conversation e.g., “My favorite color is red”. On the right-hand side, there is a skill builder checklist that shows that four stages have been completed.

We won’t monetize our skill just yet – we will cover that in a later chapter. These actions are also available in the left-hand column, which also shows the currently selected language. The items labelled ‘Custom’ can be expanded for more detail. We’ll look at the Invocation Name and Build Model in this chapter, and the Intents and Endpoint in the next. 3.2.3 The Invocation Name

Let’s start by checking the Invocation Name. Either click the right-hand column “1. Invocation Name” or expand the left-hand column and select “Skill Invocation Name”.

● 32

Chapter 3 ● Creating an Alexa Skill

Sometimes (as here) the Invocation Name says, “change me”. If necessary, change this to your skill name (it can be anything, but it seems sensible). I’ve used john’s first demo skill. The user will say “Alexa, open john’s first demo skill” to invoke the skill. Note that the Invocation Name can only be lowercase. As you’ve changed the invocation, you will have to rebuild your model. Save the model and rebuild it. Use the option we saw above (3. Build Model), or these buttons.

Whilst it’s doing that (It won’t take long), take a look at the Skill Launch Phrases and Intent Launch Phrases. Skill and Intent launch phrases enable your users to interact with your skill without stating the launch name. Neither is available until the skill is published. 3.2.4 The Intents

Click the Interaction model then Intents to see the current Intents. They are Cancel, Help, Stop, NavigateHome (they respond to the user saying ‘cancel’, ‘help’, ‘stop’ and ‘go home’), HelloWorld (triggered when the user says ‘Hello’) and Fallback. The Fallback intent is triggered by a user utterance that doesn’t match any skill intent and allows you to write code to provide the user with more details about your skill and how to interact with it.

● 33

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

If you click on an intent, you get a list of the utterances that will trigger the intent.

3.2.5 The code

Before we can test the skill, we have to deploy the code. Let’s look at the code first. Click the Code tab. The Python code example here begins by importing the libraries we need. For example, import logging imports the libraries that enable us to output debugging information to Cloudwatch logs, see section 3.8. To do this we use the code logger.info(info_to_display)

● 34

Chapter 3 ● Creating an Alexa Skill

The handlers follow. These deal with the launch request and intents such as Hello or Help. class LaunchRequestHandler(AbstractRequestHandler): class HelpIntentHandler(AbstractRequestHandler): We’ll look at these in more detail later. The rest of the intents follow. These include the Stop or Cancel intent (what to do when the user says “stop” or “cancel”); the Fallback intent (what to do if the user’s utterance doesn’t match any voice model intent); the SessionEnded intent (what to do when the session ends), on so on. The intent handlers are registered using the sb.add_request_handler code, where sb is the SkillBuilder() code. This is imported by the command:

● 35

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

‘from ask_sdk_core.skill_builder import SkillBuilder’, at the top of your code. The entry point of the code is at the end: lambda_hander = sb.lambda_handler()

This example uses Classes, you can also use Decorators. For a more detailed description of both, and the code, see: https://developer.amazon.com/en-US/docs/alexa/alexa-skills-kit-sdk-for-python/develop-your-first-skill html#using-handler-classes It’s worth noting the link mentioned in this code to more skill examples at: https://alexa.design/cookbook For the Node.js version, refer to section 3.9 at the end of this chapter. Before looking at our code in more detail, let’s make the welcome message more personal, then test our skill.

● 36

Chapter 3 ● Creating an Alexa Skill

Click the code tab. Python: • Find the line that says: speak_output = “Welcome, you can say Hello or Help. Which would you like to try?” This is about line 30:

• Change this to, say: speak_output = “Hello John, please say Hello or Help.” You can use your name instead of mine! • If you are unfamiliar with Python, you MUST ensure your code is tabbed correctly. Note that this is in the LaunchRequestHandler code, which runs when the user starts your skill. • Save the code. Click the deploy button in the top right-hand corner. Node.js: • For Node.js, find the line that says: const speakOutput = ‘Welcome, you can say Hello or Help. Which would you like to try?’; This is about line 13. Change it to, say: const speakOutput = ‘Hello John, please say Hello or Help.’; • Make sure that the quotes match and the line ends with a semi-colon.

• Save the code. Remember that, after making any changes to your code and before we can test it, we must save and deploy the code.

● 37

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Click deploy

in the top right-hand corner.

You’ll get a message when it’s been deployed, e.g., Last Deployed: Apr 13, 2022, 8:26 PM.

3.3 Testing your skill We can now test the skill. Click the Test tab. This opens the Alexa Simulator. You have to enable skill testing. Change that from ‘Off’ to ‘Development’.

You can now send any command that you might usually use with your Alexa device (try: Alexa, what’s the weather). We can now launch the skill. Type (or click the microphone button and say) “Open ” e.g., “Open johns first demo skill”. You don’t have to use the wake word (i.e., No need to say or type Alexa, open johns first demo skill).

● 38

Chapter 3 ● Creating an Alexa Skill

Congratulations! You’ve just got your first skill working. Troubleshooting If it doesn’t work, check that you have the correct invocation name, that you saved and built the model and that you saved and deployed your code. Also, if the box under Alexa Simulator e.g., English (GB) is greyed out, close the developer console and log in again. In the simulator, you can see we see the invocation: “open johns first demo skill” The response: “Welcome, you can say …” (Or whatever you changed it to) and after saying ‘hello’, our HelloWorld intent code response: “Hello”. The skill terminates after this conversation – we’ll see why later. If you type in Hello again, your program doesn’t respond - Alexa responds instead. You can save up to 30 utterances for later use by clicking the plus sign:

Note that, depending on how the user invokes your skill, it is possible that the launch request isn’t executed. For more about invoking custom skills, see: https://developer.amazon.com/en-US/docs/alexa/custom-skills/understanding-how-users-invokecustom-skills.html#about-invoking-custom-skills (or search for “invoke alexa skill”).

3.4 Skill I/O At the top of the simulator, there are options to show the Skill I/O, the Device Display, Device Log and Personalization.

The Device Display is used to show what the device will display if it has a screen. If you scroll down, you’ll see that it currently displays the skill name and how to exit. We’ll see this in action when we discuss Alexa Presentation Language in a later chapter.

● 39

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

The device log provides details of the actions of the skill, including events and directives:

Personalization. Alexa devices are capable of identifying a recognized skill user if it is set up and agreed by the user. You can use this in your code and test using personalization. For more details, see: https://developer.amazon.com/en-US/docs/alexa/custom-skills/add-personalization-to-your-skill.html Sklll I/O. Scroll down to see the JSON code.

Remember, communication uses JSON code requests and responses. This shows the JSON input (request –sent to your code) and JSON output (response – received by your code). JSON is easy to understand. If you look at the JSON output, you can see where the Welcome, you can say … occurs.

3.4.1 Skill request

The JSON input request data is sent by your Alexa device as a result of hearing the launch phrase. This contains lots of useful information such as the locale, device capabilities (does it have a screen?), the userId and so on. We’ll use some of this information in later code examples. We won’t list all the JSON input here – you can do that yourself, but I will look at a few relevant items, and we’ll see more later on.

● 40

Chapter 3 ● Creating an Alexa Skill

It starts with the version, then the session items that identify the sessionId, the application and the user’s userId. {

“version”: “1.0”, “session”: { “new”: true, “sessionId”: “amzn1.echo-api.session.27d.. e7cb”, “application”: { “applicationId”: “amzn1.ask.skill.4649.. 981” },

“attributes”: {}, “user”: { “userId”: “amzn1.ask.account.AFCVJZSYW…VY2BH6HI” }

It follows with some information on the device under “context” > “Viewports”. In this case, it refers to the Alexa simulator we’re using and tells us we have a rectangular screen capable of showing APL, it can’t rotate and has a pixel height and width of 1280 × 800. See if you can identify these in your display “context”: { “Viewports”: [ {

“type”: “APL”, “id”: “main”, “shape”: “RECTANGLE”,

It continues with information about the System. We’ll look at System>device>SupportedInterfaces later to see if the device can support APL. “System”: { “application”: { “applicationId”: “amzn1.ask.skill.4649….981” },

“user”: { “userId”: “amzn1.ask.account.AFCVJZSY…H6HI” },

“device”: { “deviceId”: “amzn1.ask.device.AGSER…DU”, “supportedInterfaces”: {}

● 41

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Finally, information about the request itself is displayed. In this case, we see that it was a launch request, and the locale was en-GB (I’m in the UK) “request”: { “type”: “LaunchRequest”, “requestId”: “amzn1.echo-api.request.a7ed3cc2-4bba-9e82-3c79e5e4eb59”, “locale”: “en-GB”, “timestamp”: “2022-04-13T19:57:52Z”, “shouldLinkResultBeReturned”: false } }

We’ll see how to extract some of this data in our code later. If we compare the request with that after entering ‘hello’, we get this: “request”: { “type”: “IntentRequest”, “requestId”: “amzn1.echo-api.request.745de8e2-7… 6fc9a”, “locale”: “en-GB”, “timestamp”: “2022-04-14T09:27:01Z”, “intent”: { “name”: “HelloWorldIntent”, “confirmationStatus”: “NONE” }

We can see that we have an IntentRequest and its name is “HelloWorldIntent”. We pick this up in our code to reply. 3.4.2 Skill response

The skill response is the one generated by your code (we haven’t looked at that yet, but the bit that creates the “Welcome, you can say Hello or Help.”). This is the response to the open skill launch request. It’s short, so we’ll look at the whole listing: {

“body”: { “version”: “1.0”, “response”: { “outputSpeech”: { “type”: “SSML”, “ssml”: “Welcome, you can say Hello or Help.” },

“reprompt”: { “outputSpeech”: { ● 42

Chapter 3 ● Creating an Alexa Skill

“type”: “SSML”, “ssml”: “Welcome, you can say Hello or Help.” } },

“shouldEndSession”: false, “type”: “_DEFAULT_RESPONSE” },

“sessionAttributes”: {}, “userAgent”: “ask-python/1.11.0 Python/3.7.12” } }

With Node.js the userAgent is: “ask-node/2.12.0 Node/v12.22.11 sample/hello-world/v1.2” The main parts here are the response > outputSpeech. This is SSML (Speech Synthesis Markup Language) which tells the device what to say initially, and the reprompt > outputSpeech which tells the device what to say if the user doesn’t respond. In this case, because “shouldEndSession” is false, the user then has 8 seconds to respond, before the device gives up. 3.4.3 Speech Synthesis Markup Language (SSML)

SSML provides additional control to your speech. You use SSML in your output by using “type”: “SSML” as seen above, instead of “type”: “PlainText” “outputSpeech”: { “type”: “SSML”, “ssml”: “This speech uses SSML” }

There is a long list of SSML tags. They can also be used in APLA, see section 8.2. They are listed at: https://developer.amazon.com/en-US/docs/alexa/custom-skills/speech-synthesis-markup-languagessml-reference.html The more commonly used ones are: • audio Plays an audio file.

• break Provides a pause. Set time with strength or time attribute.

• emphasis Gives emphasis (strong, moderate or reduced). I really like you • voice Used to speak the text using Amazon Polly voice, such as Hans, Marlene, Vicki in German locale, or Amy, Brian, Emma in en_GB. ● 43

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi



Sorry, but



I am a robot

• speak Surrounds the text to be spoken with this tag. This is what Alexa sounds like without any SSML There are strict requirements and limitations for audio when used in SSML. The main ones being: a valid MP3 file hosted on an https source, no longer than (a combined) total of 240 seconds, a bit rate of 48 kbps and sample rate of 22050 Hz, 24000 Hz, or 16000 Hz. Alexa provides a library of sound effects that you can use. It has a wide selection of short audio clips, from Air/Fire Extinguisher through Animal to Wood/Squeaks, see https://developer.amazon.com/en-US/docs/alexa/custom-skills/ask-soundlibrary.html

3.5 Code editing We edited the welcome message previously. To get familiar with using the editor, let’s change the HelloWorldIntent handler and add some debug code. 3.5.1 Edit the HelloWorldIntentHandler code

The reason the skill terminates after entering ‘Hello’ is because there was no reprompt outputSpeech message (see above). That is provided with an ‘ask’ in the return statement. Python: • Find the line of code that says: # .ask (“add a reprompt if you want to keep the session open for the user to respond”) • The # is a comment. Change the line to the following: .ask(“Say Hello again”) Node.js: • Find the line (about line 32) beginning: //.reprompt . • Remove the //. 3.5.2 Add some debug code

We’ll add a couple of lines so that we can see how to log messages and debug our code. Debug messages appear in the Cloudwatch logs, see section 3.8. Using Python, before each of the speak_output lines of code add logger.info(“In launch request”) and logger.info(“In hello world intent handler”)

● 44

Chapter 3 ● Creating an Alexa Skill

In Python, make sure they are tabbed correctly (at the same level as the speak_output). The Node.js changes are in sections 3.9 and 3.10 and use console.log() command. Your code should look like this ():

You can see how your code is steered to the correct procedure from the JSON message. The launch request had a “type”: “LaunchRequest”, which is picked up by the code: return ask_utils.is_request_type(“LaunchRequest”)(handler_input)

● 45

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

And the HelloWorldIntent is detected by: return ask_utils.is_intent_name(“HelloWorldIntent”)(handler_input) The Node.js code is steered with the following commands: return Alexa.getRequestType(handlerInput.requestEnvelope) === ‘LaunchRequest’;

and return Alexa.getRequestType(handlerInput.requestEnvelope) === ‘IntentRequest’ && Alexa.getIntentName(handlerInput.requestEnvelope) === ‘HelloWorldIntent’;

We’ll look at Intents and handler_input in more detail in the next chapter. Save and deploy your code.

3.6 Test your code This time you can say ‘Hello’ as many times as you want. Note that the Alexa simulator doesn’t time out like a real device. Try it on a device. If it doesn’t work make sure the login (and locale) for your developer is the same as that for your Alexa device.

3.7 Utility code Whilst your code is deploying, look at the other two pieces of Lambda Python code: requirements.txt and utils.py (the Node.js functions are described at the end):

requirements.txt This imports the libraries we need:

boto3==1.9.216



ask-sdk-core==1.11.0

We’ll see in a later chapter that boto3 is used to create, configure, and manage AWS services, such as the Amazon Simple Storage Service (Amazon S3). On a side note, you can search for “Why is it called Boto3?”. ask-sdk-core provides the ‘boiler plate’ code for developing our Alexa Skills. We’ll add to requirements.txt in later chapters.

utils.py The create_presigned_url code here provides temporary access to S3 storage (‘buckets’) to users who do not have AWS credentials or permissions. We’ll use that later too.

● 46

Chapter 3 ● Creating an Alexa Skill

3.8 Debugging When you’ve tested your code, Click the Code tab and then the CloudWatch logs tab.

You may have to use the drop-down box to select your region, depending on where you are. If you select the wrong region, you’ll get an error message:

You may also be prompted to complete your settings (language and region) for the CloudWatch logs to work.

● 47

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Eventually you should be taken to the CloudWatch logs page:

From there, you can select your log from the log stream. This shows you the timestamp and message.

You can see the ‘In launch request’ and ‘In hello world’ logs we added.

● 48

Chapter 3 ● Creating an Alexa Skill

3.9 Node.js differences The main Node.js code is index.js. It calls the ask-sdk-core libraries and has the same intent request and handlers.

The utils.js contains the getS3PreSignedUrl that provides access to S3 services, package. json provides the dependencies, and the local-debugger.js script has been deprecated.

● 49

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi



3.10 Node.js debugging The Node.js code has a console.log() command in the SessionEndedRequestHandler. console.log(`~~~~ Session ended: ${JSON.stringify(handlerInput.requestEnvelope)}`); This can be expanded, and you can see the request envelope:

This is the same as we saw previously.

● 50

Chapter 3 ● Creating an Alexa Skill

3.11 Summary That’s been a long chapter. We’ve looked at creating an Amazon account, creating our first Alexa-Hosted skill, the developer console, invocation, utterances and intents, and the skill model. We discussed and edited the code, saving and deploying it before testing our skill using the Alexa simulator. Furthermore, the JSON request and response were seen and how the intents are executed. We also saw how to debug the code by adding error logging and looking at the CloudWatch logs. In the next chapter, we’ll look at slots, multi-turn conversations, dialogs, and further testing of our model.

● 51

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 4 • Slots and Dialogs, Saving Session Data 4.1 Introduction In this chapter, we’ll discuss the conversation, slots and their types, adding intents and creating the interaction model. We’ll see how to pass slot values to our code, read that data and save it during a session. We’ll write a skill to ask for a user’s name and birth date and use that in the reply.

4.2 Slots in action We discussed how slots are used to pass information from your utterances to your skill. There are 9 built-in slot types, such as numbers, dates and times, and 46 built-in list types such as actors, colors, first names, etc. You can generate your own custom slot type too and add expected slot values. For instance, you may want a slot that contains Programming languages. You create a custom slot and add slot values: ‘c’, ‘python’, ‘ada’, ‘javascript’, etc. Let’s create a skill that uses these types. It will be a skill that asks for a user’s first name and birthday and records how many times the user has visited. You can already see that we can use some of the built-in slot types.

4.3 Slot skill Create a new Alexa-Hosted Skill. I’m calling it ‘My slot skill’. If you can’t remember how to do this, go to developer.amazon.com, sign in, create a skill, enter a skill name, choose Python (or Node.js), and continue using the scratch template. 4.3.1 Invocation Name

Before doing anything else, check your Invocation Name. It’s sometimes not changed to your skill name, and left as ‘change me’, but you might want something else anyway. Click Invocations > Skill Invocation Name, and change the name:

● 52

Chapter 4 ● Slots and Dialogs, Saving Session Data

4.4 Skill flow Let’s think about how the conversation might go. This is the skill flow. There are programs which make designing the flow easy for you, but won’t be covered in this book – if you’re interested, see https://github.com/alexa-games/skill-flow-builder and https://www.youtube.com/watch?v=srxwYJN_R5c Our conversation will go something like this: User: Alexa: User: Alexa:

Open My slot skill. Hello, please tell me your name. My name is Joe. Welcome Joe.

Later we’ll add: Alexa: Can you tell me the day and month of your birthday? User: My birthday is the 1st of April. Alexa: Ok, Joe, your birthday is the 1st of April. I’ll remember that for next time, bye for now. And we’ll see later how to remember the user’s name and birthday by using persistent attributes. That way the data will be stored between sessions and you won’t have to ask for it every time. Next time: User: Open My slot skill. Alexa: Hello Joe, welcome back. You were born on the 1st of April. You can see that we will need intents to ask for the user’s name and birthday, and slots for the name and birthday. We will call the intents getNameIntent (with a name slot) and a getBirthdayIntent (with a birthday slot). We’ll start by asking for the user’s name, read the slot for that and respond with the user’s name. Later we’ll ask for their birthday. You will have to think of the ways the user might say their name, as well as their birthday. For their name, they might say:

● 53

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

• My name is {name} • I’m called {name} • {name} • It’s {name} • etc. For the birthday they might say: • 1st of January • January 1st • January 1 is my birth date • I was born on the 1st of January • etc. We will use the userId to record the information for that user. The userId is automatically generated when a user enables the skill and is specific to that user and skill. It is not the same value across different skills. Remember, the userId information is passed in the JSON request under session > user > userId, but we don’t know their name. We saw this in section 3.4.1.

4.5 Add the intent to our skill First, click the Code tab and change the launch request so that it prompts for the user’s name. In Python, around line 30, change it to:

speak_output = “Hello, please tell me your name”

Node.js – it’s about line 13 – change it to:

const speakOutput = ‘Hello, please tell me your name’;

We’ll now add the intent to ask for the user’s name, to the voice model. Click the Build tab, then Skill Builder Checklist 2: Intents, Samples and slots,

● 54

Chapter 4 ● Slots and Dialogs, Saving Session Data

or use Interaction Model > Intents > Add intent.

And add a Custom Intent. Enter your Intent’s name into the box — I’ve used GetNameIntent.

You are now prompted to enter sample utterances. To add a slot, we enter it in curly brackets. Type ‘my name is {name}’. {name} will be recognized as a slot value that will be passed to your program. Clicking + or ‘Add’ will add this slot.

● 55

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

This has just added it as a slot name — we haven’t defined its type yet. When you click the plus sign to add the utterance, you can then select a slot type for your {name} slot. Click the drop-down box to display all the available slot types. I’m using AMAZON.GB_ FIRST_NAME, which is a list of common UK first names, but AMAZON.FirstName could be used instead.

Whilst you are on this screen notice: • Dialog delegation à We’ll look at this soon. • Multi-value option à Used if your slot can have more than one value, the example given is “add eggs, flour, milk, sugar to the shopping list”. • Intent Confirmation à Do you want Alexa to confirm this choice? Be careful —this can be very annoying for the user. Add some other utterances. Enter the slot {name} in curly brackets:

Think of a few yourself. You can also add slot types before adding utterances. Click ‘Slot types’, then ‘Add slot type’ and select your slot type.

● 56

Chapter 4 ● Slots and Dialogs, Saving Session Data

Click the ‘Save Model’ and ‘Build Model’ buttons, to save and build the model. You have to do this every time you change your model. Note: If you require a previous version of your model, you can get that by selecting Version and choosing the required one from the options:

4.6 Evaluate your model You can determine which intents will be used by Alexa for your utterances. Click ‘Evaluate Model’.

This opens the Utterance Profiler where you can enter utterances and see which intent will be executed:

● 57

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

In this case, we have a simple model, but you may get utterance conflicts when you build your skill. For example, if you ask the user for their name and birthday, you may have two slots for which the response could be the same, e.g., they might just say “{name}” and “{birthday}”, or “It’s {name}” and “It’s {birthday}.

Clicking on Utterance Conflicts displays the conflicts and what the behavior will be.

The conflict between the similar utterances has been detected and the behavior noted: “Alexa will attempt to disambiguate utterance between custom intents GetBirthdayIntent, GetNameIntent”.

● 58

Chapter 4 ● Slots and Dialogs, Saving Session Data

You may be happy with this, but ‘Evaluate Model’ should be used to find out conflicts and for you to address any issues before coding. If the Utterance Profiler can’t find a match for your input, it will drop to the AMAZON. FallbackIntent where you can write code to deal with the problem. You can also use Natural Language Understanding (NLU) or Automatic Speech Recognition (ASR) to evaluate your model. See section 4.17. 4.6.1 The JSON editor

This is a good time to look at the JSON editor. As you enter your utterances, intents and slot values, they are turned into a JSON file. You can see this and edit it or import new JSON files. Click on Interaction Model > JSON Editor. You will see a JSON file that contains your InvocationName, followed by the intents starting in this example with the AMAZON.CancelIntent.

If you scroll further down, you will see the GetNameIntent, with its associated slot value and (utterance) samples.

● 59

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You can edit the JSON file, for instance, change the invocation name or add or remove intents. Let’s get the user’s name interaction going and then add a birthday request.

4.7 Accessing the slot Before changing our code, let’s see how we access the slot information in our code. Remember our program receives a JSON request with all this information. This is passed to our program in the handler_input > request_envelope. To see it, you can use the logging code: logger.info(handler_input.request_envelope). Python • The slots are read using the code: - slots = handler_input.request_envelope.request.intent.slots • The slots value extracted using, - name = slots[“name”].value • Or you can add the import: - from ask_sdk_core.utils import get_slot_value • And then use - name = get_slot_value(handler_input=handler_input, slot_name=”name”) • We’ll use: - slots = handler_input.request_envelope.request.intent.slots - name = slots[“name”].value • We can also read the other items in the request JSON, e.g., the locale: - logger.info(handler_input.request_envelope.request.locale)

● 60

Chapter 4 ● Slots and Dialogs, Saving Session Data

This displays en-GB (for me) in the CloudWatch logs. You could change your greeting to match the user’s locale. We’ll see later how to use the locale information in APLA: {

“when”: “${environment.alexaLocale == ‘de-DE’}”, “string”: { “welcome”: “Willkommen.” }

}

Node.js In Node.js it is similar as a rough guideline: where you see a Python underscore, remove it and capitalize the next letter, so handler_input becomes handlerInput and the slot ‘name’ value obtained using: var name = handlerInput.requestEnvelope.request.intent.slots.name.value; Remember, console.log() is used for debugging.

4.8 The code Let’s change the code so that Alexa asks for the user’s name in the launch request and uses it in its reply. Click the Code tab. If you didn’t do so, change the speak_output code in the launch request to: speak_output = “Hello, please tell me your name”

Add a whole new intent handler code for the GetNameIntentHandler. Add this before class HelloWorldIntentHandler(AbstractRequestHandler):

● 61

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

class GetNameIntentHandler(AbstractRequestHandler):

“””Handler for Get Name Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“GetNameIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response slots = handler_input.request_envelope.request.intent.slots name = slots[“name”].value

speak_output = “Hello ” + str(name)

return ( handler_input.response_builder .speak(speak_output)

# .ask(“add a reprompt if you want to keep the session open”) .response )

It will look like this in the editor:

Now register your intent. At the end of the code, after sb.add_request_handler(LaunchRequestHandler()) add sb.add_request_handler(GetNameIntentHandler())

● 62

Chapter 4 ● Slots and Dialogs, Saving Session Data

Click the Save and Deploy buttons to save and deploy your skill. Node.js If you didn’t do so, change the speak_output code in the launch request to:

const speakOutput = ‘Hello, please tell me your name’;

Add the following intent handler: const GetNameIntentHandler = { canHandle(handlerInput) { return Alexa.getRequestType(handlerInput.requestEnvelope) ===

‘IntentRequest’

&& Alexa.getIntentName(handlerInput.requestEnvelope) ===

‘GetNameIntent’; },

handle(handlerInput) { var name = handlerInput.requestEnvelope.request.intent.slots.name.value; const speakOutput = ‘Hello ‘ + name; return handlerInput.responseBuilder .speak(speakOutput)

//.reprompt(‘add a reprompt if you want to keep the session open for

the user to respond’)

.getResponse(); } };

And include the handler in the code at the end. About line 160, change it to: exports.handler = Alexa.SkillBuilders.custom() .addRequestHandlers( LaunchRequestHandler, GetNameIntentHandler, HelloWorldIntentHandler,

● 63

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Save and deploy your skill. Full code is available in the software document. You can also refer to: https://developer.amazon.com/en-US/docs/alexa/workshops/build-an-engaging-skill/collect-slots/ step2.html Or search for ‘Alexa Cake time tutorial’. 4.8.1 Test your skill

Click the Test tab and set the testing drop-down box to ‘Development’.

Type “Open My slot skill” (or whatever your launch phrase is). If it can’t find your skill, check your invocation name (see 4.3.1).

You can see the slot in JSON input 1 (the request). The name’s value is under request > intent > slots > name > value You can find the userId and use that in your code “user”: { “userId”: “amzn1.ask.account.AH5B...VR2I”

● 64

Chapter 4 ● Slots and Dialogs, Saving Session Data

If your program doesn’t work, start again, or look at the Cloudwatch logs. For a deeper understanding, we’ll see that later.

4.9 Session attributes - saving slot values We’ll use session attributes to save slot values during a session (see section 1.7). These are deleted when the skill session ends. To save them permanently, we’ll use persistent attributes, which we’ll discuss in the next chapter (section 5.3). • To save the slot values, we use the handler_input attributes manager. • To load them we use: - attr = handler_input.attributes_manager.session_attributes • To create or save individual values we use - attr[“name”] = “Joe” • To read them we use - usersName = attr[“name”] • To save them all - handler_input.attributes_manager.session_attributes = attr 4.9.1 Remember their name

Change the GetNameIntentHandler so that it remembers the user’s name. class GetNameIntentHandler(AbstractRequestHandler):

“””Handler for Hello World Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“GetNameIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response attr = handler_input.attributes_manager.session_attributes if “userName” in attr:

userName = attr[“userName”]

speak_output = “Your name is, “ + userName else: slots = handler_input.request_envelope.request.intent.slots userName = slots[“name”].value attr[“userName”] = userName

handler_input.attributes_manager.session_attributes = attr time”

speak_output = “Thank you “ + userName + “, I’ll save that for next

return ( handler_input.response_builder

● 65

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

.speak(speak_output)

.ask(“Say hello, help or tell me your name”) .response )

I’ve also added a prompt (“Say hello, help or tell me your name”) so that the program won’t terminate immediately. It will still time out after about 8 seconds though.

Test your code. It should remember your name.

● 66

Chapter 4 ● Slots and Dialogs, Saving Session Data

Try changing the code so that it can repeat, update or delete your name. If you start a new session, it will ask for your name again. It hasn’t remembered your name between sessions. We’ll see how to save data across sessions using persistent attributes in the next chapter. You may find that you are often reading and saving the session attributes in your code. This can be done in one go by using request and response interceptors, which run before your code runs the launch request and after it’s all finished. See section 5.4. Let’s now ask for the user’s birthday and use dialog delegation to ensure all values are entered. We won’t have to write code to check for this.

4.10 Dialog delegation We’ll just ask for the day and month the user was born (it gets complicated if we ask for the year too). We will need slots for both the day and the month. Consequently, we will tick the box that says “Is this slot required to fulfil the intent?”. Define a new intent (Click Intents, Add Intent and Create Custom Intent) and name it GetBirthdayIntent. Add the following utterances:

(If you’re in the US, you might want to swap day and month) The {day} and {month} are added as new slots when you add the utterance (+).

● 67

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Click ‘Select a slot type’ for each of them, making: {day} type AMAZON.Ordinal {month} type AMAZON.Month

AMAZON.Ordinal will convert words representing ordinal numbers (e.g., first, second, third) into digits (1, 2, 3). If we add a year, we’d use AMAZON.FOUR_DIGIT_NUMBER, but the dialog gets confused between this and the AMAZON.Ordinal above and can’t easily distinguish between a year and a date of the month. Note: we don’t check for an invalid date (45th November). We can now use dialog management to let Alexa ask for all the values. Return to the sample utterance, click the {day} and edit dialog.

You can also find this from the left-hand side menu:

● 68

Chapter 4 ● Slots and Dialogs, Saving Session Data

Scroll down to enable Slot Filling (Is this slot required to fulfil the intent?).

Fill in the Alexa speech prompt box: “What will Alexa say to prompt the user to fill the slot?” Add some prompts such as “on which date of the month were you born”. Or “On what day of the month were you born”. Enter the prompt and click the plus sign. You could help by prompting “Tell me on what day were you born. Please give me a number, rather than a day”, or “I was born on the fifteenth, when were you born?” And add a user utterance reply (the more the merrier).

You now have to add your user’s possible replies, but they may give both a day and a month, e.g., “The 3rd of December”, so keep this in mind.

● 69

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You should have something like this:

Repeat for the month slot: You might have 1 Which month were you born? 2 In which month were you born? 3 Please tell me the month you were born in. Again, you could make your requests a bit more human. For instance, you could prompt: “I was born in March, when were you born?”.

● 70

Chapter 4 ● Slots and Dialogs, Saving Session Data

And user responses: • {month} • I was born in {month} • (day} {month} You can see how this affects the JSON code: “slots”: [ {

“name”: “day”, “type”: “AMAZON.Ordinal”, “confirmationRequired”: false, “elicitationRequired”: true, “prompts”: { “elicitation”: “Elicit. Slot.799842400349.666146329198” } },

And further on the information about the Elicit.Slot “id”: “Elicit.Slot.799842400349.666146329198”, “variations”: [ {

“type”: “PlainText”, “value”: “what day were you born on?”

} ]

● 71

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

To ensure Alexa does the delegation, enable auto delegation. This is below your utterances:

If you want to confirm a slot, you enable slot confirmation and add a speech prompt

You can do all the delegation yourself using a Dialog.Delegate directive. For each part of the conversation, Alexa sends an intent request with the dialogState set to STARTED, IN_PROGRESS, or COMPLETED. Your skill returns the Dialog.Delegate directive for incomplete dialogs until the dialogState is set to COMPLETED. For further information, see: https://developer.amazon.com/en-US/docs/alexa/custom-skills/delegate-dialog-to-alexa.html#ways-to-delegate-the-dialog-to-alexa You can also enable slot validation rules to accept or reject a set of values and synonyms. Save and build your model. 4.11 The Birthday code

Let’s add the GetBirthdayIntentHandler code. Change the prompt so that it asks for the user’s birthday.

● 72

Chapter 4 ● Slots and Dialogs, Saving Session Data

In GetNameIntentHandler, change the speak_output code in order to: speak_output = “Hello “ + userName + “. Please tell me your birthday.

What day and month were you born?”

or speak_output = “Thank you “ + userName + “, I’ll save that for next

time. Please tell me when were you born.”

and change the .ask in the return to: .ask(“Please tell me your name or birthday”)

Add the GetBirthdayIntent code. We don’t have to check for any empty slots, as this intent will only be executed when they are filled. class GetBirthdayIntentHandler(AbstractRequestHandler): def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“GetBirthdayIntent”)(handler_input) def handle(self, handler_input): slots = handler_input.request_envelope.request.intent.slots userName = slots[“name”].value month = slots[“month”].value day = slots[“day”].value

logger.info(“get birthday intent”)

logger.info(day) logger.info(month) speak_output = ‘OK, ‘ + userName + ‘ you were born on the {day} {month}.’.format(day=day, month=month) return ( handler_input.response_builder .speak(speak_output)

.ask(“Say hello, help or tell me your name or birthday again”) .response )

Add the skill builder handler. Put this in the sb = SkillBuilder() code: sb = SkillBuilder() sb.add_request_handler(LaunchRequestHandler()) sb.add_request_handler(GetBirthdayIntentHandler())

● 73

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

The skill works if a complete birthday is entered:

But prompts if the user does not have a complete day and month.

In the UK, we usually say day, month, year. However, it is different from country to country. You can define a different set of utterances by adding another language. It’s not fool proof though. Entering “I was born on 2nd November” results in a further request for the month.

● 74

Chapter 4 ● Slots and Dialogs, Saving Session Data

If your skill isn’t working properly, use the Evaluate Model to test the conversation.

You may therefore want to ask for day, month (and year) in separate utterances and intents. The code is available with the software document from Elektor. 4.12 Handling Yes and No intents

In section 1.5.4, we saw the Standard built-in intents. Alexa provides two intents in the built-in intent library to deal with common ‘Yes’ and ‘No’ responses, called AMAZON.YesIntent and AMAZON.NoIntent.

● 75

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You have to enter your own code for these, for example: class YesIntentHandler(AbstractRequestHandler):

“””Handler for Yes Intent.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“AMAZON.YesIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response speak_output = “You said Yes” return ( handler_input.response_builder .speak(speak_output) .ask(speak_output) .response ) class NoIntentHandler(AbstractRequestHandler):

“””Handler for Yes Intent.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“AMAZON.NoIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response speak_output = “You said No” return ( handler_input.response_builder .speak(speak_output) .ask(speak_output) .response )

● 76

Chapter 4 ● Slots and Dialogs, Saving Session Data

Don’t forget to add them at the bottom of your code:

sb = SkillBuilder()



sb.add_request_handler(LaunchRequestHandler()) sb.add_request_handler(HelloWorldIntentHandler()) sb.add_request_handler(YesIntentHandler()) sb.add_request_handler(NoIntentHandler())

You can now say ‘Yes’ or ‘No’ in your code.

Note that it also recognizes other words meaning Yes and No. Node.js code: const YesIntentHandler = { canHandle(handlerInput) { return Alexa.getRequestType(handlerInput.requestEnvelope) === ‘IntentRequest’ && Alexa.getIntentName(handlerInput.requestEnvelope) === ‘AMAZON. YesIntent’; }, handle(handlerInput) { .... // handle the Yes intent } } const NoIntentHandler = { canHandle(handlerInput) { return Alexa.getRequestType(handlerInput.requestEnvelope) === ‘IntentRequest’ && Alexa.getIntentName(handlerInput.requestEnvelope) === ‘AMAZON.

● 77

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

NoIntent’; }, handle(handlerInput) { .... // handle the No intent } }

And add the Yes and No intent to the exports.handler: exports.handler = Alexa.SkillBuilders.custom() .addRequestHandlers( LaunchRequestHandler, NoIntentHandler, YesIntentHandler,

You can also use handlerInput.requestEnvelope.request (.type and .intent.name): const YesIntentHandler = { canHandle(handlerInput) { return handlerInput.requestEnvelope.request.type === ‘IntentRequest’ && handlerInput.requestEnvelope.request.intent.name === ‘AMAZON. YesIntent’; }, handler(handlerInput) { ... } };

Note how we find out the intent name using ask_utils.is_intent_name (Python) or Alexa.getIntentName (Node.js). These extract the intent name from the skill request, see section 3.4.1. These can be combined into a custom intent called (say) YesNoIntent: class YesNoIntentHandler(AbstractRequestHandler):

“””Handler for Yes and No Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return (ask_utils.is_intent_name(“AMAZON.NoIntent”)(handler_input) or ask_utils.is_intent_name(“AMAZON.YesIntent”)(handler_input))

def handle(self, handler_input): # type: (HandlerInput) -> Response

speak_output = “You said Yes or No” return (

● 78

Chapter 4 ● Slots and Dialogs, Saving Session Data

handler_input.response_builder .speak(speak_output) .ask(speak_output) .response )

Don’t forget to include sb.add_request_handler(YesNoIntentHandler()) You don’t add a new YesNo intent. You just write the code to deal with the AMAZON.Yes and Amazon.No intents. But the YesIntent and NoIntent still need to be added. The intent name is only included with an Intent request (not a launch request), so that should also be included in the check. Have a look at the section 3.4.1 and the Node.js code below: Node.js Yes/No: const YesNoIntentHandler = { canHandle(handlerInput) { return handlerInput.requestEnvelope.request.type === ‘IntentRequest’ && (handlerInput.requestEnvelope.request.intent.name === ‘AMAZON. YesIntent’ || handlerInput.requestEnvelope.request.intent.name === ‘AMAZON. NoIntent’); }, handler(handlerInput) { ... } };

(and add the intent to the exports.handler as above)

4.13 Multiple Yes / No sources A Yes or No answer might be required by more than one intent. In this case, we store the calling intent request in session attributes and check for that in the Yes / No intent code. We find the current intent using the code: ask_utils.get_intent_name(handler_input)

and save that in session attributes as follows: attr = handler_input.attributes_manager.session_attributes

attr[“thisIntent”] = ask_utils.get_intent_name(handler_input) handler_input.attributes_manager.session_attributes = attr

In our YesIntent code we check for that:

● 79

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

class YesIntentHandler(AbstractRequestHandler): def can_handle(self, handler_input):

return ask_utils.is_intent_name(“AMAZON.YesIntent”)(handler_input)

def handle(self, handler_input): attr = handler_input.attributes_manager.session_attributes if attr[“thisIntent”] == “ChipsIntent”:

speak_output = “Ok, I’ll add chips to your order”

elif attr[“thisIntent”] == “CheeseIntent”:

speak_output = “Ok, I’ll add cheese to your burger”

else:

speak_output = “Sorry, I’m not sure about that”

For a fuller discussion of Yes /No, see: https://gist.github.com/habuma/675a4efb4a657391a4b0159e9fda78f5

4.14 AMAZON.SearchQuery Ideally, you should capture all your user input so that it leads correctly to your intent but sometimes you may have a less predictable user input. In this case, you can use the AMAZON.SearchQuery slot. The AMAZON.SearchQuery slot type is similar to that a user might enter into a search engine, e.g., “Please tell me about London/Rome/Paris”. For example, a zoo might want to direct customers to an animal or a facility. In this case, we can use an AMAZON.Animal {animal} and an AMAZON.SearchQuery {item} slot types. Add an Intent called locateItem and add the two slots —you don’t have to add any values for these, animals will be detected by Alexa and passed in {animal} and other items will be passed as {item}. Add sample utterances that must include a carrier phrase, something like this:

● 80

Chapter 4 ● Slots and Dialogs, Saving Session Data

We can test these out using the Utterance profiler (section 4.6). Utterance

Intent

Slots

“where is the tiger” “where is the dog” “where is the restaurant“

locateItem locateItem locateItem

item: not filled, animal: tiger item: not filled, animal: dog item: the restaurant, animal: not filled

We can see that non-animal items get passed as type ‘item’. We can trap this out in our code, something like this:

For more information on AMAZON,SearchQuery, have a look at https://developer.amazon.com/en-US/docs/alexa/custom-skills/slot-type-reference.html#searchquery

4.15 ASK SDK Utilities We have used some of the useful ask_utils functions. Here’s a list of them all: • getLocale(requestEnvelope: RequestEnvelope): string; • getRequestType(requestEnvelope: RequestEnvelope): string; • getIntentName(requestEnvelope: RequestEnvelope): string; • getAccountLinkingAccessToken(requestEnvelope: RequestEnvelope): string; • getApiAccessToken(requestEnvelope: RequestEnvelope): string; • getDeviceId(requestEnvelope: RequestEnvelope): string; • getUserId(requestEnvelope: RequestEnvelope): string; • getDialogState(requestEnvelope: RequestEnvelope): string; • getSlot(requestEnvelope: RequestEnvelope, slotName: string): Slot; • getSlotValue(requestEnvelope: RequestEnvelope, slotName: string): string; • getSupportedInterfaces(requestEnvelope: RequestEnvelope): SupportedInterfaces; • isNewSession(requestEnvelope: RequestEnvelope): boolean; We’ll see most of these are used in this book. There are also SsmlUtils, which will add escape to invalid SSML characters in a string, and

● 81

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

ViewportUtils that checks the viewport profile and other device characteristics. For more information: https://developer.amazon.com/en-US/docs/alexa/alexa-skills-kit-sdk-for-nodejs/utilities.html

4.16 Intent error logging As we saw previously we can use logger.info() to log in Cloudwatch any information we want. A useful one to record the intent we are in, is: logger.info(“in “ + ask_utils.get_intent_name(handler_input) (use console.log() for Node.js)

4.17 Language understanding NLU and Automatic speech recognition ASR These have recently been added to the developer console. Natural language understanding (NLU) can deduce what a speaker actually means and not just the words they say. You create and evaluate batches of utterances to improve your Skill’s model accuracy. ASR uses voice recording to evaluate your model’s speech recognition. You upload a Zip file containing an MP3, WAV, OGG, AIFF audio format. NLU: ASR:

https://developer.amazon.com/en-US/alexa/alexa-skills-kit/nlu https://developer.amazon.com/en-US/alexa/alexa-skills-kit/asr

4.18 Summary This has been another detailed chapter covering slots, their types and how that information is passed to our code and how we read and save that data during a session. We designed a skill to ask for a user’s name and birth date and repeat this back to the user. We saw how to save that data using session attributes. In the next chapter, we’ll see how to save that data permanently.

● 82

Chapter 5 ● S3 Storage and DynamoDB Database

Chapter 5 • S3 Storage and DynamoDB Database 5.1 Introduction As we have seen, you can store temporary data using session attributes. For more permanent data you can use DynamoDB or S3 storage, we’ll look at that in this chapter.

5.2 Local storage If you don’t want to use S3 or DynamoDB, there is a quick and easy way. Each function has access to 512MB of storage at no additional cost. This is in its own /tmp folder. This is only available during the session, however, it is not a permanent storage. Let’s quickly see how to do that. Start a new Alexa-Hosted Skill. Click the code button and change the Launch request def handle code to the following: def handle(self, handler_input): # type: (HandlerInput) -> Response

#”a” - Append - Opens a file for appending, creates the file non-existent f = open(“/tmp/tempfile.txt”, “a”)

f.write(“This is how we create and save a file”) f.close() #open and read the file after the appending: f = open(“/tmp/tempfile.txt”, “r”) readFile = (f.read()) f.close() speak_output = readFile + “ and this is how we read it back” return ( handler_input.response_builder .speak(speak_output) .ask(speak_output) .response )

● 83

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Save and deploy the code. Check your skill invocation name. and click the Test tab. Change the skill to Development and invoke the skill The program should run.

5.3 Persistent attributes, DynamoDB and S3 As we saw in the previous chapter, you can save temporary data in a session using session attributes, but they are lost when the session is closed. To save data across sessions, you use persistent attributes. For example, you might want to keep a record of how many times someone has used your skill and congratulate them when they reach a milestone (say every 10 visits). Persistence attributes are used to load and save data, which require some storage. The Alexa-Hosted Skill provides both DynamoDB database storage and S3 (Simple Storage Service) storage.

● 84

Chapter 5 ● S3 Storage and DynamoDB Database

For DynamoDb, a DynamoDbPersistenceAdapter is provided and for S3 there is an S3PersistenceAdapter. We’ll use the DynamoDbPersistenceAdapter here. The persistent attributes are held in the handler_input.attributes_manager • To read the persistent attributes use: - persistent_attributes = handler_input.attributes_manager.persistent_attributes • And to save them use: - handler_input.attributes_manager.session_attributes = persistent_attributes - handler_input.attributes_manager.save_persistent_attributes() This is how you might initialize them: persistent_attributes = handler_input.attributes_manager.persistent_attributes if not persistent_attributes: # First time - set attributes, e.g.:

persistent_attributes [‘counter’] = 0

persistent_attributes [‘state’] = ‘GO’

Note: You can also check for empty persistent_attributes using: if len(persistence_attr) == 0:

Note: Rather than check for non-existent persistent_attributes, you can check for individual items and initialize them, e.g: if ‘email’ not in persistent_attributes: persistent_attributes[‘email’] = “”

We also have to import the adaptor, create a database table and provide access to it. 5.3.1 Code example

Rather than duplicating a lot of code here for Node.js, see the High Low game skill at: https://github.com/alexa-samples/skill-sample-nodejs-highlowgame However, remember that often you can just remove the underscore and capitalize the next letter. In other words, what is in Python:

handler_input.attributes_manager.persistent_attributes

● 85

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

becomes in Node.js: handleInput.attributesManager.persistentAttributes We’ll write some code that records how many times a user has used the skill. Create a new Alexa-Hosted skill. Create Skill > name it > Custom, Alexa-hosted Python / Nodejs > Create skill. Then Start from Scratch and Continue with template. Check your invocation name. Click the code tab and follow these instructions: 1 Import the DynamoDB code: Find requirements.txt file in the code tab, edit it to add the Db adapter, and save it ask-sdk-dynamodb-persistence-adapter==1.15.0

2 Add this code to the top of lamba_function.py: (with the other imports) import os import boto3 from ask_sdk_dynamodb.adapter import DynamoDbAdapter

3 Add the code to initialise the Db (before the LaunchRequestHandler code) ddb_region = os.environ.get(‘DYNAMODB_PERSISTENCE_REGION’) ddb_table_name = os.environ.get(‘DYNAMODB_PERSISTENCE_TABLE_NAME’) ddb_resource = boto3.resource(‘dynamodb’, region_name=ddb_region) dynamodb_adapter = DynamoDbAdapter(table_name=ddb_table_name, create_table=False, dynamodb_resource=ddb_resource) from ask_sdk_core.skill_builder import CustomSkillBuilder from ask_sdk_dynamodb.adapter import DynamoDbAdapter

4 Add the CustomSkillBuilder. This loads the DynamoDB, add it after sb = SkillBuilder() at the bottom of the code. sb = CustomSkillBuilder(persistence_adapter = dynamodb_adapter)

● 86

Chapter 5 ● S3 Storage and DynamoDB Database

We can now load the attributes in the launch request. Change the def handle part to: def handle(self, handler_input): # type: (HandlerInput) -> Response persistent_attributes = handler_input.attributes_manager. persistent_attributes session_attributes = handler_input.attributes_manager.session_attributes if not persistent_attributes:

persistent_attributes[‘visit_counter’] = 1

session_attributes[‘number_of_hellos’] = 0

speak_output = “Hi. This is visit number “ + \

str(persistent_attributes[‘visit_counter’]) + \

“. You haven’t said Hello yet” persistent_attributes[‘visit_counter’] counter’] + 1

= persistent_attributes[‘visit_

handler_input.attributes_manager.persistent_attributes = persistent_attributes handler_input.attributes_manager.save_persistent_attributes() handler_input.attributes_manager.session_attributes = session_attributes return ( handler_input.response_builder .speak(speak_output) .ask(speak_output) .response )

And change the HelloWorldIntent def handle to: def handle(self, handler_input): # type: (HandlerInput) -> Response session_attributes = handler_input.attributes_manager.session_attributes session_attributes[‘number_of_hellos’] += 1 speak_output =

“Hi. This is hello number “ +

str(session_attributes[‘number_of_hellos’])

handler_input.attributes_manager.session_attributes = session_attributes return ( handler_input.response_builder .speak(speak_output)

.ask(“Say hello and I will count them for you”) .response

● 87

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

)

Save, deploy and run the skill a few times. Type in ‘Hello’. The hello message is stored in session attributes and should increase each time you say hello. It is reset each time the skill starts. The visit count is stored in persistent attributes and updated for each new visit.

Exit and re-run the program to see the number of visits recorded and new ‘hello’ count.

If you want to delete the persistent attributes, use: handler_input.attributes_manager.delete_persistent_attributes()

● 88

Chapter 5 ● S3 Storage and DynamoDB Database

5.3.2 DynamoDB database storage

This is a good place to look at how the data is stored in your DynamoDB table. Click the code tab and then DynamoDB.

A separate tab opens showing your database information.

We won’t cover all the items here, but you can see your data by clicking ‘Explore table items’, where you can see your visit counter.

You can the click the edit box to see and edit the items.

Click ‘JSON view’ to see it in JSON format.

● 89

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You can then edit and save changes. We’ll mention DynamoDB later, but not cover it in much further detail. It’s worth noting that the DynamoDB access provided by Alexa-Hosted Skills is limited.

5.4 Request and response interceptors Request and response interceptors are executed every time your skill code is invoked. Request interceptors run before the intent code runs and response interceptors after the code is finished. Consequently, they can be used to load and save the persistent data, rather than load and save them in each intent code. You can, of course, perform other actions in the interceptors. There is a detailed description of how they work, including Python and Node.JS code at the Amazon blog: https://amzn.to/3UNY3Hu • To use these, add more ‘from’s at the top of the code: from ask_sdk_core.dispatch_components import AbstractRequestInterceptor from ask_sdk_core.dispatch_components import AbstractResponseInterceptor

• Add the intercept code before the skill builder: class LoadPersistenceAttributesRequestInterceptor(AbstractRequestInterceptor): #Check if user is invoking skill for first time and initialize preset def process(self, handler_input): # type: (HandlerInput) -> None persistent_attributes = handler_input.attributes_manager.persistent_attributes if not persistent_attributes: # see note

● 90

Chapter 5 ● S3 Storage and DynamoDB Database

persistent_attributes[‘visit_counter’] = 1

logger.info(“Create attributes”) else:

# Convert decimals to integers - AWS SDK DynamoDB issue # https://github.com/boto/boto3/issues/369 pass return class SavePersistenceAttributesResponseInterceptor(AbstractResponseInterceptor): #Save persistence attributes before sending response to user. def process(self, handler_input, response): # type: (HandlerInput, Response) -> None handler_input.attributes_manager.save_persistent_attributes() return

• Register the intercept request and response code after sb = SkillBuilder() and after sb = CustomSkillBuilder(..), so that it’s like this: sb = SkillBuilder() sb = CustomSkillBuilder(persistence_adapter = dynamodb_adapter) # Interceptors sb.add_global_request_interceptor(LoadPersistenceAttributesRequestInterceptor()) sb.add_global_response_interceptor(SavePersistenceAttributesResponseInterceptor()) sb.add_request_handler(LaunchRequestHandler()) ..

• Change the launch request code as follows: • Comment out the code: if not persistent_attributes:persistent_attributes[‘visit_counter’] = 0

• And also comment out these lines: handler_input.attributes_manager.persistent_attributes = persistent_attributes handler_input.attributes_manager.save_persistent_attributes()

● 91

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

We could also save and load the session attributes using interceptors. Run the skill again. The visit number is stored in persistent data. Finally, the exchange is like this: User: Alexa: User: Alexa: User: Alexa: User: User: Alexa: User: Alexa: etc.

open my welcome skill Hi. This is visit number 1. You haven’t said Hello yet hello Hi. This is hello number 1 hello Hi. This is hello number 2 exit open my welcome skill Hi. This is visit number 2. You haven’t said Hello yet hello Hi. This is hello number 1

To delete persistent attributes, use: handler_input.attributes_manager.delete_persistent_attributes()

● 92

Chapter 5 ● S3 Storage and DynamoDB Database

Note: At the moment there is an issue with DynamoDB converting decimals to integers, see https://github.com/boto/boto3/issues/369 Node.js Here’s the Node.js request and response interceptor code. We’ll discuss await and async in chapter 11. const RequestInterceptor = { async process(handlerInput) { const { attributesManager} = handlerInput; let sessionAttributes = attributesManager.getSessionAttributes() let persistentAttributes = await attributesManager. getPersistentAttributes(); sessionAttributes = Object.assign({}, persistentAttributes, sessionAttributes); attributesManager.setSessionAttributes(sessionAttributes); } } const ResponseInterceptor = { async process(handlerInput) { let { attributesManager, responseBuilder } = handlerInput; let sessionAttributes = attributesManager.getSessionAttributes(); let persistentAtttributes = await attributesManager. getPersistentAttributes(); sessionAttributes = Object.assign({}, persistentAtttributes, sessionAttributes) attributesManager.setPersistentAttributes(sessionAttributes); await attributesManager.savePersistentAttributes(); } }

You also need to register the Interceptors. Add the SkillBuilders.custom() code and add the request and response interceptors to the exports.handler, so the code is similar to the following:

● 93

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

const skillBuilder = Alexa.SkillBuilders.custom(); exports.handler = Alexa.SkillBuilders.custom() .addRequestHandlers( LaunchRequestHandler, . . . SessionEndedRequestHandler, IntentReflectorHandler) ) .addRequestInterceptors(RequestInterceptor) .addResponseInterceptors(ResponseInterceptor) .addErrorHandlers(ErrorHandler) .lambda();

For more information see the Alexa blog at: https://amzn.to/2PkXqDP

5.5 DynamoDB We’ve just seen how DynamoDB is used to save persistent attributes. We edited requirements.txt, imported the DynamoDbAdapter, added the code to initialize the adaptor, and added a custom skill builder to load the dynamoDb adaptor. You can use dynamoDb to save your own database data in this way, although there are other cloud-based databases you could use, such as Airtable. DynamoDB is a ‘NoSQL’ database instead of a relational database. It stores items in a JSON document and provides scalability and low latency. It uses methods such as key-value pairs for document storage. Amazon DynamoDB supports PartiQL to query data, but that is beyond the scope of this book. For more information see: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ql-reference.html A good starting point for DynamoDB tables and queries is at: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.html To use the full capabilities of DynamoDB in your AWS skills, rather than just in Alexa-Hosted skills, you must: • Sign up for AWS (This is separate from the developer account) • Sign up for billing at https://portal.aws.amazon.com/billing/signup. • Follow the instructions. You will have to validate your details. • Get an AWS access key (an access key ID and secret access key). • Configure your credentials • Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/

● 94

Chapter 5 ● S3 Storage and DynamoDB Database

You can then access DynamoDB. We won’t cover it any further in this book.

5.6 S3 storage Amazon S3 storage is held in buckets. As part of the AWS Free Tier, you can get 5 GB of Amazon S3; 20,000 GET Requests; 2,000 PUT, COPY, POST, or LIST Requests; and 100 GB of Data Transfer Out each month. Note: Amazon S3 no longer supports creating bucket names that contain uppercase letters or underscores. Let’s see how we can store data, and access it using a pre-signed URL, which we saw with persistent data. We’ll upload an image to S3 Storage and then display it. • Start a new Alexa-Hosted Skill. • Click the S3 storage tab. It opens in a new window.

It will open S3 storage related to the skill.

• Upload an image. • Click Add files, select your image.

•

● 95

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

• Click

to upload your image.

You have limited options with an Alexa-Hosted skill. For instance, you can’t change permissions or grant public access to other AWS accounts, as you can with full AWS and S3 storage. • Close the message page and click on the file’s name to find out the ARN and http URL for the image.

Still, we’ll need the key in the bottom left-hand side:

Now we have something stored in S3 storage, let’s access it from our code. We’ll use create_presigned_url(“Media/jba.png”) to access the image. We’ll use a ‘card‘ to display our image — this is easier than using APL which we’ll discuss in a later chapter. For more information on cards, see: https://developer.amazon.com/en-US/docs/alexa/custom-skills/include-a-card-in-your-skillsresponse.html A pre-signed URL passes a reference to the media, but it doesn’t pass the actual content.

● 96

Chapter 5 ● S3 Storage and DynamoDB Database

At the top of your code, add: from utils import create_presigned_url from ask_sdk_model.ui import StandardCard, Image, SimpleCard my_image_url = create_presigned_url(“Media/jba.png”) image_data = {

“card”: { “title”: ‘My picture’, “text”: “me!”, }

} card = StandardCard(

title=image_data[“card”][“title”],

text=image_data[“card”][“text”], image=Image(

small_image_url=my_image_url, large_image_url=my_image_url ) )

And change the Launch request, to: class LaunchRequestHandler(AbstractRequestHandler):

“””Handler for Skill Launch.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool return ask_utils.is_request_type(“LaunchRequest”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response

speak_output = “Here’s your image” my_image_url = create_presigned_url(“Media/jba.png”) return ( handler_input.response_builder .speak(speak_output) .set_card(card) .ask(speak_output) .response )

● 97

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Note: The signed URL only has a lifetime of 60 s, so the create_presigned_url (and the card building) should really be called dynamically when required. Run the program to see your picture. Scroll down on the developer console to see the screen output:

5.7 Summary In this chapter, we saw how to save data for our skills permanently. You can use DynamoDB or S3 storage. We used persistent attributes in our code importing a DynamoDB Adaptor. Request and response interceptors that act to load and save our attributes were discussed, although, they can have other uses. Finally, we covered S3 storage and wrote a program to display an image on an Alexa device.

● 98

Chapter 6 ● Certification and Publishing

Chapter 6 • Certification and Publishing 6.1 Introduction Before we look into more detail about Alexa skills and communication with Raspberry Pi, we’ll discuss how you get your skills certified and published by the Alexa team. We’ll also take a brief break from coding. Once your skill has been written and tested to your satisfaction, you can get your skill certified and published. A good guide is in reference 1. Before that, however, you may want to develop your code for other locales and get it tested by a Beta tester.

6.2 Adding further languages Obviously, you should communicate with your user in their own language, and you can create skills for multiple locales. A wide range of locales is available. The voice user interface that you design for the language should include the skill invocation name, the intent schema, the sample utterances and any custom slot values. Your code should check and respond in the language it was launched. You should also deploy your code to the relevant endpoints. Alexa provides multiple endpoints: North America, Europe, and India and the Far East. If you do this, you must use the same code at all endpoints. See reference 2. Remember, we get the customer’s locale from the locale in the Launch request: “locale”: “en-GB” Nevertheless, that may not be the preferred language. When you create a skill, you choose your language (e.g., English (UK).

To add another language, select Languages settings, then Add New language.

● 99

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Select the language you want to add and save. If you choose Sync Locales, secondary locales will be added. This syncs all your JSON code and are automatically updated when you save a new interaction model or make a manifest change. You can choose which one you want to be your Primary language.

If you want different invocations, sample utterances, etc., don’t use Sync Locales. If you have used Sync Locales, return to the build tab and select your new language, you’ll see that the JSON models have been copied over for you and you can test the skill in a new language.

6.3 Distribution Clicking the distribution tab takes you to a page which requests information about your skill that will be seen by the users when published.

● 100

Chapter 6 ● Certification and Publishing

The questions are: • Public Name à Provide the name of your skill. • One Sentence Description à Give it an eye-catching description. • Detailed Description à More detailed description for someone wanting to use your skill. • What’s new? à Any updates? • Example Phrases à These can be Single Interaction (skill does one task, then terminate) or Conversational (skill listens for a response). You must give suitable examples for both. The page has some good examples. • Small Skill Icon and Large Skill Icon à You need a 108×108 and 512×512 icon. As it says in tiny writing, you can create an icon using the Alexa Skill Icon Builder at: https://developer.amazon.com/docs/tools/icon-builder.html • Category à Choose from one of the options, e.g., Games & Trivia or Lifestyle • Keywords à Choose some keyword so that users can find your skill • Privacy Policy URL à A link to a privacy policy which includes collection, use, and storage of users’ personal information. • Terms of Use URL à Add the terms of use for this skill. It is important to know that you have to complete the questions for all your locales.

● 101

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

The next screen is Privacy and Compliance.

• Does this skill allow users to make purchases or spend real money? • Does this Alexa skill collect users’ personal information? • Is this skill directed to or does it target children under the age of 13? • Does this skill contain advertising? • Export Compliance. • Testing Instructions à This is important as it is how the Alexa team will check your skill. Note the instructions at the end of the page: “To avoid delays in your skill’s certification, ensure that your skill meets the certification requirements. Please consult Alexa Skills Policy guidelines and review the submission checklist”

6.4 Availability and Beta Testing

This page sets the Availability for your skill. You choose: • Who should have access to this skill? Public or Alexa for Business Organizations • Where would you like this skill to be available? In all countries and regions where Amazon distributes skills or Selected countries and regions

● 102

Chapter 6 ● Certification and Publishing

To distribute in different languages, on the Distribution page > Skill Preview, select the language and complete the fields to display the skill in the Alexa Skills Store. In the Availability  section, select where you want to distribute your skill.

6.5 Beta Tester You can choose to add a Beta tester. You add their email address and they get notified. They then have 90 days to test your skill. If the skill isn’t ready for submission, you will be told so:

If it is ready, you get:

6.6 Validation

Checks are now run and any errors displayed:

● 103

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

And hopefully you get this:

6.7 Submission Finally, you can submit your skill.

You have the choice to certify and publish now, or publish later

As it says, one last round of checks will be made, but it will be checked by the Alexa Team.

6.8 Post Publication Once published (and Amazon lets you!), you can add Skill Launch Phrases and Intent Launch Phrases. We saw these on the Build tab. At the time of writing, these are still in Beta testing. For more information, see Name Free Interactions, reference 3.

● 104

Chapter 6 ● Certification and Publishing

6.9 Analytics You can find out how your skill is performing by looking at the analytics.

You can select a time period and see the number of sessions, Unique customers, utterances.

You can also see the percentage use of Intents (here the Stop intent has been used 12%, the Fallback 25% and the FindAWord 65%). The performance figures provide a confidence figure for the utterances and intents.

● 105

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

With this information, you can improve your skill. Your skill can now use A/B testing which allows you to run two versions of your skill and hence determine the best response.

6.10 Summary We had a look into adding further languages and locales to your skill and how to get it certified and published. This involved Distribution, Availability and Beta testing, Validation Submission, and Post Publication. Once published, you can see how your skill is performing with Analytics.

6.11 References 1. h  ttps://developer.amazon.com/en-US/docs/alexa/workshops/build-an-engaging-skill/distributiontesting/index.html 2. https://developer.amazon.com/en-US/docs/alexa/custom-skills/develop-skills-in-multiplelanguages.html 3. https://developer.amazon.com/en-US/docs/alexa/custom-skills/understand-name-free-interactionfor-custom-skills.html

● 106

Chapter 7 ● Creating Skills with Lambda and ASK CLI

Chapter 7 • Creating Skills with Lambda and ASK CLI 7.1 Introduction We have so far been using Alexa-Hosted Skills to create our skills. This is good for getting started, but has some limitations as we have already seen with S3 configurability and DynamoDB storage and read/write limits. There are two other ways you can create your skills. The first is with the AWS Lambda service and the second using ASK CLI (Alexa Skill Kit Command Line Interpreter). 7.1.1 AWS Lambda skill

With this method, you create your skill model in the developer console (as we’ve done) but this time use the AWS Lambda code editor webpage to create your lambda skill. It’s useful to have both windows open. The skill model and the code then have to be linked – the skill needs to know about the code and vice versa. You give the Lambda function the skill’s ID and the Skill the Lambda function’s ARN (Amazon Resource Number). The developer console interface has changed since making these screen shots, but the procedure is similar. Select your skill name and locale; choose experience (other) and scroll down to Provision your own and hosting region; start from scratch, then create skill. Create a skill in the developer console but this time we are NOT going to use Alexa-Hosted skill. Choose ‘Provision your own’ instead.

Now click Create skill. You can now choose from some templates. Node.js has a greater selection than Python. Choose Start from Scratch.

● 107

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Your skill model will be built and the developer console is opened. To be on the safe side, check the invocation name and set it to your choice. If you click the Code tab now, you won’t have any code.

We’ll edit our code in AWS Lambda console. We need the Skill ID, so select Endpoint and copy the skill ID:

Later we’ll fill in the Default region with our Function code ARN. Now sign into AWS Lambda at https://aws.amazon.com/ You will need an AWS account as mentioned previously.

‘Create function’ takes you to the next screen, with options such as: ‘Author from scratch’, choose a blueprint or browse serverless app repository. The serverless app repository currently has 762 public applications to choose from including ‘hello world’ for both Python and Node.js and alexa-skills-kit-nodejs-factskill. Choose Author from scratch.

● 108

Chapter 7 ● Creating Skills with Lambda and ASK CLI

Notice the permissions. We’ll use the default but you can set permissions to allow read/ write execute for code or S3, Database CloudWatch access. We’ll create permissions when we design a smart home skill. Name your function, select your runtime package (I’m using Python here) and click ‘Create function’.

The next screen provides a function overview, our ARN (needed for the skill) and a code area.

● 109

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

In the ‘Function overview’ section, you add the trigger that runs the program. We will add one from “Alexa Skills Kit”, but your code could run when an image is uploaded, or data is changed in DynamoDB, or one of many other triggers. In the Smart Home chapter, we’ll add a Smart Home trigger. Click the Add trigger button, and select Alexa Skill Kit.

Now add the Skill ID that you copied before, and ‘Add’ the trigger. This ensures that only your Skill with that specific Skill ID can invoke the Lambda function.

You’re returned to the main function screen.

● 110

Chapter 7 ● Creating Skills with Lambda and ASK CLI

Whilst you’re here, copy the ARN.

Switch to the developer console and add the ARN into the Default Endpoint. This is found on the Build tab, Endpoint option. Now you see why it’s useful to have both the developer console and the AWS Lambda screen available. Save the endpoints and rebuild your model.

Return to the AWS function and look at the editor.

This is the basic lambda Python code that will work, but of course doesn’t know anything about your intents. You can test this code. Click the Test button and save the suggested function (myTest) and execute it.

● 111

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

It says, ‘Hello from Lambda’. Note that the code starts with def lambda_handler(event, context): You can now enter your code or upload a zip file (we’ll do this later too). For this example, we’ll enter the code. I’ve modified the code from this link: https://aws.amazon.com/blogs/machine-learning/use-aws-deeplens-to-give-amazon-alexa-the-powerto-detect-objects-via-alexa-skills/ (See the software available from Elektor, chapter 7) But there are many examples available. Copy and paste the code into the editing area. Save it.

Deploy your code. You can test this from the lambda console by copying and pasting the JSON Input in the developer console into the Test event we saw before. Or test it from the developer console, launching it with the invocation name you set earlier.

● 112

Chapter 7 ● Creating Skills with Lambda and ASK CLI

We used a card before to display our image. Here we use a card in the build_speechlet_response function to display the text: ‘card’: {

‘type’: ‘Simple’, ‘title’: “SpeechletTitle - “ + title, ‘content’: “SpeechletContent - “ + output

}

This is a quick and easy way to display information on a device with a screen.

A new intent has been added in this code to show you how it’s done. def say_hello(): session_attributes = {} card_title = “Hello”

speech_output = “Well done, your intent skill works” reprompt_text = “Say hello again” should_end_session = False return build_response(session_attributes, build_speechlet_response( card_title, speech_output, reprompt_text, should_end_session))

It’s called from the def on_intent (intent_request, session): code: def on_intent (intent_request, session):

“”” Called when the user specifies an intent for this skill “”” intent = intent_request[‘intent’]

intent_name = intent_request[‘intent’][‘name’]

● 113

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

# Dispatch to your skill’s intent handlers

if intent_name == “hello”: return say_hello()

We haven’t yet added the “hello” intent in the interaction model. Go to the developer console, Interaction model intents. You’ll see there’s an AMAZON.Hello custom intent. Delete that. Now add a new intent — call it ’hello’ and add some utterances (hi, hello, good morning). Save and build your model. Test your skill again.

Now we’ll see how to do this from an Integrated Development Environment (IDE) using the command line.

7.2 ASK CLI A good introduction to ASK CLI is in reference 1. You need to install ASK CLI on your PC. To download and install the ASK CLI, see reference 2.

● 114

Chapter 7 ● Creating Skills with Lambda and ASK CLI

You need some prerequisites for using ASK CLI: 1 A Developer account, an update version of Node.js, or Python and Git access if you want to clone skills. 2 Set up an AWS IAM user and obtain AWS credentials. 3 Install and initialize ASK CLI. Use the ‘ask configure’ command to initialize ASK CLI with your Amazon and AWS credentials. 4 Use ASK CLI commands to manage your skill. Use the ‘ask new’ command to create a new skill and ‘ask deploy’ to deploy your code. 5 Test your skill. Use the ‘ask run’ command to start a debugging session to test your skill locally and the ‘ask dialog’ to send requests to your skill.

7.3 Visual Studio code One popular IDE is Microsoft’s Visual Studio code (AKA VS code). This provides a file manager, an editor, and terminal to use ASK CLI.

File manager

Editor

Terminal

● 115

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

An ASK extension is available for VS Code:

As a quick introduction, start Visual Studio Code. To create your skill type: ask new. This will create a folder containing your code and add the ‘boilerplate’ code. Answer the questions: choose Python, AWSLambda, Hello world skill, your skill name and folder name (in the example, it creates the skill in AlexaAPL folder).

When you open the folder AlexaAPL, you see the necessary files have been created for you.

● 116

Chapter 7 ● Creating Skills with Lambda and ASK CLI

The skill.json file in the skill package folder holds the overall information on the skill (the skill name, locale(s), sample phrases, interfaces, testing instructions, etc.).

Your skill model is found in skill-package > interactionModels > custom folder > en-US.json (or en-GB.json or en-DE json etc)

● 117

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

The code is found in lambda/hello_world.py.

Edit these as needed. If you are using Python, check that the runtime version in ask-resources.json is the latest — “python3.9”, otherwise you may get an error message telling you to update the version. Change it to:

“runtime”: “python3.9”,

● 118

Chapter 7 ● Creating Skills with Lambda and ASK CLI

Intents are added as we did before with classes: class SolveIntentHandler(AbstractRequestHandler):

“””Handler for Help Intent.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“SolveIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response speak_output = “Tell me the letters followed by their colour for this

attempt”

return ( handler_input.response_builder .speak(speak_output) .ask(speak_output) .response )

And register using: sb.add_request_handler(SolveIntentHandler()) Once you have modified your program and saved it, deploy the code. Type ask deploy.

● 119

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You can now go to the developer console and test the skill from there, or use ask dialog to test the code from the terminal. Here’s an example with one of my skills:

For Node.js I can recommend Stuart Pockington’s blog, reference 3.

7.4 Local debugging It is possible to debug your skill entirely on your PC. This works by routing requests to an Alexa proxy on your developer computer. The steps are quite involved. We’ll just summarize them here. Once you have created your skill, you need an access token. The AWS command for this is: ask util generate-lwa-tokens –scopes alexa::ask:skills:debug This takes you to the Log in with Amazon (LWA) page to obtain an access token (see 13.3). After logging in, it returns an access token to your CLI.

Copy the access token. You also need: • the skill_id à find this in the developer console, it starts with amzn1.ask.skill, • the handler name à use ‘handler’, • the file name (hello_world.py), • and region (I’m in the UK, but I still use NA region).

● 120

Chapter 7 ● Creating Skills with Lambda and ASK CLI

Now create a new file in your Lambda folder called local_debug.py as follows:

from ask_sdk_local_debug.local_debugger_invoker import LocalDebuggerInvoker if __name__ == “__main__”: LocalDebuggerInvoker([

‘--accessToken’, ‘’, ‘--skillId’, ‘’, ‘--skillHandler’, ‘’, ‘--skillFilePath’, ‘’ “--region”, “” # Optional argument. Region defaults to NA.]

).invoke()

Enter the values you’ve obtained from LWA into the < . . . > sections so that it’s like this:

7.4.1 Add Alexa debugger configuration

Now configure your preferred development environment (IDE). You add an Alexa debugger configuration so that your skill can connect to the Alexa proxy. This is explained in more detail in reference 4. We will create a launch.json file. Follow these steps: Open the Run menu. Select the Run icon in the Activity Bar on the side of VS Code. Or run from the menu bar at the top. Also, you can use the keyboard shortcut Ctrl+Shift+D.

● 121

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Run

à

Click ‘create a launch.json file’ and choose Add Configuration….

Select your debugger ASK: Alexa Skills Debugger (Python) (or ASK: Alexa Skills Debugger (Node.js).

This adds the configuration code to the launch.json file: {

“version”: “0.2.0”, “configurations”: [ {

● 122

“name”: “Debug Alexa Skill (Python)”, “type”: “python”, “request”: “launch”, “program”: “${command:ask.debugAdapterPath}”,

Chapter 7 ● Creating Skills with Lambda and ASK CLI

“python”: “${command:python.interpreterPath}”, “args”: [ “--accessToken”, “${command:ask.accessToken}”, “--skillId”, “${command:ask.skillIdFromWorkspace}”, “--skillHandler”, “lambda_handler”, “--skillFilePath”, “${workspaceFolder}/lambda/lambda_function.py”, “--region”, “NA” ],

“console”: “internalConsole”, “cwd”: “${workspaceFolder}/lambda” }, {

“name”: “Python: Current File”, “type”: “python”, “request”: “launch”, “program”: “${file}”, “console”: “integratedTerminal”, “justMyCode”: true

} ] }

Check if your region is correct and save the changes to launch.json (it saves it in a .vscode folder). Skip “Register a virtual Alexa device” unless you want audio output and touch events simulated. 7.4.2 Test your Alexa skill in VS code

Open the Run menu and choose Start Debugging. Your skill code starts and VS Code establishes a Web Socket connection from your computer to the Alexa proxy service. You can now use the VS Code Alexa simulator, see reference 5. In the Alexa Skills Toolkit side bar, find your skill and open it.

● 123

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Click Test skill, and then choose Open simulator. It’s now just like the developer console. Set the skill stage to Development, and the locale to that you want to test. Enter an utterance, and then click Send.

The simulator sends your request to Alexa, and the response appears. You can now debug and edit your code all in the VS code environment.

7.5 Summary In this chapter, we saw two more ways of creating Alexa skills, the first using Alexa AWS and Lambda functions, and the second developing your code using ASK CLI and VS code. We then saw how to use the Alexa VS code extension to set up and debug your code locally, thus enabling you to create and debug from one environment only.

7.6 References 1.  https://developer.amazon.com/en-US/docs/alexa/smapi/ask-cli-intro.html 2.  https://developer.amazon.com/en-US/docs/alexa/smapi/quick-start-alexa-skills-kit-command-lineinterface.html 3.  https://medium.com/@punkpocko/alexa-presentation-language-293f8cf25a09 4.  https://developer.amazon.com/en-US/docs/alexa/ask-toolkit/vs-code-testing-simulator.html#prepare-test 5.  https://developer.amazon.com/en-US/docs/alexa/ask-toolkit/vs-code-testing-simulator.html#simulator

● 124

Chapter 8 ● Alexa Presentation Language – APL

Chapter 8 • Alexa Presentation Language – APL 8.1 Introduction When Alexa devices with screens first appeared (e.g., the Echo spot), design templates were offered using Display Interfaces. This has now been deprecated in favor of Multimodal Responses and Alexa Presentation Language (APL). This is not to be confused with APL – ‘A Programming Language’ developed in the 1960s. APL provides components for audio and visual displays. They are combined in an APL document that links with your code. APL is a powerful language in its own right, and some visual examples of what can be achieved just using APL can be seen at the APL Ninja web site https://apl.ninja/ .

You can find my dot clock there.

These need no Python or Node.js code and work with both. For Node.js code though, please see reference 1.

● 125

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

There are three types of APL: APL – for screen displays (visual), APLA (for audio) and APLT (for devices with a character display). We won’t cover APLT. The audio and visual is created in the Developer Console by using Multimodal Responses:

8.2 APLA We’ll look at APLA (audio) first. The APLA interface enables Alexa devices to play audio by using an APLA document2. An APLA document is JSON code. This is Alexa’s example: { “type”: “APLA”, “version”: “0.91”, “description”: “Simple document that says hello”, “mainTemplate”: { “parameters”: [ “payload” ],

“items”: [ {

“type”: “Speech”, “content”: “Hello from A.P.L.A”

} ] }

The Alexa.Presentation.APLA.RenderDocument directive is used in your code to send the document to Alexa. This adds the APLA to the ‘directives’ part of our response to Alexa: “response”: { “directives”: [ { “type”: “Alexa.Presentation.APLA.RenderDocument”, “token”: “pagerToken”, “document”: { ● 126

Chapter 8 ● Alexa Presentation Language – APL

“type”: “APLA”, “version”: “0.91”, “description”: “Simple document that says hello”, “mainTemplate”: { “parameters”: [ “payload” ],

“items”: [ {

“type”: “Speech”, “content”: “Hello from A.P.L.A.”

} ] } },

“datasources”: {} }

]

}

See reference 3. 8.2.1 APLA components

You can use multiple components to create different audio effects. The APLA document contains a set of audio clips. These are built from text-to-speech and audio files using APL components. The components are: • speech converts plain text or SSML into speech output. • audio component plays the provided audio file, such as an MP3 file. • silence component plays silent audio for a defined amount of time, this can’t be used on its own. • sequencer component plays a series of audio clips one after another. • mixer component plays a set of audio clips at the same time. • selector component plays one clip from a list. In the following example, the locale is detected and the user greeted in their own language. {

“type”: “Selector”, “items”: [ {

“type“: “Speech”, “when”: “${environment.alexaLocale == ‘en-US’}”, “contentType”: “PlainText”, ● 127

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

“content”: “Hello!” }, {

“type”: “Speech”, “when”: “${environment.alexaLocale == ‘de-DE’}”, “contentType”: “SSML”, “content”: “ guten Tag”

} ] }

The ‘when’ clause can also be used to determine the display screen size and adjust your visual response to suit, among other things. If you want an example of these in use, open the Multimodal Responses, select Audio and click Create Audio Response:

The editor is opened, displaying a selection of the components. Part of it is shown here:

● 128

Chapter 8 ● Alexa Presentation Language – APL

You can press Preview (in the bottom left-hand corner) and play to hear the Audio response:

We won’t look in further detail at the Audio designer, preferring to look at the visual designer in the next chapter. Refer to section 3.4.3 for details of SSML that you can use in APLA and remember the audio file restrictions.

8.3 Datasources The datasources section contains data that you can bind (link) to the APL document. This separates the data from the layout. You must declare a parameter in the mainTemplate. parameters array called payload; ‘payload’ is a legacy requirement. For instance, you could use: “datasources”: { “myData”: { “user”: { “name”: “John” } } }

A dollar symbol is used to access this from our APL. In this case ‘John’ would be extracted from ${payload.myData.user.name}. We’ll find out later that we can load the datasources from a separate file which we could generate from our code as it runs.

8.4 APLA datasource example Let’s create a short skill that uses APLA and datasources to say hello. Procedure • Create an Alexa-Hosted skill using Python, choose ‘Start from Scratch’ skill and “Continue with Template”

● 129

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

• Check your invocation command (I’ve called mine ‘speech skill’) • Select the APL interface – click Interfaces and switch Alexa Presentation Language on

• Click Save Interfaces at the top and rebuild the model • Edit the code to add the APL code: At the top of your code (after all the ‘from’ statements is a good place) add the following: from ask_sdk_model.interfaces.alexa.presentation.apla import RenderDocumentDirective

Below that, add: import json def _load_apl_document(file_path): # type: (str) -> Dict[str, Any]

“””Load the apl json document at the path into a dict object.””” with open(file_path) as f: return json.load(f)

We’ll use that to load our (separate) APLA JSON file Now create a new file calling it APLA.json in the same folder (lambda) as your Python code: Type in the following code and save it as APLA.json

● 130

Chapter 8 ● Alexa Presentation Language – APL

{

“type”: “APLA”, “version“: “0.91”, “description”: “Simple document to say hello”, “mainTemplate”: { “parameters”: [ “payload” ],

“items”: [ {

“type”: “Speech”, “content”: “Hello from A.P.L.A.”

} ] } }

So that it looks like this:

Modify the HelloWorld intent to add the directive as follows: class HelloWorldIntentHandler(AbstractRequestHandler):

“””Handler for Hello World Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“HelloWorldIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response speak_output = “Hello World!”

# .ask(“keep the session open for the user to respond”)

● 131

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

return ( handler_input.response_builder #.speak(speak_output) .add_directive( RenderDocumentDirective( token=”pagerToken”,

document=_load_apl_document(“APLA.json”), datasources={} ) ) .response )

Note the add_directive includes the RenderDocumentDirective and that we load our document using: document=_load_apl_document(“APLA.json”) Save and deploy your code. Test the program, enable testing and invoke your skill (type in your skill name). After that, type or say “Hello”. You should hear “Hello World!” from the speak_output and then “Hello from A.P.L.A.”

If it doesn’t work, look at your Cloudwatch logs. Let’s modify this to use datasources. Currently in the APLA.json we have “content”: “Hello from A.P.L.A.” Change this to “content”: “${payload.myData.text.welcome}” Now we need a datasource. Create a new file (in the lambda folder, as we did with APLA. json), call it datasource.json. Add the following text:

● 132

Chapter 8 ● Alexa Presentation Language – APL

{

“myData”: { “text”: { “welcome”: “Hi. This uses data sources” } }

}

Can you see how “Hi. This uses data sources” is extracted to be used by our APL text? Finally change the code in the HelloWorldIntent from datasources:{ }

to: datasources=_load_apl_document(“datasource.json”)

(see below) Save all the files, deploy and test your code. It should now say “Hi. This uses data sources”.

8.5 Adding an APLA reprompt You can add a standard response to your APLA (just use .ask(“…”) as we’ve seen before) or add an APLA reprompt by using add_directive_to_reprompt. Create a file called reprompt.json in your lambda folder as you did before with APLA.json with this content:

● 133

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

{

“type”: “APLA”, “version”: “0.91”, “description”: “Simple document to say hello”, “mainTemplate”: { “parameters”: [ “payload” ],

“items”: [ {

“type”: “Speech”, “content”: “This is a reprompt”

} ] } }

And change the HelloWorld intent to add the reprompt directive as follows: return ( handler_input.response_builder .speak(speak_output) .add_directive( RenderDocumentDirective(

token=”pagerToken”,

document=_load_apl_document(“APLA.json”),

datasources=_load_apl_document(“datasource.json”) ) ) .add_directive_to_reprompt( RenderDocumentDirective(

token=”pagerToken”,

document=_load_apl_document(“reprompt.json”),

) ) .response )

To enable this, you must have the latest version of ask-sdk-core so modify the requirements.txt to: ask-sdk-core==1.16.1

The current default for Alexa-Hosted Skill is 1.11.1

● 134

Chapter 8 ● Alexa Presentation Language – APL

The reprompt doesn’t work on the developer console, try it on a device. But you can see it in the Alexa response json: “items”: [ {

“type”: “Speech”, “content”: “This is a reprompt”

}

Note If your code uses both APL and APLA, try using the code: from ask_sdk_model.interfaces.alexa.presentation.apl import RenderDocumentDirective as APLRenderDocumentDirective from ask_sdk_model.interfaces.alexa.presentation.apla import RenderDocumentDirective as APLARenderDocumentDirective

then use APLRender… or APLARender… as required

8.6 Summary We introduced the Alexa Presentation Language and saw how to create them using Multimodal response from the Developer Console. We looked at the Audio response and saw that this is more JSON code. Moreover, we added this to our Alexa response by using RenderDocument directive and loaded a separate JSON file into our Python code using load_apl_document. We covered some of the basic APLA components and how we use them. We studied an example on how to bind datasources to our APL documents. In the next chapter, we’ll look at the APL Visual responses.

8.7 References 1.  https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/apla-interface.html 2.  https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/apla-interface.html 3.  https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/apla-interface.html

● 135

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 9 • APL Visual Multimodal Responses 9.1 Introduction We introduced APL in the previous chapter, how to create APL JSON files and how to incorporate APL in our code using the RenderDocumentDirective. We’ll now look at Visual Multimodal Responses. There are some excellent examples of APL at the Ninja web site: https://apl.ninja/

9.2 Creating an APL Visual Response Create a new skill. Add the APL interface and rebuild the skill. The Authoring tool is accessed from the developer console using Multimodal Responses and choosing Visual, the default.

You can now create using the Response Builder or the Authoring Tool.

Response Builder ‘Create with Response Builder’ requests a name and displays a list of common templates, for example: Simple text and image, Multiple choice, text list, Audio player, Video Player, etc.

● 136

Chapter 9 ● APL Visual Multimodal Responses

You then simply change the items you want then ‘Preview and Test’.

Authoring Tool Clicking on ‘Create with Authoring Tool’ also displays a selection of responses that may suit your needs:

● 137

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Select ‘Blank document’ to start designing your Visual Response. You are presented with the Authoring Tool (editor) and a blank screen:

At the top left-hand side, you can switch between Code View or GUI View.

The default is GUI View, with a blank screen. Soon, we’ll see this populated with the component’s layout in the left-hand column and the component’s properties in the righthand column.

● 138

Chapter 9 ● APL Visual Multimodal Responses

If you select Code View, it displays the skeleton code for the blank screen.

You can edit this directly.

The left-hand side of the screen allows you to select from APL code, data, sources, etc. At the bottom of this list, you can find the commands and OnMount – what to do when your APL loads. At the top of the screen, you can select the display you want your APL to use.

● 139

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

And on the right, there is a selection of components that you can drag onto your screen.

As you can see, there is a lot to the visual designer. We’ll cover just enough to get you going.

9.3 Visual Components Like the APLA, there are visual components with associated properties that you use with your visual response. The main components are: • Container – simply contains multiple child items. • EditText – displays an editable box with a single line of text. • Frame – holds a single child and has a border and background colour (color). • GridSequence – displays a set of components in a fixed grid layout. • Image – draws an image on the screen. • Pager – displays a series of pages in a horizontal or vertical direction. • ScrollView – has a single child and scrolls vertically. • Sequence – displays a long scrolling list of components. • Text – displays a single or multiple lines of text. • TouchWrapper – wraps a single child component which responds to touch events. • VectorGraphic – displays a scalable vector graphic image (AVG or SVG). • Video – displays a video player. In itself it has no controls. Components that can receive input from tap, cursor or keyboard events have Actionable component properties, and those from tap or pointer events have Touchable properties.

9.4. APL component example This example APL shows a Frame component containing a Text component:

● 140

Chapter 9 ● APL Visual Multimodal Responses

{

“type”: “Frame”, “borderRadius”: 50, “height”: “100”, “backgroundColor”: “grey”, “item”: { “type”: “Text”, “text”: “Hi”, “width”: “50”, “height”: “50”, “textAlign”: “center”, “textAlignVertical”: “center” }

}

This creates a Frame containing a Text component with text displaying ‘Hi’. You can type it into the Code View and see how it displays between the square brackets for items:.

A circular Frame can be made if the “borderRadius” is set to half the “height” and “width” (e.g., 50,100,100). . Not much happens with You can press the Preview button to see your APL execute this APL but it will execute commands and play videos if they are in the APL. You can also display the APL by sending directly to any of your registered devices:

● 141

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

9.5 Using the Authoring Tool You can use either the Code View or the GUI view to create your visual design. The GUI view makes this easy for you. You can either select the component you want to place on your screen (display) from a drop-down box, or drag it from the selection on the right-hand side. We’ll create a simple APL display with an image and text. We’ll use the GUI View and use both selection boxes and then drag and drop. Select GUI View then the mainTemplate. Click ‘Add component’ then Child components and Container:

This places a container that takes up the whole screen. Scroll through the properties and set the height and width to 100%.

● 142

Chapter 9 ● APL Visual Multimodal Responses

We’re going to add two more containers, one at the top of the screen, holding an image component, and one below that, holding a text component. Ensure that the Container is selected and repeat the above to add another container within the first. (Add Component, select Child Components, ‘Container’, Add) Set the height of this container to 80% and the alignItems to the center.

Check that the first container is selected again and repeat to add a third Container. Set the height of this container to 20% and the alignItems to the center. You can drag components from the left-hand options at any time. This container will be placed below the second container. If you want it the other way around, just drag the container in the GUI View

When you click on the containers in the GUI, you can see where they are placed on your display. Let’s place an image component. Select the 80% container but this time drag and drop the Image component from the right-hand side.

onto the container. It should be centrally placed as we Drag an Image component have set AlignItems to ‘centre’. Select the component in the GUI View and change the height to 480px and width to 640px. You don’t have to select the container to drop the component. Select the second container (with the image) and now drag a Text component from the right-hand side and drop it

● 143

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

onto the 20% container. Select the text component and change the height and width as you desire. Change the text and find an image to display. You can try mine at: https://raw.githubusercontent.com/jallwork/savingTemporaryAlexaFiles/main/perkin.png (But I can’t guarantee it will always be there) You should now have something like this:

Put the text and image source in the datasources:

Here’s my mainTemplate code: “items”: [ {

“items”: [ {

“alignItems”: “center”, “type”: “Container”, “width”: “100%”, “height”: “80%”, “items”: [ {

● 144

Chapter 9 ● APL Visual Multimodal Responses

“source”: “${payload.myData.image.perkin}”, “type”: “Image”, “width”: “640px”, “height”: “480px” } ] }, {

“alignItems”: “center”, “type”: “Container”, “width”: “100%”, “height”: “20%”, “items”: [ {

“text”: “${payload.myData.text.myText}”, “fontSize”: “32dp”, “fontStyle”: “normal”, “type”: “Text”, “width”: “300dp”, “height”: “64dp” } ] } ],

“alignItems”: “center”, “type”: “Container”, “width”: “100%”, “height”: “100%” } ]

Save your APL (I’ve called it videoAPL) and download it. That will incorporate the APL and the datasources.

If it doesn’t save, check that you’ve added the APL interface in your skill.

● 145

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

9.6 Integrating APL and code Let’s change the code so that it displays our image when the user says ‘Hello’. You can link your APL with your code in two ways: a) copy the APL JSON to your code or b) use a link to the document saved in the Authoring Tool. a) Copy the code and save it in a file along with your lambda code. Here it’s saved in a separate folder (which I’ve called ‘apl’ – we’ll do this later).

Use this Python code to load the JSON file. aplDocument = _load_apl_document(‘./apl/videoAPL.json’) Remember from the previous chapter that we use the RenderDocumentDirective in our code. We can extract the separate document and datasources: .add_directive( RenderDocumentDirective( token=”pagerToken”,

document= document=aplDocument[‘document’],

datasources=aplDocument[‘datasources’] )

b) Link to your APL document in the Authoring Tool You use this as your link: doc://alexa/apl/documents/ Python code: .add_directive( RenderDocumentDirective( token=”imageToken”,

document = {

“src” :”doc://alexa/apl/documents/videoAPL”, “type” : “Link”,

}, datasources= {} ) )

● 146

Chapter 9 ● APL Visual Multimodal Responses

Note: You must rebuild the interaction model after making changes to your APL. Node.js code is here: https://developer.amazon.com/ja-JP/docs/alexa/alexa-presentation-language/apl-changes-1-4.html We’ll use a) and copy the code into a JSON file. Return to your code editor. At the top of your code, add the following (after all the ‘from’ statements is a good place): from ask_sdk_model.interfaces.alexa.presentation.apl import RenderDocumentDirective from ask_sdk_model.interfaces.alexa.presentation.apl import ExecuteCommandsDirective So that we can load our APL JSON file, below that add: import json def _load_apl_document(file_path): # type: (str) -> Dict[str, Any]

“””Load the apl json document at the path into a dict object.””” with open(file_path) as f: return json.load(f)

Create a new folder called ‘apl’ in our lambda folder. In there, create a file called videoAPL.json. Copy all the downloaded JSON code into this file. It should look like this:

Modify the HelloWorld intent.

● 147

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

9.6.1 Check for screen support First, we’ll add a check in our code to see if the user’s device has a screen. When the user’s device has a screen that supports APL, the request envelope contains context.system.device.supported_interfaces which includes alexa_presentation_apl

In Python you can use: if handler_input.request_envelope.context.system.device.supported_interfaces.alexa_ presentation_apl is not None:

Or neater, import the library: from ask_sdk_core.utils import get_supported_interfaces

and use if get_supported_interfaces( handler_input).alexa_presentation_apl is not None: # OK

device has a screen

The Node.js code is: const Alexa = require(‘ask-sdk-core’);

if (Alexa.getSupportedInterfaces(handlerInput.requestEnvelope)[‘Alexa.Presentation.

APL’])

{ // OK - device has a screen }

Add the RenderDocumentDirective to the response. All of the HelloWorldIntent handler is now: class HelloWorldIntentHandler(AbstractRequestHandler):

“””Handler for Hello World Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“HelloWorldIntent”)(handler_input)

● 148

Chapter 9 ● APL Visual Multimodal Responses

def handle(self, handler_input): # type: (HandlerInput) -> Response

speak_output = “Here’s your A. P. L.” aplDocument =

_load_apl_document(‘./apl/videoAPL.json’)

if handler_input.request_envelope.context.system.device.supported_ interfaces.alexa_presentation_apl is not None: return ( handler_input.response_builder .add_directive( RenderDocumentDirective( token=”imageToken”,

document = aplDocument[“document”],

datasources= aplDocument[“datasources”] ) ) .response ) else:

speak_output = “Sorry your device doesn’t have a screen” return ( handler_input.response_builder .speak(speak_output) .response

)

You should see your image and text displayed when you run your skill:

● 149

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

9.7 APL Commands Commands are sent in response to a user’s utterance using the APL.ExecuteCommands directive, or event handlers in your APL document that trigger a command (e.g., response to a touch screen or the user saying ‘play video’). https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/apl-standardcommands.html Commands are split into Standard, Media and Custom types. 9.7.1 Standard Commands

Some of the more common commands are: • AnimateItem – runs an animation on an item, such as a transform (translation and rotation). • OpenURL - opens the specified URL in a web browser or application. • Parallel and Sequential - run a series of commands in parallel or sequentially. • Reinflate - rebuilds the entire document with updated properties. • Scroll - scrolls a ScrollView or Sequence forward or backward. • SendEvent – sends an Alexa.Presentation.APL.UserEvent request to your skill. • SpeakItem - reads the contents of a component on the screen. • SpeakList - read the contents of items inside a container. The item scrolls into view before being read. 9.7.2 Media Commands

These are used to control your media. They are split into two groups: PlayMedia - plays the media on a video component (see above) and ControlMedia - controls a media player to play, pause, next, rewind etc. 9.7.3 User-defined commands

APL provides a “commands” property which is used to define a set of commands. The new command can be made from standard and media commands. 9.7.4 Execute Commands directive

The APL.ExecuteCommands directive sends one or more commands to a linked document. This can be in the same or to a previous response and is linked by the same token. Commands can be used for the following actions: (Not an exhaustive list) • Navigate • Scrolling • Change the displayed page • Change a component • Control media • Read the audio content from component(s) • Send commands to an APL extension e.g.: ControlMedia commands are: play pause, next previous, rewind, seek, setTrack If you have Execute directives, you’d also add an ExecuteCommandsDirective for the

● 150

Chapter 9 ● APL Visual Multimodal Responses

action to take (e.g., play). Note: When your response includes both RenderDocument and ExecuteCommands directives the device displays the specified document first, then runs the commands, provided the token matches in both directives.

9.8 Responsive components and Alexa Layouts In the above example, we created a frame that held some text. To make this responsive (like a button we can click), we would have had to wrap it in a touch wrapper. Moreover, we would have had to make sure it worked for all our screen formats. Responsive components combine this for you. For example, when you use the AlexaHeadline template, you can expect it to work well on all of the supported viewport profiles. There is a wide selection of components, including Button, Image, Progress Bar (various types), TextListItem, And of templates: GridList, ImageList, PaginateList. The full list is here: https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/apl-layouts-overview html#available-responsive-components-and-templates To use the responsive components and templates, add the alexa-layouts package to the import: “import”: [ {

“name”: “alexa-layouts”, “version”: “1.5.0”

} ]

We won’t cover this in more detail here.

9.9 Converting Text to speech – using Transformers Transformers convert data in an APL data source into different types, e.g.: • ssmlToSpeech – Converts SSML to speech, • ssmlToText – Converts the input SSML to plain text, • textToHint – Converts the text input to a hint, • textToSpeech – Converts plain text (no SSML tags) into speech. The transformer JSON has an inputPath, an outputName and the transform itself:

● 151

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

“transformers”: [ {

“inputPath”: “oddFactText”, “outputName”: “outputSpeech”, “transformer”: “textToSpeech”

} ]

We’ll see it working in APL design then how to add it to code using Execute Command Directive. 9.9.1 Transformer APL design

We’ll study a simple design that will speak the text from a Text component when it is loaded using the OnMount event. Create a new Alexa-Hosted skill. Add Alexa Presentation Language, choose interfaces > APL > save interfaces and rebuild the model. Create the APL. Select the Build tab and then Multimodal responses > Visual > create with authoring tool > blank document Click the data icon. The transformer takes the text from an inputPath, transforms it and sends it to an outputName. Add the following to the data source: {

“data”: { “properties”: { “oddFactText”: “Flamingos bend their legs at the ankle, not the knee” },

“transformers”: [{ “inputPath”: “oddFactText”, “outputName”: “outputSpeech”, “transformer”: “textToSpeech” }] } }

● 152

Chapter 9 ● APL Visual Multimodal Responses

Be careful – you may have a valid JSON, but the layout may be wrong and not work. We’ll add a Text component to do the speech. It has a speech property that is used to say the words. The speech comes from the outputName of the transform (outputSpeech), and the input coming from the inputPath (oddFactText) which is linked using the ID of the Text component (oddFactText). The speech property of the Text will be: “speech”: “${payload.data.properties.outputSpeech}”, “id”: “oddFactText”, We need an event for the speech to occur. We’ll use OnMount. We’ll add this in the APL first, then later in the Python code (e.g., in the LaunchRequest). Click the APL icon and choose Code View as it is easiest.

Change the APL code to the following. {

“type”: “APL”, “version”: “1.6”, “mainTemplate”: { “parameters”: [ “payload” ],

“items”: [ ● 153

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

{

“items”: [ {

“text”: “${payload.data.properties.oddFactText}”, “type”: “Text”, “speech”: “${payload.data.properties.outputSpeech}”, “id”: “oddFactText”, “onMount”: [ {

“type”: “SpeakItem”, “componentId”: “oddFactText”

} ],

“width”: “100vw”, “height”: “10vh” } ],

“type”: “Container” } ] } }

9.9.2 Operation

We have a container that holds a 10vh (10% viewport height) high Text component. When the Text is loaded (mounted) the OnMount event occurs and speaks the item found in the componentId (oddFactText) which is the speech item that comes from payload. data.properties.outputSpeech, the transformer’s output. The transformer’s input comes from the payload.data.properties.oddFactText. Save your template (I called mine oddFact). Find your code. We’ll load the APL, so create two files: main.json and data.json as we’ve done before. Save the data source in data.json and the APL in main.json. Now modify the code. Add at the top: from ask_sdk_model.interfaces.alexa.presentation.apl import RenderDocumentDirective from ask_sdk_model.interfaces.alexa.presentation.apl import ExecuteCommandsDirective from ask_sdk_core.utils import get_supported_interfaces

● 154

Chapter 9 ● APL Visual Multimodal Responses

import json def _load_apl_document(file_path): # type: (str) -> Dict[str, Any]

“””Load the apl json document at the path into a dict object.””” with open(file_path) as f: return json.load(f)

Replace the Launch request code with the following: class LaunchRequestHandler(AbstractRequestHandler):

“””Handler for Skill Launch.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool return ask_utils.is_request_type(“LaunchRequest”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response return ( handler_input.response_builder #.speak(speak_output)

.ask(“Keep the session open”) .add_directive( RenderDocumentDirective( token=”pagerToken”,

document=_load_apl_document(“main.json”),

datasources=_load_apl_document(“data.json”) ) ) .response )

Test your code. On launch, it should say “Flamingos …”

We can execute code to do this too. Edit the main.json Remove the OnMount from main.json, so that it becomes: {

“type”: “APL”, “version”: “1.6”, “mainTemplate”: { ● 155

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

“parameters”: [ “payload” ],

“items”: [ {

“type”: “Container”, “items”: [ {

“text”: “${payload.data.properties.oddFactText}”, “type”: “Text”, “width”: “100vw”, “height”: “10vh”, “speech”: “${payload.data.properties.outputSpeech}”, “id”: “oddFactText”

} ] } ] } }

Now change the LaunchRequestHandler to include the ExecuteCommandsDirective class LaunchRequestHandler(AbstractRequestHandler):

“””Handler for Skill Launch.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool return ask_utils.is_request_type(“LaunchRequest”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response return ( handler_input.response_builder #.speak(speak_output)

.ask(“Keep the session open”) .add_directive( RenderDocumentDirective( token=”pagerToken”,

document=_load_apl_document(“main.json”),

datasources=_load_apl_document(“data.json”) ) ) .add_directive(

● 156

Chapter 9 ● APL Visual Multimodal Responses

ExecuteCommandsDirective( token=”pagerToken”,

commands =

[{

“type” : “SpeakItem”, “componentId” : “oddFactText” }]

) ) .response )

Try it.

9.10 Summary In this chapter, we saw how to create a Visual Multimodal response. We saw how to use the authoring tool and the components and created a Visual design with both GUI and Code view. We integrated the APL code with our Lambda code using RenderDocumentDirective and saw it execute. We discussed the Execute commands directive, and Alexa layouts that support responsive components. Finally, we saw how to use transformers to convert text to speech, using them in both APL and code. The code is in Appendix 9 of the software.

9.11 References I have a YouTube video at: https://youtu.be/x6425OyqD8E If you haven’t done so, watch the Alexa evangelist video at: https://www.youtube.com/watch?v=IGsvRT_Ak_M This video will tell you more about creating components and data binding.

● 157

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 10 • Alexa In-skill Purchasing (ISP) 10.1 Introduction In this chapter, we’ll see how to extend your skill to add in-skill purchases and possibly make some money. You can offer in-skill products with the following payment models: One-time purchases: Unlock access to features or content within a skill. One-time purchases don’t expire. Examples include game expansion packs, unlocked features, extra characters, etc. Consumables: Content or features that a user can purchase, deplete, and purchase again. For example, hints for a game, in-game currency, extra lives, or “day passes” for premium content. Subscriptions: Offers access to premium content or features for a period of time, charged on a recurring basis until the user cancels the subscription. We’ll look at a one-time purchase. You’ll have to pay to access extra jokes in the skill. Recently Amazon have announced Alexa Shopping Kit which enables you to sell products sold on Amazon though your skill. For the user experience, see https://www.youtube.com/watch?v=8J4IDMhyAOs. To see how you implement this in your code, see https://www.youtube.com/watch?v=QUBc_s9MoZk about 17 minutes in. To implement in-skill purchases in your skill you: a) Create your skill and add in-skill product(s). b) Add ISP support to your skill. You update your interaction model to include custom intents to support user requests to: - Buy or shop for products. - Ask for a refund or cancel a subscription. c) Add code to handle the purchase requests and use your products. Get a list of products that the user is eligible to purchase. Your code then passes the purchase request to Amazon’s purchase flow. Resume the skill correctly after the purchase flow completes. e.g., update the user’s inventory of purchased items and keep track of those items as they are used up. d) Test your skill and get your skill certified.

● 158

Chapter 10 ● Alexa In-skill Purchasing (ISP)

When your skill is live, you can view the purchases as well as the earning and payments for your products.

10.2 Create your ISP skill We’ll create a skill that gives you access to some jokes, but the user can purchase more jokes. Create an Alexa-Hosted skill (Custom > Alexa-hosted > From scratch). Check the invocation. (I’ve used johns joke skill) Add ISP support to your skill. Click Build > Models > Monetize your skill, Select Add in-skill products:

Note the three types of ISP options are here.

Name it paidforjoke and select One-Time Purchase. Then save. (Note: It’s best to give it a simple name.)

● 159

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Now we need to answer the questions about the skill (supported languages, price, icon, testing instructions, etc) Add supported language (English(UK)) and answer the questions: • Display name – use paid joke – again use a simple name that the user can say. • One sentence description – This gives you access to more jokes. • Detailed description – Buy this to gain access to more jokes. • Small icon for this in-skill product – The image used for the product. * • Large icon for this in-skill product – The image used for the product. * * To create an icon, you can use https://developer.amazon.com/docs/tools/icon-builder.html . It downloads both 108 and 512 files sizes. • Purchase prompt description – What the customer hears when making a purchase – Thank you, you’ve now got access to more jokes. • Purchase confirmation description – You’ve just bought access to more jokes. Click Save. Answer the other questions: • Price and Availability: the minimum price is 0.99. • For access, I’ve selected Amazon.com and Amazon.co.uk. • Tax category: Software • For Testing, use: “Start a game. Say ‘upgrade’ to purchase the product”. This is more important if you are going to really publish the skill. Save and click ‘Link to skill’.

Add more languages if you want. If you return to ‘Monetize your skill’, you will also see other products you have available to link and can also add them:

● 160

Chapter 10 ● Alexa In-skill Purchasing (ISP)

That’s it for creating the ISP. Let’s see how we access them from our code.

10.3 Accessing your ISP code For more information, see reference 1. Add a new file called joke.py and insert the joke code. See software Chapter 10 for the code. The short-form version of joke.py is: import random def freejokes():

jokes = [“I was awake all night wondering where the sun went, but then it

dawned on me”,

“What‘s brown and sticky? A stick.” ] randNo = random.randint(0,len(jokes)-1) return jokes[randNo]

def paidjokes():

jokes=[“I‘m reading a book about anti-gravity. I can‘t put it down.”, “What do you call it when one cow spies on another? A steak out!”“ ] randNo = random.randint(0,len(jokes)-1) return jokes[randNo]

● 161

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Now go to lambda_function.py. Add the code: import joke at the top of the code and add this line of code below the speak_output in launch request: speak_output = joke.freejokes()

Deploy and test Let’s check that free and paid jokes work in the HelloWorld handler. We’ll use session attributes to check for paid joke access: Change the code in the HelloWorld handler to class HelloWorldIntentHandler(AbstractRequestHandler):

“””Handler for Hello World Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“HelloWorldIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response

ask_output = “Here’s a free joke: “ + joke.freejokes()

speak_output = “Here’s your free joke: “ + joke.freejokes() + “. You can

purchase more jokes, just say “What can I buy”

session_attr = handler_input.attributes_manager.session_attributes if “paid_jokes” in session_attr:

if session_attr[“paid_jokes”] == True:

speak_output = “Here’s your paid joke: “ +joke.paidjokes()

return ( handler_input.response_builder

● 162

Chapter 10 ● Alexa In-skill Purchasing (ISP)

.speak(speak_output) .ask(ask_output) .response )

Deploy and test. Say ‘hello’ to get a free joke. You can test this for paid for jokes if you set session_attr[“paid_jokes”] =True in the Launch handler: session_attr = handler_input.attributes_manager.session_attributes session_attr[“paid_jokes”] = True

handler_input.attributes_manager.session_attributes = session_attr

Check that you get a joke from the paidjokes list. Later, we’ll prompt for a joke, so add ‘Tell me a joke” to the HelloWorldIntent: (Build > Intents> HelloWorldIntent)

And add “Tell me a joke” to the utterances.

● 163

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

10.4 Retrieve in-skill products, get their information and purchase. To do this, we must add an API client. Add the following line to requirements.txt boto3==1.9.216 ask-sdk-core==1.11.0 ask_sdk

In your code change sb = SkillBuilder() to: sb = StandardSkillBuilder() . Add these imports at the top of the code from ask_sdk.standard import StandardSkillBuilder from ask_sdk_model.services.monetization import ( EntitledState, PurchasableState, InSkillProductsResponse, Error, InSkillProduct) from ask_sdk_model.interfaces.monetization.v1 import PurchaseResult from ask_sdk_model import Response, IntentRequest from ask_sdk_model.interfaces.connections import SendRequestDirective

At the top of your code, add all these functions, which we’ll use later: def in_skill_product_response(handler_input):

“””Get the In-skill product response from monetization service.””” # type: (HandlerInput) -> Union[InSkillProductsResponse, Error] locale = handler_input.request_envelope.request.locale ms = handler_input.service_client_factory.get_monetization_service() return ms.get_in_skill_products(locale)

def get_all_entitled_products(in_skill_product_list):

“””Get list of in-skill products in ENTITLED state.””” # type: (List[InSkillProduct]) -> List[InSkillProduct] entitled_product_list = [ l for l in in_skill_product_list if ( l.entitled == EntitledState.ENTITLED)] return entitled_product_list

● 164

Chapter 10 ● Alexa In-skill Purchasing (ISP)

def get_speakable_list_of_products(entitled_products_list):

“””Return product list in speakable form.””” # type: (List[InSkillProduct]) -> str

product_names = [item.name for item in entitled_products_list] if len(product_names) > 1:

# If more than one, add and ‘and’ in the end

speech = “ and “.join(

[“, “.join(product_names[:-1]), product_names[-1]])

else: # If one or none, then return the list content in a string speech = “, “.join(product_names) return speech skill_name = ”Johns joke skill”

View Product The in_skill_product_response provides information about the product, e.g: {‘in_skill_products’: [{‘active_entitlement_count’: 0, ‘entitled’: ‘NOT_ENTITLED’, ‘entitlement_reason’: ‘NOT_PURCHASED’, ‘name’: ‘paid joke’, ‘object_type’: ‘ENTITLEMENT’, ‘product_id’: ‘amzn1.adg.product.d0aed689-6761-483c-baf9-788’, ‘purchasable’: ‘PURCHASABLE’ ‘purchase_mode’: ‘TEST’, ‘reference_name’: ‘paidforjoke’, ‘summary’: ‘This gives you access to more jokes’}], ‘is_truncated’: None, ‘next_token’: None}

Note the ‘entitled’ and ‘entitlement_reason’. We’ll use these later. Also note the ‘product_id’ we’ll need that later to buy the product in View Product (section 10.4). If you want to view the products, you can add the following to your code in_skill_product_list = in_skill_product_response(handler_input) logger.info(“in_skill_product_list “) logger.info(in_skill_product_list )

● 165

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

We have several scenarios now: 1 The user hasn’t purchased access to the paid jokes. We need to tell them what’s available and ask if they want to purchase. 2 The user has access to the paid skills. Tell a paid joke when asked. 3 The user asks for paid membership and either confirms or declines. 4 The user has ‘paid membership’ and wishes to cancel. See references 2 and 3. Let’s do this a bit at a time. Change the launch request to: tell the user what‘s available so that it becomes: Change the launch request to tell the user what‘s available class LaunchRequestHandler(AbstractRequestHandler):

“””Handler for Skill Launch.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool return ask_utils.is_request_type(“LaunchRequest”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response

speak_output = “Welcome, you can ask for a joke or say hello” in_skill_response = in_skill_product_response(handler_input) if isinstance(in_skill_response, InSkillProductsResponse): entitled_prods = get_all_entitled_products(in_skill_response. in_skill_products) if entitled_prods: speak_output = (

“Welcome to {}. You currently own {} products. “ “To hear a paid joke, say, ‘Tell me a joke.”).format( skill_name, get_speakable_list_of_products(entitled_prods))

else:

logger.info(“No entitled products”) speak_output = (

“Welcome to {}. To hear a joke you can say “ “’Tell me a joke’, or to hear about the paid for jokes “ “for purchase, say ‘What can I buy’, or ask for help”

).format(skill_name)

reprompt = “I didn’t catch that. What can I help you with?” else:

logger.info(“Error calling InSkillProducts API: {}”.format( in_skill_response.message))

● 166

Chapter 10 ● Alexa In-skill Purchasing (ISP)

history.”

speak_output = “Something went wrong in loading your purchase reprompt = speak_output

return ( handler_input.response_builder .speak(speak_output) .ask(speak_output) .response )

Save, deploy and test. At the moment, we aren’t entitled to any ISP, but you can ask for a joke.

10.5 Produce detail and purchase Now let’s add the purchase options – ‘tell me more’ and ‘buy’. We’ll need to add the intents. First, we’ll add a ‘tell me more’ intent that will give details about the product and ask if the user wants to buy it. Add a TellMeMore intent, either using the console:

● 167

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

or edit the JSON: {

“name”: “TellMeMoreIntent”, “slots”: [], “samples”: [ “What can I purchase”, “What can I get”, “Tell me more”, “What can I buy” ]

}

Save and build the model In your code, at the top of the sb.add_request_handler at the end of the code, add this line: sb.add_request_handler(TellMeMoreHandler())

Add the TellMemoreIntent code (say before class LaunchRequestHandler.. ) class TellMeMoreHandler(AbstractRequestHandler): def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“TellMeMoreIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response

logger.info(“In TellMeMoreHandler”) # Tell the user about the product(s)

speak_output = ‘You can upgrade to hear more jokes. Just say, “Upgrade” to

purchase this product.’

reprompt = “I didn’t catch that. What can I help you with?” return handler_input.response_builder.speak(speak_output).ask( reprompt).response

Note: If the user just says ‘buy’, Alexa may hear ‘Bye’ and close the skill, so it’s best to make sure the user says something like ‘Buy johns joke skill’. If you have more than one product you can get a list of them. Your code would be something like this:

● 168

Chapter 10 ● Alexa In-skill Purchasing (ISP)

See reference 1. Save the model and deploy and test the code.

10.6 Purchase Nearly there. Let’s add the Buy intent. Remember, don’t use ‘buy’ as an utterance, Alexa hears ‘Bye’ and terminates Console:

● 169

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Or JSON: {

“name”: “BuyIntent”, “slots”: [], “samples”: [ “let’s buy it”, “upgrade”, “Can I buy it”, “purchase”, “I want to buy it” ]

},

If you have more than one product, your utterances will be something like: let’s buy {productCategory} pack Although it’s best to include this even if you only have one product. See the Alexa Sample skill for more information. Now let’s add the ‘Buy’ code. We will need two intent handlers here. One to purchase the product (which passes this to Amazon) and a second one to deal with the response from Amazon with what the user says (e.g., Yes or No to ‘Buy’). We will do this in two steps. We’ll need the product ID. We saw that before (## in View Product). You can also find it by returning to Build > Monetize and choose the product:

Copy the Product ID. It’s something like: amzn1.adg.product.b6c09ad1-da89-4c54-a924-7f10d5ef1234

● 170

Chapter 10 ● Alexa In-skill Purchasing (ISP)

Add the sb handler as before, so that the code is like this: sb = StandardSkillBuilder() sb.add_request_handler(TellMeMoreHandler()) sb.add_request_handler(BuyHandler()) sb.add_request_handler(LaunchRequestHandler())

Add the following Buy intent handler code (below your TellMeMoreIntent code) class BuyHandler(AbstractRequestHandler):

“””Handler for letting users buy the product. User says: Alexa, upgrade

“””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“BuyIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response logger.info(“In BuyHandler”)

product_id = “PASTE YOUR PRODUCT ID HERE” return handler_input.response_builder.add_directive( SendRequestDirective( name=”Buy”,

payload={

“InSkillProduct”: { “productId”: product_id }

},

token=”buyToken”) ).response

This code sends a ‘Buy’ request to Amazon with the product ID. Note that we use an add_directive here. This handles the purchase, then returns control to our skill along with the purchase response. After the purchase flow completes, Alexa launches your skill again. We need a code to handle the purchase response and continue with the skill. We’ll add a ‘buy response handler’. This handles the Connections.Response event. For more detail regarding this, see reference 4.

● 171

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

We receive a payload.purchaseResult which is either ACCEPTED, DECLINED, ALREADY_PURCHASED, or ERROR. We will deal with all these cases. Add the sb add_request as before, so that it is: sb.add_request_handler(TellMeMoreHandler()) sb.add_request_handler(BuyHandler()) sb.add_request_handler(BuyResponseHandler())

Add the BuyResponseHandler code: class BuyResponseHandler(AbstractRequestHandler):

“””This handles the Connections.Response event after a buy occurs.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return (ask_utils.is_request_type(“Connections.Response”)(handler_input)

and handler_input.request_envelope.request.name == “Buy”) def handle(self, handler_input): # type: (HandlerInput) -> Response

logger.info(“In BuyResponseHandler”)

if handler_input.request_envelope.request.status.code == “200”: speech = None reprompt = None purchase_result = handler_input.request_envelope.request.payload.get(

“purchaseResult”)

if purchase_result == PurchaseResult.ACCEPTED.value: session_attr = handler_input.attributes_manager.session_attributes session_attr[“paid_jokes”] = True

handler_input.attributes_manager.session_attributes = session_attr speech = “You have bought the access to more jokes, enjoy” reprompt = “You can ask for more jokes”

elif purchase_result in ( PurchaseResult.DECLINED.value, PurchaseResult.ERROR.value, PurchaseResult.NOT_ENTITLED.value):

speech = “Thanks for your interest. Ask for another free joke” reprompt = “Ask for a joke”

elif purchase_result == PurchaseResult.ALREADY_PURCHASED.value: logger.info(“Already purchased product”)

another joke”

● 172

speech = “ You’ve already purchased more jokes, just ask for reprompt = “Ask for a joke, help or exit”

Chapter 10 ● Alexa In-skill Purchasing (ISP)

else: # Invalid purchase result value

logger.info(“Purchase result: {}”.format(purchase_result))

speech = “Sorry, there was an error, please try again” reprompt = “Ask for a joke, help or exit”

return handler_input.response_builder.speak(speech).ask( reprompt).response else:

logger.log(“Error: {}”.format( handler_input.request_envelope.request.status.message))

return handler_input.response_builder.speak(

“There was an error handling your purchase request. “ “Please try again or ask for help”)

Note that we update the session attribute. If necessary, reset the purchases (Build > Monetize> Reset) and test the skill:

Example output saying ‘yes‘ and ‘no‘ to purchase:

● 173

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

And if we’ve already got it:

However, if you try this, you see that we are still getting only the free jokes. Add this code to the LaunchRequest, after entitled_prods = get_all_entitled_products(speak_output = in_skill_response.in_skill_ products) in_skill_response = in_skill_product_response(handler_input) if isinstance(in_skill_response, InSkillProductsResponse): if entitled_prods and len(entitled_prods) > 0: session_attr = handler_input.attributes_manager.session_attributes session_attr[“paid_jokes”] = True

handler_input.attributes_manager.session_attributes = session_attr

10.6.1 Failed Purchase

You should check for a failed purchase. We do that in the BuyResponseHandler code: else: # Invalid purchase result value

logger.info(“Purchase result: {}”.format(purchase_result))

speech = “Sorry, there was an error, please try again” reprompt = “Ask for a joke, help or exit”

10.6.2 Refunds

You must implement a refund / returns policy. As previously, add a new Intent. Call it RefundPurchase, and add utterances: “Refund”, Refund purchase”, “return goods”, “I want a refund”, etc.:

● 174

Chapter 10 ● Alexa In-skill Purchasing (ISP)

Or add the JSON code: {

“name”: “RefundPurchase”, “slots”: [], “samples”: [ “i want a refund”, “return goods”, “refund purchase”, “refund” ]

}

Now add the refund code. Again, like the purchase, we will have two handlers: one will use an add_directive to send a SendRequestDirective with type “Cancel” and the second one to handle the response from Amazon/Alexa (remember the error we had before: Unable to find a suitable request handler) Add the sb handlers sb.add_request_handler(TellMeMoreHandler()) sb.add_request_handler(BuyHandler()) sb.add_request_handler(BuyResponseHandler()) sb.add_request_handler(RefundPurchaseHandler()) sb.add_request_handler(CancelResponseHandler())

class RefundPurchaseHandler(AbstractRequestHandler): def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“RefundPurchase”)(handler_input) def handle(self, handler_input):

● 175

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

# type: (HandlerInput) -> Response

logger.info(“In RefundPurchaseHandler”) product_id= ‘ADD YOUR PRODUCT ID HERE’ return handler_input.response_builder.add_directive( SendRequestDirective( name=”Cancel”,

payload={

“InSkillProduct”: { “productId”: product_id }

},

token=”correlationToken”) ).response

This simply sends a Cancel RequestDirective with the product ID. We need to know the product ID for the item being refunded, to Alexa. If you have more than one, refer to the Alexa code, about line 550 in reference 5: We also need the Cancel response handler. Add the request handler. You could just set the paid_joke session attribute to False: session_attr = handler_input.attributes_manager.session_attributes session_attr[“paid_jokes”] = False

handler_input.attributes_manager.session_attributes = session_attr

But this code handles with other possibilities and gracefully, checking purchase ACCEPTED, and DECLINED. class CancelResponseHandler(AbstractRequestHandler): #This handles the Connections.Response event after a cancel occurs. def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return (ask_utils.is_request_type(“Connections.Response”)(handler_input) and

handler_input.request_envelope.request.name == “Cancel”) def handle(self, handler_input): # type: (HandlerInput) -> Response

logger.info(“In CancelResponseHandler”) in_skill_response = in_skill_product_response(handler_input) product_id = handler_input.request_envelope.request.payload.

get(“productId”)

● 176

Chapter 10 ● Alexa In-skill Purchasing (ISP)

if in_skill_response: product = [l for l in in_skill_response.in_skill_products if l.product_id == product_id]

if handler_input.request_envelope.request.status.code == “200”: speech = None reprompt = None purchase_result = handler_input.request_envelope.request.payload. get(

“purchaseResult”) purchasable = product[0].purchasable if purchase_result == PurchaseResult.ACCEPTED.value: session_attr = handler_input.attributes_manager.

session_attributes

session_attr[“paid_jokes”] = False handler_input.attributes_manager.session_attributes =

session_attr

speech = (“You have successfully cancelled your paid joke

access.”)

reprompt = “Ask for a free joke” if purchase_result == PurchaseResult.DECLINED.value: if purchasable == PurchasableState.PURCHASABLE:

speech = (“You don’t currently have paid joke access.”)

else:

speech = “Ask for a free joke”

reprompt = “Ask for a free joke”

return handler_input.response_builder.speak(speech).ask( reprompt).response else:

logger.log(“Connections.Response indicated failure. “

“Error: {}”.format(

handler_input.request_envelope.request.status.message)) return handler_input.response_builder.speak(

“There was an error handling your cancellation “ “request. Please try again or contact us for “ “help”).response

Don’t forget to remove the paid_joke session attribute. Please note the code in italics. This cancels the user’s access immediately. But the refund workflow doesn’t actually process the refund, it just directs the customer to contact Amazon for a refund, which they may or may not do. You can rely on the entitlement state of the product when the customer launches the skill. When a refund has gone through

● 177

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

then the customer will no longer have entitlement set to ENTITLED. (Thanks to Andy Whitworth). And here’s the response:

See the Alexa code if you have more than one purchased product, reference 1, (line 526): For a Node.js version, see: http://alexaskillstutorials.com/2020/01/23/make-money-with-alexa-in-skill-purchasing-2020/ That’s it! All the software is in Chapter 10 of software.

10.7 References 1.  https://github.com/alexa-samples/skill-sample-python-fact-in-skill-purchases/blob/master/lambda/py/ lambda_function.py 2.  https://developer.amazon.com/en-US/docs/alexa/paid-skills/test-paid-skills.html#test-purchase-flows 3.  https://developer.amazon.com/en-US/docs/alexa/paid-skills/overview.html 4. https://developer.amazon.com/en-US/docs/alexa/in-skill-purchase/ add-isps-to-a-skill.html#handle-results 5. https://github.com/alexa-samples/skill-sample-python-fact-in-skill-purchases/blob / 136b6cd0a78fc7268f38d31a69e08cbb292257d4/lambda/py/lambda_function.py Other references: https://github.com/alexa-samples/skill-sample-java-premium-hello-world/tree/master/lambda/custom/ handlers http://alexaskillstutorials.com/2020/01/23/make-money-with-alexa-in-skill-purchasing-2020/ https://developer.amazon.com/en-US/docs/alexa/in-skill-purchase/ https://github.com/alexa-samples/skill-sample-python-fact-in-skill-purchases

● 178

Chapter 11 ● Progressive Response - Accessing the Internet

Chapter 11 • P  rogressive Response - Accessing the Internet 11.1 Introduction A progressive response can be used to generate an interim response whilst your skill obtains further information before completion. Your skill may need to access information from other web sites before it can respond to the user. For instance, it may have to wait on a taxi company or pizza delivery time to respond, which may take a short while. In the meantime, your skill can use a progressive response to tell the user that it is still working. This can be some speech or music. However, your skill still only has 8 seconds before timing out. Your code can access the Internet using the pypi aiohttp functions See reference 1, but the basic code is: def get(url): with aiohttp.ClientSession() as session: with session.get(url) as resp: return resp.text()

11.2 Steps to Send a Progressive Response To send a progressive response, call the Progressive Response API and send a directive: • Get the apiAccessToken and requestId and construct your API request. • Call the Progressive Response API send a directive (e.g., VoicePlayer.Speak). • Complete your normal skill processing. • Once the progressive response call completes, return your full-skill response object. Reference 2 has a Node.js example video.

11.3 Progressive response example We’ll call the progressive response and then access a web site to find the number of astronauts in space but add a delay to show how it works. This is performed in the get_progressive_response function below using the code: directive_request = SendDirectiveRequest( .. )

# get access token

directive_service_client.enqueue(directive_request)

# send request

and

If you want more details on the API calls, see reference 4. Procedure: Start a new Python Alexa-Hosted skill (I’ve called it progressive response). We’ll add a new intent that finds the number of people currently in space. Add a new intent à call ‘getAstronauts’.

● 179

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Add some utterances: “how many astronauts in space”, “astronauts in space”, “people in space”, “how many people in space” Save and build the model. Click the code tab and edit requirements.txt. Add: aiohttp == 3.8.0. Add the imports: import aiohttp import asyncio import json import time

With the rest of the ‘froms’, add: from ask_sdk_model.services.directive import ( SendDirectiveRequest, Header, SpeakDirective) from ask_sdk_core.skill_builder import CustomSkillBuilder from ask_sdk_core.api_client import DefaultApiClient sb = CustomSkillBuilder(api_client=DefaultApiClient())

We’re using a CustomSkillBuilder for the API client, so comment out the line at the end: #sb = SkillBuilder()

Add the functions we need: async def get(url): time.sleep(5)

# only here to provide long response

async with aiohttp.ClientSession() as session: async with session.get(url) as resp: return await resp.text() def get_progressive_response(handler_input): # type: (HandlerInput) -> None request_id_holder = handler_input.request_envelope.request.request_id directive_header = Header(request_id=request_id_holder) speech = SpeakDirective(speech=”Ok - wait”) directive_request = SendDirectiveRequest( header=directive_header, directive=speech)

● 180

Chapter 11 ● Progressive Response - Accessing the Internet

directive_service_client = handler_input.service_client_factory.get_directive_service() directive_service_client.enqueue(directive_request) return

Change LaunchRequest speak_output code to: speak_output = “Hi, you can ask for how many people in space” Do the same for ‘Hello world’ intent if you want. Add the getAstronauts intent code (say before HelloWorld intent handler). Make sure your intent_name agrees with what you called the intent in the build: class GetAstronautsIntentHandler (AbstractRequestHandler): # Handler for Astronaut Count Intent def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name (“getAstronauts”)(handler_input) def handle(self, handler_input): speech_text = “OK - wait” try: url = “http://api.open-notify.org/astros.json” get_progressive_response(handler_input) htmlData = asyncio.run(get(url)) # extract data jsonResponse = people =

json.loads(htmlData)

jsonResponse.get(‘number’)

speech_text = “There are currently “ + str(people) + “ people in

space”

except Exception as ex:

speech_text = “ An exception occurred”

return ( handler_input.response_builder .speak(speech_text) #.ask(speech_text) .response )

● 181

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Add the intent handler to the skill builder at the end of your code. If you didn’t do so, comment out the sb = SkillBuilder() that’s provided: #sb = SkillBuilder() sb.add_request_handler(LaunchRequestHandler()) sb.add_request_handler(GetAstronautsIntentHandler())

Save, deploy, test, enable development and try your skill. BUT the developer console doesn’t execute progressive responses properly, so try it on a device. Remember, your response still has only 8 seconds to do it all. 11.3.1 Code response

The web site responds with the JSON code that looks (in short form) like this: {

“number”: 10, “people”: [{

“craft”: “ISS”,

“name”“: “Mark Vande Hei” }, {

“craft”: “ISS”,

“name”: “Pyotr Dubrov”

}, {

“craft”: “ISS”,

“name”: “Megan McArthur”

}, {

“craft”: “Shenzhou 13”,

“name”: “Ye Guangfu” } ],

“message”: “success” }

We extract the number of people from this using people = jsonResponse.get(‘number’) You could extract the other information of course.

● 182

Chapter 11 ● Progressive Response - Accessing the Internet

11.4 asyncio, async and await - awaiting a web response We need to wait until our web resource responds with its data. To do this, we use a coroutine. A coroutine is a function that can pause its execution whilst waiting for another operation to complete. In the code, the asyncio.run() function is used to run the coroutine get(url), using the code asyncio.run(get(url)) To create and pause a coroutine, we use the Python async (create) and await (pause) keywords. Here we use async def get(url) and await resp.text(). Note that the await command only works inside the async function; it pauses your code and waits for the function to finish. Before calling the coroutine, we call the progressive response handler using get_progressive_response(handler_input)

For further information, see https://docs.python.org/3/library/asyncio-task.html and reference 3.

11.5 References 1. https://pypi.org/project/aiohttp/ 2. Video: https://www.youtube.com/watch?v=-Vk9Ei6bT4g 3. https://docs.python.org/3/library/asyncio-task.html#running-tasks-concurrently 4.  https://developer.amazon.com/en-US/docs/alexa/alexa-skills-kit-sdk-for-python/ call-alexa-service-apis.html#directiveserviceclient also https://developer.amazon.com/en-US/docs/alexa/custom-skills/ send-the-user-a-progressive-response.html#steps-to-send-a-progressive-response

● 183

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 12 • Creating a Raspberry Pi IoT Thing 12.1 Introduction Now that we know how to create and display our Alexa skill, we can move on to the hardware. In this chapter, we will create an AWS IoT Thing and obtain security certificates to enable MQTT to publish/subscribe to messaging. We will write code for our Pi, which will listen to messages from our AWS console. We will send messages from our AWS console and see them received on our Pi Thing.

12.2 Create a Raspberry Pi IoT The stages involved in creating an IoT are as follows: • Create an Internet of Things (IoT) and its certificates and policies. • Create and run the Python code on the Pi. • Send MQTT messages to the Pi. • Create an Alexa-Hosted Skill and test it with the Pi. • Try it from an Alexa device. Prerequisites. I’m using a Raspberry Pi 4 and Raspberry Pi OS Buster (Debian 10) as the OS. The latest release of is Debian 11 (bullseye), which should also work. My Pi has an explorer HAT I/O board that controls a Pimoroni robot, but you don’t need those, you could just switch an LED on and off. Use US (East Virginia) for the AWS server. You will need an AWS account. The Pi will subscribe to and listen for messages sent to it via our Alexa voice command, which will run some Python lambda code that publishes (sends) the messages (e.g., go forward, stop, reverse turn left / right, etc.) This will be performed using MQTT (Message Queuing Telemetry Transport). MQTT is a lightweight messaging protocol for IoT devices. We will communicate using Publisher/ Subscribe (AKA Pub/Sub). 12.2.1 a) Create our ‘Thing’ and its certificates

Go to the AWS IoT console (https://console.aws.amazon.com/iot/ ) and select Manage > Things. Click ‘Create things’.

● 184

Chapter 12 ● Creating a Raspberry Pi IoT Thing

And choose ‘Create a single thing’. Name your Thing (MyThing). Ignore the Additional Configurations and leave Device Shadow as No shadow. Device Shadow allows devices to sync states with AWS.

Click Next. The next page is ‘Attach policies to certificate’. Click Create policy (opens in new window). A policy controls access to AWS (not necessarily just IoT). Name the policy (MyThingPolicy). To give IoT permissions on the Thing, add the action “iot:*”, resource ARN”*” and check “Allow”. The easiest way is to click the JSON tab and cut/paste/edit to the following:

{

“Version”: “2012-10-17”, “Statement”: [ {

“Effect”: “Allow”, “Action”: [ “iot:*” ],

“Resource”: [ “*” ] } ] }

● 185

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

In the bottom right-hand corner, click “Create”. You should have created a policy.

Return to your previous window, select the policy and click ‘Create Thing’.

Now your certificates are displayed.

These will authenticate our Thing with AWS IoT. Download and save them. They are xxx.pem.crt, public and private pem.key, Root CA1 and CA3 certificates. Click Done and you’ll see that you have successfully created a Thing and certificate.

You can download your certificate again should you need to by visiting AWS IoT > Secure > Certificates.

In the ‘Action’ section select download. This downloads a certificate.pem.crt file.

You have now a ‘Thing’ with a certificate and an attached policy.

● 186

Chapter 12 ● Creating a Raspberry Pi IoT Thing

To see your Thing go to Manage > Things:

You can see your certificates from here or obtain the Thing’s endpoint. 12.2.1 b) Thing’s endpoint

For our skill to communicate with our Thing, we need to know the Thing’s endpoint address. This in IoT > Settings. Click

. You get the notice:

You can click Setting here, or from the menu on the left-hand side.

Copy the endpoint. MQTT clients and AWS IoT Device SDKs use this endpoint. It’s something like: abcdefghijk123-ats.iot.us-east-1.amazonaws.com This will be the endpoint in the Pi’s listener code. We’ll now transfer the certificates to our Pi.

● 187

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

12.2.1 c) Transfer the certificates to your Pi

Create a folder on your Pi (I’ve called it MyThing). Within that folder, create folders called certs.

Transfer the files and certificates to your RPi MyThing/certs folder. I use WinSCP to do this. Rename the xx.private.pem.key file to MyThing-private.pem.key (you don’t need the public cert) rename the xx certificate.pem.crt to MyThing-certificate.pem.crt and use AmazonRootCA1.pem. If you do this, you won’t have to change them in the program. Your folder contents should be like this:

12.2.2 Create and run the Python code

Install the files for the AWSIoT on your Pi, use: sudo pip install AWSIotPythonSDK We need a listener program. This will print “hello” on the terminal when ‘hello’ is sent as a payload[‘directive’] to the ‘/myPi’ topic (we’ll explain that soon). Enter the code and save it in the MyThing folder as listener.py. The software from section 12.1 is the software section. The important parts are: # Create MQTT Client and connect createMQTTClient = AWSIoTMQTTClient(“MyThing”) createMQTTClient.configureEndpoint(‘abcedf12345-ats.iot.us-east-1.amazonaws. com’, 443)

● 188

Chapter 12 ● Creating a Raspberry Pi IoT Thing

# Check these certificate names createMQTTClient.configureCredentials(“/home/pi/MyThing/certs/AmazonRootCA1. pem”, “/home/pi/MyThing/certs/MyThing-private.pem.key”, “/home/pi/MyThing/ certs/MyThing-certificate.pem.crt”) createMQTTClient.connect() # Subscribe to topics and listen createMQTTClient.subscribe(“/myPi”, 1, driveCallback) print (“Listening on /myPi”) and the driveCallback code where the Pi does the work: def driveCallback(client, userdata, message):

print(f”Received {message.payload} from {message.topic}”)

payload = json.loads(message.payload) command = payload[‘directive’]

print(f”Processing command: {command}”)

if command == “hello”: print(“hello”)

else:

print(“Command not found”)

Don’t forget to get the correct path and names for the certificates and have your endpoint in the line of codes highlighted. Run the code (type: python3 listener.py) and check that it works. You should get a message saying “Listening on /myPi”.

If it fails, it will be a problem with your endpoint or certificate. 12.2.3 Send messages to your Pi

You can now send MQTT messages to your Pi from AWS. Log into the AWS console and go to IoT core > Test > Device Advisor > MQTT test client.

● 189

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

First, subscribe to a topic.

Enter /myPi and click Subscribe.

Now select ‘Publish to a topic’, and enter the following into the Message Payload box: {

“directive”: “hello”

}

Check that your Pi is still waiting for messages and click Publish.

● 190

Chapter 12 ● Creating a Raspberry Pi IoT Thing

You should see ‘hello’ on your Pi.

Well done! You’ve sent an MQTT message to your Pi. Troubleshooting If it doesn’t work check that you have entered the correct directive, ‘hello’ not ‘Hello’. Examine that you have copied the correct certificates into the correct folder. Check you have the correct path and certificate names in the createMQTTClient.configureCredentials line of code. Make sure that you have the correct name for your Thing in the code. Extend the code Now we can change our Pi code to do anything we want when we receive payload directive messages. We can publish the following messages to /myPi: { “directive”: “forward”, “data”: “10” }

They are extracted in our code using: command = payload[‘directive’] distance = payload[‘data’] Now we have to write our lambda function code and create our Alexa skill. 12.2.4 Create an Alexa-Hosted Skill

The skill will send an MQTT message when the user says ‘hello’. We’ll add the following command to the HelloWorld intent: send_mqtt_directive(“/myPi”, “hello”, data) Create an Alexa-Hosted (Python) skill – (I’ve called it ‘my hosted thing’) and choose ‘start from scratch’. Open the code tab.

● 191

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Create a folder in the lambda folder called certs. Within that create files called AmazonRootCA1.pem, MyThing-certificate.pem.key and MyThing-private.pem.key. Edit them to include the certificates you downloaded previously. The private.pem.key begins -----BEGIN RSA PRIVATE KEY----and the other two begin with -----BEGIN CERTIFICATE-----

Edit requirements.txt to include AWSIoTPythonSDK==1.5.2 So that it becomes:

Open lambda.py and add the following at the top of the code: from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient and the import: import json Before class LaunchRequestHandler add: “”” Make sure MyThing is the same name as the Thing you created

“””

createMQTTClient = AWSIoTMQTTClient(“MyThing”)

“””

Change this endpoint to suit yours

“””

createMQTTClient.configureEndpoint(‘a3egq123456-ats.iot.us-east-1.amazonaws.com’, 443)

“””

# Check these certificate names if necessary

● 192

Chapter 12 ● Creating a Raspberry Pi IoT Thing

“””

createMQTTClient.configureCredentials(“./certs/AmazonRootCA1.pem”, “./certs/

MyThing-private.pem.key”, “./certs/MyThing-certificate.pem.key”) createMQTTClient.configureAutoReconnectBackoffTime(1, 32, 20)

createMQTTClient.configureOfflinePublishQueueing(-1) # Infinite offline Publish queueing createMQTTClient.configureDrainingFrequency(2) # Draining: 2 Hz createMQTTClient.configureConnectDisconnectTimeout(10) # 10 sec createMQTTClient.configureMQTTOperationTimeout(5) # 5 sec createMQTTClient.connect() def _load_apl_document(file_path):

“””Load the apl json document at the path into a dict object.””” With open(file_path) as f: return json.load(f)

def format_mqtt_message(directive, data): payload = {}

payload[‘directive’] = directive payload[‘data’] = data

return json.dumps(payload) def send_mqtt_directive(topic, directive, data = {}): payload = format_mqtt_message(directive, data)

createMQTTClient.publish(topic, payload, 1)

Change the Launch request speak_output to remind the user to run their Pi code. speak_output = “Make sure your Pi is listening on my Pi topic and say hello” And change the Hello intent code to send the MQTT message to the myPi topic, with a directive ‘hello’ and some data (1234): speak_output= “If your Pi is receiving messages, look at the console output” data = “1234” send_mqtt_directive(“/myPi”, “hello”, data) Deploy your skill. 12.2.5 Test the skill

Start your Pi listening and run your skill: Click ‘Test’ in the Developer followed by the drop-down box from Off to Develop and launch your skill.

● 193

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

When the user says ‘hello’, the skill sends an MQTT message to our Pi. Look at the Pi to see the data’s been received.

Try it from your Alexa device. Congratulations, now that you’ve done that, you can go on to control hardware from your Pi. Troubleshooting If it doesn’t work, check the items suggested above: certificates, policy, topic, thing name, correct folders and file names, etc. Check the Cloudwatch logs.

12.3 Add intents to the Alexa skill Now we’ll add our Alexa intents and utterances. Add a ‘MoveDirectionIntent’ and a ‘StopMovingIntent’. Add utterances for Move: “go {direction}”, “travel {direction}”, and “move {direction}”. N.b. To accommodate commands for both the robot and the LED, we’ll include the command “go on” and “go off” (or “move on” and “move off”) to switch the LED on and off. The utterances for Stop are: “stop going backward”, “stop going back”, “quit going back”, “quit going forward”, “quit moving”, “stop turning”, “stop moving”, etc. Note that these have to be different from the ‘Stop’ utterance that will close the skill.

● 194

Chapter 12 ● Creating a Raspberry Pi IoT Thing

The code will send an MQTT message with either ‘left’, ‘right’, ‘forwards’ or ‘backwards’ as the directive. We won’t send any data – just an empty data field. Either add the intents or edit the JSON. Note the MoveDirectionIntent has slots for left, right, forward(s) and backward(s). The full JSON is in software 12.3 “IoT JSON” MoveDirectionIntent

Where {direction} has a user-defined slot type called ‘directions’:

And slot type ‘directions’ has slot values: off, on, left, right, back, backward, backwards, forward, forwards.

The full JSON is available in the software: 12.3 IoT JSON. You can copy that and paste into the JSON editor.

12.4 Control the robot First, check that your hardware works. You connect an LED to GPIO pin17 of your Pi or use the Pimoroni robot and explorer HAT.

● 195

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

a) LED

Connect the long leg (positive-anode) of an LED to GPIO 17 and the short leg via a resistor (say 1.2 k-Ohms) to GND. You can follow the instructions here: https://projects.raspberrypi.org/en/projects/physical-computing/2 b) For the Pimoroni robot controlled from an explorer HAT. Plug in the Hat. Install the software: sudo pip3 install explorerhat Test this with: cd /home/pi/Pimoroni/explorerhat/examples and run: python test.py More information on installing the explorer HAT is at: https://github.com/pimoroni/explorer-hat Note that at the time of writing the explorer HAT Python library does not currently work with the latest version of Raspberry Pi OS (Bullseye). We will see how to control the robot or switch the LED later. We’ll concentrate on how to send the commands from the Alexa and decode them in the Pi code, which you will have to change if your hardware is different.

12.5 Add intent handlers to the skill code Now we will edit the skill code to add the intents that a) send a directive as the direction (or LED on/off), and b) a directive ‘stop’. Later, we’ll edit the Pi code to control our robot or LED. Add the StopMoving and MoveDirection intent code: class StopMovingIntentHandler(AbstractRequestHandler):

“””Handler for stop moving Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“StopMovingIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response

● 196

Chapter 12 ● Creating a Raspberry Pi IoT Thing

data = {}

speak_output= “Stoping the robot”

send_mqtt_directive(“/myPi”, “stop”, data) return ( handler_input.response_builder .speak(speak_output)

.ask(“I’ve stopped the robot”) .response ) class MoveDirectionIntentHandler(AbstractRequestHandler):

“””Handler for move Intent.”””

def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_intent_name(“MoveDirectionIntent”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response speak_output= “Moving” data = {} slots = handler_input.request_envelope.request.intent.slots logger.info(“slots”) logger.info(slots)

if slots[“direction”].value == ‘left’:

send_mqtt_directive(“/myPi”, “left”, data)

elif slots[“direction”].value == ‘right’:

send_mqtt_directive(“/myPi”, “right”, data)

elif slots[”direction”].value == ’on’: # for LED on send_mqtt_directive(”/myPi”, ”on”, data)

elif slots[”direction”].value == ’off’: # for LED off send_mqtt_directive(”/myPi”, ”off”, data)

elif slots[“direction”].value == ‘back’ or slots[“direction”].value ==

‘backward’ or slots[“direction”].value == ‘backwards’: send_mqtt_directive(“/myPi”, “back”, data) else:

send_mqtt_directive(“/myPi”, “forward”, data)

return ( handler_input.response_builder .speak(speak_output)

.ask(“you can tell me to turn left or right or stop moving”) .response )

● 197

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

And after sb = SkillBuilder(), add: sb.add_request_handler(StopMovingIntentHandler()) sb.add_request_handler(MoveDirectionIntentHandler())

Save your model and deploy the skill. Run listener.py on your Pi. Invoke the skill and send some commands, e.g.: ‘move forward’ or ‘stop’ Check that your Pi is receiving the commands.

Now we’ll change our Pi code to control the hardware.

12.6 Modify your code Now modify your code to suit your hardware. This section shows how to do it for a simple LED or explorerhat 12.6.1 Modify your Pi code - LED

Edit the listener.py code. Add the imports: from gpiozero import LED led = LED(17)

and change the driveCallback code to the following (or whatever you need to control your motor). # Custom MQTT message callbacks def driveCallback(client, userdata, message):

print(f”Received {message.payload} from {message.topic}”)

payload = json.loads(message.payload) command = payload[‘directive’]

print(f”Processing command: {command}”)

if command == “on”: led.on()

● 198

Chapter 12 ● Creating a Raspberry Pi IoT Thing

elif command == “off”: led.off() else:

print(“Command not found”)

The full code can be found in the software, section 12.1.1. 12.6.2 Modify your Pi code - explorerhat

Edit the listener.py code. Add the imports: import explorerhat and change the driveCallback code to the following (or whatever you need to control your motor) # Custom MQTT message callbacks def driveCallback(client, userdata, message):

print(f”Received {message.payload} from {message.topic}”)

payload = json.loads(message.payload) command = payload[‘directive’]

print(f”Processing command: {command}”) if command == “left”: explorerhat.motor.one.forwards(50) explorerhat.motor.two.backwards(50)

elif command == “right”:

explorerhat.motor.one.backwards(50) explorerhat.motor.two.forwards(50)

elif command == “back”:

print(“reversing at 50% speed”)

explorerhat.motor.backwards(50)

elif command == “forward”:

print(“forward at 50% speed”)

explorerhat.motor.forwards(50)

elif command == “stop”:

explorerhat.motor.stop() else:

print(“Command not found”)

The full code can be found in the software, section 12.1.2.

● 199

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

12.7 Test your robot or LED We should now be able to control our robot or the LED using voice commands. Start your program running on your Pi and launch the skill either in the developer console or with your Alexa device. If you have a robot, try a few commands such as: go forwards, stop, go left, etc. If you have an LED, try the voice commands ‘go on’ and ‘go off’.

12.8 Summary In this chapter, we have created a Raspberry Pi IoT device along with its policies. We created a Thing in AWS and wrote code for the Pi to receive MQTT messages and sent some to the Pi. We extended this to create an Alexa skill that sends messages that are received by the Pi that can control some hardware.

● 200

Chapter 13 ● Smart Home Devices

Chapter 13 • Smart Home Devices 13.1 Introduction By using an Alexa Smart Home skill, users can control their devices from your skill. The smart home interfaces use predefined utterances that users say to control your device, such as switching a friendly named light on or off (“Alexa turn the Raspberry Pi LED off”). There is no need to start a skill by saying “Alexa, open...”. The Alexa Smart Home skill enables you to describe your devices and their supported properties, events, and directives. Your code provides this information in a Discover.Response event after receiving a Discover directive from Alexa during the device discovery process. We’ll connect our device to Alexa using Smart Home Endpoints, but you can also connect without a skill using Smart Home Device Connection Options or integrate a hardware module into your product using the Alexa Connect Kit.

13.2 Alexa Interfaces Alexa provides a wide range of interfaces. They use a pre-built voice interaction model and handle requests from Alexa. The user can just say “Alexa, turn off the light” or “Alexa, cook the salmon in the microwave for three minutes”. They do not have to open a skill to do this. Typical smart home interfaces include: • Alexa.PowerController — Turn devices on and off. • Alexa.PowerLevelController – Control the power level of devices. • Alexa.BrightnessController – Control the brightness of devices, • Alexa.ColorController – Change the color of devices, e.g., light bulbs. • Alexa.Speaker – Control the volume of devices that have speakers. Some interfaces are for any devices (e.g., PowerController) whereas others are for specific devices such as cooking skills (oven or microwave temperature or time controllers) or security skills (view camera feed, lock and unlock devices). We’ll see how to set up a simple virtual device and switch it on and off, see reference 1. I hope by now that you have an Amazon Web Services (AWS) account in addition to your Amazon developer account. Before we create our skill and function, we’ll set up the security.

13.3 Login with Amazon (LWA) When a user enables your skill, an account linking process takes place and an access token is provided by your system. This token enables your skill to access the user’s account when the user calls your skill. The easiest way to do this is to use Login with Amazon (LWA).

● 201

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

13.3.1 Create a security profile

Go to the LWA page 2: https://developer.amazon.com/loginwithamazon/console/site/lwa/overview.html You can find that from the developer console dashboard.

Click ‘Create a New Security Profile’.

Enter a description of your product and a URL for your privacy policy.

And save it. Your profile now has a Client ID and Client Secret credentials. To view the credentials, click ‘Show Client ID’ and ‘Client Secret’.

Copy these for use later.

● 202

Chapter 13 ● Smart Home Devices

13.4 Create your Smart Home Skill We’re not using an Alexa-Hosted skill. We’re going to create a Smart Home skill and link that to a Lambda function as we’ve seen previously. For more in-detail information, see references 3 and 5. Create a new skill from the developer console (I’ve called it my smart skill). Choose Smart home as the type of experience from the Experience, Model Hosting page. This time choose the ‘Smart Home’ skill.

Copy the skill ID, e.g., amzn1.ask.skill.9e6293a1-9b54-4dc2-bc4f-e4d343012345 We can’t do any more here yet so leave this page open and see # later in (13.8 Link the function to the skill).

13.5 Create a Lambda function I used my local Ireland server (eu-west-1). If your function isn’t triggered by the smart home detecting devices, try a different server from that you have chosen. Go to the AWS Lambda console and create a new function (my-smart-home-function). It should be Python 3.9 now.

Use the default execution profile.

● 203

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Now add the trigger that will execute your function. Click ‘Add Trigger’ and select ‘Alexa Smart Home’ from the drop-down box.

Now enter the skill Id you saved earlier in the Application ID box:

13.6 Lambda skill code Your skill code will execute when the function receives a trigger from Alexa. Click the code tab and copy and paste the (Python or Node.js) code from here: https://developer.amazon.com/en-US/docs/alexa/smarthome/implement-skill-tutorial.html#add-code You can also upload a zipped file from https://github.com/alexa/alexa-smarthome

● 204

Chapter 13 ● Smart Home Devices

File > Save and Deploy your code.

13.7 Test your Lambda function Before testing our function from Alexa, we’ll test the function from this page by creating a test event. We’re going to send a trigger to the function and see the result. We’ll create a discovery event and a power on event. In your Functions > my-smart-home-skill page, select the Test tab. Then select ‘New event’. Give it a name (e.g., DiscoveryTest). Enter the following code into the Event JSON editor and save it: {

“directive”: { “header”: { “namespace”: “Alexa.Discovery”, “name”: “Discover”, “payloadVersion”: “3”, “messageId”: “Some id” },

“payload”: { “scope”: { “type”: “ BearerToken “, “token”: “access-token-from-skill” } } } }

Repeat for a power on test event, with the following code saving it as PowerOnEvent: {

“directive”: { “header”: { “namespace”: “Alexa.PowerController”, ● 205

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

“name”: “TurnOn”, “messageId”: “SomeNumber”, “correlationToken”: “SomeTokenNumber”, “payloadVersion”: “3” },

“endpoint”: { “scope”: { “type”: “BearerToken”, “token”: “access-token-from-skill” },

“endpointId”: “sample-light-01”, “cookie”: {} },

“payload”: {} } }

Test your function. Select DiscoverEvent from the drop-down box.

Click Test. You should get ‘Execution result: succeeded’:

Click ‘logs’ to see the result in CloudWatch.

Repeat for the PowerOn event.

● 206

Chapter 13 ● Smart Home Devices

We now know our skill will respond to a discovery and power on events.

13.8 Link the function to the skill At the moment, our function knows about our skill, but not vice versa. At the top of the Functions > my-smart-home-skill page find and copy the function ARN.

# Return to your developer skill page (see 13.4). Create a Smart Home Skill above, and on the Smart Home page, paste the ARN for the Lambda function in Default endpoint. This This is found in the Build tab, Endpoint option.

13.9 Configure account linking Click ‘Save’ and ‘Setup Account Linking’ at the bottom of the page.

We’ll return here once we have some more information. Go back to your skill in the Alexa developer console. If not selected, click ‘ACCOUNT LINKING’. If you want to know more about account linking, see: https://developer.amazon.com/en-US/docs/alexa/account-linking/configure-authorizationcode-grant.html

● 207

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Enter the following information. Remember you saved your ClientId and security profile. • Your Web Authorization URI, enter https://www.amazon.com/ap/oa • Access Token URI, enter https://api.amazon.com/auth/o2/toke • Your Client ID, paste the client ID from your security profile. • Your Secret, paste the client secret from your security profile. • Your Authentication Scheme, select HTTP basic (Recommended). • Scope, click +Add scope, and then enter profile:user_id

Click Save at the top of the page. Save the redirect URLs. They will be something like: https://alexa.amazon.co.jp/api/skill/link/MW7PX8KRHZABS https://pitangui.amazon.com/api/skill/link/MW7PX8KRHZABS https://layla.amazon.com/api/skill/link/MW7PX8KRHZABS Click save. Add redirect URLs to your security profile. Alexa uses one of these URLs to access the authorization service when the account is linked. Copy the Alexa Redirect URLs from the bottom of the page.

● 208

Chapter 13 ● Smart Home Devices

Return to the LWA page, click the gear icon by your security profile and select web settings.

Click Edit and paste the return URLs. Use ‘Add another’ to add them all.

Ignore Allowed origins. Go to https://alexa.amazon.com in your browser. Make sure that you log in with same Amazon developer account. Now we can test our skill.

13.10 Enable and Link the skill Enable the skill and link the account. Open your Alexa App on your phone, or sign into alexa.amazon.com on your desktop. Find Skills > All Skills > Your Skills > Dev Skills.

● 209

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Find your skill (I know it was a long time ago that we named it). If you can’t find it, check you have the correct developer account.

Clicking ENABLE takes you to the login page. Sign-in. and Click ‘Allow’.

You should get this message:

Close the window. You’re taken back to the previous window. Discover your device.

Alexa looks for new devices.

● 210

Chapter 13 ● Smart Home Devices

If your device isn’t discovered, check the CloudWatch logs to see if the function was triggered. If not, check you have the copied the ClientID and secret correctly, verify the account linking is correct. If the function is triggered, check the logs to see that a discover was sent, and look at your program’s response. You can also test your function again from the Lambda function. When device is discovered, you can test from the dev console. See reference 3.

You can also run test from the developer console.

● 211

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

13.11 Clean up

If you want to remove all the items you’ve created, go to LWA and delete your Security Profile, go to Lambda and delete your function, and to the developer console and delete your skill. 13.12 Troubleshooting

If it doesn’t work, try these. Remember what happens. Your user discovers the devices (does that work?). The function is triggered by the smart home event (is the function being triggered?). Your function runs (is there a problem with your code?). You can tell this from the CloudWatch logs. If the function isn’t triggered, check your hosting region.

13.13 Summary We saw how to set up security for a smart home device and how to create a smart home skill linked to a lambda function. We tested that function locally before linking to the smart home skill and configuring account linking. Finally, we discovered your skill from Alexa and (virtually) powered on and off our smart home device. In the next chapter, we’ll communicate between our Lambda function and our Raspberry Pi, controlling an LED display on the Pi.

● 212

Chapter 13 ● Smart Home Devices

13.14 References 1.  https://developer.amazon.com/en-US/docs/alexa/device-apis/alexa-interface-reference.html 2.  https://developer.amazon.com/loginwithamazon/console/site/lwa/overview.html 3.  https://github.com/alexa/alexa-smarthome/wiki/Build-a-Working-Smart-Home-Skill-in-15-Minutes 4.  h ttps://developer.amazon.com/alexa/console/ask/test/amzn1.ask.skill.06b36e2a-8fcf-4aa0baaa-475e820300b9/development/en_GB/ 5.  https://developer.amazon.com/en-US/docs/alexa/smarthome/smart-home-skill-tutorial.html

● 213

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 14 • C  ontrolling a smart home raspberry Pi with SQS 14.1 Introduction We have seen how to send MQTT messages to a Pi and how to set up a Smart Home device. In this chapter, we will control a Smart Home device from our Alexa using Simple Queue Service (SQS) messaging1 to communicate between Alexa and our Pi. Of course, we could also use MQTT. Our Pi python code will read the SQS message queue and control a Raspberry Pi Sense Hat (reference 4) switching it on and off or changing the color. These will use the ALEXA. discovery, Power.Controller and the Color.Controller capabilities. The design could be extended to use the MODE controller, say to set specific fire types (a specific combination of flame and color e.g., ‘roaring’, campfire’ or ‘candle’) or use RANGE to control the size of the fire. (Full on to pilot light)2 We’ll start by creating an SQS queue and writing the Pi code to receive a message and control a Sense HAT display. (See section 15.8, but if you don’t have one of those, you can just control an LED). Then we’ll create a Smart Home skill to control this. We should then be able to control our Pi from our Alexa device.

14.2 Create an SQS Queue We’ll need a queue. Find Simple Queue Service from AWS dashboard https://eu-west-1.console.aws.amazon.com/sqs/v2/home

You can choose a standard or a FIFO queue. Leave the queue type as Standard, and give it a name (senseHatQueue). This means messages may arrive in any order, but is simpler.

● 214

Chapter 14 ● Controlling a smart home raspberry Pi with SQS

We’ll let anyone access our queue (not recommended), so click Advanced in the Access Policy and enter the following JSON: {

“Version”: “2012-10-17”, “Id”: “SenseHat_Policy_UUID”, “Statement”: [ {

“Sid”: “Queue1_AnonymousAccess_All”, “Effect”: “Allow”, “Principal”: “*”, “Action”: “sqs:*”, “Resource”: “arn:aws:sqs:eu-west-1:559144301234:senseHatQueue”

} ] }

Change your ARN to suit. Click Create queue. It should successfully be created

Once your queue has been created you can test it by sending and receiving messages:

● 215

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You can list your available queue(s) from your Pi with this Python code: import boto3

client = boto3.resource(‘sqs’,region_name=’eu-west-1’)

print (“Available Queue URL”)

for queue in client.queues.all(): print(queue.url)

Make sure the region name is correct. I get: Available Queue URL https://eu-west-1.queue.amazonaws.com/559144301234/senseHatQueue

14.3 Raspberry Pi SQS code Now we have a queue that we can send messages to, we’ll write some code to receive the messages and control our Pi display. Messages sent by the Smart Home skill are JSON code. They contain a namespace (e.g., “Alexa.PowerController” or ‘Alexa.ColorController’) and a name or value. For instance, the ColorController message is: {

‘endpointId’: ‘sense-hat-01’, ‘namespace’: ‘Alexa.ColorController’, ‘name’: ‘color’, ‘value’: {‘hue’: 350.0, ‘saturation’: 0.25, ‘brightness’: 1.0} }

The full code to receive Smart Home messages is in Software, appendix 14.1. The SQS message is read and then deleted with the code (the fire function controls the Sense Hat): def get_sqs_msg(queue): json_msg = “”

for message in queue.receive_messages(): json_msg = json.loads(message.body) message.delete() return json_msg

● 216

Chapter 14 ● Controlling a smart home raspberry Pi with SQS

The main code is: def main():

powerState = “on” colour =

(255, 0, 0) # This is HSV, not RGB (see ‘value’ above)

fire(powerState, colour) while True: fire(powerState, colour) time.sleep(1) msg_text = get_sqs_msg(queue) if “namespace” in msg_text:

print(msg_text[“namespace”])

if msg_text[“namespace”] == “Alexa.PowerController”: colour = red

if msg_text[“value”] == “ON”: powerState = “on”

colour = red fire(powerState, colour) else:

powerState = “off” colour = (0,0,0) fire(powerState, colour)

if msg_text[“namespace”] == “Alexa.ColorController”: powerState = “on”

colour = msg_text[“value”]

colour = hsv2rgb(colour) fire(powerState, colour) else: # no new message, send last command fire(powerState, colour)

We read the values using msg_text[“namespace”] and msg_text[“value”] Enter the Appendix14_1 SQSsenseHat.py code and run it on your Pi. As we have allowed universal access to the SQS queue, we don’t need any ID or key security!

● 217

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Send the following JSON from the SQS queue {

“endpointId”: “sense-hat-01”,

“namespace”: “Alexa.ColorController”,

“name”: “color”, “value”: {

“hue”: 350.0,

“saturation”: 0.25,

“brightness”: 1.0 } }

Your Pi Sense Hat display should turn light pink.

We know now we can access our queue from our Pi. Let’s create our Smart Home skill.

14.4 Create a Smart Home skill We saw this in chapter 13. From the developer console, create the skill, give your skill a name (e.g., Sense Hat), choose Smart Home and create your skill. You will see a Smart Home service endpoint with your AWS Lambda ARN Skill ID. It starts with amzn1.ask.skill. Copy this for later in the next section.

14.5 Create the function We’ve seen this before, so briefly: Go to AWS Lambda, create function. Author from scratch (we’ll upload the code later), choose Python, name it and click create function. Click Add trigger > Alexa Smart Home and enter the Smart Home skill ID you saved # in the previous section. Click Add. Save the function ARN for section 14.7, e.g.: arn:aws:lambda:eu-west-1:559144301234:function:sensehatfunction

● 218

Chapter 14 ● Controlling a smart home raspberry Pi with SQS

14.6 Create a security profile This is to provide access to the Smart Home skill, not the SQS queue. As before, we’ll use Login with Amazon (LWA). You can use a previously created profile, but if you haven’t got one or don’t want to, the procedure is: • Go to the LWA page: developer.amazon.com/loginwithamazon/console/site/lwa/overview.html • Create a new security profile, enter a description, save it. Save the Client ID and Client Secret credentials We’ll set up the Allowed Return URLs in the next section

14.7 Configure the smart home skill Return to your skill. Enter the AWS Lambda ARN Skill ID you saved earlier (in section 14.5) into the Default Endpoint box. (See Link the function to the skill in chapter 13) Click Save and then Setup Account Linking. As before enter the following information. • Your Web Authorization URI, enter https://www.amazon.com/ap/oa • Access Token URI, enter https://api.amazon.com/auth/o2/tokenn • Your Client ID, paste the client ID from your security profile. • Your Secret, paste the client secret from your security profile. • Your Authentication Scheme, select HTTP basic (Recommended). • Scope, click +Add scope, and then enter profile:user_id Copy the redirect URLs for later and click Save Return to the LWA page, click the gear icon and select web settings. Click Edit and paste the return URLs, see the previous section.

14.8 Add the function code We’ll modify the Alexa’s Jeff Nunn code (thanks Jeff!). This has a specific folder layout, so download the code as advised at https://github.com/alexa-samples/skill-sample-smarthome-fireplace-python/blob/main/instructions/ deploy-the-sample-code.md#package-the-sample-code The zipped code is also available at: https://bookcode.s3.amazonaws.com/fireplace.zip Browse to the skill-sample-smarthome-fireplace-python/lambda/ directory. Zip the index.py file and the alexa, capabilities, and events/ folders into a file named package.zip.

● 219

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Upload this to your function code. You should have a layout similar to the following:

Change the code, or overwrite the lambda_function code with that from software – Appendix14.2 - lambda_function.py • Change the name index.py to lambda_function.py • Add the color controller json [ {

“type”: “AlexaInterface”, “interface”: “Alexa.ColorController”, “version”: “3”, “properties”: { “supported”: [ {

“name”: “color”

} ],

“proactivelyReported”: false, “retrievable”: true } } ]

• Change region in code and sqs

● 220

Chapter 14 ● Controlling a smart home raspberry Pi with SQS

config = Config(region_name = ‘eu-west-1’) # Replace with your region, if necessary sqs_client = boto3.client(‘sqs’, config=config) • Add color to discovery: ## ColorController

color_controller_capabilities = load_capability_definition(“ColorController”) capabilities.append(color_controller_capabilities)

• still in discovery, change code to response.add_payload_endpoint({

‘endpointId’: ‘sense-hat-01’, ‘friendlyName’: ‘Sense Hat display’, ‘manufacturerName’: ‘Developer’, ‘description’: ‘A Raspberry Pi Sense Hat Smart Home skill’, ‘capabilities’: capabilities

})

Change if namespace == ‘Alexa.ColorController’: to: if namespace == ‘Alexa.ColorController’:

endpoint_id = request[‘directive’][‘endpoint’][‘endpointId’] token = request[‘directive’][‘endpoint’][‘scope’][‘token’]

correlation_token = request[‘directive’][‘header’][‘correlationToken’]

color_value = request[‘directive’][‘payload’][‘color’] message = {

‘endpointId’:endpoint_id, ‘namespace’:’Alexa.ColorController’, ‘name’:’color’, ‘value’:color_value

} if send_device_state_message(message): response = AlexaResponse({

‘correlationToken’: correlation_token, ‘token’: token, ‘endpointId’: endpoint_id

}) response.add_context_property({

‘namespace’:’Alexa.ColorController’, ‘name’: ‘color’, ● 221

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

‘value’: color_value }) response = response.get() else: response = AlexaResponse({

‘name’: ‘ErrorResponse’, ‘payload’: { ‘type’: ‘ENDPOINT_UNREACHABLE’, ‘message’: ‘Unable to set endpoint state.’ }

}).get() return send_response(response)

14.9 Test the function Make sure there are no errors in your code by testing the function with the events. You can do this from the Code or the Test section. From the code section, click the down arrow next to Test, and choose the directive you want to send. They are in software – appendix 14.3 – Event test JSON

Then click Test. The response and logs are displayed.

14.10 Discover your device Link your skill. We did this in chapter 13. Go to alexa.amazon.com, find your skill. Click ENABLE, log in and Click Allow. Your device should be discovered.

Run the code on your Pi.

● 222

Chapter 14 ● Controlling a smart home raspberry Pi with SQS

You can now test your smart home skill from the developer console. You don’t need a launch phrase.

14.11 Test from an Alexa device Finally you can test the skill from an Alexa device. Just say “Alexa, turn Sense Hat display green”.

14.12 Clean up Delete your SQS queue or change the access policy so that access is only available using an ID and secret key, reference 3.

14.13 Summary In this chapter we discussed Simple Queue Service, SQS, created an SQS queue and saw how we can send and receive SQS messages. On our Raspberry Pi we executed a program that listed available SQS queues and another program to receive messages and control our Sense Hat display. This we controlled by sending messages from AWS SQS console. We moved on to create a Smart Home device and then uploaded and modified code in the lambda function. The lambda function was tested using events and the code and skill were linked. Finally we discovered our smart home device and tested it from the developer console and finally from an Alexa device.

● 223

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

14.14 References 1.  https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html 2.  https://github.com/alexa-samples/skill-sample-smarthome-fireplace-python 3.  https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ sqs-basic-examples-of-sqs-policies.html 4. Raspbery Pi Sense HAT https://www.raspberrypi.com/products/sense-hat/ Node.js code at 5.  https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ sqs-getting-started.html

● 224

Chapter 15 ● IoT, Pi and Node-RED

Chapter 15 • IoT, Pi and Node-RED 15.1 Introduction Node-RED is a visual flow-based programming language for devices and IoT. In this chapter, we’ll see how to install and run Node-RED on a Pi, starting with a simple ‘Hello world’ program, expanding that to accept inputs from a Sense Hat and display the temperature. We’ll create an AWS IoT thing and, in a node-RED design, configure an MQTT out node to publish messages. Finally, we’ll send messages from our IoT to our Pi and display them on a Pi Sense Hat.

15.2 Prerequisites I used a Raspberry Pi 4. Node-RED says “If you are using Raspberry Pi OS, Buster is the currently supported version”: https://nodered.org/docs/getting-started/raspberrypi You can find your version using cat /etc/os-release

If you need, you can also find your nodejs, node and npm versions using -v. Node-RED can be installed from the Pi’s Preferences > Recommended Software, but it’s best to use the node-RED installation at: https://nodered.org/docs/getting-started/raspberrypi

● 225

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

15.3 Installation Node-RED 3.0 is now available to install, which means that Node.js 14.x or later is required. If updating from version 2, follow the instructions at: https://nodered.org/blog/2022/07/14/version-3-0-released#migrating-from-node-red-2x Node-RED says: “We provide a script to install Node.js, npm and Node-RED onto a Raspberry Pi. The script can also be used to upgrade an existing install when a new release is available.” Use the command: bash import).

● 239

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

I deselected the Motion and Joystick events. Save and deploy. Open a new browser tab and enter http://:1880/ui/ The user interface should appear:

For more information, see the GitHub repo at: https://github.com/node-red/node-red-dashboard

● 240

Chapter 15 ● IoT, Pi and Node-RED

15.10 Sense Hat output We can display messages on the Sense HAT screen. Start a new node and drag an inject node and Sense HAT output on to the canvas. Connect them and add a debug node if you want:

Set up the inject node as we did in Hello World previously. I gave my Sense Hat a name too, but there’s nothing else to do. Save and deploy. Inject a signal and watch your Sense Hat display ‘Hello World!”.

Of course, you can see it in the debug window too:

● 241

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

15.11 IoT - Receiving MQTT messages Node-RED provides MQTT input (subscribe) and output (publish) nodes for IoT communication.

To communicate with AWS IoT we need certificates. We spoke about this previously in Chapter 12.

15.12 Create a new IoT thing for MQTT communication. Have a look at Chapter 12 again, but briefly the procedure is: choose AWS > IoT core > Manage > Things > Create things > create single thing. Thing name > NodeREDthing Ignore additional types and leave ‘default No shadow’. Next screen: choose ‘Auto-generate a new certificate (recommended)’. In Policy, click ‘Create Policy’ (name: NodeREDPolicy). For Actions choose ‘All AWS IoT actions’, and ‘Policy Resource’ to *.

In the next screen, add the policy to the thing and create thing. Download your certificates.

● 242

Chapter 15 ● IoT, Pi and Node-RED

You should now have the thing.

● 243

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

We will need the IoT endpoint, so click IoT > Test > Settings and copy the Endpoint.

15.12.1 Subscribe to a topic

Before we start receiving messages from our Raspberry Pi, choose Test > MQTT test client and ‘Subscribe to a topic’. We saw this before. This time, choose node-red as the topic (you can choose whatever you want, but make sure it’s the same in the node-RED setup).

Now go to your Node-RED application.

15.13 Node-RED IoT Application Start node-RED. We’ll start by sending an MQTT message to AWS IoT. Connect an inject node to an MQTT out node.

Leave the inject node alone – it will just inject a timestamp. But we need to configure the MQTT node. Find Properties.

● 244

Chapter 15 ● IoT, Pi and Node-RED

Give the Server a name (it lets you choose the server if you have more than one). Add a topic name (we will subscribe to this in AWS IoT) - node-red. I’ve also renamed the node.

Now click the pencil

to edit the connection and add the certificates.

Enter the AWS IoT server name you saved previously. Change the port to 8883. Select TLS and click the pencil to add the TLS security.

● 245

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

For the certificate, find the file ending: certificate.pem.crt For the Private Key, use the file ending: private.pem.key For the CA certificate, use AmazonRootCA1.pem Leave Verify Server Certificate checked, but the other fields blank. Name it (AWSIoT) and click Add Click Done and Deploy. Your MQTT node should say connected.

Check that you are receiving messages on your AWS IoT MQTT test, and inject a timestamp from your node-RED flow. You should see a timestamp appear on your AWS IoT.

You can change the inject node to inject a string message, which is received with an error:

● 246

Chapter 15 ● IoT, Pi and Node-RED

To correct this click ‘Edit’ and change the payload display.

Well done — you’ve sent a payload message from Node-RED to AWS IoT.

15.14 Receiving MQTT messages Let’s receive some messages by using an MQTT-in node on node-RED. Connect an MQTT-in node to a debug node.

The properties of the MQTT node remembers the AWS server we set up previously. I’ve selected ‘Quality of Service’ as 0 (AWS IoT only supports levels 0 and 1).

Subscribe to the node-red topic as we did before. Add node-red as the topic and click

● 247

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

done and deploy. Your MQTT in should connect.

Return to the AWS IoT page and select ‘Publish to a topic’ (the topic name should be remembered as node-red), type something into the message payload box and click ‘Publish’.

Return to your node-RED screen and see that you have indeed received a message:

Finally connect a Sense HAT output node.

Send another message from AWS IoT and watch your Sense Hat display ‘hello’.

● 248

Chapter 15 ● IoT, Pi and Node-RED

15.15 Summary We have observed how to install Node-RED on a Raspberry Pi and run it from a PC. We have described the Node-RED user interface and seen how to create a simple Node-RED flow that interfaces with a simple LED and switch. This was extended to accept inputs from a Sense Hat and display the temperature. This was also displayed on the node-RED dashboard. IoT MQTT messaging was then covered. We created a new IoT thing from AWS and generated security certificates. In a Node-RED design, we configured an MQTT out node to publish messages and subscribed and received those messages on the AWS IoT thing. Finally, we sent messages from our IoT to our Pi, displaying those messages on the Raspberry Pi Sense Hat.

● 249

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 16 • P  roactive Events – Sending Raspberry Pi Alexa Notifications 16.1 Introduction In this chapter, we will see how we can send a proactive event or notification, from a Raspberry Pi to an Alexa device. For example, you might want to send an hourly weather (tornado!) update or an intruder alert. We’ll use Python, for Node.js see reference 1. This means that the user doesn’t have to open a skill to find out if there’s anyone at the door, or if it’s raining. They just get notified from your skill. The user has to give permission for this. First, we’ll discuss Alexa notifications, then we’ll see how we code this using a Raspberry Pi. Alexa Notifications Amazon determines the types of notification you can use: Event description

Example notification

Event name

Weather alert Sports event update Message reminder Order status update Reservation confirmation Trash collection reminder Media content notification Game invitation

Tornado alert England 5, Wales 1 3 new messages Order sent Dentist confirmed Bin day today Band on Sunday Invitation for cards

AMAZON.WeatherAlert.Activated AMAZON.SportsEvent.Updated AMAZON.MessageAlert.Activated AMAZON.OrderStatus.Updated AMAZON.Occasion.Updated AMAZON.TrashCollectionAlert.Activated AMAZON.MediaContent.Available AMAZON.SocialGameInvite.Available

We’ll use the AMAZON.MessageAlert.Activated notification. This provides a notification like: “You have message(s) from .” For example: You have 3 NEW (OVERDUE, UNREAD, FLAGGED) messages from John.

16.2 The Lambda function Please note, Amazon says: “You must develop your skill using ASK CLI because some features supported in SMAPI are not supported on the Amazon Developer Portal.” Consequently, we’ll be using the ASK CLI and Visual Code. See Chapter 7 for details on setting up the CLI, or Getting started with the AWS CLI. I’m using Visual Studio code (https://code.visualstudio.com/) and ASK (Alexa Skill Kit). Briefly, the commands are: ask new, create an en-GB.json (copy the en-US.json), add en-GB locale to the skill.json, ask deploy, ask dialog, ‘open skill name’.

● 250

Chapter 16 ● Proactive Events – Sending Raspberry Pi Alexa Notifications

Use ask new and answer the questions. I’ve called the skill ‘intruder alert’. C:\Users\john\Documents\Alexa> ask new Please follow the wizard to start your Alexa skill project -> ? Choose the programming language you will use to code your skill: Python ? Choose a method to host your skill’s backend resources: AWS Lambda Host your skill code on AWS Lambda (requires AWS account). ? Choose a template to start with: Hello world Alexa’s hello world skill to send the greetings to the world! ? Please type in your skill name: intruder alert ? Please type in your folder name for the skill project (alphanumeric): intruderalert Project for skill “intruder alert” is successfully created at ..Documents\Alexa\ intruderalert Project initialized with deploy delegate “@ask-cli/lambda-deployer” successfully. Change to the intruderalert folder and type ask deploy. You’ll get a successfully deployed message. If you get the error message: “[Warn]: File ..\lambda\ask-resources.json not exists. If this is a skill project managed by v1 ask-cli, please run ‘ask util upgrade-project’ then try the command again.” You’re in the wrong folder. Move to the folder above the lambda. Or if you get git push messages, you’ve chosen Alexa-Hosted skill, instead of AWS Lambda. Start again. Your skill has been added to the list at developer.amazon.com. You can see it there if you want to. Open up intruder alert in VS code. In skill package > interaction models, change the en-US.json code invocation: “invocationName”: “intruder alert”, e.g., to: {

“interactionModel”: { “languageModel”: { “invocationName”: “hello world”, “intents”: [ ● 251

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

My Alexa devices are registered in the UK, so I add en-GB.json (I copy the en-US one). Add the new locale to the skill.json as follows: “en-US”: { “summary”: “Sample Short Description”, “examplePhrases”: [ “Alexa open hello world”, “hello”, “help” ],

“name”: “intruder alert”, “description”: “Sample Full Description” },

“en-GB”: { “summary”: “UK Short Description”, “examplePhrases”: [ “Alexa open hello world”, “hello”, “help” ],

“name”: “intruder alert”, “description”: “Sample Full Description” } },

Open skill.json and copy the URI “apis”: { “custom”: { “endpoint”: { “uri”: “arn:aws:lambda:us-east-1:5591444306123:function:asksendnotification-default-default-1614362517059” } } },

Still in the skill manifest (skill.json) , under the apis code: “apis”: { … },

● 252

Chapter 16 ● Proactive Events – Sending Raspberry Pi Alexa Notifications

Add the following write notification permissions, and replace the arn:aws:lambda:useast-1:0000000000000:function:sampleSkill with the one you copied. “permissions”: [ {

“name”: “alexa::devices:all:notifications:write”

} ],

“events”: { “publications”: [ {

“eventName”: “AMAZON.MessageAlert.Activated”

} ],

“endpoint”: { “uri”: “arn:aws:lambda:us-east-1:0000000000000:function:sampleSkill” },

“subscriptions”: [ {

“eventName”: “SKILL_PROACTIVE_SUBSCRIPTION_CHANGED”

} ],

“regions”: { “NA”: { “endpoint”: { “uri”: “arn:aws:lambda:us-east-1:0000000000000:function:sampleSkill” } } } },

Don’t forget the ending comma. In the code, edit helloworld.py to add the proactive event handler: • Add the following line of code in the sb = SkillBuilder() section. sb.add_request_handler(LaunchRequestHandler()) sb.add_request_handler(HelloWorldIntentHandler()) sb.add_request_handler(ProactiveEventHandler()) sb.add_request_handler(HelpIntentHandler())

● 253

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

• and add the proactive event handler itself (say after the HelloWorldIntent handler): class ProactiveEventHandler(AbstractRequestHandler):

“””Handler for Hello World Intent.””” def can_handle(self, handler_input): # type: (HandlerInput) -> bool

return ask_utils.is_request_type(“AlexaSkillEvent.

ProactiveSubscriptionChanged”)(handler_input) def handle(self, handler_input): # type: (HandlerInput) -> Response

logger.info(“Proactive event changed userId, api endpoint and

subscription”)

logger.info(ask_utils.request_util.get_user_id(handler_input)) logger.info(handler_input.request_envelope.context.system.api_endpoint) logger.info(handler_input.request_envelope.request.body.subscriptions) return ( handler_input.response_builder .response )

Run ask deploy again. Now we will enable notifications in our Alexa Skill. Your end user will do this via their Alexa skill. Go to Alexa.amazon.com > Settings > Alexa Account > notifications

● 254

Chapter 16 ● Proactive Events – Sending Raspberry Pi Alexa Notifications

The notification should appear. If not, check your locale.

Sign in if necessary and turn notification on.

This triggers the ProactiveEventHandler with AlexaSkillEvent.ProactiveSubscriptionChanged and ‘AMAZON.MessageAlert.Activated’ event. If you open CloudWatch and look at the logs, you’ll see under subscriptions, ‘event_name’: ‘AMAZON.MessageAlert.Activated’

The other logs give us the endpoint and user information: • Endpoint: https://api.eu.amazonalexa.com • User: amzn1.ask.account.AHIBPIPN”… You can use that information to keep a record of the user’s consent to notifications and to send messages to individuals. We’ll just send a multicast message later, so won’t save the user ID.

● 255

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

When the user turns off the subscription, you get the message: logger.info(handler_input.request_envelope.request.body.subscriptions) AttributeError: ‘NoneType’ object has no attribute ‘subscriptions’

You will need to check for that in your code if you are going to add and remove users from a database.

16.3 Send a notification This is done by sending POST requests to the Amazon servers. First, we obtain an access token, then we use that token to send the notification. To request a token, we send a POST message to Amazon with our Client id and Client secret. The Client id and Client secret are obtained from developer console > alexa skill > model > permissions

You can use postman (https://www.postman.com/ ) to get an access token and then send notification. Here’s the response you get when requesting a token:

We send a request to the URI: https://api.amazon.com/auth/O2/token. We saw this before with LWA. Use the header: headers = {

“Content-Type”: “application/json;charset=UTF-8”

}

● 256

Chapter 16 ● Proactive Events – Sending Raspberry Pi Alexa Notifications

And the body (see above) with: token_params = {

“grant_type” : “client_credentials”, “scope”: “alexa::proactive_events”, “client_id”: CLIENT_ID, “client_secret”: CLIENT_SECRET

}

Pi code to obtain an Access Token We write a Python program to do this. You need AWS credentials. I’ve put my AWS credentials in a separate file called credentials.py. Remember get these from developer > skill > models > permissions. We will post to https://api.amazon.com/auth/o2/token using requests (e.g., requests.post ) and will need to import the requests package. For more information see: pypi.org/project/requests/ Install using: python -m pip install requests The credentials.py code simply contains your ClientID and Client secret: key = {

‘CLIENT_ID’ : ‘put your client id here’, ‘CLIENT_SECRET’ : ‘put your client secret here’

}

Add your CLIENT_ID and CLIENT_SECRET.

16.4 Code to get the access token The code sends a POST request to “https://api.amazon.com/auth/o2/token” which responds with an authorization token. import time import datetime import json import requests import credentials # constants

UTC_FORMAT = “%Y-%m-%dT%H:%M:%S.00Z”

TOKEN_URI = “https://api.amazon.com/auth/o2/token” # Token access constants

● 257

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

CLIENT_ID = credentials.key[‘CLIENT_ID’]

CLIENT_SECRET = credentials.key[‘CLIENT_SECRET’] def get_access_token(): token_params = {

“grant_type” : “client_credentials”, “scope”: “alexa::proactive_events”, “client_id”: CLIENT_ID, “client_secret”: CLIENT_SECRET

} token_headers = {

“Content-Type”: “application/json;charset=UTF-8”

} response = requests.post(TOKEN_URI, headers=token_headers, data=json. dumps(token_params), allow_redirects=True) print(“Token response status: “ + format(response.status_code))

print(“Token response body

: “ + format(response.text))

if response.status_code != 200: print(“Error calling LWA”) return None access_token = json.loads(response.text)[“access_token”] return access_token def main():

“”” Request an Authorization token from AMAZON “”” token = get_access_token() print (token) if __name__ == “__main__”: main()

● 258

Chapter 16 ● Proactive Events – Sending Raspberry Pi Alexa Notifications

Here’s the output:

We want a response code of 200. The access token is the all the highlighted part that begins with Atc| The response text is in JSON format: {

“access_token”: “Atc|….. xoFKIQWpM”, “scope”: “alexa::proactive_events”, “token_type”: “bearer”, “expires_in”: 3600

}

It expires in 3600 seconds (1 hour).

16.5 Send the notification We will use the access token we received to POST to the URI: https://api.eu.amazonalexa.com/v1/proactiveEvents/stages/development This time the header includes the token: header = {

“Authorization”: “Bearer {}”.format(token), “Content-Type”: “application/json;charset=UTF-8”

}

The body contains our event(s), These can be unicast (to a single user) or multicast. The payload will contain our event notification, in our case AMAZON.MessageAlert.Activated This is described at https://developer.amazon.com/en-US/docs/alexa/smapi/proactive-events-api.html with the Message alert event in more detail at: https://developer.amazon.com/en-US/docs/alexa/smapi/schemas-for-proactive-events.html#message-alert

● 259

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

We’ll send a body and event payload like this: {

“timestamp”: “2021-03-01T15:17:58Z”, “referenceId”: “someUniqueReferenceNumber” “expiryTime”: “2021-03-02T15:17:58Z”, “event”: { “name”: “AMAZON.MessageAlert.Activated”, “payload”: { “state”: { “status”: “UNREAD”, “freshness”: “NEW” },

“messageGroup”: { “creator”: { “name”: “Johns python program” },

“count”: 1 } } },

“localizedAttributes”: [ {

“locale”: “en-GB”, “providerName”: “Alexa Events Example”, “contentName”: “Some content”

} ],

“relevantAudience”: { “type”: “Multicast”, “payload”: { } } }

If you need to send to a specific user (type: Unicast), you will have to retrieve the userID that you saved when you handled the Proactive Event. To get a timestamp and expiry time, use the code: timestamp

= time.strftime(UTC_FORMAT, time.gmtime(seconds))

seconds +=3600 expiry_time =

# 1 hour for demo time.strftime(UTC_FORMAT, time.gmtime(seconds))

And to get a reference Id, use uuid: reference_id = str(uuid.uuid4())

● 260

Chapter 16 ● Proactive Events – Sending Raspberry Pi Alexa Notifications

The notification will be “You have 1 NEW UNREAD message from John’s python program”. The whole code is in software Appendix 16. And here’s the response when it is executed:

Your Alexa should ping with a notification message. Response 202 is good! You can now implement that into any code that you want to send a notification.

16.6 Summary We have seen what Alexa notifications are and their specific types. We have created a Lambda function using ASK CLI, added a UK locale, added permissions and edited the code to add a proactive event handler. Furthermore, we have seen how we enable notifications from our skill, and enable them. This triggers an event that we can see in CloudWatch. We wrote and executed code to obtain an access token using Client id and Client secret obtained from the developer console. Finally, we examined the code that sent a notification to everyone using a skill.

16.7 References: 1.  https://github.com/alexa/alexa-cookbook/blob/master/feature-demos/ skill-demo-proactive-events/order.js And others that may help: https://developer.amazon.com/en-US/docs/alexa/smapi/proactive-events-api.html https://developer.amazon.com/en-US/docs/alexa/smapi/schemas-for-proactive-events.htm l https://developer.amazon.com/blogs/alexa/post/7822f3ee-1735-4eaa-9aa6-5b8e39953c07/ proactiveeventsapi-launch-announcement https://developer.amazon.com/en-US/docs/alexa/smapi/proactive-events-api.htm

● 261

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 17 • R  aspberry Pi as a Stand-alone Alexa Device 17.1 Introduction In the last chapter, we’ll see how to turn our Raspberry Pi into a stand-alone Alexa device. We’ll install the Alexa Amazon Voice Service (AVS) on your Raspberry Pi

17.2 Raspberry Pi setup Prerequisites. Use the fastest Pi you can find. I used a Raspberry Pi 4 and for the OS, Raspbian GNU/Linux 10 (buster). Ideally, start with a fresh build using Raspberry Pi Imager (https://www.raspberrypi.com/software/ ). If you don’t know how to do that, see https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up Or watch this video: https://www.youtube.com/watch?v=rGygESilg8w If you‘re not starting from a new build, then before you begin, update your Pi software and restart your Pi as follows: • Update the repository packages: • Run the update command:

sudo apt update sudo apt dist-upgrade

Follow any instructions and wait for the Pi to update. • Remove unwanted files: • And restart your device:

sudo apt clean sudo reboot

If you need to know which OS you’re running on your Pi, enter cat /etc/os-release You can find out your cpu info using cat /proc/cpuinfo and your memory with cat / proc/meminfo Attach your microphone and speakers/headphones to your Pi and check that they work. You can use the following commands: arecord -D hw:2,0 -d 5 -f cd test.wav -c 1 sudo arecord --format=S16_LE --duration=5 --rate=64000 --file-type=raw out.raw

You can test the combined microphone and speaker using: arecord --format=S16_LE --rate=16000 | aplay --format=S16_LE --rate=16000

You can set the volume of your record and playback devices using the command: alsamixer Use the up and down keys to alter the settings. You will need a good-quality microphone.

● 262

Chapter 17 ● Raspberry Pi as a Stand-alone Alexa Device

17.3 Procedure You can set up the AVS Device SDK from source1 or script2. If set up from the source, the AVS uses tap-to-talk and consequently needs a screen and keyboard. I used VNC to wireless link to my Pi. We’ll follow the script instructions. Steps: • Register your AVS device with Amazon. • Download and install the AVS SDK. • Run and authorize the sample app. • Use the sample app. 17.3.1 Register your AVS device with Amazon

Three steps: a) Log into the Amazon developer account (see section 2.2). b) Register an AVS built-in device. Log into the developer console. We’ve seen this before, but this time select Alexa Voice Service.

Select ‘Manage your products’ from the next screen.

● 263

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

And then ‘Add new product’.

If you’ve created AVS products before, they are shown here, e.g.,:

In the next screen add the information about your device. The Product name is shown to end users in the Alexa App and the Product ID is used to identify the device during authorization. Choose a Product name and ID (I used Raspberry2 and Raspberry2), for Product type choose Device with an Alexa Built-in and No to companion app, Other for Product category. Enter something for the description (e.g., prototype), and Hands-free for the interaction.

● 264

Chapter 17 ● Raspberry Pi as a Stand-alone Alexa Device

Ignore upload an image and say No to the rest of the questions. Finally click Next. c) Set up your security profile. In the next screen, you can select an existing security profile or create new profile. Click ‘Create a new profile’ and add a Profile name and description.

When you click NEXT, you’re provided with a Security Profile ID. Copy that, e.g.: amzn1.application.8ebda94958345678881551dca2e Select ‘Other devices and platforms’.

● 265

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Click Generate ID and copy the Client ID, e.g.: amzn1.application-oa2-client.049c89d3584a4a4f83b5f23cd4c54321

Click DOWNLOAD to obtain the config.json file that we’ll need for our Pi. This simply contains the clientId and productId: {

“deviceInfo”: { “clientId”: “amzn1.application-oa2-client.049c89d3584…”, “productId”: “Raspberry2” }

}

Agree to the terms and Finish. Your product should be created:

And it should be listed in your products.

● 266

Chapter 17 ● Raspberry Pi as a Stand-alone Alexa Device

If you select your product, you can add capabilities such as timers, reminders, location services, etc.

If you want to enable your security profile for commercial distribution, go to LWA (login with Amazon), choose the security profile you created, add a link to your privacy policy and save. We now have our device registered, and we need to install Alexa Voice Service on the Pi. 17.3.2 Download and install the AVS SDK

Install Alexa Voice Service on the Pi. I ran this on a Raspberry Pi 4 and Raspbian GNU/ Linux 10 (buster). If you haven’t updated your Pi, do so now. Type ‘sudo apt-get update’ followed by ‘sudo apt-get upgrade’ and then ‘sudo reboot’. Transfer the config.json file you downloaded onto your Pi’s /home/pi/ folder. I use WinSCP:

● 267

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Enter the following commands: cd /home/pi/ wget https://raw.githubusercontent.com/alexa/avs-device-sdk/master/tools/ Install/setup.sh wget https://raw.githubusercontent.com/alexa/avs-device-sdk/master/tools/Install/genConfig.sh wget https://raw.githubusercontent.com/alexa/avs-device-sdk/master/tools/Install/pi.sh

Note the Amazon warning:

Currently when you run setup.sh, you get an error message:

Edit setup.sh to replace the git command: git://github.com/alexa/avs-device-sdk.git

with https://github.com/alexa/avs-device-sdk.git

● 268

Chapter 17 ● Raspberry Pi as a Stand-alone Alexa Device

The setup.sh script builds the SDK and installs dependencies to maintain the HTTP connection with AVS, play music, record from the microphone and store data in a persistent database. Run the setup.sh script using your config.json file and a device serial number (DSN) as arguments. For example, for the DSN 1234 use: sudo bash setup.sh config.json [-s 1234] AGREE to the terms Worryingly, Alexa says if the installation process stops, your Pi may have overheated. Unplug the Pi, let it cool down, and start again with a cooling fan. Eventually you should get:

If you have any problems, refer to the troubleshooting page: https://developer.amazon.com/en-GB/docs/alexa/avs-device-sdk/troubleshooting.html 17.3.3 Run and authorize the sample app

From home/pi run the command: sudo bash startsample.sh

You should see:

● 269

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

You should get a message saying your device is not currently authorized and instruct you to enter a code: To authorize, browse to: ‘https://amazon.com/us/code’ and enter the code: {XXXX}

I found this scrolled past too quickly, so pipe the command into ‘less’: sudo bash startsample.sh | less

Or use control-S and control-Q to stop and start scrolling. Browse to the web site, log in to your Amazon developer account, enter the code and select Allow.

● 270

Chapter 17 ● Raspberry Pi as a Stand-alone Alexa Device

Confirm, and you should get:

On the PI, wait for the program to authorize and Alexa is waiting for you.

17.4 Use the sample app You can now use your Pi just like a normal Alexa device. Try saying: “Alexa, what’s the weather” or “Alexa, tell me a joke” or one of many other Alexa commands. Note: Some microphones aren’t receptive enough, so use a good one.

17.5 Summary In this chapter, we saw how to set up a Raspberry Pi as a stand-alone Alexa device. We checked our Pi hardware and created an Alexa Voice Service product by registering our device with AVS and set up a security profile. The config.json file that was created was transferred to the Pi and AVS SDK was installed on our Pi. The sample app was run and we authorized our device by logging on and registering with Amazon. Finally, we used the sample app to run our Raspberry Pi Alexa device.

17.6 References 1. https://developer.amazon.com/en-GB/docs/alexa/avs-device-sdk/raspberry-pi.html 2. https://developer.amazon.com/en-GB/docs/alexa/avs-device-sdk/raspberry-pi-script.html

● 271

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

Chapter 18 • Conclusion 18.1 Conclusion The idea behind this book was to help you write Alexa skills and control your Raspberry Pi from an Alexa device. In doing so, it has covered a lot of ground; from how Alexa Voice Service works which involved request and response; setting up accounts for Alexa and the charges by Amazon; writing a simple Alexa skill, understanding utterances, intents and slots, following that with APL visual designs; S3 storage and DynamoDB database; In-skill purchasing; LWA security; distribution, certification and publishing. Different ways of creating skills were covered (AWS lambda and CLI). In the latter chapters, we moved on to controlling the Raspberry Pi, creating an IoT ‘thing’; introducing Smart Home devices; creating a Raspberry Pi Smart Home device; controlling the Pi with Node-RED; using the Pi to proactively send messages to our Alexa device; finally using our Pi as a stand-alone Alexa device. I hope that the book has provided enough to get you going, that the examples have been clear, and that it has spurred you on with your own designs. Of course, there is still a lot more you can learn, from more detailed S3 storage, dyanamoDB database work, APL designs, and new Alexa Conversations to mention a few. Please look at my GitHub repo at https://github.com/jallwork and my YouTube videos (search for John Allwork) for updates. I hope that you found this book useful and that it has given you the background and confidence to discover more for yourself. I wish you all the best in your designs.

● 272

Index

Index A

P

Alexa-Hosted Skill 19, 23, 27 Alexa Presentation Language - APL 125 APL Commands 150, APL Visual Responses 135 ASK CLI 23, 107, 114 AVS 262, 263, 264 await and async 93

Persistent attributes Pi SQS code Proactive events Progressive Response

C

S

Certification and publishing CloudWatch logs

99, 272 34, 44, 47

D Datasources developer console dialogs Discover your device DynamoDB

127, 129, 132 19, 23, 25 51, 52, 72 210, 222, 19, 23, 28

E Execute Commands directive

150,157

I In-skill purchasing intent interaction model interceptor IoT Thing

Request and response

S3 storage Sense Hat output session attributes Skill request Skill response Slots Smart home devices SQS SQS Queue SSML stand-alone

20, 90, 98

46, 83, 95 241, 248 19, 20, 65 40 42 16, 17, 17 201, 272 214, 216, 217 214 20, 42, 43 262, 271

T 151

U Utterance

18, 57, 58

V Visual Components

L Lambda Login with Amazon LWA

R

Transformers 158,272 17, 18, 20 15, 19, 29 90, 91, 93 184, 225, 242

19, 53, 65 216 250 179, 183

140

15, 16, 19 25, 201, 219 25, 120, 121

M MQTT

184, 187, 188

N Node-RED Node-RED dashboard Node-RED IoT Application Node-RED user interface

225, 226, 227 237, 249 244 227, 249

● 273

books books

books

with Alexa and Raspberry Pi The book is split into two parts: the first part covers creating Alexa skills and the second part, designing Internet of Things and Smart Home devices using a Raspberry Pi. The first chapters describe the process of Alexa communication, opening an Amazon account and creating a skill for free. The operation of an Alexa skill and terminology such as utterances, intents, slots, and conversations are explained. Debugging your code, saving user data between sessions, S3 data storage and Dynamo DB database are discussed. In-skill purchasing, enabling users to buy items for your skill as well as certification and publication is outlined. Creating skills using AWS Lambda and ASK CLI is covered, along with the Visual Studio code editor and local debugging. Also covered is the process of designing skills for visual displays and interactive touch designs using Alexa Presentation Language. The second half of the book starts by creating a Raspberry Pi IoT “thing” to control a robot from your Alexa device. This covers security issues and methods of sending and receiving MQTT messages between an Alexa device and the Raspberry Pi. Creating a smart home device is described including forming a security profile, linking with Amazon, and writing a Lambda function that gets triggered by an Alexa skill. Device discovery and on/off control is demonstrated. Next, readers discover how to control a smart home Raspberry Pi display from an Alexa skill using Simple Queue Service (SQS) messaging to switch the display on and off or change the color. A node-RED design is discussed from the basic user interface right up to configuring MQTT nodes. MQTT messages sent from a user are displayed on a Raspberry Pi. A chapter discusses sending a proactive notification such as a weather alert from a Raspberry Pi to an Alexa device. The book concludes by explaining how to create Raspberry Pi as a stand-alone Alexa device.

SKU20400_COV_Programming Voice-controlled IoT Applications_v02.indd Alle pagina's

John Allwork graduated from Sheffield University where he developed an interest in computers and gained his MSc at UMIST. After two years working for ICL as a design engineer, he returned to UMIST where he graduated with a PhD in ‘Design and Development of Microprocessor Systems’. He worked for several years in technical support and as manager in electronics distribution, working closely with Intel engineers and later designing Inmos Transputer systems. Having taught electronics at Manchester Metropolitan University, he retired in 2011 but retained his interest in electronics and programming. His other occupations consist of traveling, walking, geocaching and spending time on his allotment.

Elektor International Media www.elektor.com

Programming Voice-controlled IoT Applications • Dr John Allwork

Programming Voice-controlled IoT Applications

Programming Voice-controlled IoT Applications with Alexa and Raspberry Pi

tials import creden

” # constants T%H:%M:%S.00Z d % m % Y % “ = h/o2/token” t u a / m o c UTC_FORMAT . n o z a m https://api.a TOKEN_URI = “ s constants # Token acces CLIENT_ID’] ‘ [ y e k . s l a i t n T’] rede ‘CLIENT_SECRE CLIENT_ID = c [ y e k . s l a i t n e = cred CLIENT_SECRET ken():

s_to def get_acces

= { ”, token_params t_credentials n e i l c “ : ” e p “grant_ty ve_events”, i t c a o r p : : a x e “scope”: “al LIENT_ID, C : ” d i _ t n e i l “c CLIENT_SECRET : ” t e r c e s _ t n “clie }

= { set=UTF-8” r a h c ; n o s j / token_headers n o ”: “applicati “Content-Type } oken_headers t = s r e d a e h , I _UR Dr John Allwork ts.post(TOKEN s e u q e r = e s n respo

16-01-2023 11:08