Raspberry Pi Full Stack 9781907920950, 9783895763786, 1907920951

Table of contents :
About the author
Part 1: Getting started with the Raspberry Pi Full Stack
Chapter 1 • What is this book about?
Chapter 2 • A walk-through the Full Stack project
Chapter 3 • Required hardware
Chapter 4 • How to get help
Chapter 5 • The code repository
Part 2: Raspberry Pi, Arduino, and Raspberry Pi Zero W
Chapter 6 • Raspberry Pi vs Arduino high level comparison
Chapter 7 • Need for efficiency: the Raspberry Pi Zero W
Chapter 8 • Need for speed: the Raspberry Pi 4 (and 3)
Part 3: How to set up the operating system
Chapter 9 • Operating systems for the Raspberry Pi
Chapter 10 • What is a 'headless' operating system
Chapter 11 • How to download and install Raspbian
Chapter 12 • How to set up SSH and WiFi in headless mode
Chapter 13 • How to set a hostname
Chapter 14 • Boot into Raspbian for the first time
Chapter 15 • How to set a fixed IP address
Chapter 16 • Basic configuration
Chapter 17 • Working as the 'root' user
Part 4: How to backup and restore your SD card
Chapter 18 • Backup an SD card - MacOS
Chapter 19 • Restore an SD card - MacOS
Chapter 20 • Backup an SD card - Windows
Chapter 21 • Restore an SD card - Windows
Part 5: Pins, GPIOs and how to control them with Python
Chapter 22 • Raspberry Pi pins, roles, and numbers
Chapter 23 • A taste of Python on the Command Line Interpreter
Chapter 24 • Python functions
Chapter 25 • A simple Python program
Chapter 26 • Wire a simple circuit
Chapter 27 • Control an LED with GPIOZERO
Chapter 28 • Control an LED with rpi.gpio
Chapter 29 • Read a button with GPIOZERO
Chapter 30 • Read a button with RPi.GPIO
Chapter 31 • Control an LED with a button
Chapter 32 • Set up the DHT22 sensor with Git
Chapter 33 • Use the DHT22 sensor
Part 6: Set up the Web Application Stack
Chapter 34 • The Web Application Stack
Chapter 35 • The Python Virtual Environment
Chapter 36 • Increase the disk swap file size
Chapter 37 • Set up system Python - preparation
Chapter 38 • Download, compile and install Python 3
Chapter 39 • Set up the app Python virtual environment
Chapter 40 • Set up Nginx
Chapter 41 • Set up Flask
Chapter 42 • A tour of a simple Flask app
Chapter 43 • UWSGI installation
Chapter 44 • Nginx configuration
Chapter 45 • UWSGI configuration
Chapter 46 • UWSGI and Nginx configuration testing
Chapter 47 • Configure systemd to auto-start uwsgi
Part 7: Set up the database
Chapter 48 • Install the SQLIte3 database
Chapter 49 • Hand-on with the SQLite3 CLI
Part 8: Styling with Skeleton
Chapter 50 • Static assets and the Skeleton boilerplate CSS
Chapter 51 • Set up the static assets directory
Chapter 52 • Introducing the Skeleton boilerplate CSS
Chapter 53 • Copying files using SFTP
Chapter 54 • Flask templates
Chapter 55 • Debugging a Flask app
Part 9: Capture and record sensor data
Chapter 56 • Introduction to Part 9
Chapter 57 • Install the DHT library and the rpi-gpio module
Chapter 58 • Display the current sensor values in the browser
Chapter 59 • Create a database to store sensor data
Chapter 60 • Capture sensor data with a Python script
Chapter 61 • Schedule sensor readings with Cron
Chapter 62 • Update the application and template file
Part 10: Implement the date range selection feature
Chapter 63 • Introduction to Part 10
Chapter 64 • Prototype datetime range of records in SQLite CLI
Chapter 65 • Prototype datetime range in the browser
Chapter 66 • URL querystring validation
Chapter 67 • Quick tidying up
Chapter 68 • Use radio buttons for easy timedate range selection
Chapter 69 • Provision the Python script to work with the radio buttons
Part 11: Google Charts and datetime widgets
Chapter 70 • Introduction to Part 11
Chapter 71 • Implement Google Charts
Chapter 72 • Test Google Charts
Chapter 73 • The datetime picker widget
Chapter 74 • Im plement the datetime picker widget
Chapter 75 • Test the datetime picker widget
Part 12: Dealing with time zones
Chapter 76 • Adjust datetimes to local time zone on the client side
Chapter 77 • Introduction to Arrow
Chapter 78 • Implement Arrow
Chapter 79 • Upload timezone changes and test
Chapter 80 • Link the two pages of the application
Part 13: Charting with Plotly
Chapter 81 • What is Plotly and how to install it
Chapter 82 • Try out Plotly on the CLI
Chapter 83 • Implement Plotly support on the client side
Chapter 84 • Add Plotly support on the server side
Chapter 85 • How to debug Javascript
Chapter 86 • Server side debugging example
Part 14: Access your application from the Internet
Chapter 87 • How to access your application from the Internet?
Chapter 88 • Set a static IP address
Chapter 89 • Expose your app to the Internet with port forwarding
Chapter 90 • Create a self-signed certificate for application
Chapter 91 • Edit Nginx configuration to use SSL
Chapter 92 • Test SSL in Firefox, Safari, Chrome
Part 15: Data logging with Google Sheet
Chapter 93 • What is data logging, and why Google Sheet?
Chapter 94 • Set up Google API credentials
Chapter 95 • Set up the Python libraries and Google Sheet
Chapter 96 • Implement of Google Sheet data logging
Part 16: Set up a remote Arduino sensor node with the nRF24
Chapter 97 • Why set up an Arduino remote node?
Chapter 98 • The Arduino node wiring
Chapter 99 • The Arduino node sketch
Chapter 100 • Raspberry Pi and nRF24 wiring
Chapter 101 • The Raspberry Pi nRF24 receiver script
Chapter 102 • How to install the Python nRF24 modules on the Raspberry Pi
Chapter 103 • Test the nRF24 communications
Chapter 104 • Modify the front end of the application to show remote node data
Part 17: If This Then That alerts
Chapter 105 • An introduction to If This Then That
Chapter 106 • Create an IFTTT webhook and applet
Chapter 107 • Add IFTTT code in the application and testing
Chapter 108 • Install the node listener script as an systemd service
Part 18: Wrapping up
Chapter 109 • Make lab_en v_db page update every 10 minutes
Chapter 110 • Recap and what's next
Part 19: Project extension: Text messaging using Twilio
Chapter 111 • What is this project extension all about?
Chapter 112 • An introduction to Twilio
Chapter 113 • Set up a Twilio account
Chapter 114 • Create a useful bash shell script
Chapter 115 • Add Twilio support to Raspberry Pi
Chapter 116 • Install Twilio CLI
Chapter 117 • Create local and public DNS hostnames
Chapter 118 • Create trusted SSL/TLS certificate
Chapter 119 • Send text alert messages
Chapter 120 • Receive text message commands
Index

Citation preview

---

Raspberry Pi Full Stack ● Dr. Peter Dalmaris

LEARN DESIGN SHARE

Raspberri Pi Full Stack UK 200609.indd 3

06-08-20 15:30

●

This is an Elektor Publication. Elektor is the media brand of

Elektor International Media B.V. 78 York Street, London W1H 1DP, UK Phone: (+44) (0)20 7692 8344

●

All rights reserved. No part of this book may be reproduced in any material form, including

photocopying, or storing in any medium by electronic means and whether or not transiently or incidentally to some other sue of this publication, without the written permission of the copyright holder except in accordance with the provisions of the Copyright Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licencing Agency Ltd., 90 Tottenham Court Road, London, England W1P 9HE. Applications for the copyright holder's permission to reproduce any part of the publication should be addressed to the publishers.

●

Declaration

The author and publisher have used their best efforts in ensuring the correctness of the information contained in this book. They do not assume, or hereby disclaim, any liability to any party for any loss or damage caused by errors or omissions in this book, whether such errors or omissions result from negligence, accident or any other cause..

●

British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library Cover designer: Harmen Heida Prepress Production: D-Vision, Julian van den Berg Printed in the Netherlands First Printing: 2020 ISBN: 978-1-907920-95-0 ISBN (PDF): 978-3-89576-378-6 ISBN (epub): 978-3-89576-379-3

Elektor is part of EIM, the world's leading source of essential technical information and electronics products for pro engineers, electronics designers, and the companies seeking to engage them. Each day, our international team develops and delivers high-quality content - via a variety of media channels (including magazines, video, digital media, and social media) in several languages - relating to electronics design and DIY electronics. www.elektormagazine.com

LEARN DESIGN SHARE

Raspberri Pi Full Stack UK 200609.indd 4

06-08-20 15:30

Contents About the author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 How to read this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 The book web page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Did you find an error? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Part 1: Getting started with the Raspberry Pi Full Stack . . . . . . . . . . . . . . . . . . . . 13 Chapter 1 • What is this book about? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Chapter 2 • A walk-through the Full Stack project . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Chapter 3 • Required hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Chapter 4 • How to get help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Chapter 5 • The code repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Part 2: Raspberry Pi, Arduino, and Raspberry Pi Zero W . . . . . . . . . . . . . . . . . . . . 27 Chapter 6 • Raspberry Pi vs Arduino high level comparison . . . . . . . . . . . . . . . . . . . . 28 Chapter 7 • Need for efficiency: the Raspberry Pi Zero W . . . . . . . . . . . . . . . . . . . . . 33 Chapter 8 • Need for speed: the Raspberry Pi 4 (and 3) . . . . . . . . . . . . . . . . . . . . . . 36 Part 3: How to set up the operating system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Chapter 9 • Operating systems for the Raspberry Pi . . . . . . . . . . . . . . . . . . . . . . . . . 39 Chapter 10 • What is a 'headless' operating system . . . . . . . . . . . . . . . . . . . . . . . . . 42 Chapter 11 • How to download and install Raspbian . . . . . . . . . . . . . . . . . . . . . . . . . 45 Chapter 12 • How to set up SSH and WiFi in headless mode . . . . . . . . . . . . . . . . . . . 48 Chapter 13 • How to set a hostname . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Chapter 14 • Boot into Raspbian for the first time . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Chapter 15 • How to set a fixed IP address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Chapter 16 • Basic configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Chapter 17 • Working as the 'root' user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Part 4: How to backup and restore your SD card . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Chapter 18 • Backup an SD card - MacOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Chapter 19 • Restore an SD card - MacOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Chapter 20 • Backup an SD card - Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Chapter 21 • Restore an SD card - Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Part 5: Pins, GPIOs and how to control them with Python . . . . . . . . . . . . . . . . . . . 73 Chapter 22 • Raspberry Pi pins, roles, and numbers . . . . . . . . . . . . . . . . . . . . . . . . . 74

●5

Raspberri Pi Full Stack UK 200609.indd 5

06-08-20 15:30

Raspberry Pi Full Stack Chapter 23 • A taste of Python on the Command Line Interpreter . . . . . . . . . . . . . . . 76 Chapter 24 • Python functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Chapter 25 • A simple Python program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Chapter 26 • Wire a simple circuit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Chapter 27 • Control an LED with GPIOZERO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Chapter 28 • Control an LED with rpi.gpio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Chapter 29 • Read a button with GPIOZERO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Chapter 30 • Read a button with RPi.GPIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Chapter 31 • Control an LED with a button . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Chapter 32 • Set up the DHT22 sensor with Git . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Chapter 33 • Use the DHT22 sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Part 6: Set up the Web Application Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Chapter 34 • The Web Application Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Chapter 35 • The Python Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Chapter 36 • Increase the disk swap file size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 Chapter 37 • Set up system Python - preparation . . . . . . . . . . . . . . . . . . . . . . . . . 124 Chapter 38 • Download, compile and install Python 3 . . . . . . . . . . . . . . . . . . . . . . . 125 Chapter 39 • Set up the app Python virtual environment . . . . . . . . . . . . . . . . . . . . 129 Chapter 40 • Set up Nginx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Chapter 41 • Set up Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Chapter 42 • A tour of a simple Flask app . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Chapter 43 • UWSGI installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Chapter 44 • Nginx configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 Chapter 45 • UWSGI configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Chapter 46 • UWSGI and Nginx configuration testing . . . . . . . . . . . . . . . . . . . . . . . 146 Chapter 47 • Configure systemd to auto-start uwsgi . . . . . . . . . . . . . . . . . . . . . . . 148 Part 7: Set up the database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Chapter 48 • Install the SQLIte3 database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 Chapter 49 • Hand-on with the SQLite3 CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Part 8: Styling with Skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Chapter 50 • Static assets and the Skeleton boilerplate CSS . . . . . . . . . . . . . . . . . . 159 Chapter 51 • Set up the static assets directory . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

●6

Raspberri Pi Full Stack UK 200609.indd 6

06-08-20 15:30

Contents Chapter 52 • Introducing the Skeleton boilerplate CSS . . . . . . . . . . . . . . . . . . . . . . 164 Chapter 53 • Copying files using SFTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 Chapter 54 • Flask templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 Chapter 55 • Debugging a Flask app . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 Part 9: Capture and record sensor data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Chapter 56 • Introduction to Part 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 Chapter 57 • Install the DHT library and the rpi-gpio module . . . . . . . . . . . . . . . . . 181 Chapter 58 • Display the current sensor values in the browser . . . . . . . . . . . . . . . . 184 Chapter 59 • Create a database to store sensor data . . . . . . . . . . . . . . . . . . . . . . . 189 Chapter 60 • Capture sensor data with a Python script . . . . . . . . . . . . . . . . . . . . . . 191 Chapter 61 • Schedule sensor readings with Cron . . . . . . . . . . . . . . . . . . . . . . . . . 194 Chapter 62 • Update the application and template file . . . . . . . . . . . . . . . . . . . . . . 196 Part 10: Implement the date range selection feature . . . . . . . . . . . . . . . . . . . . . . 201 Chapter 63 • Introduction to Part 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Chapter 64 • Prototype datetime range of records in SQLite CLI . . . . . . . . . . . . . . . 204 Chapter 65 • Prototype datetime range in the browser . . . . . . . . . . . . . . . . . . . . . . 206 Chapter 66 • URL querystring validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Chapter 67 • Quick tidying up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Chapter 68 • Use radio buttons for easy timedate range selection . . . . . . . . . . . . . . 218 Chapter 69 • Provision the Python script to work with the radio buttons . . . . . . . . . . 221 Part 11: Google Charts and datetime widgets . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Chapter 70 • Introduction to Part 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Chapter 71 • Implement Google Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Chapter 72 • Test Google Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Chapter 73 • The datetime picker widget . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 Chapter 74 • Implement the datetime picker widget . . . . . . . . . . . . . . . . . . . . . . . . 238 Chapter 75 • Test the datetime picker widget . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Part 12: Dealing with time zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Chapter 76 • Adjust datetimes to local time zone on the client side . . . . . . . . . . . . . 244 Chapter 77 • Introduction to Arrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 Chapter 78 • Implement Arrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Chapter 79 • Upload timezone changes and test . . . . . . . . . . . . . . . . . . . . . . . . . . 254

●7

Raspberri Pi Full Stack UK 200609.indd 7

06-08-20 15:30

Raspberry Pi Full Stack Chapter 80 • Link the two pages of the application . . . . . . . . . . . . . . . . . . . . . . . . . 256 Part 13: Charting with Plotly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Chapter 81 • What is Plotly and how to install it . . . . . . . . . . . . . . . . . . . . . . . . . . 260 Chapter 82 • Try out Plotly on the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 Chapter 83 • Implement Plotly support on the client side . . . . . . . . . . . . . . . . . . . . 267 Chapter 84 • Add Plotly support on the server side . . . . . . . . . . . . . . . . . . . . . . . . 272 Chapter 85 • How to debug Javascript . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 Chapter 86 • Server side debugging example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 Part 14: Access your application from the Internet . . . . . . . . . . . . . . . . . . . . . . . 284 Chapter 87 • How to access your application from the Internet? . . . . . . . . . . . . . . . 285 Chapter 88 • Set a static IP address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Chapter 89 • Expose your app to the Internet with port forwarding . . . . . . . . . . . . . 290 Chapter 90 • Create a self-signed certificate for application . . . . . . . . . . . . . . . . . . . 295 Chapter 91 • Edit Nginx configuration to use SSL . . . . . . . . . . . . . . . . . . . . . . . . . . 300 Chapter 92 • Test SSL in Firefox, Safari, Chrome . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Part 15: Data logging with Google Sheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 Chapter 93 • What is data logging, and why Google Sheet? . . . . . . . . . . . . . . . . . . 308 Chapter 94 • Set up Google API credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Chapter 95 • Set up the Python libraries and Google Sheet . . . . . . . . . . . . . . . . . . . 322 Chapter 96 • Implement of Google Sheet data logging . . . . . . . . . . . . . . . . . . . . . . 328 Part 16: Set up a remote Arduino sensor node with the nRF24 . . . . . . . . . . . . . . 330 Chapter 97 • Why set up an Arduino remote node? . . . . . . . . . . . . . . . . . . . . . . . . 331 Chapter 98 • The Arduino node wiring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 Chapter 99 • The Arduino node sketch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Chapter 100 • Raspberry Pi and nRF24 wiring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Chapter 101 • The Raspberry Pi nRF24 receiver script . . . . . . . . . . . . . . . . . . . . . . 343 Chapter 102 • H  ow to install the Python nRF24 modules on the Raspberry Pi . . . . . . 348 Chapter 103 • Test the nRF24 communications . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 Chapter 104 • M  odify the front end of the application to show remote node data . . . . 355 Part 17: If This Then That alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 Chapter 105 • An introduction to If This Then That . . . . . . . . . . . . . . . . . . . . . . . . 360 Chapter 106 • Create an IFTTT webhook and applet . . . . . . . . . . . . . . . . . . . . . . . . 362

●8

Raspberri Pi Full Stack UK 200609.indd 8

06-08-20 15:30

 Chapter 107 • Add IFTTT code in the application and testing . . . . . . . . . . . . . . . . . . 371 Chapter 108 • Install the node listener script as an systemd service . . . . . . . . . . . . 374 Part 18: Wrapping up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 Chapter 109 • Make lab_env_db page update every 10 minutes . . . . . . . . . . . . . . . 377 Chapter 110 • Recap and what's next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378 Part 19: Project extension: Text messaging using Twilio . . . . . . . . . . . . . . . . . . . 380 Chapter 111 • What is this project extension all about? . . . . . . . . . . . . . . . . . . . . . 381 Chapter 112 • An introduction to Twilio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 Chapter 113 • Set up a Twilio account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385 Chapter 114 • Create a useful bash shell script . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 Chapter 115 • Add Twilio support to Raspberry Pi . . . . . . . . . . . . . . . . . . . . . . . . . 394 Chapter 116 • Install Twilio CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 Chapter 117 • Create local and public DNS hostnames . . . . . . . . . . . . . . . . . . . . . . 400 Chapter 118 • Create trusted SSL/TLS certificate . . . . . . . . . . . . . . . . . . . . . . . . . . 405 Chapter 119 • Send text alert messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 Chapter 120 • Receive text message commands . . . . . . . . . . . . . . . . . . . . . . . . . . 416 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426

●9

Raspberri Pi Full Stack UK 200609.indd 9

06-08-20 15:30

Raspberry Pi Full Stack

About the author Dr. Peter Dalmaris is an educator, electrical engineer, electronics hobbyist, and maker. He is the creator of online video courses on DIY electronics and author of several technical books. Peter is the author of 'Maker Education Revolution', a book about how making is changing the way we learn and teach in the 21st century. As a Chief Tech Explorer since 2013 at Tech Explorations, the company he founded in Sydney, Australia, Peter's mission is to explore technology and help educate the world. Tech Explorations offers educational courses and Bootcamps for electronics hobbyists, STEM students, and STEM teachers. A lifelong learner, Peter's core skill lies in explaining difficult concepts through video and text. With over 15 years of tertiary teaching experience, Peter has developed a simple yet comprehensive teaching style that students from all around the world appreciate. His passion for technology and the world of DIY open-source hardware has been a dominant driver that has guided his personal development and his work through Tech Explorations.

● 10

Raspberri Pi Full Stack UK 200609.indd 10

06-08-20 15:30

How to read this book

How to read this book All examples, descriptions and procedures have been tested on a Raspberry Pi Zero W, running Raspbian Buster. I have also tested the exact same project on a Raspberry Pi 4, Raspberry Pi 3 and Raspberry Pi 1. This book has a web page with resources designed to maximise the value it delivers to you, the reader. Please read about the book web page, what it offers and how to access it in the section 'The book web page', later in this introductory segment. Finally, you may be interested in the video course version of this book. This course contains detailed demonstrations and explanations of the Full Stack project. The video lectures capture techniques and procedures that are just not possible to do in text. Please check in the book web page for updates on this project. Be sure to subscribe to the Tech Explorations email list so I can send you updates.

Requirements To make the most out of this book, you will need a few things. You probably already have them. The list of hardware requirements is available on the Tech Explorations website: https://techexplorations.com/parts/rpifs-parts/ Please source these items before you embark on this project.

● 11

Raspberri Pi Full Stack UK 200609.indd 11

06-08-20 15:30

Raspberry Pi Full Stack

The book web page As a reader of this book, you are entitled access to its online resources. You can access these resources by visiting the book's web page at http://txplo.re/rpifsp. The two available resources are: 1. The book discussion space on the Tech Explorations Community. This is a place where you can ask book-related questions and have a conversation about your projects. I will be spending time in the forum weekly, answering questions and participating in discussions. 2. An errata page. As I correct bugs, I will be posting information about these corrections in this page. Please check this page if you suspect you have found an error. If you have found an error that is not listed on the errata page, please use the error report form in the same page to let me know about it. From time to time, I will be posting additional Raspberry Pi Full Stack resources and updates on this page, so please check regularly. By subscribing to the Tech Explorations email list, you'll be sure to receive my regular book updates and news. The subscription form is in the book page.

Did you find an error? Please let us know. Using any web browser, go to http://txplo.re/rpifsp, and submit a ticket. Please take care to provide enough details in your ticket so I can find the bug. Please remember to include the page number of the PDF version of the book. I'll get it fixed right away.

● 12

Raspberri Pi Full Stack UK 200609.indd 12

06-08-20 15:30

Part 1: Getting started with the Raspberry Pi Full Stack

Part 1: Getting started with the Raspberry Pi Full Stack

● 13

Raspberri Pi Full Stack UK 200609.indd 13

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 1 • What is this book about? Think of this book as your guide to a adventure for determined learners. Yes, an adventure. An adventure of courage, determination, and collaboration. If you choose to stay to path, you will be rewarded with knowledge. And, with my admiration. Because this adventure is not going to be a walk in the park. You will struggle. You will fail. Many times. You will curse me :-) But each time, you will get up, dust yourself off, and continue your journey to complete this full stack application. I will be there to help you at every step of the way. I will guide you and show you the way. But you will have to do the hard walk. You will have to write the code, solder the circuits, test the connections and search the database At the end, you will have more than knowledge: you will feel the sweet sense of accomplishment, and you will know that you are strong enough to conquer the next challenge. The adventure I am talking about is a project, in which your weapon is a Raspberry Pi. Your objective is to create a useful application that runs on the Raspberry Pi and spans the Internet, and your enemy is your lazy self. While most books are filled with micro-projects, this one is the opposite. The whole book is one BIG project.

● 14

Raspberri Pi Full Stack UK 200609.indd 14

06-08-20 15:30

Chapter 1 • What is this book about?

This project brings together multiple technologies: electronics, computers, operating systems, networking, multiple programming languages, a database, Internet of Things platforms, and more. You will learn how to use these technologies to create something that actually works. This will be an application with many moving parts. In terms of its architecture and the specific components that it contains, it is an accurate analog of commercial Internet of Things applications in domains such as Smart Homes, Smart Cities and Manufacturing. The application you will build is scalable to a global scale with a few targeted modifications, and it is a platform on which you can build.

● 15

Raspberri Pi Full Stack UK 200609.indd 15

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 2 • A walk-through the Full Stack project Ok, so you know that this project is going to be awesome. But what exactly is it that you are going to do? What exactly is it that you are going to learn? I'm glad you asked. Let's dive in. The image below depicts the path you will follow through this course (Figure 2.1).

Figure 2.1: The path you will follow through this course. The journey begins with a look at the base hardware, the Raspberry Pi. The application that you will eventually create works well on any Raspberry Pi, however in this book I have concentrated on the Raspberry Pi Zero W (for "Wireless"). The Raspberry Pi Zero represents an excellent balance of price versus performance versus size. I am amazed by how much you can achieve with this low-cost computer. In the video version of this book, I have implemented the exact same application on the Raspberry Pi 4. The implementation of the application only has a couple of small differences between the Raspberry Pi 4 and the Raspberry Pi Zero, which I will highlight when the time comes. In the first section of the book, I will introduce you to the Raspberry Pi, and focus on the Raspberry Pi Zero as our base hardware platform. This base hardware represents the lowest level of the Full Stack application.

● 16

Raspberri Pi Full Stack UK 200609.indd 16

06-08-20 15:30

Chapter 2 • A walk-through the Full Stack project

In the second section of the book, step two of the project, you will learn about the Raspbian operating system. This is the software that makes it possible for us, the application developers, to implement our application using high-level tools like the Python programming language and the SQLite3 database. You will learn that the Raspberry Pi, as a true multi-purpose computer, can work with a wide range of operating systems, including various flavours of Linux and even Microsoft Windows. In this project, you will be using a minimal version of the Raspberry Pi preferred operating system, Raspbian. Because it is only a matter of "when", and not "if" for disaster to strike, in section three of this book I will show you how to make backups of your Raspberry Pi's SD card, and then how to restore from a backup file. Making a mistake during the implementation of a project which makes it hard or even impossible to continue is something that you should expect when you are learning something new. I have been there many times. If you are prepared for such occurrences, with full backups taken at regular intervals, you will be always be able to restore your project to a previous working state and continue from there, instead of having to rebuild from scratch. The real work begins in section four. There, you will learn about the Raspberry Pi's General Purpose Inputs and Outputs (GPIO), and how to use them with simple Python programs. I will explain how to refer to a specific GPIO and show you the basic Python commands you need to perform simple tasks, like turn an LED on and read the state of a button. All big things start with the first simple step, and for this project, the first step is to blink an LED. In section five, you will learn about the application stack and the components and services that are present in each level. This is where you will create the framework of this project, and learn about where in the stack various back-end and front-end technologies are placed such as the Flask micro-framework for building Python web applications, the uWSGI application server, the Nginx high-performance web server, the SQLite3 database server, HTML/CSS/JavaScript/JQuery, and, of course, Internet of Things platforms like Google's various Cloud APIs, IFTTT and Plotly. Yes, you will learn how to use all of these technologies but not in one big brain dump. We'll spread the learning throughout the full project, so you can relax. In section six, you will start building the front end of your application, and give it persistence. Persistence is the ability of an application to store data, and retrieve it later. This is where you will set up the first version of a simple Flask Python-powered web application, and give it a basic (but elegant) user interface using HTML and CSS. For persistence, you will use SQLite3, a simple yet powerful open-source database server. From section seven and onward, you are really getting into the nitty-gritty of the application development process. In section seven, you will create the core of your application, and flesh-out that most important functions: you will implement the sensor integration so you can get readings and store them in the database, and automate measurements so that data-login begins.

● 17

Raspberri Pi Full Stack UK 200609.indd 17

06-08-20 15:30

Raspberry Pi Full Stack

In section eight, you will learn how to work with dates in the database, retrieve a subset of sensor records from the database using the URL query string, and how to integrate graphical widgets in your application's front end to make it easy for the end user to select date and time ranges. You are now into the most rewarding and fast-paced part of the project, where with each chapter you implement new features. In section nine, you will learn how to create charts from the stored data, and display multiple datapoint across selectable date and time ranges graphically. In section ten, you will revisit the date and time topic that you started in Section eight, but this time at the front-end. You will learn how to use JavaScript/JQuery and Python to convert the date and times stored in your application's database into the correct date/time for your timezone. Of course, once you complete the implementation of this feature, the conversion will happen automatically, without any input from the user. In section eleven, you will revisit one more topic, the graphical representation of data, and learn how to create temperature and humidity charts using Plotly. This is an opportunity to learn how to interact with a powerful Cloud service, and increase the value of your application by giving the user the opportunity to process their environment data with Plotly's powerful graphical analysis tools. In section twelve, you will learn how to access your Raspberry Pi Full Stack application from anywhere in the world. By doing so, you will be able to check your application on your phone no matter where you might happen to be in the world. Most people would stop here and call it a day, but not us. There's more to do. In section thirteen, you will learn to record data from your sensors on a Google Sheet using the Google Drive and Sheet APIs. This work will give you a unique insight into how you can integrate a sophisticated Cloud service, like Google Drive and Sheet, including the confusing processes around creating and using its authentication and encryption certificates. In section fourteen you will learn how to expand your Raspberry Pi Full Stack application by connecting (wirelessly) Arduino nodes with their own sensor. With this extension, your Raspberry Pi will be recording data from an arbitrary number of Arduino nodes (as well as from its own sensor), both on its local database, and on Google Sheet. With the newly expanded application from section fourteen, you will jump into section fifteen and learn how to use another popular Cloud service, If This Then That (IFTTT). You will learn how to implement a configurable email alert app that will email you when the temperature or humidity in one of your application nodes exceeds a threshold. Once you know how to do this, you will be able to integrate any IFTTT capability with your application. It is incredible what you can achieve in just two steps (step 1: "if this", step 2: "then that").

● 18

Raspberri Pi Full Stack UK 200609.indd 18

06-08-20 15:30

Chapter 2 • A walk-through the Full Stack project

To conclude this project, in Section sixteen, you will learn how to secure your application with SSL. This is important especially if you have chosen to expose your application to the Internet, as you learned in Section twelve. One you implement the security components and configuration on the web server, you will be able to access your application page with the secure HTTPS protocol instead of the un-encrypted HTTP. There you have it, this is the project you are about to embark on. Ready?

● 19

Raspberri Pi Full Stack UK 200609.indd 19

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 3 • Required hardware This project is hands-on. While you will be spending most of your implementation time on the software side, you will need to assemble the hardware first. Luckily, the hardware requirements are few and simple. Below, you will find the list of hardware components you will need. Please use this list as a quick references, but check out the online list of parts to ensure that you have the latest details about the hardware. You can find the online parts list here: https://techexplorations.com/parts/rpifs-parts/ A question I frequently receive from students of Raspberry Pi Full Stack is "which Raspberry Pi" and "which Arduino" should I use? Here are my guidelines: Which Raspberry Pi? It doesn't really matter. I tested this project on the original Raspberry Pi 1 Model A. In fact, my Raspberry Pi 1 is still running the original code for this project, going strong since 2014. I also tested with the Raspberry Pi 2, 3 and 4 (all, Model B), as well as the Raspberry Pi Zero W. In all cases, the application works flawlessly. Your choice of Raspberry Pi for this project comes down to these considerations: • Do you have a spare Raspberry Pi? • Do you mind waiting a little longer for the software compilation and installation to complete? • Do you prefer not to use the more recent and expensive Raspberry Pi's (especially the Raspberry Pi 4 with 4GB RAM). • Do you prefer a smaller footprint? Once you have completed the application and it is working on your Raspberry Pi, it will be hard to "feel" the performance difference between the various Raspberry Pis, with the exception of the Raspberry Pi 1. The Raspberry Pi 1 is and feels slow to the end user, so I would recommend against it. So, let's exclude the Raspberry Pi 1 from the remainder of this discussion. The application is currently running on a Raspberry Pi 4, 3 and Zero and I can hardly feel the difference. Next, consider the performance of each computer during development. Installing and compiling C code is computationally intensive. To compile the Python interpreter and RF24 drivers you will need anywhere between 30 minutes to one hour.

● 20

Raspberri Pi Full Stack UK 200609.indd 20

06-08-20 15:30

Chapter 3 • Required hardware

This is where the Raspberry Pi 4 beats all other Raspberry Pi's by a long way. It is fast, and this means you will not have time to get coffee or tea during compilation. The Raspberry Pi Zero is the slowest of the modern Raspberry Pi's because of its low clock speed and single core. Still, I didn't find the performance unworkable. I just went to get coffee or worked on my email queue while the Raspberry Pi Zero was "sweating" with the compilation. If you are the kind of person that goes for top raw performance, go for the Raspberry Pi 4. It has so much power that you can easily run the Full Stack application using the full GUI version of the operating system, as well as run other applications on it, like the Workbench Automation Computer application, and more. The Raspberry Pi 4 is super-fast, but it also quickly gets super-hot. This heat tends to effect the temperature sensor which consistently shows a higher temperature on the Raspberry Pi 4, skewing your temperature data by one or two degrees Celsius. If you are concerned by this, and want to ensure more accurate measurements, you should avoid the Raspberry Pi 4, or just ignore the data from its on-board sensor and instead use data from Arduino nodes. If you are not thrilled by the expensive Raspberry Pi 4 and the amount of heat it produces, consider the Raspberry Pi Zero W (Wireless). It is not crazy fast, but is very cheap, fast enough during development, and has a tiny footprint. You can use it with a regular USB power supply (instead of the more expensive and specially-designed power supply for the Raspberry Pi 4). The Raspberry Pi Zero W is my recommendation. As you can see in Figure 3.2, the Raspberry Pi Zero W, with the custom project HAT that contains the sensor, button, LEDs and RF24 transceiver measures around 7 cm in length, 4 cm in height and 5 cm in width.

● 21

Raspberri Pi Full Stack UK 200609.indd 21

06-08-20 15:30

Raspberry Pi Full Stack

Figure 3.2: My Raspberry Pi Zero W with my custom project HAT. It emits minimal heat and costs less than $20. Unless you have specific reasons to use a different model , go for the Zero. Which Arduino? For the Arduino, the guidelines are much simpler. Just use the Arduino Uno or something compatible, or whichever Atmega328-based Arduino board you happen to have. I have tested the circuit and sketch with an Arduino Uno and Arduino Pro Mini boards, and the operation in both cases was flawless. List of hardware For a more complete and up-to-date list of hardware components, please check https:// techexplorations.com/parts/rpifs-parts/. The following is a simplified list that is centered around the Raspberry Pi Zero W. • A Raspberry Pi Zero W • An Arduino Uno or 100% compatible • SanDisk Ultra 8GB (or larger, I use 16GBytes) Class 10 UHS-I MicroSDHC Memory Card with Adapter • 2 x 10KΩ resistors (a pull-up for the button, and a pull-up for the sensor Data pin) • 2 x 330Ω resistors (for the LEDs) • 2 x LED Diodes (choose your preferred colours) • 2 x DHT22 temperature and humidity sensors • 2 x NRF24L01+ 2.4GHz Wireless RF Transceiver • A breadboard-friendly momentary button • Mini breadboard • Jumper wires

● 22

Raspberri Pi Full Stack UK 200609.indd 22

06-08-20 15:30

Chapter 3 • Required hardware

The HAT PCB I designed a tiny HAT that I invite you to use in your project. You can see a photograph this HAT in Figure 3.2. I found using this Raspberry Pi HAT instead of the breadboard made my work with the application faster. I did not have to worry about loose jumper wires, and it also looks great. You can download a copy of its Gerber files here, and order your boards from your preferred manufacturer. Alternatively, you can go to this URL and order your PCBs from my shared project: https://www.pcbway.com/project/shareproject/Raspberry_ Pi_Full_Stack_RF24_and_DHT22_HAT.html If you choose to work with the custom PCB instead of the breadboard, you will not need to source the through-hole components from the list, numbered 4 to 11. Instead consult the components list in my shared project for the SMD components you will need to solder on the PCB.

● 23

Raspberri Pi Full Stack UK 200609.indd 23

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 4 • How to get help At Tech Explorations, we support our students through our community spaces. Raspberry Pi Full Stack has its own space, which you can reach at https://community. techexplorations.com/c/raspberry-pi-full-stack. If you purchased this book directly from the Tech Explorations website, you should have already received an invitation to join our community. Please look for it in your inbox (also check your spam folder). In this email, you will find a link to accept my invitation. Click on the link to accept the invitation and fill in the form to create your free community account. If you purchased this book from one of our partners, you can still join our community, but you will need to complete these steps: Go to our support page at https://txplo.re/support. Click on the support option "Join the Makers Club". In the field "Where did you purchase your course", choose the most appropriate option from the list. In the text box "Comments", indicate that you wish to become a member of the Raspberry Pi Full Stack community space. Important: please copy your book order details. In the field "Email configuration to", include the email address where you would like us to send you the invitation. That's it. Please allow us up to 24 hour to process your request.

● 24

Raspberri Pi Full Stack UK 200609.indd 24

06-08-20 15:30

Chapter 4 • How to get help

Figure 4.3: The Community Space for the Raspberry Pi Full Stack project.

● 25

Raspberri Pi Full Stack UK 200609.indd 25

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 5 • The code repository You can find the source code for this project in the course code repository on Github. Bookmark this URL: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian The repository contains the source code in various stages of development. You can copy this code into your project instead of typing it. This will save you time and frustration that is inevitable with typing. Apart from Python code, the repository also contains my command line sessions and various configuration files. I have tested each and every one of these files and I am confident that they work. When you run into a problem, check the source code at Github so that you can trust that it works, and use it to try and identify the problem in your code. Please note that the code in the repository is "alive". This means that I frequently update it to fix bugs or improve functionality. It is likely that you will find differences between the code I have captured in this book, and the "live" code in the repository.

Figure 5.4: The Raspberry Pi Full Stack project code repository on Github.

● 26

Raspberri Pi Full Stack UK 200609.indd 26

06-08-20 15:30

Part 2: Raspberry Pi, Arduino, and Raspberry Pi Zero W

Part 2: Raspberry Pi, Arduino, and Raspberry Pi Zero W

● 27

Raspberri Pi Full Stack UK 200609.indd 27

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 6 • Raspberry Pi vs Arduino high level comparison For many of us, the Arduino is the technology that introduced us into the world of programmable electronics. It is relatively simple to learn, forgiving of many common beginner mistakes, with a whole universe of documentation, example code, circuits, and projects. We start exploring other technologies once we feel that we have grasped at least the basics of the Arduino. For many of us, the next logical step is the Raspberry Pi. The Raspberry Pi opens up new possibilities, especially when it comes to interfacing our projects with advanced online or computationally intensive features. A question I regularly receive is about the differences and similarities between the Raspberry Pi and the Arduino. I think it is a very good question because once you understand the differences and similarities you will be able to devise ways for the two to work together, in a simple project. The Raspberry Pi and Arduino, or, to be more precise, the Arduino Uno, are complementary technologies. While they share some common characteristics, they are different in almost every aspect that matters. And yet, because of their differences, they are each other's perfect complement. In this chapter, I present an high-level comparison between the Raspberry Pi and the Arduino, based on the table in Figure 6.5.

Figure 6.5: A high-level comparison between the Raspberry Pi and the Arduino Uno. So, here are the 10 most important differences between the Raspberry Pi and Arduino.

● 28

Raspberri Pi Full Stack UK 200609.indd 28

06-08-20 15:30

Chapter 6 • Raspberry Pi vs Arduino high level comparison

Microcomputer vs Microcontroller The Raspberry Pi is a regular computer, similar to the one on your desk. It just comes in a small package. The Raspberry Pi has all the usual features and capabilities that you expect a computer to have. For example: • • • • •

You can connect a large, high-definition monitor, capable of playing 1080p video. You can connect a keyboard, a mouse, and external hard disks. It can connect to the Internet via Ethernet and WiFi. It requires an operating system, and can run your choice of Linux or Windows. For a tiny computer, it has respectable specifications, including a 64-bit quad core CPU with hardware video acceleration and 4GB RAM. • It has multiple USB connectors for peripherals. For the price and size of the Raspberry Pi, these specifications are impressive. On the other hand, the Arduino is a totally different machine. The most important difference between the Arduino and the Raspberry Pi, is that the latter is not a computer. It is a microcontroller. Think of a microcontroller as a very small computer without any of the things that we expect from a desktop or a laptop computer. Take the RAM, for example, and focus on the Arduino Uno. The Arduino Uno has tiny amount of RAM, just 2 kilobytes, and can't to connect to a network like the Internet for example, without additional components. Compare this to the 4 gigabytes of RAM that are available on the Raspberry Pi 4! The Arduino Uno only has 32 kilobytes of flash memory for storing your program, whereas the Raspberry Pi can use very large amounts of storage space on removable SD cards (and you can also connect terabytes of external disk space via USB). Another big difference is performance. The Arduino Uno is powered by an 8-bit, 16Mhz microcontroller that contains a single-core CPU. The Raspberry Pi is powered by a 64-bit quad core ARM CPU, clocked at 1.5GHz. The comparison is far from fair. The Raspberry Pi can do real-time video processing on applications like object recognition and artificial intelligence, while the Arduino Uno would be stretched to it limits trying to identify a loud noise. The Arduino Uno does not use an operating system. Instead, your programs are compiled to run directly on the hardware, and that is why their compiled version is called "firmware". As you can see, the two devices that are totally different. That's because a microcontroller is designed for doing tasks that are small, constantly and repeatedly. A microcomputer is designed as a general-purpose device, with a very large repertoire of possible applications.

● 29

Raspberri Pi Full Stack UK 200609.indd 29

06-08-20 15:30

Raspberry Pi Full Stack

For example, if you want a gadget that measures the temperature and humidity in a room, and then depending on those single two values to turn on a fan or an air conditioner, then something like the Arduino or a microcontroller in general would be the best technology on which you can build such a gadget. If you want something that can recognize a person's face from a video stream and then sent out an alert or turn on an alarm and log that event in an online database, play a movie on Netflix, and check your email, then you will need a Raspberry Pi. The Raspberry Pi is great for general computer tasks, whereas the Arduino is great for things that are small in complexity, small in programming size, and repeat constantly. A great aspect of the Arduino is all of its input and output pins and ability to connect them to actuators and sensors, so things such as relays, switches and motors, then the Arduino is really a good choice, really optimized for sensing and controlling the world around it. Just like the Arduino, the Raspberry Pi does have a bunch of general-purpose input output pins that you can connect to actuators and sensors, just like the Arduino. So there is definitely a good case to be put forward that says that the Raspberry Pi can actually do everything that the Arduino can, and therefore you don't really need an Arduino. But there is a flip side to that: complexity and robustness. The great thing about the Arduino is that it's simple and forgiving. The fact that there is no operating system to worry about means that you don't have to learn how to use one just to turn on a relay. For beginners, I always say that the best way to get started with electronics and with control applications is through the Arduino, gain an understanding of its simple hardware, and build-up skill using the bare minimal infrastructure that a microcontroller provides before moving onto something more complicated. When a new person comes into the world of electronics and control, their objectives tend to be fairly simple; just blinking an LCD very often is a great start. A simple start like that is a significant achievement that can lead to much greater things. You can achieve wins like this very quickly quickly with the Arduino. Programming Another big difference between the Arduino and the Raspberry Pi is programming. The Arduino is programmed in C++. Normally, C++ is not a language that I would recommend to a beginner. It is fairly complicated, and many concepts and techniques that are not intuitive. A language like Python or Ruby is much more suitable as a first programming language. However, the Arduino did not become so popular despite the difficulty in

● 30

Raspberri Pi Full Stack UK 200609.indd 30

06-08-20 15:30

Chapter 6 • Raspberry Pi vs Arduino high level comparison

learning C++. The people that created Arduino were keenly aware of this, so they created a collection of libraries that hide most of the complexity of C++. It makes programming so simple that learners don't realise they are using C++ until well into their learning journey. We often refer to these libraries as the "Arduino language", even though we are really using the hardcore C++ that programmers use to create the Linux operating system. In addition to the language, the Arduino developer team created a very simple development environment that hides all of the complexity of "regular" C++ development environments and compilers that, again, can confuse even those of us with many years experience of using them. In summary, the Arduino is an excellent choice for beginner because it is based on relatively simple hardware, and has a very simple programming interface and environment. On the other hand the Raspberry Pi, being a full microcomputer, has a wide range of languages that you can use to program it. Any language that you can use in Linux is also available for Raspberry Pi. By default it comes with support for C and C++ and Python. You can also install other languages, like Ruby, PHP, JavaScript, Lua, etc. Virtually any programming language is available on the Raspberry Pi. If you're proficient in any language, chances are that you'll be able to use it with Raspberry Pi. Keep in mind that most of the Raspberry Pi documentation is written for Python. Python is, by many accounts, the most widely used high level language in the world. Compared to C++, it has a very short learning curve. Even if you've never used Python, you'll be able to pick it up fairly quickly. Hardware Hardware-wise, the Raspberry Pi is more complicated than the Arduino Uno because of the amount of peripheral hardware that is present on its circuit board. The Raspberry Pi 4 comes with one or two HDMI video output ports, a camera port (so you can do things like face or object recognition), a 1 gigabit Ethernet port, Wi-Fi and Bluetooth, and multiple USB ports (so you can connect your keyboards, mice, external video cameras, and anything else that pretty much has a compatible driver for the Raspberry Pi running linux). The Raspberry Pi is unique among computers because it exposes general-purpose input/ output pins that you can use to interface with other external modules using communications protocols such as I2C, SPI, and UART serial. Sensors motor controllers, LCD displays, relays, and many other devices can be connected directly to the Raspberry Pi via GPIOs.

● 31

Raspberri Pi Full Stack UK 200609.indd 31

06-08-20 15:30

Raspberry Pi Full Stack

In contrast, the Arduino Uno looks like it's lacking in almost every respect. It doesn't have a USB host suitable for things like a keyboard or external storage. It's single USB port allows you to connect your Arduino to your computer for power and serial communication only. However, the Arduino Uno does contain a small number of GPIO pins, that you can use to connect sensor, actuators, displays and other peripherals using the I2C, SPI and UART protocols. Power Finally, I'd like to mention a few things about power consumption. As a computer with a lot more resources and hardware to power, the Raspberry Pi consumes much more current compared than the Arduino. The Raspberry Pi consumes around 2.8 W of power when it's idle and not doing anything useful. It requires more than that if you have connected things such as touch screens and cameras. It requires a power supply with at least 3A at 5V. The Arduino, on the other hand, has very low power consumption needs. This makes it suitable for low-power, battery applications. It can operate with as little as 20 milli amps, for example, versus 700 milli amps for the Raspberry Pi. For a microcontroller, 20 milli amps is not negligible, but with the power management features built into the chip, it can be reduced significantly. What is power management? It is one of the advantages of microcontrollers over regular computers. Microcontrollers are built to be able to programmatically control the power consumption of their various components when not needed in order to reduce power draw. It is possible to program a custom version of an Arduino to operate for many months on a small alkaline battery by turning of its subsystems and "waking" them up when needed. When the job is done, the Arduino can go back to sleep and conserve power. The ability to programmatically control the power consumption of the microcontroller is a very important advantage of the Arduino and is particularly useful when you want to build a gadget that needs to be deployed out in the field, away from power sockets.

● 32

Raspberri Pi Full Stack UK 200609.indd 32

06-08-20 15:30

Chapter 7 • Need for efficiency: the Raspberry Pi Zero W

Chapter 7 • Need for efficiency: the Raspberry Pi Zero W The Raspberry Pi Foundation brought its first computer to the market in 2012. This was the original Raspberry Pi Model B, with a 26-pin GPIO header, 85.60 mm x 56.5 mm in size, which created a precedent for the Raspberry Pis that followed. The foundation has created several newer models since the original board design: the Raspberry Pi 2 (2015), the Raspberry Pi 3 (2016), and the Raspberry Pi 4 (2019). In 2015, the Raspberry Pi Foundation released the Raspberry Pi Zero, which had the same CPU as the original Raspberry Pi, a 46-pi GPIO header, but a much smaller footprint: just 65 × 30 mm. Perhaps the most amazing aspect of the Raspberry Pi Zero was its pricing: just $5. This model was so popular that I remember purchasing unit limits and long delays in deliveries. The Zero was perfect for applications that needed a small footprint, and many creators embedded it into their designs without any modifications. Prior to the Zero, people had to manually remove unwanted components, such as video and USB ports in order to reduce the size of the board. The Raspberry Pi Zero had one major weakness: it had no built-in radios. This meant no WiFi, and or Bluetooth. This was corrected in 2017, with the introduction of the Raspberry Pi Zero W. The Zero W looked the same as and was like the Zero in every aspect: same CPU, same RAM, same GPIO header. But it had one important difference: integrated WiFi and Bluetooth. The price went from $5 to $10, still much cheaper than all other Raspberry Pi's (except for the Zero), and even cheaper then all official Arduinos. For many makers, the Raspberry Pi Zero W was a game changer because it made it possible to create tiny Internet of Things applications, at a small cost.

● 33

Raspberri Pi Full Stack UK 200609.indd 33

06-08-20 15:30

Raspberry Pi Full Stack

Figure 7.6: My Raspberry Pi Zero W, powered by a USB power supply that I found in a drawer. When I created the first edition of the Raspberry Pi Full Stack course, I used an original Raspberry Pi Model B and tested it against the Raspberry Pi 2. I have since created the second edition of the course on a Raspberry Pi 3, and tested it with a Raspberry Pi 4. Working with the Raspberry Pi 4 is amazing because of its speed. It is, however, very expensive, and "fiddly". The Raspberry Pi 4, with a recommended power supply, huge heatsink and SD card, can cost around $150. With the Raspberry Pi Zero W, you don't need a beefed-up and expensive power supply or heatsink. For $10 plus an SD card, you can power it from any USB port on your computer or a regular USB charger, and you can start working. You can see my Raspberry Pi Zero W with the regular USB power supply I use for it in Figure 7.6. Of course, the Raspberry Pi Zero W will not be as fast as the Raspberry Pi 2, 3 or 4. But for the purposes of this project, this is not a problem. Only a few operations during this project can take advantage of high CPU throughput. One is the compilation of the Python interpreter and the C++ source code of the RF24 modules. Another is when the user retrieves a large number of records from a database. For everything else, it is hard to "feel" what kind of Raspberry Pi you are working with.

● 34

Raspberri Pi Full Stack UK 200609.indd 34

06-08-20 15:30

Chapter 7 • Need for efficiency: the Raspberry Pi Zero W

Figure 7.7: My Raspberry Pi Zero W, with the project HAT attached. Another aspect of the Raspberry Pi Zero W that I appreciate, is that it provides an excellent physical framework on which to implement the project hardware. In Figure 7.7, you can see that the HAT I have designed for the project, with all its components populated, fits perfectly over the Raspberry Pi. There is very little wasted space. If you are looking for the most efficient hardware platform on which to build your Raspberry Pi Full Stack application, I recommend the Raspberry Pi Zero W. If you are looking for the highest possible performance specifications on a Raspberry Pi, read the next chapter.

● 35

Raspberri Pi Full Stack UK 200609.indd 35

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 8 • Need for speed: the Raspberry Pi 4 (and 3) The Raspberry Pi 4 is the complete opposite of the Raspberry Pi Zero. It is crazy fast, large, with a full array of ports and connectors. It is a total energy hog. For the price of one Raspberry Pi 4 with 4 GBytes of RAM, you can buy 5x Raspberry Pi Zero W. The Raspberry Pi 4 is a full computer on a single board. It can run a 64-bit operating system, including Windows, can support two high-definition displays, has a true Gigabit Ethernet port, dual-band WiFi, and even supports USB 3 devices (great for connecting external storage). Of course, you can use a Raspberry Pi 4 to complete this project. Because the requirements of the project are minimal, it is true that using a Raspberry Pi 4 can be seen as an overkill. However, it is not if you consider the possibility of using a single Raspberry Pi for multiple projects.

Figure 8.8: My Raspberry Pi 4, with a huge heat sink, running the full-stack application. For example, I have used my Raspberry Pi 4, 4 GB RAM, to implement the full-stack project, but also as a desktop computer for experimenting with Mathematica, and browsing the web. You can see my Raspberry Pi 4, with its large heatsink, running the full stack application in in Figure 8.8. In a similar way, I am using a Raspberry Pi 3 to implement the full stack project as well as a workbench computer. The workbench computer project is one of my most popular projects, in which you use a Raspberry Pi with a TFT screen to build a graphical user interface (using Python) to control bench appliances (like a soldering iron and a fan), as well as take photos

● 36

Raspberri Pi Full Stack UK 200609.indd 36

06-08-20 15:30

Chapter 8 • Need for speed: the Raspberry Pi 4 (and 3)

and videos of my work on the workbench, and record environmental data. You can learn more about this project on the Tech Explorations website1.

Figure 8.9: My Raspberry Pi 3, as a Bench Computer, also running an early version of the Full Stack application. You can see my Raspberry Pi 3 as the Bench Computer, in Figure 8.9. In the front view, you can see the large touch screen which I used to implement a touch-based user interface, programmed in Python. Under the "hood", you can see the Raspberry Pi, connected to a wiring HAT and a relay HAT, plus part of the DHT22 sensor. An important component of the hardware setup that is not visible in these photographs is the external mains power box which is controlled by a relay connected to the Raspberry Pi. If you choose to use a Raspberry Pi 3 or 4 for this project, you will enjoy a significant performance boost compared to the Raspberry Pi Zero W. Because these models are so powerful, you will be able to use the same board for more than one project which then makes this a good choice.

1.

L earn more about the project course "Raspberry Pi: Make a Workbench Automation Computer": https://techexplorations.com/so/raspberry-pi-workbench-computer/

● 37

Raspberri Pi Full Stack UK 200609.indd 37

06-08-20 15:30

Raspberry Pi Full Stack

Part 3: How to set up the operating system

● 38

Raspberri Pi Full Stack UK 200609.indd 38

06-08-20 15:30

Chapter 9 • Operating systems for the Raspberry Pi

Chapter 9 • Operating systems for the Raspberry Pi As you learned in the previous section, one of the most important differences between the Raspberry Pi and Arduino is that the Raspberry Pi (as with any computer) needs an operating system, while the Arduino does not. The operating system is the software that allows the end-user software (such as our Python scripts or C++ programs) to use the hardware resources of the computer. While it is possible to write software that uses these resources directly (usually low-level assembly code), that would be a very tedious and slow job, that requires us to have detailed knowledge of the hardware and its architecture. Only specialist programmers go to this kind of effort and they have good reasons for doing so. For example, if you wanted to write a specialized BIOS (Basic Input/Output System), you would need to write low-level C or assembly code. This code would be executed by the microprocessor to prepare the computer for operation before it initializes the operating system. Anyway, I digress. If you want to learn more about the BIOS, have a look at this article 2. The Raspberry Pi, especially the last two iterations, are powerful enough to be fully capable of using a variety of modern operating systems. You can see several of these Operating Systems in Figure 9.10.

Figure 9.10: Example operating systems that work with the Raspberry Pi.

2.

https://en.wikipedia.org/wiki/BIOS

● 39

Raspberri Pi Full Stack UK 200609.indd 39

06-08-20 15:30

Raspberry Pi Full Stack

These operating systems are (almost) all derivatives of Linux, and tend to emphasize particular features. For example, Ubuntu MATE emphasises a clean and easy to use Linux desktop environment , while PiNet emphasises classroom and education-friendly features such as centralised network accounts for students and shared folders. In this project, you will use the Raspbian operating system that is developed and maintained by the Raspberry Pi Foundation. Raspbian is perfect for the Raspberry Pi because it is tuned and tested for the specific hardware that exists on a Raspberry Pi board. The documentation that comes with it is extensive and specifically written for the Raspberry Pi. With most other operating systems that are compatible with the Raspberry Pi, the documentation is common among different hardware platforms which makes it often hard to find instructions for common things, such as how to configure the build-in WiFi module. Within Raspbian, the Raspberry Pi Foundation offers three variants (Figure 9.11).

Figure 9.11: The Raspbian operating systems has three variants. At the time I am writing these lines, the latest version of Raspbian is codenamed "Buster". Buster comes in three variants: The first variant is the full graphical desktop with all recommended options, such as development modules for writing programs and end-user applications such as web browsers, a Scratch programming editor and compiler, NodeRed, Mathematica, and lots more. This option is loaded with software that you will probably never use, and, for the purposes of this project, is useless. In addition, this variant has a very large download size and you will need a large SD card to install it on. I don't recommend you use this variant.

● 40

Raspberri Pi Full Stack UK 200609.indd 40

06-08-20 15:30

Chapter 9 • Operating systems for the Raspberry Pi

The second variant contains the graphical desktop with only the necessary software options, such as a web browser and a text editor. Nothing fancy. It is half the download size of the fully-loaded variant. If you are working on the project with a Raspberry Pi 3 or 4, then you consider using this variant. I am using this variant to implement the project on my Raspberry Pi 4, and it is working very well with a 16 GByte SD card. The third variant is "Lite". It does not contain the graphical components of the operating system, and none of the software options that require a GUI interface. It is a plain operating system that depends on the command-line interface for interaction with the user. As a result, it is very light on resources, and only a fraction of the download size of the other two variants. If you are using a Raspberry Pi Zero W, this is the recommended operating system. This is the operating system I will be using in this book and I encourage you to do the same. With Raspbian Lite, we have a basic operating system with a very small footprint, so we can use a small, low cost SD card. Another advantage is that it does not contain components that we don't need, including things such as background services that might consume precious resources on our Raspberry Pi Zero W. Also, this means that we don't have to maintain software that we don't need by downloading updates and patches. On the other hand, we will be able to install anything we need, and know exactly what it is. As I mentioned earlier, Raspbian Lite is a base operating system without a graphical user interface. There are no "windows", drop-down menus or pointer. We interact with Raspbian Lite using a keyboard and terminal (no mouse!). This kind of operating system is known as "headless". Want to know more about headless OS? Jump to the next chapter.

● 41

Raspberri Pi Full Stack UK 200609.indd 41

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 10 • What is a 'headless' operating system In this project you will be using Raspbian Lite which, as you know, is a Linux-based operating system with no graphical user interface and only the bare-minimum pre-installed software. In addition, you will set up your Raspberry Pi Zero W without a monitor, keyboard or mouse. The only cable you will connect to it is a mini-USB power cable. This type of setup is known as "headless". A headless computer is one that is configured without typical input and output interfaces (monitor, keyboard, mouse). A user will interact with a headless computer through a network interface and a seperate, client computer. In this project, you will set up your regular work computer with a terminal program, like Putty, iTerm or Windows Console and SSH (Secure SHell) protocol to connect to your Raspberry Pi via your WiFi network. You will issue text commands on the command line, as in the example provided in Figure 10.12.

Figure 10.12: Example of an SSH session using the iTerm terminal emulator. If you have not used a terminal emulator before, or used text commands to interact with a computer, beware that there is a small learning curve. This book will help get you started, and by the end of the project you will be a command line wizard. There are a few more differences between a headless computer system and a GUI-based one that I want to highlight. I have summarised these differences in Table 1.

● 42

Raspberri Pi Full Stack UK 200609.indd 42

06-08-20 15:30

Chapter 10 • What is a 'headless' operating system

Headless

GUI

No monitor, mouse, keyboard.

Requires monitor, mouse, keyboard.

OS occupies much less disk space.

OS is larger to accomodate for the graphical components and applications.

Operated using a seperate computer via a

Can be operated directly, no need for seper-

remote connection.

ate computer.

Best for server or remote applications.

Best for desktop applications.

Faster to backup and update (due to its smaller Longer to backup and update (due to its footprint).

larger footprint).

Improved reliability (fewer things to break).

Acceptable reliability for desktop use.

Improved performance that is memory-bound,

RAM has to be shared with video and

RAM is dedicated to applications.

applications.

User must be familiar with the command line.

User can use point and click interface.

Table 1: The main differences between a headless and GUI computer. Apart from the differences I have already mentioned, the above table also highlights a few other important differences. A headless operating system without GUI components and applications has a very small footprint on the disk. This means maintenance and backup/restore operations are faster simply because there are fewer bytes to work with. In this project, I will show you how to backup and restore your work using an SD card. Backing up an 8GByte SD card (which can easily accomodate Raspbian Lite) takes half the amount of time that you need for the same thing when you use a 16GByte SD card (which is the smallest recommended size for the full Raspbian). Another advantage is reliability. Headless computers typically run minimal operating systems so that the risk software bugs and conflicts between services is minimised. These computers can run for years without a reboot, reliably servicing end user requests. This is impossible to ask from a full desktop operating system. Even the best of these must be restarted regularly. In this project, you will create an application that operates as an automated service, reliably, non-stop, forever. My Raspberry Pi 1 is still running the original Full Stack application and Raspbian Wheezy (the first version of Raspbian from 2013) from 2014. I admit that I had to restart it a few times due to power outages. The longest uninterrupted stretch was 9 months, which is amazing. Another advantage of headless computers and Lite operating systems is performance. A regular GUI operating systems contains hundreds of background services that are constantly running. These services do things such as checking for mouse and keyboard events, redrawing the screen and handling user notifications. All these things take up valuable CPU

● 43

Raspberri Pi Full Stack UK 200609.indd 43

06-08-20 15:30

Raspberry Pi Full Stack

and memory resources. In a modern computer like a PC with multiple cores, and lots of RAM, this is not a big consideration. After all, these computers are designed for GUI interfaces from the ground up. However, for a Raspberry Pi Zero, with a single processing core and 512MB RAM, useless services quickly become performance bottlenecks. A Lite and headless operation releases resources that we can better use to develop and operate our web application.

● 44

Raspberri Pi Full Stack UK 200609.indd 44

06-08-20 15:30

Chapter 11 • How to download and install Raspbian

Chapter 11 • How to download and install Raspbian Time to roll up your sleeves and get to work. In this first step, you will: • Download a copy of the Raspbian Lite operating system disk image. • Expand the compressed image file. • Copy (sometimes referred to as "burn") this image onto an SD card. I will show you how to do these three items in this chapter. In the next chapter, I'll show you how to configure SSH (for command-line shell access) and WiFi by editing specific files on the SD card, before you boot your Raspberry Pi for the first time. Let's start by downloading the Raspbian Lite operating system disk image. The disk image is a large file that contains a compressed copy of the operating system as it will eventually exist on an SD card. Use your browser, and navigate to https://www.raspberrypi.org/downloads/raspbian/. Click on the "Download ZIP" button under Raspbian Buster Lite (Figure 11.13). As I mentioned earlier, "Buster" is the code name of the current version of the Raspbian operating system at the time I am writing this book.

Figure 11.13: Click on the "Download ZIP" button under Raspbian Buster Lite. When the download finishes, you will have a ZIP file in your Downloads folder, titled "2020-02-13-raspbian-buster-lite.zip", or similar. Extract the contents of the ZIP archive. You can do this by double-clicking on the file (on a Mac). If you are using a Windows computer, right-click on the file and click on "Extract All…", then select the location where you'd like the extracted image file to be placed. You should now have something like the example in Figure 11.14 with the ".img" extension. Also notice the size of this file, 1.85 GB. You will

● 45

Raspberri Pi Full Stack UK 200609.indd 45

06-08-20 15:30

Raspberry Pi Full Stack

need an SD card that is larger than this, with plenty of left-over available space for the Raspberry Pi to use over time. The smallest SD card you should consider is 8GBytes.

Figure 11.14: The Raspbian Lite disk image file. You now have the Raspbian image file ready to be copied to an SD card. There are several ways which you can do this, but the easiest is to your a small utility called "Balena Etcher". This utility is available for Mac, Windows and Linux, and you can download it from https:// www.balena.io/etcher/ (Figure 11.15).

Figure 11.15: Balena Etcher is a simple utility for copying a disk image onto an SD card. Download Etcher and install it on your computer like you do with any other program. Now, start Etcher. There are three steps to the copying process (Figure 11.16): • Select the Raspbian Lite image. • Select the SD card. • Copy the image on to the SD card (or "flash it").

● 46

Raspberri Pi Full Stack UK 200609.indd 46

06-08-20 15:30

Chapter 11 • How to download and install Raspbian

Figure 11.16: There are three steps to preparing the SD card with Raspbian Lite. Click "Select image" and select the image file. In my case, it is "2020-02-13-raspbian-buster-lite.img". Then click "Select target" to choose your SD card. Be aware that anything store on this SD card will be wiped out by the content of the image. I have selected a 16GByte SD card plugged in to my external SD card reader device. Finally, click on Flash to start the copy process. This is going to take several minutes, so be patient. When Etcher finishes, continue to the next chapter to set up SSH and WiFi.

● 47

Raspberri Pi Full Stack UK 200609.indd 47

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 12 • How to set up SSH and WiFi in headless mode Now that you have Raspbian Lite on your SD card, it's time to do some simple configuration. The purpose of this configuration is: • To enable the SSH deamon so that you can access your Raspberry Pi via the SSH protocol and your terminal emulator. • To set up WiFi so that your Raspberry Pi can connect automatically to your local network once you start it for the first time. The second item is very important because the Raspberry Pi Zero W does not have an ethernet network interface. This means that you will not be able to access it without WiFi. Also, because you are working in headless mode, you will not be able to plug in a monitor and a keyboard to set up WiFi directly. Start by inserting your Raspberry Pi SD card into your computer. Use your computer's file browser (Finder in MacOS, Windows Explorer in Windows) to navigate to the disk with name "boot". The Raspberry Pi looks inside this partition when it boots for special files that contain instructions like the ones we are about to create. First, lets enable SSH. Use a text editor and create a new empty file. Save it to the boot disk with the file name "ssh". No extension, and no content. Just an empty file named "ssh" (Figure 12.17).

Figure 12.17: Create an empty text file titled "ssh" in the boot disk to enable SSH. That's it.

● 48

Raspberri Pi Full Stack UK 200609.indd 48

06-08-20 15:30

Chapter 12 • How to set up SSH and WiFi in headless mode

Let's continue with the WiFi configuration. Go to the project repository on Github, and grab a copy of the wpa_supplicant.conf.buster file. It looks like this: country=US ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev update_config=1 network={ ssid="---your-SSID--" psk="---your-wifi-passkey---" }

Use your favourite text editor to create a new text file, and copy the content from the sample supplicant file in it. Take care to type in the correct network name and password in the "ssid" and "psk" fields. Then, save the file to the "boot" disk with the name "wpa_supplicant.conf" (Figure 12.18).

Figure 12.18: The wpa-supplicant.conf file in the boot disk configures the WiFi interface before the first boot. When you have both the "ssh" and "wpa_supplicant.conf" files saved in the boot disk, you can eject it from your computer. In the next chapter, you will use it to boot your Raspberry Pi for the first time.

● 49

Raspberri Pi Full Stack UK 200609.indd 49

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 13 • How to set a hostname Your SD card with Raspbian Lite is ready. You've enabled SSH and configured the WiFi settings. You are eager to use it and start your Raspberry Pi for the first time. But wait. There is one more thing to do before you boot for the first time. In this chapter, I will show you how to define the hostname of your Raspberry Pi, and give it a fixed IP address. Although these two configurations are not essential, they will save you time in the long-term. By giving your Raspberry Pi a hostname, you will be able to access it with an easy to remember name, instead of an IP address. For example, I can access the home page of my application in my browser with this: http://raspberrypi-zero.local/lab_env_db

Instead of this: http://192.168.1.45/lab_env_db

It's much better with the hostname, right? Once you have a hostname, you will be able to identify your Raspberry Pi in your router by name, instead of IP address. This way, you will be able to use your router's administration tools to assign it with a fixed IP address. This is not critical, however it is good practice to give computers that offer services to the network (such as file, web, or printing services) a fixed/permanent IP address. You can certainly skip this step and set your hostname after you boot your Raspberry Pi for the first time. But, because there is no monitor connected to it, you will need to use a network scanner application or information from your router to find out your Raspberry Pi's assigned IP address. Once you know the IP address, you will be able to connect to it via SSH and edit the hostname. This is something I will show you how to do very soon. But here, it is worth looking into the option of setting the hostname before the first boot so that you sort out the hostname early in the process (and get the easy access benefits). At the same time, you will learn about the internal structure of your Raspberry Pi's SD card. Let's start.

● 50

Raspberri Pi Full Stack UK 200609.indd 50

06-08-20 15:30

Chapter 13 • How to set a hostname

At the time I am writing this, there is no reliable way to set the hostname of a Raspberry Pi computer running Raspbian by setting a configuration file in the root partition. I am aware of efforts by members of the community to create a solution for this based on a script, however this solution is not reliable at the moment. Ideally, the Raspberry Pi Foundation will provide a solution similar to the SSH and wpa_supplicant method that you learned about in the previous chapters. But, it is still possible to set a hostname before we boot the Raspberry Pi for the first time. The Raspbian SD card contains two partitions. You already know about the "boot" partition. You modified the contents of this partition in previous chapters to enable the SSH deamon and set the WiFi configuration. The second partition is named "rootfs". This partition is formatted with the extFS4 filesystem. If you are working on a Linux computer, you will be able to access this partition natively, like with any other disk. If you are using a MacOS (as I do), or a Windows computer, you will need to install an extFS4 driver first. I use a utility called "extFS for Mac3". There is a version for Windows4. This utility allows you to mount an extFS3 or extFS4 disk in your file system, and use it as you would with a native disk. An alternative to the paid solution from Paragon is the MacFuse project (https://osxfuse.github.io/) for the MacOS, or the Ext2Fsd Project for Windows (http:// www.ext2fsd.com/). The Ext2Fsd Project seems to not be active though, with the latest release being in November of 2017. Install one of these solutions to enable your computer to work with extFS4 drives, and then insert your Raspbian SD card into your computer. Then, select the rootfs drive partition. In extFS For Mac, it looks like the example in Figure 13.19.

Figure 13.19: To set the hostname, edit the "hostname" file under rootfs/etc. 3. 4.

Linux File System for Mac: https://www.paragon-software.com/home/extfs-mac/ Linux File System for Windows: https://www.paragon-software.com/home/linuxfs-windows/#

● 51

Raspberri Pi Full Stack UK 200609.indd 51

06-08-20 15:30

Raspberry Pi Full Stack

Once you mount the rootfs partition on your computer, browse to rootfs/etc/. Then, open the "hostname" file in a text editor, and type in the name that you'd like to refer to your Raspberry Pi by. My Raspberry Pi hostname is: raspberrypi-zero.local

Save this text file with the single line, close the text editor and eject the rootfs and boot partitions from your computer. You are now ready to boot your Raspberry Pi for the first time, and access it with its new hostname. Let's do that next.

● 52

Raspberri Pi Full Stack UK 200609.indd 52

06-08-20 15:30

Chapter 14 • Boot into Raspbian for the first time

Chapter 14 • Boot into Raspbian for the first time In this chapter you will boot your Raspberry Pi for the first time, and connect to it using SSH. If you have set the hostname, you will use this to access the Raspberry Pi. If you have not, no problem: I will show you how to determine the IP address that your router has assigned to your Raspberry Pi, and use that IP address to access it (again, via SSH). To boot your Raspberry Pi Zero W and operate it, apart from the Raspbian SD card, you will need a USB power supply and a USB cable with a Micro Type B plug at one end, and the regular USB Type A plug at the other end (Figure 14.20).

Figure 14.20: To operate your Raspberry Pi Zero W, apart from the Raspbian SD card, you will need a USB power supply and a cable. Insert the Raspbian SD card into the SD card slot of the Raspberry Pi, and plug the USB power supply into the mains power socket. Then, plug the Micro USB connector into the USB port that is marked "PWR IN". In Figure 14.21 you can see my Raspberry Pi Zero W, powered by USB. The tiny green LED is showing disk activity by blinking irregularly as the computer is booting.

● 53

Raspberri Pi Full Stack UK 200609.indd 53

06-08-20 15:30

Raspberry Pi Full Stack

Figure 14.21: When you apply power to your Raspberry Pi, the green LED will blink irregularly to indicate disk activity. Allow at least two minutes for your Raspberry Pi to boot and connect to your WiFi network. You will know when it is ready: The disk activity indicator (the green LED) stays on. If you set a hostname, as described in the previous chapter, now is time to use it. Start your terminal emulator (such as iTerm or Terminal on MacOS, and Putty or PowerShell on Windows), and issue this command: $ ssh [email protected]

Replace "raspberrypi-zero.local" with the host name of your Raspberry Pi. The default password for user "pi" is "raspberry". Let's break down this instruction: • "ssh" is the protocol you are using for the communication session. • "pi" is the user name for the account you are logging into. • "raspberrypi-zero.local" is the hostname of your Raspberry Pi In Figure 14.22 you can see an example of what to expect when you issue this command.

● 54

Raspberri Pi Full Stack UK 200609.indd 54

06-08-20 15:30

Chapter 14 • Boot into Raspbian for the first time

Figure 14.22: Connect to the Raspberry Pi using SSH and a terminal emulator. Your SSH client will ask you to confirm that you want to connect to the new host because it has never seen its "fingerprint" before. Answer "yes". Your computer will store your Raspberry Pi's fingerprint and will not ask you again. After this, SSH will ask you for the password for user "pi". Type "raspberry", and hit enter. You're in! Before doing anything else, I'd like to show you how to connect to your Raspberry Pi in case you have not set a hostname. In this case, you will need to find out the IP address that your router assigned to your Raspberry Pi. There are two ways to do this: • With the help of a network scanner • By looking in your routers DHCP page Method 1: network scanner You can use a network scanner like "iNet Network Scanner" for MacOS and Advanced IP Scanner for Windows. Either utility will scan your network and make a list of all the hosts that it can find. One of those hosts will be your Raspberry Pi, and the scanner will give you its IP address. In Figure 14.23 you can see that my network scanner has found my Raspberry Pi, and is showing relevant information such as its hostname (which I set earlier), IP address, and MAC address.

● 55

Raspberri Pi Full Stack UK 200609.indd 55

06-08-20 15:30

Raspberry Pi Full Stack

Figure 14.23: My network scanner has found my Raspberry Pi. As you can see, the IP address assigned to my Raspberry Pi is 192.168.111.70. You can use the IP address instead of the hostname to connect via SSH: $ ssh [email protected]

Method 2: Router DHCP table An alternative to the network scanner is to look for the same information in your router's DHCP table. You will need to log in to your router's admin panel, and look for the DHCP page, usually available under a diagnostics menu item. In Figure 14.24 you can see the DHCP table of my router, which gives the IP address of my Raspberry Pi.

● 56

Raspberri Pi Full Stack UK 200609.indd 56

06-08-20 15:30

Chapter 14 • Boot into Raspbian for the first time

Figure 14.24: My router's DHCP table show the IP address of my Raspberry Pi. Use one of these methods to find your Raspberry Pi IP address. Without these, or the hostname, you will not be able to connect to your Raspberry Pi, and therefore will not be able to do the following work. In the next chapter, I will show you how to set a fixed IP address for your Raspberry Pi, which is good practice on any computer that offers services (such as a web server) to the network.

● 57

Raspberri Pi Full Stack UK 200609.indd 57

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 15 • How to set a fixed IP address In addition to setting a hostname, it is good practice to set a fixed IP address to network hosts that provide services to other hosts. This way, a client will be able to use the same IP address for all its requests to the server, instead of first doing a lookup request to the DHCP server. To set a fixed IP address for your Raspberry Pi, you will need to log in to your router's admin panel. Every router has an admin panel with its own "branded" design elements, but in general there is a page titled "DHCP" or "Bind IP" that allows you to edit the configuration of a host's IP address. In Figure 15.25 you can see the relevant page in my router's administration panel. This page is available under the LAN menu item.

Figure 15.25: My router allows me to bind an IP address to a MAC address. To bind an IP address to a MAC address, first select the Raspberry Pi by identifying its hostname from the ARP Table ("1"), then click on the "Add" button ("2"). You may change the IP address to something else, or accept the one that DHCP has already assigned). In the IP Bind List box, you can now see the fixed IP address for your Raspberry Pi ("3").

● 58

Raspberri Pi Full Stack UK 200609.indd 58

06-08-20 15:30

Chapter 16 • Basic configuration

Chapter 16 • Basic configuration All network-related configurations are now complete, so now you can go ahead with some basic and essential configurations of the Raspberry Pi. Start by logging in: $ ssh [email protected]

Or (if you prefer the IP address method): $ ssh [email protected]

Raspbian comes with a configuration utility called "raspbi-config". This utility allows you to easily turn components on or off like the SPI and I2C interfaces, set how much RAM to dedicate to video, and even to turn on overclocking so that your Raspberry Pi can operate faster (I do not recommend doing this). Start raspi-config like this: $ sudo raspi-config

You will see the utility home page from where you can access the subcategories (Figure 16.26):

Figure 16.26: The home screen of the raspi-config utility. You can navigate the utility pages with the arrow keys (up, down), TAB (change to next button), and Return (accept selection) keys.

● 59

Raspberri Pi Full Stack UK 200609.indd 59

06-08-20 15:30

Raspberry Pi Full Stack

These are the changes that you want to make (remember that some options, such as SSH and the hostname, are already enabled): • Under Interfacing Options: - SPI: Enable • Under Advanced Options: - Expand Filesystem: this will ensure that you are using the full available space on the SD card. - Memory Split: Make this 8MB (perhaps 0MB is possible, however I prefer to allow some memory access to the GPU to avoid potential instability). Select "Finish" and allow your Raspberry Pi to reboot. Give it a couple of minutes, and try to log in again to ensure we can continue: $ ssh [email protected]

Great, we are almost done with the operating system setup. In the next chapter, I'll show you how to set up the root user. By default, the root user is disabled, but we need to use it with our SFTP client later in the project.

● 60

Raspberri Pi Full Stack UK 200609.indd 60

06-08-20 15:30

Chapter 17 • Working as the 'root' user

Chapter 17 • Working as the 'root' user By default, the root user is disabled, but we need it for use with our SFTP client later in the project. An SFTP client is a program that allows us to transfer files between two computers using the same secure protocol used in SSH. When we are working on the Raspberry Pi using the terminal emulator and SSH, we can simply switch to the root user with the "sudo" command: $ sudo su

This means that the root (or "super user", hence "su") user is available, but is only accessible via the "sudo" command. It is not enabled for remote access, such as SSH or SFTP. Once you become "super user" with "sudo", you can work as if you were logged in as "root", and access parts of the file system that are only permitted to the root user, or edit files owned by root. To exit the super user and go back to the normal "pi" user, type: $ exit

Let's enable external log in to the root account to make our life a bit easier. First, log in to your Raspberry Pi as "pi": $ ssh [email protected]

Then, change into the super user: pi@raspberrypi-zero:~ $ sudo su

And finally, use the nano text editor to edit the sshd_config file: root@raspberrypi-zero:/home/pi# nano /etc/ssh/sshd_config

Scroll down the sshd_config file to find the PermitRootLogin directive. Its argument should be "yes". Remove the "#" to uncomment it (Figure 17.27).

● 61

Raspberri Pi Full Stack UK 200609.indd 61

06-08-20 15:30

Raspberry Pi Full Stack

Figure 17.27: The sshd_config file. Type Control-X to exit and save the changes in the file . Next, restart the SSH daemon so that the changes become effective: root@raspberrypi-zero:/home/pi# /etc/init.d/ssh restart

You are now almost ready to log in to your Raspberry Pi using the root user . The last thing to do is to set a password for the root user . Do this using the "passwd" command: root@raspberrypi-zero:/home/pi# passwd root New password: Retype new password: passwd: password updated successfully

The utility will ask for the new password for the root user . I use "raspberry", since I have no security concerns . If I was working to create a security-sensitive application, I would only enable the root user during development, and disable it for production . Time to test your "new" root user . Exit super user, then exit the pi user, and then log in to your Raspberry Pi using "root": root@raspberrypi-zero:/home/pi# exit exit pi@raspberrypi-zero:~ $ exit logout Connection to raspberrypi-zero.local closed. Peters-iMac:~ peter$ ssh [email protected] [email protected]'s password: Linux raspberrypi-zero.local 4.19.97+ #1294 Thu Jan 30 13:10:54 GMT 2020 armv6l

● 62

Raspberri Pi Full Stack UK 200609.indd 62

06-08-20 15:30

Chapter 17 • Working as the 'root' user

The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. SSH is enabled and the default password for the 'pi' user has not been changed. This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password. root@raspberrypi-zero:~#

In the command line session above, I have marked in bold the "root" user in my new SSH command. As you can see, I was able to log in to my Raspberry Pi using "root". Good work so far. You can log out from root. Your Raspberry Pi is now ready for development, but there's one more critical set of operations I'd like to show you: backing up and restoring your Raspberry Pi SD card. This will save you countless hours and frustration when things go wrong, and give you a reliable way to recover from disaster.

● 63

Raspberri Pi Full Stack UK 200609.indd 63

06-08-20 15:30

Raspberry Pi Full Stack

Part 4: How to backup and restore your SD card

● 64

Raspberri Pi Full Stack UK 200609.indd 64

06-08-20 15:30

Chapter 18 • Backup an SD card - MacOS

Chapter 18 • Backup an SD card - MacOS Learning how to backup and restore your Raspbian SD card is an essential skill. Any work you do on a computer is at risk of total loss due to hardware failure or software glitch. The only way to mitigate this is to the a simple but effective backup strategy, and to implement it. When you work with a Raspberry Pi, all of your data and software is stored on the SD card. Backing up the contents of the SD card ensures that if you lose your work due to hardware failure or by just doing the wrong thing on the command line will not waste hundreds of hours of work. With access to a recent backup, you can just restore your work and continue where you left off. In this chapter, I'll show you how to create a backup of your SD card. Because taking a backup is not enough on its own if you can't restore it, in the next chapter I will show you how to copy a backup file back to an SD card so you can continue your work. The backup and restoration method I will demonstrate differs between MacOS and Windows. To keep things simple, in this and the next chapter I will demonstrate the process for MacOS. In the last two chapters of this part of the book, I will do the same for Windows 10. You can follow along if you wish. I assume your Raspberry Pi is in operation. Before you backup your Raspbian SD card, you must first shutdown. On the command prompt, type this: pi@raspberrypi-zero:~ $ sudo shutdown -h now

This will shutdown the operating system. It will save all outstanding files and terminate running processes in an orderly manner so that no damage is inflicted on the file system. Wait for at least 30 seconds, and when there no activity on the green LED indicator of the Raspberry Pi, remove the USB power cable, and remove the SD card from the Raspberry Pi. Insert the SD card into your computer's SD card slot. The first thing you need to do is to find out the mount location of the SD card. On the Mac, the easiest way to do this is by using the terminal. Bring up the terminal, and type this command: $ diskutil list

The command will respond with a long list of the various disks and volumes that are connected to your Mac. One of them is the Raspbian SD card. It will look a bit like this:

● 65

Raspberri Pi Full Stack UK 200609.indd 65

06-08-20 15:30

Raspberry Pi Full Stack

/dev/disk6 (external, physical): #:

TYPE NAME

SIZE

IDENTIFIER 0:

FDisk_partition_scheme

1:

Windows_FAT_32 boot

2:

Linux

*15.9 GB

disk6

268.4 MB

disk6s1

2.0 GB

disk6s2

This output gives you all the information you need for the backup. In the first line, you can see that my SD card is mounted at location /dev/disk6. Now you can make the backup. Again, there are several ways to do this, but the easiest one is to use the "dd" utility that is also available via the command line. This is a command that requires super user, and looks like this: $ sudo dd if=/dev/disk6 of=/Volumes/Lacie\ RAID/Raspberry\ Pi\ backups/ RaspbianStretchSystemd_Backup_1_Feb_2018.dmg

You can use the command "man dd" to learn more about the various arguments used, but here I'll give you a summary: • if: The input file. In my case, this is the mount location for my SD card, so I type "/dev/disk6". • of: output file. This is where you would like to store the backup file that "dd" will create. I am directing the backup file to my external RAID drive. Because I am planning to make many backups, I use a very descriptive file name that includes the date and a summary of the contents of the backup. In Figure 18.28 you can see three of my backup files as they appear in my Raspberry Pi backup folder.

Figure 18.28: Three of my Raspbian SD card backups. That's all there is to it. The backup will take a relatively long amount of time because every one of the SD card's 16 billion bytes have to be copied and verified. Be patient and let this process complete. Now that you have your backup, you will need to know how to restore it. This is even easier, as I'll show you in the next chapter.

● 66

Raspberri Pi Full Stack UK 200609.indd 66

06-08-20 15:30

Chapter 19 • Restore an SD card - MacOS

Chapter 19 • Restore an SD card - MacOS In the previous chapter you learned how to make a backup of your Raspbian SD card on MacOS. A backup is not much use if you don't know how to restore it, so in this chapter I'll show you how to do this. Imagine that you are working on your Raspberry Pi Full Stack application and you do something (you can't remember exactly what, but you know you were working as the super user) that wrecks the application. You have spent an hour trying to fix it, but the problem persists. Then you remember, you have a backup! You won't have to start from scratch! Perfect, that's what backups are for. Lets restore the Raspbian SD card to the latest available backup. Start by inserting a new SD card in your computer's SD card slot. Of course, you can use the current Raspbian SD card. Just be aware that whatever happens to be on this SD card will be replaced by the contents of the backup file. Next, start Etcher. Remember Etcher? It's the utility you used to install Raspbian on your SD card a few chapters ago. To restore the SD card, you are working with an image of a file system, just like you did when you installed a fresh copy of Raspbian. Select the backup file with the "img" extension, the destination, and click on Flash (Figure 19.29).

● 67

Raspberri Pi Full Stack UK 200609.indd 67

06-08-20 15:30

Raspberry Pi Full Stack

Figure 19.29: You can restore your Raspbian SD card from a backup file with Etcher. A few minutes later, the restoration will be complete. Eject the SD card from your computer, insert it into the Raspberry Pi, and connect the USB power cable. Log in, and continue your from where you left off. It is important to develop a strong backup habit. Every time you finish a new feature, take a backup. At the end of your work day, take a backup. You don't need to keeps all backups since each one takes several gigabytes of disk space, but at the very least keep the most recent two.

● 68

Raspberri Pi Full Stack UK 200609.indd 68

06-08-20 15:30

Chapter 20 • Backup an SD card - Windows

Chapter 20 • Backup an SD card - Windows Learning how to backup and restore your Raspbian SD card is an essential skill. Any work you do on a computer is in risk of total loss due to hardware failure or software glitch. The only way to mitigate this risk is to employ a simple but effective backup strategy, and implement it. When you work with a Raspberry Pi, all of your data and software is stored on the SD card. Backing up the contents of the SD card ensures that if you lose your work due to hardware failure or just doing the wrong thing on the command line will not waste hundreds of hours of work. With access to a recent backup, you can just restore your work and continue where you left off. In this chapter, I'll show you how to create a backup of your SD card. Because taking a backup is not enough on its own if you can't restore it, in the next chapter I will show you how to copy a backup file back to the SD card so you can continue your work. The backup and restoration method I will demonstrate differ between MacOS and Windows computers. To keep things simple, in this and the next chapter I will demonstrate the process for MacOS computers. In the next chapter of this part of the book, I will do the same for Windows computers. You can follow along if you wish. I assume that your Raspberry Pi is in operation. Before you backup your Raspbian SD card, you must first shutdown. On the command prompt, type this: pi@raspberrypi-zero:~ $ sudo shutdown -h now

This will shutdown the operating system. It will save all outstanding files and terminate the processes running in an orderly manner so that no damage is inflicted on the file system. Wait for at least 30 seconds, and when there no activity on the green LED indicator on the Raspberry Pi, remove the USB power cable, and the SD card from the Raspberry Pi. Insert the SD card into your computer's SD card slot. On Windows 10 (or other version), the easiest way to both backup and restore an SD card is by using the free utility Win32 Disk Imager. This utility uses a Graphical User Interface. You can download it from https://sourceforge.net/projects/win32diskimager/ Install the utility, and start it. In Figure 20.30 you can see the utility in action.

● 69

Raspberri Pi Full Stack UK 200609.indd 69

06-08-20 15:30

Raspberry Pi Full Stack

Figure 20.30: Win32 Disk Imager is a simple, free, utility for backing up and restoring an SD card. First, select the letter of our SD card. In my case, it happens to be "E". Second, select the destination folder and type the name of the backup file. Lastly, click on "Read" to start the backup process. "Read" will read the contents of the SD card and copy it into a new image file. That's all there is to it. The backup will take a relatively long amount of time because every one of the SD card 16 billion bytes have to be copied and verified. Be patient and let the process complete. Now that you have your backup, you will need to know how to restore it. This is even easier, as I'll show you in the next chapter.

● 70

Raspberri Pi Full Stack UK 200609.indd 70

06-08-20 15:30

Chapter 21 • Restore an SD card - Windows

Chapter 21 • Restore an SD card - Windows In the previous chapter you learned how to make a backup of your Raspbian SD card on Windows. A backup is not much use if you don't know how to restore it, so in this chapter I'll show you how it's done. Imagine that you are working on your Raspberry Pi Full Stack application and you do something (you can't remember exactly what, but you know you were working as the super user) that wrecks the application. You have spend an hour trying to fix it, but the problem persists. Then you remember, you have a backup! You won't have to start from scratch! Perfect, that's what backups are for. Lets restore the Raspbian SD card to the latest available backup. Start by inserting a new SD card in your computer's SD card slot. Of course, you can use the current Raspbian SD card. Just be aware that whatever happens to be on this SD card will be replaced by the contents of the backup file. Next, start Etcher (you can also use Win32 Disk Imager, the same utility that you used to create the backup). Select the backup file with the "img" extension, the destination, and click on Flash (Figure 21.31: restore_windows>).

Figure 21.31: You can restore your Raspbian SD card from a backup file with Etcher. A few minutes later, the restoration will be complete. Eject the SD card from your computer, insert it into the Raspberry Pi, and connect the USB power cable. Log in, and continue your work where you left off.

● 71

Raspberri Pi Full Stack UK 200609.indd 71

06-08-20 15:30

Raspberry Pi Full Stack

It is important to develop a strong backup habit. Every time you finish a new feature, make a backup. At the end of your work day, make a backup. You don't need to keep all backups since each one can take up several gigabytes of disk space. At the very least, keep the most recent two.

● 72

Raspberri Pi Full Stack UK 200609.indd 72

06-08-20 15:30

Part 5: Pins, GPIOs and how to control them with Python

Part 5: Pins, GPIOs and how to control them with Python

● 73

Raspberri Pi Full Stack UK 200609.indd 73

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 22 • Raspberry Pi pins, roles, and numbers It is time to dig into your first hands-on experiments. In the next few chapters you will learn how to work with an LED and button using Python. This is the first step towards doing much more interesting things. On the Raspberry Pi, you can connect external devices, like buttons and LEDs to the various pins that are exposed through the 40-pin header. Each of the pins perform specific functions. Most of the pins can perform multiple functions, as you will see later. There are two attributes that are most important when it comes to the Raspberry Pi pins: • • • • • • •

The pin number, which allows you to refer to a pin inside your Python scripts The pin capabilities, so that you know what it is that you can do with a pin. Let's have a look at some examples. Pins 1 and 17 provide 3.3V power. Pins 2 and 4 provide 5V power Pins 6, 9, 14, 25, 30, 34 and 39 are GND. Pin 22 is a general-purpose input/output (GPIO) pin. You can use it to drive an LED or sense the state of a button. • Pin 14 is a GPIO, but also the transmit pin of the UART (serial interface). • Pin 32 is a GPIO, but also a PWM pin. In Figure 22.32 you can see a full map of the Raspberry Pi 40-pin header. The map shows the primary and secondary function of each pin.

Figure 22.32: This is a map of the Raspberry Pi 40-pin header, with information about the primary and secondary role of each pin (Courtesy of https://www.raspberrypi.org/documentation/usage/gpio/).

● 74

Raspberri Pi Full Stack UK 200609.indd 74

06-08-20 15:30

Chapter 22 • Raspberry Pi pins, roles, and numbers

A handy tool to have on your Raspberry Pi is pinout, a component of a library called GPIO Zero. With pinout, you can see a simple header map and basic information about your Raspberry Pi on the command line. It looks like the example in Figure 22.33.

Figure 22.33: Pinout is a handy utility that shows basic information about your Raspberry Pi and a map of its header. Pinout is already installed on the full version of Raspbian, but not Lite. To install pinout on Raspbian Lite, issue this command: $ sudo apt install python3-gpiozero

To show the pin map, type this: $ pinout

As you can see in these two maps, each pin has two numbers. For example, with reference to Figure 22.33>, the second pin from the top in the left column is "GPIO2 (3)". This means that you can refer to this pin in two ways from your Python scripts: • With its GPIO reference number, which is "2". • With its board reference number, which is "3". Similarly, for "GPIO11 (23)", the GPIO number is 11, and board number is 23. In the scripts in this project, we will use the GPIO numbering system unless noted otherwise.

● 75

Raspberri Pi Full Stack UK 200609.indd 75

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 23 • A taste of Python on the Command Line Interpreter Let's play with Python! The Raspberry Pi is a full Linux computer, so you can use virtually any programming language with it . However, Python is the one language that stands out . Python is the language with the widest adoption among Raspberry Pi users . This means that it has (by far) the best and most plentiful documentation, widest range of available libraries that you can use in your programs, and drivers for all sorts of external hardware . For this reason, we will be using Python in this project . In this chapter, you will try out a few simple experiments with Python that do not require external hardware . This is an opportunity to get used to using Python on the command line . If you have never worked with Python before, do not worry . We'll take this one step at a time and gradually build the skills you will need for the project . Let's begin . Log into your Raspberry Pi with the pi user . For me, the log in process looks like this (in bold is the command that I entered): Peters-iMac:~ peter$ ssh [email protected] [email protected]'s password: Linux raspberrypi-zero.local 4.19.97+ #1294 Thu Jan 30 13:10:54 GMT 2020 armv6l The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Tue Mar 24 00:12:09 2020 from 192.168.112.13 SSH is enabled and the default password for the 'pi' user has not been changed. This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password. pi@raspberrypi-zero:~ $

At the time I am writing this, Python is available in two versions . Python 2, and Python 3 . Python 2 is a legacy version for which support is ending . So, we'll be working with Python 3 . You can use Python is two "modes" .

● 76

Raspberri Pi Full Stack UK 200609.indd 76

06-08-20 15:30

Chapter 23 • A taste of Python on the Command Line Interpreter

Firstly, you can write traditional Python programs and then execute them. We'll do that very soon. Secondly, you can write interactive Python programs in the Command Line Interpreter (CLI). The CLI is a utility that evaluates a Python instruction as you issue it. It is a very good way to quickly try out an idea, or learn how to use a function. Let's now have a look at using Python on CLI. On the command line, type this: pi@raspberrypi-zero:~ $ python3 Python 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>

This will invoke the CLI, and present you with the ">>>" prompt. You can type a Python command on the prompt, and the the CLI will evaluate it immediately. Try out these commands (lines that start with ">>>" are where I have entered a command. Lines without the prompt are where Python prints out the result of the command, and the numbers before the ">>>" are there for reference): 1-

>>> 1+1 2

2-

>>> 17/3 5.666666666666667

3-

>>> 17/3 # This is a division 5.666666666666667

4-

>>> division_result = 17/3

5-

>>> division_result 5.666666666666667

6-

>>> print(division_result) 5.666666666666667

7-

>>> round(division_result,2) 5.67

8-

>>> 'Tech Exporations' 'Tech Exporations'

9-

>>> str = 'Tech Exporations' 10-

>>> str

'Tech Exporations' 11- >>> print(str) Tech Exporations 12- >>> print(str + ' ' + 'RPi FS') Tech Exporations RPi FS

● 77

Raspberri Pi Full Stack UK 200609.indd 77

06-08-20 15:30

Raspberry Pi Full Stack

13- >>> 3 * str 'Tech ExporationsTech ExporationsTech Exporations' 14- >>> str[5] 'E' 15- >>> str[5:7] 'Ex' 16- >>> str[0] 'T' >>>

Let's take some time to dissect some of these examples. • In the first two prompts ("1+1" and "17/3"), you did simple arithmetic. You used the "+" and "/" operators to do addition and division. So, you can use Python as a calculator. • In the third prompt I added a comment using the "#" symbol. Any text after the "#" is ignored by Python. • In the fourth prompt, you created a variable with the name "division_result" and stored the result of the division in it. In Python, variables can store numbers, strings, arrays, and all kinds of objects. • In the fifth prompt, you typed in the name of the variable only, and what came back from Python is the value stored in it. • In the sixth prompt you used the "print" function to show the content of the variable. The result is the same as if you have just typed in the name of the variable, and this is specific to how the CLI behaves in these cases. • In the seventh prompt, you used the "round" function to round the result of the division to two decimal points. • In prompt 8, you created a string. Notice that you used single quotes to enclose the string. • In prompt 9, you stored the string in a variable titled "str". • In prompt 10, you retrieved the value stored in "str". • In prompt 11, you did the same by using the "print" function. • In prompt 12, you used the "+" operator to concatenate three smaller strings into a large one. The "+" operator concatenates strings when it is used with strings, and adds numbers when it is used with numbers.

● 78

Raspberri Pi Full Stack UK 200609.indd 78

06-08-20 15:30

Chapter 23 • A taste of Python on the Command Line Interpreter

• In prompt 13, you used the "*" operator to multiply a string by a number. The result is the same string coming out multiple times (3 times, in this example). • In prompts 14, 15, and 16, you use the string as an array of characters. Inside the square brackets, you make reference to the cell of this array from where you want to extract the character. Inside the "str" variable, you stored the string "Tech Explorations". The letter in cell "0" is "T", and in cell "5" is "E". You can also specify cell ranges, so the letters in the range of 5 (inclusive) to 7 (not inclusive) are "Ex". Python, while very powerful, is very intuitive for beginners. You can be productive very quickly, without any formal study of the language. In the next chapter, I will show you the basics of functions. Functions are a programming construct that allows us to bundle functions together, and call them with a single name.

● 79

Raspberri Pi Full Stack UK 200609.indd 79

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 24 • Python functions Unless you are working on a trivial program, you will need a way to bundle multiple lines of code together. Modern programming languages have a handy way to do this: functions (also known as "methods"). A function is a block of code that does something useful. Think of functions as mini programs that are part of a larger program. They are an extremely useful programming building block, and, of course, you will use them a lot in this project. Since we are working with Python and the Command Line Interface, I'd like to show you how to create a simple function in this chapter. Log in to your Raspberry Pi as "pi", and start Python CLI: $ python3

Now, let's create a function. In Python, we use the keyword "def" to define a new function. A function definition needs a name (so we can use this name to call the function from somewhere else in our program), and an optional one or more attributes. An attribute can be a variety of things, like strings, numbers, variables, and arrays. Here's an example Python function, named "str_operation", that has the single argument "strn": def str_operation(strn):

Notice the "def" keyword, followed by the function name, the parameter in parentheses, and the ":" in the end that indicates the end of the function definition. In the following lines you can write the code that makes up the function block. Let's do this on the CLI: >>> def str_operation(strn): ...

The "…" in the CLI shows that Python is aware that we are creating a function. It is waiting for us to write the function code, and is not evaluating anything until we "tell" it our function is complete. So remember, when you see "…", Python is waiting for you to type in an instruction that is part of the function. Let's continue to the second line. Press the space bar three times to move the start of the next instruction towards the right by three spaces. Type this in: print(3 * strn)

● 80

Raspberri Pi Full Stack UK 200609.indd 80

06-08-20 15:30

Chapter 24 • Python functions

Your CLI should look like this: >>> def str_operation(strn): ...

print(3 * strn)

...

So, after the function definition, you have a line of code that multiplies the value stored in the "strn" variable by 3 and prints it to the console. Notice the third line also begins with "…". Python is not evaluating anything yet. It is waiting for your to give it the next instruction. It is important to remember that white space is very important in Python. This is how Python can figure out which instruction belongs to a particular block. If you did not include the three spaces in the second line of this example, Python would assume that the new instruction does not belong to the function. This dependency on white space is a known pain point for Python programmers, and something that you will have to endure. To keep this example super-simple, let's end the function here, with only a single line of code. Your cursor is on line three. Hit Enter one more time. You should see this: >>> def str_operation(strn): ...

print(3 * strn)

... >>>

After the blank line, the CLI shows the familiar ">>>" prompt. Whatever you now type now will be evaluated by python. Your new function exists in memory, and will produce a result on the screen when you call it by name and pass an argument to it. Now, try this: >>> str_operation("Hello ")

You should get this back: >>> str_operation("Hello ") Hello Hello Hello >>>

The string "Hello" has been copied three times in the console. Nice!

● 81

Raspberri Pi Full Stack UK 200609.indd 81

06-08-20 15:30

Raspberry Pi Full Stack

You can modify this function like this: >>> def str_operation(strn): ...

a = 3 * strn

...

print(a)

... >>>

This will evaluate the expression "3 * strn" and store the result in variable "a", then print it to the console. When you call it, the result is exactly the same as in the first version: >>> str_operation("Hello ") Hello Hello Hello >>>

Another version of this example is the following: >>> def str_operation(strn, num): ...

a = num * strn

...

print(a)

... >>>

In this version, I added a new parameter, called "num". In the second line, I replaced the original "3" with "num", so that I can repeat the string stored in "strn" as many times as I like without having to modify the program. I can call this function like this: >>> str_operation("Hello ",4) Hello Hello Hello Hello >>>

Notice that inside the parentheses I provided the string argument, and number argument. In this call, the arguments appear in the same order as the function definition; this is very important to remember as Python expects the arguments to be in the order defined in the definition of the function. As you can see, we use functions to bundle instructions together. We can call a function by its name, and pass any required parameters to it in order to have its instructions executed. But functions can also return values to the caller. Have a look at this variation of the "str_operation" function: def str_operation(strn, num): return num * strn

● 82

Raspberri Pi Full Stack UK 200609.indd 82

06-08-20 15:30

Chapter 24 • Python functions

In this example, the function definition is the same as in the original example. Instead of the function doing the calculation and printing the result to the console, it simply returns the result of the calculation using the "return" keyword. The caller of the function can then decide what to do with the returned value. Here is the example on the CLI: >>> def str_operation(strn, num): ...

return num * strn

... >>> a = str_operation("Hello ", 3) >>> print(a) Hello Hello Hello

In this example, I defined the new version of str_operation to return a value. Then, I call the function with its parameters, and store the returned value in a variable. Finally, I print the value stored in the "a" variable in the console. There is a lot of flexibility in writing functions, using the return function, and how you structure your Python programs in general. For example, if you prefer a multi-step approach, you can write the function like this: def str_operation(strn, num): new_string = num * strn return new_string

Here, your function does the calculation in one step, and the return of the value in another. You could choose a more compact way of coding, like this: def str_operation(strn, num): return num * strn print(str_operation("hello ", 3))

In this example, you condensed four lines of code into 2 (the return line, and the print line), skipping the intermediate steps that involve the variables. My personal preference is to code for clarity. I imagine myself or other people reading my code some time in the future. Will I (or my readers) be able to understand what my code does? Will they be able to improve it, or add new features? This means that usually it is best to be more verbose, with sensible function and variable names, and a logical program structure. From here, functions and programs get more elaborate and complicated, but you should now be familiar with the basic principles.

● 83

Raspberri Pi Full Stack UK 200609.indd 83

06-08-20 15:30

Raspberry Pi Full Stack

Ready for something new? In the next chapter, I will show you how to write a Python script that you can call from the command line, without using the CLI. You will build on the knowledge gained in this chapter.

● 84

Raspberri Pi Full Stack UK 200609.indd 84

06-08-20 15:30

Chapter 25 • A simple Python program

Chapter 25 • A simple Python program Let's convert the program your wrote on the CLI in the last chapter into a proper Python program. At the same time, you will learn the basics of Vim editor, which you will use extensively in this project. You will write this program over multiple iterations. Let's start with the first version. To write it, you will use a powerful editor called "Vim". This editor has been around since the beginning of time (at least the beginning of computer time), and you will find it on every Linux computer. Log in to your Raspberry Pi as "pi". Before you write your first program, you must install the Vim text editor. You can install Vim using the Debian "apt-get" utility, like this: pi@raspberrypi-zero:~ $ sudo apt-get install vim

The "sudo" command will temporarily raise your user level to "root" so that you can install Vim on the system. Once Vim is installed, use it to create a Python program: pi@raspberrypi-zero:~ $ vim example.py

You are creating a new file called "example.py" with the Vim editor. You will see a blank editor window, like the one in Figure 25.34.

Figure 25.34: A blank Vim editor waiting for your program. Vim is extremely powerful when you memorise the few hundred shortcuts. There is no mouse support, and no obvious way of doing anything with it. But you can do everything you need with just two or three keyboard shortcuts. Even my dog can remember two or three commands. The first thing you will want to do is, of course, enter some text.

● 85

Raspberri Pi Full Stack UK 200609.indd 85

06-08-20 15:30

Raspberry Pi Full Stack

To do this type "i". The "i"command will instruct Vim to enter "insert mode" so that you can type text in its buffer. You know that its OK to type text because as soon as you type "i", Vim will update its status to "- - INSERT - -", as you can see in Figure 25.35.

Figure 25.35: Use "i" to change Vim mode to "insert". Now that you are in Insert mode, go ahead and copy the below text into Vim: def str_operation(strn, num): a = num * strn print(a) str_operation('Hello ', 3)

Be careful with the white space, and remember to leave a blank line after the third line of the function. In Vim, while you are in Insert mode, you can use the space bar to leave a white space, the Enter key to create a new line, and Backspace/Delete key to reduce indentation. This program first defines the function, and then calls it. This is exactly what you did earlier in the CLI. When you finish typing, you will have your Python program in the Vim buffer, like in the example below (Figure 25.36):

● 86

Raspberri Pi Full Stack UK 200609.indd 86

06-08-20 15:30

Chapter 25 • A simple Python program

Figure 25.36: Your first Python program in Vim. The program looks ready to execute. First you must save it to disk and exit Vim. As you are still in Insert mode, you need to hit the Esc key once to go into command mode. Your editor should look like the example in Figure 25.37.

Figure 25.37: Hit the ESC key to go into Command Mode. Notice the status line in Vim is empty. Next, type ":wq", as in Figure 25.38.

● 87

Raspberri Pi Full Stack UK 200609.indd 87

06-08-20 15:30

Raspberry Pi Full Stack

Figure 25.38: Type ":wq" to write the buffer to disk, and quit. The ":" prepares Vim to receive an instruction. With "w", you are telling Vim to write the contents of the buffer to the disk. With "q" you are telling it to quit. You now know enough about Vim complete this project. Remember there's much more you can learn to become a Vim Ninja. You should now be back on the command line. Let's run the program. To run a Python program, you must pass it as an argument to the Python interpreter. For us, the interpreter is Python3, so execute your program like this: pi@raspberrypi-zero:~ $ python3 example.py Hello Hello Hello

Easy! What I'd like to do next is to parametrise the program. Instead of the program outputting the exact same thing every time you run it (=boring), how about you make a few small changes so that you can control what it outputs when you invoke it? For example, you could write something like this: pi@raspberrypi-zero:~ $ python3 example.py 'This is AWESOME! ' 5

And your program would output "This is AWESOME!" five times. The way you do this is by providing input to your program from the command line. You program needs a way to take your input from the command line and process it. Luckily, Python has a simple way to do this. There is a build-in library called "sys" which contains an attribute called "argv". Argv is really an array, and you can retrieve command line arguments from it like this:

● 88

Raspberri Pi Full Stack UK 200609.indd 88

06-08-20 15:30

Chapter 25 • A simple Python program

• To grab the first command line argument, use sys.argv[1]. • To grab the second command line argument, use sys.argv[2]. • Etc. Ok, let's modify our program. Again, use Vim to open and edit it: pi@raspberrypi-zero:~ $ vim example.py

This time you will not see a blank buffer, but your program. Remember how to go into Insert mode? That's right, type "i". Use the keyboard cursors to navigate up and down, and use the Enter key to add a new line, and the Delete key to delete text. Add a new line above the function definition and type "import sys". Replace the arguments in the function call of the last line with the calls to sys.argv. There is one more detail to note: sys.argv will return a string for each command line argument. For our multiplication purposes, we want the "num" variable to contain a number. You must explicitly instruct Python to convert the string from the second argument into a number. Python has a keyword for this: "int". Enclose the second sys.argv as a parameter for "int". Here is the completed program: import sys def str_operation(strn, num): a = num * strn print(a) str_operation(sys.argv[1], int(sys.argv[2]))

Your program should now look like this (Figure 25.39):

Figure 25.39: To modify an existing program, go into Insert mode with "i".

● 89

Raspberri Pi Full Stack UK 200609.indd 89

06-08-20 15:30

Raspberry Pi Full Stack

Ready to try it out? Save the buffer and quit Vim (":wq"). Then invoke your program: pi@raspberrypi-zero:~ $ python3 example.py 'This is AWESOME! ' 5 This is AWESOME! This is AWESOME! This is AWESOME! This is AWESOME! This is AWESOME! pi@raspberrypi-zero:~ $

Try different combinations of strings and multipliers, and each time your program will produce something different. Python is an extremely flexible multiple purpose language, while at the same time being friendly to beginners. Are you ready for the next step? How about you learn how to use Python to control an LED? In the next chapter, we will construct a simple circuit.

● 90

Raspberri Pi Full Stack UK 200609.indd 90

06-08-20 15:30

Chapter 26 • Wire a simple circuit

Chapter 26 • Wire a simple circuit You may not feel like it, but you now know enough Python to be able to manipulate simple hardware connected to your Raspberry Pi. In this chapter, you will assemble a simple circuit that contains an LED and button. Once you have your circuit ready, you'll learn how to work with it through using several simple Python scripts. Before you plug anything on to your Raspberry Pi's header pins, you must shut it down and remove the USB power cable. Unlike the Arduino, the Raspberry Pi is much more sensitive to things like static electricity and short circuits. It is easy to damage. By turning the power off, you will protect your Raspberry Pi. On the command line, type this: pi@raspberrypi-zero:~ $ sudo shutdown -h now Connection to raspberrypi-zero.local closed by remote host. Connection to raspberrypi-zero.local closed. Peters-iMac:~ peter$

The shutdown command must be given as a super user, hence the "sudo" infront of it. The "-h" switch instructs Raspbian to "halt", ie. not restart. And "now" will cause the shutdown to take place immediately. You can also schedule the shutdown for some time in the future. Once the green activity LED shows no activity, remove the power USB cable. You can now safely plug external wires on to the header pins. Use the circuit depicted in Figure 26.40 to assemble a simple circuit that contains a button, an LED, and two resistors. Don't worry about the two components on the right side of the schematic (marked J2 and R3); you will implement this part in a later chapter.

● 91

Raspberri Pi Full Stack UK 200609.indd 91

06-08-20 15:30

Raspberry Pi Full Stack

Figure 26.40: This schematic will guide you in the assembly of a simple circuit Start with the LED and its 330 Ω current-limiting resistor. The anode (long pin) of the LED is connected to GPIO 4, which is physical pin 7. For the button, be careful to connect one side to physical pin 1 (which provides 3.3V power), and the other side to physical pin 9 (GND) via a 10 kΩ pull-down resistor5 and to GPIO14 (physical pin 8). You can see my assembled LED-button circuit in Figure 26.41. It isn't easy to copy the circuit from this photograph, so I advise you to work from the schematic in Figure 26.40.

5.

If, in the testing you will do later, you find that the button presses are not reliably detected by the Raspberry Pi, try switching the 10KΩ pull-up resistor for a larger one, such as a 50KΩ resistor. In my testing, the larger pull-up makes the button press detection more reliable.

● 92

Raspberri Pi Full Stack UK 200609.indd 92

06-08-20 15:30

Chapter 26 • Wire a simple circuit

Figure 26.41: This is my assembled LED-button circuit. When you are ready, do a final check to ensure the wiring is correct, and connect the power USB cable. Give your Raspberry Pi a couple of minutes to boot and connect to your WiFi network. Continue on to the next chapter.

● 93

Raspberri Pi Full Stack UK 200609.indd 93

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 27 • Control an LED with GPIOZERO You have an LED connected to GPIO4. How can you make it blink? I'll show you in this chapter. As you can probably guess, there are many ways to achieve the same result when it comes to programming and electronics. I will show you two ways to manipulate the LED. The first one involves using a feature of the GPIO Zero library (https://gpiozero.readthedocs. io/en/stable/) that you installed in the first chapter of this part of the book. Back then, you only used this library to print out a pin map on the console. The library contains a lot of functionality, including simple functions to manipulate an LED. Let's try it out. Log in to your Raspberry Pi as the "pi" user. Then, invoke the CLI for Python 3: pi@raspberrypi-zero:~ $ python3 Python 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>

Next, import the LED module from the gpiozero library: >>> from gpiozero import LED

This gives you access to the specific functions that allow you to manipulate the state of an LED. Before you can turn the LED on, you must define the GPIO to which it is connected. Remember you have connected the anode of the LED to GPIO4. Issue this command: >>> led = LED(4)

What you just did is create an LED object that is connected to GPIO4 and assigned it to the "led" variable. Now, you can call the "on()" and "off()" functions of this object to turn it on and off. Try it: >>> led.on()

Did your LED turn on? It should look like this (Figure 27.42):

● 94

Raspberri Pi Full Stack UK 200609.indd 94

06-08-20 15:30

Chapter 27 • Control an LED with GPIOZERO

Figure 27.42: My LED, on. Can you turn it off? Try: >>> led.off()

Here's the complete CLI session: pi@raspberrypi-zero:~ $ python3 Python 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from gpiozero import LED >>> led = LED(4) >>> led.on() >>> led.off() >>> exit()

You can also achieve the same result by writing a small Python program and executing it with the Python3 interpreter. Create a new text file with Vim, and call it "led_blink_gpiozero.py". Type this code into the buffer: from gpiozero import LED from time import sleep led = LED(4)

● 95

Raspberri Pi Full Stack UK 200609.indd 95

06-08-20 15:30

Raspberry Pi Full Stack

while True: led.on() sleep(1) led.off() sleep(1)

Your Vim should look like this (Figure 27.43):

Figure 27.43: This program will blink an LED on and off in 1 second intervals. Save your program and exit Vim (":wq"). Now, execute it like this: $ python3 led_blink_gpiozero.py

Your LED will blink on and off. In the code for this program, we imported the "time" Python module because it contains the "sleep" function. The "sleep" function is useful when you want to delay the execution of a program for a while. In our case, we used it to keep the LED on or off for 1 second. Feel free to replace "1" with another number if you want to create a different delay. To end the program, use this keyboard shortcut: Ctrl-C. Next, let's do the same thing (blink an LED) using a second method. When you are ready, continue on to the next chapter.

● 96

Raspberri Pi Full Stack UK 200609.indd 96

06-08-20 15:30

Chapter 28 • Control an LED with rpi.gpio

Chapter 28 • Control an LED with rpi.gpio In this chapter I'll show you how to control an LED using another Python library, "rpi.gpio". This is a library we will be using later in the course. You can install it easily using the Python package manager "pip", instead of the "apt-get" Linux installer that you used for GPIOZero. Raspbian comes with Python3, but not pip, so you will need to install it before you can install a pip package. Install pip for Python3 like this: $ sudo apt-get install python3-pip

Accept the prompts, and give it a few minutes to complete. Next, install rpi.gpio like this: $ pip3 install rpi.gpio

This will take another few minutes. You are now ready to play with the LED. Start the CLI for python3: $ python3 Python 3.7.3 (default, Dec 20 2019, 18:57:59) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>

Copy the following program into the CLI: >>> import RPi.GPIO as GPIO >>> GPIO.setmode(GPIO.BCM) >>> led = 4 >>> GPIO.setup(led, GPIO.OUT) >>> GPIO.output(led, GPIO.HIGH) >>> GPIO.output(led, GPIO.LOW)

The instruction "GPIO.setmode(GPIO.BCM)" allows you to set the numbering system that you want to use. I have selected "BCM" in this example so that we can use the GPIO numbering system. The alternative is to use "GPIO.BOARD", in which case you can opt for a physical number. If you use GPIO.BOARD in this example, then you will need to change "led = 4" to "led = 7", since GPIO4 is physical pin 7 on the header. You also need to set the direction of the pin, before you can use it. Because we are working with an LED, on the fourth line of the program you have set the direction to GPIO.OUT, i.e. an output. If this was a button (as you will see in the next chapter), you would set the direction to "GPIO.IN", i.e. an input.

● 97

Raspberri Pi Full Stack UK 200609.indd 97

06-08-20 15:30

Raspberry Pi Full Stack

If you get a complaint about a "channel being already in use", type the following this to clear the GPIO tables and try the program again: >>> GPIO.cleanup()

The rpi.gpio library follows an alternative stylistic path that resembles the way that we control pin state on the Arduino. People that are familiar with Arduino, find rpi.gpio easier to understand. You can see my session in Figure 28.44.

Figure 28.44: My CLI LED blinking session using rpi.gpio. As with GPIOZERO, you can write a proper Python program and invoke it on the command line. While you are at it, add the "time" module to make the LED blink every 1 second. Use Vim to write a new program called "led_blink_rpigpio.py": $ vim led_blink_rpigpio.py

In the buffer, copy this program: import RPi.GPIO as GPIO from time import sleep GPIO.setmode(GPIO.BCM) led = 4 GPIO.setup(led, GPIO.OUT) while True: GPIO.output(led, GPIO.HIGH) sleep(1) GPIO.output(led, GPIO.LOW) sleep(1)

● 98

Raspberri Pi Full Stack UK 200609.indd 98

06-08-20 15:30

Chapter 28 • Control an LED with rpi.gpio

Your Vim buffer should look like this (Figure 28.45):

Figure 28.45: My CLI LED blinking program using rpi.gpio. Run your new program with Python3: $ python3 led_blink_rpigpio.py

You will probably receive a warning about the "channel being already in use". This is just a warning, not an error. You can suppress it by inserting the below code in your program, just below the two import statements: GPIO.setwarnings(False)

You are now familiar with two ways to control the state of an LED, or any other on/off device connected to a GPIO. In the next chapter, you will learn how to read the state of a button.

● 99

Raspberri Pi Full Stack UK 200609.indd 99

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 29 • Read a button with GPIOZERO You now know how to "do" output. It's time to learn "input". A momentary button is an input device. It is essentially a sensor with only two possible states: "pressed" and "not pressed". In your circuit, you have connected the output of the button to GPIO14. You Python program will check the voltage at GPIO14 to determine if the button has been pressed or not. Just like with the LED, I'll show you two ways to do this. First, we'll use the GPIOZERO library. Unlike with the LED example, this time we'll go straight to writing a regular Python program. This is because we want to get Python to read the state of GPIO14 many times per second, so that as soon as we press the button, Python can print a message on the screen to let us know if it has detected the press. We can also do this on the CLI, but it will be a clunky implementation which I prefer to avoid. Use Vim to create a new Python program: $ vim button_gpiozero.py

Copy this code into the Vim buffer: from gpiozero import Button import time button = Button(14,None,True) while True: if button.is_pressed: print("Button is pressed") else: print("Button is not pressed") time.sleep(0.1)

Save the buffer to disk, and quit Vim (":wq"). There are a few new elements in this code, so let's take a moment to learn. In the first line, you are importing the Button module, another member of the gpiozero library. In the second line you import the "time" module, so that you can insert a tiny delay later in your program. This is important because without this delay, you program will sample the button GPIO as fast as your Raspberry Pi can, leaving little time for anything else. We only want to read the state of a button, not to totally dominate the CPU.

● 100

Raspberri Pi Full Stack UK 200609.indd 100

06-08-20 15:30

Chapter 29 • Read a button with GPIOZERO

Line three is a little challenging because it contains three parameters. You can find detailed information about these parameters in the gpiozero documentation for the Button object. The first one is the GPIO number. Since the button is connected to GPIO14, you'll enter "14" in the first parameter. The second parameter controls the type of pull-up/down resistor we are using. The Raspberry Pi can use internal pull-up resistors resistor, but in our circuit we have provided an external pull-down resistor. This resistor ensures that voltage at GPIO14 is equal to GND when the button is not pressed.6 Because of the presence of the external pull-down, we use "None" as the value of the second parameter. In the third parameter, we include the active state value. Because when the button is pressed the voltage at GPIO14 is "HIGH", and when the button is not pressed the voltage is "LOW", we use the "True" value for this parameter. If the voltages are reversed (i.e. pressed buttons created a "LOW" voltage, and not pressed button created "HIGH"), the third parameter we would write "False". You have saved the configured Button object in the "button" variable, and then got into an infinite loop. Your program simply reads the state of the button and if it is pressed, it prints "Button is pressed" to the console. If it isn't pressed, it prints "Button is not pressed". Run the program like this: $ python3 button_gpiozero.py

Now press the button and see how the message on the console changes accordingly. It should look like this (Figure 29.46):

Figure 29.46: A button was just pressed. It works! We'll do something more interesting very soon, but first let's achieve the same output using rpi.gpio in the next chapter. 6.

F or an article on pull-up/down resistors, please have a look at the Tech Explorations website (https://techexplorations.com/blog/ electronics/blog-what-are-pull-up-and-pull-down-resistors)

● 101

Raspberri Pi Full Stack UK 200609.indd 101

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 30 • Read a button with RPi.GPIO Let's read the state of a button using the rpi.gpio library now. Create a new Python program using Vim: $ vim button_rpigpio.py

Copy this code in the buffer: import RPi.GPIO as GPIO import time button = 14 GPIO.setmode(GPIO.BCM) GPIO.setup(button,GPIO.IN) while True: value = GPIO.input(button) if value: print("Pressed") else: print("Not Pressed") time.sleep(0.1) GPIO.cleanup()

Surprisingly, the code seems clearer than the gpiozero equivalent because there are no fancy parameters in any of the functions. You simply import the modules, set the pin numbering system ("GPIO.setmode"), set up the GPIO as an input ("GPIO.setup"), and you are off in the loop. In the loop, you take a reading of the state of GPIO14, and depending on whether it is HIGH or LOW (pressed or not pressed), you print out a message. By default, rpi.pgio assumes that you are using an external pull-down resistor. You can define an internal pull-up or pull-down resistor in the setup method, as explained in the library documentation7. Execute the program like this: $ python3 button_rpigpio.py

Press the button a few times to see how the message in the console changes. Here's what it looks like (Figure 30.47):

7.

Learn more about the various setup parameters in the documentation for GPIO inputs: https://sourceforge.net/p/raspberry-gpio-python/wiki/Inputs/

● 102

Raspberri Pi Full Stack UK 200609.indd 102

06-08-20 15:30

Chapter 30 • Read a button with RPi.GPIO

Figure 30.47: A button was just pressed. Let's take this one step further in the next chapter: use the button to control the state of the LED. When you press the button, the LED will turn on.

● 103

Raspberri Pi Full Stack UK 200609.indd 103

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 31 • Control an LED with a button In this chapter, you will combine the programs you wrote in the previous chapters so when you press the button, the LED turns on. I will show you how to do this using rpi.gpio, but feel free to implement the gpiozero version as an exercise. Create a new Python program with Vim: $ vim button_led.py

In the Vim buffer, copy this code: import RPi.GPIO as GPIO

## Import GPIO Library

import time button = 14

## Button connected to pin 8, GPIO14

led = 4

## LED connected to pin 7, GPIO4

GPIO.setwarnings(False)

## Turn off warnings

GPIO.setmode(GPIO.BCM)

## Use BCM pin numbering

GPIO.setup(button, GPIO.IN)

## Set button to INPUT

GPIO.setup(led, GPIO.OUT)

## Set led to OUTPUT

while True:

## Do this forever

value = GPIO.input(button)

## Read input from button

print(value) if value:

## If button is released

print("Pressed") GPIO.output(led, GPIO.HIGH) else:

## Turn LED on ## Else button is pressed

print("Not Pressed") GPIO.output(led, GPIO.LOW)

## Turn LED off

time.sleep(0.1) GPIO.cleanup()

In this example, I have included comments in most lines to help you understand what each does. You don't have to copy these comments into your program. You can also find this source code in the project code repository on Github8. Save the buffer and close Vim. At this point, you should be able to understand what each line does. Remember, this program is a combination of programs you have already written and executed.

8.

The source code for this program is available at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/button_led.py

● 104

Raspberri Pi Full Stack UK 200609.indd 104

06-08-20 15:30

Chapter 31 • Control an LED with a button

Execute the program with: $ python3 button_led.py

Press the button and see the LED turn on. You can see my circuit working in Figure 31.48.

Figure 31.48: When I press the button, the LED turns on. In the console, you can see the messages coming from your program changing depending on the state of the button (Figure 31.49).

Figure 31.49: When I press the button, the LED turns on. The text that you see in the console is produced by the two "print" statements inside the "while" loop. The first print statement prints out the raw value stored in the "value" variable. This is the value that is evaluated by the "if" function. Inside the 'if" block, there are two more print statements that print out the "Pressed" and "Not Pressed" messages.

● 105

Raspberri Pi Full Stack UK 200609.indd 105

06-08-20 15:30

Raspberry Pi Full Stack

You have come a long way since the beginning of Part 5 of this project, but really, the fun is just starting. Buttons and LED are very useful, but are only the "tip of the iceberg". In the next chapter I'll show you how you can use a digital sensor to measure temperature and humidity. This sensor is a core component of the application that you implement in this project.

● 106

Raspberri Pi Full Stack UK 200609.indd 106

06-08-20 15:30

Chapter 32 • Set up the DHT22 sensor with Git

Chapter 32 • Set up the DHT22 sensor with Git You should now be more comfortable working on the command line and writing small Python programs. You know how to control an LED and read the status of a button. Both are simple devices with only two possible states each: either on or off, or pressed/not-pressed. Sensors are sophisticated integrated devices that you can use to measure a range of environment factors, such as temperature and humidity, the strength of the Earth's magnetic field (or a small magnet), various forces and acceleration, even the concentration of various chemicals in the air. In this and the next chapter I'll show you how to set up and use a common environment sensor, the DHT22. DHT22 can measure temperature and humidity. This is an integrated digital sensor that uses a communication protocol to "talk" to its host. The host can be a microcontroller, like an Arduino, or computer, like the Raspberry Pi. The protocol is a language that the two devices use to communicate. The DHT22 will respond to a request for a reading from the Raspberry Pi with the raw numbers that represent the temperature and humidity. Unlike the button, which only has two states, a sensor like the DHT22 is a real "chat box" that sends several bytes of data, encoded according to its protocol that the host has to decipher. As you can see from this description, using a sensor is a bit more involved than using a button. As the programmer, you will need to understand the intricacies of the communications protocol, which include things like register addresses, specific timings for requests and events, and a deeper understanding of computer concepts like "Big-endian" and "Little-endian" and checksums. That's too much if you just want to measure temperature. Luckily, all this low-level functionality is nicely packaged nicely inside libraries that we can easily integrate into our programs. We can then extract the information we need from the sensor using a simple command. This is what we'll do with the DHT22. The process involves installing one of the many DHT22 libraries for Python and Raspberry Pi, and then running one of the example programs. First, let's add the DHT22 sensor to the breadboard and connect it to the Raspberry Pi. I would like to remind you of the schematic from earlier in this section. Go ahead and wire the right part of this schematic, including the DHT22 sensor and 10 kΩ resistor (Figure 32.40).

● 107

Raspberri Pi Full Stack UK 200609.indd 107

06-08-20 15:30

Raspberry Pi Full Stack

Figure 32.40: This schematic will guide you in the assembly of a simple circuit. In Figure 32.50 you can see the assembly of my circuit on a mini breadboard.

Figure 32.50: The circuit, assembled on a mini breadboard. Don't forget to turn off your Raspberry Pi before you connect the sensor. Do it like this: $ sudo shutdown -h now

● 108

Raspberri Pi Full Stack UK 200609.indd 108

06-08-20 15:30

Chapter 32 • Set up the DHT22 sensor with Git

Wait for the shutdown to complete, and then remove the power cable. Once you have assembled and confirmed the circuit, apply the power and wait for a couple of minutes, then log in to your Raspberry Pi as "pi". You will install the Adafruit DHT library for the Raspberry Pi. This library is hosted on Github. There are a few ways by which you can transfer the library (or any file) to your Raspberry Pi, but for now, and because this library is hosted on Github, the easiest is to use Git. With Git, you can clone (copy) a repository from Github to your Raspberry Pi. But first, you need to install Git on your Raspberry Pi. This is easy. Assuming you are logged in as Pi, copy this command: $ sudo apt-get install git-core

It will install the Git tool on your Raspberry Pi. Wait for the installation to complete, and then configure the tool for use: pi@raspberrypi-zero:~ $ git config --global user.email [email protected] pi@raspberrypi-zero:~ $ git config --global user.name "Peter"

Of course, use your own email address and name. Now you can go ahead and clone the repository into your pi user home folder. Copy this into the command line: pi@raspberrypi-zero:~ $ git clone https://github.com/adafruit/Adafruit_ Python_DHT.git Cloning into 'Adafruit_Python_DHT'... remote: Enumerating objects: 325, done. remote: Total 325 (delta 0), reused 0 (delta 0), pack-reused 325 Receiving objects: 100% (325/325), 98.35 KiB | 234.00 KiB/s, done. Resolving deltas: 100% (176/176), done.

That's it, you now have a copy of the DHT22 library repository in your local folder. The repository contains a few example programs that you can play with, as well as the library itself. Before you can play with the examples, you must install the library. Follow this process: 1. Go in the repository directory: $ cd Adafruit_Python_DHT/

● 109

Raspberri Pi Full Stack UK 200609.indd 109

06-08-20 15:30

Raspberry Pi Full Stack

2. Run the setup utility: $ sudo python3 setup.py install

The setup utility will produce a lot of output in the console, and eventually show you something like "Finished processing dependencies for Adafruit-DHT==1.4.0". The library is now installed. Let's play with it in the next chapter.

● 110

Raspberri Pi Full Stack UK 200609.indd 110

06-08-20 15:30

Chapter 33 • Use the DHT22 sensor

Chapter 33 • Use the DHT22 sensor Let's try out the DHT22 sensor with the Adafruit library. From the root of your "pi" directory, change into the Adafruit DHT directory, and then into the examples directory: $ cd Adafruit_Python_DHT/ $ cd examples/

Then, run the AdafruitDHT.py example, like this: $ python3 AdafruitDHT.py 2302 17 Temp=23.6*

Humidity=60.8%

Notice there are two arguments this example needs: the type of sensor you are using ("2302") and the GPIO to which the sensor data pin is connected ("17"). When the example program runs, you can an output with the temperature and humidity. Now that you know that your sensor works, you can take a closer look at the example program to learn how it works. You can see inside any text file using the "cat" command, like this: $ cat AdafruitDHT.py

I am copying the program here, but to economise on space I have removed most of the multi-line comments: ~/Adafruit_Python_DHT/examples $ cat AdafruitDHT.py #!/usr/bin/python # Copyright (c) 2014 Adafruit Industries # Author: Tony DiCola # … other comments import sys import Adafruit_DHT # Parse command line parameters. sensor_args = { '11': Adafruit_DHT.DHT11, '22': Adafruit_DHT.DHT22, '2302': Adafruit_DHT.AM2302 } if len(sys.argv) == 3 and sys.argv[1] in sensor_args: sensor = sensor_args[sys.argv[1]] pin = sys.argv[2] else: print('Usage: sudo ./Adafruit_DHT.py [11|22|2302] ') print('Example: sudo ./Adafruit_DHT.py 2302 4 - Read from an AM2302 connected to GPIO pin #4')

● 111

Raspberri Pi Full Stack UK 200609.indd 111

06-08-20 15:30

Raspberry Pi Full Stack

sys.exit(1) # Try to grab a sensor reading.

Use the read_retry method which will retry up

# to 15 times to get a sensor reading (waiting 2 seconds between each retry). humidity, temperature = Adafruit_DHT.read_retry(sensor, pin) # Un-comment the line below to convert the temperature to Fahrenheit. # temperature = temperature * 9/5.0 + 32 # Note that sometimes you won't get a reading and # the results will be null (because Linux can't # guarantee the timing of calls to read the sensor). # If this happens try again! if humidity is not None and temperature is not None: print('Temp={0:0.1f}*

Humidity={1:0.1f}%'.format(temperature, humidity))

else: print('Failed to get reading. Try again!') sys.exit(1)

The two import statements at the start of the program import the library itself ("Adafruit_ DHT"), and the "sys" module. The "sys" (or "system") module contains functions like "sys. argv", used to get input from the command line, that you have seen in previous examples. The "sensor_args" list matches our input of the sensor model to the internal code for this model that the library understands. So, when we specify "2302", the library will internally will use "Adafruit_DHT.AM2302". Let's jump to the interesting part. This line: humidity, temperature = Adafruit_DHT.read_retry(sensor, pin)

… is where the program asks the sensor for humidity and temperature readings. We can get both readings in a single line of code. The result is a multiple assignment, where the humidity value is stored in the "humidity' variable, and the temperature value is stored in the "temperature" variable. The "read_retry" function needs two parameters: the type of sensor you are using, and the GPIO to which it is connected. Both of those values we specify in the command line when we invoke the program. If you prefer temperature in Fahrenheit, you can uncomment the conversion calculation just after the measurement is taken.

● 112

Raspberri Pi Full Stack UK 200609.indd 112

06-08-20 15:30

Chapter 33 • Use the DHT22 sensor

Another interesting segment in the program is this: if humidity is not None and temperature is not None: print('Temp={0:0.1f}*

Humidity={1:0.1f}%'.format(temperature, humidity))

else: print('Failed to get reading. Try again!') sys.exit(1)

This is where the program tests for the validity of the result from the sensor. If the result is a number (instead of "None"), it will print out the results. Otherwise it will print out a "failed" message. Behind the scenes, a fairly sophisticated choreography of actions takes place. But from our perspective and the end users of the sensor module and the library, we have accomplished what we wanted with a single line of code. This is an example of the power of Python and its library ecosystem. Rapid prototyping.

● 113

Raspberri Pi Full Stack UK 200609.indd 113

06-08-20 15:30

Raspberry Pi Full Stack

Part 6: Set up the Web Application Stack

● 114

Raspberri Pi Full Stack UK 200609.indd 114

06-08-20 15:30

Chapter 34 • The Web Application Stack

Chapter 34 • The Web Application Stack Welcome to Part 6 of this project. In the chapters that make up this Section, I will show you how to implement the infrastructure of your Full Stack Application. This is the perfect time to learn about the meaning of the term "Full Stack". By doing so, you will understand why this web application requires so many software components, what their individual role is, and how they relate to each other. In this chapter, I will give you an overview of the Web Application Stack. Starting in the next chapter, you will implement the web application stack on your Raspberry Pi. In the remainder of the project, you will gradually add features and capabilities, always within the framework that you will create in Part 6 of the project.

Figure 34.51: A graphical depiction of the Web Application Stack. In Figure 34.51 you can see a depiction of the web application stack. In the centre, depicted as blocks, you can see the major components of the stack, such as the operating system, code application, and application server. Outside of the blocks, you can see the specific software I have chosen to use and implement for each component. For example, I have chosen to use uWSGI to implement the application server, and NGinx to implement the web server. While the architecture of the stack tends to be the same (or very similar) among different implementations, the exact technologies used for each component can vary significantly. What I show you in this project is just one possible selection of high-quality, open source technologies. As you start understanding how the stack works, you will begin to consider other options that are more appropriate for your specific needs. For example, you may choose to swap the SQLite3 database that we will use in this project (good for low-loads and single-user applications) for a database that permits higher client loads in a production environment.

● 115

Raspberri Pi Full Stack UK 200609.indd 115

06-08-20 15:30

Raspberry Pi Full Stack

Now, let's walk through the individual components that make up the web application stack. (1) Hardware (a.k.a. "metal") At the bottom of the stack is the hardware. This is where you will find the processor, memory, and other peripherals. In this project, the hardware consists of a Raspberry Pi Zero W. Much of what happens in the hardware layer is abstracted by the operating system, the various device drivers and other components of the stack, so that we, as the end users, don't have to worry about the details. It is possible to swap the Raspberry Pi Zero W with another Raspberry Pi, and without changing anything else, our application will still work. It is also possible to change our hardware for an alternative with a totally different architecture, such as a Beaglebone Black. With only small modifications in the rest of the stack, the application will still work. I am intrigued when I think that the hardware, while it is responsible for doing the actual work of running our application, is the component of the stack that I have to worry the least about. (2) Server operating system Operating systems come in two big types: • Server operating systems • Desktop operating systems A desktop operating system is typically one that runs productivity software (or games!) that a single user will use. This is software like web browsers, spreadsheets, and email clients. These operating systems are optimised to run a graphical user interface with quick reaction times to user events, like clicking a button. A server operating system is optimised to respond to client requests that arrive via a network interface. Because these requests come from a network interface (not a keyboard or mouse) and the responses are also routed to the client via the same network interface (not a screen), a server operating system does not use a graphical user interface. Its resources are dedicated to running the services required to respond to client requests, such as databases and web servers. The application you are building in this project is, essentially, a web application. The end user will use a browser to make a HTTP request to the application server. This request will take place through a network interface, not a keyboard and/or mouse. This scenario fits best with a use case for a server operating system. This is why I have chosen to use Raspbian Lite, instead of the full version of Raspbian with the graphical components. Of course, consider that modern operating systems are extremely versatile. You can take Raspbian Lite, and install the missing GUI packages to convert it to a full desktop operating system, while still running your web application. And, you can take the full Raspbian and use it to run your web application. Anything is possible, but in each case there is an optimisation towards one of the two major use cases.

● 116

Raspberri Pi Full Stack UK 200609.indd 116

06-08-20 15:30

Chapter 34 • The Web Application Stack

(3) Application framework An application framework is a collection of software, templates and decisions designed to help you create an application quickly. There are many application frameworks that you can use to build virtually anything: web applications, games, mobile applications, artificial intelligence applications etc. When it comes to building web application, we are spoilt for choice. There are frameworks like Django, Turbogears and web2py (only three out of many dozens) that allow you to quickly build fully-features web applications. There are many other web-application frameworks for other languages, such as Ruby-onRails for Ruby, Angular for Javascript, ASP.NET for C#, and many more. In this project you are going to build a simple web application. There's a lot that your application will not need to do. For example, there is no need for authentication, ecommerce, or a custom storage system. Therefore, you will not need a fully-featured web application development framework, which while it can do "anything", will require a lot more time to learn. This is why I have chosen the Flask "micro-framework". We call it "micro" because it has the benefits of a full framework (efficiency in development time), without the bloat that comes with frameworks that want to do everything. With Flask, you can build an application (like the one in this course) with a single Python file, and learn why you need to do this within a few minutes (not days or weeks). You will build your full-stack application within the confines of the Flask framework. This is what I represent with the orange "Application" box inside the blue "App framework" box marked as "3" in Figure 34.51. (4) & (5) Application server and Web server You can build an application with the help of a development framework, and you can also run a test instance to test it. However, to properly publish your application, you will need two additional components: an application server, and web server (I will discuss the web server later in this chapter). The application server is a program that has one purpose: to manage your web application. While the application you are building in this project is small, and is likely to never have more than one or two users, it is an opportunity to think about how to build an application that can potentially have thousands of users. A web application server is responsible for managing the application, so that it can scale up. For example, imagine you have built an amazing application on your Raspberry Pi, that a lot of people use. One Raspberry Pi is not enough to deal with the load, so you add another five to your new cluster. The application server is there to help you manage the available resources of your cluster, so that each user request is given a quick response. This is just one example.

● 117

Raspberri Pi Full Stack UK 200609.indd 117

06-08-20 15:30

Raspberry Pi Full Stack

In the case of both, (1) the potential for scalability, and (2) my decision to make this project as realistic as possible, I should also add the last component of the stack, the web server (marked "4" in Figure 34.51). The web server is responsible for taking a HTTP request from a client, and passing it on to the application for processing. The web server, in a way, protects the application by being an intermediary. The webserver is placed between the application and end user, and evaluates incoming requests. Anything in the request that the webserver can fulfil on its own, it will, and therefore preserve application resources for requests that need actual processing. For example, you will later learn that the NGinx webserver (my choice of technology for this layer of the stack), is capable of serving static assets, such as Cascading Style Sheet and Javascript files directly, bypassing the application server. For anything else, it will forward the request to the application server, which in turn will forward it to the Flask-Python application. If you have a cluster of Raspberry Pis each running its own copy of your Flask-Python application, the application server will use the most appropriate one to process the incoming request. For example, if the request is asking for temperature and humidities recorded in the last 6 hours (not a static resource), NGinx9 will forward it to uWSGI (my choice of web application server technology for the fourth layer of the stack), which will run your Flask-Python application and evaluate the response to the web browser. As you can see, a web application is composed of multiple layers, from hardware, to the operating system, all the way to the publicly accessible webserver. The actual code that you write to implement the functionality of your application is only a small part of the code that is needed to publish. Most of this code has to do with managing hardware and software resources, and protecting your application from the outsider world. I hope that this discussion has given you a better understanding of the full stack components. In the next chapter, I will discuss another critical component of the stack that I have marked as "6" in Figure 34.51: the Python Virtual Environment.

9.

While it is potentially possible for NGinx to be able to work with our Flask application directly, at the time I am writing this, such capability is not implemented. It is likely that the NGinx developers prefer to continue to work on making the software better as a web server rather than implementing features that are already satisfied through other software, such as uWSGI. On the other hand, uWSGI can fulfil the role of a front-facing webserver, in place of Nginx. However, it is not good as Nginx in certain areas, including efficiency by which Nginx fulfils requests for static assets. Therefore, a combination of the two helps us achieve excellent performance with only a small increase in the complexity of our setup. You may want to read this blog article from more details on the topic: https://www.ultravioletsoftware.com/single-post/2017/03/23/An-introduction-into-the-WSGI-ecosystem

● 118

Raspberri Pi Full Stack UK 200609.indd 118

06-08-20 15:30

Chapter 35 • The Python Virtual Environment

Chapter 35 • The Python Virtual Environment Just when you think you have learnt all there is to know about the architecture of the full stack application, I come along to say that there's one more thing… The "Python Virtual Environment". What is that? I assure you, there is no trickery. In general, "virtualisation" is a technique that has become very popular in the last 20 years that helps us squeeze much more computing performance from our servers than what it would be possible without it. You are probably familiar with the concept of a "Virtual Machine" (VM). A VM is a full computer implemented in software. Yes, it is a software computer. Like any computer, physical or virtual, a software computer contains a dedicated operating system and all of its application software. Of course, it has to be hosted on a physical computer, a machine with real hardware resources. But, here's the amazing thing: this physical computer is a host of one or more Virtual Machines. With virtualisation, our hardware is better utilised. This is the principle behind Cloud computing, which is simply an infrastructure composed of millions upon millions of networked physical computers, able to host billions of Virtual computers (VMs). Apart from the enhanced utilisation of our computing hardware, Virtual Machines also give us immense flexibility. We can have ten VMs on a single physical computer, and each of these VMs can be independent. If you change something in the operating system or application software of one of the VMs, the other will not be affected. You can restart or stop a VM, update or change its software, even crash it deliberately, without the other VMs on the same host caring. The concept of virtualisation is so powerful that it has been adopted in many other areas of computer science. One of them, as you might expect, is in running Python applications. Please consider the diagram in Figure 35.52.

● 119

Raspberri Pi Full Stack UK 200609.indd 119

06-08-20 15:30

Raspberry Pi Full Stack

Figure 35.52: A graphical depiction of the Python Virtual Environment. Imagine a computer, like your Raspberry Pi. When you install Raspbian on this computer (like most flavours of Debian Linux), it comes equipped with a Python interpreter. Because this interpreter ships with the operating system, we can refer to it as "system Python". You can use system Python to execute any number of Python programs on your computer. If your program require additional modules, you can install them with pip (the Python package manager) or "apt-get", the Debian package manager. Now imagine that you have two programs that require the same Python package. But, each needs a different version. You now have a problem. Your two applications require two different versions of the same Python package, but you only have one Python interpreter (the system interpreter) with a single location for storing these packages. Yes, it is possible to specify the version of a Python package at the application level, but that increases the complexity of the application. Now, imagine these two scenarios: 1. You have two Python applications, but each requires a specific version of the Python interpreter. You now have to install and maintain two Python interpreters. 2. You update several system-level Python packages, and unknowingly break the functionality of one of your Python applications that is not compatible with the new version. Your application is broken until you go back to the previous version of the offending package. And then, you have to figure out what the problem is, whilst the rest of your applications have to operate with an old version of the offending package.

● 120

Raspberri Pi Full Stack UK 200609.indd 120

06-08-20 15:30

Chapter 35 • The Python Virtual Environment

In Figure 35.52, the situation I have just described is depicted on the left side. The red application boxes are utilising the same system, Python interpreter and share the same Python packages. Changes to the bottom two layers will affect all applications. All of these problems are solved using dedicated Python Virtual Environments. These environments are equivalent to the VMs I mentioned earlier. In Figure 35.52, you can see there is an additional layer above the system Python layer, that consisting of Python Virtual Environments. On top of each virtual environment, you have an application. In other words, each application is executed inside a dedicated Python Virtual Environment. Each virtual environment has its own Python interpreter and collection of packages, all hand-picked to match the exact requirements of the application. Any changes to the dedicated environment do not impact any of the other applications because there is no connection between them. The environments are isolated. When we use Python Virtual Environments, the system Python is used to generate the virtual environment only. From the moment the virtual environment is generated, it becomes an independent bubble, and your application runs completely within it. To be sure, your various applications operating within their individual virtual environments will be sharing other operating system resources. For example, they may share a database server, and web server. However, their resources have their own way of securely working with different clients (=applications) that don't compromise the flexibility and safety of the Python Virtual Environment. For these reasons, I chose to use a Python Virtual Environment for the purposes of this project. This is best practice of Python applications, and, as you are about to see in the next few chapters, it is very easy to implement. Let's begin to implement the full-stack application Python Virtual Environment in the next chapter. >> This is a good time to take a full backup of your SD card.

● 121

Raspberri Pi Full Stack UK 200609.indd 121

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 36 • Increase the disk swap file size If you are using a Raspberry Pi Zero W (or a Raspberry Pi will less than 512 Mbytes of RAM) in this project, you should follow the instructions in this chapter. I will show you how to set up a RAM swap file of 1 GByte. This swap file is an extension of the RAM on the SD card. Without it, it is likely that some of the operations later on in this project that involve the compilation of C programs will fail due to insufficient RAM. A warning is required here: A swap file on an SD card should be used as a temporary measure because it can drastically reduce the life of your SD card. An SD card wears out faster when there is a lot of read/write activity. RAM operations have a lot more read/write cycles compared to SD card or disk operations, and as a result, when we treat an SD card as if it is RAM, we expose it to much more activity than it is designed for. To mitigate the issues, I suggest you expand the size of the swap file to make it possible to set up your application during the development of the application, and then reduce it to the default size once development is finished. By default, Raspbian defines a 100 Mbyte RAM swap file. This is small, but enough to improve the stability of the system in cases where RAM is not enough. You can find out how much swap space is used on your Raspberry Pi with the "free -m" command: $ free -m total

used

free

shared

buff/cache

available

Mem:

480

82

149

24

248

320

Swap:

99

0

99

In this example, my Raspberry Pi is set to use 100 Mbytes of swap space, shown here as "99". All of this space is free at the moment. To change the size of the swap file, you must edit the file /etc/dphys-swapfile: $ sudo vim /etc/dphys-swapfile

In this file, look for the line that starts with "CONF_SWAPSIZE", and change the value to "1024": CONF_SWAPSIZE=1024

Save the buffer to disk and save the file with ":wq".

● 122

Raspberri Pi Full Stack UK 200609.indd 122

06-08-20 15:30

Chapter 36 • Increase the disk swap file size

Now, restart the swap file deamon: $ sudo /etc/init.d/dphys-swapfile restart [ ok ] Restarting dphys-swapfile (via systemctl): dphys-swapfile.service.

And run "free -m" once again to confirm that the size of the new swap file is 1024 Mbytes: $ free -m total

used

free

shared

buff/cache

480

82

145

24

251

1023

0

1023

available Mem: 319 Swap:

As you can see in my example above, the new Swap file total is 1023 Mbytes, all of which is free. Much of this available space will be used later as you compile various C packages. Remember to reset the swap file back to the default 100 Mbytes to prolong the lifespan of your SD card. To do this, simply reverse the process I described above: 1. In file "/etc/dphys-swapfile" set "CONF_SWAPSIZE" to "100". 2. Restart the swap file deamon: "sudo /etc/init.d/dphys-swapfile restart" Let's continue on to the next chapter where you will install several utilities and packages that you will need later in the project.

● 123

Raspberri Pi Full Stack UK 200609.indd 123

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 37 • Set up system Python - preparation In this chapter you will install various packages and tools that are needed later in the project. All these packages come from the Debian repository, in which there is a special sub-project for Raspbian. At the time I am writing this chapter, Buster is the current version of Raspbian. You can find information about all the packages that are part of Buster here: https://packages.debian.org/buster/. At the same location, you can find information about the packages that you will be installing in this chapter. For example, if you are curious to know what exactly is "build-essential", then have a look at https://packages.debian.org/buster/build-essential. Similarly, if you want to know what is in "libncurses5-dev", have a look at https://packages. debian.org/buster/libncurses5-dev. For any of these packages, you can go to the Buster base URL "https://packages.debian. org/buster/" and add the exact name of the package at the end. Start by logging into your Raspberry Pi as the "pi" user: $ ssh [email protected]

Then issue these commands, one at a time: $ sudo apt-get update $ sudo apt-get upgrade $ sudo apt-get install build-essential $ sudo apt-get install libncurses5-dev libncursesw5-dev libreadline6-dev libffi-dev $ sudo apt-get install libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libsqlite3-dev libgdbm-dev libssl-dev openssl $ sudo apt-get install libboost-python-dev $ sudo apt-get install libpulse-dev $ sudo apt-get install python-dev $ sudo apt-get install vim

The apt-get tool may suggest additional actions, such as to running it with the "--fix-missing" parameter. Keep an eye on output on the console and follow any advice provided to ensure all of these modules are successfully installed before you proceed. That's it. In the next chapter, you will install the latest available version of Python at system level. You will then use this Python to set up the Virtual Python Environment for your application. >> This is a good time to take a full backup of your SD card.

● 124

Raspberri Pi Full Stack UK 200609.indd 124

06-08-20 15:30

Chapter 38 • Download, compile and install Python 3

Chapter 38 • Download, compile and install Python 3 Raspbian ships with a Python 3 interpreter, however it tends to be a few months old. It is preferable to use newer versions of software whenever possible to take advantage of performance and security enhancement. For this reason, I encourage you to install the latest available version of Python on your Raspberry Pi system, and then use this version to set up the Python Virtual Environment for your application (Figure 38.53).

Figure 38.53: In this chapter you will install an updated version of Python at the system level. At the time I am writing this chapter, the latest version of Python is 3.9.0. However, I have thoroughly tested this project with version 3.8.2. To keep the instructions in this book consistent, I will use 3.8.2 as the "official" version of Python for this project. I do thorough testing of newer versions of Python every ~6 months. To find out which new versions of Python are available, browse to https://www.python.org/ ftp/python/.

● 125

Raspberri Pi Full Stack UK 200609.indd 125

06-08-20 15:30

Raspberry Pi Full Stack

Figure 38.54: The current latest version of Python is 3.9.0, however I have tested up to 3.8.2. Click on "3 .8 .2" to go into the folder that contains the downloadable materials for Python 3 .8 .2 . In this folder you will see various download options, including binaries for various operating systems, and, of course, the source code we need to compile Python for our hardware and OS . Right-click on "Python-3 .8 .2 .tgz" and select "Copy Link" or the equivalent option of your browser to get the URL for this archive file (Figure 38 .55) .

Figure 38.55: Copy the URL of the source code archive for Python 3.8.2. You will get a URL like this: https://www .python .org/ftp/python/3 .8 .2/Python-3 .8 .2 .tgz You will use this URL in a moment .

● 126

Raspberri Pi Full Stack UK 200609.indd 126

06-08-20 15:30

Chapter 38 • Download, compile and install Python 3

Now, log in to your Raspberry Pi as the "pi" user. Ensure you are in the "pi" home directory, create a new directory titled "python-source", and change to it: $ cd ~ $ mkdir python-source $ cd python-source/

While in "python-source", use the "wget" utility to download the source code archive file from the URL you copied earlier: $ wget https://www.python.org/ftp/python/3.8.2/Python-3.8.2.tgz

Next, use the "tar" utility to extract the contents of the archive: $ tar zxvf Python-3.8.2.tgz

The contents of the archive are stored in a new directory titled "Python-3.8.1". Switch to this directory, and the run the configure utility to set up your new Python installation: $ cd Python-3.8.2/ $ ./configure --prefix=/usr/local/opt/python-3.8.2

It's now time to do the actual compilation. This process will take a long time, especially on a relatively slow Raspberry Pi Zero W. Type below to invoke the "make" utility which will compile the source code into a full binary of Python 3.8.2: $ make

Get up to stretch your legs and perhaps get some tea or coffee. When the compilation is complete, use "make install" to copy the binary files to the correct location on Raspbian. Because you will be changing folders that are owned by the root user, you must use "sudo": $ sudo make install

If all went well, you now have an updated version of system Python along side older versions. You can confirm that your new Python works by invoking it, and passing the "version" switch: $ /usr/local/opt/python-3.8.2/bin/python3.8 --version Python 3.8.2

● 127

Raspberri Pi Full Stack UK 200609.indd 127

06-08-20 15:30

Raspberry Pi Full Stack

The response is the actual version of Python that you just installed, proof that the installation was successful. >> This is a good time to take a full backup of your SD card.

● 128

Raspberri Pi Full Stack UK 200609.indd 128

06-08-20 15:30

Chapter 39 • Set up the app Python virtual environment

Chapter 39 • Set up the app Python virtual environment In this chapter you will set up a dedicated Python environment for your application based on system Python 3 .8 .2 which you installed in the previous chapter . To do so, you will also create the directory in which your application will exist . I depict this graphically in Figure 39 .56 .

Figure 39.56:A dedicated Python environment for your application. Start by logging into your Raspberry Pi as user "pi" . Then, switch to the root user so that you are able to create a directory and work in the "var" folder: $ sudo su

Then create the new directory under "var", called "www", and within "www" the actual application directory called "lab_app": # mkdir /var/www # mkdir /var/www/lab_app/ # cd /var/www/lab_app/

Now, install the Python Virtual Environment by calling the "venv" module of Python 3 .8 .2: # /usr/local/opt/python-3.8.2/bin/python3.8 -m venv .

In the command you just entered, the "-m" switch runs the "venv" module which sets up the virtual environment in the current directory . Let's try out the new Python environment . While still in /var/www/lab_app/, activate the environment like this: # . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

● 129

Raspberri Pi Full Stack UK 200609.indd 129

06-08-20 15:30

Raspberry Pi Full Stack

Notice the command prompt line starts with the name of the Python Virtual Environment in parentheses: "(lab_app)". This is how you know the virtual environment is activated. Let's get the latest version of the virtual environment Python: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python --version Python 3.8.2

It is 3.8.2, the same as the system Python that you used to generate the virtual environment. Let's do one last test. Let's find out the location of the Python interpreter that you are using in this virtual environment. To do this, invoke the Python CLI using your virtual environment Python (just type "python"): (lab_app) root@raspberrypi-zero:/var/www/lab_app# python Python 3.8.2 (default, Mar 14 2020, 01:38:54) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> print(sys.executable) /var/www/lab_app/bin/python >>>

When the CLI starts, import the "sys" module, and then "print(sys.executable)". The response from the print command is the location of the Python interpreter that you are using, which is inside the "bin" directory of the new application directory. Congratulations! Your Python virtual environment is set up and ready to use. Let's jump over to the next chapter to set up the web server, and after this, create your first Flask web application.

● 130

Raspberri Pi Full Stack UK 200609.indd 130

06-08-20 15:30

Chapter 40 • Set up Nginx

Chapter 40 • Set up Nginx In this chapter you will install NGinx, which is item (5) in the Full Stack diagram of Figure 40.51. Apart from testing to ensure it works after installation, you are not going to do much more with it (i.e. configure it to work with your application) until later.

Figure 40.51: A graphical depiction of the Web Application Stack. Start by logging in to your Raspberry Pi as "pi" user. Then, switch to the root user with "sudo su", and install the "nginx" package: $ sudo su # apt-get install nginx

Nginx is a system-level service, so it does not matter if your application Python virtual environment is activated. When the installation process completes, your new NGinx webserver should be working. To test it, bring up your web browser and point it to the IP address or host name of your Raspberry Pi (Figure 40.57).

● 131

Raspberri Pi Full Stack UK 200609.indd 131

06-08-20 15:30

Raspberry Pi Full Stack

Figure 40.57: NGinx start working as soon as installation is complete. Ignore Nginx for now. At a later time, you can go back to it for configuration with your Flask application. Before you can do this, you have to set up the Flask application itself. You will do this in the next chapter.

● 132

Raspberri Pi Full Stack UK 200609.indd 132

06-08-20 15:30

Chapter 41 • Set up Flask

Chapter 41 • Set up Flask Flask is a Python micro-framework for rapid development of simple web applications. Its gentle learning curve and simplicity are the main reasons I decided to use Flask in this project. In this chapter, I will show you how to install it and create a very simple "hello world" application. In Figure 41.51, you will work in layer 3 of the stack. Unlike NGinx, which is a system-level server, your Flask application will operate inside your applications Python Virtual Environment. In fact, this Flask application is the centre of the web application for this project.

Figure 41.51: A graphical depiction of the Web Application Stack. Start by logging into your Raspberry Pi as the "pi" user, change into the application directory, switch into the superuser ("su"), and activate the Python virtual environment: $ cd /var/www/lab_app pi@raspberrypi-zero:/var/www/lab_app $ sudo su root@raspberrypi-zero:/var/www/lab_app# . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

Now, use "pip" to install the Flask module in the Python virtual environment: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install flask

At the end of the process, you should see something like this: Successfully installed Jinja2-2.10.3 MarkupSafe-1.1.1 Werkzeug-0.16.0 click-7.0 flask-1.1.1 itsdangerous-1.1.0

● 133

Raspberri Pi Full Stack UK 200609.indd 133

06-08-20 15:30

Raspberry Pi Full Stack

This indicates that Flask was installed successfully. You may also see a message that is suggesting that the Python package manager needs an upgrade. You can do it like this: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install --upgrade pip

Now that Flask is installed, let's try out. Use Vim to create simple Flask application called "hello.py". In this file, copy this content: from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Hello World!" if __name__ == "__main__": app.run(host='0.0.0.0', port=8080)

Don't worry about what is happening in this program now; I will explain in detail in the next chapter. Save the buffer to disk and exit Vim (":wq"). Back on the command line, start your new Flask application like this: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python hello.py * Serving Flask app "hello" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)

Notice the output. It is telling you that your Flask application is running on http://0.0.0.0:8080/ The IP address "0.0.0.0" is unusual, but simply indicates that your application is accessible from clients on the network. Open your browser and point it to your Raspberry Pi's IP address or hostname. Remember to include the port number "8080" at the end. In Figure 41.58 you can see my browser fetching the sample page from my test Flask application, on port 8080.

● 134

Raspberri Pi Full Stack UK 200609.indd 134

06-08-20 15:30

Chapter 41 • Set up Flask

Figure 41.58: Access the "Hello World" Flask application. It works! Next up, let's take a closer look at the simple Flask program. It will be our launching pad to what will come.

● 135

Raspberri Pi Full Stack UK 200609.indd 135

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 42 • A tour of a simple Flask app You can find a detailed description of how the minimal Flask application works on the Flask documentation website10. Here I summarise a few points of interest, and also mention a couple of different ways by which you can execute this (and any) Flask application. Here is the program, with line numbers: 1.

from flask import Flask

2.

app = Flask(__name__)

3.

@app.route("/")

4.

def hello():

5.

return "Hello World!"

6.

if __name__ == "__main__":

7.

app.run(host='0.0.0.0', port=8080)

Let's start (naturally) with line 1: from flask import Flask

Here, your program imports the Flask module so that it can then create an instance of the application. This is what happens in line 2: app = Flask(__name__)

This is where your program creates the web application object. The Python keyword "_ _name _ _" will return "_ _main_ _" if we are executing this program from the command line when we are doing testing. This information is useful at the end of the program, in lines 6 and 7: if __name__ == "__main__": app.run(host='0.0.0.0', port=8080)

In line 6, the program is checking to see if it was started by the user on the command line. If _ "_name_ _" is equal to "_ _main_ _", then the answer is "yes", so Flask will start the web application and make it available to any client on port "8080". This is a point of confusion for many students. Remember this: when you are developing your web applications, you will most often start it on the command line like this: # python hello.py 10. A minimal Flask application (tutorial): https://flask.palletsprojects.com/en/1.1.x/quickstart/#a-minimal-application

● 136

Raspberri Pi Full Stack UK 200609.indd 136

06-08-20 15:30

Chapter 42 • A tour of a simple Flask app

When you have completed your application and are ready to deploy it, you will set up the uWSGI application server to control your Flask application. When this happens, the web application will not be started by a command line invocation, but by a seperate system module (uWSGI). In this case, you don't want to start your application and make it available on port 8080, but on whichever URL the application server is configured for. This is the role for the code on lines 6 and 7. On line 3, the program defines a route "/", and matches it with a function that will handle a web client request for this route. This function is defined on line 4. The association between the route and the function handler is made by positioning the route definition just before the function definition. On line 3, the defined route is "/". This route is called when we call this URL: http://raspberrypi-zero.local/

You can define any route you want in a similar way. For example, this: @app.route("/test)

… is accessible through a URL like this: http://raspberrypi-zero.local/test

On line 5, the program defines the function that will handle the root route. Inside this function, there is only a return statement that prints "Hello World!" to the client. This is the message that you saw in your browser when you tested the application in the previous chapter. Of course, the body of the function can do a lot more, including accessing a database, doing calculations, and sending a fully styled HTML document back to the client. This is something you will do in later parts of this project. For now, let's continue with the next step of this project: installing uWSGI.

● 137

Raspberri Pi Full Stack UK 200609.indd 137

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 43 • UWSGI installation Let's continue the project by installing uWSGI, the application server. I remind you that uWSGI is the component of the stack that manages the Python Flask application, and processes incoming client requests passed to it through NGinx. In Figure 43.51, the uWSGI application server is depicted in layer "4", in-between NGinx in layer 5 and Flask in layer 3.

Figure 43.51: A graphical depiction of the Web Application Stack. To install uWSGI, be sure to log in to your Rapsberry Pi as "pi", and then switch to super user, changing to the application directory: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install uwsgi

At the end, you will see something like this on the command line: Successfully installed uwsgi-2.0.18

You can check the installation was successful, and also get the installation location and version by using the "show" sub command of the pip package manager: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip show uwsgi Name: uWSGI Version: 2.0.18 Summary: The uWSGI server Home-page: https://uwsgi-docs.readthedocs.io/en/latest/ Author: Unbit

● 138

Raspberri Pi Full Stack UK 200609.indd 138

06-08-20 15:30

Chapter 43 • UWSGI installation

Author-email: [email protected] License: GPLv2+ Location: /var/www/lab_app/lib/python3.8/site-packages Requires: Required-by:

You can also look for the uwsgi executable binary inside the "bin" folder of your application folder: (lab_app) root@raspberrypi-zero:/var/www/lab_app# ls -al bin/ total 16068 drwxr-xr-x

2 root root

4096 Mar 14 02:56 .

drwxr-xr-x 12 root root

4096 Mar 31 23:16 ..

-rw-r--r--

1 root root

2202 Mar 14 01:53 activate

-rw-r--r--

1 root root

1254 Mar 14 01:53 activate.csh

-rw-r--r--

1 root root

2406 Mar 14 01:53 activate.fish

-rw-r--r--

1 root root

8471 Mar 14 01:53 Activate.ps1

-rwxr-xr-x

1 root root

235 Mar 14 02:26 chardetect

-rwxr-xr-x

1 root root

247 Mar 14 01:20 easy_install

-rwxr-xr-x

1 root root

247 Mar 14 01:20 easy_install-3.8

-rwxr-xr-x

1 root root

225 Mar 14 02:17 flask

-rwxr-xr-x

1 root root

227 Mar 14 02:31 jsonschema

-rwxr-xr-x

1 root root

233 Mar 14 02:31 jupyter

-rwxr-xr-x

1 root root

233 Mar 14 02:31 jupyter-migrate

-rwxr-xr-x

1 root root

238 Mar 14 02:31 jupyter-troubleshoot

-rwxr-xr-x

1 root root

266 Mar 14 02:31 jupyter-trust

-rwxr-xr-x

1 root root

238 Mar 14 02:19 pip

-rwxr-xr-x

1 root root

238 Mar 14 02:19 pip3

-rwxr-xr-x

1 root root

238 Mar 14 02:19 pip3.8

-rwxr-xr-x

1 root root

226 Mar 14 02:26 pyrsa-decrypt

-rwxr-xr-x

1 root root

226 Mar 14 02:26 pyrsa-encrypt

-rwxr-xr-x

1 root root

224 Mar 14 02:26 pyrsa-keygen

-rwxr-xr-x

1 root root

247 Mar 14 02:26 pyrsa-priv2pub

-rwxr-xr-x

1 root root

220 Mar 14 02:26 pyrsa-sign

-rwxr-xr-x

1 root root

224 Mar 14 02:26 pyrsa-verify

lrwxrwxrwx

1 root root

lrwxrwxrwx

1 root root

lrwxrwxrwx

1 root root

9 Mar 14 01:19 python -> python3.8 9 Mar 14 01:19 python3 -> python3.8 41 Mar 14 01:19 python3.8 -> /usr/local/opt/

python-3.8.2/bin/python3.8 -rwxr-xr-x

1 root root 16343896 Mar 14 02:56 uwsgi

(lab_app) root@raspberrypi-zero:/var/www/lab_app#

You can see that "uwsgi" is listed at the end of the folder. Installation is complete, so let move swiftly to the next chapter where I'll show you how to configure NGinx so we can use it as our application's public-facing web server.

● 139

Raspberri Pi Full Stack UK 200609.indd 139

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 44 • Nginx configuration In the previous chapter you installed uWSGI, the application server, which is responsible for managing the operation of your Flask application. As per the architecture in Figure 44.51, when a web browser requests a page from your application, the request will first "hit" the NGinx public-facing webserver. NGinx will then pass the request to uWSGI (unless the request is for a static file), and finally uWSGI will invoke the relevant function of your Python Flask application. In this chapter, you will configure NGinx to work with uWSGI.

Figure 44.51: A graphical depiction of the Web Application Stack. By default, after installation, Nginx has a simple placeholder configuration file at /etc/ nginx/sites-enabled/default. This file is what configures the new NGinx to present this "Welcome" page:

Figure 44.57: NGinx start working as soon as installation is complete.

● 140

Raspberri Pi Full Stack UK 200609.indd 140

06-08-20 15:30

Chapter 44 • Nginx configuration

You want to replace the placeholder configuration file with a new one that points to your application. So, go ahead and delete the "default" file (as usual, you should work as root): (lab_app) root@raspberrypi-zero:/var/www/lab_app# rm /etc/nginx/ sites-enabled/default

Now, you can go ahead and create a new configuration file11 for Nginx, using Vim. You can create it in the application folder, and use a symbolic link to link it to the NGinx "sites-enabled" folder so that NGinx can read it: (lab_app) root@raspberrypi-zero:/var/www/lab_app# vim lab_app_nginx.conf

In the new Vim buffer, copy this content (I'll explain how this work in a moment): server { listen

80;

server_name localhost; charset

utf-8;

client_max_body_size 75M; location /static { root /var/www/lab_app/; } location / { try_files $uri @labapp; } location @labapp { include uwsgi_params; uwsgi_pass unix:/var/www/lab_app/lab_app_uwsgi.sock; } }

Next, create a symbolic link between this new file and the "sites-enabled" folder: (lab_app) root@raspberrypi-zero:/var/www/lab_app# ln -s /var/www/lab_app/ lab_app_nginx.conf /etc/nginx/conf.d/

Finally, restart NGinx so that it reads the new configuration file and applies the settings: (lab_app) root@raspberrypi-zero:/var/www/lab_app# /etc/init.d/nginx restart [ ok ] Restarting nginx (via systemctl): nginx.service.

The application is not ready to test yet, because you also need to complete the configuration of the uWSGI application server. You will do this in the next chapter. But before you get to this, give me a moment to explain what is happening inside the NGinx configuration file. 11. You can find this file in the project repository on Github: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_app_nginx.conf

● 141

Raspberri Pi Full Stack UK 200609.indd 141

06-08-20 15:30

Raspberry Pi Full Stack

Inside the lab_app_nginx.conf file, you can see one large block, titled "server". Inside this block you have some basic configuration directives12, such as "listen" and "server_name". The "listen" directive instructs NGinx to expose your website to port 80, which is the default port for HTTP applications. The "server_name" directive sets the name of the virtual server, which is "localhost". All of these directives on the header of the server block are default, and I haven't changed them for this application. Inside the "location" directive you can see parameter "/static", and a small block that contains the "root" directive. This "location" block is configuring Nginx so that if a client requests a file that is inside the "/static" directory, Nginx will look for the assets that the client has requested in the "/var/www/lab_app/" physical location on the server. At the moment, your application does not have any static assets, so this directive does not do anything useful. You will be using it later on, when you have HTML, CSS and Javascript static assets as part of your application. After "/static", you can see two more locations: 1. "location /" 2. "location @labapp" This is where you link the NGinx webserver to uWSGI so that dynamic requests can be evaluated. When the client requests an asset at the root of the test web application (such as http://raspberripi-zero.local/), NGinx will pass it to uWSGI. NGinx and uWSGi communicate via a socket file. The physical location of this file is /var/ www/lab_app/lab_app_uwsgi.sock. In the next chapter, I will show you how to configure uWSGI so that it creates and manages the socket file. Let's configure uWSGI.

12. You can see a full list of directives at this URL: https://nginx.org/en/docs/http/ngx_http_core_module.html

● 142

Raspberri Pi Full Stack UK 200609.indd 142

06-08-20 15:30

Chapter 45 • UWSGI configuration

Chapter 45 • UWSGI configuration Time to link everything together. In this chapter, you will configure the uWSGI application server so that it becomes the link between NGinx (which you configured in the previous chapter) and your "Hello World" Python Flask web application. In Figure 45.51, you are about to configure layer 4.

Figure 45.51: A graphical depiction of the Web Application Stack. The following instructions assume you are in the application directory ("/var/www/lab_ app"), and are working as super user (you should have the "#" symbol as your command prompt). Use Vim to create a new initialization13 file for uWSGI, titled "lab_app_uwsgi.ini14": root@raspberrypi-zero:/var/www/lab_app# vim lab_app_uwsgi.ini

In the Vim buffer, copy this content: [uwsgi] #application's base folder base = /var/www/lab_app #python module to import app = hello module = %(app)

13. You can learn more about uWSGI initialization files here: https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files 14. You can find this file in the project repository on Github: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/ master/lab_app_uwsgi.ini

● 143

Raspberri Pi Full Stack UK 200609.indd 143

06-08-20 15:30

Raspberry Pi Full Stack

home = %(base) pythonpath = %(base) #socket file's location socket = /var/www/lab_app/%n.sock #permissions for the socket file chmod-socket = 666 #the variable that holds a flask #application inside the module #imported at line #6 callable = app #location of log files logto = /var/log/uwsgi/%n.log

Save the buffer to file, and exit (":wq"). The initialization file starts with setting up the local variable "base" to store the path to the application on the file system. You can see this variable is used further down in this file. uWSGI configuration files support so-called "magic variables". These variables can inject various values in the configuration file when it is parsed by uWSGI. The documentation contains detailed information about available magic variables15. For example look at the line that starts with "socket". The socket directive determines the location and filename of the uWSGI socket (which NGinx uses for its communication with uWSGI). We have set the value of the socket to "/var/www/lab_app/%n.sock". Because the filename of the uWSGI ini file is "lab_app_uwsgi.ini", we get the value stored in "%n" to be ""lab_app_uwsgi". As a result, the filename of the socket file is ""lab_app_uwsgi.sock". We do the exact same thing at the end of the ini file where we set up the log file. An important line to notice is the one that starts with "app", towards the top of the file. This is where we define the application name. In this example, we set the value of the "app" variable to "hello", because the name of our Flask application file is "hello.py". This is how uWSGI knows which Python program to call. Before you can test your application, you must create the directory where the uWSGI log file will exist. The log file and its location is defined in the last line of the ini file: /var/log/ uwsgi/. Create the directory like this: (lab_app) root@raspberrypi-zero:/var/www/lab_app# mkdir -p /var/log/uwsgi 15. H  ere is a list and details of the available magic variables: https://uwsgi-docs.readthedocs.io/en/latest/Configuration. html#magic-variables

● 144

Raspberri Pi Full Stack UK 200609.indd 144

06-08-20 15:30

Chapter 45 • UWSGI configuration

Ok, that's it. Your test Python Flask application, uWSGI and NGinx are all configured and ready to fire up.

● 145

Raspberri Pi Full Stack UK 200609.indd 145

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 46 • UWSGI and Nginx configuration testing You have completed the set up of your Python Flask web application, the uWSGI application server, and NGinx webserver. These are layers 3, 4 and 5 in Figure 46.51. How do you know it works? Through testing, of course!

Figure 46.51: A graphical depiction of the Web Application Stack. The NGinx service is already running. To be sure, you can check the status of the service like this: (lab_app) root@raspberrypi-zero:/var/www/lab_app# sudo service nginx status nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-03-26 23:38:25 GMT; 5 days ago Docs: man:nginx(8) Process: 420 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_ process on; (code=exited, status=0/SUCCESS) Process: 427 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 428 (nginx) Memory: 9.9M CGroup: /system.slice/nginx.service ├─428 nginx: master process /usr/sbin/nginx -g daemon on; master_ process on; └─429 nginx: worker process Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

● 146

Raspberri Pi Full Stack UK 200609.indd 146

06-08-20 15:30

Chapter 46 • UWSGI and Nginx configuration testing

The output provides information about the processes running that belong to the NGinx service, and indicates that it is "active". As for uWSGI, you will need to start it manually. In the next chapter, I will show you how to set up uWSGI so that it starts automatically, as with NGinx. To manually start uWSGI, pass the uwsgi initialization file to the uwsgi binary that is located in the bin directory: (lab_app) root@raspberrypi-zero:/var/www/lab_app# bin/uwsgi --ini /var/www/ lab_app/lab_app_uwsgi.ini [uWSGI] getting INI configuration from /var/www/lab_app/lab_app_uwsgi.ini

In the output, you will get confirmation that uWSGI received its configuration from the ini file, and then wait for a request from NGinx. Next, open your browser and direct it to your Raspberry Pi. Remove the port number "8080" that you included in the previous test of the Flask application. In my setup, I pointed my browser to http://raspberrypi-zero.local (Figure 46.59).

Figure 46.59: The Hello World Flask application controlled by uWSGI. My browser made its request to NGinx; NGinx passed it on to uWSG, and uWSGi evaluated the response by invoking the Flask application. The implementation of the architecture in Figure 46.51 works! To wrap up this part of the project, there is only one thing left to do: set up uWSGI so that it begins automatically when your Raspberry Pi starts.

● 147

Raspberri Pi Full Stack UK 200609.indd 147

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 47 • Configure systemd to auto-start uwsgi On a production server, applications that work in the background to respond to client requests (like a web server or a database server) should start automatically after the computer boots. It should not be necessary for the administrator to start these services manually. On our current setup, there are two services that should be able to start automatically: NGinx and uWSGI. NGinx was set to start automatically on installation. uWSGI is not. In this chapter, I will show you how to set uWSGI as a system service that starts automatically after your Raspberry Pi boots up. Debian operating systems (to which Raspbian belongs) have a provision for this scenario by providing a system service called "systemd16". Systemd is useful for a wide range of system tasks. We'll use it to control the status of our uWSGI application server, and autostart it. To make this happen, create a new configuration file for uWSGI called "emperor.uwsgi. service", and save it in /etc/systemd/system/. Use Vim for this task: root@raspberrypi-zero:/var/www/lab_app# vim /etc/systemd/system/emperor. uwsgi.service

In the Vim buffer, copy this content17: [Unit] Description=uWSGI Emperor After=syslog.target [Service] ExecStart=/var/www/lab_app/bin/uwsgi --ini /var/www/lab_app/lab_app_uwsgi. ini # Requires systemd version 211 or newer RuntimeDirectory=uwsgi Restart=always KillSignal=SIGQUIT Type=notify StandardError=syslog NotifyAccess=all [Install] WantedBy=multi-user.target

16. You can learn more about systemd here: https://wiki.debian.org/systemd 17. You can find this file in the project repository on Github: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/emperor.uwsgi.service

● 148

Raspberri Pi Full Stack UK 200609.indd 148

06-08-20 15:30

Chapter 47 • Configure systemd to auto-start uwsgi

The important directives in this configuration file are in the "[Service]" block. The first line in "[Service]" is "ExecStart". This contains the command line needed to start the uWSGI service for your application. Notice it is the same command line you used to start uWSGI in the previous chapter. The rest of the directives are fairly standardised, and you can learn more about them in the documentation18. Save the buffer and quit Vim (":wq"). To start the uWSGI service, use the systemctl command: root@raspberrypi-zero:/var/www/lab_app# systemctl start emperor.uwsgi. service

You can check the status of the service like this: root@raspberrypi-zero:/var/www/lab_app# systemctl status emperor.uwsgi. service emperor.uwsgi.service - uWSGI Emperor Loaded: loaded (/etc/systemd/system/emperor.uwsgi.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2020-04-01 02:28:00 BST; 27min ago Main PID: 14683 (uwsgi) Status: "uWSGI is ready" Memory: 31.6M CGroup: /system.slice/emperor.uwsgi.service └─14683 /var/www/lab_app/bin/uwsgi --ini /var/www/lab_app/lab_ app_uwsgi.ini Apr 01 02:27:48 raspberrypi-zero.local systemd[1]: Starting uWSGI Emperor... Apr 01 02:27:48 raspberrypi-zero.local uwsgi[14683]: [uWSGI] getting INI configuration from /var/www/lab_app/lab_app_uwsgi.ini Apr 01 02:28:00 raspberrypi-zero.local systemd[1]: Started uWSGI Emperor.

It is working! You can refresh your browser to confirm your web application can still respond to client requests. This time, the uWSGI service operates as a system service, instead of having been invoked manually on the command line. Last, enable autostart: root@raspberrypi-zero:/home/pi# systemctl enable emperor.uwsgi.service

To confirm that uWSGI can start automatically, reboot your Raspberry Pi like this: root@raspberrypi-zero:/var/www/lab_app# reboot 18.

 ou can find detailed information about the various systemd configuration directives here: Y https://www.freedesktop.org/software/systemd/man/systemd.directives.html

● 149

Raspberri Pi Full Stack UK 200609.indd 149

06-08-20 15:30

Raspberry Pi Full Stack

Wait for a few minutes, and then refresh your web browser. If all goes well, you will see the familiar "Hello World!" Message, confirming the uWSGI service started automatically and is managing your Flask application. Congratulations. You have completed the setup of the infrastructure of your application. Starting with Part 7 of this project, you will start implementing the features that make up its functionality.

● 150

Raspberri Pi Full Stack UK 200609.indd 150

06-08-20 15:30

Part 7: Set up the database

Part 7: Set up the database

● 151

Raspberri Pi Full Stack UK 200609.indd 151

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 48 • Install the SQLIte3 database Your full-stack application will be collecting data from the Raspberry Pi sensor. If you choose to implement the Arduino extension as part of this project, your application will also be collecting sensor data from remote nodes. You need an efficient way to store and retrieve this data. This is why databases exist. Databases are designed for the storage of organised data so that retrieval is efficient and quick. When it comes to open source databases, we are spoilt for choice. There are dozens to choose from, each with an emphasis in a specific aspect of its operation. However, an important feature of many modern databases is the programming language that we, developers, use to communicate with them. This language is at the top 3 or 4 of our priorities as we set out to choose one for our application. There will be more the programming language later in this chapter. To choose an appropriate database for this full-stack application, I outlined the following criteria: • It must support standard SQL ("Structured Query Language") for interfacing with the user or other programs. • It must be able to operate with minimal resources on a Raspberry Pi Zero W, and be gentle with the SD card. • It must be easy to install. • It must be open-source (of course). • It must be well documented. I think the last four criteria are self-explanatory, so I will not spend any digital ink and time to explain them. I'd like to take a few minutes to expand on the first one. The Structure Query Language is a high-level language that is commonly found in data-related applications, especially databases. As a programming language, it is designed specifically for dealing with data stored in tables. Therefore, it is the opposite of a "general-purpose" language, like Python. SQL is very specific in its purpose. It is an example of a "domain-specific language". Since its beginning in the 1970s, SQL had grown to be the de-facto language used in a huge variety of databases, open or closed-source. Databases as diverse as MariaDB, MS SQL, MySQL, DB2 and Oracle, all support SQL. This means that you can learn a bit of SQL, and use this knowledge across all databases.

● 152

Raspberri Pi Full Stack UK 200609.indd 152

06-08-20 15:30

Chapter 48 • Install the SQLIte3 database

For our application, this is important. It means that if we choose a database that supports SQL and then write our application with standard SQL for storing and retrieving data, it is possible to switch to a different database if needed without having to change anything in our code. This, in turn, means that I can choose a simple, light weight, low volume database system to begin with. If my full stack application becomes popular and requires a more "beefy" database to support it, I can simply switch to the new database without changing any database-related code. After some extensive research on this matter, I decided that SQLite319 fits all five criteria. SQLite3 is a small, fast, reliable, and full-featured database. Many people don't realise it, but SQLite3 is the most widely used database in the world. You will find an SQLite3 database in every smart phone, tablet, and computer, and even in many embedded devices. As you can guess from its name, SQLite3 supports SQL. It is tiny, with the Linux pre-compiled executable measuring just 1.94 MB. It is designed so that the SQLite file format is stable and backwards compatible until at least 2050. It is also open-source. In my third criterion, I wrote that the database I'd like to work with must be very easy to install. SQLite3 is easy to install, as you are about to learn. First, log in to your Raspberry Pi as "pi" and switch to the super user: pi@raspberrypi-zero:/var/www/lab_app $ sudo su

Then use apt-get to install SQlite3: root@raspberrypi-zero:/var/www/lab_app# apt-get install sqlite3

That's it. A few seconds later you'll have SQLite3 installed in Raspbian. When you are ready, continue to the next chapter where I'll show you how to perform a few simple operations with your new database.

19.

Learn more about SQLite3 here: https://www.sqlite.org/index.html

● 153

Raspberri Pi Full Stack UK 200609.indd 153

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 49 • Hand-on with the SQLite3 CLI You can interact with SQLite3 programmatically, as you will learn very soon. That is, your Python program will contain code that will create new database records with your sensor data, and will also query the database to retrieve previously stored data. You can also interact with SQLite3 on the command line. SQLite3 and a CLI ("Command Line Interface") that is very similar to the one that Python has. In this chapter, I will show you the basics of the SQLite3 CLI20, and take the opportunity to teach you some basic SQL operations. To call the SQLite3 CLI, simply type "sqlite3" on the command line: root@raspberrypi-zero:/var/www/lab_app# sqlite3 SQLite version 3.27.2 2019-02-25 16:06:06 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite>

This will bring up the CLI, and give a prompt to which you can type SQLite3 directives, and regular SQL statements. To learn about the available SQLite3 directives, type ".help": sqlite> .help .archive ...

Manage SQL archives

.auth ON|OFF

Show authorizer callbacks

.backup ?DB? FILE

Backup DB (default "main") to FILE

.bail on|off

Stop after hitting an error.

.binary on|off

Turn binary output on or off.

.cd DIRECTORY

Change the working directory to DIRECTORY

.changes on|off

Show number of rows changed by SQL

Default OFF Default OFF

(… continues …)

This will display a long list of the available directives21. I have only included the first few lines in the command line except above. One directive to keep in mind is ".quit". This stops the SQLite3 CLI and takes you back to the Raspbian command line. Also keep in mind the ".tables" command, which shows you a list of tables in the working database.

20. 21.

You can find an excellent tutorial of the SQLite3 CLI here: https://www.sqlite.org/cli.html You can find details about all SQLite3 keywords in the Keyword Index: https://www.sqlite.org/keyword_index.html

● 154

Raspberri Pi Full Stack UK 200609.indd 154

06-08-20 15:30

Chapter 49 • Hand-on with the SQLite3 CLI

Because you entered SQLite3 by simply invoking "sqlite3", any work you do on the CLI (such as creating a new table and storing data in it), will not be persistent. As soon as you call ".quit", your data will be lost. Therefore, before you do any work, let's make sure that it's saved. Firstly, quit the SQLite3 CLI: sqlite> .quit root@raspberrypi-zero:/var/www/lab_app#

Secondly, go back in the SQLite3 CLI, but this time, pass the name of a file. This will become your SQLite3 database file, and it will contain your sensor data: root@raspberrypi-zero:/var/www/lab_app# sqlite3 lab_app.db SQLite version 3.27.2 2019-02-25 16:06:06 Enter ".help" for usage hints. sqlite>

Next, create a new table so that you can store data in it. For this purpose, you will use simple SQL22 statements. Create a new database table with the name "temperatures", in which you can store a date and time of creation for a new record, a sensor ID that we can use to keep track of the origin of the data, and, the temperature from the sensor. Each of those data items has a specific datatype23. Date and time are of type "rDatetime", the sensor ID is of type "text", and temperature is of type "numeric". Here's the statement for the creation of the "temperatures" table: sqlite> create table temperatures (rDatetime datetime, sensorID text, temp numeric);

This statement is composed of three parts: • The "create table" SQL command, used to create a new table. • The name of the table that you are about to create, in this case "temperatures". • The list of columns, including the name and datatype of each one. Confirm that the new table exists like this: sqlite> .tables temperatures

22. 23.

You can learn details of the SQL language as implemented in SQLite3 here: https://www.sqlite.org/lang.html You can learn more about SQLite3 datatypes here: https://www.sqlite.org/datatype3.html

● 155

Raspberri Pi Full Stack UK 200609.indd 155

06-08-20 15:30

Raspberry Pi Full Stack

You can see that "temperatures" is listed. This means that you can store data in it. Do the same once more, and create a table named ".humidities": sqlite> create table humidities (rDatetime datetime, sensorID text, hum numeric);

Confirm you have created two tables: sqlite> .tables temperatures humidities

Very good. Now, let's create a couple of records in each table. For this purpose, you will use the "insert" SQL command. sqlite> insert into temperatures values (datetime(CURRENT_TIMESTAMP),"1",25); sqlite> insert into temperatures values (datetime(CURRENT_TIMESTAMP),"1",25.10);

These two statements will create two new records in the "temperatures" table. Just like with the "create" statements, the "insert" statements are composed of three parts: • The "insert" command, used to create a new record. • The name of the table in which you want to create the new record after the "into" keyword. In this example, it is "temperatures". • The values for the new record after the "values" keyword. It is important to list the values in the same order they were created, with the correct datatype. As you can see in the example, the "sensor id" is enclosed inside double-quotes, because it is of datatype "text". The third parameter is a number, so it is simply a number without quotes. In this example, I have used the SQL function "datetime24" to get the current time and date stamp from the "CURRENT_TIMESTAMP25" SQLite keyword and converted it into the appropriate format. This saved me the hassle of entering the date and time manually. Now that you have created a couple of new records in the database, lets try to retrieve them. For this, use the "select" function: sqlite> select * from temperatures; 2020-04-03 04:47:00|1|25 2020-04-03 04:47:25|1|25.1

24. 25.

Learn more about "datetime": https://www.sqlite.org/lang_datefunc.html Learn more about the available SQLite keywords (including CURRENT_TIMESTAMP) here: https://www.sqlite.org/lang_keywords.html

● 156

Raspberri Pi Full Stack UK 200609.indd 156

06-08-20 15:30

Chapter 49 • Hand-on with the SQLite3 CLI

There's your data! As a mini-exercise, use the CLI to create two new records in the "humidities" table, and then retrieve the data. Before we conclude this chapter, I'd like to show you a few of other useful SQL and SQLite3 commands. The first ones are "begin" and "commit;", which allow you to bundle together multiple SQL statements. Here is an example: sqlite> begin; sqlite> create table humidities (rDatetime datetime, sensorID text, hum numeric); sqlite> insert into humidities values (datetime(CURRENT_TIMESTAMP),"1",51); sqlite> insert into humidities values (datetime(CURRENT_TIMESTAMP),"1",51.10); sqlite> commit;

In this example, you can enclose multiple statements between "begin" and "commit". When you hit the return key after "commit;", the enclosed statements will be executed together. The second one is the ".schema" SQLite command. It works like this: sqlite> .schema temperatures CREATE TABLE temperatures (rDatetime datetime, sensorID text, temp numeric);

The ".schema" will show you the "create tables" command that you used to create a new table. This is a simple way to inspect a table to confirm its columns and datatypes. To exit the SQLite3 CLI, use ".exit". Of course, there is a lot more to learn. SQL is rich and can get very complicated. You can achieve a lot with very little. For the purposes of this project, there's only a couple of additional SQL features that you will be using. We will learn these later.

● 157

Raspberri Pi Full Stack UK 200609.indd 157

06-08-20 15:30

Raspberry Pi Full Stack

Part 8: Styling with Skeleton

● 158

Raspberri Pi Full Stack UK 200609.indd 158

06-08-20 15:30

Chapter 50 • Static assets and the Skeleton boilerplate CSS

Chapter 50 • Static assets and the Skeleton boilerplate CSS Modern web pages are composed of static and dynamic assets. A static asset is a component of a web page that doesn't change. A dynamic asset, on the other hand, is anything on a page that changes depending on user or browser settings. In Figure 50.60 you can see several static and dynamic elements of this page of your fullstack application.

Figure 50.60: Examples of static and dynamic assets in a webpage. In this figure, the Dynamic labels point to the temperature chart, and to the table of temperatures. These two elements are constructed from data that is fetched from the SQLite3 database. The shape of the chart and date depend on the date range that the user has selected in the date widgets. The Static label points to a button, some text elements, and radio widgets. These items are always the same, regardless of user choices. The are generated by static HTML, and are rendered always in the same way by the browser. Let's have a closer look at static and dynamic assets, and zoom into the code that handles them. You will write this code later in this project. In Figure 50.61, you can see an excerpt of the lab_env_db.html file that produces the page you see in Figure 50.60.

● 159

Raspberri Pi Full Stack UK 200609.indd 159

06-08-20 15:30

Raspberry Pi Full Stack

Figure 50.61: Examples of static code in a webpage. The page contains a lot of static HTML code. The two arrows point to two CSS ("Cascading Style Sheet") files that are stored in the /static/css/ directory of the full stack application, and are responsible for controlling the look and feel of the page. I will write more about these CSS files shortly. What is important to point out here, is that these two files are static, not changing, and always loaded when the page loads. You can see another CSS file on line 15, which is fetched from fonts.googleapis.com. This is also static, but stored on the Cloud instead of the Raspberry Pi. In Figure 50.62 you can see another excerpt of the lab_env_db.html file that produces the page you see in Figure 50.60.

Figure 50.62: Examples of dynamic code in a webpage.

● 160

Raspberri Pi Full Stack UK 200609.indd 160

06-08-20 15:30

Chapter 50 • Static assets and the Skeleton boilerplate CSS

In this example, the arrows are pointing to two "div" elements where the browser will eventually render the two charts. The charts are generated by Javascript that exists elsewhere in the HTML file, using data generated by the Python application. You will spend a lot of time working with Javascript later. But first, let's work with CSS. CSS is a language that complements HTML. It is responsible for making a web page look good. It controls the way text looks and the position of various elements like headers and images. Over the years, CSS has become very sophisticated, and although not Turing-complete26, it offers a lot of capabilities for a general-purpose programming language. For us, CSS provides a way to style our full-stack application so it looks pleasant and useful. As with other languages, it is not necessary to write CSS code from scratch. Personally, my web page design skill is very limited, so I prefer to build on the work of others. There are many boilerplate CSSs in the public domain that you can integrate into your project which will make your site look great from the very beginning. One such option is "Skeleton". Skeleton is a simple to use CSS boilerplate set of CSS files. It is also responsive, which is a special term meaning that your Skeleton-styled webpages will work on a mobile browser, as well as on desktop browsers. Skeleton is also very small. Its main CSS file is only around 400 lines. Skeleton has everything we need for this project: • • • • • •

Styled headers and paragraph text, Buttons Widgets (like radio buttons an text fields). Lists Tables Ability to display content in columns of various sizes

So, with Skeleton, you will make your full-stack application pages look good. First things first: in the next chapter, I will show you how to set up the directories in which you will store the necessary static assets (such as the Skeleton CSS files and Javascript).

26.

Learn more about Turing Completeness: https://en.wikipedia.org/wiki/Turing_completeness

● 161

Raspberri Pi Full Stack UK 200609.indd 161

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 51 • Set up the static assets directory In this chapter I will show you how to set up the directories that will hold your application's static assets (CSS, Javascript, and image files) . I will also show you how to set up a simple HTML file so that you can test the Skeleton CSS boilerplate . In the chapter titled "Chapter 44 - Nginx configuration", you created a configuration file for Nginx . I have copied this configuration file here for easy reference: server { listen

80;

server_name localhost; charset

utf-8;

client_max_body_size 75M; location /static { root /var/www/lab_app/; } location / { try_files $uri @labapp; } location @labapp { include uwsgi_params; uwsgi_pass unix:/var/www/lab_app/lab_app_uwsgi.sock; } }

In bold, I have marked the "/static/" location in which Nginx (because of this configuration file), expects to find any static files . The "static" folder does not exist yet, so lets create it . Log in to your Raspberry Pi as "pi" . Switch to the super user, and change to the application directory: pi@raspberrypi-zero:/var/www/lab_app $ sudo su root@raspberrypi-zero:/var/www/lab_app#

Create a new directory, called "static": root@raspberrypi-zero:/var/www/lab_app# mkdir static

Move into the new directory: root@raspberrypi-zero:/var/www/lab_app# cd static

● 162

Raspberri Pi Full Stack UK 200609.indd 162

06-08-20 15:30

Chapter 51 • Set up the static assets directory

And here, create three new directories: root@raspberrypi-zero:/var/www/lab_app/static# mkdir css root@raspberrypi-zero:/var/www/lab_app/static# mkdir javascript root@raspberrypi-zero:/var/www/lab_app/static# mkdir images

Now, create a simple HTML file that you can use to test your Skeleton CSS styling in the next lecture. Use Vim, and name this file "a_static_file.html". You can find the code for this simple HTML file in the project repository27.

Static page

This is an example of a static page

Neat, isn't it?

Nothing fancy here, just a bare-minimal HTML file. Use your browser to fetch this file. Point your browser to http://raspberrypi-zero.local/static/a_static_file.html (of course, replace the domain name with your Raspberry Pi's actual IP address or hostname). You should see something like the example in Figure 51.63.

Figure 51.63: Examples of dynamic code in a webpage. The browser renders the HTML with its default styling. With the help of the Skeleton CSS, and just a single extra line of code in the HTML file, you can make this example page look noticably better. I'll show you how to do this in the next chapter.

27.

 ere is the simple HTML file in the project Github repository: H https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/a_static_file.html

● 163

Raspberri Pi Full Stack UK 200609.indd 163

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 52 • Introducing the Skeleton boilerplate CSS In the last chapter, you created a simple HTML file titled "a_static_file.html ". The web browser will render it using its default setting. It looks ok, but not great. In this and the next couple of chapters, you'll use Skeleton to make this page look great. I'll show you how to install it in your Python Flask web application, and how to configure a_static_file.html to use it. Start by visiting getskeleton.com from where you can download the required files. On the home page, you'll find the Download button. Click on it to download the boilerplate ZIP file (Figure 52.64).

Figure 52.64: Click on Download to get the Skeleton boilerplate files. The ZIP file should take a couple of seconds to download. When the download is complete, expand it. The expanded archive contains two directories: "css" and "images" (Figure 52.65).

Figure 52.65: The contents of the Skeleton boilerplate CSS archive.

● 164

Raspberri Pi Full Stack UK 200609.indd 164

06-08-20 15:30

Chapter 52 • Introducing the Skeleton boilerplate CSS

Inside the "css" directory are two css files. The one titled "skeleton.css" is the file that contains the CSS code that will style your web application pages. Open this file in a text editor and have a look at its code. Here is a small segment: /* Typography –––––––––––––––––––––––––––––––––––––––––––––––––– */ h1, h2, h3, h4, h5, h6 { margin-top: 0; margin-bottom: 2rem; font-weight: 300; } h1 { font-size: 4.0rem; line-height: 1.2;

letter-spacing: -.1rem;}

h2 { font-size: 3.6rem; line-height: 1.25; letter-spacing: -.1rem; } h3 { font-size: 3.0rem; line-height: 1.3;

letter-spacing: -.1rem; }

h4 { font-size: 2.4rem; line-height: 1.35; letter-spacing: -.08rem; } h5 { font-size: 1.8rem; line-height: 1.5;

letter-spacing: -.05rem; }

h6 { font-size: 1.5rem; line-height: 1.6;

letter-spacing: 0; }

This code styles the header tags and controls the font size and spacing around each header. There is a lot of CSS code in this file, controlling how elements like columns, buttons, links, forms and tables look. You can learn how Skeleton CSS works and looks by looking at the examples available on the Get Skeleton website and comparing them against the actual code that generates them. For example in Figure 52.65, you can see how Skeleton-styled buttons look, and the CSS code that produces the result.

● 165

Raspberri Pi Full Stack UK 200609.indd 165

06-08-20 15:30

Raspberry Pi Full Stack

Figure 52.66: A button example from the Skeleton CSS. It is worth spending a bit of time to become familiar with the Skeleton CSS file. The second file I recommend you to look at is the index.html, at the root of the Skeleton folder. This is a sample HTML file that shows you how to use Skeleton in your webpages. Open it in a text editor. The most important lines from this file are these:



● 166

Raspberri Pi Full Stack UK 200609.indd 166

06-08-20 15:30

Chapter 52 • Introducing the Skeleton boilerplate CSS

The first line imports the font that Skeleton uses from the Google fonts repository: Raleway. The second line (ignoring the comments, imports the normalize.css file. The purpose of this file is to reset the browser's default stylesheet to a common baseline. The third line imports the actual skeleton.css file which is responsible for the final styling. You will need to use the same lines inside each page of your web application to make use of Skeleton. Let's do this in the next chapter.

● 167

Raspberri Pi Full Stack UK 200609.indd 167

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 53 • Copying files using SFTP In this chapter you will apply Skeleton CSS files to your web application. To do this, you must complete two steps: 1. Copy the Skeleton files to the Raspberry Pi, in the already created static folder. 2. E  dit your test HTML file so that it imports the Google font and the two Skeleton CSS files. Begin with the copying of the files. There are a few different ways to do this. The easiest is to use SFTP, or "Secure File Transfer Protocol". SFTP is based on SSH, and you can choose to use it on the command line, or via a graphical utility. I choose to use a graphical STFP utility because it is easier to use, especially when it comes to manipulating files in bulk. My SFTP utility of choice is Cyberduck28. There are many excellent utilities that work in the same way, so feel free to use whichever you are comfortable with. Install Cyberduck and start it. Go to the bookmarks pane. This is where you can create bookmarks for the various resources you want to work with. You can see mine in Figure 53.67.

Figure 53.67: A button example from the Skeleton CSS. If this is the first time you are using this utility, the list will be empty. To create a bookmark, click on the "+" button, marked as "1" in Figure 53.67. This will present you with a dialogue like in Figure 53.68. 28.

Get your copy of Cyberduck from https://cyberduck.io/

● 168

Raspberri Pi Full Stack UK 200609.indd 168

06-08-20 15:30

Chapter 53 • Copying files using SFTP

Figure 53.68: My Cyberduck bookmark for my Full Stack application. Here are the required settings: • • • • • •

Type of connection: "SFTP". Nickname: Something reasonable. Hostname: Your Raspberry Pi IP address or hostname. Username: root Password: whichever you set for the Raspberry Pi root user. Path: /var/www/lab_app/static

Close the window to commit the changes, and then double-click on the bookmark. You will be asked if you allow connection to a host with an unknown fingerprint. You should choose "Allow", and you will not get this question again. You should now see the file system on your Raspberry Pi. You may need to navigate to the /var/www/lab_app/static/ folder if Cyberduck did not take you straight there. You can expand the directories, and interact with the filesystem as you normally do on your computer (Figure 53.69).

● 169

Raspberri Pi Full Stack UK 200609.indd 169

06-08-20 15:30

Raspberry Pi Full Stack

Figure 53.69: With this SFTP utility, you can browse the file system of your Raspberry Pi. With your mouse, select the two CSS files from your computer, and drag and drop them inside the CSS directory of your web application's static directory . You can see the result in Figure 53 .69 . You now have the Skeleton CSS files in your static application directory . Let's use them . Use Vim to open the test HTML file titled "a_static_file .html" that you created previously . Copy and paste the three import lines you learned about in the previous chapter just before the close "head" tag . Your HTML file should now look like this:

Static page



● 170

Raspberri Pi Full Stack UK 200609.indd 170

06-08-20 15:30

Chapter 53 • Copying files using SFTP

This is an example of a static page

Neat, isn't it?

I have highlighted the inserted code in bold font. Save the file, and exit Vim. Go to your browser and fetch the test HTML page: http:// raspberrypi-zero.local/static/a_static_file.html. It should look like the example in Figure 53.70.

Figure 53.70: With this SFTP utility, you can browse the file system of your Raspberry Pi. As you can see, the text in this page is now using Skeleton styling instead of the browser default. Excellent! While we are still on the topic of HTML and web pages, in the next chapter I will show you how to create, style and use a HTML template in Flask.

● 171

Raspberri Pi Full Stack UK 200609.indd 171

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 54 • Flask templates Your web page will not be very useful as a static HTML file. To create a truly useful web application, pages must contain dynamic content. For example, in Figure 54.71, you can see the current values for temperature and humidity in my lab. These values update every ten seconds, and are examples of dynamic content.

Figure 54.71: The temperature and humidity values in this page are examples of dynamic content. Flask has a templating mechanism. A Flask template is a HTML file that contains placeholders for dynamic content. You can pass dynamic content as variables of any kind (as long as it is valid in Python), and then display the value stored in the variables content in the rendered HTML page. You will use Flask templates extensively in your full stack application. In this chapter, I will show you how to use Flask templates with a very simple example. You will work on the hello.py file that you created in a previous chapter. You can find detailed information on Flask templates at https://flask.palletsprojects.com/en/1.1.x/quickstart/#rendering-templates. Here's the process: • Enable the templating capability in hello.py. • Enable debugging so that you can troubleshoot programming errors (more about this in the next chapter). • Replace the static content of the example route with a variable. • Create a simple Flask template that displays the value of the variable from step 3.

● 172

Raspberri Pi Full Stack UK 200609.indd 172

06-08-20 15:30

Chapter 54 • Flask templates

Let's start . Use Vim to edit hello .py . Edit the file so that it looks like this: from flask import Flask from flask import render_template app = Flask(__name__) app.debug = True @app.route("/") def hello(): return render_template('hello.html', message="Hello World!") if __name__ == "__main__": app.run(host='0.0.0.0', port=8080)

I have used the bold font to highlight the new code . You must import the "render_template" method because this makes it possible to process template files . The code "app .debug = True" turns on debugging . If an error is present in the application, error messages or warnings will be printed on the console . This is very useful during the application development process, as you will see in detail in the next chapter . Inside the "hello()" method, I replaced the original return ""Hello World!" " statement (which returned a static value to the browser) with a call to the "render_template" method . This method take two arguments . The first is the name of the HTML template to use . The second is a list of one or more variables that you want to pass to the template . In this case, I am passing a single variable with a string value . Simple, right? Your application file (hello .py) is complete . Next, create the template file . Browse to the root of your application directory and create a new directory with the name "templates": root@raspberrypi-zero:/var/www/lab_app# mkdir templates

The name is important . Flask expects to find template files inside the "templates" directory . Create a new file called "hello .html" . This is the filename for the template file that the render_template method will call . In this file, copy the following content:

Static page



{{ message }}

Most of this should be familiar to you . I have highlighted with bold font the only new element . This is the variable you defined in hello .py, enclosed in curly brackets . This is the placeholder for the dynamic content of this web page . The value stored in the "message" variable in hello .py will appear within the "h1" brackets in the hello .html file when it is rendered in your web browser . Save the file . Because you have changed the Python program which is controlled by uWSGI, you must restart uWSGI . This will fetch a fresh copy of the program from the disk . Try it out: root@raspberrypi-zero:/var/www/lab_app/templates# systemctl restart emperor. uwsgi.service

Go to your browser, and fetch the page . You can see my example in Figure 54 .72 .

Figure 54.72: This content is produced by a Flask template. The "Hello World!" string was passed as a variable.

● 174

Raspberri Pi Full Stack UK 200609.indd 174

06-08-20 15:30

Chapter 54 • Flask templates

What you see in your browser has been generated by the hello.html template, with the value of the variable that was passed through the render_template method. As you can see, Flask provides us with a simple but powerful templating mechanism. You will use this mechanism to pass large volumes of data to the template pages, that will render large tables and graphics to the user's browser. Things are about to get more complicate, as you are about to begin the implementation of the application's features. Before this, you should be familiar with the process of debugging your Flask application. I'll show you how to do this in the next chapter.

● 175

Raspberri Pi Full Stack UK 200609.indd 175

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 55 • Debugging a Flask app Debugging is an important aspect of application development. Python (like every modern programming language) provides debugging tools to make finding and correcting defects in your code efficient. In this chapter, I will show you how to easily debug simple defects that will find their way into your Python programs. By no means is this demonstration is complete! A web application is made of many components. Python and Flask are just a couple of them. You application can have defects elsewhere. Javascript, HTML/CSS, database, and Cloud API errors are also possible. Each one has its own tools and techniques for detecting and correcting. You have to start somewhere, and Python debugging is a good place. I have deliberately introduced a defect in my hello.py file. When I use my browser to access the "/" route, I see an error message from the web server. It looks like the example in Figure 55.73.

Figure 55.73: A programming or configuration bug is also always responsible for an Internal Server Error. Let's try to fix this bug. In the previous chapter, remember you inserted this line in your hello.py program: app.debug = True

This turns on the debugging tool that can provide you with a lot of information relating to programming defects. Often, this information points directly to the bug. To make use of the debugger tool, you must run the Flask application on the command line, instead of uWSGI: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python hello.py

Refresh your browser. Don't forget to append ":8080" to the Raspberry Pi hostname or IP address since you are now operating the application on the command line. You will see a very detailed error message, like in the example in Figure 55.74.

● 176

Raspberri Pi Full Stack UK 200609.indd 176

06-08-20 15:30

Chapter 55 • Debugging a Flask app

Figure 55.74: A programming or configuration bug is also always responsible for an Internal Server Error. The output of the debugging tool contains a lot of information. Most of it is not very useful. Somewhere in there you will find clues of what the problem might be. In this example, Python has indicated the exact location and nature of the problem. There is an undefined entity in line 9. You can find this information in the console. In Figure 55.75 you can see the same bug report as the one in the browser.

Figure 55.75: You can find clues in the command line.

● 177

Raspberri Pi Full Stack UK 200609.indd 177

06-08-20 15:30

Raspberry Pi Full Stack

With this information, you can go into the application file and make the fix. Type Ctrl-C to stop the application, and use Vim to open hello.py, removing the offending content on line 9. To confirm you fixed the problem, run the application again on the command line and refresh your browser. It should work properly this time. When you are satisfied your program is free from defects, you can turn off the debugger. In hello.py, change: app.debug = True

To: app.debug = False

With the basics of debugging a Flask application covered, lets continue to the next part of the project, where you will start with the development of the first feature: capturing sensor data, storing it in a database, and displaying it in a browser.

● 178

Raspberri Pi Full Stack UK 200609.indd 178

06-08-20 15:30

Part 9: Capture and record sensor data

Part 9: Capture and record sensor data

● 179

Raspberri Pi Full Stack UK 200609.indd 179

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 56 • Introduction to Part 9 At this point, you have created the architecture of your full-stack application, and have set up the main components. You have implemented the web server, application server, and database server. From now on, you will work towards implementing features in the Flask application file, and HTML template files (which also include the Javascript code). In the chapters of this section, you will implement the code that handles the DHT22 sensor. You will write code that takes sensor readings, stores and retrieves the readings from the database, and displays them in the command line and browser. You have already learned how to install the DHT22 Python library and use it in a previous chapter. You will repeat the process in the application's Python Virtual Environment, and then set up your Flask application to make use of the DHT22 sensor. You will then expand on the knowledge you gained in Part 7 and the SQLite3 CLI to record sensor readings to the database from the Flask application. Without delay, let's go ahead and install the DHT Python library in the application's Python Virtual Environment.

● 180

Raspberri Pi Full Stack UK 200609.indd 180

06-08-20 15:30

Chapter 57 • Install the DHT library and the rpi-gpio module

Chapter 57 • Install the DHT library and the rpi-gpio module In this chapter, you will set up the rpi-gpio and DHT Python modules. This is work you have done before. Now, you'll do it inside your application's working directory, and the Python Virtual Environment. Once completed, your Flask application will have access to these resources. Let's begin. Start by logging into your Raspberry Pi, as user "pi". Switch to super user, and change to the application's working directory: pi@raspberrypi-zero:~ $ sudo su root@raspberrypi-zero:/home/pi# cd /var/www/lab_app root@raspberrypi-zero:/var/www/lab_app# . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

Ensure "(lab_app)" and "root" appear in the front of the command line. Next, install rpi.gpio. It's a good practice to update pip if you haven't done so for a few days: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install --upgrade pip Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Requirement already up-to-date: pip in ./lib/python3.8/site-packages (20.0.2) (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install rpi.gpio Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Successfully installed rpi.gpio-0.7.0

To make sure that rpi.GPIO works on your Python virtual environment, try the blinking LED script, using the Python interpreter of your Python Virtual Environment. Using Vim, create a new Python script called "blink.py": (lab_app) root@raspberrypi-zero:/var/www/lab_app# vim blink.py

Copy this content: import RPi.GPIO as GPIO

## Import GPIO Library

import time

## Import 'time' library (for 'sleep')

GPIO.setwarnings(False) pin = 7

## We're working with pin 7

GPIO.setmode(GPIO.BOARD)

## Use BOARD pin numbering ## You can use BCM 4 if you prefer

● 181

Raspberri Pi Full Stack UK 200609.indd 181

06-08-20 15:30

Raspberry Pi Full Stack

GPIO.setup(pin, GPIO.OUT)

## Set pin 7 to OUTPUT

for i in range (0, 20):

## Repeat 20 times

GPIO.output(pin, GPIO.HIGH)

## Turn on GPIO pin (HIGH)

time.sleep(1)

## Wait 1 second

GPIO.output(pin, GPIO.LOW)

## Turn off GPIO pin (LOW)

time.sleep(1)

## Wait 1 second

GPIO.cleanup() And run it: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python blink.py

The LED on physical Pin 7 should blink. This means your rpi.GPIO module is successfully installed, so you can continue. Before continuing with the DHT module, try out the button script. Create the script: (lab_app) root@raspberrypi-zero:/var/www/lab_app# vim button.py

Copy this content: import RPi.GPIO as GPIO

## Import GPIO Library

inPin = 8

## Switch connected to pin 8

GPIO.setmode(GPIO.BOARD)

## Use BOARD pin numbering

GPIO.setup(inPin, GPIO.IN)

## Set pin 8 to INPUT

while True:

## Do this forever

value = GPIO.input(inPin) ## Read input from switch if value:

## If switch is released

print("Pressed") else:

## Else switch is pressed

print("Not Pressed") GPIO.cleanup()

And run it inside the Python Virtual Private Environment: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python button.py Not Pressed Not Pressed Not Pressed Not Pressed Pressed

● 182

Raspberri Pi Full Stack UK 200609.indd 182

06-08-20 15:30

Chapter 57 • Install the DHT library and the rpi-gpio module

Pressed Pressed

Excellent! Let's continue with the DHT module. Git is already installed on your Raspberry Pi. Use it to fetch the DHT module repository into your application directory: (lab_app) root@raspberrypi-zero:/var/www/lab_app# git clone https://github. com/adafruit/Adafruit_Python_DHT.git

Move into the directory that Git created, titled "Adafruit_Python_DHT", and set up the module: (lab_app) root@raspberrypi-zero:/var/www/lab_app# cd Adafruit_Python_DHT (lab_app) root@raspberrypi-zero:/var/www/lab_app/Adafruit_Python_DHT# python setup.py install running install running bdist_egg running egg_info writing Adafruit_DHT.egg-info/PKG-INFO writing dependency_links to Adafruit_DHT.egg-info/dependency_links.txt writing top-level names to Adafruit_DHT.egg-info/top_level.txt reading manifest file 'Adafruit_DHT.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' … Installed /var/www/lab_app/lib/python3.8/site-packages/Adafruit_DHT-1.4.0py3.8-linux-armv6l.egg Processing dependencies for Adafruit-DHT==1.4.0 Finished processing dependencies for Adafruit-DHT==1.4.0

Do a quick test to ensure that the DHT module is properly installed. Run the example script in the examples directory: (lab_app) root@raspberrypi-zero:/var/www/lab_app/Adafruit_Python_DHT# cd examples/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/Adafruit_Python_DHT/ examples# python AdafruitDHT.py 2302 17 Temp=24.8*

Humidity=52.7%

Did you get a temperature and humidity reading from the sensor? Excellent. This means the Python DHT sensor module is properly installed on your application Python Virtual Environment and you will be able to use it with your Flask application.

● 183

Raspberri Pi Full Stack UK 200609.indd 183

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 58 • Display the current sensor values in the browser You are building a web application, so you want to implement all end-user interactions that happen via the web browser. This application will eventually contain several individual features. In this chapter, you will create the very first feature: The ability to show current sensor values in the web browser, and update the page contents automatically. In the previous chapter, you installed the Python DHT module on the application's Python Virtual Environment, and tested it to make sure it works. Now, you'll take the sample code from the example Python script, and use it in a new Flask application file. You will also create a simple HTML template that will work with the Flask route that you are about to create. Start by logging into your Raspberry Pi, as user "pi". Switch to super user, and change to the application's working directory: pi@raspberrypi-zero:~ $ sudo su root@raspberrypi-zero:/home/pi# cd /var/www/lab_app root@raspberrypi-zero:/var/www/lab_app# . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

Ensure that "(lab_app)" and "root" appear in the front of the command line. Use Vim to create a new Flask application file titled "lab_app.py": (lab_app) root@raspberrypi-zero:/var/www/lab_app# vim lab_app.py

In the buffer, copy the following code (this is file "lab_app_v1.py" from the project repository29). from flask import Flask, request, render_template import sys import Adafruit_DHT app = Flask(__name__) app.debug = True # Make this False if you are no longer debugging @app.route("/") def hello(): return "Hello World!" @app.route("/lab_temp") def lab_temp(): humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.AM2302, 17) 29.

You will find this file at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_app_v1.py

● 184

Raspberri Pi Full Stack UK 200609.indd 184

06-08-20 15:30

Chapter 58 • Display the current sensor values in the browser

if humidity is not None and temperature is not None: return render_template("lab_temp.html",temp=temperature,hum=humidity) else: return render_template("no_sensor.html")

if __name__ == "__main__": app.run(host='0.0.0.0', port=8080)

Save the buffer and exit Vim. The contents of this file should look familiar. I have preserved the base route "/" from the previous examples and added a new route called "/lab_temp". This route has a method called "lan_temp", in which you can see code I copied from the Adafruit example. The page is constructed by using a template file called "lab_temp.html". There are two parameters, "temp" and "hum". Their values can be rendered in the browser. I have also added a second template, called "no_sensor.html", in case that the sensor is not working properly. For the template files, go into the "templates" directory and use Vim to create two new files: "lab_temp.html", and "no_sensor.html": (lab_app) root@raspberrypi-zero:/var/www/lab_app# cd templates/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/templates# vim lab_temp. html (lab_app) root@raspberrypi-zero:/var/www/lab_app/templates# vim no_sensor. html

Here are the contents of "lab_temp.html" (from "lab_temp_v1.html" in the project repository30):

Lab Conditions by RPi



30.

You will find this file at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_temp_v1.html

● 185

Raspberri Pi Full Stack UK 200609.indd 185

06-08-20 15:30

Raspberry Pi Full Stack





Real time lab conditions Temperature: {{"{0:0.1f}".format(temp) }}°C Humidity: {{"{0:0.1f}".format(hum)}}%

This page refreshes every 10 seconds

Points to note: In the header, you will see the "refresh" meta tag, and content "10". This instructs the browser to refresh the page every 10 seconds. In the body, you will see the template parameters "temp" and "hum", formatted with a single decimal point. Flask uses a formatting language called "Jinja". You can learn more about Jinja in its documentation31. I have applied the Skeleton boilerplate as in the previous examples. Here are the contents of "no_sensor.html" (from "no_sensor_v1.html" in the project documentation32):



Sorry, can't access the sensor!

31. 32.

You can learn more about Jinja here: https://jinja.palletsprojects.com/en/2.11.x/templates/ You will find this file at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/no_sensor_v1.html

● 186

Raspberri Pi Full Stack UK 200609.indd 186

06-08-20 15:30

Chapter 58 • Display the current sensor values in the browser

There's not much to say about this file… Before restarting uWSGI, you must make a subtle change to the uWSGI configuration file "lab_app_uwsgi .ini" . You must update the "app" variable to point to the new Flask application file ("lab_app") instead of the old one ("hello") . Let's do it now: (lab_app) root@raspberrypi-zero:/var/www/lab_app# vim lab_app_uwsgi.ini

Here is the new configuration file, with the only change highlighted in bold: [uwsgi] #application's base folder base = /var/www/lab_app #python module to import app = lab_app module = %(app) home = %(base) pythonpath = %(base) #socket file's location socket = /var/www/lab_app/%n.sock #permissions for the socket file chmod-socket = 666 #the variable that holds a flask #application inside the module #imported at line #6 callable = app #location of log files logto = /var/log/uwsgi/%n.log

Save the file, and continue with the test . Restart uWSGI so the application server reloads the Flask file: (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl restart emperor. uwsgi.service (lab_app) root@raspberrypi-zero:/var/www/lab_app#

In your browser, fetch the new "/lab_temp" route . In my implementation, the full URL looks like this: "http://raspberrypi-zero .local/lab_temp" . You can see what this page looks like in Figure 58 .76 .

● 187

Raspberri Pi Full Stack UK 200609.indd 187

06-08-20 15:30

Raspberry Pi Full Stack

Figure 58.76: The current sensor values in my lab. If you see an error message, try the debugging procedure you learned about in the debugging chapter. This will give you clues about where the problem is and how to fix it. In most cases, problems are caused by typos, or errors in copy/paste. In the next step, prepare the database to store the sensor data.

● 188

Raspberri Pi Full Stack UK 200609.indd 188

06-08-20 15:30

Chapter 59 • Create a database to store sensor data

Chapter 59 • Create a database to store sensor data In a previous chapter, you learned how to interact with SQLite3 via the CLI. If you completed the practical exercise in the chapter, you already have a database file titled "lab_app. db". This file implements your application database. The application database (that persists in the lab_app.db file) contains two tables: • temperatures • humidities These tables are ready to use, and store data from your sensor. I'll show you how to store sensor data with the help of a simple Python script in the next chapter. For now, I suggest you double check your application database file is properly set up and ready to work with the Python script. As usual, start by logging into your Raspberry Pi, as user "pi". Switch to super user, and change to the application's working directory: pi@raspberrypi-zero:~ $ sudo su root@raspberrypi-zero:/home/pi# cd /var/www/lab_app root@raspberrypi-zero:/var/www/lab_app# . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

Start SQLite3, and load the application database so you can inspect its tables and table schema: (lab_app) root@raspberrypi-zero:/var/www/lab_app# sqlite3 lab_app.db SQLite version 3.27.2 2019-02-25 16:06:06 Enter ".help" for usage hints. sqlite>

First, check that the two tables are there: sqlite> .tables humidities

temperatures

Great, the tables are there. Now, let's check their schema. The schema of a table is a term that describes table structure: columns, and the data type you can store in each column. For this, use the ".schema" command: sqlite> .schema temperatures CREATE TABLE temperatures (rDatetime datetime, sensorID text, temp numeric); sqlite> .schema humidities CREATE TABLE humidities (rDatetime datetime, sensorID text, hum numeric);

● 189

Raspberri Pi Full Stack UK 200609.indd 189

06-08-20 15:30

Raspberry Pi Full Stack

As you can see, the tables are properly constructed with a datetime, sensorID, and sensor value columns.

● 190

Raspberri Pi Full Stack UK 200609.indd 190

06-08-20 15:30

Chapter 60 • Capture sensor data with a Python script

Chapter 60 • Capture sensor data with a Python script In this chapter you will learn how to use a simple Python script that to fetch temperature and humidity data from the DHT22 sensor, and store in the SQLite3 database. To make the code more readable, I will show you the condensed version of the script here. You can find the full and annotated version in the project repository33. Use Vim to create a new file titled "env_log.py". In the Vim buffer, copy this code: import sqlite3 import sys import Adafruit_DHT def log_values(sensor_id, temp, hum): conn = sqlite3.connect('/var/www/lab_app/lab_app.db') curs = conn.cursor() curs.execute("""INSERT INTO temperatures values(datetime(CURRENT_TIMESTAMP, 'localtime'),(?), (?))""", (sensor_id,temp)) curs.execute("""INSERT INTO humidities values(datetime(CURRENT_TIMESTAMP, 'localtime'),(?), (?))""", (sensor_id,hum)) conn.commit() conn.close() humidity, temperature = Adafruit_DHT.read_retry(Adafruit_DHT.AM2302, 17) if humidity is not None and temperature is not None: log_values("1", temperature, humidity) else: log_values("1", -999, -999)

This code should already be familiar to you. It starts by importing the required Python modules. It defines a method, named "log_values", that containing the database-related code (more about this in a moment). This method is called to store temperature and humidity in the database. It calls the "read_retry" method of the Adafruit_DHT object to fetch humidity and temperature from the sensor, and stores the values in relevant variables. It does a simple validity check to ensure the sensor has returned a valid response, before calling the log_values method. 33.

 he full and annotated version of this script is available here: T https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/env_log.py

● 191

Raspberri Pi Full Stack UK 200609.indd 191

06-08-20 15:30

Raspberry Pi Full Stack

Let's take some time to look into the details of the log_values method. The method takes in three parameters: "sensor_id", "temp", and "hum". The values of these parameters are then stored in the database. Inside log_values, the code makes extensive use of the sqlite3 Python interface. You can find details about the relevant functions in the documentation34. sqlite3.connect opens a connection to the application database file. Notice that the parameter contain the full path to the database file. This is important because in a later lecture you will automate the execution of this script via the Cron utility. The Cron utility runs outside the regular user shell, so does it not "know" where it is in the file system. Any reference to a file must be made using absolute paths. conn.cursor is an object that allows us to navigate through the database. It is able to execute an SQL query. curs.execute takes two arguments: a valid SQL statement and list of values. In this Python script, we construct the SQL statement using Python string35 interpolation. The three double-quotes allow us to define a string that spans multiple lines (even though in this example we can make it fit in a single line). The "(?)" in the SQL statement are placeholders for the values that appear in the second parameter of the execute function. You can set up multiple SQL statements by calling curs.execute once per statement. conn.commit() will execute the SQL statements that were created earlier. conn.close() will close the connection to the database file and release the file. It is important to do this, especially on a Raspberry Pi with an SD card because you can ensure the file will be safe if, for example, power is lost. Because a sensor may malfunction, or simply not be properly connected, it is good practice to ensure the values we get from the Adafruit_DHT.read_retry method are proper numbers. The database table schemas expect numbers, nothing else. This is why we test that the contents of the temperature and humidity variables are, indeed, numbers. The script does this in an indirect way. If the sensor was malfunctioning or not connected properly, the "read_retry" method would return the "None" value. If the value is not "None" we know it is a number (it can't be anything else). If the sensor value is not valid (i.e. it is "None"), then the script will write "-999" in the database tables. We can use this value to determine a problem with the sensor, as "-999" is not possible as temperature or humidity. Think of "-999" as an error code. If you see it in your database, it means that your DHT22 sensor is not working.

34. 35.

For documentation on the sqlite3 Python module, please see https://docs.python.org/3/library/sqlite3.html Learn about Python strings: https://docs.python.org/3/tutorial/introduction.html#strings

● 192

Raspberri Pi Full Stack UK 200609.indd 192

06-08-20 15:30

Chapter 60 • Capture sensor data with a Python script

Try out the script on the command line. Always working as root, with the Python Virtual Environment enabled, run env_log.py: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python env_log.py

If the script works, there will be a pause for a few seconds, and the command line will return. Enter the SQLite3 CLI and use the SQL "select" statement to get the records from the temperatures and humidity tables: (lab_app) root@raspberrypi-zero:/var/www/lab_app# sqlite3 lab_app.db SQLite version 3.27.2 2019-02-25 16:06:06 Enter ".help" for usage hints. sqlite> select * from temperatures; 2020-04-03 04:47:00|1|25 2020-04-03 04:47:25|1|25.1 2020-04-08 05:12:17|3|20.9

In the example above, the new record is the one created by my script test. Use "select * from humidities;" to see the records in the humidities table.

● 193

Raspberri Pi Full Stack UK 200609.indd 193

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 61 • Schedule sensor readings with Cron The Python script responsible for taking sensor values and storing them to the database is tested and ready. It is not practical to have to manually invoke it. Instead, you want to use an automatic scheduler that will run the script at regular intervals. Linux provides the Cron36 utility for this purpose. With Cron, you can execute a program at any interval you want. Every 5 seconds, 5 minutes, hours, days etc. You can learn more about Cron in the Ubuntu documentation37. Cron is a system deamon. It operates in the background, and reads scheduling settings from a system Cron file. To execute a program at whichever interval you wish, you must edit this file and set the schedule and the program to be executed. For this purpose, you can use the Cron editor, a simple text editor. To edit the Cron scheduler file, use "crontab -e". Continuing from the previous chapter session: (lab_app) root@raspberrypi-zero:/var/www/lab_app# crontab -e

If this is the first time you have run crontab, Cron will ask to select your preferred editor. Choose Vim, of course. In Vim, go to the end of the buffer, and enter this line: */10 * * * * /var/www/lab_app/bin/python /var/www/lab_app/env_log.py >/dev/ null 2>&1

Save the buffer and exit Vim. Cron is now going to run the env_log.py script every 10 minutes. This code instructs Cron to run "/var/www/lab_app/bin/python /var/www/lab_app/env_ log.py" every 10 minutes. Any output from this command is to be sent to ">/dev/null 2>&1", which in Linux is the void. In other words, Cron will not record or output any text produced by the script. The first part of the code is the schedule: */10 * * * *

There are 5 placeholders: minute, hour, day of month, month and day of week. I find that a simple and error-free way to create a schedule in this format is to use a crontab calculator, like crontab.guru. With Crontab Guru, you can create a Cron schedule and translate it into 36. 37.

Learn more about Cron: https://en.wikipedia.org/wiki/Cron The Cron documentation at Ubuntu: https://help.ubuntu.com/community/CronHowto

● 194

Raspberri Pi Full Stack UK 200609.indd 194

06-08-20 15:30

Chapter 61 • Schedule sensor readings with Cron

English so that you can be sure it will do what you think. Also, notice the scheduling instruction is using full paths, both for the Python interpreter, and Python script. As mentioned in the previous chapter, Cron does not operate inside a user shell, therefore it does not know "where" it is. As it does not know where it is, it is not able to use relative paths.

● 195

Raspberri Pi Full Stack UK 200609.indd 195

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 62 • Update the application and template file The "env_log.py" Python script you created created earlier, and then scheduled for automatic execution by Cron are the part of your application that records sensor data in the database. However, this script only records data. If you want to view actual sensor data, you will have to do so in the SQLite3 CLI, using a "select" SQL statement. Wouldn't it be much better if you could access these records in your browser? Of course. This is what you will implement in this and the next chapter. To make it possible to view your application database records in the browser, you will need to make some changes to the Flask Python application file and its HTML template. Specifically, you will need to include a method you can call from the browser that to retrieve database records, and then a HTML template file that can render these records in a suitable table format in the browser. By the end of this work, you will be able to view your database records in a browser window, as in the example in Figure 62.77.

Figure 62.77: In this and the next lecture, you will be able to see your database records in the browser. Begin with the lab_app.py file. Use Vim to open "lab_app.py". You created this file in an earlier part of the project. You can get the new version of the file from the repository38 and replace the current version, or copy the new code across:

38.

You can find this file here: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_app_v2.py

● 196

Raspberri Pi Full Stack UK 200609.indd 196

06-08-20 15:30

Chapter 62 • Update the application and template file

@app.route("/lab_env_db") def lab_env_db(): conn=sqlite3.connect('/var/www/lab_app/lab_app.db') curs=conn.cursor() curs.execute("SELECT * FROM temperatures") temperatures = curs.fetchall() curs.execute("SELECT * FROM humidities") humidities = curs.fetchall() conn.close() return render_template("lab_env_db.html",temp=temperatures,hum=humidities)

The new code creates a new route, "/lab_env_db", which is implemented by the new "lab_ env_db" function. The content of the function is similar to the code in the "env_log.py" that you created earlier. TThe only difference is that the SQL query is using a SELECT statement, instead of CREATE. The cursor is calling the "fetchall()" function to return all records that result from the select statement. The last line of the new code calls the "lab_env_db.html" template, and passes two collections as parameters: temp and hum. These collections contain all the records fetched from the database. In the template file, we'll use iteration to create a table that will contain all of their values. Save the buffer to disk and quit Vim. Continue to work on the template file. Switch into the "templates" directory and use Vim to create a new file named "lab_env_ db.html". You can copy the code from the repository39:



39.

 ou can find the code for this file here: Y https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v1.html

● 197

Raspberri Pi Full Stack UK 200609.indd 197

06-08-20 15:30

Raspberry Pi Full Stack







58.

 he full source code of the new version of lab_env_db.html is available on the project repository: T https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v3.html

● 229

Raspberri Pi Full Stack UK 200609.indd 229

06-08-20 15:30

Raspberry Pi Full Stack





From date

To date





I have highlighted in bold text the most interesting components of the new code: • The form has action attribute set to "/lab_env_db" and a method set to "GET" . When the user clicks on the Summit button, the form will submit its data to the lab_env_db route on the server, using the GET method . • There are two input fields of type text . Each has a unique ID . We will use these IDs to attach the datetimepicker widgets to each input field . • Unlike the radio buttons you created in Part 10 of the project, the user will need to click on the Submit button to submit the form . Therefore, at this time, the submit button is part of the form . 62.

You can see the fill source code for the new version of lab_env_db.htm (version 4) here: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v4.html

● 238

Raspberri Pi Full Stack UK 200609.indd 238

06-08-20 15:30

Chapter 74 • Implement the datetime picker widget

This is the part of the user interface that is visible. Let's continue with Javascript. Here it is (I have omitted code that relates to the radio buttons to keep the discussion as simple as possible):



I have positioned this code just below the line that imports jQuery. Here is what each line of code does: • Import the CSS file that controls the visual elements of the widget. • Import the Javascript that implements the widget. • Inside the "

As usual, we are getting the code from CDN . You can periodically check CDN67 for a new version of the library . At the time I am writing this chapter, the newest version is 1 .0 .7 . Next, we need to update the Javascript code that handles the submission of each form . The existing version of the lab_env_db .html file already has code that handles the submission of the "range_select" form . Simply insert the time zone code (in bold text): jQuery("#range_select input[type=radio]").click(function(){ timezone = jstz.determine(); jQuery(".timezone").val(timezone.name()); jQuery("#range_select").submit(); });

This code will get the browser timezone and insert it in the elements that belong to the "timezone" class .

67.

The jstimezonedetect library page on the CDN is https://cdnjs.com/libraries/jstimezonedetect

● 246

Raspberri Pi Full Stack UK 200609.indd 246

06-08-20 15:30

Chapter 76 • Adjust datetimes to local time zone on the client side

For the second form, named "datetime_range", use this code:

This code detects when the user has clicked on the form's submit button, and then gets the browser time zone and inserts it in the field with class name "timezone". You have now implemented the new functionality on the client (browser) side. Before you test, you need to implement the necessary changes on the server side. In the next chapter I will introduce you to the Arrow module: a Python component that is specifically written for time and date calculations. Arrow is responsible for getting timezone information from the client and converting sensor data datetime stamps to the user's timezone.

● 247

Raspberri Pi Full Stack UK 200609.indd 247

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 77 • Introduction to Arrow The HTML template of your application now has the ability to detect the browser's system timezone, and send this information to the server. On the server side, your application will use this information to convert the sensor data timestamps to the user's timezone. A Python module that is perfect for this task is Arrow68. With Arrow, your Python application can do time and date calculations such as: • Getting the current time at UTC. • Adding or subtracting any number of hours, minutes or seconds from the current time (or any other time). • Displaying a time and date using an arbitrary format. • Displaying a time in a humanised way. Instead of "11:00", Arrow can print "an hour ago". It can even do this in various languages. • Displaying the current time in any timezone specified in the Olson format. • Parsing string representations of times and dates into Arrow time objects. You can find the full listing of Arrow's features, with examples, in its user guide69. In this chapter, I will show you how to install Arrow in the application Python Virtual Environment, and then perform a few experiments using Arrow in the Python CLI. In the next chapter, I'll show you how to implement the necessary changes in lab_app.py to support the automatic timezone conversion feature. Begin by logging into your Raspberry Pi, switching to super user, changing to the application directory, and activating the Python Virtual Environment: pi@raspberrypi-zero:~ $ sudo su root@raspberrypi-zero:/home/pi# cd /var/www/lab_app root@raspberrypi-zero:/var/www/lab_app# . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

Next, use pip to install Arrow: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install arrow

Ask Raspbian for the current time and date using the "date" command. It should appear in UTC. (lab_app) root@raspberrypi-zero:/var/www/lab_app# date Fri 17 Apr 02:23:31 UTC 2020

68. 69.

The home page for the Python Arrow module is https://arrow.readthedocs.io/en/latest/. Arrow user guide: https://arrow.readthedocs.io/en/latest/#user-s-guide

● 248

Raspberri Pi Full Stack UK 200609.indd 248

06-08-20 15:30

Chapter 77 • Introduction to Arrow

Let's use Arrow to get the same information, and then do some simple time and date operations: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python Python 3.8.2 (default, Mar 14 2020, 01:38:54) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import arrow >>> print(arrow.utcnow()) 2020-04-17T02:23:50.685483+00:00

As you can see, I used the "import" command to import the arrow module. I then used "arrow.utcnow()" to get the current time, in UTC. The two time stamps are approximately the same, with the format being a little different. Let's find out the current time for a variety of timezones70. For example: • • • •

Africa/Djibouti America/Argentina/Jujuy America/Indiana/Petersburg Australia/NSW

Back on the CLI, the conversions look like this: >>> print(arrow.utcnow().to('Africa/Djibouti')) 2020-04-17T05:27:21.618628+03:00 >>> print(arrow.utcnow().to('America/Argentina/Jujuy')) 2020-04-16T23:27:33.156310-03:00 >>> print(arrow.utcnow().to('America/Indiana/Petersburg')) 2020-04-16T22:27:44.644656-04:00 >>> print(arrow.utcnow().to('Australia/NSW')) 2020-04-17T12:27:55.205273+10:00

Just call the ".to" functions on "utcnow", and pass the Olson timezone name as a parameter. How about humanising one of these times. >>> print(arrow.utcnow().to('Australia/NSW').humanize()) just now

Try to add one week to the current time: >>> print(arrow.utcnow().to('Australia/NSW').shift(weeks=+1)) 2020-04-24T12:31:07.967959+10:00

70.

You can find a full listing of timezones and names in Wikipedia: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

● 249

Raspberri Pi Full Stack UK 200609.indd 249

06-08-20 15:30

Raspberry Pi Full Stack

Again, humanise it: >>> print(arrow.utcnow().to('Australia/NSW').shift(weeks=+1).humanize()) in a week

Let's control the format: >>> print(arrow.utcnow().to('Australia/NSW').format('YYYY-MM-DD HH:mm:ss ZZ')) 2020-04-17 12:32:05 +10:00

You can even get time spans: >>> print(arrow.utcnow().to('Australia/NSW').span('hour', count=2)) (, )

This tells us the time span between now and two hours from now starts at "2020-0417T12:00:00+10:00" and finishes at 2020-04-17T13:59:59.999999+10:00. From these examples, the most relevant to our purposes in this project is the one that shows the use of the "to" function. With "to", you can convert a sensor datetime stamp (from UTC) to any timezone. Let's finish the implementation in the next chapter.

● 250

Raspberri Pi Full Stack UK 200609.indd 250

06-08-20 15:30

Chapter 78 • Implement Arrow

Chapter 78 • Implement Arrow In this chapter I will show you how to implement the server side functionality necessary for implementing the timezone automation feature. The full source code of the new version of lab_app.py is available in the project repository71. In the discussion that follows, I will focus on the changes. Start by importing the Arrow module into the script: import arrow

Inside the lab_env_db method, grab the timezone from the query string. Remember, the query string is processed by the "get_records" method, which we'll look at further down in this chapter. temperatures, humidities, timezone, from_date_str, to_date_str = get_records()

In the same method, we convert all datetime stamps (from UTC) to the required timezone: for record in temperatures: local_timedate = arrow.get(record[0], "YYYY-MM-DD HH:mm:ss"). to(timezone) time_adjusted_temperatures.append([local_timedate.format('YYYY-MM-DD HH:mm:ss'), round(record[2],2)]) for record in humidities: local_timedate = arrow.get(record[0], "YYYY-MM-DD HH:mm:ss"). to(timezone) time_adjusted_humidities.append([local_timedate.format('YYYY-MM-DD HH:mm:ss'), round(record[2],2)])

To keep things simple, instead of modifying the temperatures and humidities collections, we have created two new ones: • time_adjusted_temperatures.append • time_adjusted_humidities.append The script will append a new record after each datetime stamp is converted to the target timezone. Then, the script passes the timezone, and the two new collections to the HTML template.

71.

 he full source code of the new version of lab_app.py (version 9) is available at T https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_app_v9.py

● 251

Raspberri Pi Full Stack UK 200609.indd 251

06-08-20 15:30

Raspberry Pi Full Stack

return render_template("lab_env_db.html",

timezone = timezone,

temp = time_adjusted_temperatures, hum = time_adjusted_humidities, from_date = from_date_str, to_date = to_date_str, temp_items = len(temperatures), query_string = request.query_string, hum_items = len(humidities))

The important changes to notice in the code above are: • The timezone value is passed to the template even though we are not making use of it. If you wish, you can print it out in the web page so the user can see their system timezone. • In the previous version of the application, "temp" and "hum" collections were given values "temperatures" and "humidities". In the new version, we have replaced these with values "time_adjusted_temperatures" and "time_adjusted_ humidities", which contain the converted datetime stamps. In the get_records() method, you will find the following changes. Firstly, grab the user timezone: timezone

= request.args.get('timezone','Etc/UTC');

If a timezone key-value pair does not exist in the URL, the default will be "Etc/UTC". You can change this to whichever default timezone you prefer. We make the necessary datetime conversions before hitting the database. Because we now work with the assumption that the user will be working in their local timezone, we must convert any datetime coming from the front-end into UTC. Then, we use the UTC version of the datetimes in the SQL statements. if isinstance(range_h_int,int): arrow_time_from = arrow.utcnow().shift(hours=-range_h_int) arrow_time_to

= arrow.utcnow()

from_date_utc

= arrow_time_from.strftime("%Y-%m-%d %H:%M")

to_date_utc

= arrow_time_to.strftime("%Y-%m-%d %H:%M")

from_date_str

= arrow_time_from.to(timezone).strftime("%Y-%m-%d %H:%M")

to_date_str = arrow_time_to.to(timezone).strftime("%Y-%m-%d %H:%M") else: from_date_utc

= arrow.get(from_date_obj, timezone).to('Etc/UTC').

strftime("%Y-%m-%d %H:%M") to_date_utc

= arrow.get(to_date_obj, timezone).to('Etc/UTC').

strftime("%Y-%m-%d %H:%M")

● 252

Raspberri Pi Full Stack UK 200609.indd 252

06-08-20 15:30

Chapter 78 • Implement Arrow

If a quick-select datetime range is defined (by clicking on one of the radio buttons), we'll convert to UTC and format it appropriately for the database search. If a custom datetime range was selected from the widgets, we'll convert the from and to values to UTC. The last change in this method is this: return [temperatures, humidities, timezone, from_date_str, to_date_str]

I have added the timezone value in the array that is returned to the lab_env_db method. Ready to test the new feature? Let's do this in the next chapter.

● 253

Raspberri Pi Full Stack UK 200609.indd 253

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 79 • Upload timezone changes and test Time to test the new automatic timezone detection and conversion feature. As always, restart uWSGI so the changes in "lab_app.py" become effective: (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl restart emperor. uwsgi.service

Use your browser to fetch this URL: http://raspberrypi-zero.local/lab_env_db The result should look like the example in Figure 79.93.

Figure 79.93: The URL does not contain timezone information. Timestamps are in UTC. Because I used the base URL without any timezone information, the datetimes in the table and charts are UTC. As I was working on this chapter, I discovered an interesting "bug". This bug is the reason for the weird right side of the line chart, marked with the arrow in Figure 79.93. I discovered that my Raspberry Pi was set to BST (British Summer Time) instead of UTC. At the time I am writing, BST is one hour ahead of UTC. When I changed my Raspberry Pi time to UTC from BST (using raspi-config utility), I essentially I turned its clock one hour behind, which resulted in the artefact you can see in the chart.

● 254

Raspberri Pi Full Stack UK 200609.indd 254

06-08-20 15:30

Chapter 79 • Upload timezone changes and test

Now, try clicking one of the radio buttons. The URL contains this information: http://raspberrypi-zero.local/lab_env_db?timezone=Australia%2FSydney&range_h=3

Notice the timezone key-value pair is part of the query string. The new webpage looks like the example in Figure 79.94.

Figure 79.94: The URL contains timezone information. Timestamps are in my local timezone. As you can see, the datetime stamps that appear in the table and the chart are now in my local timezone. Experiment with the widgets to confirm the timezone key-value pair still appears in the query string. The new feature has made this application more useful. Working with time zones is an essential component of working with date and time data. In the next chapter, I'll show you how to quickly link the two pages that make up your application: "lab_env_db" and "lab_temp", so you can jump from one to the other with a click.

● 255

Raspberri Pi Full Stack UK 200609.indd 255

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 80 • Link the two pages of the application To complete this part of the project, let's link the two pages that make up our application so that it is possible to go from one to the other without having to edit the URL. At the moment, we have the two pages, as in Figure 80.95.

Figure 80.95: At the moment, the two pages of this web application are not linked. To improve the usefulness of the application, we link the two pages. In the lab_env_db page, a simple link to lab_temp will suffice. For the opposite, we can do better. Instead of using a simple link from lab_temp to lab_env_db, we'll use quick select radio buttons, like the ones you already implemented in lab_env_db.html. We'll edit both template pages. The full source code of the updated files are in the project repository72. By the end of this chapter, the two pages will be linked as in the example of Figure 80.96.

72. You can find the updated version of lab_env_db.html (version 6) at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v7.html You can find the updated version of lab_temp.html (version 3) at  https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_temp_v3.html.

● 256

Raspberri Pi Full Stack UK 200609.indd 256

06-08-20 15:30

Chapter 80 • Link the two pages of the application

Figure 80.96: We'll work to link the two pages. Let's start with lab_env_db .html . Here's the addition (in bold):

Plotly

Current

You can ignore the reference to Plotly . This is something you will implement in the next part of the project . This code generates a simple link to the "/lab_temp" route . Continue with lab_temp .html . I have added a new row, with the form that implements the radio buttons form from lab_env_db .html . I have simply copied the code across:



3hrs

6hrs

● 257

Raspberri Pi Full Stack UK 200609.indd 257

06-08-20 15:30

Raspberry Pi Full Stack

12hrs

24hrs



This form includes the timezone hidden field which relies on the "jstimezonedetect" Javascript plugin, and the code that inserts the browser timezone in the field. So, let's add the relevant code at the end of the file:



The linking of the two pages is complete. Go ahead to test. Start at "http://raspberrypi-zero.local/lab_temp" and click on any of the radio buttons. If your click on the "3hrs" button, your browser will go to "http://raspberrypi-zero.local/lab_env_db?timezone=Australia%2FSydney&range_h=3" (in your own timezone, and hostname). Click on "Current" to go back to the current temperature page.

● 258

Raspberri Pi Full Stack UK 200609.indd 258

06-08-20 15:30

Part 13: Charting with Plotly

Part 13: Charting with Plotly

● 259

Raspberri Pi Full Stack UK 200609.indd 259

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 81 • What is Plotly and how to install it Plotly is a website dedicated to data visualisation. If you are a data scientist or just interested in data analysis using powerful visualisation tools and techniques, Plotly is perhaps the best tool on the web. Over the last few years, Plotly has expanded its product offerings. For the purposes of this project, we will be using the Plotly Chart Studio product. Plotly also offers Dash, a product for building web-based analytics apps, which we will not be using. You may be wondering: why do we need to use Plotly if we are already using Google Charts? There are four reasons I think are important: • As makers and learners, it is importnat to be familiar with a variety of tools and techniques. Google Charts are awesome, but are not the only option. Plotly offers an alternative that is worth exporing. • By looking into Plotly, you will learn how to transfer your sensor data onto the your account in the Plotly Cloud, where you can experiment with numerous visualisation and analysis options. By simply learning how to transfer the data across will expose you to an alternative way to interact with Cloud applications (more will follow later in this project when you will work with IFTTT). • Plotly will give us the opportunity to integrate a new feature into the full-stack application, requiring changes in the front (the user interface) and back end (the Python Flask application). • Plotly offer capabilities that are not available with Google Charts, at least without significant additional work. For example, Plotly Charts are easily shared with a link, and can be embedded into blog posts etc. Plotly can even generate code in languages like Python that you can use in your programs. This is particularly useful for data scientists. Let's take a quick tour of the Plotly Chart Studio. To access Plotly Chart Studio, go to https://chart-studio.plotly.com/. Create a new free account, and log in. Bring up your Profile page by clicking on your username (top right corner of the page) and selecting "Profile". You can see my profile in Figure 81.97.

● 260

Raspberri Pi Full Stack UK 200609.indd 260

06-08-20 15:30

Chapter 81 • What is Plotly and how to install it

Figure 81.97: My Plotly Chart Studio profile page. If your account is new, your Profile will not have much content. In my example, you can see several charts and data tables. All of them are generated by my full-stack application. For each chart and table, you have the option of invoking an Editor or Viewer. These options do what you probably think they do. The Viewer allows you to view and interact with a chart or table. You can see an example in Figure 81.98.

● 261

Raspberri Pi Full Stack UK 200609.indd 261

06-08-20 15:30

Raspberry Pi Full Stack

Figure 81.98: The Viewer of one of my Charts. You can use your mouse to zoom in/out, pan, and select specific regions of the chart. The Editor looks like the example in Figure 81.99.

Figure 81.99: The Editor of one of my Charts.

● 262

Raspberri Pi Full Stack UK 200609.indd 262

06-08-20 15:30

Chapter 81 • What is Plotly and how to install it

Within the editor, you can manipulate raw data, change chart type, style, and theme. You can also annotate the chart with text, shapes or images, and analyse using techniques such as curve fittings and moving averages. In this project, most of your interaction with Plotly will be done via the Python CLI and Flask program. You can find documentation about on how to create Plotly charts with Python on the Plotly documentation site73. In the next chapter, I will show you how to install Plotly on your Raspberry Pi, and how to create a simple chart on the Python CLI. You will use this knowledge in your full stack application. Let's begin.

73.

Plotly Python documentation: https://plotly.com/python/

● 263

Raspberri Pi Full Stack UK 200609.indd 263

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 82 • Try out Plotly on the CLI At the time I am writing this chapter, the latest version of the Plotly module for Python is 4.6.0. This module only allows us to create Plotly charts in "offline" mode. That is, the charts you will create using the latest version of the Python module for Plotly will be hosted on your Raspberry Pi, and you will not be able to take advantage of Plotly's online tools. I prefer to retain the online functionality because I want to be able to use Plotly's online chart tools, its editor, and the ability to share my charts. Instead of installing the latest version of the Python module, we'll install version 3.10. This is the latest version of the Python module that allows both online and offline operations. Begin by logging in to your Raspberry Pi. Switch to super user, changing to the application directory, and activating the Python Virtual Environment: pi@raspberrypi-zero:~ $ sudo su root@raspberrypi-zero:/home/pi# cd /var/www/lab_app root@raspberrypi-zero:/var/www/lab_app# . bin/activate (lab_app) root@raspberrypi-zero:/var/www/lab_app#

Next, use pip to install the Plotly Python module, and specifically request version 3.10: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install plotly==3.10

The Plotly module for Python is now installed. Because your Python program will be interacting with your Plotly account, you need to set up your account credentials74. The easiest way to do this is to set up your credentials through the Python CLI. Start the CLI like this, and import the "plotly" module: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python Python 3.8.2 (default, Mar 14 2020, 01:38:54) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import plotly >>>

You will need your account username and API key, which you can get from https://chartstudio.plotly.com/settings/api. Go back to the Python CLI and enter this instruction (replace the username and api_key with your own):

74.

You can find the source documentation for this process here: https://plotly.com/python/getting-started-with-chart-studio/ Beware the documentation uses "chart_studio" as the name of the module instead of "plotly". This is because the documentation refers to the latest version of the module, instead of version 3.10 that we are using in this project. Remember to replace "chart_studio" with "plotly".

● 264

Raspberri Pi Full Stack UK 200609.indd 264

06-08-20 15:30

Chapter 82 • Try out Plotly on the CLI

>>> plotly.tools.set_credentials_file(username='your_username', api_key='your_api_key)

You are now ready to use Plotly in online mode. If you need to make changes to your credentials, re-run the "set_credentials_file" instruction, or manually edit the ".credentials" JSON file at "/root/.plotly". Let's create a simple line chart. Here's the code to copy into the Python CLI: >>> import plotly.plotly as py >>> from plotly.graph_objs import * >>> trace0 = Scatter(x=[1, 2, 3, 4],y=[16, 5, 11, 9]) >>> trace1 = Scatter(x=[1, 2, 3, 4],y=[10, 15, 13, 17]) >>> data = [trace0, trace1] >>> py.plot(data, filename = 'basic-line') 'https://plot.ly/~futureshocked/181' >>>

Start by importing the "plotly" class from the "plotly" module, and set up the "py" shortcut to reference it by. Then import the graphing objects from the "plotly" module. Create two traces by creating the Scatter object for each one. Simply list the X and Y coordinates for each datapoint. Combine the two traces into a single data array, called "data". Create the chart, call the "plot" method of the "py" class, and pass the data array and a file name for the chart. The file name can be anything you want to help you recognise the chart on your dashboard. After a few seconds of processing, Plotly will return a URL for your new chart. Copy this URL to your browser to see the chart. You can see mine in Figure 82.100.

● 265

Raspberri Pi Full Stack UK 200609.indd 265

06-08-20 15:30

Raspberry Pi Full Stack

Figure 82.100: I created this simple chart programmatically on the Python CLI. You can now play with this new chart using the editor. Change the plot type, colors, legends etc. What you just did is the process you will follow to create Plotly charts from your full stack application sensor data. Let's begin work towards this in the next chapter, where you will make the necessary changes to the front-end.

● 266

Raspberri Pi Full Stack UK 200609.indd 266

06-08-20 15:30

Chapter 83 • Implement Plotly support on the client side

Chapter 83 • Implement Plotly support on the client side At the moment, you have installed the Plotly Python module, and tested it on the Python CLI. You can create a Plotly Chart using the Python module, and can get a URL that points to the programmatically created chart so that you can view or edit it in your web browser. In this chapter, you will begin the integration of the Plotly feature in your full stack application. As usual, you will update the front end with the user interface element you want your users to interact with, and then complete the integration with the necessary modification to the back end. Before you begin the implementation, let's define exactly what you are going to do. There are several different ways you can go about this. For example: • Your application can automatically create a Plotly chart when you click one of the quick select radio buttons. • You can set a Cron job to generate a Plotly chart every hour, and email you the chart URL. • Your application can generate a Plotly chart when temperature and/or humidity go above or below a set threshold, and email you the chart URL. • You can define a hyperlink the user can click on when he wants to capture sensor data, as they appear in the "lab_env_db" as a Plotly chart. • You can… (write your own idea here) __________________________________ Regardless of the choice, the process is always the same. It looks like this: • A trigger (hyperlink, timer, event) kickstarts the process. • Read the relevant data from the database. • Create the Plotly dataset and request an online chart from Plotly. • Forward the new Plotly chart URL to the user. The trigger exists in the user interface, which is what this chapter is about. The rest of the process takes place in the back end. I have chosen to proceed with option 5. I will create a hyperlink, and place it in the "lab_ env_db" page. The user will be able to click on this link to trigger steps 2, 3, and 4 of the process. When the Plotly chart is created and the URL returned, the application will display a new hyperlink with this URL that the user can click on to view the chart.

● 267

Raspberri Pi Full Stack UK 200609.indd 267

06-08-20 15:30

Raspberry Pi Full Stack

In Figure 83.101, the arrow shows the hyperlink that triggers the chart creation process.

Figure 83.101: The user will click on the Plotly link to generate a new chart. Because it takes a few seconds for the operation to complete, it is good user interface design practice to display a message that informs the user that work is happening in the background. In Figure 83.102, the arrow points to a simple message that appears in the browser after the user has clicked on the "Plotly" hyperlink.

Figure 83.102: While the application is generating the chart in the background, let the user know. When the process is complete and the chart is generated, the user should be able to access their new chart. We can easily do this with another hyperlink that takes the user to Plotly where they can view and interact with the new chart. In Figure 83.103, the arrow points to this hyperlink. The URL of this hyperlink is the one returned by Plotly.

Figure 83.103: The user can access the new chart via a hyperlink.

● 268

Raspberri Pi Full Stack UK 200609.indd 268

06-08-20 15:30

Chapter 83 • Implement Plotly support on the client side

To summarise the above, here is what you are about to do: • Create a Plotly hyperlink that will trigger the chart creation process. • Set a HTML element that will be used to display the "work in progress" notification, and then the hyperlink to the new chart. You can find the complete source code for the new version of the lab_env_db.html template file in the project repository75. Here, I will highlight the changes and new code. The Plotly chart trigger hyperlink is created by this code: Plotly

Notice the href attribute is empty. This is because the our page will handle the GET request via Javascript. We do this because we want to include the necessary query string information so that the server side can retrieve the appropriate sensor data from the database. In Javascript, we will define a click event that will detect when the user clicks on the Plotly link, and implement the GET request. For this purpose, we use the "id" attribute. When the user clicks on the hyperlink with "id" being "plotly", it will trigger the function that creates the GET request to our application. More about this later in this chapter. Next, we define the HTML elements that will contain the Plotly chart URL and the "working" notification:



Again, nothing fancy here. There is a hyperlink with no href attribute, since we don't know what the URL is for the chart until after it is created. For notification, you use a simple span element, that has no value. The "id" attributes of both elements are unique, and allow us to target these elements from our Javascript and update their values and attributes programmatically. This feature is powered by jQuery. The jQuery script is responsible for detecting the click event on the Plotly link, preparing and executing the GET request, and handling the response from the server. Here it is: jQuery("#plotly").click( function(){ jQuery("#plotly_wait").text("Sending data..."); jQuery("#plotly_url").text(""); jQuery.get("/to_plotly", {from:"{{from_date}}", to:"{{to_date}}", 75.

 he new version of lab_env_db.html is 7, and you can find it at T https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v7.html.

● 269

Raspberri Pi Full Stack UK 200609.indd 269

06-08-20 15:30

Raspberry Pi Full Stack

timezone:"{{timezone}}"}) . done(function( data ) { jQuery("#plotly_url").attr("href",data); jQuery("#plotly_url").text("Click to see your plot"); jQuery("#plotly_wait").text(""); }); return false; });

There's a lot going on in this small code segment. We start by registering a click event76 for the element with id "plotly", and then we define the handler function that is called when this event fires. When the event handler function is called, three operations take place: • jQuery writes the text77 "Sending data…" the the element with id "plotly_wait". • jQuery clears the text value of the element with id "plotly_url". This is useful in case the user has requested a Plotly chart earlier, and this element already contained a URL. • jQuery uses the "get"78 function to do a GET request on the server. Notice the GET request URL is "to_plotly", which is a route we will implement on the back end in the next chapter. Along with the URL, we also create the query string to contain the "from_date", "to_date", and "timezone" key-value pairs. This information is inserted by the template using the variable data it receives from the application. At this point, the GET request is complete, and the browser has sent it to the server. • A few moments after the GET request is send, the response come back from the server. If it is successful, the "done"79 method is called, and the code inside its block is executed. • jQuery uses the "attr"80 function to change the value of the "href" attribute of the element with id "plotly_url". This will provide the user with a hyperlink to the Plotly chart.

76. 77. 78. 79.

Learn more about the "click" event in jQuery: https://api.jquery.com/click/. jQuery uses the "text" function to change the text value of an element. Lean more: https://api.jquery.com/text/#text2. Learn more about the GET request in jQuery: https://api.jquery.com/jQuery.get/#jQuery-get1 The jQuery GET request function will call "done" when the request has a successful response. If the request fails, it will call the "fail" function. I have not implemented the "fail" function in this example, but you can easily do so if you wish. Please see the examples at https://api.jquery.com/jQuery.get/#jQuery-get1, and implement "fail" to display an appropriate message in the "plotly_wait" element. 80. Learn more about "attr": https://api.jquery.com/attr/#attr-attributeName.

● 270

Raspberri Pi Full Stack UK 200609.indd 270

06-08-20 15:30

Chapter 83 • Implement Plotly support on the client side

• jQuery will use the "text" function to update the text value of the element with id "plotly_url". • jQuery will use the "text" function to clear the text value of the element with id "plotly_wait". At this point, you can completed the modifications on the front-end of the application. In the back end, you will need to implement the new route "to_plotly" responsible for handing the Plotly GET request. You can try out loading the new version of the "lab_env_db". Click on the "Plotly" hyperlink. You should see the notification "Sending data…", but that's about it. The provision does not yet exist in the back end to process this request. Let's go ahead and implement the missing route in the next chapter.

● 271

Raspberri Pi Full Stack UK 200609.indd 271

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 84 • Add Plotly support on the server side On the client side, the new feature is ready. On the server side, there is one outstanding task to complete: implement the "to_plotly" route that is called by the jQuery GET request. You already have all the knowledge needed to implement the new route. You know how to use the Plotly Python module, how to extract information from the query string, and how to interact with the database. In the "to_plotly" route, you will simply apply your existing knowledge to a new configuration. The full source code of new version of the lab_app.py script that containing the new functionality is available in the project repository81. Below, I have copied the code that implements the new route, plus two new lines from the script's header. The new route will make use of the Plotly module, so let's include it in the header of the script: import plotly.plotly as py from plotly.graph_objs import *

Below is the full code of the new route. I have only removed the comments to make it easier to format for this page. @app.route("/to_plotly", methods=['GET']) def to_plotly(): ###### PART 1: handle query string and database query ####### temperatures, humidities, timezone, from_date_str, to_date_str = get_records() time_series_adjusted_temperatures

= []

time_series_adjusted_humidities = [] time_series_temperature_values = [] time_series_humidity_values = [] for record in temperatures: local_timedate = arrow.get(record[0], "YYYY-MM-DD HH:mm:ss").to(timezone) time_series_adjusted_temperatures.append(local_timedate.format('YYYY-MMDD HH:mm:ss')) time_series_temperature_values.append(round(record[2],2)) for record in humidities: local_timedate = arrow.get(record[0], "YYYY-MM-DD HH:mm:ss").to(timezone) time_series_adjusted_humidities.append(local_timedate.format('YYYY-MM-DD 81.

Version 10 of lab_app.py is available at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_app_v10.py.

● 272

Raspberri Pi Full Stack UK 200609.indd 272

06-08-20 15:30

Chapter 84 • Add Plotly support on the server side

HH:mm:ss')) time_series_humidity_values.append(round(record[2],2)) ###### PART 2: handle Plotly operations ####### temp = Scatter( x

= time_series_adjusted_temperatures,

y

= time_series_temperature_values,

name

= 'Temperature'

) hum = Scatter( x

= time_series_adjusted_humidities,

y

= time_series_humidity_values,

name

= 'Humidity',

yaxis = 'y2' ) data = Data([temp, hum]) layout = Layout( title

= "Temperature and humidity in Peter's lab",

xaxis

= XAxis(

type

= 'date',

autorange = True ), yaxis title

= YAxis( = 'Celsius',

type

= 'linear',

autorange

= True

), yaxis2 = YAxis( title

= 'Percent',

type

= 'linear',

autorange

= True,

overlaying = 'y', side

= 'right'

) ) fig

= Figure(data = data, layout = layout)

plot_url = py.plot(fig, filename = 'lab_temp_hum') return plot_url

● 273

Raspberri Pi Full Stack UK 200609.indd 273

06-08-20 15:30

Raspberry Pi Full Stack

The method that implements the new route "to_plotly" consists of two parts: 1. The first part extracts the datetime and timezone data from the query string, and queries the database for relevant records. All the code in this part of the method should be familiar to you. It is almost an exact copy of the code in the "get_records" method. The outcome of this part are the two collections: - time_series_temperature_values - time_series_humidity_values 2. The second part performs the Plotly operations, as per the experiment on the command line interface two chapters ago. The last line of the method is "return plot_url". Unlike the other two routes, this one does not call a HTML template. Instead, it sends the single string that contains the chart URL to the jQuery GET request "done" handler function. I remind you this is the "done" function from the previous chapter: .done(function( data ) { jQuery("#plotly_url").attr("href",data); jQuery("#plotly_url").text("Click to see your plot"); jQuery("#plotly_wait").text(""); });

The return statement of the "to_plotly" method returns the value stored in the "plot_url" variable (which contains the URL of the new chart returned from Plotly) to the jQuery "done" function. In the "done" function, the URL string is stored in the "data" variable. Inside the "done" function block, we write the value stored in "data" to the "href" attribute of the element with id "plotly_url". Implement these changes and try out the new functionality. Restart uWSGI so the changes in "lab_app.py" become effective: (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl restart emperor. uwsgi.service

Use your browser to fetch this URL: http://raspberrypi-zero.local/lab_env_dbhttp://raspberrypi-zero.local/lab_env_db Click on one of the radio buttons, or enter a customer datetime range so that the application can detect your browser timezone. Then, click on the Plotly hyperlink. Your browser will take you to your new Plotly chart, as in Figure 84.104.

● 274

Raspberri Pi Full Stack UK 200609.indd 274

06-08-20 15:30

Chapter 84 • Add Plotly support on the server side

Figure 84.104: Click on the "Click to see your plot" button view the new chart. In "real life", implementing new features does not always go smoothly. Bugs are part of creating anything new. Just as important to knowing how to create new features, is knowing how to deal with defects. In the next two chapters, I will give you examples of how to use client and server side tools that can help you find and fix bugs in your code.

● 275

Raspberri Pi Full Stack UK 200609.indd 275

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 85 • How to debug Javascript Modern web applications depend on client-side functionality coded using Javascript . Combined with libraries such as jQuery, Javascript code can quickly grow in complexity . As your Javascript code gets complex, the risk of introducing defects increases . Luckily, modern web browsers come equipped with developer tools that we can use to debug Javascript code . In this chapter, I'll show you how you can use the developer tools in Chrome to find and fix a simple bug in the Javascript that we wrote to implement the Plotly feature . If you prefer to to use another modern web browser, such as Firefox, Safari and Edge, know that they come with equivalent developer tools . I find that once I understand how to debug Javascript code in a single browser, I can find my way around the tools that come with any other browser . This specific skill is easily transferable across platforms and operating systems . To demonstrate, I will introduce a bug in the Javascript in templates/lab_env_db .html . You can use vim for this: (lab_app) root@raspberrypi-zero:/var/www/lab_app# vim templates/ lab_env_db.html

Look for this line of code:

Change it to this (change in bold):

Instead of fetching the full jQuery library, I am fetching the slim version of the library . This version is smaller in download size, but is missing several functions, including the GET function that our Javascript needs for Plotly functionality . Save your "bugged" version of the template file . For the debugging example, I will use Google Chrome . As I mentioned earlier, you can follow the exact method using equivalent developer tools in other modern browsers . Start Chrome, and access your full stack application . I use this URL for my testing: http://raspberrypi-zero .local/lab_env_db?from=2020-0424+00%3A00&to=2020-04-24+22%3A26&timezone=Australia%2FSydney

● 276

Raspberri Pi Full Stack UK 200609.indd 276

06-08-20 15:30

Chapter 85 • How to debug Javascript

At this point, the application looks fine . And it is . Click on the Plotly link . The application seems to react as expected . It will show the "Sending data…" message . But, the message goes away and it is not replaced by the chart URL hyperlink . Oh dear! We have a bug . Let's fire up the browser developer tools . Right click anywhere in the page and select "Inspect" (Figure 85 .105) .

Figure 85.105: Firing up the browser developer tools. This will divide the browser window into two parts . The top part is showing your webpage . The bottom contains developer tools . You now have access to several developer tools, such as: • Elements, which allow you to inspect the HTML code of your page . • Console, which allows you to interact with the page using a command-line environment . • Sources, which gives you information about the various files that make up your webpage, such as the HTML file, and various Javascript imports (like jQuery) .

● 277

Raspberri Pi Full Stack UK 200609.indd 277

06-08-20 15:30

Raspberry Pi Full Stack

• Network, which gives your information about the network transfer of the files that make up your webpage, such as type, size, the HTTP response code, and how long it took for your browser to download them. In this investigation I will use the Console. Click on the Console tab to select it (Figure 85.106).

Figure 85.106: In this investigation, we'll use the Console tool. Let's take a step back. At this point, let's assume I don't know what the problem is (even though I created the problem myself). Let's say it was a typo. However, I know something is not right because the click on the Plotly hyperlink did not produce the expected result. As the application developer, I know that specific feature depends on Javascript. This suspicion prompts me to enable the Console tool as it will show me Javascript error messages just like the Python CLI shows errors with my Python code when I execute it. With the console enabled, click on the "Plotly" hyperlink to trigger the handler function. The Console will output several lines of text. But what is most interesting are the lines in red (Figure 85.107).

● 278

Raspberri Pi Full Stack UK 200609.indd 278

06-08-20 15:30

Chapter 85 • How to debug Javascript

Figure 85.107: The problem with my Javascript is highlighted in red. In red text, the Console shows exactly where the problem is. Javascript error messages are not always as clear-cut as this, but there is always enough information to get your bearings and to solve a problem. This is the error message (marked as "1" in Figure 85.107): Uncaught TypeError: jQuery.get is not a function

You can click on the black triangle to expand the message. This alone should give you a strong about the indication of the location of the bug. The Console can help you reveal the location in the browser itself. Click on the link in the first line of the error message, marked as "2" in Figure 85.107. This will switch your developer tools to the Sources tool, and display the exact line of code where the bug is located (Figure 85.108).

● 279

Raspberri Pi Full Stack UK 200609.indd 279

06-08-20 15:30

Raspberry Pi Full Stack

Figure 85.108: Found the bug! Now you can see exactly where the problem is. The call to the "get" function is causing the script to fail. At this point you know where the problem is, but not how to fix it. When I come across issues like this, it takes further investigation to understand why this problem is happening (not just where it is happening). Once I understand the "why", I can devise a way to fix it. I admit that fixing the "get" bug took me around 15 minutes. I was confident the jQuery library contained all the functionality I needed, and it wasn't until I had ruled out other problems that it dawned on me that the "slim" version of the jQuery library did not contain the AJAX functions, one of which is "get". When I suspected the functionality I needed for the application was missing from the "slim" version of the jQuery library, I looked for more information on this at https://jquery.com/ download/. Here, I found this: You can also use the slim build, which excludes the AJAX and effects modules: Aha! Now I was 99.99% sure that this bug would be fixed by removing the "slim" from the import code in the template file:

Test again, and voila, no Javascript error!

● 280

Raspberri Pi Full Stack UK 200609.indd 280

06-08-20 15:30

Chapter 86 • Server side debugging example

Chapter 86 • Server side debugging example Earlier in the project, I demonstrated a simple method to test and debug a Flask application. This method consists of running the application using the Flask built-in web server on the command line. In this chapter, I will show you how to debug a server side bug of an application that is running in "production" mode. This is, an application that is operating through NGinx and managed by uWSGI. Imagine using your browser to access your application. You will see this (Figure 86.109):

Figure 86.109: Oh no! A server-side bug is causing this. Some server-side bugs break the application to such as extent that the application can't return anything useful to the client (the web browser). The browser will just produce a generic error message like the one in Figure 86.109. Less dramatic problems can also occur. For example, incorrect calculations, and errors in SQL queries that result in the wrong records (or even no records) to be returned from the database will not produce an Internal Server Error in the browser, but will show signs of incorrect behaviour in an otherwise correct looking web page. In this example I chose to break the application so you can learn how to deal with an extreme situation. Remember when you configured uWSGI, you defined a log file. The log file is configured in this line of the lab_app_uwsgi.ini file: logto = /var/log/uwsgi/%n.log

Any output from our application, such as messages produced with a "print" command, or error and warnings from Flask and Python are stored in this log file. In most cases, you will want to look at the last few lines of the content of this log.

● 281

Raspberri Pi Full Stack UK 200609.indd 281

06-08-20 15:30

Raspberry Pi Full Stack

Use the "tail" command, with "-n 20" as an argument to show the last 20 lines of the log file (the command is in bold text): (lab_app) root@raspberrypi-zero:/var/www/lab_app# tail -n 20 /var/log/uwsgi/ lab_app_uwsgi.log your mercy for graceful operations on workers is 60 seconds mapped 64400 bytes (62 KB) for 1 cores *** Operational MODE: single process *** added /var/www/lab_app/ to pythonpath. Traceback (most recent call last): File "/var/www/lab_app/lab_app.py", line 56, in from plotly.graph_objss import * ModuleNotFoundError: No module named 'plotly.graph_objss' unable to load app 0 (mountpoint='') (callable not found or import error) *** no app loaded. going in full dynamic mode *** uWSGI running as root, you can use --uid/--gid/--chroot options *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI worker 1 (and the only) (pid: 31139, cores: 1) --- no python application found, check your startup logs for errors --[pid: 31139|app: -1|req: -1/1] 192.168.112.13 () {42 vars in 956 bytes} [Sat Apr 25 03:03:00 2020] GET /lab_env_db?from=2020-04-24+00%3A00&to=202004-24+22%3A26&timezone=Australia%2FSydney => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (1 switches on core 0) --- no python application found, check your startup logs for errors --[pid: 31139|app: -1|req: -1/2] 192.168.112.13 () {40 vars in 716 bytes} [Sat Apr 25 03:03:00 2020] GET /favicon.ico => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (1 switches on core 0) --- no python application found, check your startup logs for errors --[pid: 31139|app: -1|req: -1/3] 192.168.112.13 () {42 vars in 956 bytes} [Sat Apr 25 03:15:31 2020] GET /lab_env_db?from=2020-04-24+00%3A00&to=202004-24+22%3A26&timezone=Australia%2FSydney => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)

The last 20 lines of the log file contain a lot of content . You can ignore anything marked as "Warning", or printouts from the application . Look for errors . There is one that stands out: ModuleNotFoundError: No module named 'plotly.graph_objss'

This error is complaining about an error on line 56 of the application file . According to this error message, Python is trying to load a module called "plotly .graph_objss" (Figure 86 .110) .

● 282

Raspberri Pi Full Stack UK 200609.indd 282

06-08-20 15:30

Chapter 86 • Server side debugging example

Figure 86.110: A typo has broken the application. The arrow in Figure 86.110 shows an extra "s" in the name of the module, on line 56 of the program. That's the bug! Remove the extra "s" so the module name is "plotly.graph_objs". Restart uWSGI: (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl restart emperor. uwsgi.service

Refresh your browser. You application should be back to normal. Next up, lets expose your application to the internet so that you can access it from anywhere in the world.

● 283

Raspberri Pi Full Stack UK 200609.indd 283

06-08-20 15:30

Raspberry Pi Full Stack

Part 14: Access your application from the Internet

● 284

Raspberri Pi Full Stack UK 200609.indd 284

06-08-20 15:30

Chapter 87 • How to access your application from the Internet?

Chapter 87 • How to access your application from the Internet? Your application is already providing you with an interesting platform on which you can build new features and capabilities, and, learn new technologies along the way. Later in this project, you will implement additional features, such as the ability to receive sensor data from remote Arduino nodes and create email alerts. Before adding new features, I want to show you how to access your full-stack application from anywhere in the world. Right now, your application is only accessible from a computer that is part of your local area network. Wouldn't it be good to be able to check on your lab temperature and humidity as you travel on a train or go out for a walk? The key to this capability is in your local area network router. On most home networks, there is a single device that combines a router82 with a modem83. Many people, myself included, have a network setup that has a discreet modem and router. Either way, the principal of exposing an internal network server to the Internet is the same. It is commonly known as "port forwarding84". Here is the outline: Public IP address. Your router is given a public IP address from your Internet Service Provider (ISP). This IP address is important, because it is accessible from anywhere on the Internet, and we will use it as the IP address of your Raspberry Pi server. Service Port number. Because you may be running multiple services inside your local area network, and also for the sake of providing a slightly more secure setup, we must "multiplex" the single public IP address provided by the ISP to multiple services inside the local network. One of those services is your full-stack application. The IP protocol gives us a way to do this: the Port85 number. There are almost 65000 possible port numbers, and only a small number of them has been given an official designation86. You can choose any non-assigned number87 to your full stack application. Combine the public IP address with your chosen Port number. Once you choose a Port number, say 4000, you will be able to access your full stack application from anywhere in the world by combining your router's public IP address and chosen port number.

82. 83. 84. 85. 86. 87.

Need to refresh your knowledge on routers? Please read this: https://en.wikipedia.org/wiki/Router_(computing) Need to refresh your knowledge on modems? Please read this: https://en.wikipedia.org/wiki/Modem Wikipedia's article on port forwarding is recommended reading: https://en.wikipedia.org/wiki/Port_forwarding Learn more about the "Port" in computer networking: https://en.wikipedia.org/wiki/Port_(computer_networking) See list of TCP and UDP port numbers and their designations: https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers I should be more precise about this: You can actually choose any port number you want, not just a port number without an official designation. As long as there is no other service inside your local network that is already using this port, your port forwarding setup will work.

● 285

Raspberri Pi Full Stack UK 200609.indd 285

06-08-20 15:30

Raspberry Pi Full Stack

For example, if your router's public IP address is "129.15.7.62", you will be able to get your lab's current sensor readings with this URL: 129.15.7.62:4000/lab_temp. Set up your router to do port forwarding. As you already know, your full stack application is a service running on your Raspberry Pi. You Raspberry Pi has an IP address on your LAN, and knows nothing about your router's public IP address. It also knows nothing about the port number that you have arbitrarily given to your application to enable external access. Your router is responsible for matching the external IP address and port number, to the application's internal IP address and port number. For example, your router must be able to receive requests at 129.15.7.62:4000 and then forward these requests to 192.168.111.70:80. I will show you how to do this using my own router as an example. Because every router does port forwarding slightly differently depending on the manufacturer and model, you will need to translate my instructions to your situation. Port forwarding is more reliable when the target resource is using a static IP address88, which I will also show you how to do. Secure your website. You are now able to access your full-stack application webpages from anywhere on the internet. At the moment, your webpage data flows through the internet unencrypted, making it easy for anyone that is interested in this sort of thing to intercept and read it. As a way of increasing your application's security, I'll show you how to encrypt your webpages as they travel through the Internet using SSL certificates on the nGinx web server. Let's jump into the implementation of port forwarding setup first, and then work to secure the application website with SSL.

88.

A static IP address is an address that you can allocate to a device (like your Raspberry Pi) that does not change. Normally, a router will assign a dynamic IP address to a device. A dynamic address is leased by the device, and can change over time.

● 286

Raspberri Pi Full Stack UK 200609.indd 286

06-08-20 15:30

Chapter 88 • Set a static IP address

Chapter 88 • Set a static IP address In this chapter, I will show you how to assign a static (permanent) IP address to the Raspberry Pi that is hosting your full stack application. There are a couple of simple methods to do this. The first method involves setting the static IP address of the Raspberry Pi itself. This requires you to edit the /etc/dhcpd.conf file, and set your desired IP address using the "static ip_address" directive. I do not favour this method for two reasons: 1. It can potentially conflict with your router's DHCP service which is responsible for assigning IP addresses to the network devices. To deal with this conflict, you will need to exclude your Raspberry Pi's static IP from the pool of addresses your router can allocate. This means that if you choose this method, you will need to implement the setup in two different locations. 2. It decentralises the way that IP address are managed on your network. You want the exact opposite: network management should be centralised so you have a single place to manage, with clear authority on who is responsible for managing IP addresses, with the additional benefit that there is no chance of conflict. Let's go for the second method: Use your router to permanently assign an IP address to your Raspberry Pi. I will show you how to do this on my router. The principle is the same across any router that has a DHCP89 service. My router is a Draytek Vigor 2926. On this router, the feature that allows me to assign a fixed IP address to any device is called "IP binding". Log in90 to the router's admin panel and click on LAN (1), "Bind IP to MAC" (2). You can see what this screen looks like in Figure 88.111.

89. 90.

Learn about the DHCP service: https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol To find out how to login to your router's admin panel, please consult your router's documentation.

● 287

Raspberri Pi Full Stack UK 200609.indd 287

06-08-20 15:30

Raspberry Pi Full Stack

Figure 88.111: Bind an IP address to the Raspberry Pi. I remind you that unless you have the same router, you will need to "translate" the following steps to work with your router. In the "Bind IP to MAC" screen, my router shows me the devices that are part of my network inside the pane titled "ARP table". In this table, locate the Raspberry Pi (3). Ideally, you will be able to recognise your Raspberry Pi based on its host name ("raspberrypi-zerolocal"), or IP or MAC address. You can get the IP and MAC91 address of your Raspberry Pi by logging in to your Raspberry Pi and typing "ifconfig" in the command line. You can see the result in Figure 88.112, where "1" points to the IP address, and "2" points to the MAC address.

91.

Learn about the MAC address: https://en.wikipedia.org/wiki/MAC_address

● 288

Raspberri Pi Full Stack UK 200609.indd 288

06-08-20 15:30

Chapter 88 • Set a static IP address

Figure 88.112: Use "ifconfig" to get the IP and MAC address of your Raspberry Pi. You can double-click on this Raspberry Pi row. This will copy the row to the "IP Bind List" table (4). Click on "OK" to commit the change (5). That's it, your Raspberry Pi now has a static IP address. You are now ready to continue with the next step in the process, where you will set up port forwarding.

● 289

Raspberri Pi Full Stack UK 200609.indd 289

06-08-20 15:30

Raspberry Pi Full Stack

Chapter 89 • Expose your app to the Internet with port forwarding Let's set up port forwarding. As in the last chapter, I will show you the configuration as it applies in my Draytek Vigor2926 router. You will need to reference your router's documentation for the correct settings that apply to your network setup. First, collect the necessary parameters for port forwarding: • • • •

Your router's public IP address. Your Raspberry Pi's internal IP address. The public IP port that you have assigned to your full-stack application. The internal IP port of your application.

Let's determine the router's IP address. This is not strictly necessary with my Draytek router because you can simply set port-forwarding to work with any incoming packet to the WAN (Wide Area Network) interface. However other routers may need this information to be explicitly provided. To get the public IP address on the Draytek, log in to the routers administrator website, and click on Dashboard (Figure 89.113).

Figure 89.113: The IP address allocated from y ISP to my router. The arrow is pointing to my router's public IP address: 100.92.123.123.

● 290

Raspberri Pi Full Stack UK 200609.indd 290

06-08-20 15:30

Chapter 89 • Expose your app to the Internet with port forwarding

Next, let's configure port redirection. Click on NAT (1), Port Redirection (2) to access the appropriate screen (Figure 89.114).

Figure 89.114: The Port Redirection screen. This screen contains 40 slots you can use to define port redirections. Click on any available slot to define a redirection. I'll go for slot "1" (3). Click on index 1 to bring up the port redirection form for this slot (Figure 89.115).

Figure 89.115: The Port Redirection screen for slot 1. Let's complete the form like this: • Mode: Single. We only want to configure a single port redirection, not a range of ports.

● 291

Raspberri Pi Full Stack UK 200609.indd 291

06-08-20 15:30

Raspberry Pi Full Stack

• Service name: A name for this redirection that makes sense to you. This name will appear in the Service Name column of the Port Redirection screen (Figure 89.114). • Protocol: TCP. • WAN Interface: My router has two WAN interfaces, but only one is connected at the moment, so I'll choose WAN2. If your router has one WAN interface, you will not have this option. • WAN IP: This is where you must enter the WAN (public) IP address of your router which you found to be 100.92.123.123. The Draytek knows its own WAN IP so it has filled it in for me. • Public port: 4000. This is the port number you will use to access your application from the Internet. • Source IP: I'd like to be able to access my application regardless of the IP address allocated to my mobile device or computer. So, set source IP to "Any". • Private IP: This is the internal IP of your Raspberry Pi. In my case, its 192.168.111.70. • Private port: This is the internal IP Port of your Raspberry Pi. In my case, its 80, as set in the nGinx configuration chapter. Click on "OK" to accept the settings and create the port forwarding. Ensure the new slot is active (notice the checked box marked "4" in Figure 89.114). Now let's test. There are a couple of ways to do this. The easiest way is to use your smartphone. First, disconnect from your local WiFi so that your phone will use the cellular network exclusively. Next, use a browser on your phone to access your local network's public IP address, with your full-stack application's public port number. In my case92 this is "http://100.92.123.123:4000/lab_temp". You can see the current sensor readings page rendered on my phone browser in Figure 89.116.

92.

In these examples I am using a random IP address instead of my real public IP in order to protect my network's privacy.

● 292

Raspberri Pi Full Stack UK 200609.indd 292

06-08-20 15:30

Chapter 89 • Expose your app to the Internet with port forwarding

Figure 89.116: I can access my full-stack app on my phone, while connected to a 4G network. Another way of testing is via your VPN connection service. With VPN, you can connect your computer to a gateway anywhere in the world, and effectively disconnect from your local network. In Figure 89.117 you can see my browser accessing the full-stack application while my computer is connected to a gateway in Brazil.

● 293

Raspberri Pi Full Stack UK 200609.indd 293

06-08-20 15:30

Raspberry Pi Full Stack

Figure 89.117: I can access my full-stack app while connected to a Brazilian gateway via a VPN service. Now you have configured your port forwarding so that you can access the application from anywhere in the world, let's continue by improving its security . In the next few chapters of this part of the project, I will show you how to encrypt the application webpage traffic using SSL .

● 294

Raspberri Pi Full Stack UK 200609.indd 294

06-08-20 15:30

Chapter 90 • Create a self-signed certificate for application

Chapter 90 • Create a self-signed certificate for application Your full stack application is now published so that you can access it from anywhere on the Internet . However, all communication between your remote browser and webserver that is running on your Raspberry Pi is travelling around the Internet unencrypted and anyone can read it . Granted, this communication does not contain any top-secret information, but it is 2020, and information security should be one of our considerations even for the most trivial application . To add a layer of security, I will show you how to install SSL page-level encryption on your website . Let's evaluate the current security situation of your website . Access the application, either from its public, or private IP address . First, try this: http://raspberrypi-zero.local/lab_temp

Your browser should show the webpage, as usual . Now, try this: https://raspberrypi-zero.local/lab_temp

You should see something like the example in Figure 90 .118 .

Figure 90.118: The server is unable to respond to a HTTPS request.

● 295

Raspberri Pi Full Stack UK 200609.indd 295

06-08-20 15:31

Raspberry Pi Full Stack

As per its current configuration, our webserver cannot respond to a HTTPS request. A HTTPS request uses encryption at the page level that makes it possible to obfuscate the contents of a web page as it travels between the server and client. Let's start to work towards making HTTPS requests work with our Nginx server. The main component of the process is the creation of two files: the SSL key and SSL certificate. You can purchase an SSL certificate from a reputable online certificate authority. These certificate authorities are trusted, in a way similar to how a government is trusted to issue a passport by other governments. A trusted certificate authority will issue an SSL certificate to "promise" that a website is legitimate. However, a certificate from a trusted authority costs anywhere from $100 to a few thousand dollars depending on the level. If you are running an ecommerce website, you must use a certificate from a trusted certificate authority. However, in this project we are not implementing an ecommerce site. Our full stack application will be used by yourself and perhaps a few other people. Therefore, we don't need to purchase a certificate. Instead, we'll create one on the Raspberry Pi, using open-source software that is already on Raspbian. Raspbian ships with openssl93, a cryptography and SSL/TLS toolkit. This toolkit contains several small utilities that allow you to create SSL and TLS keys and certificates. You will need to create two files, as mentioned earlier: • The SSL private key • The SSL public certificate The private key will stay secure on your webserver and is the component that proves the identity of the webserver to the client. The public certificate is shared with the client web clients. It is used by the client to decrypt the information it receives from the server. The client also uses this public certificate to encrypt any information it sends to the server. Public key cryptography is a real mathematical achievement. The private and public keys are mathematically impossible to calculate from each other, yet they work together. A client with the public key can encrypt data in a way only the owner of the private key can decrypt. This is why the private key must be stored securely on the server.

93.

You can learn more about Openssl here: https://www.openssl.org

● 296

Raspberri Pi Full Stack UK 200609.indd 296

06-08-20 15:31

Chapter 90 • Create a self-signed certificate for application

To create the two keys, use the openssl utility like this (ensure that you are logged in to your Raspberry Pi as the super user, in the application directory): # openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/ private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt Generating a RSA private key .............................................................+++++ ...........................................+++++ writing new private key to '/etc/ssl/private/nginx-selfsigned.key' ----You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]:NSW Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []:raspberrypi-zero.local Email Address []:

I have highlighted the command I have entered with bold text . This command contains several components94 that I explain below, starting from the end: • The public self-signed certificate is one of two outputs for this operation (using the "-out" switch), and will be stored in /etc/ssl/certs/nginx-selfsigned .crt . We'll be using this path in a later step . • The private self-signed key is the second output of this operation (using the "-keyout" switch) and is stored in /etc/ssl/private/nginx-selfsigned .key . We'll be using this path in a later step . • With the switch "-newkey" we instruct openssl to create a new certificate and key using the RSA algorithm . The key and certificate will have a length of 2048 bits . • With the switch "-days", we set the expiry of the certificate to 365 days . • With the switch "-nodes", respecify the private key must not be encrypted . This makes it easier to use with Nginx . As long as the server is protected from the outside world, the security of our private key is not affected . 94.

You can find more information about the various configuration options of openssl here: https://www.openssl.org/docs/man1.0.2/man1/openssl-req.html

● 297

Raspberri Pi Full Stack UK 200609.indd 297

06-08-20 15:31

Raspberry Pi Full Stack

• With the switch "-x509", openssl will output a self-signed certificate instead of a certificate request. If you were planning to purchase a certificate from a certificate authority, you would generate a certificate request by omitting this switch. When you hit "enter" to execute this command, your Raspberry Pi will calculate the public certificate and private key, and then ask a few questions about your server. You can accept all defaults by hitting "enter", except for the Common Name. In the Common Name question, type the hostname of your Raspberry Pi. This name will appear in the certificate as it appears on the client web browser. When the certificate and key creation is complete, your new SSL files will be stored in /etc/ ssl/certs/ and /etc/ssl/private/. You can have a quick look inside if you wish: (lab_app) root@raspberrypi-zero:/var/www/lab_app# cat /etc/ssl/certs/nginxselfsigned.crt -----BEGIN CERTIFICATE----MIIDrTCCApWgAwIBAgIUNx7dHsmf1/pfFZwuFNdwOz8ik00wDQYJKoZIhvcNAQEL BQAwZjELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEfMB0GA1UEAwwWcmFzcGJlcnJ5cGkt emVyby5sb2NhbDAeFw0yMDA1MDYwMDM2NDlaFw0yMTA1MDYwMDM2NDlaMGYxCzAJ BgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5l dCBXaWRnaXRzIFB0eSBMdGQxHzAdBgNVBAMMFnJhc3BiZXJyeXBpLXplcm8ubG9j YWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/Vrl/5/bgylJVFKSD c/ec8OUJym84RIsqJR7uyDegwKqbBR/Hz/zbiQ/VSvGs/tjFMVB8b7hf+pYtG3+x JWQVN9z2GM1bl0TiLY3JDfBrZNFKiccWCC36TnjkSXJZwFywRwiaU/vLEmfd8X0Q B/pnyhYC2TNCtXncThF9HqN9KBpB31R25GflgEWeKdeXUewJPYYg59zwwelZEE2J GSiexfv9DsKT04GM3pM/WNdNTU93CXusJW9fS/rK63OqQdRZgVQtj73Emegm2KIW JLp5d4v3XTjWWew8LoRdoMEpHSOqiWktgu78CZsC6BYpYeFO2utDrEnoI2t97lJB 71Y3AgMBAAGjUzBRMB0GA1UdDgQWBBSjx6m4wR11+3KXIgggAdudgA4C3jAfBgNV HSMEGDAWgBSjx6m4wR11+3KXIgggAdudgA4C3jAPBgNVHRMBAf8EBTADAQH/MA0G CSqGSIb3DQEBCwUAA4IBAQBGs8XwlsXtTTIjnkJDndAcsxogCH77GfQ/8/ZlBEa2 vzY8tywwuWQKI7w/0Dyk3OSognAYBlO+0G7diGwGxr6LVmgJKVSVyuFFMgl10oSy eQ2SI9A4bOK0u4DQAC0yuxgrUuz4kWi+4Ew1GFvJ2Q7dIN1I6UhLlqsMlyMxPfKT kxAVfirYDmA9X2zYO9Rxs/fyDDqDhm8zQ2KxVwqr343DqeQLgxuUCgtc8lYw53eo 9jWjmCjSInWcTiJMn5N6a0l6J3CzZ6h7ey6ivB74Y5eFftJbihT8Qza6mb88Aogw iXNSu8oHXGV9uKB13NcmnnMJDvPEJL2YBvzZEeo+GmrU -----END CERTIFICATE----And: (lab_app) root@raspberrypi-zero:/var/www/lab_app# cat /etc/ssl/private/ nginx-selfsigned.key -----BEGIN PRIVATE KEY----MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC/Vrl/5/bgylJV FKSDc/ec8OUJym84RIsqJR7uyDegwKqbBR/Hz/zbiQ/VSvGs/tjFMVB8b7hf+pYt

● 298

Raspberri Pi Full Stack UK 200609.indd 298

06-08-20 15:31

Chapter 90 • Create a self-signed certificate for application

G3+xJWQVN9z2GM1bl0TiLY3JDfBrZNFKiccWCC36TnjkSXJZwFywRwiaU/vLEmfd 8X0QB/pnyhYC2TNCtXncThF9HqN9KBpB31R25GflgEWeKdeXUewJPYYg59zwwelZ EE2JGSiexfv9DsKT04GM3pM/WNdNTU93CXusJW9fS/rK63OqQdRZgVQtj73Emegm 2KIWJLp5d4v3XTjWWew8LoRdoMEpHSOqiWktgu78CZsC6BYpYeFO2utDrEnoI2t9 7lJB71Y3AgMBAAECggEADlu9s+F+X0QPhZ708YBkVnylU2TDj99k6ha7CYQw2Hlc 1CFAT0hGW1sfuwvSsv602adnuM7RcTJmzp+/p1V8+zKj/DvEPVlOrAJrH2lSX1zD b+qC7B5VBPGalNcGI8IRspDCDoK65HRPIxKaJffTTLKXkEMoW5sut9+L5G3GKwAU Et88SXWLvY/hsdNztnH4jacOKvkQ0Oaoj/X0fgePIsAcpDJ737VDsWt17fRgD1uY EPLX1/UCApXlHH0ZntovBNfQxCrbzJ0nyEHdUWmLzs0A5ZBsi4rwDoPlRCN4cYfR qyOjWjDJSy7f+TPHw1xs8YFnHvYYZ1ENQsCQFf+1YQKBgQDwr3obEMTTBXF2QcgH ESR6xFXt/oMYAgudrc/1nfCHHREj5XBGwYSNYZjvr6V98YqgPuSLVYGAEYQ/JyQ0 xQG/sfZFBOE67fCT7jLEitCeCuyqtqC+1Ivz3VdkLHt7zHBC72GXnxzmRUc86nUA QIJcmF5zfC8J/XLvaWu2mdFEbwKBgQDLg3Cdnj71oyy6Pgbz7Q1TTktxTQNwm3Qc eeClKsQApQm5KAJsTK3uTavn/YSapcyAInFP+LDzLU28I+xoBTiyHY65KsiKJ0uB 9HnZoDWLScAYMSfOO0x7QwvQwZu2/sG9s3zZROQ9L4PXywGplB6yG3PLW8doPKZ1 2AhyS/4+uQKBgQDVfKDPbxFWqCZbIcAeXJtB9fLOdomreuz4wNqMX7qg7ixs7a+o YzCDA4XXXLJrQxuWRhMMcBy9D0yBg+N+lxyU/3KVB94MGk/ht0/6u7KN7Ny+E94D Rp6ZAcTpivdA+Ta+eHzVM4I70kl9A+4h+hZnsZNd1lIXB2OyboznXAqeKwKBgAwl 5GsELy3qd5kGatPUqp72ZXp15maqYNvySn6RdvGy6EmtIbflO9yltkU61NeGXhan 96uWZLmfUqyQWGQ9K402Rna3HxFgFxnCxiM/dPLDjDvlc0LUN1SERAhKBkl+G4J/ XEx72EPuuif/bjH5LXvkN51D9Ts7o5QVOSvZbWvBAoGBAMdmDwIqm158y0WlYijK 6Px+EmORowx4LbESl+uqWpnZKIhGl3YcEsoxvYXZt+hPbDTaGBtzoXW9oeOcX0nO rxqH950ivZXwTee1WJZWrGgeasc3CgXQ/NFEGOZBHNBUSP6u6Uco1xKUSgqSYh16 utR574dBHXKobzhHXfFNa1qQ -----END PRIVATE KEY-----

Next, let's configure nGinx so it can use these SSL certificates and enable HTTPS.

● 299

Raspberri Pi Full Stack UK 200609.indd 299

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 91 • Edit Nginx configuration to use SSL In the previous chapter, you created these two files: • The private self-signed key at /etc/ssl/private/nginx-selfsigned.key • The public self-signed certificate at /etc/ssl/certs/nginx-selfsigned.crt In this chapture, you will edit the application's Nginx95 configuration file to allow a web client to use HTTPS. The Nginx configuration can be modularised using snippet files. This makes it easier to organise the various directives so that we don't end up with a gigantic configuration file. I have broken down the SSL/HTTPS configuration process in four steps. Ensure that you are working as the super user throughout. Step 1: Create a configuration snippet for the key and certificate files Use Vim to create a file named "self-signed.conf", and store it in /etc/nginx/snippets/: vim

/etc/nginx/snippets/self-signed.conf

Copy this content in self-signed.conf: ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt; ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

This is how Nginx will be able to find the SSL files. Step 2: Create a second snippet with the SSL configuration Use Vim to create a file named "ssl-params.conf", and store it in /etc/nginx/snippets/: vim /etc/nginx/snippets/ssl-params.conf

Copy this content in ssl-params.conf: ssl_protocols TLSv1.2; ssl_prefer_server_ciphers on; ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSAAES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384; ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0 ssl_session_timeout 10m; ssl_session_cache shared:SSL:10m; ssl_session_tickets off; # Requires nginx >= 1.5.9 ssl_stapling on; # Requires nginx >= 1.3.7 95.

You may want to consult the Nginx documentation for more details on the various configuration directives: https://docs.nginx.com/nginx/technical-specs/

● 300

Raspberri Pi Full Stack UK 200609.indd 300

06-08-20 15:31

Chapter 91 • Edit Nginx configuration to use SSL

ssl_stapling_verify on; # Requires nginx => 1.3.7 resolver 8.8.8.8 8.8.4.4 valid=300s; resolver_timeout 5s; add_header X-Frame-Options DENY; add_header X-Content-Type-Options nosniff; add_header X-XSS-Protection "1; mode=block";

This file sets the various SSL parameters, such as the version of protocols allowed (TLS v1 .2), and the ciphers to be used . Step 3: Edit the Nginx main application configuration file to use SSL The main Nginx configuration file for our application can be found here: /var/www/lab_app/ lab_app_nginx .conf . You must edit it in order to enable the HTTPS port (port 443), and link it to the configuration snippets . We also have the option of blocking unencrypted requests to port 80, or simply retain port 80 and add encrypted traffic capability through port 443 . Use Vim to open the nginx configuration file: vim /var/www/lab_app/lab_app_nginx.conf

Edit its content like this (I have highlighted the new content in bold text): server { listen 443 ssl; listen [::]:443 ssl; server_name localhost raspberrypi-zero.local 192.168.111.70; include snippets/self-signed.conf; include snippets/ssl-params.conf; charset utf-8; client_max_body_size 75M; location /static { root /var/www/lab_app/; } location / { try_files $uri @labapp; } location @labapp { include uwsgi_params; uwsgi_pass unix:/var/www/lab_app/lab_app_uwsgi.sock; } } server { listen

80;

server_name localhost;

● 301

Raspberri Pi Full Stack UK 200609.indd 301

06-08-20 15:31

Raspberry Pi Full Stack

charset

utf-8;

client_max_body_size 75M; location /static { root /var/www/lab_app/; } location / { try_files $uri @labapp; } location @labapp { include uwsgi_params; uwsgi_pass unix:/var/www/lab_app/lab_app_uwsgi.sock; } }

What you have just done is add a new server configuration block above the existing server block. In the new block, you permit the server to listen for incoming requests on port 443, which is the standard port for HTTPS. You used the "include" directive to include the certificate and SSL configuration snippets. The rest of the server block is identical to the original. Step 4: Restart nGinx Before restarting nGinx so that the new configuration becomes effective, lets's test the configuration files for possible errors: (lab_app) root@raspberrypi-zero:/var/www/lab_app# nginx -t nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/etc/ssl/certs/nginx-selfsigned.crt" nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

In my case, there is only one harmless warning relating to the fact that the certificate is self-issued instead of produced by a certificate authority. Let's restart nGinx: (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl restart nginx (lab_app) root@raspberrypi-zero:/var/www/lab_app#

No error message is a good thing! Your full stack application should now be accessible via HTTPS. Let's test it in the next chapter.

● 302

Raspberri Pi Full Stack UK 200609.indd 302

06-08-20 15:31

Chapter 92 • Test SSL in Firefox, Safari, Chrome

Chapter 92 • Test SSL in Firefox, Safari, Chrome Time to test that your new HTTPS setup works. I suggest you test on multiple browsers, and also try to access your application from outside your network. Use both HTTP and HTTPS to ensure that no functionality is lost during the change of configuration. 1. Test from the local network I have tested this setup on Firefox, Chrome and Safari. I used my browsers to access: • http://raspberrypi-zero.local/lab_temp • https://raspberrypi-zero.local/lab_temp With all three browsers, the HTTP request was successful. All good, no change here. Next, I tested with HTTPS. I found that Firefox and Safari will detect that I am using a self-signed certificate, and they will warn me against continuing (Figure 92.119).

Figure 92.119: Safari is warning me that this website may not be trustworthy. However, they will give me the option to proceed (by clicking on "Show Details"), and access the page (Figure 92.120). Because I am the one who created the certificate, I chose to proceed (by clicking on the tiny link "visit this website" in Figure 92.120).

● 303

Raspberri Pi Full Stack UK 200609.indd 303

06-08-20 15:31

Raspberry Pi Full Stack

Figure 92.120: Safari will allow me to proceed by clicking on "visit this website". Note if this was a website I stumbled upon on the Internet, I would choose not to proceed without a way to verify ownership of this website. Once on the lab_temp page (Figure 92.121), I clicked the padlock icon (see the arrow in the figure below) in the URL field to get more information about the certificate. You can see the certificate belongs to a server named "raspberry-zero.local", which is what I entered in the "Common Name" field when I created this certificate.

Figure 92.121: You can get information about the certificate by clicking on the padlock.

● 304

Raspberri Pi Full Stack UK 200609.indd 304

06-08-20 15:31

Chapter 92 • Test SSL in Firefox, Safari, Chrome

The same process takes place when I access the HTTPS version of my website on Firefox . Once your browser knows that you trust the certificate, it will not request confirmation again . Google Chrome Version 81 .0 .4044 .122 (Official Build) (64-bit) did not allow me to visit my site at all, and I have been unable to find a workaround . It seems that I'll be using Safari and Firefox with my self-signed certificate . 2 . Test from the Internet Earlier in this part of the project, you exposed your application to the Internet by configuring your router to do port forwarding . In the existing port forwarding configuration of your router, you have set it to forward requests from public port 4000 to private port 80 . For example (on my setup), at present, a request to "http://100 .92 .123 .123:4000/lab_ temp" will be forwarded to "https://192 .168 .111 .70:80" . To make port forwarding work with HTTPS request, you must set up a new port-forwarding rule in your router that will forward requests to a different public port, say "4001" to the private HTTPS port "443" . Go ahead and add a new port-forwarding rule in your router, as per the example in Figure 92 .122 .

Figure 92.122: Port-forwarding HTTPS requests. Notice that I am using public port "4001" and private port "443", which is the one I used in the new nGinx configuration file . Once the new port-forwarding rule is effective, test it . Use your smartphone connected to a cellular network with WiFi disabled, or your computer connected to a VPN service, and request https://100 .92 .123 .123:4001/lab_temp (of course, change the IP address to your router's actual public IP address) .

● 305

Raspberri Pi Full Stack UK 200609.indd 305

06-08-20 15:31

Raspberry Pi Full Stack

In Figure 92.123, notice that the SSL padlock is shown, indicating my phone fetched the page from the webserver using HTTPS.

Figure 92.123: Port-forwarding HTTPS request from my smartphone while connected to a cellular network. Nice work! You now have a relatively secure web application published on the Internet, accessible from anywhere in the world. Let's continue this project by adding new features to the full-stack application. Next up, add support for cataloguing using Google Sheet.

● 306

Raspberri Pi Full Stack UK 200609.indd 306

06-08-20 15:31

Part 15: Data logging with Google Sheet

Part 15: Data logging with Google Sheet

● 307

Raspberri Pi Full Stack UK 200609.indd 307

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 93 • What is data logging, and why Google Sheet? Data logging is a term that refers to the recording of data over time. Typically, we want to do data logging in order to maintain historical data for later processing and analysis. You have already set up your full stack application to log data. The data is captured by the "env_log.py" script, and stored in the SQLite3 database. You can view the data in your application by fetching the "lab_env_db" in your web browser (Figure 93.124).

Figure 93.124: Viewing the data logged by your application. You can also process and analyse the logged data in Plotly (Figure 93.125).

● 308

Raspberri Pi Full Stack UK 200609.indd 308

06-08-20 15:31

Chapter 93 • What is data logging, and why Google Sheet?

Figure 93.125: The Plotly viewer showing a chart from the data logged by your application. I'd like to take data-logging a step further. In this part of the project, I will show you how to transfer data from the application on to Google Sheet so you can then do further processing, charting perhaps, or use other Google Drive facilities. By uploading the sensor data to Google Sheet, you will have a copy of all data stored in the database. Google Sheet provides powerful data processing features that you will be able to apply to the data. Aside from the specific features available in Google Chart, the value in implementing this extension to the project is that the process is similar among many Cloud services, outside of Google. You will be able to reuse the knowledge you are about to gain elsewhere. I've had my full-stack application uploading data to Google Sheet for a couple of weeks now. At the time I am writing this sentence, my application has stored 109,658 records in my "Temperature and Humidity" Google Sheet. A new record is automatically inserted every few seconds (Figure 93.126).

● 309

Raspberri Pi Full Stack UK 200609.indd 309

06-08-20 15:31

Raspberry Pi Full Stack

Figure 93.126: A segment of my data-logger Google Sheet In the next few chapters, I will show you how to implement data logging to Google Sheet.

● 310

Raspberri Pi Full Stack UK 200609.indd 310

06-08-20 15:31

Chapter 94 • Set up Google API credentials

Chapter 94 • Set up Google API credentials The first step to using any Google service programmatically is to set up the necessary credentials. These credentials are used to prove to Google that the application requesting to use its resources is authorised to do so. When you complete the process of creating the credentials for your application, you will have a credentials JSON file similar to this: { "type": "service_account", "project_id": "rpifsv3", "private_key_id": "3b0f6096b78dddf7039c9e3eaas64934e49dd5f501dd71", "private_key": "-----BEGIN PRIVATE KEY-----\ nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDK463w2cSJhk9A\ n92hCKEFC+2Fs1dY0K1OE/bDgdP7sKnYtVZZpAfpiiSARP7BOdtUsbeCGFRhtv4gG\ nwc9JMBnNfXDeDT9+M8BB+8pkaka0Ci45ScMmTaX9N7jF2eVVsdKwsWaReDGZaHkW\ nus0lV8evfDyqkugg2B+p6tGSIXUg6uHq6Ix+kJGGmvl7KNyaKF3bkpR+nRs3+553\ nn2hyVDLRFtL2wtxxigRnSZOOhkAqYJfxkoYyenR3TJlhdep0Unu3cvlgu4DEPRsZ\ nbdRNBSZhJnXXFIadfdaf@#@$FD9oi9ZTHSM+js+2mkig6N5FVhlOTJxS4EmWvCkVeqZCwZp\ nRW4KyFKVAgMBAAECggEACFpt5antndTmC/Bzvi+Omk2jwfme31h+UI4qW19txHbu\ nFSk+L1YZwzdOlu82mNBrL/BZK2QQdKpLVS3jFSoSEeQNgtLsT4udaYoOl+pJPW3i\ n39Bx5LCTOjTyWlEBkHDpEgXHyYr3jc+flqg7vIMb8VVt4P756CpFyXblyHywqBl4\ nsmR/K0GXbBvFidmMzVJ0wpbvuCMzbnwJ/x/RVcjSd/wedVVk+X4X0wG6VAbiCu3m\ nBPws4ex2Q4LAsYAVmYF3hSxBT4wSSlC7rJWTtH6QbWmg6o/7CUEQwu9qImqmty91\ nt2ed+pBm0S9f11L67bLHl2BujlmrtpA7TWrWiV/aUQKBgQD9rh9E9A5xZZSfHSKl\ nRu02hisX1HOBSJEA1fJ8T9/7o32rp+R9pjcP0QG2ncsTJJkWW9GiOSMIfSrFLqRK\ nUhb6SEh9fltc1+3tnTY2ZzYFHxVKZ+gBs673c7fKurC31NenhZcwDHYH8wTGvit4\ nmKGE7NH58g5iniSTyk+4epifhQKBgQDMvqdO0pDcoRblHMPDuwBSkbQF2UzFKFJC\ nD3cJWMeKVPCcdgr05jcOMbT2di2pR+nnWdITYQhG9EWgFmCGAq4E0x2hNpDTyXzK\ nW1euZj7/GtSVaq1aLvMjb9UcRMfjupvivqmhGc/m2OW0wvzorLFbrd/4u7K2L7Q/\ nR1oXvXPr0QKBgE/L4Mwo1Q44fMqcOJSyvdVj2Nadkjahds635SDFGWt5uvRmEwoflRrqIUZ\ nrsJL5XUTJ5jq7zjIQyX7QRjffn+uD0HNlqq96AeJoqWOyqnuxOnSB30O7PyNuL4o\ nTkZKYSrlFMeOrRNmA0u677nKaNs3YKc7o7sTX213Ba1hv3G5tt/5uF6dAoGBALlM\ n33zuA186h88Jjn9pkiwoZAZbGSjiijfxaSlTaRxgLXddOiNU64T0JbRdVov7Ys0Y\ nNJGHYwdcclslhWwjpr0FGK7cBI04sLEvpBZZwS/+Z2FnL8cL8M8kmnpAIayQKh83\ nsav1AX0K1jIXuu8VxFqV7btenKpYWEhg2j/cHbrRAoGBAMlAW4fF1VjJ/inNFCVZ\ nR/71Wl7ATsOc2JVq3gv/do2+pThJ+GdM+isTEmdJBe9jAEPiNSH4uRtSZ7jepIwF\ nUQXoRuUJnzu59vdyo/J5NobLc/gOReKjjS29PFqo3XQhjAYKdkyRaRHPVkCey0uU\ n8nh7MtpeHQ7mX77IhpuMnrcF\n-----END PRIVATE KEY-----\n", "client_email": "[email protected]", "client_id": "11696447548129394453077", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/ certs",

● 311

Raspberri Pi Full Stack UK 200609.indd 311

06-08-20 15:31

Raspberry Pi Full Stack

"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/ x509/peters-python-script%40rpifsv3.iam.gserviceaccount.com" }

Start by logging on to your Google account, and head over to https://console.developers. google.com. The Console will look a like the example in Figure 94.127.

Figure 94.127: The Google APIs console. On the top left of the page, click on the black triangle button to bring up the projects pane (indicated by the arrow). The project pane contains a list of your exciting projects, and also allows you to create a new project (Figure 94.128).

● 312

Raspberri Pi Full Stack UK 200609.indd 312

06-08-20 15:31

Chapter 94 • Set up Google API credentials

Figure 94.128: The Google APIs console. I already have a couple of projects that relate to my full-stack application here, but I will go ahead and create a new one. Click on the New Project buttons (indicated by the arrow in (Figure 94.128). You can see the New Project form in Figure 94.129.

Figure 94.129: The New Project form.

● 313

Raspberri Pi Full Stack UK 200609.indd 313

06-08-20 15:31

Raspberry Pi Full Stack

Give your project a sensible name. The form also asks for a billing account, even though this is not a billable project. I have blurred out the Organisation and Location fields because they are private, and depend on how you have set up your Google account. Click on the blue "CREATE" to start the creation process. This is going to take about half a minute. Eventually, you will receive confirmation that the new project is ready (Figure 94.130).

Figure 94.130: Your new project is ready. Click on the VIEW link to go to the new project Dashboard. The Dashboard will look similar to the example in Figure 94.131.

Figure 94.131: My new project dashboard.

● 314

Raspberri Pi Full Stack UK 200609.indd 314

06-08-20 15:31

Chapter 94 • Set up Google API credentials

Click on "Explore and enable APIs" in the "Getting started" box. This will take you to the APIs & Services page (Figure 94.132).

Figure 94.132: Getting started with APIs and Services. Click on Enable APIs and Services. We want to access to the drive, Google Drive API. This will display a search box. Type "drive" in the search box to find the most relevant available API, Google Drive API. Click on "Google Drive API" to select it, and then click on the blue "ENABLE" button to enable it (Figure 94.133).

Figure 94.133: Enable the Google Drive API. With this, the Google Drive API is enabled for our Google API project. You can confirm this by ensuring Google Drive API dashboard is under your project name. In my example of Figure 94.134, you can see the project name is "Pi-zero-FS", as pointed to by the arrow marked "1".

● 315

Raspberri Pi Full Stack UK 200609.indd 315

06-08-20 15:31

Raspberry Pi Full Stack

Figure 94.134: The Google Drive API is enabled for the RPi-zero-FS project. The next thing to do is to create the credentials. Notice the Google Drive API dashboard is informing you to do this via a message box. Click on the "CREATE CREDENTIALS" button to start the wizard (arrow "2" in Figure 94.134). You can see the Wizard form and the appropriate selected options in Figure 94.135.

● 316

Raspberri Pi Full Stack UK 200609.indd 316

06-08-20 15:31

Chapter 94 • Set up Google API credentials

Figure 94.135: Part 1 of the credentials wizard form. Set the credentials attributes as you see them in Figure 94.135: • • • •

Which API are you using? Google Drive API Where will you be calling the API from? Web server What data will you be accessing? Application data Are you planning to use this API with App or Compute Engine? No

● 317

Raspberri Pi Full Stack UK 200609.indd 317

06-08-20 15:31

Raspberry Pi Full Stack

Next, click on the blue button entitled "What credentials do I need?". This will take you to the second part of the credentials wizard. Fill in the form as per the example in Figure 94.136.

Figure 94.136: Part 2 of the credentials wizard. Here is some information about the form fields: • Service account name: use a name that makes sense to you. I have used a name that describes itself instead of something more cryptic. • Role: choose "editor". • Service account ID: this is an email address that the wizard will populate automatically based on the service account name text. Make a note of this email address because you will need to use it in your Python script in the next two chapters. • Key type: choose JSON. When you are ready, click on the blue "Continue" button. The Wizard will create the credentials file and download it to your computer (Figure 94.137).

● 318

Raspberri Pi Full Stack UK 200609.indd 318

06-08-20 15:31

Chapter 94 • Set up Google API credentials

Figure 94.137: The Credentials file is generated and downloaded. Look for this file in your downloads folder and open it with a text editor. Be careful to not introduce any changes to this file or authentication with the Google API will fail! Your credentials file will look similar in structure to the example I gave at the start of the chapter. Of course, the key it contains will be totally different. Click on "CLOSE" to dismiss the wizard box. You can confirm the new service account has been created by looking in the Service Accounts list under "Credentials". Again, ensure you are looking on the correct project by looking at the header of the page (in my example, the project name is "RPi-zero-FS" (Figure 94.138).

● 319

Raspberri Pi Full Stack UK 200609.indd 319

06-08-20 15:31

Raspberry Pi Full Stack

Figure 94.138: The Credentials page shows my new service account for the current project. Before you continue with the next chapter, you must enable another Google API: the Google Sheet API on top of the Google Drive API. From the Credentials page, click on "Dashboard" and then "+ ENABLE APIS AND SERVICES". Search for "google sheet", and click on the Google Sheets API result to select it. Click on the blue "ENABLE" button. Go back to the project dashboard, and confirm that both APIs are listed, as in the example in Figure 94.139.

● 320

Raspberri Pi Full Stack UK 200609.indd 320

06-08-20 15:31

Chapter 94 • Set up Google API credentials

Figure 94.139: The enabled API for this project are listed in the Dashboard. With this, you have completed the setup on the Google side. You created an API project that gives your program access to Google Drive and Google Sheets. You also have a credentials JSON file and service account that your Python program will use to access the Google resources. In the next chapter, I will show you how to use Python to add new rows in a Google sheet. We'll experiment with this in the Python CLI before implementing the new feature in the full-stack application script.

● 321

Raspberri Pi Full Stack UK 200609.indd 321

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 95 • Set up the Python libraries and Google Sheet In the previous chapter you created the credentials file and a service account so your Python program can access Google Drive and Sheet APIs. In this chapter, I will show you how to write a small Python program that will add new rows in a Google Sheet. During prototyping (which is what you will do in this chapter), you will use the Python CLI to experiment with Google Sheets. In the next chapter, you will implement the new Google Sheets feature as an extension to the existing application. Before you can do anything with a Google Sheet, you must create one. Go to https://drive. google.com/, create a new folder for your Google Sheets experiments, and create a new Google Sheet (Figure 95.140).

Figure 95.140: Create a Google Sheet. Give your new sheet a name. Pick a sensible name and make a note of it because you will use it in your Python program later. I called mine "Temperature and Humidity - RpiZero".

● 322

Raspberri Pi Full Stack UK 200609.indd 322

06-08-20 15:31

Chapter 95 • Set up the Python libraries and Google Sheet

Figure 95.141: Give your new sheet a name. You must allow access to this sheet to the service account that you created in the last chapter. If you don't remember what it is, the easiest way to find out is to look at the bottom of the credentials JSON file. My service account is this: peter-s-python-script%40rpi-zero-fs.iam.gserviceaccount.com

The segment "%40" is a URL-encoded entity96, used to replace the character "@" in the JSON file. You can manually change that back to "@", so your service account email address looks normal: [email protected]

Click on the share button on the right side of the Google sheet to bring up the share pane.

96.

Learn about URL encoding (or "percent encoding"): https://en.wikipedia.org/wiki/Percent-encoding

● 323

Raspberri Pi Full Stack UK 200609.indd 323

06-08-20 15:31

Raspberry Pi Full Stack

Figure 95.142: Allow access to this Sheet to your service account. In the text box, copy the email address of your service account, and set its access level to "editor". Click "Share" to finish the process. Before continuing with the Raspberry Pi, add some random data in the sheet so that your Python program has something to retrieve (Figure 95.143).

● 324

Raspberri Pi Full Stack UK 200609.indd 324

06-08-20 15:31

Chapter 95 • Set up the Python libraries and Google Sheet

Figure 95.143: I have added sample data for my Python program to retrieve. Let's continue with the Raspberry Pi. Log in to your Raspberry Pi as "pi", change to the super user, go into your application directory, and activate the Python virtual environment. It is time to upload the credentials file you created in the previous chapter to your Raspberry Pi. Use Cyberduck or any SFTP utility to do this. It is also possible to simply copy and paste the contents of the credentials file into a new text file in your application directory using Vim. After all, the credentials file contains plain text. I chose to upload the JSON file using Cyberduck. Ensure the file is in the application directory: (lab_app) root@raspberrypi-zero:/var/www/lab_app# ls -al *.json -rw-r--r-- 1 root root 2334 May

8 02:23 RPi-zero-FS-cdd8c3b3bab4.json

To achieve our the objective of interacting with a Google Sheet through Python, you must install two Python libraries: "gspread97" and "oauth2client98". The former allows your Python script to interact with a Google Sheet. The later takes care of the authentication. Install the two libraries with pip: (lab_app) root@raspberrypi-zero:/var/www/lab_app# pip install gspread oauth2client

Now, let's do some testing on the Python CLI. Start the Python CLI with "python": (lab_app) root@raspberrypi-zero:/var/www/lab_app# python Python 3.8.2 (default, Mar 14 2020, 01:38:54) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> 97. 98.

The documentation for "gspread" is at https://gspread.readthedocs.io/en/latest/. The documentation for "oauth2client" is at https://oauth2client.readthedocs.io/en/latest/

● 325

Raspberri Pi Full Stack UK 200609.indd 325

06-08-20 15:31

Raspberry Pi Full Stack

Here is the full test program, followed by a description: >>> import gspread >>> from oauth2client.service_account import ServiceAccountCredentials >>> scope = ['https://spreadsheets.google.com/feeds','https://www. googleapis.com/auth/drive'] >>> creds = ServiceAccountCredentials.from_json_keyfile_name('RPi-zero-FScdd8c3b3bab4.json', scope) >>> client = gspread.authorize(creds) >>> sheet = client.open("Temperature and Humidity - RpiZero").sheet1 >>> my_sensor_data = sheet.get_all_values() >>> print(my_sensor_data) [['1'], ['2'], ['3'], ['4'], ['5'], ['6'], ['7']] >>>

The program begins by importing the two Python libraries, "gspread" and "oauth2". It creates a new array variable called "scope" in which we store the resources that we'll be using further in the program. These are the only resources that our security credentials give access to. We then create a credentials object called "creds" by reading the JSON credentials file and passing the scope array. Remember to replace the example filename of the credentials file with your own. The next line uses the "authorize" method of the "gspread" library to authorise our program to access the sheet. If this authorisation request is successful, our program can continue to work with the sheet. After authorisation is granted, the program creates a new sheet object titled "sheet". Remember to replace the example name of the Google sheet with yours. Once we have the sheet object, we can interact with it, reading, editing or writing values. We use the "get_all_values" method of the "sheet" object to get all existing data we entered in the sheet. Finally, the program uses the "print" method to print data in the console. You can see the output contains the test data that I entered in my Google Sheet earlier. Now that you know your Python program can access your Google Sheet, you can continue to the next chapter. There, I will show you how to extend the functionality of your full stack application so that sensor data is logged in Google Sheet.

● 326

Raspberri Pi Full Stack UK 200609.indd 326

06-08-20 15:31

Chapter 95 • Set up the Python libraries and Google Sheet

Once you have completed the addition of the new code, go ahead and test your new feature. If you don't want to wait for Cron to trigger env_log.py, run it on the command line: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python env_log.py

Wait for a few seconds, and then look for the new row in your Google Sheet (Figure 95.144):

Figure 95.144: The first row with actual sensor data from the updated env_log.py logging script

● 327

Raspberri Pi Full Stack UK 200609.indd 327

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 96 • Implement of Google Sheet data logging Time to add the new Google Sheet cataloguing feature to the application . Before you proceed, take a moment to clear the test data from your Google Sheet . Your application is already logging data to the SQLite3 database . The Python script that is responsible for data-logging is "env_log .py", and it is invoked automatically through a Cron schedule . The simplest way to implement the Google Sheet data-logging feature is to add the necessary code in the existing file instead of creating a new one . Let's do this . You can see the current version of the env_log .py99 file in the project repository . The new version of this file100, containing the Google sheet feature is also in the repository, so you can easily reference both . I have copied the relevant parts of the new version of env_log .py below, and highlighted the new code in bold . import sqlite3 import sys import Adafruit_DHT from time import gmtime, strftime import gspread from oauth2client.service_account import ServiceAccountCredentials import RPi.GPIO as GPIO

## Import GPIO Library

def log_values(sensor_id, temp, hum): GPIO.output(pin, GPIO.HIGH)

## Turn on GPIO pin (HIGH)

conn=sqlite3.connect('/var/www/lab_app/lab_app.db')

#It is

important to provide an #absolute path to the database #file, otherwise Cron won't be #able to find it! curs=conn.cursor() curs.execute("""INSERT INTO temperatures values(datetime(CURRENT_ TIMESTAMP, 'localtime'), (?), (?))""", (sensor_id,temp))

#This will store

the new record at UTC curs.execute("""INSERT INTO humidities values(datetime(CURRENT_ TIMESTAMP, 'localtime'), (?), (?))""", (sensor_id,hum))

#This will store

the new record at UTC conn.commit() conn.close() # Create a new record in the Google Sheet 99. The current version of env_log.py: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/env_log.py 100. The new version of env_log.py with the Google Sheet feature: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env-GSHEET.py

● 328

Raspberri Pi Full Stack UK 200609.indd 328

06-08-20 15:31

Chapter 96 • Implement of Google Sheet data logging

scope = ['https://spreadsheets.google.com/feeds','https://www. googleapis.com/auth/drive'] creds = ServiceAccountCredentials.from_json_keyfile_name('/var/www/ lab_app/raspberry-pi-full-stack-1-f37054328cf5.json', scope) client = gspread.authorize(creds) sheet = client.open('Temperature and Humidity').sheet1 row = [strftime("%Y-%m-%d %H:%M:%S", gmtime()),sensor_id,round(temp,2),round(hum,2)] sheet.append_row(row) GPIO.output(pin, GPIO.LOW)

## Turn off GPIO pin (LOW)

As you can see, the original code is left intact. I have simply inserted the new code that is relevant to the Google Sheet feature at the end of the log_values method. The only difference between the code you saw in the previous chapter and the code in this chapter are the last two lines: row = [strftime("%Y-%m-%d %H:%M:%S", gmtime()),sensor_id,round(temp,2),round(hum,2)] sheet.append_row(row)

To insert a new row in a Google sheet, use the "append_row" method of the sheet object, and pass a row as a parameter. A row is an array of values. Each row contains 4 columns: • The time, acquired by the Python gmtime() method, and formatted using the "strftime" method. • The sensor ID. • The temperature value, rounded to two decimal points. • The humidity value, rounded to two decimal points.

● 329

Raspberri Pi Full Stack UK 200609.indd 329

06-08-20 15:31

Raspberry Pi Full Stack

Part 16: Set up a remote Arduino sensor node with the nRF24

● 330

Raspberri Pi Full Stack UK 200609.indd 330

06-08-20 15:31

Chapter 97 • Why set up an Arduino remote node?

Chapter 97 • Why set up an Arduino remote node? In this part of the project, you will expand your Raspberry Pi gadget so it can receive sensor data from a remote node. The node consists of an Arduino Uno, which is connected to a nRF24 transceiver module, in addition to a DHT22 sensor. This Arduino node will be able to communicate with the Raspberry Pi base module and will be able to send it data from the sensor. The Raspberry Pi will receive the data and record it in its database, in addition to the data from its own sensor. Despite the fact the implementation of the remote node functionality requires extensive work on the back end of the application, changes to the front-end are minimal. There is only a new text field that allows the user to type in the sensor ID for the records to be retrieved (Figure 97.145).

Figure 97.145: The only change in the front end is the addition of the Sensor ID field. In Figure 97.145, I use the arrow to point to the only new element of the page, the Sensor ID text field. In this example, I have retrieved the records that fall within the specific datetime range for my remote Arduino node, that has a Sensor ID of "3". The Sensor ID of my Raspberry Pi is "1". Over the next few chapters, I will show you how to implement this new feature. Please keep in mind that although in my implementation I have used a single remote Arduino, you can implement dozens using the same architecture. This is because of the nRF24 transceiver hardware and network module of the RF24 library for the Arduino. The

● 331

Raspberri Pi Full Stack UK 200609.indd 331

06-08-20 15:31

Raspberry Pi Full Stack

hardware and software combined make it possible to create nRF24 networks with multiple nodes. In the next chapter, I'll show you how to connect an Arduino Uno to the nRF24 transceiver, and then discuss the firmware (the "Arduino sketch") that drives this hardware. Later, you will continue your work with the Raspberry Pi.

● 332

Raspberri Pi Full Stack UK 200609.indd 332

06-08-20 15:31

Chapter 98 • The Arduino node wiring

Chapter 98 • The Arduino node wiring The Arduino node will use a nRF32 transceiver to communicate with the Raspberry Pi. The nRF24 is a mature and low-cost wireless technology that uses the 2.4 GHz band to implement low-power but high-performance communications between microcontrollers. The nRF24L chips are manufactured by Nordic Semiconductor, and use the Enhanced ShockBurst protocol (ESB). Whilst there are many options you can choose that allow for wireless communications between an Arduino and Raspberry Pi, the nRF24 stands out. There are a few reasons for this: 1. Y  ou can use the exact same hardware with the Arduino and Raspberry Pi. No need for converters, level shifters, or "special" modules". 2. T  he software libraries on the Arduino are mature and complete. The library you will use has been around for many years and is very stable and efficient. 3. O  n the Raspberry Pi, there is a very efficient and also stable C library for the nRF24. In addition, this library comes with a convenient wrapper for Python, so you can use the nRF24 directly from your Python programs. 4. C  ost: This is the cheapest transceiver module I have come across, at around US$2.5. For its simplicity of use and reliability, nothing comes close. Take a minute to connect your nRF24 and DH22 to your Arduino. Here is the list of parts: • • • • •

An nRF24 transceiver module. A DHT22. A 10KΩ resistor. A 22μF capacitor. A breadboard to hold everything.

The resistor is used to pull-up the data line of the sensor. The capacitor is used as a bypass for the nRF24. In Figure 98.146 you can see the nRF24 module attached to a simple breakout101 I designed. The breakout includes a socket for the bypass capacitor. Because the nRF24 header consists of 2 rows, it is not compatible with the breadboard. The breakout converts the 2x4 header of the transceiver to a 1x8 row that you can plug into the breadboard, as in the picture below.

101. You can download the Gerber files for this breakout so that you can order a copy from any online PCB manufacturer. Please get the ZIP from the course parts list page: https://techexplorations.com/parts/rpifs-parts/

● 333

Raspberri Pi Full Stack UK 200609.indd 333

06-08-20 15:31

Raspberry Pi Full Stack

Figure 98.146: The nRF24L breakout. You can see the list of connections in Figure 98.147.

Figure 98.147: The connections details for the Arduino node Notice the nRF24 module Vcc pin should be connected to the Arduino 3.3V pin. Take care to connect the capacitor in the correct orientation, and the rest of the pins as I show in Figure 98.147. I have decided to power the DHT22 sensor from one of the Arduino digital pins instead of the 5V pin. This way, I will be able to completely turn off the sensor when not in use. By turning off the sensor we can reduce the amount of power drawn from the power supply (or the battery) when the sensor is not being used to take a reading. Double-check before you apply power!

● 334

Raspberri Pi Full Stack UK 200609.indd 334

06-08-20 15:31

Chapter 98 • The Arduino node wiring

You can see the end result in Figure 98.148.

Figure 98.148: A view of my Arduino node. In this photograph, you can see how I connected the nRF24 breakout PCB to the breadboard. A mini breadboard is large enough to accomodate the nRF24 breakout and the DHT22 with its pull-up resistor. When you are ready, continue with the next chapter where you will work on the Arduino sketch.

● 335

Raspberri Pi Full Stack UK 200609.indd 335

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 99 • The Arduino node sketch With the nRF24 transceiver connected to your Arduino, let's continue with the Arduino sketch. The purpose of this sketch is to take readings from the DHT22 sensor on a periodic schedule and transmit the values to the Raspberry Pi. A desirable feature of this sketch is to make efficient use of the limited hardware it will operate on. Imagine this hardware will operate away from mains power, and will depend on batteries. The sketch, aside from reading and transmitting sensor data, should also be good at minimising current draw so the battery can last as long as possible. The sketch I have written for the Arduino node is based on three libraries: • The RF24 main library102, and its subordinate RF24Network module103. This will allow us to easily use the nRF24 module in networking mode to reliably transmit sensor values as data packets. The documentation104 for all parts of the RF24 project, including for C and Python components used on the Raspberry Pi are recommended reading. • The Adafruit DHT library105. This will allow us to work with the DHT22 sensor. This library has two dependencies: the Adafruit DHT Unified Sensor, and the Adafruit ADXL343 libraries. If you use the Arduino IDE Library Manager, it will prompt you to install the dependencies along side the DHT library. You should choose "Install all" at the prompt. • The LowPower library106. This will make it very easy to put the Arduino to sleep when it not doing anything useful. At the time I am writing, LowPower is not available through the Arduino IDE Library Manager, so you should install it manually. Go ahead and install the necessary libraries in your Arduino IDE. You can use the Library Manager for RF24, RF24Network, and Adafruit DHT. You will need to install the LowPower library manually. You can see the full source code of the Arduino sketch in the project repository107. In this chapter, the code is not complicated. Please take a few minutes to read and become familiar with it. In the remainder of this chapter, I will focus on specific components and features of the sketch.

102. You can find the RF24 for Arduino main library on Github: https://github.com/nRF24/RF24 103. You can find the RF24Network for Arduino library on Github: https://github.com/nRF24/RF24Network 104. The documentation for the RF24 project is at http://tmrh20.github.io/RF24/ 105. You can find the Adafruit DHT library on Github: https://github.com/adafruit/DHT-sensor-library 106. You can find the LowPower library on Github: https://github.com/rocketscream/Low-Power 107. The full source code of the Arduino sketch: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/Arduino-rf24-dht22.ino

● 336

Raspberri Pi Full Stack UK 200609.indd 336

06-08-20 15:31

Chapter 99 • The Arduino node sketch

Power management I envision Arduino RF24 modules used in remote locations, away from mains power, dependent on a battery. Naturally, we want to prolong the life of the node before we have to replace the battery, so it is important to reduce power consumption. A typical approach to power management entails these considerations: • Program any useful work to take place periodically. The frequency should be as low as possible, and the work should be completed as quickly as possible. In the case of our Arduino node, the work consists of taking a measurement from the sensor and transmitting the readings to the Raspberry Pi. A reasonable frequency is once every 5 or 10 minutes. The whole cycle of reading and transmitting should be possible to complete within 2 to 5 seconds. If, say, the frequency is set to 10 minutes and the read & transmit cycle can be completed in 5 seconds, then the Arduino will not be doing anything for 99% of the time. • Identify any peripherals that can be switched off when not needed. For example, the DHT sensor is only needed for a few seconds at a time when the Arduino has to take a measurement. For the rest of the time, it can be turned off. • The Arduino is only doing useful work when it reads the sensor and transmits data to the Raspberry Pi. This is only 1% of the time. For the rest 99%, the Arduino, and specifically its microcontroller should be placed in low-power or sleep mode. Based on the above considerations, once we understand how the Arduino will operate, we can proceed to design a circuit and sketch to reduce power consumption as much as possible. I have made a few simple provisions to do so: • The DHT22 power pin (4) draws power from an Arduino digital pin instead of the regular 5V. The digital pin can provide enough current to feed the sensor. When we need the sensor to be turned on, we can simply take the digital pin to HIGH. When we want to turn it off, we take the digital pin to LOW. In the sketch, you can see that on line 45, I defined pin 5 as the DHT power pin. In the loop, on lines 107 and 114, I take pin 5 HIGH (to turn on the sensor), and LOW (to turn off the sensor). Also notice that on line 110 I inserted a 2 second delay to ensure the DHT sensor has enough time to "boot" before taking a reading. • I use the facilities of the LowPower library to put the ATMega328 microcontroller to low-power mode when it is not doing something useful. This is done with a single line of code on line 104, inside the loop(). There, I set the low-power mode to have a duration of 8 seconds, and turn most of the microcontroller's subsystems off. The 8 seconds value is the highest the library provides, but there is the option to keep the low-power mode forever, until an external interrupt connected to (for example) pin 2 takes the microcontroller out of sleep. This would require an external timer circuit, and for keeping this project as simple as possible I de-

● 337

Raspberri Pi Full Stack UK 200609.indd 337

06-08-20 15:31

Raspberry Pi Full Stack

cided to adopt the 8 seconds option. Note you can put the microcontroller into low-power mode multiple times, for 8 seconds each time, thus creating much longer, low-power periods. With this simple setup, the Arduino will enter a new loop, sleep for 8 seconds, wake up, turn on the DHT sensor, wait for the sensor to boot, take a reading, transmit the data to the Raspberry Pi, turn off the DHT sensor, and repeat the loop. The nRF24 module cannot be switched on and off in the same way as the DHT sensor. I have chosen to power it from the Arduino 3.3V pin which will always supply current (even when the Arduino is in low-power mode). Data transmission The sensor data must be transmitted to the Raspberry Pi as a packet. The packet must contain the correct header which includes the recipient and sender addresses. On lines 59 and 60 I have set the address of the Arduino node ("3") and Raspberry Pi ("0"). To learn how to address your nodes, you should read the RF24Network documentation108 and in particular the section titled "Octal Addressing and Topology". Later in this part of the project, in the chapter entitled "The Raspberry Pi nRF24 receiver script", I will show you how to set the address for the Raspberry Pi. Because the RF24 network topology is dependent on a master node, the header of each data packet must contain the address of the master node. On line 63, I set the header address to be that of the Raspberry Pi. In an RF24 network, a recipient will send an acknowledgement back to the sender. We want to capture the acknowledgment so the recipient knows the last packet it send was received successfully. The RF24 transceiver will capture it and place the acknowledgement in a buffer, until we request it in our sketch. This is what line 120 is about. On line 120, I call the "update()" function of the network object. The update() function is essential for other functions, such as routing packets through the network, receiving and sending. Without calling the update function regularly, the network will stop working. The interesting part of the sketch starts on line 129. Then, I start to construct the data packet based on the values read from the sensor. The packet is conracted as a string. The C language offers a variety of methods to create strings, but the easiest one (though, not most efficient) is to use the String object. We can concatenate substrings together to end up with a string that contains all the information we want to transmit. In this example, between lines 129 and 132, I construct a string that looks like this: 51.23,23.09

108. Read about addressing and topology of an RF24 network: http://tmrh20.github.io/RF24Network/

● 338

Raspberri Pi Full Stack UK 200609.indd 338

06-08-20 15:31

Chapter 99 • The Arduino node sketch

The two values are separated by a comma. As long as the recipient knows about this format (which it does since will program it accordingly), it will be able to decode the string into the individual values. On line 141, I use the "write()" function to transmit the data packet. This function requires to packet to be formatted as a character array. It will not work with a String object. To convert a String object to a character array, we can use the "toCharArray()" function which is available to String objects. This is what I do on line 138. The "toCharArray()" function requires a character array in which it will store the data from the String object, and the size of the String object. I create a character array on line 137, called "payload". Notice the size of the array is "12", which is the total number of characters in the String object "values_string" (such as the example "51.23,23.09"), plus one. Compile and upload to Arduino I hope the discussion above has helped to clarify how this sketch works. Go ahead and compile the sketch, and then upload it to the Arduino. Then, set the Arduino aside and continue your work with the Raspberry Pi.

● 339

Raspberri Pi Full Stack UK 200609.indd 339

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 100 • Raspberry Pi and nRF24 wiring It's time to turn our attention to the Raspberry Pi. The Raspberry Pi will act as the base node of the RF24 network, and is responsible for processing the sensor values it receives from the Arduino node (or nodes, if you have more than one). In this chapter, I will show you how to connect the RF24 to the Raspberry Pi. You can choose to do the wiring on a breadboard, or to use a special HAT I designed for this purpose. I implemented my Raspberry Pi RF24 module using the HAT breakout. You can see it in action in Figure 100.149. The HAT accommodates the nRF24 transceiver module, the DHT22, a momentary button, and two LEDs (power indicator, and activity indicator). You can download the Gerber files for the HAT from the project's part list page109. You can also order this PCB directly from PCBWay110.

Figure 100.149: The Raspberry Pi HAT on a Raspberry Pi Zero W. If you prefer to prototype this circuit using the breadboard, please consult the connections table in Figure 100.150, and the schematic diagram of the HAT PCB in Figure 100.149. You can get a high resolution version of the HAT schematic from the same project parts list111.

109. Get the Gerber files for the HAT board from the project list of parts page: https://techexplorations.com/parts/rpifs-parts/ 110. You can order this HAT PCB (un-populated) from PCBWay: https://www.pcbway.com/project/shareproject/Raspberry_Pi_Full_Stack_RF24_and_DHT22_HAT.html 111. A high-resolution version of the schematic is available at the project parts list: https://techexplorations.com/parts/rpifs-parts/

● 340

Raspberri Pi Full Stack UK 200609.indd 340

06-08-20 15:31

Chapter 100 • Raspberry Pi and nRF24 wiring

In the following list I include, all of the parts that you will need to implement that schematic diagram of Figure 100.151. With some of these parts, such as the button and DHT22 sensor, you have already connected in earlier parts of this project. If you are working your way through the project, the only new components are the nRF24 transceiver and its bypass capacitor. Parts needed: • • • • • • •

One DHT22 One nRF24 breakout. Two resistors, 10 kΩ Two resistors 330 Ω One LED red (power indicator) One LED blue (activity indicator) One electrolytic capacitor 220 μF or similar

Figure 100.150: The connection details for the nRF24 and DHT22 modules.

● 341

Raspberri Pi Full Stack UK 200609.indd 341

06-08-20 15:31

Raspberry Pi Full Stack

Figure 100.151: The schematic diagram of the Raspberry Pi HAT for the nRF24 and DHT22 breakout. Once you have assembled the circuit, continue to the next chapter where you will implement the Python script that handles the nRF24 communications.

● 342

Raspberri Pi Full Stack UK 200609.indd 342

06-08-20 15:31

Chapter 101 • The Raspberry Pi nRF24 receiver script

Chapter 101 • The Raspberry Pi nRF24 receiver script In this chapter, I will explain the functionality of the Python receiver script that runs on the Raspberry Pi and takes care of the nRF24 communications. Note this script will not work "out of the box". It depends on the RF24 and RF24Network C-language drivers and the Python wrappers that I will show you how to set up in the next chapter. For now, let's concentrate on the Python receiver script. You can see the full source code of this script in the project repository112. I have written this script so that eventually it can run as a background process controlled by systemd (similar to the way you already set up the web application script earlier in this project). I will show you how to do this later, as first we must be sure the script runs properly on the command line. Here is the script, with most comments removed to make it fit in these pages (you should always check the repository for an updated version of the script, before copying it to your project): from __future__ import print_function import time from struct import * from RF24 import * from RF24Network import * import sqlite3 import sys from time import gmtime, strftime import gspread from oauth2client.service_account import ServiceAccountCredentials import requests import RPi.GPIO as GPIO def log_values(sensor_id, temp, hum): GPIO.output(pin, GPIO.HIGH) conn=sqlite3.connect('/var/www/lab_app/lab_app.db') curs=conn.cursor() print("Update database...") curs.execute("INSERT INTO temperatures values(datetime(CURRENT_ TIMESTAMP, 'localtime'), ?, ?)", (sensor_id,float(temp))) curs.execute("INSERT INTO humidities values(datetime(CURRENT_ TIMESTAMP, 'localtime'), ?, ?)", (sensor_id,float(hum))) conn.commit() 112. The source code of the rf24_receiver.py script is available on the project repository: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/rf24_receiver.py

● 343

Raspberri Pi Full Stack UK 200609.indd 343

06-08-20 15:31

Raspberry Pi Full Stack

conn.close() print("Update Google Sheet...") scope = ['https://spreadsheets.google.com/feeds','https://www. googleapis.com/auth/drive'] creds = ServiceAccountCredentials.from_json_keyfile_name('/var/www/ lab_app/raspberry-pi-full-stack-1-f37054328cf5.json', scope) client = gspread.authorize(creds) sheet = client.open('Temperature and Humidity').sheet1 row = [strftime("%Y-%m-%d %H:%M:%S", gmtime()),sensor_id,temp,hum] sheet.append_row(row) GPIO.output(pin, GPIO.LOW) GPIO.setwarnings(False) pin = 7

## GPIO 7 connected is nRF24 CE

GPIO.setmode(GPIO.BOARD)

## Use BOARD pin numbering

GPIO.setup(pin, GPIO.OUT)

## Set pin 7 to OUTPUT

radio = RF24(RPI_V2_GPIO_P1_26, RPI_V2_GPIO_P1_24, BCM2835_SPI_SPEED_8MHZ) network = RF24Network(radio) octlit = lambda n: int(n, 8) this_node = octlit("00") radio.begin() time.sleep(0.1) network.begin(90, this_node)

# channel 90

radio.printDetails() packets_sent = 0 last_sent = 0 while 1: network.update() while network.available(): header, payload = network.read(12) print("payload length ", len(payload)) print(payload.decode()) values = payload.decode().split(",") try: print("--------") print("Temperature: ", values[1]) print("Humidity: ", values[0]) print("Sensor ID: ", header.from_node)

● 344

Raspberri Pi Full Stack UK 200609.indd 344

06-08-20 15:31

Chapter 101 • The Raspberry Pi nRF24 receiver script

print("Header Type: ", str((chr(header.type)))) print("--------") temperature = float(values[1][0:5]) humidity

= float(values[0][0:5])

log_values(header.from_node, temperature, humidity) except: print("Unable to process the data received from node ", header. from_node, ". Data: ", values ) print("---------") time.sleep(1)

Much of this script should be familiar to you. The content of the function "log_values()" is an almost perfect copy of the same function in script env_log.py that is already set to run based on a Cron schedule. This function will simply receive sensor values as parameters and store them in the local database and Google Sheet on the Cloud. Below I list and discuss the elements of the script that you have not seen before, and in particular those that relate to the nRF24 communications. • Right after the definition of the log_values() function, the script sets up the nRF24 module. It first initialises the "radio" variable using the RF24 constructor. This constructor is part of the RF24 Python wrapper library that allows us to use the RF24 C driver from within our Python script. You can learn about this function by studying the driver's source code113. • Once the radio object is created and initialised, the script creates the "network" object using the RF24Network constructor. This is the object that makes it possible for the Raspberry Pi to receive a transmission from the Arduino node. The only parameter needed to create the network object is the "radio" object we created on the previous line. You can learn more about the RF24Network constructor in the source code of RF24Network.h114, on line 373. • On the next line, with "octlit = lambda n: int(n, 8)" we create a lambda function which we use later to convert a decimal number into an octal number. We do this because the RF24 Network driver uses the octal system for the node addresses. In Python, a "lambda"115 function is similar to a regular function (where you use the "def" keyword), but has no name. They are convenient to use when you want to do things like evaluate a single expression, as is the case here: we pass a decimal number to "octlit", and the lambda function will return its octal-base equivalent using the Python "int()"116 function. 113. See line 137 of RF24.h: https://github.com/nRF24/RF24/blob/master/RF24.h#L137 114. See line 373 of RF24Network.h: https://github.com/nRF24/RF24Network/blob/master/RF24Network.h#L373 115. Learn about "lambda expressions": https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions 116. Learn about "int()": https://docs.python.org/3/library/functions.html#int

● 345

Raspberri Pi Full Stack UK 200609.indd 345

06-08-20 15:31

Raspberry Pi Full Stack

• In "this_node = octlit("00")", we use the "octlit" lambda to get the octal-base equivalent of "00", and store the value in the "this_node" variable. This is the RF24 network address of the Raspberry Pi. • In the next six lines, until the "while" block, the script will: - start the RF24 radio, - wait for 0.1 seconds for the radio to become ready, - start the network at channel 90, - print the radio and network configuration to the console, - reset the "packets_sent" counter to zero, - and reset the "last_sent" counter to zero. • Now the radio and network are ready, the script enters an infinite loop during which it waits for a transition from an Arduino node. • At the start of each loop, it calls the "update()"117 function of the network object. This checks for you messages. • If there is no new message, the script goes to sleep for 1 second. After this the loop restarts. • If there is a new message, the script uses the "read()"118 function to read the 12 bytes (the payload) of the message, and pass them the "payload" variable. The read function will also get the header of the message and pass it to the "header" variable. Remember in the Arduino sketch, we created a 12-byte payload that looks like this: "51.23,23.09". • On the next two lines, I use "print" to print out the payload to the console. This was useful as I was working to decode the payload into numbers in the "try" block that follows. • The "payload" variable contains a string, that looks like this: "51.23,23.09". The script uses "decode()"119 to convert it into a UTF-8 encoding, and the "split()"120 function to slip the temperature and humidity values into an array of two substrings, using the "," as the delimiter. The two values are stored in the "values" variable (an array of strings). • In the "try" block, the script uses the "float()"121 function to extract the numbers stored in the payload character array and convert them into a floating point number. The "float()" function receives a string, and if it can be converted into a number, it returns it. 117. See https://github.com/nRF24/RF24Network/blob/master/RF24Network.h#L413 118. See https://github.com/nRF24/RF24Network/blob/master/RF24Network.h#L467 119. Learn about "decode()": https://docs.python.org/3/library/stdtypes.html#bytes.decode 120. Learn more about "split()": https://docs.python.org/3/library/stdtypes.html#str.split 121. Learn about "float()": https://docs.python.org/3/library/functions.html#float

● 346

Raspberri Pi Full Stack UK 200609.indd 346

06-08-20 15:31

Chapter 101 • The Raspberry Pi nRF24 receiver script

• With "float(values[1][0:5])", the script takes the first 4 characters of the string stored in index 1 of the "values" array, and uses the "float()" function to convert it into a number. This is the numerical value stored in the "temperature" variable. • The same happens with the humidity value using "float(values[0][0:5])". • If any of these two conversations fail, the "try" block exits and an error message is printed on the console. A conversion can fail if the payload string is not formatted properly by the Arduino, or perhaps the payload becomes corrupted during transmission and receipt. Interference may be the cause of such corruption. Take as much time as you need to study this code so you are comfortable with it. When you are ready, copy it into your application directory. Use Vim to create a new file titled "rf24_receiver.py". In the Vim buffer, copy the code from the project repository.122 Before you can test the RF24 communications, you need to compile the RF24 and RF24Network C language drivers and Python wrappers. You will do this in the next chapter. But before you do this, I want to show you what the script you just learned looks like when running (Figure 101.152). I have annotated this screenshot so you can see the printouts embedded in the script.

Figure 101.152: Example RF24 receiver script output. Let's continue to the next chapter with the setup of the RF24 and RF24Network drivers. 122. This is the latest version of the file rf24_receiver.py: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/rf24_receiver.py

● 347

Raspberri Pi Full Stack UK 200609.indd 347

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 102 • H  ow to install the Python nRF24 modules on the Raspberry Pi In this chapter I will show you how to compile and install the C-language drivers and relevant Python wrappers for the nRF24 module. There are two modules involved: the main RF24 module, and RF24Network sub-module. There are several forks of the RF24 project, but the one you will be using is maintained by TMRh20123. This is optimised for the Raspberry Pi. You can obtain the source code for the two modules from their respective Github repositories: • • • • • •

RF24 repository: https://github.com/nRF24/RF24 RF24Network repository: https://github.com/nRF24/RF24Network For each module, the installation process includes these steps: Download the source code of the module from Github. Compile and install the C driver. Compile and install the Python wrapper.

Before you begin, ensure the SPI interface on your Raspberry Pi is enabled. The nRF24 module uses SPI to communicate with the Raspberry Pi. To check this (and enable SPI if not already enabled), log in to your Raspberry Pi and type the following on the command line: sudo raspi-config

Use the keyboard arrows and Enter key to navigate to "Interfacing Options", and then "SPI". Enable SPI if necessary, and exit raspi-config. Continue by updating Raspbian: sudo apt-get update sudo apt-get upgrade

You will download the source code of the modules in your application directory. You also want to compile the Python wrappers with your application Python virtual environment activated so that they are available to your application Python scripts. Prepare your work with these commands: $ sudo su # cd /var/www/lab_app/ # . bin/activate # mkdir rf24libs # cd rf24libs 123. The "home" of this project is at https://tmrh20.github.io/RF24/

● 348

Raspberri Pi Full Stack UK 200609.indd 348

06-08-20 15:31

Chapter 102 • How to install the Python nRF24 modules on the Raspberry Pi

You are now ready to begin the compilation and installation of the two modules . The following contains a lot of input/output in the console . I have highlighted my commands with bold text . Text that is not bold is output from the console . RF24 compilation and installation Create a clone of the RF24 source code, from https://github .com/tmrh20/RF24 .git (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs# git clone https://github.com/tmrh20/RF24.git RF24 Cloning into 'RF24'... remote: Enumerating objects: 46, done. remote: Counting objects: 100% (46/46), done. remote: Compressing objects: 100% (41/41), done. remote: Total 3545 (delta 16), reused 15 (delta 4), pack-reused 3499 Receiving objects: 100% (3545/3545), 1.54 MiB | 679.00 KiB/s, done. Resolving deltas: 100% (2119/2119), done.

In your source code directory you now have a new directory: "RF24" . (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs# ls -al total 12 drwxr-xr-x

3 root root 4096 May 13 01:38 .

drwxr-xr-x 14 root root 4096 May 13 01:39 .. drwxr-xr-x

9 root root 4096 May 13 01:38 RF24

Enter the RF24 directory and install the driver: (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs# cd RF24/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24# make install [Running configure] [SECTION] Detecting arm compilation environment. [OK] arm-linux-gnueabihf-gcc detected. [OK] arm-linux-gnueabihf-g++ detected. … (omitted output)… [Installing Libs to /usr/local/lib] [Installing Headers to /usr/local/include/RF24]

The C driver is now installed . Let's work on the Python wrapper . This operation can take several minutes on the slower Raspberry Pi Zero W . (lab_app) root@rpi4:/var/www/lab_app/rf24libs/RF24/pyRF24# cd pyRF24/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24/pyRF24# python setup.py install

● 349

Raspberri Pi Full Stack UK 200609.indd 349

06-08-20 15:31

Raspberry Pi Full Stack

running install running bdist_egg running egg_info creating RF24.egg-info … (omitted output)… Installed /var/www/lab_app/lib/python3.8/site-packages/RF24-1.3.4-py3.8linux-armv6l.egg Processing dependencies for RF24==1.3.4 Finished processing dependencies for RF24==1.3.4

The RF24 module is now installed, and the Python wrapper is ready to use. Continue with RF24Network. RF24Network Go back to the root of the rf23libs directory and create a clone of the RF24Network source code, from https://github.com/nRF24/RF24Network.git. (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24/pyRF24# cd ../.. (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs# git clone https://github.com/nRF24/RF24Network.git Cloning into 'RF24Network'... remote: Enumerating objects: 2249, done. remote: Total 2249 (delta 0), reused 0 (delta 0), pack-reused 2249 Receiving objects: 100% (2249/2249), 1.60 MiB | 733.00 KiB/s, done. Resolving deltas: 100% (1342/1342), done.

The cloning operation has created a new directory, "RF24Network". Change into the new directory and compile the source code: (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs# cd RF24Network/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24Network# make install g++ -Wall -fPIC

-c RF24Network.cpp

g++ -shared -Wl,-soname,librf24network.so.1

-o librf24network.so.1.0

RF24Network.o -lrf24-bcm [Install] [Installing Headers]

The driver is installed. Continue with the Python wrapper for RF24Network. (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24Network# cd ../RF24/pyRF24/pyRF24Network/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24/pyRF24/

● 350

Raspberri Pi Full Stack UK 200609.indd 350

06-08-20 15:31

Chapter 102 • How to install the Python nRF24 modules on the Raspberry Pi

pyRF24Network# python setup.py install running install running build running build_ext building 'RF24Network' extension … (omitted output)… running install_egg_info Removing /var/www/lab_app/lib/python3.8/site-packages/ RF24Network-1.0-py3.8.egg-info Writing /var/www/lab_app/lib/python3.8/site-packages/ RF24Network-1.0-py3.8.egg-info

The Python wrapper RF24Network is installed. Let's test it. Testing The easiest way to test that the RF24 and RF24Network modules are working properly is to run the Python example programs that ship with the source code. You will find these examples in the "examples" directory of "pyRF24Network". Run the "rx" example like this: (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24/pyRF24/ pyRF24Network# cd examples/ (lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24/pyRF24/ pyRF24Network/examples# ls helloworld_rx.py

helloworld_tx.py

(lab_app) root@raspberrypi-zero:/var/www/lab_app/rf24libs/RF24/pyRF24/ pyRF24Network/examples# python helloworld_rx.py ================ SPI Configuration ================ CSN Pin CE Pin



= CE0 (PI Hardware Driven)

= Custom GPIO22

Clock Speed

= 8 Mhz

================ NRF Configuration ================ STATUS

= 0x0e RX_DR=0 TX_DS=0 MAX_RT=0 RX_P_NO=7 TX_FULL=0

RX_ADDR_P0-1 = 0xccccccccc3 0xcccccccc3c RX_ADDR_P2-5 = 0x33 0xce 0x3e 0xe3 TX_ADDR

= 0xe7e7e7e7e7

RX_PW_P0-6 = 0x20 0x20 0x20 0x20 0x20 0x20 EN_AA

= 0x3e

EN_RXADDR RF_CH

= 0x3f

= 0x5a

RF_SETUP = 0x07 CONFIG

= 0x0f

DYNPD/FEATURE Data Rate Model

= 0x3f 0x04

= 1MBPS

= nRF24L01+

● 351

Raspberri Pi Full Stack UK 200609.indd 351

06-08-20 15:31

Raspberry Pi Full Stack

CRC Length = 16 bits PA Power = PA_MAX

Notice the output contains statistics from the RF24 module as it is starting up. It contains valid values for the RX, an address (that is, non-zero values) as well as the rest of the NRF configuration. If the RF24 and RF24Network modules were not installed properly, or if the Raspberry Pi could not communicate with the nRF24 module via SPI, you would either get an error message from the example program (it would not start at all), or the configuration values would be blank. Now the RF24 modules are installed, the Python script you installed in the last chapter will be able to work. Let's go on to the next chapter to test that communications between the Raspberry Pi and Arduino node are working.

● 352

Raspberri Pi Full Stack UK 200609.indd 352

06-08-20 15:31

Chapter 103 • Test the nRF24 communications

Chapter 103 • Test the nRF24 communications Let's do a full test of the nRF24 communications between the Raspberry Pi and Arduino node. Some preparation first: • The Arduino node must be turned on, with the nRF24 sketch uploaded, and the nRF24 module and DHT22 sensors correctly connected. • The Raspberry Pi is correctly connected to the nRF24 module. • The RF24 modules are properly installed. Ensure you are at the root of the application directory, with the Python virtual environment activated, and working as the super user. Then run the rf24_recevier.py script on the command line: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python rf24_receiver.py

The output will contain the SPI and NRF configuration, and they should look similar to this example: ================ SPI Configuration ================ CSN Pin CE Pin



= CE0 (PI Hardware Driven)

= Custom GPIO7

Clock Speed

= 8 Mhz

================ NRF Configuration ================ STATUS

= 0x0e RX_DR=0 TX_DS=0 MAX_RT=0 RX_P_NO=7 TX_FULL=0

RX_ADDR_P0-1 = 0xccccccccc3 0xcccccccc3c RX_ADDR_P2-5 = 0x33 0xce 0x3e 0xe3 TX_ADDR

= 0xcccccccc3c

RX_PW_P0-6 = 0x20 0x20 0x20 0x20 0x20 0x20 EN_AA

= 0x3e

EN_RXADDR RF_CH

= 0x3f

= 0x5a

RF_SETUP = 0x07 CONFIG

= 0x0f

DYNPD/FEATURE Data Rate Model

= 0x3f 0x04

= 1MBPS

= nRF24L01+

CRC Length = 16 bits PA Power = PA_MAX

The nRF24 module is waiting for a new data packet from the Arduino node. Depending on the frequency you have set on the Arduino, give your Raspberry Pi some time. I have set the transmission frequency of my Arduino node to 8 second intervals.

● 353

Raspberri Pi Full Stack UK 200609.indd 353

06-08-20 15:31

Raspberry Pi Full Stack

When a new packet arrives, the script will print out a notification like in this example: payload length

12

39.50,19.60 -------Temperature: Humidity:

19.60

39.50

Sensor ID:

3

Header Type:

t

-------Update database... Update Google Sheet...

This is success! The script received the data packet from the Arduino node, decoded the payload, stored the sensor values in the database, and sent a copy of these values to your Google Sheet. Leave the script running for the time being. I am planning to make a modification to the script in the next and final part of this project. This will make it possible to generate alert notifications via If This Then That. Once we have implemented this new feature, we can set up the script to operate in the background as a system service using Systemd. Until then, continue to run it on the regular command line so your database collects data from the Arduino node. In the next chapter, I'll show you how to modify the application so the end user can retrieve sensor data for any available node.

● 354

Raspberri Pi Full Stack UK 200609.indd 354

06-08-20 15:31

Chapter 104 • Modify the front end of the application to show remote node data

Chapter 104 • M  odify the front end of the application to show remote node data The back-end of your application is now recording sensor data coming from your remote Arduino node. The data is being stored in the Raspberry Pi SQLite3 database, and also in the Google Sheet. However, there is no way for the user to view the data. The existing web user interface only shows the data stored from the Raspberry Pi's own sensor, and there is no way yet to choose a different sensor ID. In this chapter, I will show you how to update the web application so you can see data from any sensor. Firstly, you will make a simple change to the HTML page that implements the "lab_env_db" route. Secondly, you will make the corresponding changes in the application file "lab_app.py". Edit the "lab_env_db.html" file You can see the complete version of the new "lab_env_db.html" file in the project repository124. Below, I have copied the only change in this file. Here is the new code:

Sensor ID

I added this code in the form with id "datetime_range" on the top of the page. It created a small text box where you can type in the sensor ID you want to retrieve the data of. Notice there is also a template variable "sensorid" that can be passed from the application to the template so the field is pre-filled with the most recent value typed by the user. In Figure 104.153 you can see the result of this change.

124. You can get the full source code of version 8 of lab_env_db.html from the project repository: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v8.html

● 355

Raspberri Pi Full Stack UK 200609.indd 355

06-08-20 15:31

Raspberry Pi Full Stack

Figure 104.153: The new Sensor ID field. That's it with the HTML page . Let's switch over to the Python application script . Edit the "lab_app .py" file Let's edit the application lab_app .py file so it can read the sensor ID typed in by the user, and create the appropriate SQL query to retrieve the appropriate sensor readings . You can see the complete source code of the new version of this file in the project repository125 . Below, I copied the only change in this file . Here is the new code: Firstly, in the function "lab_env_db()": At the very start, retrieve the "sensor_id" from the "get_records()" function . temperatures, humidities, timezone, from_date_str, to_date_str, sensor_id = get_records()

At the end, pass the censored to the HTML template: return render_template("lab_env_db.html", timezone = timezone, temp = time_adjusted_temperatures, hum = time_adjusted_humidities, from_date = from_date_str, to_date = to_date_str, temp_items = len(temperatures), 125. You can see the full source code of version 11 of lab_app.py in the project repository: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_app_v11.py

● 356

Raspberri Pi Full Stack UK 200609.indd 356

06-08-20 15:31

Chapter 104 • Modify the front end of the application to show remote node data

query_string = request.query_string, hum_items = len(humidities), sensorid = sensor_id)

Secondly, in the "get_records", retrieve the sensor ID from the URL query string: sensor_id

= request.args.get('sensor_id','1')

Then insert the sensor_id in the SQL query string: temp_sql = "SELECT * FROM temperatures WHERE (rDateTime BETWEEN '%s' AND '%s') AND sensorID = %s" % (from_date_utc.format('YYY-MM-DD HH:mm'), to_ date_utc.format('YYY-MM-DD HH:mm'), sensor_id) curs.execute(temp_sql) temperatures = curs.fetchall() hum_sql = "SELECT * FROM humidities WHERE sensorID = ? AND (rDateTime BETWEEN ? AND ?)", (sensor_id, from_date_utc.format('YYYY-MM-DD HH:mm'), to_ date_utc.format('YYYY-MM-DD HH:mm')) curs.execute(hum_sql) humidities = curs.fetchall()

There are also changes elsewhere in the script. All of them involve the "sensor_id" variable. To make this chapter as readable as possible, I chose to not list the changes here. Instead, you can use your browser to access the full source code of this file and search for "sensor_id". This will reveal all relevant changes. Go ahead to implement the changes in lab_app.py. Then restart the application: (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl restart emperor. uwsgi.service

Use your browser to test the application. I pointed my browser to http://raspberrypi-zero. local/lab_env_db, selected a recent "from" and "to" dates, and typed "3" in the Sensor ID text box. You can see the result in Figure 104.154.

● 357

Raspberri Pi Full Stack UK 200609.indd 357

06-08-20 15:31

Raspberry Pi Full Stack

Figure 104.154: The sensor data from Sensor ID 3. I then typed "1" in the Sensor ID text box to make sure I can still access the values from the Raspberry Pi's sensor . You can see the result in Figure 104 .155 .

Figure 104.155: The sensor data from Sensor ID 1. It looks like everything is working as expected . You now have the infrastructure to increase the number of Arduino nodes that are part of your RF24 network .

● 358

Raspberri Pi Full Stack UK 200609.indd 358

06-08-20 15:31

Part 17: If This Then That alerts

Part 17: If This Then That alerts

● 359

Raspberri Pi Full Stack UK 200609.indd 359

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 105 • An introduction to If This Then That In the last few chapters, you reworked your application so that it has support for one or more remote Arduino nodes using the RF24 transceiver to communicate with the Raspberry Pi. What we're going to do now is use the "If This Then That" Web service to generate email alerts like the one in Figure 105.156.

Figure 105.156: An alert email, triggered by the application, delivered by IFTTT. The logic I implemented that resulted in this email notification was to test each sensor reading values against a humidity and temperature threshold, and it the values were larger than the threshold, to generate a notification. The application uses the "If This Than That" IoT platform (IFTTT.com) to generate the email based on a template that belongs to the IFTTT applet. In this part of the project, I will show you how to do this. Apart from IFTTT, we will make use of a Python module called "Requests". With the help of this module, a Python program can easily send a HTTP request to anywhere on the web. In this project, we will use the HTTP request to trigger the IFTTT applet that generates the email notification when one of the two threshold conditions are satisfied.

● 360

Raspberri Pi Full Stack UK 200609.indd 360

06-08-20 15:31

Chapter 105 • An introduction to If This Then That

In the next chapter I will show you how to create an IFTTT applet that will receive a HTTP request from our Python application program and generate a notification email. After this, I will show you how to modify your application Python script so it can generate the HTTP request that works with IFTTT.

● 361

Raspberri Pi Full Stack UK 200609.indd 361

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 106 • Create an IFTTT webhook and applet In this chapter I will show you how to create a simple app on If This Then That, that can send you an email message with custom text. The text will contain the values captured by your Raspberry Pi Full Stack application. The app must be somehow triggered, and perhaps the easiest and most "standardised" way to do this is by using a webhook. A web hook is a simple HTTP or HTTPS request that calls a special service URL on the IFTTT domain, and carries any data you want to pass through in the URL as a query string, or in the body of the request as a JSON message. If This Then That employs a simple wizard making it simple to create such applications. The name of this service encapsulates the way it operates: • If a appropriate trigger is detected, then… • Perform an action. You should take a few minutes to learn more126 about the kinds of triggers that are available, and some of the hundreds of actions your app can perform. Right now, I will show you how to create a web hook and then create a email simple app that is triggered by the web hook. Start by logging into your account on ifttt.com, or create a free account if you don't have one. With reference to Figure 106.157, on the top left corner of the IFTTT page, click inside the search text box and type "webhook" (1). This will bring back a single service result (2), titled "webhooks". Click on the Webhooks service box (3).

126. Learn more about IFTTT.com services: https://ifttt.com/discover

● 362

Raspberri Pi Full Stack UK 200609.indd 362

06-08-20 15:31

Chapter 106 • Create an IFTTT webhook and applet

Figure 106.157: Search for the Webhooks service. The Webhooks service page will appear. Click on "connect" (4) (Figure 106.158).

Figure 106.158: Create a new web hook. Your web hook is ready. Click on the Setting button (5) to retrieve the new web hook URL (Figure 106.159).

● 363

Raspberri Pi Full Stack UK 200609.indd 363

06-08-20 15:31

Raspberry Pi Full Stack

Figure 106.159: The web hook URL is available in the Setting page. The settings page contains the web hook URL and its status. You can see my web hook URL in Figure 106.160, and its status is "active".

Figure 106.160: The web hook URL and status are available in the Setting page.

● 364

Raspberri Pi Full Stack UK 200609.indd 364

06-08-20 15:31

Chapter 106 • Create an IFTTT webhook and applet

You will insert this URL in your application Python script in the next chapter, in a line of code that looks like this (in bold I have marked the components of the web request URL that derive from the web hook URL): requests.post("https://maker.ifttt.com/trigger/RPiFS_report/with/key/ Y_COim_0pwyNGOZSxSOSPmUrwoLEsXUBSrnylw", data=report)

Notice the web request URL contains additional components, such as the "trigger" and "RPiFS_report" slugs . The trigger is a keyword that involves the service named in the next slug . In this case, the service triggered is called "RPiFS_report" . You are about to create this service next . Now that you have a web hook, continue to create an app to trigger with it . Go back to the home page of your IFTTT account, and click on the "+" button here to create a new app or applet (Figure 106 .161) .

Figure 106.161: Create a new applet In the applet creation page, click on the "+" sign to create the trigger and start the wizard (Figure 106 .162) .

● 365

Raspberri Pi Full Stack UK 200609.indd 365

06-08-20 15:31

Raspberry Pi Full Stack

Figure 106.162: The app contains two components: trigger and action. In step 1 of the wizard, type "webhook" in the search box to find the "webhook" service (Figure 106.163). Click on the "Webhooks" service.

Figure 106.163: IFTTT new app wizard, step 1. Because you have already configured the webhook, step 2 of the process will simply ask you to give a name to this app. I called mine "RPiFS_report" (Figure 106.164). Whichever name you choose, you will use in the HTTP request in your Python script in the next chapter.

● 366

Raspberri Pi Full Stack UK 200609.indd 366

06-08-20 15:31

Chapter 106 • Create an IFTTT webhook and applet

Figure 106.164: IFTTT new app wizard, step 2. Click on "Create trigger" to continue. In the next page (Figure 106.162), click on the "+" button (11) next to "That" .

Figure 106.165: IFTTT new app wizard, configure the action. In the next few steps of the wizard, you will configure the action that will be executed after the webhook trigger is captured. In the search box of the 3rd step of the wizard, type "email", and click on the "Email" service box.

● 367

Raspberri Pi Full Stack UK 200609.indd 367

06-08-20 15:31

Raspberry Pi Full Stack

Figure 106.166: IFTTT new app wizard, step 3. In step 4 of the wizard, you will choose the action to be performed by the email service. The email service only has one possible action: "Send me an email". Other services have multiple possible actions. Click on "Send me an email" (13), and continue with Step 5 (Figure 106.167).

Figure 106.167: IFTTT new app wizard, step 4. In step 5, fill in the subject and body text for the email message (Figure 106.168). In the subject and body text you can use IFTTT ingredients which contain data passed to the app from the Raspberry Pi application via the HTTP request.

● 368

Raspberri Pi Full Stack UK 200609.indd 368

06-08-20 15:31

Chapter 106 • Create an IFTTT webhook and applet

Figure 106.168: IFTTT new app wizard, step 5. Here is how I filled in the Subject line and Body of my email notification. Any text contained in double curly brackets is an IFTTT ingredient. Subject: The event named "{{EventName}}" occurred on the Maker Webhooks service.

Body: Your lab environment values on {{OccurredAt}}:

* Device: {{Value1}}
* Temperature: {{Value2}} °C
* Humidity: {{Value3}} %


Kind regards,
RPi Robot

When you are ready, click on the "Create action button" to complete step 5. In Step 6, click on Finish (Figure 106.169).

● 369

Raspberri Pi Full Stack UK 200609.indd 369

06-08-20 15:31

Raspberry Pi Full Stack

Figure 106.169: IFTTT new app wizard, step 6. Your app is now ready to receive a HTTP request from your Raspberry Pi application. You can always modify its settings (such as the email's subject and body text) by going into the app's settings page. In the next chapter, I will show you the necessary changes in the application's Python script titled "rf24_receiver_v2.py".

● 370

Raspberri Pi Full Stack UK 200609.indd 370

06-08-20 15:31

Chapter 107 • Add IFTTT code in the application and testing

Chapter 107 • Add IFTTT code in the application and testing Your new IFTTT app is ready, and waiting for a HTTP request so that it can work and generate a notification email. In the back end of the application, you can set up the necessary Python code in a couple of locations. For example, you an add code in these scripts: • the env_log.py script, that runs on a cron schedule, and reads data from the Raspberry Pi's own sensor. • the lab_app.py script, which is the centrepiece of the application. In particular, you would place the IFTTT code in the "lab_temp" route, which reads data from the Raspberry Pi's own sensor. • the rf24_receiver.py script, that reads data from the remote Arduino node's sensor. In fact, you can add the IFTTT code on all of these locations if you want notifications to be fired based on data from all sensors connected to the application. To keep things simple, I will show you how to add IFTTT support to the rf24_receiver.py script so that email notifications are generated when values from the remote Arduino node sensor go above a certain threshold. If you wish to expand on this, you can use the same code elsewhere. I recommend not adding IFTT code in the main application file ("lab_app. py"). This is because the HTTP request is generated by the IFTTT code will "lock" your application and reduce its performance. The project repository127 contains the full source code of the new version of rf24_receiver.py (version 2). Below, I copied the new content only (the "…" symbol indicates no change in the source code from version 1128): import requests … def log_values(sensor_id, temp, hum): … print("Email alert if needed...") if temp > 40 or hum > 60: # convert string to floats so we can do this test email_alert(sensor_id, temp, hum) GPIO.output(pin, GPIO.LOW)

## Turn off GPIO pin (LOW)

127. ou can find the source code here: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/rf24_receiver_v2.py 128. The first version of rf24_receiver.py is available on the project repository: https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/rf24_receiver.py

● 371

Raspberri Pi Full Stack UK 200609.indd 371

06-08-20 15:31

Raspberry Pi Full Stack

def email_alert(device_id, temp, hum): report = {} report["value1"] = device_id report["value2"] = temp report["value3"] = hum requests.post("https://maker.ifttt.com/trigger/RPiFS_report/with/key/ hxHjeYu_IjKEKbGHDsIzj", data=report)

That's all . Here is a walkthrough of this code: • Start by importing the "requests" Python library . This library makes it easy to generate HTTP requests . Learn more about this library in the official documentation129 . • Inside the "log_values" functions, define the thresholds for temperature and humidity above which a notification email should be generated . In this example, I have set the temperature threshold to be 40°C, and humidity 60% . If any of these values is exceeded, the script will call the "email_alert" function (see below) . • The new "email_alert" function takes care of generating the email notification . It does this by using the Requests library to send a HTTPS request to IFTTT . Firstly, it creates a list of the sensor values and device IDs that that we want to include in the email message (the "report" variable) . Secondly, it generates a POST request, directed to the webhook URL you created in the previous chapter (under "webhook settings") . In the code above, I highlighted in bold text the segment of the URL that relates to the triggering of the RPiFS_report app that you created in the previous chapter . This app is responsible for creating and sending the email . Also, as part of the POST request, the new code will attach the "report" object that contains the sensor data and ID . Also, you must change the key (marked in italics) with your actual IFTTT key . Edit the "rf24_receiver .py" to contain the new code . When ready, test it out . Ensure the IFTTT app is running, and that your trigger thresholds are low enough to generate the HTTP request . Then, run the RF24 receiver script: (lab_app) root@raspberrypi-zero:/var/www/lab_app# python rf24_receiver.py

129. Requests: HTTP for Humans™, documentation: https://requests.readthedocs.io/en/master/

● 372

Raspberri Pi Full Stack UK 200609.indd 372

06-08-20 15:31

Chapter 107 • Add IFTTT code in the application and testing

Figure 107.170: the RF24 receiver script in operation. In Figure 107.170 you can see the RF24 receiver script in operation. The first payload from the remote Arduino node has been received, and the database and Google sheet have been updated. An email notification it sent if the thresholds were exceeded (you can edit this script so the output indicates if an email was actually sent). The script will run in the console until you stop it with Ctrl-C. It will print out a message to indicate that a threshold was exceeded, and when that happens, you will receive a notification email from IFTTT. When you are confident the script works, change the thresholds to more reasonable values to prevent your inbox being flooded with useless notifications.

● 373

Raspberri Pi Full Stack UK 200609.indd 373

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 108 • Install the node listener script as an systemd service In this chapter, I will show you how to run the rf24_receiver .py script as a background service so you don't need to keep it running in the console . To do this, you will use the systemd130 service manager . Remember you have already learned how to use systemd earlier in this project . Back then, you used it to automatically start uwsgi, and with it, your application . I will now show you how to apply the same knowledge to automatically start the rf24_receiver .py script . To do this, you need a service configuration file . Use Vim to create a new file (ensure that you are working as root) inside the application directory . Title this file "rf24_receiver .service" . You can see the contents of the file in the project repository131 . I also copy it below: [Unit] Description=RF24 Receiver Service After=syslog.target [Service] ExecStart=/var/www/lab_app/bin/python /var/www/lab_app/rf24_receiver.py WorkingDirectory=/var/www/lab_app/ StandardOutput=inherit StandardError=inherit Restart=always User=root [Install] WantedBy=multi-user.target

I highlighted the most important parts of the configuration with bold text . Ensure that the ExecStart variable is pointing to the absolute path to the Python script that implements the service . Save the buffer and exit Vim when you are finished working on the file . Now, you must install the service . Use the commands I list below to do this: (lab_app) root@raspberrypi-zero:/var/www/lab_app# cd /var/www/lab_app/ (lab_app) root@raspberrypi-zero:/var/www/lab_app# cp rf24_receiver.service / etc/systemd/system/rf24_receiver.service (lab_app) root@raspberrypi-zero:/var/www/lab_app# chmod 644 /etc/systemd/ 130. Learn more about Systemd: https://wiki.debian.org/systemd 131. The contents of rf24_receiver.service can be copied from https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/rf24_receiver.service

● 374

Raspberri Pi Full Stack UK 200609.indd 374

06-08-20 15:31

Chapter 108 • Install the node listener script as an systemd service

system/rf24_receiver.service (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl start rf24_ receiver.service (lab_app) root@raspberrypi-zero:/var/www/lab_app# systemctl enable rf24_ receiver.service

Using the "start" command, you start the background service . With the "enable" command, you set the service to autostart when your Raspberry Pi boots or restarts . You can stop the service with the "stop" command . To make sure that the RF24 Receiver Service is really running, use the "htop" utility . Type "htop" in the command line, then hit "F4" to go into search mode, and type "rf24" (the first few letters of the service file name) to narrow down the listing to just one item .

Figure 108.171: The RF24 Receiver Service is running. In Figure 108 .171, you can see a single line that contains the process details of the RF24 Receiver Service . This is proof that it works . You can exit "htop" with the F10 key .

● 375

Raspberri Pi Full Stack UK 200609.indd 375

06-08-20 15:31

Raspberry Pi Full Stack

Part 18: Wrapping up

● 376

Raspberri Pi Full Stack UK 200609.indd 376

06-08-20 15:31

Chapter 109 • Make lab_env_db page update every 10 minutes

Chapter 109 • Make lab_env_db page update every 10 minutes Before you wrap up this project, I want to suggest one last, super-quick enhancement . It involves setting the lab_env_db .html template page to automatically refresh periodically, say every 1 minute . You have already done this in the lab_temp page, by using this code:

The "content" parameter is in seconds, and instructs the browser to refresh the current page . Because the lab_env_db .html does not need to be refreshed very often, a period of 10 minutes is enough . Here is what the updated lab_env_db .html file looks like (this is version 8 of this file, and you can see the full source code in the project repository132):

Lab Conditions by RPi



I have marked the new line of code with bold characters . Insert the new line in the header of the lab_env_db .html file and load it in your browser . Ten minutes later, your browser will automatically refresh the page .

132. The full source code for the template file lab_env_db.html is at https://github.com/futureshocked/RaspberryPiFullStack_Raspbian/blob/master/lab_env_db_v8.html

● 377

Raspberri Pi Full Stack UK 200609.indd 377

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 110 • Recap and what's next Congratulations for completing the project! You're now familiar with a wide range of technologies that start with the Raspberry Pi and go far beyond it. Let's recap what you've learned: • You know how to install and configure Raspbian. • You know how to install and use Python and the Python virtual environment. • You understand the Web application stack and its components. • You understand the use of the naming systems of the Raspberry Pi GPIO so that you can interact with simple components and sensors connected to the GPI shows. • Using Python, you are familiar with a variety of fundamental open source software and services such as NGinx and Flask, uSWGI, Skeleton boilerplate CSS, Cron, JavaScript and JQuery. • You have experience with popular web services like Google Charts, Plotly and IFTTT. • And lastly, you created a full stack application on your Raspberry Pi on which you applied all these components in a single system. So what's next? Well, that's up to you. I can suggest a couple of ideas: • How about you extend your existing application? You can start by simply adding a home page. You can continue by adding more sensors. Automatic notifications for when these sensors reach a trigger value. You can also use elements from my course "Raspberry Pi: Make A Bench Automation Computer" so that you can control various appliances remotely or based on sensor data. • You can also continue to work on the user interface. You can experiment with different widgets or data visualizations. • Figure out how to publish your data to data logging services like dweet.io.

● 378

Raspberri Pi Full Stack UK 200609.indd 378

06-08-20 15:31

Chapter 110 • Recap and what's next

Or, forget about the application you just created and start from scratch. Think big, and think deep. It is up to you. It depends on what drives you and the way you learn. I hope this project has inspired you to take the next step.

● 379

Raspberri Pi Full Stack UK 200609.indd 379

06-08-20 15:31

Raspberry Pi Full Stack

Part 19: Project extension: Text messaging using Twilio

● 380

Raspberri Pi Full Stack UK 200609.indd 380

06-08-20 15:31

Chapter 111 • What is this project extension all about?

Chapter 111 • What is this project extension all about? This book started its life as a video course. Over the years, many makers added their own extensions to the Full Stack application. Some implemented home automation features. Other added sensors, or improved its user interface. In the extension of the project that is documented in this part of the book, Billy Hare shows how to use the popular Twilio service to add two-way text messaging to our Full Stack application. Once you implement this extension, you will be able to use any mobile phone to send instructions to your Raspberry Pi. You will also be able to receive text messages from your Raspberry Pi. For example, you can text your Raspberry Pi for the latest sensor value. You can also include the sensor node that you are interested in. Or, you can get your Raspberry Pi to text you alerts when temperature and humidity cross a certain threshold. Here's an example interaction between me (or, more accurately, my phone), and my Raspberry Pi, after implementing Billy's instructions (Figure 111.172):

Figure 111.172: An example conversation between the Full Stack app and my phone.

● 381

Raspberri Pi Full Stack UK 200609.indd 381

06-08-20 15:31

Raspberry Pi Full Stack

The chapters in this part of the book are written by Billy Hare, with only light editing from me. Here are Billy's contact details: Email: [email protected] LinkedIn: https://www.linkedin.com/in/billy-hare-b24b6410/ GitHub: https://github.com/bhare-code

Let's continue with the project.

● 382

Raspberri Pi Full Stack UK 200609.indd 382

06-08-20 15:31

Chapter 112 • An introduction to Twilio

Chapter 112 • An introduction to Twilio In earlier chapters you added email alerts to the application using the "If This Then That" web service. We're now going to expand the application to include the ability to support text messaging using a service provided by Twilio133 . We'll begin by setting up a Twilio account and obtain a phone number which will be used for programmatic text messaging. We'll then add Twilio support to the Raspberry Pi including the Twilio Command Line Interface (CLI) and required credentials to communicate with the Twilio server. At this point we will enhance remote access to the Raspberry Pi by setting up a publicly accessible domain name and use a trusted SSL/TLS certificate for secure HTTP-S communications instead of relying on a self-signed certificate. This is required to support incoming text messages since Twilio does not allow the use of self-signed certificates for secure access. Next, the application will be enhanced to send text alert messages in a similar way that it sends email alert messages (Figure 112.173).

Figure 112.173: An example text message sent by the Full Stack application. Finally, the application will be further enhanced to provide the ability to handle incoming text messages containing commands which will be executed on the Raspberry Pi.

133. Learn more about Twilio: https://www.twilio.com/

● 383

Raspberri Pi Full Stack UK 200609.indd 383

06-08-20 15:31

Raspberry Pi Full Stack

Figure 112.174: This example shows a variety of messages and operations you can implement via text messaging. With this capability we will be able to request the current status of any sensor via text messaging. The server will respond to our request with a text message containing the current status. Additional commands will also be implemented just to show the possibilities of using this technique.

● 384

Raspberri Pi Full Stack UK 200609.indd 384

06-08-20 15:31

Chapter 113 • Set up a Twilio account

Chapter 113 • Set up a Twilio account Navigate to the Twilio web site at https://www.twilio.com/ to create an account (Figure 113.175).

Figure 113.175: Click on "Sign up" to create a new account on Twilio. Click on "Sign up", provide your account information, accept the Twilio Terms of Service and Privacy Statement then click on "Start your free trial" (Figure 113.176).

Figure 113.176: Step 1 of the account creation process Twilio will send an email to the address provided to verify your email address. Click on the "Confirm Your Email" link provided in the email. When you click on the link to verify your email address, Twilio will ask you to verify your mobile phone number (Figure 113.177).

● 385

Raspberri Pi Full Stack UK 200609.indd 385

06-08-20 15:31

Raspberry Pi Full Stack

Figure 113.177: Step 2 of the account creation process. Add the mobile phone number that you will use to test your Twilio service with and send commands to the web server on the Raspberry Pi. Check the box at the bottom of the screen if you do not want Twilio to contact you on your mobile phone then click "Verify" (Figure 113.178).

Figure 113.178: Step 3 of the account creation process. You'll receive a verification code on your mobile phone. Enter the code in the form provided then click "Submit". Answer the questions presented by Twilio to "customize your experience". Be sure to choose Python as the language of choice and that you wish to create a project to send and receive text messages.

● 386

Raspberri Pi Full Stack UK 200609.indd 386

06-08-20 15:31

Chapter 113 • Set up a Twilio account

At this point you'll be taken to your Twilio Dashboard (Figure 113.179).

Figure 113.179: Your new Twilio Dashboard. Click on "Get a Trial Number" (Figure 113.180).

Figure 113.180: Get a phone number, step 1. Select a phone number in your region with Short Message Service (SMS) capabilities. We won't be using voice or MMS in the project. Click on "Choose this Number" to continue (Figure 113.181). Note from Peter: It seems that Twilio will only give a free trial phone number to customers inside the USA. In other countries, you will need to purchase a phone number. I am in Australia, and a mobile phone number cosst me AUD 6 per month.

● 387

Raspberri Pi Full Stack UK 200609.indd 387

06-08-20 15:31

Raspberry Pi Full Stack

Figure 113.181: Get a phone number, step 2. Make a note of the phone number provided. We'll need it later. Click on "Done" to continue. You'll return to your Twilio Dashboard. Notice there is a balance in your account. The small balance will allow you to test the Twilio service against the verified personal mobile phone number you provided (Figure 113.182).

Figure 113.182: Your new phone number appears in the Dashboard. Make a note of ACCOUNT SID and AUTH TOKEN provided on your Dashboard. The copy icon to the right of each can be used to copy the values. We'll also need Twilio API credentials for our project. Click on the Settings (i.e. gear) icon on the top right of the screen and choose "Settings" (Figure 113.183).

● 388

Raspberri Pi Full Stack UK 200609.indd 388

06-08-20 15:31

Chapter 113 • Set up a Twilio account

Figure 113.183: Note your account SID. From the list of options on the left, select "API Keys" (Figure 113.184).

Figure 113.184: Create a new API key, Step 1. Click on the "Create new API Key" button at the bottom of the screen to generate the key credentials (Figure 113.185).

● 389

Raspberri Pi Full Stack UK 200609.indd 389

06-08-20 15:31

Raspberry Pi Full Stack

Figure 113.185: Create a new API key, Step 2. Provide a FRIENDLY NAME for the key. Leave the KEY TYPE set to Standard. Click on "Create API Key" (Figure 113.186).

Figure 113.186: Create a new API key, Step 3. Copy the API Key SID and API Key Secret. Save them for later. Check the "Got it! I have saved my API Sid and Secret in a safe place to use in my application" box and then click on the "Done" button on the bottom of the screen.

● 390

Raspberri Pi Full Stack UK 200609.indd 390

06-08-20 15:31

Chapter 114 • Create a useful bash shell script

Chapter 114 • Create a useful bash shell script Before making changes on the Raspberry Pi, I like to mention a simple technique that can be used to save time. When you find yourself executing certain commands or a sequence of commands over and over, especially when the syntax is complex, it can be useful to create a bash script134. For example, during development of this project we had to reset or get the status of our systemd services multiple times. In an upcoming section we will also edit one of our systemd configurations. We can use a custom bash shell script to make things a little bit easier to remember. Do the following to create a bash shell script that can be used to manipulate any of our systemd services. The script will reside in our lab_app directory since this is the location in which we're typically in during development. $ sudo su # cd /var/www/lab_app

Create a new file called, for example, myservice. # vim myservice

Add the following bash script text to the file. #!/bin/bash SERVICE=$1 COMMAND=$2 if [ -z "$SERVICE" ] then echo echo "Service options..." echo " echo "

nginx

uwsgi

echo "

- nginx"

rf24

- emperor.uwsgi.service" - rf24_receiver.service"

else if [ "$SERVICE" = "uwsgi" ] then SERVICE="emperor.uwsgi.service" elif [ "$SERVICE" = "rf24" ] then SERVICE="rf24_receiver.service" fi fi 134. For more information on bash scripts refer to https://www.gnu.org/software/bash/manual/html_node/Shell-Scripts.html.

● 391

Raspberri Pi Full Stack UK 200609.indd 391

06-08-20 15:31

Raspberry Pi Full Stack

if [ -z "$COMMAND" ] then echo echo "Command options..." echo "

restart - restart service"

echo "

start

- start service"

echo "

stop

- stop service"

echo "

enable

- enable service at bootup"

echo "

disable - disable service at bootup"

echo "

edit

- edit service configuration"

echo "

status

- get service status"

else echo "sudo systemctl $COMMAND $SERVICE" sudo systemctl $COMMAND $SERVICE fi if [ "$COMMAND" = "start" ] then echo "reloading..." echo "sudo systemctl daemon-reload" sudo systemctl daemon-reload fi

As you can see, the syntax is not quite as readable as Python. The script expects two arguments: service name and service action. The entire service name, e.g. emperor.uwsgi.service can be provided or simply uwsgi. Likewise, either rf24_receiver.service can be used or rf24 to specify the rf24 receiver service. Before executing the script, we need to make the new file executable. The following command will add execution permission for all users. # chmod +x myscript

To display help text, simply execute the command without the required arguments. Notice the ./ before the filename which is needed to run the file as a command since the lab_app directory is not in the Linux user's execution PATH135 . # ./myscript Here's the output. Service options... nginx

- nginx

135. Linux PATH definition: http://www.linfo.org/path_env_var.html.

● 392

Raspberri Pi Full Stack UK 200609.indd 392

06-08-20 15:31

Chapter 114 • Create a useful bash shell script

uwsgi

- emperor.uwsgi.service

rf24

- rf24_receiver.service

Command options... restart - restart service start stop

- start service - stop service

enable

- enable service at bootup

disable - disable service at bootup edit

- edit service configuration

status

- get service status

When the start option is provided, the script executes an additional command: systemctl daemon-reload. This command is typically needed when starting a systemd service after making changes to files associated with it while it's stopped. Even though I'm providing this script, all the instructions that follow will include the original systemd commands. The script is provided only for convenience and to illustrate its applicability.

● 393

Raspberri Pi Full Stack UK 200609.indd 393

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 115 • Add Twilio support to Raspberry Pi Let's continue with our project by installing the Twilio Python package into the virtual environment. $ sudo su # cd /var/www/lab_app # . bin/activate (lab_app)# pip install twilio

Set up all Twilio-related environment variables that will be needed by the web application and the Twilio CLI for the pi user. We're setting up environment variables instead of using hard-coded credentials into our Python Flask application for security reasons. If, for example, you decide to submit your code to an online repository such as Github, you won't want to inadvertently include your Twilio account or API credentials. NOTE: the same technique can be applied to other credentials that have been added to the project in previous chapters. Edit the .bashrc file for the pi user. $ cd $ vim .bashrc

Add the following to the bottom of the file. The account SID and auth token are available via the Dashboard on Twilio and were copied in an earlier section. The API key was also created earlier. The MY_PHONE_NUMBER environment variable must be set to the phone number of a mobile phone you have in your possession and verified earlier through Twilio when setting up your account. Here's the code to add. # Twilio export TWILIO_ACCOUNT_SID= export TWILIO_AUTH_TOKEN= export TWILIO_API_KEY= export TWILIO_API_SECRET= export TWILIO_PHONE_NUMBER= export MY_PHONE_NUMBER= Here's an example. # Twilio

● 394

Raspberri Pi Full Stack UK 200609.indd 394

06-08-20 15:31

Chapter 115 • Add Twilio support to Raspberry Pi

export TWILIO_ACCOUNT_SID=AC11223344556677889900aabbccddeeff export TWILIO_AUTH_TOKEN=01234567890abcdef01234567890abcdef export TWILIO_API_KEY=SK01234567890abcdef1122334455667788 export TWILIO_API_SECRET=AbccdEFghijkl0123456789MNOpqrstu export TWILIO_PHONE_NUMBER=+14045551212 export MY_PHONE_NUMBER=+17705551212

Save the file and run the following command to apply the changes: $ source .bashrc

At this point, it's always a good idea to create a 2nd SSH connection to your Raspberry Pi and log in with the pi user. When you log in the newly updated .bashrc file will be executed. If there's an error in the file and it fails to execute, you won't be able to log in! If this occurs, you can correct the issue while logged in using the original connection. Once you've verified you can still log in, you can release the 2nd SSH connection. By the way, running the source command above has a similar effect. The newly updated .bashrc file is executed making all Twilio-related environment variables active. You can verify the environment variables are active by executing the following command against any of the newly configured options. $ echo $MY_PHONE_NUMBER

The phone number you configured for MY_PHONE_NUMBER will be printed to the screen. Since our full stack application runs as root, we also need to make the same change for the root user: $ sudo su # cd # vim .bashrc

Add the same environment variable code to the bottom of the .bashrc file, and then source the file and verify the changes.

● 395

Raspberri Pi Full Stack UK 200609.indd 395

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 116 • Install Twilio CLI The Twilio Command Line Interface (CLI) can be used to manage your Twilio account. Specifically, we'll use it to set up the webhook needed for receiving text messages. The Twilio CLI will also be used for initial and subsequent testing of our application outside of the Nginx web server with a secure connection provided by Ngrok. Ngrok is integrated within the Twilio CLI to provide the secure tunnel needed to receive text messages. We will now install the Twilio CLI which requires Node.js. Let's install the latest version. Check the Node.js web site at https://nodejs.org/en/ to get the latest version. As of this writing the latest version is 14.3. Adjust the commands below as necessary. Run the following commands as the pi user to install Node.js. The update command is used to update the list of available packages on the Raspberry Pi. It's always a good idea to run this command before installing a new package. $ cd $ sudo apt update $ curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash $ sudo apt install -y nodejs

Verify Node.js is installed correctly by checking the current version. $ node –v

Install the latest version of the Node Version Manager (NVM) which is needed to install the Twilio CLI according to the Twilio CLI Quickstart for Linux. Check the NVM version website at https://github.com/nvm-sh/nvm/releases to get the latest version. As of writing this, the latest version is v0.35.3. Adjust the commands below as necessary. These commands must be run as the pi user since the Twilio CLI cannot be installed as root. $ cd $ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.1/install.sh | bash

A device reboot is needed for the changes to be fully applied before moving on. $ sudo reboot now

Log in to the Raspberry Pi again as the pi user and then execute the following to use the current Long-Term Support (LTS) version of Node.js (Figure 116.187). $ nvm install --lts

● 396

Raspberri Pi Full Stack UK 200609.indd 396

06-08-20 15:31

Chapter 116 • Install Twilio CLI

Figure 116.187: Install the Long Term Support version of Node.js. Note the LTS version of nvm returned which is v12.16.3 in this example then run the following command to use this version. Adjust the version information of the command as needed. $ nvm use v12.16.3

Install the libraries needed for storing credentials required by the Twilio CLI. $ sudo apt install libsecret-1-dev $ sudo apt install gnome-keyring

Now all of the prerequisites are installed, it's time to install the Twilio CLI using the Node Package Manager (NPM) (Figure 116.188). $ npm install twilio-cli -g

Figure 116.188: Install the Twilio command line interface..

● 397

Raspberri Pi Full Stack UK 200609.indd 397

06-08-20 15:31

Raspberry Pi Full Stack

To make sure we have the latest version of the Twilio CLI, execute the following command. This particular command can be executed any time to update the CLI (Figure 116.189). $ npm install twilio-cli@latest -g

Figure 116.189: Confirm that you are using the latest version of the Twilio CLI. Even though we just installed the Twilio CLI there was one package updated. There's no need to create a Twilio profile as implied in the screenshot above. We already did this by adding Twilio credentials to the .bashrc file for the pi user. Do the following to enable Twilio CLI command auto-completion: Change to the home directory of the pi user since that's the location of the .bashrc file that will be updated. The first command after the cd command simply displays information on auto-completion and how to enable it. The last command enables auto-completing by making an update to the pi user's .bashrc file (Figure 116.190). $ cd $ twilio autocomplete bash

● 398

Raspberri Pi Full Stack UK 200609.indd 398

06-08-20 15:31

Chapter 116 • Install Twilio CLI

Figure 116.190: Enable Twilio CLI command auto-completion. $ printf "$(twilio autocomplete:script bash)" >> ~/.bashrc; source ~/.bashrc Verify that the Twilio CLI installed successfully. $ twilio

If you see the help screen for Twilio, the installation was successful.

Figure 116.191: Test the Twilio CLI is properly installed.

● 399

Raspberri Pi Full Stack UK 200609.indd 399

06-08-20 15:31

Raspberry Pi Full Stack

Chapter 117 • Create local and public DNS hostnames To use HTTP-S both locally and remotely, we need to set up two DNS hostnames: one for local network access and another for public access. Let's Encrypt will not allow a certificate to be created against an IP address. A public domain name must be used. Let's Encrypt won't allow a certificate to be assigned to the local domain name. We're creating it more for convenience. To create or modify an existing local DNS hostname, use the Raspberry Pi configuration tool (Figure 117.192). $ sudo raspi-config

Figure 117.192: Set the local hostname for your Raspberry Pi, Step 1. Select "Network Options". Use the arrow keys to navigate the menu up and down and the TAB key to choose between and . Press the ENTER key when the desired selection is highlighted (Figure 117.193).

Figure 117.193: Set the local hostname for your Raspberry Pi, Step 2.

● 400

Raspberri Pi Full Stack UK 200609.indd 400

06-08-20 15:31

Chapter 117 • Create local and public DNS hostnames

Select "N1 Hostname". A warning message will be displayed to provide information about the required syntax for hostnames (Figure 117.194).

Figure 117.194: Set the local hostname for your Raspberry Pi, Step 3. Press the ENTER key to proceed (Figure 117.195).

Figure 117.195: Set the local hostname for your Raspberry Pi, Step 4. Enter a valid hostname and then press the TAB key to highlight OK. Then press ENTER. The tool will return to the home screen. Press the TAB key until is highlighted. Then press the ENTER key. A reboot request window will appear (Figure 117.196).

● 401

Raspberri Pi Full Stack UK 200609.indd 401

06-08-20 15:31

Raspberry Pi Full Stack

Figure 117.196: Set the local hostname for your Raspberry Pi, Step 5. Select using the TAB key and then press ENTER to reboot the Raspberry Pi for the hostname modification to take effect. The local domain name assigned to your Raspberry Pi depends on the router you're using. In most cases it will be either .local (e.g. RPiFSv2.local) or .home (e.g. RPiFSv2.home). Refer to your router documentation if you're unsure. Once you know the full local domain name, use a web browser to navigate to it from a device on your local network. Now that we have a local domain name, let's move on to creating a public domain name. There are many services available to create publicly accessible DNS hostnames. Two free services are Duck DNS136 and No IP137. Either can be used to set up your own domain name. Both provide mechanisms to automatically update the mapping between your domain name and router's IP address. We'll use Duck DNS in our project. Create an account. From the DuckDNS home page for your account, make a note of the Token assigned. You'll need it later (Figure 117.197).

136. For information on Duck DNS visit https://www.duckdns.org. 137. For information on No IP visit https://www.noip.com/.

● 402

Raspberri Pi Full Stack UK 200609.indd 402

06-08-20 15:31

Chapter 117 • Create local and public DNS hostnames

Figure 117.197: Set the public domain name for your Raspberry Pi, Step 1. From the home page, create a new domain name. Choose a unique sub domain name, e.g. myAwesomeRPi which will be part of the URL that will be used to access your Raspberry Pi remotely, e.g. https://myAwesomeRPi.duckdns.org. Configure the IP address to be anything valid, but different from your actual IP address of your home router. We're doing this to later verify the IP address is updated automatically by the Raspberry Pi. You can get either your router's IP address from your router's web page or browse to https://whatismyipaddress.com/ or similar service from any device connected to your local network. Configure the Raspberry Pi to periodically update the IP address for the Duck DNS138 domain name you created. $ sudo su # cd # mkdir duckdns # cd duckdns # vi duck.sh

Add the following code into the new bash shell script file and then save the file. Change "exampledomain" to the subdomain name that you chose, e.g. "myAwesomeRPi". Change the example token to your account token copied earlier. Be sure to leave the "&ip=" portion of the command as it is. This is required to instruct the Raspberry Pi to use the detected router IP address instead of a hard-coded value. echo url="https://www.duckdns.org/update?domains=exampledomain&token=a7c4d0 ad-114e-40ef-ba1d-d217904a50f2&ip=" | curl -k -o ~/duckdns/duck.log -K -

Modify the permissions of the duck.sh bash script to allow execution by the root user.

138. Refer to the DuckDNS Install site at https://www.duckdns.org/install.jsp for additional details.

● 403

Raspberri Pi Full Stack UK 200609.indd 403

06-08-20 15:31

Raspberry Pi Full Stack

# chmod 700 duck.sh

Set up a cron job to update the IP address every five minutes. # crontab -e

Add this line to the bottom of the crontab file for the root user. */5 * * * * ~/duckdns/duck.sh >/dev/null 2>&1

Test the bash shell script. # ./duck.sh # cat duck.log

Figure 117.198: The duck.sh script automatically updates Duck DNS. If the log printed is "OK" as it is above, the command successfully executed. If "KO" is returned the script failed to execute successfully. Check your Duck DNS account to verif the IP address for your newly created domain name was updated correctly. You may need to log out then log back in to see the change.

● 404

Raspberri Pi Full Stack UK 200609.indd 404

06-08-20 15:31

Chapter 118 • Create trusted SSL/TLS certificate

Chapter 118 • Create trusted SSL/TLS certificate Let's Encrypt is a Certificate Authority (CA) that provides free certificates for SSL/TLS encryption which is what HTTP-S utilizes. Visit the Let's Encrypt website at https://letsencrypt.org/ (Figure 118.199).

Figure 118.199: You can get a free certificate from Let's Encrypt. Click on the "Get Started" button to visit https://letsencrypt.org/getting-started/ (Figure 118.200).

Figure 118.200: Accept the suggestion to use Certbot. Since we have shell access to the Raspberry Pi we can use the Certbot ACME Client to install the certificate. Click on Certbot to visit https://certbot.eff.org/ (Figure 118.201).

Figure 118.201: This wizard will help you get the appropriate code.

● 405

Raspberri Pi Full Stack UK 200609.indd 405

06-08-20 15:31

Raspberry Pi Full Stack

Scroll down to where it says "My HTTP website is running..." and select Nginx for the Software and "Debian 10 (buster)" for the System since Raspbian Buster is based on Debian 10 (buster). The page will redirect to https://certbot.eff.org/lets-encrypt/debianbuster-nginx to provide installation and set up instructions. Set up a port forwarding rule on your router to map public port 80 to internal port 80 on the Raspberry Pi. This is needed to obtain the SSL/TLS certificate. Let's Encrypt will verify your domain is accessible on this port when the certificate acquisition request is made. Prepare Nginx by removing the self-signed certificate created earlier. $ sudo rm /etc/ssl/certs/nginx-selfsigned.crt

Edit the configuration file for Nginx. $ sudo vi /var/www/lab_app/lab_app_nginx.conf

Make the following modifications: 1. A  dd your local and remote access domain names created in the last chapter to the server_name line for the server sections for port 80 and port 443. Here's an example: server_name myawesomerpi.duckdns.org localhost rpifsv2.local 192.168.0.132;

2. R  emove or comment out the line of code that reference the snippet file for the self-signed certificate: include snippets/self-signed.conf;

3. If you enabled HTTP to HTTP-S, redirect when implementing the self-signed certificate and then comment out the redirect request by changing this: return 302 https://$server_name$request_uri; # …

to this: #return 302 https://$server_name$request_uri; # …

Check the configuration for Nginx. # nginx -t

If unsuccessful, correct the issue. When the configuration is valid, restart Nginx. # systemctl restart nginx

● 406

Raspberri Pi Full Stack UK 200609.indd 406

06-08-20 15:31

Chapter 118 • Create trusted SSL/TLS certificate

Use a web browser to navigate to your domain name using HTTP without a port number included. Port 80 will be used by default. Verify you can access the web server and that it does not redirect to using HTTP-S. Install and configure Certbot on the Raspberry Pi. $ sudo su # apt update # apt install certbot python-certbot-nginx # certbot certonly --nginx

Include the certonly option since we will update the nginx configuration file manually. Answer the questions when prompted: • Provide your email address • Read and agree to the Terms of Service • Choose if you want, an email from the Electronic Frontier Foundation

Figure 118.202: Choose the public domain name you want to use with this SSL certificate. At this point, Certbot will provide a list of domain names associated with your Raspberry Pi. These correspond to the ones configured earlier in the lab_app_nginx.conf file.

● 407

Raspberri Pi Full Stack UK 200609.indd 407

06-08-20 15:31

Raspberry Pi Full Stack

Since we want to be able to securely access the Raspberry Pi remotely, select the option corresponding to the public domain name that you created earlier via your Duck DNS account. Edit the configuration file for Nginx. $ sudo vi /var/www/lab_app/lab_app_nginx.conf

Add the following lines after the server_name line for the server section using port 443: ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

Change "example.com" to your Duck DNS domain name in both lines. Test Nginx configuration again. # nginx -t

When successful restart Nginx. # systemctl restart nginx

Test secure access to your web server by opening a new browser window and navigating to https://YOUR_SUB_DOMAIN_NAME.duckdns.org:YOUR_PUBLIC_PORT, where YOUR_SUB_DOMAIN_NAME is the sub domain name configured via your Duck DNS account and YOUR_PUBLIC_PORT is the port number that you configured via your router earlier for remove access. You should no longer receive a warning about using a self-signed certificate. When using the Chrome browser, a lock icon will appear. Click on the icon. A window appears (Figure 118.203).

Figure 118.203: The connection is secure.

● 408

Raspberri Pi Full Stack UK 200609.indd 408

06-08-20 15:31

Chapter 118 • Create trusted SSL/TLS certificate

You can also click on "Certificate" to view the new certificate you created via Certbot and Let's Encrypt (Figure 118.204).

Figure 118.204: Certificate information. Now try to access your Raspberry Pi from another device on your local network by browsing to either its IP address or local domain name created in the last chapter. You'll notice a similar issue to using a self-signed certificate since the local (or private) domain name does not match the public domain name in which the certificate is associated. When using Chrome, for example, you'll see a warning such as this (Figure 118.205):

● 409

Raspberri Pi Full Stack UK 200609.indd 409

06-08-20 15:31

Raspberry Pi Full Stack

Figure 118.205: A Chrome warning. Click on the "Proceed…" link to access the web server. Chrome shows the connection is "Not secure". Click on "Not secure" for more information. A window appears (Figure 118.206).

Figure 118.206: A Chrome warning, more information. Click on "Certificate (Invalid)" to get even more information (Figure 118.207).

● 410

Raspberri Pi Full Stack UK 200609.indd 410

06-08-20 15:31

Chapter 118 • Create trusted SSL/TLS certificate

Figure 118.207: Chrome certification information. As you can see, the certificate created against your public domain name is used. This will work fine since we have fully secured the connection when accessed remotely - which is what we need to be able to have the server receive text message commands via Twilio. The server certificate will automatically be renewed. You can close access to port 80 on your router by removing the port forwarding rule added earlier. Renewal does not require HTTP access to your website by Let's Encrypt. Do the following to test automatic renewal: # certbot renew --dry-run

The command will output a success message if the dry-run is successful (Figure 118.208).

Figure 118.208: Result of the dry-run certificate renewal. During SSL/TLS cert installation Certbot installed a cron job in the /etc/cron.d/certbot file. It is not used because the Raspberry Pi is using the services provided by systemd. Instead, the Raspberry Pi is set up to use a system timer which is managed by systemd.

● 411

Raspberri Pi Full Stack UK 200609.indd 411

06-08-20 15:31

Raspberry Pi Full Stack

Use this command to list all running timers (Figure 118.209). # systemctl list-timers

Figure 118.209: Result of the dry-run certificate renewal. The timer set up by Certbot is called "certbot.timer". Run the following command for details of the timer (Figure 118.210). # systemctl show certbot.timer

Figure 118.210: The certbot.timer. The timer is configured to run Certbot twice per day. This will allow the certificate to be renewed prior to expiration. Press the spacebar to display additional information about the timer or the "q" key to quit.

● 412

Raspberri Pi Full Stack UK 200609.indd 412

06-08-20 15:31

Chapter 119 • Send text alert messages

Chapter 119 • Send text alert messages In this chapter, we'll start by testing the Twilio installation by creating a simple Python program139 to send a text message to our verified mobile phone. After this, we'll enhance our web application to send text alert messages. We'll need our Python virtual environment to test the ability to send a test message from the Raspberry Pi. $ sudo su # cd /var/www/lab_app # . bin/activate Create a file called send_sms.py that contains the following140. import os from twilio.rest import Client account_sid = os.environ["TWILIO_ACCOUNT_SID"] auth_token = os.environ["TWILIO_AUTH_TOKEN"] my_twilio_phone_number = os.environ["TWILIO_PHONE_NUMBER"] receive_phone_number = os.environ["MY_PHONE_NUMBER"] test_text = "Hello from my Raspberry Pi!" client = Client(account_sid, auth_token) client.messages.create(to = receive_phone_number, from_ = my_twilio_phone_number, body = test_text)

Next, run the Python script to send the test text message (Figure 119.211): (lab_app) # python send_sms.py

A text message will be sent to the phone number configured as MY_PHONE_NUMBER above.

139. For more information see the Twilio Quickstart Python tutorial at https://www.twilio.com/docs/sms/quickstart/python. 140. Copy this script from https://gist.github.com/futureshocked/1810d1bec426d7a41994ee093a6b36b4

● 413

Raspberri Pi Full Stack UK 200609.indd 413

06-08-20 15:31

Raspberry Pi Full Stack

Figure 119.211: A text message sent from the Raspberry Pi. Modify both env_log.py141 and rf24_receiver.py142 to include support for sending text alert messages. Add the following import lines in addition to the others: import os from twilio.rest import Client

Add this function definition143 : def text_alert(device_id, temp, hum): account_sid = os.environ["TWILIO_ACCOUNT_SID"] auth_token = os.environ["TWILIO_AUTH_TOKEN"] my_twilio_phone_number = os.environ["TWILIO_PHONE_NUMBER"] receive_phone_number = os.environ["MY_PHONE_NUMBER"] deg_sign = u"\N{DEGREE SIGN}" report = f'Device {device_id} reported a temperature of {temp}{deg_sign} C and {hum}% humidity.' client = Client(account_sid, auth_token) client.messages.create(to = receive_phone_number, from_ = my_twilio_phone_number, body = report)

Add this line after the email_alert(sensor_id, temp, hum) line: text_alert(sensor_id, temp, hum)

Modify the corresponding print message to be as follows: print("Email and text alert if needed...") 141. Copy the updated script from https://gist.github.com/futureshocked/5ecdaf202202398fccfdfbfea228494d 142. Copy the updated script from https://gist.github.com/futureshocked/040eb55e0f457205c37ca22e11f7d8d7 143. I'm using a new feature of Python introduced in version 3.6 called f-stings for string formatting. Refer to PEP-0498 at https://www.python.org/dev/peps/pep-0498/ for more information.

● 414

Raspberri Pi Full Stack UK 200609.indd 414

06-08-20 15:31

Chapter 119 • Send text alert messages

Since the systemd service needs to access some of our Twilio related environment variables, we need to update the configuration file. $ sudo systemctl stop rf24_receiver.service $ sudo vim /var/www/lab_app/rf24_receiver.service

Add the following within the [Service] section of the file144. Environment="TWILIO_ACCOUNT_SID=" Environment="TWILIO_AUTH_TOKEN=" Environment="TWILIO_PHONE_NUMBER=" Environment="MY_PHONE_NUMBER=