Shared projects - Industries, Unique & fun DIY electronics and kits

Abstract Figure 1: The SmartWristband device being worn Monitor patients sent to self-isolate with the coronavirus by collecting their temperature and heart rate through a comfortable wristband and sending this data to the hospital through an interactive dashboard and ensure that they respect the isolation by alerting appropriate authorities if the user leaves their home. I decided to create a solution that would help care for people which have contracted a weak form of the virus. This is a category of patients which is often overlooked as their symptoms are not serious to warrant very close surveillance. However, this does not mean that their condition isn’t volatile. Due to this fact, I believe that monitoring the health of these patients is a priority. As it turns out, countries have different approaches for tending to these patients: some will hospitalise them (crowding hospitals with patients that could be monitored off-campus), while others send them to self-isolate at home. But there is a problem with the second option: a patient’s condition can quickly deteriorate. As a result, all patients must be monitored. Sure, there exists heavy, bulky equipment that does this. But the problems are obvious; as the majority of the sick population will exert weak forms of the virus, these expensive, uncomfortable devices do not have the capacity to scale and be deployed to the entire population. There are also situations in which patients who exhibit symptoms may grow restless and wish to leave their home, putting themselves and others at risk. So that this does not occur, the condition and status of patients should be monitored. Figure 2: The SmartWristband device being worn SmartWristband is a wristband held device that monitors the patient’s vitals and geolocation, caring for their safety while they fight the virus within the constraints of their home. The device is powered by an Arduino MKR GSM 1400 and is equipped with both an IR temperature and heart rate sensor collecting crucial information about the patient’s health. Elevated temperature and irregular heart rate are signs of the progression of the virus. The patient will not notice these changes immediately, SmartWristband aims to do this and stream the data live and securely to the hospital monitoring the patient. The device will also track the user’s location ensuring that they are at home at all times. All of this data is centralised in a scalable database allowing for thousands of these devices to be deployed internationally. Figure 3: The dashboard detailing the heart rate, temperature, location and other parameters for a specific patient The hospital will have access to a dashboard where they will be able to see the health of all the patients being monitored at a glimpse as well as a specific report going into the detailed state of each patient. Machine learning algorithms are also implemented in the application to predict the temperature and heart rate of the patient in the near future so that the hospital knows what to expect and when to hospitalise the patient. The patient also has access to a mobile dashboard showing them their heart rate and temperature as well as when they will need to charge their wristband. Figure 4: The mobile dashboard available to the patient displaying data collected by the device The device is comfortable enough for the patient to wear day and night without noticing it. SmartWristband will also notice if the user took the device off, alerting the hospital in this case. SmartWristband is a low-cost, scalable, open source solution for caring for patients sent to self-isolate with weak forms of the virus monitoring the development of their condition and alerting medical staff through machine learning and a simple-to-use dashboard experience if a patient requires attention. This way, hospitals can send patients home and monitor them without the need to allocate a lot of human resources. SmartWristband can ensure that these patients are contained at home and are taken care of, helping the world in the fight with the virus. Project Sumup This section will look at the functionality of the application and give a high-level understanding of what it is doing. Data CollectedData is collected by the device at intervals of 5 minutes. This is done to provide a real-time view of the condition of the patient while using the lowest possible battery consumption. The device collects two types of data: vital and location. Vital data refers to the heart rate and temperature of the patient while the location data refers to their geolocation. Application Frontend Figure 8: The frontend architecture diagramThe diagram above shows the architecture of the application running on the device. These are the steps taken by the device every 5 minutes. The device collects all the data, checks if it is worn by the user and then packages them and sends them to the backend through GSM. The device also keeps track of the time, it gets the time from a designated server and syncs its onboard RTC to this timestamp when it wakes up for the first time or after it is turned off. Application Backend Figure 9: The backend architecture diagramThe data sent by the device is received in the Azure IoT Hub. A stream analytics job extracts this data from the hub at short intervals of time and pushes it into a Cosmos DB database. The patient and doctor Power Bi reports then query this data at intervals of time from the database, run machine learning algorithms built into Power Bi and then display the data. The doctor’s dashboard then collects all patients monitored and provides a brief of their health. Why is this data collected? An analysis of the patient’s temperature and heart rate readings can indicate whether their current state is degenerating. Studies show that an irregular heart rate and high temperature are signs of the presence of the virus. If these vital signs degenerate over time (i.e. the temperature rises gradually) this is a sign that the condition of the patient may be worsening. The patient may not realise this until later, but the device will pick these signs up as they appear. The geolocation data is also collected by the device, this is done to ensure that the patient is staying at home in quarantine and is abiding by the rules of the isolation. Azure checks if the user is at home and will report this on the doctor dashboard. A look at the device I created the prototype with comfort in mind. The device is easily placed on the wrist of the patient, either on the left or right hand. It is equipped with a heart rate sensor and IR temperature sensor that stick out at the bottom of the device to touch the skin of the patient. These sensors provide precise measurements without causing discomfort. Note that for further temperature precision, the sensor should be placed in a location such as the under the arm in a pad. The device will use the raw data outputted by the heart rate sensor to check if it is being worn by the patient. The sensor will output a very specific pattern of data when it is in contact with skin and hence the lack of this pattern represents the device not being worn. Battery Life Figure 5: The patient dashboard highlighting the battery level of the device and the number of hours of battery leftRigorous testing was conducted to precisely measure the battery life of the device. The device has a total battery life of 8 days with a single charge. This means that if the patient is to be surveyed for two weeks, the battery would have to be recharged once. The battery can be charged while the device is being worn. Alternatively, data could be collected at intervals of 10 minutes, essentially doubling the device’s battery life as it would be sleeping between reads, consuming very little battery. The remaining battery life as well as the number of hours of battery life left is displayed on the patient and doctor dashboards. Cost of Development The cost of development in the case of the project consists of once-off and recurring costs. The cost of development for one device would be €70 including the enclosure. Recurring costs occur in the backend. I am using Hologram as my service provider costing me €5 to send data for 2 weeks per device. The backend costs including data streaming and the database are at €30 for 2 weeks. The total cost for 100 devices is approximately €7,035. Note that these devices are reusable. Note that the backend costs are scalable and will be the same regardless of the number of devices deployed. Costs can be dramatically reduced when buying components in bulk and using custom circuits. The Doctor Dashboard It is very important for the hospital to be able to monitor the health of these patients sent at home at all times and get alerts if they are not ok. The hospital has access to a dashboard showing the vitals of all patients monitored as well as a specific report for all of them. This allows doctors to be notified if a patient needs assistance and then can look at the patient in more detail to see exactly what the problem is. The doctor will receive a notification in the form of an email and alert in the dashboard if a patient is not at home, not wearing the device or has irregular temperature and/or heart rate. Figure 6: Tiles on the doctor dashboard highlighting core data from all patients being monitoredThe doctors will also get alerts through emails when a patient’s vitals are abnormal allowing action to be taken immediately. The report seen in the dashboard is split into 3 sections: the patient (giving details about the vitals), the location (location of the patient and if at home) and the device (battery level, ID). Machine learning is also used to predict the health of the patient in the near future showing the doctor if the patient’s health is getting worse over time. Patient Dashboard I think that patients should also be able to access this data to know how they are doing and become self-aware. Both a desktop and mobile experience is available for the patient so they can get their data on the fly. Note that forecasting and geolocation data is not displayed on the patient’s dashboard because they would not need that information. Figure 7: The dashboard the patient has access to on a desktop showing the data collected from the deviceMass Production and Scalability The device is suitable for mass production and scalable because of its small price point and ease of construction. The device can be operational in any place in the world with cellular coverage protecting people world-wide. 100% Open Source When developing this project, I ensured that all assets used, libraries and others, were open source to ensure that the application could be developed and released as soon as possible without bureaucracy. This project is fully open source and available on GitHub. Data Security The application involves the collection of user location information and its processing. All communications performed between the frontend and backend of the application are encrypted to prevent man-in-the-middle type attacks. Azure IoT Hub is also considered to have state-of-the-art security. Going Further While developing this project, numerous ideas to further develop and enhance the product came to mind, this section details them: Collection of blood oxygen level can be implemented without the need to change any hardware as this value can be extracted from the existing heart rate sensor. The SpO2 levels in the blood are an indication of the damage done by the virus. The development of a 3D printed enclosure would be preferable in a future iteration. The size of the device can also be reduced dramatically making it even more comfortable to wear. Market Research Many countries have been implementing alternatives to the monitoring of patients with coronavirus while they self-isolate at home. Taiwan has implemented a revolutionary tracking system where the patients are tracked via their mobile phone to ensure that they stay at home. Other countries have also implemented ideas to monitor the health of people in self-isolations and some systems work better than others. But no product collects both the vital information about the patient’s state and his/her location in a small, comfortable package. Target Audience This device is aimed at people that are sent by the hospital at home to self-isolate because they have been tested positive for a weak form of the virus or are asymptomatic. Benefits of using SmartWristband Ensuring that patients sent to self-isolate are staying at home Monitoring the patients’ vitals without the need of them being in hospital Liberating space in hospitals for people with severe manifestations of the virus Alerting doctors to the state of all patients in self-isolation 25/7 Providing a comfortable monitoring solution for the patient Providing the hospital with a forecast of the patient’s health Informing the patient about their health through a mobile and desktop dashboard Cost effective and scalable Project Conduct This section will guide you through the construction of the project. Step 1: Getting everything together The first thing we need to do is get all the components needed together. The components and services needed are listed in the BOM above. Figure 10: An image depicting all the components required for the projectUse the tutorials linked below to set up the software tools necessary for the project. Get the Arduino IDE or use the Web Editor Create your Hologram account and connect your SIM cardCreate a Microsoft Azure accountCreate a Power Bi account Step 2: Working with the MKR GSM Before we start making the project itself, we need to configure the MKR GSM to connect to Azure IoT. For this, we need to generate and store a certificate on the device to be able to verify the connection with the backend. There is a tutorial on Hackster that covers this for the MKR NB. Please follow along with this tutorial from the start and note that you should stop after you generated the certificate. Please ensure to save the SHA1 string as we will need it later. The tutorial is here. Step 3: Connecting everything together The next thing we must do is get everything wired up. Below is an image depicting the schematics for creating this project. Note that your components probably do not look like the ones in the schematics. Ensure that you connect the temperature sensor to A4 on the device and the heart rate sensor to A5. Don’t forget to put the SIM in! Figure 11: A diagram of the schematics for the SmartWristband deviceStep 4: Onto the IoT Hub Ok, now that we have all of that done, we can move on with preparing the IoT hub. We will be working in the backend for the next few steps to get that out of the way first. Ensure that you have an Azure account before continuing. The first thing you need to do is navigate to your console (portal.azure.com) and click on the create a resource button at the top left of the screen. Now type IoT Hub and select the option from the dropdown. Click on create after the setup was completed. Note that in this example, we will be using a free tier IoT hub so that we will not be charged for its usage. If deploying this solution, consider changing the tier to S1 (or S2/S3 depending on the scale). Create a new resource group and name your IoT hub. Now click on the Size and Scale option from the steps above. Now click Review and Create and deploy the resource. Wait for the IoT Hub to create and then open it by clicking onto All Resources from the menu on the right of the screen. We now need to create a device and attach it to the IoT hub. Navigate to the IoT Devices option on the menu on the right of the screen and click on the option. Click on the New button to add a new device to the IoT hub. Input the following credentials: Figure 12: A screenshot of the device tab in IoT HubMake sure you replace the Primary Thumbprint and the Secondary Thumbprint with your SHA1 value extracted in the second step of the project conduct.Click save and we are ready to go!Step 5: Creating the Cosmos DB DatabaseNow we need to create the database where the data sent by our device will be stored. I am using Azure’s Cosmos DB as it is accustomed to storing JSON files which is what we are sending from the device.Firstly, navigate back to the Create a Resource button from the menu on the right and click it. Search for Cosmos DB and then select the option.Select the resource group created with the IoT Hub and give it a name, keep the rest of the settings the same. Do ensure that the region is the same on both the IoT hub and the database to prevent region transfer costs.And that’s it! Wait for the resource to deploy and then we are ready to move on!Step 6: Creating the Streaming Analytics JobSo now that we have our IoT hub and database set up, we need to connect the two together so that the data inputted into our IoT Hub is streamed into the database as it is received. Start off by creating a new resource by typing Stream Analytics Job. Now select the resource group used up until now and ensure that the number of streaming units is set to 1. If delivering a high quantity of devices, you may want to consider creating more streaming units to handle the greater number of devices.Wait for the resource to deploy and then select it from the All Resources button on the menu on the left.Now that we are in the resource, select the Inputs button from the menu on the left to select our input. Click on Add Stream Input and then select IoT Hub from the list of options.Enter the name of your IoT hub and then select it from the menu shown. Do not change any other settings. Click Save. Now click on the Outputs option from the menu on the left of the screen and select the Cosmos DB option from the list provided.Name the output and then select the database from the options provided. Select Create New under Database and set the database name to SmartWristband and the container name to data. Leave the document ID empty. Now click Save.The final thing we need to do is click on Query from the options. We need to supply the service with code to route all the data from the IoT hub into the database. Copy and paste the code below into the space provided. The code will map all inputted values to outputs which will then be stored in a JSON format in the Cosmos DB.SELECT time as time, id as name, geoLat as latitude, geoLng as longitude, isWorn as isWorn, heartrate as heartrate, temperature as temperature, battery as battery INTO [universum-smartwristband-db] FROM [universum-smartwristband-hub] After that is done, click on the Save Query button from the menu at the top of the screen and we are ready to go. Now all we need to do is click back on Overview from the menu on the left and then start the streaming by pressing the Start button.Step 7: Libraries RequiredAll libraries used in this project are open source. Below is the list of all libraries used and the links to their GitHub repositories. Please make sure you download and install them before moving on.MKRGSM Library - https://github.com/arduino-libraries/MKRGSM RTCZero LibraryArduino Bear SSL Library ArduinoECCX08 Library Arduino MQTT Client Library Arduino Low Power Library DFRobot Heartrate Library Step 8: Preparing the Variables Now we can focus on the frontend of the application. Open the code from the GitHub repo in the Arduino IDE or web IDE and follow along with the steps below to edit the variables in the document so that the MKR GSM will be able to connect to Azure IoT. The first thing we will need to do is navigate to the IoT hub created and copy the hostname shown. Now open the secrets.h file from the project and input the string into the SECRET_BROKER field. The next thing is getting the device name. This is simply the name given to the device in the IoT Hub. Copy the name displayed and then paste it in the SECRET_DEVICEID field in the secrets.h file. At the end, your file should look something like this. Figure 13: Inserting the credentials in the Arduino sketchThe last thing you need to do is change the DEVICE_ID. I included this feature so that all devices would have a specific ID when deployed. You can set the ID to the name of the patient or a string to be able to identify the devices and distinguish them.Finally, I want to talk about the MODE option. When I created this project, I needed to have a way to easily test and troubleshoot the application before deploying it. The mode of the application solves this. There are two modes in which the application can be set:DEVELOP_TIME is the developing mode. Here, the device prints verbose to the serial (the device only works connected to the computer with the serial monitor on). The device also wakes up and collects data every 10 seconds. RELEASE_TIME is the release mode. Here, the application works as expected, it can work on the field without the need of a connection to a computer, it will also take samples every 5 minutes as opposed to 10 seconds. We will toggle between these modes later, for now, leave it in DEVELOP_MODE. Step 9: Flashing the Developer Version Now that all the setup is done, we need to test the code out to ensure it is running in the DEVELOP_MODE. To do this, ensure that you follow all the steps above and then connect your device to your PC or mac via USB and then upload the sketch. Open the serial monitor (ctrl/cmd-shift-m) and view the verbose. It will take a while to configure the first time, ensure that the data is being sent to the IoT Hub correctly. Then visit the IoT Hub and check the graphs to see if data was received. You are now ready to deploy the application! If you need help troubleshooting, you can contact me by the email provided below. Step 10: Shipping the Release Version To change to the release version, all you need to do is change the MODE definition in the secrets.h file to RELEASE_TIME. Now upload this code to the device, ensure the LiPo battery is connected and then unplug it from the computer. You should see the onboard LED of the device flashing every 10 minutes indicating that a cycle is running. Check the IoT Hub and Cosmos DB database to ensure that the data is going through. Now you are ready to deploy the device! Ensure you send some data to the backend before continuing as we will need this when creating the dashboard. Step 11: Onto the Dashboard In this section I will explain working with the data in the Power Bi dashboards and reports, I will not go over how to design the dashboard because this is up to personal preference but will link tutorials that show you how to do this in detail. Doctor Dashboard I will start with the doctor dashboard. Make sure you have Power Bi desktop installed on your PC. Mac users can use the Power Bi online version to create this report, everything is basically the same. Launch the application and click on the Get Data button. Now navigate to the Azure option and select Cosmos DB from the options. Now go to your Cosmos DB resource and copy the URI link and paste it in the field in Power Bi. Sign in if this is required. Now we need to go to the keys window in the Cosmos DB dashboard and copy the PRIMARY KEY and paste it into the Account Key field. Now navigate through the database and ensure that the data file is checked. Click load and then we are ready to go! Now that you are in the report, find the Transform data button from the menu at the top of the screen. Click it to get a table with data. Click on the button next to the title of the table and select the fields shown above. You should see your table expand. Now we need to change the data types of the rows. Identify the Data Type button from the menu at the top of the screen and change the values accordingly. Now click on Close and Apply and you will be brought back to the dashboard. Your dashboard should look something like this. Figure 14: Screenshot illustrating the data points in the Power Bi reportThe next thing we will need to do is create measures to process data and convert it into different forms in the report. Open the Stream Analytics Script folder from the GitHub repo in Visual Studio code. I installed an extension called DAX for Power Bi to get context highlighting. There are a number of files which have to be edited specifically for the patient as they hold the patient’s information. The list of files that must be edited is below: Gender.dax Age.dax Address.daxOpen these files in VS code and edit them there. Now we will create new measures in Power Bi using these files. To do so, open the Power Bi report and click on the New Measure button from the menu at the top of the screen. Copy and paste the first file’s content into the text bar. Ensure that the Format and Data Category are set accordingly (i.e. set the category of latitude data to latitude). Now repeat this step for all other files, creating a new measure for each. Now we are moving on to the fun part – designing the dashboard. Please look at the tutorials below to get an idea of how to create infographics and style the dashboard: Basic Power BiComplete Guide to Power Bi Free Udemy Course on Power Bi Forecasting data (ML) in Power Bi (used on temperature, heart rate and battery level) Now after the dashboard is designed, we need to publish it on the web so that we can access it from anywhere, on any device. Note that the data in the report is automatically refreshed when the user opens it (there is also a refresh button). Click on the Publish button from the menu at the top of the screen and select My Workspace as the destination. Wait for the dashboard to upload. To continue working on this, open the Power Bi report in the browser. We will now be working on the dashboard that summarises all patients. To do this, click on the tile depicting the status of the patient to see a menu. From the menu, select the pin visual button and create a new dashboard. This visual will pin to the dashboard. Repeat this for the following cards: Name Status Temperature Heart Rate The doctor will be able to see a glimpse of all the patients being monitored on this dashboard. Finally, click on the temperature card in the dashboard and then on the menu button, select the Manage Alerts button to open a menu where you can set alerts for the visual. Configure this as below and repeat the step with heart rate (set the threshold to 100). And that’s it, we now need to work on the patient dashboard.Patient DashboardWe will start this dashboard by copying the doctor dashboard as we will be using all the measures and data collected there but display it in a different way. I kept all the data there but removed the forecasting for the temperature and heart rate sections (to prevent patients from worrying too much about it) and removed the location section (because the patient knows where they are). I also developed a mobile version of this dashboard. To do so, navigate to the View menu and then select Mobile Layout. From there, we only need to drag and drop the existing tiles onto the mobile phone to create this layout. After this is done, we will upload the dashboard online again as done before. This dashboard can be shared with the patient through the Power Bi app where they can get a mobile version of the application showing them all the information they need as well as an online view.Step 12: Finally, the enclosure So now that everything is done, we need to create an enclosure for the project, now ideally it would be 3D printed, but because I don’t have a 3D printer (I really need to get one), I cut pieces of plastic out and assembled them that way. I do though have a 3D design made for the enclosure of the prototype. The device is quite big in this instance and a future and industrial version will obviously be reduced in size (I just didn’t want to solder all the components onto a PCB so I could reuse them in future projects). And that’s it! The project is done!Conclusion SmartWristband is a compact wearable device which monitors a coronavirus patient’s temperature, heart rate and alerts authorities should they violate their lockdown requirements. The information can be viewed to both patients and doctors on a minimal dashboard powered by Microsoft Power Bi. I was inspired by the Taiwanese government because of their smart tracking devices that ensure that people confirmed with the virus stay at home and abide by the rules of the quarantine. I felt like this system could be improved to further enhance the safety of the patient by also monitoring their heart rate and temperature while at home. Coronavirus is a tricky problem to tackle and truly needs a lot of cooperation and devotion to overcome. I believe that this application is implementable at global level to help monitor the health of patients confirmed with weak forms of the virus allowing them to stay at home and liberate space in hospitals that can be assigned to more urgent cases. The thing I enjoyed the most out of the making of this project was the dashboard, I really enjoyed designing it (I am a visual person). I encountered some problems connecting the Arduino MKR GSM to Azure IoT. This was because I set the expiry date of the device’s certificate to something like 2 minutes (took me a while to realise).

AbstractCityGuide is a digital tour guide application which allows people to sightsee throughout a city at their own pace without the need of a large group or tour guide. The application consists of a device that locally tracks the location of the user, without sharing it externally. Whenever the user is close to a landmark, the device will play back a recording talking about the landmark. Project OverviewFigure 1: CityGuide placed on a map Typically, I when I go abroad, I get a tour wherever I go to cover the main landmarks in the area, but now, due to the recent COVID-19 pandemic I am afraid that this will expose me to a large group of people. On top of that, to be honest, I find myself to be a rather impatient person. I really don’t care about every statue in the city centre, nor am I very fond of finding out where famous people used to live. But if you go on a, typical tour you will end up seeing things you really don’t care about. Then you have the tour itself, maybe the tour guide is too quiet, or perhaps the group is moving so slow that you end up spending half the day going from one place to another. Honestly, I would prefer to get the tour over and done with and then have dinner somewhere. Figure 2: CityGuide being held by a tourist while a tram is passing by CityGuide has been my response to this problem. CityGuide is a small and pocketable device that acts as a digital, hands-free guide to the city. The device tracks the location of the tourist locally, without sharing it externally. Whenever a landmark is in viewing distance from the tourist, the device will play back an audio file talking about the landmark the tourist is looking at. CityGuide is geared towards empowering tourism agencies with digital services that can be offered to tourists interested in experiencing the city at their own pace. The agency can offer a map or app highlighting all the landmarks in the city. The tourist can then go out around town and visit only the landmarks that interest them, be it by walking or cycling. The device has been created with ease in mind. All tourism agencies have to do is record the audio files for each landmark and then add them to an SD card and then input it into the device together with the geolocation of the landmark. This allows for easy set up across tens of devices, allowing the agency to focus on gaining clients. CityGuide enables a new kind of sightseeing adapted to our busy and dynamic lives. The application empowers the tourist to create their own tour around the city at their own pace, they can stop by a café or go into a museum and then resume the tour when they come back. The tourist can focus on the landmarks they want to see instead of being dragged around the city. Most importantly, CityGuide helps tourism flourish through creating a safe way to organise sightseeing. Instead of congregating in large groups moving across the city, CityGuide allows individuals to maintain their social distance while enjoying a tour of the city. CityGuide is an application that paves the way to a sightseeing provided by agencies, where the tourist is given the flexibility to visit whatever landmarks they wish, at whatever pace they desire. Project In-Depth This section will delve into the way the project works. The Device: Sony Spresense Board Figure 3: The Sony Spresense Device This project has the Sony Spresense device as its brains, the board is highlighted in figure 3. Honestly, I think it’s one of the best development boards there is out there due to the way complex features such as audio decoding, geolocation and other features are handled by the device without much coding. All features are accessible via Arduino libraries, this allows the device to be programmable through the Arduino IDE. The Spresense board has a mother board and an expansion board which adds ports such as a headphone jack and a micro-SD card reader. The best part about it is that all the features needed for this project are built into the device. This means that there is very little wiring to be done. The device’s built in SD-card reader is used to interface with the SD-card loaded with the audio files played back and the headphone jack is used to output the audio to the user via earphones. Setup Process Figure 4: Setup process diagram Figure 4 depicts the process through which the device goes on start-up. The device can be kept offline when not in use and then turned back on when a tourist wants to use it. When the device turns on, it will first lock its GPS co-ordinates and get a precise geolocation. The next thing it will do is load the processes needed to decode MP3 files, it will then initialise the SD-card reader and wait until an SD card is present. When a card is present, the device will scan the contents of the card looking for an index file and mp3 files (go to Data Storage for more info). The device will then parse the index file into a dictionary in memory to be referenced when needed. The device will then start running the application. Project Architecture Figure 5: Project architecture diagram While the device is online, it will repeat the same algorithm constantly. Figure 5 shows this process. The device will start off by getting its current location using GPS. This data will be stored locally until refreshed again. The device will then cross reference the current location with the location of all landmarks stored in the dictionary. This is done to check if the current location is close to any of the landmarks. If this is the case, the device will check the index file for the audio file that corresponds with the landmark the tourist is at and start playing it. The device will wait until the tourist is no longer in the range of the landmark and then repeat the process. If there are multiple landmarks close to one another, the device will play the file for the first landmark in the file and then move on to the second one directly. The device will keep track of the landmarks the user went to. The device is set to not play the same media file again if the tourist already visited the specific landmark. This prevents playback errors from occurring. Geolocation and Data Privacy CityGuide is built to respect the user’s data. The device does not share the geolocation of the user with anyone, it is kept local to the device. The device stores this in memory and all geolocation references are deleted when the device is restarted. The geolocation is received from an onboard GPS module, this reduces the ability to tamper with the module as it is not directly accessible. This adds a layer of certainty and security for the user as they do not need to worry about their location data getting in the hands of wrong people. Battery Life The device is designed to survive about 12 hours of use. I chose to power the device with a power bank as it allows the device to last longer. This is also a practical choice because the power bank could be disconnected from the device when it is not in use and charged with batteries from other devices. This increases the ease of operation and makes it easier to operate with large quantities of devices. I used a 2,200mAh power bank. Data Storage and File Formatting Figure 6: File Formatting Diagram The device uses a micro-SD card for storing the sound files which are played when the tourist arrives at a landmark. The micro-SD card will have two types of files: a source file, and the music files. The source file contains a list of all landmarks and respective geolocations as well as the name of the media file that should be played at that location. The device will index this file when it boots up and save the data in the file in a runtime dictionary. The device must be restarted if there are changes to the file so the changes can be indexed. Figure 6 illustrates the format of the data file; it represents one landmark. There are three pieces of information that must be included for each landmark. The first is the name of the landmark, for example GPO.Followed by a space, the geolocation of the landmark written in terms of latitude and longitude with a comma separating them. Finally, the name of the media file that contains the audio that should be played when the tourist reaches the specified landmark should be added separated by a space. It is crucial that the space character is only used to separate the three different data points and should not be used in names or anywhere else in the list. Each landmark is separated by a carriage return (new line). Figure 7 illustrates how a datafile with four landmarks added should look. Figure 7: The source file with 4 locations added Volume Control The device is equipped with a potentiometer which allows the tourist to adjust the volume of the media playing at landmarks. Debug LEDs The Sony Spresense board is equipped with 4 debug LEDs that can be interfaced by an application. The project uses 3 of these LEDs for debug purposes. The table below illustrates what the different LEDs represent. Table 1: Debug LEDs These LEDs are helpful for debugging the project before deploying in the field. They indicate whether the device works properly and is connected to GPS, they can also help troubleshoot problems playing back media. Adding Media Files The Sony Spresense device requires the installation of a media decoder to play back files. This is described in the Project Conduct section. The device can decode mp3 and wav files and play them back to the tourist. The files added to the SD card should be in either format and the file extension should be included in the source file. The Enclosure Figure 8: Enclosure Model The device necessitates a strong enclosure to ensure its robustness during use. The enclosure can be either 3D printed, or CNC machined using acrylic or other plastics. The enclosure is relatively simple and consists of 3 parts: the base, the sides, and the top. The enclosure should be fully closed off except for a hole for the potentiometer and another for the cable connecting the device to the power bank on the outside of the enclosure. All parts can be 3D printed using open-source files available in the project’s GitHub repo and stuck together using superglue or a similar adhesive. An exploded view of the enclosure can be seen in figure 8. Deployment Overview Figure 9: Deployment overview diagram This section will walk through the procedure that is undergone when preparing the device for deployment. Figure 9 illustrates this process. The first step is to plan which landmarks should be included with the device. The agency that wishes to employ the product should compile a list of the landmarks they consider people will visit. The agency should then get the coordinates of these landmarks. This can be done through Google Maps or another mapping software. The agency should then record the audio files that will be played at each landmark. These recordings can be made by an external company if desired. All the data should then be centralised in the source file. The source file and the media files should be burned to an SD card. This card can be duplicated and then the SD cards can be inputted in the boards. The code can then be uploaded to the device. A power bank should be connected after the code uploaded. The program will then run on the device Market Research This section will talk about the project as a product in the market. The Product in the Market CityGuide was designed to be easily implementable by existing tourism agencies and provide an alternative to traditional sight-seeing. The device encourages tourism through agencies and does not aim to replace traditional ways of sightseeing but provide an alternative for interested tourists. The product can be fully managed by the tourism agency and implemented as the company sees fit. There is a lot of flexibility for the way the company can implement the project. It could for example decide to provide specific routes which tourists can follow with CityGuide providing information on each landmark along the way. The agency can also allow the tourists to create their own route around the city and visit whatever landmarks they want. Alternatively, a bike tour could be organised by the agency and tourists could cycle around landmarks in the city. Deployment by Companies I planned for the product to be deployed by tourism agencies as opposed to the consumer for example purchasing the device themselves. This provides tourism agencies with an alternative means to provide tours to tourists in a cheaper and personalised way while maintaining traditional tours. CityGuide does not aim to shut down the tour industry but provide an alternative that can be supplied to willing buyers. This allows agencies to expand their reach while not decimating traditional tours. CityGuide needs custom audio files to be added for each landmark. While such product can be sold on shelves, the cost would be too high for a single use and it would be difficult to record all landmarks if the product is sold at a national or even international level. Tourism agencies can focus devices on a specific city or part of a city and record audio files for specific landmarks, this is way more efficient. Devices can also be provided on a leasing basis to tourists who can return them. A deposit can be collected from the tourist which will be refunded on the return of the device. Target Market Not everyone would like to use this system as opposed to a traditional tour led by a tour guide. CityGuide is aimed at a young age demographic as these people are more likely to engage with the modern sight-seeing solution and are more impatient going on traditional tours. These people are also more energetic and may want to plan their own tour around the city. The product is aimed at tourists on a city break that want to get a tour of the city. The service can be provided for a cheap price which will attract tourists as expenses for operating the device are low. User PersonaJohn is a tourist visiting Dublin, Ireland on a city break. He wishes to designate half a day to sight-seeing but notices that most tours offered don’t cover the whole city but only small parts of it. John enjoys planning his own way around things and would love to choose the landmarks he wants to see himself but also get some history on the landmark narrated to him. John finds out about a local agency providing the CityGuide sight-seeing solution and reads about the offering. He finds out that he can plan his own way around landmarks and only go to the ones he is interested in. John can also go at his own pace and enter sights such as museums and then continue with the tour after he finished visiting them. John is delighted and books a device for a few hours of sightseeing. Open Source and Market Readiness The project has been developed in accordance with the open-source standard. All code for the device as well as schematics and source files are provided open source. This means that anyone can develop and deploy a replica of the system without needing to worry about patenting and other rights. The project is market ready. An enclosure can be easily designed for the device. The components can be soldered together manually and then the device can be placed in the enclosure and sealed. An external power bank is used to power the device, this ensures that the battery source can be easily recharged and replaced if need be. International Operation The device uses GPS to get its location data. This service is available worldwide and is free to use. This ensures that the project can operate anywhere in the world and cuts down operation costs drastically. Cost of Deployment Table 2: Cost Analysis Table 2 shows the cost of producing one device. The main expense is the Sony Spresense device. The enclosure would cost about $30 to 3D print. Other components needed would cost around $5 and labour needed for assembly would cost about $15. In total, the cost of producing one device would be about $110. Operation cost is estimated at around $5-$10 per device per month. This takes repairs and replacements into consideration. Making the Project This section will walk over constructing this project. Step 1: Required Apparatus Figure 10: All the components needed for the project This project does not require a lot of components as most of the needed sensors and modules are built into the Spresense board, the BOM can be seen above.Step 2: Connecting the Circuit Figure 11: Project Schematics The potentiometer must be soldered to the Sony Spresense board. Figure 11 illustrates the connections. Step 3: Installing the Arduino Spresense Libraries Moving on to the software side, we need to include the libraries necessary to operate the Spresense board using the Arduino IDE. This is done easily using the board manager. Follow the steps below to guide you through the steps for setting up the environment. Open the Arduino IDE on your computer Navigate to preferences by pressing cmd/ctrl + , Locate the field labelled additional boards manager urls Paste the following link in the dialect: https://github.com/sonydevworld/spresense-arduino-compatible/releases/download/generic/package_spresense_index.json Click Ok to close the window and go to tools -> Board… -> Board Manager in the Arduino App Search for spresense in the dialect that opens. A package should appear. Install the package Step 4: Installing the Port Utility The guide below will display the steps needed to download and install the required port utility to send commands to and from your Spresense. To get started, download the appropriate utility for your OS from the Sony Spresense website. Follow the installation steps to install the utility. Note that Arduino boards cannot operate with the utility installed, it is crucial to uninstall it to program and use Arduino or similar boards on your computer. To uninstall the driver. locate the uninstaller script in the package and run it using your preferred terminal. Step 5: Updating the Device Firmware In order to operate the project on your Spresense board; the device needs to be updated to the latest firmware. The guide below walks through the steps required to update the device's firmware. Connect the Spresense device to your computer Select the Spresense board in the Arduino IDE by going to Tools -> Board… -> Spresense Now click on Tools -> Port and select the serial port binded to the Spresense board connected Next, navigate to Tools -> Programmer and select Spresense Firmware Updater Below that in the tools menu, you should see the Burn Bootloader option. Select it. Step 6: Installing the MP3 Decoder In order to play MP3 files on the Spresense board, an MP3 decoder is required on the device. This decoder can be installed either on the device's SPI Flash or on an SD card. We will install it on the Flash so we will not need a specific SD card to run the project. The steps below list the steps to install the decoder. The first thing to do is to choose the type of files wanted to decode – choose between MP3 and WAV at your convenience and download the respective code from this link. Open the code in the Arduino IDE Make sure you have the correct port and board selected Ensure the device is connected to your computer and then upload the code to the device Open the serial monitor (cmd/ctrl + ^ + m) You will be asked where to install the decoder, select flash from the options If you are deploying on the field, you may want to consider flashing to an SD card. This would allow the SD card to be duplicated and eliminate the need to flash the decoder on all devices separately. Step 7: Setting up the SD card To use the SD card with your project, it has to be appropriately formatted in the FAT partition. You can check the card’s format and reformat it by following this link. Plug your SD card into the Spresense board. Your setup should look something like figure 12 (you may have already soldered the potentiometer). Figure 12 shows the potentiometer connected to the Sony Spresense device. Figure 12: Plan view of device Step 8: Preparing the Files Now, we must set up the files on the SD card, these will be used in order to identify the landmarks for the device and supply it with the sound files to play at the landmarks. The following files are needed on the SD card.Datalog.txt is the text file onto which the landmarks and related data is inputted by the agency mp3 files used as the files that the device will play back when it arrives at the appropriate landmark. The datalog.txt file is the heart of the device, all the data that is needed for the project's operation is included in this file. The agency can add as many landmarks as they wish to this file, all they have to do is include an entry for each landmark. The format is described in the Data Storage and File Formatting section. The steps below will walk through setting everything up. Research and make a list of the landmarks wished to be included in the file Search for these on a mapping service such as Google Maps You need to extract the latitude and longitude coordinates of the landmark; you can do this by right clicking on the map if using Google Maps. After all landmarks are identified, the next step is to record the media that will be played at each landmark. Make sure the audio is in an MP3 or WAV file depending on the decoder you installed. Now, create a new .txt file on your computer using your favourite editor Add the names of the landmarks (e.g. GPO) followed by the coordinates (separated by a comma). Make sure there is a space between the name and the coordinates, and each landmark is on its own line. Following each coordinate, add another space followed by the name of the file that will play when the device is at the respective landmark (don’t forget about the extension). Your file should look something like figure 7. Save this file and connect the SD card to your computer Burn the datalog file and the media files to the SD card. You can duplicate the SD card if you wish when deploying many devices. Finally, insert the SD card into the Spresense device Step 9: Acknowledging the Code There are 3 main sections to the code, each one of these sections is quite big. Please clone the project’s repository off GitHub to follow along. Get SD Setup Get and Process GPS Play Audio File They are all detailed below.Get DS SetupThis section of the code is composed of 2 methods. Overall, this section reads the data from datalog.txt and processes it. getData() reads the file and extracts the data from it, it detects the number of locations inputted and the details attached to all of these. drawGrid() draws a grid around all of the inputted geolocations, if the client is inside one of these grids, the device plays back the appropriate sound file. Get and Process GPS This section is composed of 3 methods. It oversees setting up and enabling the GNSS module and connecting and retrieving the data from satellites. startGPS() initialises the GNSS module, this is not done in the setup() loop as the GNSS module is disabled before playing back the Audio and has to be enabled again. processGPS() processes the data returned by the GNSS module after a read was made in the void() loop and extracts the geolocation. checkLocation() checks the geolocation received by the GNSS module against the grids generated earlier to check if the client is at a landmark. Play Audio File This section of the code controls the mp3 file playback, it sets up the file and then plays it. attachFile() sets up the playback, it starts the playback procedure, checks for the presence of the file and verifies to ensure that the mp3/wav file is formatted correctly and the frames are readable. playFile() starts off by disabling the GNSS module to start the playback, finalises setup and then actually plays the file until it is done, the user can control the volume of the playback using the potentiometer while the file is playing. Step 10: Setting Up the Variables Now that the code has been described, some variables have to be set up in order to get the best experience out of the project. The first variable to edit is proDebug, this variable controls the serial prints. If it is enabled (set to true), the device requires to be connected to the computer and the Serial Monitor enabled. It is set to false by default as it must be set to false to operate on the field. The second variable does not need to be edited, but its index must be adjusted. A structure called Landmark is used to keep all the landmark related data, a code snippet with the initialiser is included below. Landmark landmark[10]; The device needs to know how many landmarks are included in the project. The index of the initialiser method that creates instances of the Landmark class needs to be set to the number of landmarks included in the file. Note that code can be written for the device to automatically identify the number of landmarks included in the file and create that many instances if preferred. Step 11: Uploading the Code Now that the code is ready to go, it is time to upload it to the Sony Spresense device. Make sure the device is connected to your computer. Upload the code to the device. You can now disconnect it from your computer and connect it to a power bank. The code will run on the device. Nearly there! Step 12: Creating the Enclosure Figure 13: The device enclosed on the desk Libraries UsedSDHCI - Copyright (c)2018 Sony Semiconductor Solutions Corporation under the GNU Lesser General PublicLicense this library is in the public domain GNSS - Copyright (c)2018 Sony Semiconductor Solutions Corporation under the GNU Lesser General PublicLicense this library is in the public domain Audio - Copyright (c)2018 Sony Semiconductor Solutions Corporation under the GNU Lesser General PublicLicense this library is in the public domain ConclusionI built this project around the Sony Spresense device. I was quite excited about its launch a while back and thought about an interesting project to build using it. I came up with CityGuide. I was inspired by other applications that provide digital tours, but I wanted to make a physical device that could be provided by tourism agencies. One thing I loved was the documentation for the device. There are step-by-step tutorials provided by Sony online into setting everything up. There are even a lot of examples to help build the project. I was able to create the project is a relatively small amount of time. I hope you enjoyed the read!