In my previous post I proposed two possible directions for the final project. One involved an interactive video frame on a 7” screen, and the other one was a virtual road trip using Google Street View imagery on a smaller, 2.8” screen. I chose the second direction due to lack of access to video equipment and delays in shipping electronic components during this period.
I did a good amount of small display research on Adafruit, and ended up deciding to get the 2.8” TFT with capacitive display shield for Arduino UNO, which includes a MicroSD card slot. However, Adafruit stopped their shipping operations during the COVID-19 crisis, so I went to Tinkersphere and ordered a similar shield.
The sheild’s specifications on the Tinkersphere website mention it is compatible with the Adafruit_TFTLCD library, but I was not able to get it working (even though they use the same ILI9341 display.) I also tried the Adafruit_ILI9341 library without any luck. After doing some online research, I realized the Tinkersphere shield is different from the Adafruit one – they actually built the MCUFRIEND 2.4” TFT Display Shield, which uses different components from Adafruit’s version. Fortunately, they have an Arduino library, MCUFRIEND_kbv, got the display showing the test graphics and eventually images from my MicroSD card after some tinkering.
In terms of software and gathering all the Street View images for an entire trip, I built a script using the Google Maps APIs. At a high level, my script takes two GPS coordinates as an input (origin and destination), asks the API for the driving directions from origin to destination, parses the directions and decodes the polyline for each segment of the route, determines the viewing angle between each two consecutive coordinates on the path and then asks the Street View API for an image at each coordinate, looking in the correct direction of the road.
Here you can see results from two queries – one of them a short trip in New York, the other a longer trip in the desert – at different display frame rates (computer rendering, not Arduino.)
An interesting artifact I hadn’t thought of before is the fact that the Street View car only travels in one direction for some road segments, so some of the footage will be backwards (the rear-view of the car, played in reverse.) The road motion is the same, but you can notice it in the cars passing by.
I played the frames on Arduino as well. However, the display refresh rate is currently very slow: an image takes about 2.5 seconds to be loaded from the SD card and displayed on screen.
There’s something I really like about the very slow quality of the trip, but at the same time I would like to have more control over the frame rate, or the visual aspect of updating the screen. I’m currently using buffering, with a buffer size of ~80 pixels, which is about the maximum that will fit in Arduino’s 2kb memory. If I managed to get somewhere at 4-8 FPS that’d be great. But that means an image needs to be able to render in ~100-200ms, which is 20 times faster than my current time. Some ideas for optimizations include:
– reducing the bit depth of my BMP files from 24 to 16 or 8. This would reduce the file sizes, therefore theoretically also reducing the read time of reading each frame from the MicroSD card.
– lower image resolutions (currently 360x240), displayed on a portion of the screen. Same considerations as above, plus faster render times. This could work, given that I want to add text elements to the screen as well (compass direction, maybe city/town or coordinates.) However, anything less than 75% of the screen size would be too small.
– replace the Arduino UNO with a board that has more SRAM. If I managed to fit one or two frames inside of the RAM, rendering could be significantly faster. However, I would need hundreds of kilobytes of SRAM per frame. Teensy boards have 64k, which would still allow me to have a more sizeable buffer.
Another limitation for my initial idea is the API limit for Google Street View. Using the free Google Cloud Credits I’ve accumulated, I could make about 100,000 requests (1 image / request.) This will not be enough for the original NY – LA trip I had planned, but I can shorten my trip.