Ordered parts for one set out of four sets of sensor and actuator modules, microprocessors, and auxiliary parts
The first version of the project website was created. The website had info about the project's concept and brief explanations about technical requirements and specifications.
Started on developing the app. Wireframes of the app created. Basic layout with all activities and fragments described in wireframes.
We had trouble getting the Medialink USB Bluetooth Adapter to work with the BeagleBone Black. There seems to be a device compatibility issue in powering the dongle. We switched to a SoundBot Bluetooth USB adapter which seems to be more compatible with the board.
We decided to have our sub-modules collect sensor/actuator data, and then send the data to main board. Due to limitations of I2C over long lengths (> 1ft) in keeping the clock synced, we decided to have a dedicated Atmega328P on each of the custom sub-modules. Each sub-module will communicate with the main processor via UART because it is a asynchronous serial protocol, avoiding the clock skew issues. This design also allows for simple computation on the sub-module, which could be used to decrease the processing load on the main board.
After testing out the Lilypad boards, we found some issues. The Lilypad had limitations in terms of the # of pins available and the raw processing power capable (for machine learning). We decided to switch out the board for a Beagle Bone black in the mean time.
The accelerometers and gyrosensors were initially tested to check how they hooked up with microprocessors and what types of data they gave. By refering to data sheets online and checking tutorial codes that were provided, having the sensors to work was not a big issue. However, the form factors of the senosrs were designed for lilypads and clothings, so we had to order the same sensors in different forms where we could wire them easily.
App can store fake data from testing input buttons on a fragment of the application. Application can also query and display most current posture. Can search for bluetooth device and read device name but cannot connect and receive data from custom bluetooth module yet.
Bluetooth USB adapter is able to scan for devices and connect to other devices. We had trouble initiating the pairing from a different device such as a laptop, but seem to be successful with connecting from an Android phone. Since this will be our intended device, this configuration should work for our project.
Our sub-module design previously used a separate chip for the gyroscope and accelerometer. We found the Invensense MPU6050 sensor chip which conveniently combines both functions. We incorporated the chip into our PCB design, but we later realized that the MPU6050 SMD chip was tiny and extremely difficult to hand solder. Thus, we decided to use a breakout board instead, which we will solder on to the custom PCB sub-module.
After some testing, the beagle bone black was feasible for our project's scope but soon after working on it, we ran into a large issue. The BBB, due to running a full Linux kernel, had a much larger power consumption and on our current battery would last only an hour or two at most. Started looking for alternative hardware. Ordered the Auduino Yun and FRDM K22F boards.
The newest version of the project website (currently viewed version) was created. The design of the website was overhauled by using CSS frameworks, and more details and diagrams of the project were added. Also, new pages and links were provided to host future documents and project logs.
Since both gyro and accelerometer sensor values could now be dealt by one single microprocessor, the next job was to feed them to a process coded with a higher language so that the data could be processed. Python was initially selected due to the amount of libraries that could support serial communication and machine learning.
Machine learning alogrithms were implemented by using logistic regression. The main libraries used for implementing this were scikit-learn, pandas, scipy, and numpy. The software would intially collect data from the sensors for a specific label provided by the user. After data for a few different states, such as leaning or standing straight, are collected, the user could initiate the training process and have the alogrithm predict a state for newly fed data. This process would later serve as the foundation for the process where the user has a button for calibrating the jacket.
Can now display the saved postures via a pie chart by using MPAndroidChart. Still having trouble detecting and connecting to arbitrary device and accepting arbitrary data. Moving forward with detecting and connecting to specific device and specific data.
After deciding to switch from the BeagleBone Black to the FRDM-K22F, we had to change our Bluetooth module hardware. Since there is no direct USB support, we decided to switch to use a BLE breakout board for our Bluetooth connectivity. The module we are now using is the SparkFun BLE Mate 2, which allows for access to the BC118 module via UART.
We created a quick program running on the Arduino Uno that establishes a Bluetooth connection with a mobile device and successfully sends/receives data. Since the Arduino Uno only has one serial port for communicating with the laptop, we had to use software serial, which is known to have issues with Bluetooth data. This issue should not affect our project because we will be using a board that will have a dedicated serial port for communication with the Bluetooth module.
We finished our sub-module PCB design. We also decided to have another custom PCB to fit on top of our main board (FRDM-K22F) for the following reasons. Aesthetically, the additional PCB will cover the chips and parts that lie on the FRDM-K22F board, giving our product a more sleek design. Also, we can include premade connectors that will allow for easy connections to each of the sub-modules. In addition, we will include a powerjack for our power supply to the board, as the FRDM-K22F board originally does not have one.
Decided on the FRDM K22F as the main board due to:
The next step was to find out how to have the software for receiving the sensor data get compiled without using the Arduino IDE, which was restricted for this project. The solution was found by using open source makefiles that utilized avrdude/avr-gcc/avr-g++ and could easily compile codes that previously used Arduino libraries.
There were talks about having the algorithm ported to C so that it could be run on Atmel microprocessors directly. It turned out that AVR microcontrollers did not have standard C/C++ libraries supported, which made it hard to have suitable data structures for holding data. Also, the 8 bit processor seemed limited for performing floating point arithmetics. Thus, after an amount of research, it was decided best to have the maching learning code run on a more powerful processor/OS that can compile python/java along with its machine learning libraries.
Can successfully connect to our BLE module using MelodySmart libraries. There are no visible problems with receiving data from the BLE module and storing into the phone application based on our designed protocol.
After looking through several board choices (Arduino Due, Freescale Boards, Arduino Yun), we decided to settle on the Yun for the following reasons:
As soon as the board was decided, we had to start out customizing the OS running on the Yun, since it didn't have a sufficient amount of RAM on board. As we already had a 8GB microSD card with us, we used that to provide additional swap memory to the OS and also provide more storage on which we could store our code and data later on. Also, the Yun was prepared so that python and javascript could be used by installing both pip and node.
Due to the fact that the OpenWrt OS on the Arduino Yun couldn't compile several python libraries that were used for building the prototype, we had to switch to Node.js in order to accomodate exisiting machine learning algorithms. Fortunately, OpenWrt worked well with NPM, and there were already lots of machine learning libraries that were being publicly distributed through NPM.
As the AR9331 processor wasn't a particularly powerful one, it took around 80 seconds to train the algorithm with 200 data sets with a dimension of 12 parameters. However, the prediction took around 1~2ms after the training was finished, and thus we decided the training delay wouldn't negatively affect our system (since there is only one booting process).
Updated the android application for better user experience. Included more visual indications of postures with whimsical figures. Updated the main screen of the application with new logo and layout for a more professional look.
As the training phase took to long to run, it turned out that it was inefficient to have the 32U4 request a script to run on the AR9331 over again when it needed to predict a new data. In order to overcome this, we had to have one main script that runs once and continuously waits for any incoming serial communication. Since the bridge between the 32U4 and the AR9331 on the Yun was already occupied by the system by default, we had to change the settings of the OS so that it doesn't occupy the serial port whenever it boots up. This way, we could freely use the bridging port solely for our system.
We had yet finished assembling all parts into one, but we started planning out how we will prepare and train the data. Since we were planning on having only two sub-modules, there were a total of 12 parameters to be expected from the algorithm: x,y,z values of the accelerometer and x,y,z values of the gyrosensor coming from each sub-module. For each of the five posture types available, we decided to collect around 30~40 sample data so that the training phase doesn't take too much time.
The in-class system demo was performed on May 3 on the 4th floor of CIC. Since this was yet a prototype before the final demo, we had the modules all temporarily taped onto a hoodie. Overall, we were able to show a working full system that had the sub-modules, main-module, and smartphone all communicating with each other. Several feedbacks we had after the demo were the following:
The final system demo was done over at the University Center on May 8. We were able to fix the issues we had during the in-class demo by applying the following fixes: