Some shots of a native iPad app that I have been making. I finished designing most of it, and have implemented the screens below. More details coming soon.
Cricketainment iPad App Sneak Peek
Reply
Category Archives: Technical
Presenting Peekaboo
Today, we presented our final year design project, Peekaboo, to the University of Waterloo. I thought it was only fair that I post it online on the same day as well.
Peekaboo is an iOS app that lets you find out more about people around you. It accomplishes this by using facial detection and recognition. Simply snap a picture and Peekaboo tries to find out who the people in it are. You can login to the app using Facebook, and it goes through your tagged Facebook photos to create a model of your face. Don’t have or don’t want to connect with Facebook? Peekaboo can train you using your phone’s front-facing camera.
Watch the short video
I thought a video would explain it better than words can. This was taken on my iPhone 4S and recorded using AirPlay. It’s a slightly old build.
The idea
We started with the idea of wanting to know more about people in university. If you think about it, there are 10,000 like-minded students in a university campus but I only know 0.01% of them, due to a random moment that made me meet them. That 0.01% is responsible for most of the joy and friendships that I have created in the last 4 years. So what if this 0.01% was larger? We felt that there should be a better way of knowing more about people around you, so that you can meet more like-minded individuals. Is Peekaboo the right approach? I’m not totally sure, and it certainly has polarizing aspects, but it sure was fun to make.
The technology
I mentioned how I designed Peekaboo’s system architecture in an earlier post. Some things have changed since then. We found face.com provided us improved facial recognition and had a faster execution time, so we went with that. This allowed us to simplify a lot of the code on the C++ side, effectively making the system a combination of the iOS app and a Node.js server on Heroku that acted as an API that linked to the various services, Parse and Face.com.
I was responsible for coding the majority of the Node.js server and the iOS app. My goal throughout this process was to become proficient with Objective-C and the iOS SDK and I think I’ve got close.
Some pictures
Here are some screenshots and Instagrams of Peekaboo and the design symposium.
How we used free + open source tools to build a facial recognition app
Update: See my new post introducing Peekaboo.
Here’s a post I’ve been meaning to write for a while, but just never got around to it.
As a Waterloo Engineering student, we have to complete a final year design project under the guidance of a supervisor in groups of three.
What have I been making?
My project is called Peekaboo. It uses facial detection and recognition to allow a user to find out more about people around them.
I’ve been working on this for about 5 months now with my group members. We’ve made sufficient progress and are now at a stage where I can talk about our systems architecture and some of the design decisions that we made.
System Overview
Peekaboo is comprised of 3 components, shown in the diagram below.
System Overview
iOS App + Face Detection + Parse
We have a native iOS app that uses Parse for it’s mobile backend. We’ve been very happy with Parse, and it was painless to set up. Face detection is performed on the device itself through Face.com’s API. We went with Face.com over OpenCV’s Face detection due to higher accuracy and the availability of more data. Face.com is able to give us JSON back with eye positions, mouth posiitions, head tilt and yaw, gender, and more.
I strongly believe user experience should be important to any project. We use a lot of great open source iOS UI components available at Cocoa Controls.
Web Server on Heroku
We wrote a web server in Node.js deployed over Heroku to act as an API. Once again, Heroku makes this painless to set up and deploy. The web server is the mediator used by the iOS app and the desktop C++ code. The web server talks to Parse to get information and feeds it to the C++ code (and vice versa). It uses standard authentication.
The web server has a MySQL database which we usually query instead of directly querying Parse. This way, we stay below the rate limits. Parse is only queried when there is a face that needs recognition.
- We found Sequelize to be a good ORM for MySQL (available as an npm module)
- We found Request to be a useful module for simplifying HTTP Requests.
Facial Recognition Algorithms
I won’t lie, I’m not the best at C++. I find web languages and Objective-C a lot easier to code in. Nonetheless, when doing Face Recognition, we had to use C++. It’s mainstream and fast. Of course, we went with OpenCV, one of the most popular computer vision libraries out there (See my post on how to set up OpenCV on Mac).
Like I said, we don’t use OpenCV for Face detection. We use it to work with images, do some basic image processing (histogram equalization) and use OpenCV’s Eigenfaces implementation. We have also implemented another Facial Recognition algorithm known as LDA (Linear Discriminant Analysis). This was done mostly through reading research papers and contacting people smarter than us. Google Scholar is a useful resource. LDA gives us a 10-15% improved recognition rate over EigenFaces.
Communication Layer
Since I’m not a big C++ guy, I wrote a pretty thin Objective-C layer on top of the C++ code that communicates with the web server and figures out when to execute the facial recognition code. I was able to leverage a lot of my iOS knowledge for this, using standard libraries such as ASIHTTPRequest and SBJSON (ASIHTTPRequest has recently been stopped being maintained, but for our project, it does the job).System diagram of how facial recognition works
Final Thoughts
Some parts of our code are a little hacked together. This is partly due to the time crunch we are under and partly due to the fact that we are still learning. However, I am pretty proud of what we have achieved so far. We can accurately (over 70%) recognize a person who has been trained by our system (database size is 70 people currently). We’ve used $0 on this project, which is a testament to how open and amazing our software engineering industry is. We have learned a lot through out this process. I’ll put another post up with a live demo sometime soon.
If you have questions, ask them below!
Get your Facebook Cover Photo using FQL
Facebook’s Graph API doesn’t have a method to easily retrieve a cover photo. However, provided that you (or a user) has given your app access to their Photo Albums, you can get the photo pretty easily.
The best way to get it through a single request is using the following FQL query.
SELECT cover_object_id from album where aid IN (SELECT aid, name FROM album WHERE owner=me() AND name="Cover Photos")
Note that you will need an access token to use this query. It definitely works for me, but I’m still testing with permissions to see what’s needed to make it work with other users. Will post an update if anything changes.
Update: Yep, this works. Have it running on my iPhone right now. Remember to change “me()” with the Facebook ID of the user. “me()” will only work from the FQL console.