Following up on our submission to the Netflix Challenge, the DataRush team is improving on our entry and tuning our Collaborative Filter into an application that you can use for your own "recommender" system. Netflix offered a hefty million dollar prize to the team that could improve recommending appropriate movies to viewers by 10% or more. Needless to say, we were motivated.
Using the Pervasive DataRush engine, our original final entry had an acceptable 6.5% improvement in accuracy over the course of the challenge with amazing performance. We could crank through the full Netflix data in less than 17 minutes where other teams were taking a few hours. Since the challenge, our DataRush team has been making some improvements. Our performance is still orders of magnitude faster than the other submission and we are using a commodity 16-core system (too bad they weren't looking for performance speed alone!). We have improved accuracy nearly 22% (8.3% vs. 6.5%) since the challenge with room for growth on accuracy and performance. These accuracy results would have put us in 34th place out of thousands of submissions in the Challenge.
Our soon-to-be-released updated Collaborative Filter is based off DataRush 4.2 and brings a set of improvements, including accuracy and performance. This is big news for E-tailers looking to recommend better products to their customers, like Amazon. Or companion sites, like e-harmony, to shoot a straighter arrow. The possibilities are endless...
Stay tuned because both DR4.2 and the Collaborative Filter should be released by the end of the year.