Catching Up With KillrVideo

What’s happening with everyone’s favorite reference application (and cat video source)

One of the challenges of working with new technology is ramping up on how it should be used, learning best practices and understanding which pitfalls you’ll want to avoid. You may have a concept of how to use a technology at a high level, but struggle with the practical details. 

If you’re trying to get up to speed on DataStax Enterprise and related technologies like Apache Cassandra, we’ve got your back. Our goal as an evangelist team (and developer relations in general) is to help our users become awesome at using our products. One of the best ways we’ve found to to do that is through reference applications like KillrVideo, an application for sharing and watching videos.

“KillrVideo is not a demo!” - Patrick McFadin

What do we mean when we say KillrVideo is not a demo? It means that KillrVideo is not a toy, or a contrived sample that breaks down when you try to copy code from it to a real application, or just eye candy with no real substance behind it.

KillrVideo is a reference application - a real, working application that actually does something. KillrVideo is an application that allows you to upload, watch, rate and comment on videos. But in addition to that, it is a great learning tool. We know that developers tend to copy and paste from sources like this, so we are careful to make sure we’re demonstrating the right way to use our products.

At the same time, we’ve tried to model more than just proper usage of DSE to incorporate best practices for distributed applications in general. To help explain, let me share a bit about the architecture and our technology choices.

A cloud-based, microservice architecture

KillrVideo has a three-tiered architecture common for cloud-scale applications, with a web application, services tier and database tier, as shown in the figure below. We’ve adopted a microservice approach with multiple stateless services.

• The web application is built using Node.js, React, and Falcor. (Although we may look at GraphQL in the future)
• The services tier provides APIs defined using Google’s GRPC, and we offer implementations of the those APIs in various languages including Java, C#, and JavaScript (node.js). We also make use of etcd for service discovery and are investigating use of a service mesh in the future.
• The database tier is provided by DataStax Enterprise. The basic data store used for KillrVideo is DSE Core, our distribution of Apache Cassandra, and we leverage capabilities built on top of that core, including DSE Search, DSE Analytics and, for the first time, DSE Graph.

You can read more about the architecture at: https://killrvideo.github.io/docs/guides/architecture/.

As a cloud-based application, we’re also trying to exemplify best practices in continuous deployment, making usage of Travis CI to automate builds, testing, packaging and deployment of code. This includes creating Docker images for the various runtime components, registering the images with Docker Hub, and deploying them to our live instance, which we’ll talk about more in a minute.

A reference application is never “done”

KillrVideo is probably a lot like most applications that you have worked on, in that it is never done. There are always new features that we’d like to add to the product, and new features of DSE for us to test out ourselves and put to work. And the cast of evangelists working on KillrVideo has changed over time, from Luke Tillman who created the initial version, to DuyHai Doan who created the first Java implementation of the services, to David Gilardi and Jeff Carpenter who have taken up the torch to carry the project forward. Here’s some of the things we’ve been up to lately...

Killrvideo.com is back!

One of the things that makes KillrVideo a great learning tool is the fact that you can download the entire application and run it on your desktop in Docker containers, or even run portions of it in your IDE so you can experiment with changing and debugging the code. 

Running your own local instance of KillrVideo is still the best way to experience the app and get the most out of it, but we also felt it was important for our reference application to have a real online presence, as it did for a time when it was deployed to Azure.

We’re excited to share that KillrVideo.com is back online, hosted in Amazon Web Services (AWS). The biggest difference about this new KillrVideo.com is that the evangelist team is not maintaining the DSE cluster behind it. Instead, KillrVideo is one of the first clients of our new DataStax Managed Cloud. We had a lot of great interactions with our managed cloud team as they have been working to turn on each new feature in the managed cloud.

Take your own guided tour

In prior versions of KillrVideo, there was a “Tour” feature that helped many to come up to speed on KillrVideo. We’re excited to announce that the tour is back! The tour walks you through each of the screens in KillrVideo, explaining how the various capabilities of DataStax Enterprise are used behind the scenes to implement the features you see on the site, even down to the detail of walking you through CQL schemas and example queries.

Explore KillrVideo data with DataStax Studio

There’s an extra feature available if you download and run KillrVideo in your local environment. We’ve configured our Docker environment to run DataStax Studio, which is a fantastic developer tool for experimenting with CQL and graph queries, analyzing data, and documenting your results. We’ve included a notebook with sample queries and results to help you get started on exploring KillrVideo data.

A graph-based recommendation engine

We’ve saved the most exciting new feature for last - our new recommendation engine. For the first time, KillrVideo is making use of DSE Graph.

The new recommendation engine uses a graph populated with data about users, ratings, and viewing statistics, in order to implement an algorithm that does something like the following:

Using the videos I really like (rating 4-5), find other users who also really like the same videos, and grab videos they really like while excluding any videos I have watched

We actually did two implementations of this algorithm. The first implementation just used the Gremlin query language to build a traversal. We then created a domain specific language (DSL) for KillrVideo data which allowed us to abstract some of the complexity and make the service code even easier. This is an area we’re super excited about and will be blogging about quite a bit in the near future.

What Comes Next?

We’ve got plenty of ideas for where to take KillrVideo next. First up is the addition of additional recommendation engine implementations and the ability to switch between them. We’re also building up additional Studio notebooks with more ways for you to interact with KillrVideo data. 

We’re always interested in your contributions to the project and are happy to accept your pull requests on GitHub. In fact, one of the reasons we’re looking to make the recommendation engine pluggable is that we received two different contributed recommendation engine implementations at a hackathon we held recently with the DataStax engineering team, one of which led to the implementation you saw above, with the other implementation coming soon.