The Challenge

When Jeff Glatstein, Senior Manager of Cyberinfrastructure, was hired by WHOI to work on the OOI program, one of his tasks was making sure that the Cassandra architecture is optimized. While the development team was good at figuring out how to load data into Cassandra, Glatstein and his colleagues realized that they weren’t using the best configuration of Cassandra, which was leading to performance issues.

“We really had no eyes into Cassandra at all in terms of performance,” Glatstein says. “At the time, we had no idea our partitions were too big and we didn’t know that our configuration actually varied from node to node.”

The cluster started to lose nodes. Initially, maybe one node would go down every couple of months. Then, all of a sudden, three nodes went down in two days. Due to a bug in the version of Cassandra the WHOI team was using, it was taking them over 72 hours to bring a node back online.

Something needed to change.

The Solution

After researching their options, the WHOI team signed up for DataStax Luna, an expert support offering for Cassandra.

“We were getting to the point where we knew we wouldn’t be able to operate, and that’s when we contacted Luna support,” Glatstein says.

As a result, the Program has been able to optimize their Cassandra architecture, improving performance, and helping stabilize access to oceanographic observation data.

“It just opened up a world to us in terms of realizing we had more tables than we probably should,” Glatstein explains. “That was news to us that there was a limit.”

Prior to enlisting Luna, the WHOI team was encountering a number of issues in their Cassandra deployment.

“Luna was able to get in and work us through those issues right away, and it cleared up a lot of mystery,” Glatstein continues.

The Results

The decision to use DataStax Luna to support their Cassandra architecture proved to be a wise one. The Program has already experienced a number of benefits thanks to Luna.

Faster recovery

Instead of having to wait days to bring nodes back online, OOI is able to put nodes in as needed.

“Our Cassandra cluster hadn’t been bounced in three years—which is not good practice—and the Luna team talked us through bouncing it,” Glatstein continues. “By doing that, it alleviates us from that 72-hour waiting period.”

Improved performance

DataStax Luna has also helped improve performance across their cluster. For example, the Luna team was able to identify that there was corruption in some of the files, which impacted performance. Additionally, Luna helped expedite garbage cleanup by standardizing configuration across all nodes.

“Our garbage cleanup was taking way, way too long,” Glatstein explains. “We fixed that so that our compaction and garbage cleanup is no longer impacting our performance.”

Increased productivity

As a small team, any time spent maintaining or searching health problems is time that someone’s not spending on moving the system forward. Thanks to Luna, OOI’s developers can focus on more pressing support matters instead of tuning a database.

“What should have taken an hour from the Luna team’s point of view took five hours, and from our point of view, would have taken us days,” Glatstein explains. “So, right there was a productivity gain.”

Maximized returns from Cassandra

Thanks to DataStax Luna, the Program has the peace of mind that comes with knowing their Cassandra deployment is optimized—not just crossing their fingers and hoping that’s the case.

“I recommend anybody using Cassandra who is unsure if they are getting the most out of it to try Luna,” Glatstein concludes. “It’s been great.”