Whether it's staying updated on the latest social trends or making informed investment decisions, the need for real-time data has never been more crucial. In a recent fireside chat at the Gartner Apps Summit with Curtis Cook, Commonstock’s director of back-end engineering, we delve into their experience of evaluating different streaming options for real-time data-in-motion at the San Francisco-based investment social network platform. We also explored how Commonstock leveraged DataStax Astra Streaming to build a robust architecture that enables high-velocity application development, data distribution, and next-generation AI.
Commonstock enables users to share their investment track records and real-time trades linked to their brokerage accounts. The platform facilitates verified knowledge exchange between investors so they can make well-informed decisions.
Cook highlighted Commonstock’s need for a managed solution and the initial considerations of popular streaming technologies such as Kafka and Apache Pulsar. Kafka was the obvious choice in the beginning because it was a popular choice, and some managed solution was better than none. After Commonstock was introduced to Astra Streaming, a multi-cloud messaging and event streaming platform powered by Apache Pulsar, it showed promise as an alternative that addressed some of Commonstock’s concerns around Kafka’s architectural debt. The decision to explore Asta Streaming was a shrewd move, as they discovered it provided more than double the throughput and significantly lower and consistent latency in the latest benchmarks.
“With Astra Streaming, Commonstock can now quickly create Pulsar instances, manage clusters, scale across cloud regions, and manage Pulsar resources like topics, connectors, functions, and subscriptions,” Cook said.
We also discussed the challenges of transitioning from batch processing to a real-time/stream processing mentality. Cook explained that Commonstock’s architecture was built around Pulsar as a streaming framework to ensure real-time capabilities. As a finance-oriented organization, Commonstock understands the importance of instant gratification and the need for timely data processing. It adopted a primary use case of an event bus framework, where the service generating the message becomes the source of truth and other services process messages into the most useful format for them. While batch processing is still utilized for reconciliation purposes, they focus on near-real-time data syncs and processing.
Reliability became a focal point of our discussion, and Cook emphasized its significance in its streaming and distribution technology.
“We prioritize reliability and uptime to ensure uninterrupted services for their users,” he said. “Opting for a managed service like Astra Streaming allows us to offload the responsibility of managing the Pulsar architecture, ensuring minimal downtime and preserving user trust. We strive to make their streaming messages idempotent to enable at least once’ processing, further enhancing reliability.”
The author and Commonstock's Curtis Cook (right).
Commonstock also explored the incorporation of AI into application design patterns and the role Astra Streaming plays in their vision. Cook acknowledged being more of a consumer of AI models than a producer, leveraging existing pre-trained models from platforms like HuggingFace and OpenAI. Streaming acts as an event bus, enabling Commonstock’s services to run these models asynchronously, potentially on cloud servers with GPUs. This approach enables multiple models to run simultaneously on different machines without the complexities of multiprocessing or threading, ensuring a seamless and responsive user experience.
Looking ahead, streaming is expected to play a crucial role in Commonstock's AI/ML strategy. While they currently rely on pre-trained models, they recognize the possibility of training their own models in the future.
Cook emphasized the importance of a high-performing and durable streaming solution. “Streaming facilitates a many-to-one relationship, allowing AI/ML services to listen to messages from various other services and train models accordingly,” he said. “Commonstock already leverages this capability in its recommendation engine to deliver relevant content to its users.”
Commonstock’s journey sheds light on the transformative power of streaming technologies, particularly Astra Streaming, in enabling organizations to overcome challenges and unlock new possibilities. By embracing real-time processing, high-velocity application development, and seamless data distribution, organizations can pave the way for next-generation AI and contextual decision-making.
To learn more about Commonstock’s Astra Streaming deployment, check out this case study.
