DataStax Apache Kafka™ Connector is one of the new and long awaited functionality in DataStax Enterprise 6.7. It allows seamless data integration between Apache Kafka™ DataStax enterprise using Kafka Connect.
Chris Splinter, product manager, explains us what the connector is able to do achieve and provided some details about how it can be used to help and empower developers.
Highlights!
00:15 - Cedrick: Introducing Guest + release of DSE 6.7
01:00 - Cedrick: Chris, what is Apache Kafka?
01:20 - Chris details Kafka definition and purposes :
- keep track of events
- some kind of ETL middleman to some transformation
- buffering data in front of DSW clusters
03:00 - Cedrick: Using SAMACK architecture we could already read data from Kafka so what’s new with this connector?
04:00 - Chris exposes what was and still possible before :
- write your own code
- use spark streaming
04:50 - Chris details the advantages of the connector :
- You don't have to write anything
- Copy from Kafka to DSE period, no extra runtime needed
06:00 - Chris introduces Kafka Stream and KSQL
06:49 - Cedrick: Can you tell us more about Kafka Connect?
07:00 - Chris;
- It is Include in Kafka framework
- dedicated framework
- our framework is a sink
07:30 - Cedrick: Is Kafka Connect resilient and scalable?
08:30 - Chris details
- how we developed the connector
- How you can install the connector
09:30 : Cedrick: What about the mapping? DataType, Format,
10:10 : Chris list mappings you can do and how
11:30 : Cedrick: What about performance, tell us about benchmarks?
12:00 : Chris introduces the result we got in our tests
13:20 : Cedrick: Can I access metrics when using the connector?
13:30 : Chris expose the metrics available through JMX
14:00 : Cedrick: Is there any required version for DSE or KAFKA?
14:30 : Chris tell us connector can be used with any version from 5.0 and any distributon including DDAC
15:40 : Cedrick reminds documentation on DataStax website
15:55 : Cedrick reminds Training on academy
16:03 : Cedrick reminds working samples on github and whitepaper.
16:47 : Chris ask to provide us some feedback