![]() ![]() I have tested by changing the retention.ms parameter using kafka config tool and the change was applied on the topic immediately. Yes, it is possible to change the configuration parameter at runtime. Iii) Will MSK provide some way to update some cluster or tuning configuration? Like aws glue provides parameter change for spark executor and driver memory in their managed environment. Start the schema registry service schema-registry-start schema-registry.properties &Įxactly once semantics is apache kafka's feature and although I haven't tested it on msk, I believe it should support this feature since it's part of open-source apache kafka only. ** modify the bootstrapserver and zookeeper(connection) url/ips. sudo yum install confluent-schema-registryĬreate file name schema-registry.propertie and add the following content- listeners= Installing and setting up schema registry is similar to confluent kafka-rest proxy. You can use avro message along with 'Schema Registry' to achieve schema evolution and schema maintenance. Get a list of topics curl " curl " In order to the access from on-premise or local machine, makesure the ec2 instance on which the rest server is running has public ip or elastic ip attached. Start rest server kafka-rest-start kafka-rest.properties &Īccess the MSK via rest API with curl or rest client/browser. = ** modify the bootstrapserver and zookeeper url/ips. This framework is not full-fledged kafka client and doesn't allow all operation of kafka client, but you can do most operation on the cluster starting from fetching metadata of cluster, topic information, producing and consuming the message,etc.įirst set up the confluent repo and ec2 instance security group (Refer - Section-1: Pre Install or set up- additional kafka components) and then install/setup kafka rest proxy. To access the MSK cluster from local machine or on-premise systems, you can set up kafka Rest Proxy framework open-sourced by Confluent to acess the MSK cluster from the outside world via rest api. To access through kafka client, you need to launch ec2 instance in the same vpc of MsK and execute kafka client(producer/consumer) to acess msk cluster. Because the broker url, zookeeper connection string are private ip's of the msk cluster vpc/subnet. You cannot access MSK directly from on-premise or local machine using kafka client or kafka stream. I) How can I access AWS MSK with the kafka clients running on my on premise system? I will try to answer your questions based on my personal experience: ![]() Also if you have specific use case or requirement that is not documented, I will suggest you contact AWS support for further clarification regarding the managed part of kafka cluster (maximum number of broker allowed, reliability, cost). Ideally, it should be able to perform all/most things that open source Kafka supports. MSK is basically the vanilla apache kafka cluster customized and managed by aws (with predefined configuration settings based on cluster instance type, number of brokers,etc) tuned for the cloud environment. Vii) Also how reliable is MSK compared to other cloud-based kafka cluster from Azure/confluent and any performance benchmark compared to vanilla kafka? And what is the maximum number of brokers that can be lauched in cluster? Answers ![]() Vi) How can I perform real-time predictive analysis of data flowing through MSK? V) Can I use streaming sql with MSK through ksql? How can I set up KSQL with MSK? Iv) Is it possible to integrate MSK with other AWS service (e.g Redshift,EMR,etc)? Iii) Will MSK provide some way to update some cluster or tuning configuration? Like aws glue provides parameter change for spark executr and driver memory in their managed environment. Ii) Does MSK support schema evolution and exactly once semantics? I) How can I access AWS MSK with the Kafka clients running on my on-premise system? I tried setting up the msk cluster and was validating whether msk could fulfill all the use case/requirement of our company, but currently, it lacks documentation and example. I'm evaluating AWS Managed Service Kafka (MSK) and I know that currently, it is in preview mode, hence might not have all features or proper documentation. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |