9

I am planning to start to implement the below IoT use case.

Use case

The IoT devices will send 100k messages/minute to the gateway via repeaters and the gateway will transfer the messages to the cloud. I want to track the employees in an organization. The sensors will be fixed on their ID card. The sensor sends the location related data (approx. 15KB/message) to the gateway via repeaters. It's for the analytical purpose. After the data passed to the cloud, I'll do some analytics and store into the DB and display on a web page. Based on this analytics data, I'll show the user's current location and also the user's moving locations of a certain passed time span (last 1hr or 2hr or 1day).

I'll do some processing over the data and send it to the front end/DB.

I have gone through the IoT basics and its architecture. Then I decided to use "SMACK" stack (Spark, Mesos, Akka, Cassandra, Kafka) architecture.

I decided to use "Kafka native client" in the gateway to publishing the messages to the cloud.

Should I use MQTT protocol to transfer the message to Kafka? Or MQTT is not needed for the above use case?

If yes, what would be the benefit of using MQTT with the "SMACK" architecture?

Helmar
  • 8,450
  • 6
  • 36
  • 84
SKK
  • 261
  • 1
  • 5

2 Answers2

5

You don't have to use MQTT. Kafka clients installed on employee ID cards can send data to the kafka broker in the cloud directly. So while you are using Kafka for the gateway, you can actually use kafka for the sensors itself.

Kafka and MQTT are not interchangeable, they have different strong sides(energy consumption, bandwidth consumption,throughtput...) but I think you know it , assuming from the complexity of your stack. Kafka can handle 100k messages/minute.

If you decide to use MQTT anyway , here is a blog post on how IBM uses MQTT and Kafka in the same project. Their project is also about mobility, so I think it might actually help.

Managing IoT devices with Kafka and MQTT

atakanyenel
  • 477
  • 2
  • 5
2

There is no problem with almost any kind MQTT broker to handle this load, especially for qos=0 (probably in your case) messages. We have constant load to our broker with incoming 100.000 messages (0.5KB) per second (+SSL). The problem may appear from traffic side, not from pps.

Regarding architecture of your system my personal advice - try to make it as simple as possible. And simple mean - just a few intermediate components/services. If you can connect directly two services - do it. You will always have the possibility to make it more complex when you will start adding features.

shal
  • 806
  • 5
  • 4