IoT the real world challenges and how to overcome with Azure

Difficulty Level:    

I’m very interested in IoT area, spending a lot of time to study, practice and join IoT projects this year. Combining information, I gathered from Golden Sachs’s report, Texas Instrument’s report, IDG NetworkWorld’s articles, Microsoft’s documentation (refer to references section for more information) and my own experience together I wrote this post with the purpose to provide you a comprehensive picture of IoT real world challenges and how to overcome with Azure.

Internet of Things and key requirements

The Internet of Things (IoT) is emerging as the third wave in the development of the Internet. The 1990s’ Internet wave connected 1 billion users while the 2000s’ mobile wave connected another 2 billion. The IoT has the potential to connect 28 billion “things” to the Internet by 2020, ranging from bracelets to cars.


A number of significant technology changes have come together to enable the rise of the IoT, these include the following.

  • Cloud provides elastic platform and compute capacity to reduce complexity of supporting IoT.
  • More products are adding intelligence with MCUs to support more sophisticated control.
  • Adding connectivity is getting easier & cheaper.
  • Low-power semiconductors allow for more battery-powered applications.
  • Wi-Fi and internet access broadly available.
  • Tablets, PCs and Smartphones broadly available – can be leverage as a gateway.
  • Connectivity brings control, sensing and ability to update system software over internet.

Today, things, people and cloud services getting connected via the Internet to enable business models.


  • Innovative ways to sense and deliver information from the physical world to the cloud.
  • Variety of wired and wireless connectivity standards are required to enable different application needs.
  • Many IoT applications need to run for years over batteries and reduce the overall energy consumption.
  • Protecting users’ privacy and manufacturers’ IP; detecting and blocking malicious activity.
  • IoT application development needs to be easy for all developers, not just to experts.
  • IoT applications require end-to-end solutions including cloud services.


Figure 1 – Internet of Things and Key Requirements.

Challenges & how to overcome with Azure

The most exciting and practical IoT applications are happening in the Industrial Internet of Things: smart agriculture, smart cities, smart factories and smart grid. That said, the IoT brings with it tremendous amount of complexity. This is the dark side of IoT, and if we don’t overcome the challenges its presents, we will be heading for trouble.

The easiest way to see these challenges in action is to explore a possible IoT deployment. In my case, I would like to share with you the GREEN CORP.’s story (the name is changed due to security restriction), GREEN CORP. is manufacture leading in climate control products for heating, ventilation, air conditioning and refrigeration. They started internal IoT program since early 2014 with Microsoft Azure as IoT platform in order to

  • Monitor and control the heating and cooling system remotely.
  • Control in-house electronic devices such as lighting system.
  • Energy optimization and control by national organization.

Some of the requirements of the deployment include.

  • Several dozen zones, placed strategically in the layout of the warehouse/ factory/ building to optimize energy efficiency.
  • Several thousand thermostats as well as ventilation and humidity sensors inside warehouse/ factory/ building.
  • Real-time notification providing the required devices and sensors.
  • The elimination of security breaches and malfunctions that could cause a temperature imbalance and ruin inventory.

With this use case in mind, let’s explore today’s top three IoT challenges, what we need to solve them, and how we solve them.

Problem #1: Too much data
The challenge

In some cases, the total amount of data being collected may be so great that moving it over the network to a central location may not be viable. Consider an individual outside temperature sensor on the warehouse. To serve its various purposes, including maintenance, it transmits temperature, humidity, hardware version, software version, battery level, motion/position changes, etc. The list can go on and on.

This information may be sent every 30 seconds – maybe even every second depending on the criticality – and there are several hundred sensors on the outside of the large warehouse. And this is only one type of perhaps dozens of types of sensors.


What we need to solve

What’s needed is an integration/ data ingestion solution with the ability to aggregate only the desired data from wherever it resides, normalize it into common data models, and make it accessible as needed for monitoring, reporting, maintenance, and other scenarios. For example, the warehouse solution should be able to pull the outside temperature and humidity readings from only the 50 outer wall sensors in Zone 3 to check the climate balance for Zone 3.


Existing solution and the debts

The existing solution is in-house solution which built on top of Azure Service Bus and Azure Table Storage. Azure Service Bus with Relay Message mode to exchange the data/ command across devices and cloud services. Azure Table Storage to store telemetry data which is directly connect to the Service Bus consumer. This is a well-design and deployable solution but cannot serve millions of request per second.

How to solve the debts

Azure IoT Hub is recommended to replace current existing solution to serve millions devices, millions transaction per second and 50MB data per second.


Figure 2 – Data Ingestion with Azure IoT Hub

IoT Hub is a fully managed service that enables reliable communications between millions of IoT devices and solution back end. IoT Hub can serve up to 300 million messages/day per unit.

Tier Sustained thought Sustained send rate
S1 Up to 1111 KB/minute per unit

(1.5 GB/day/unit)

Average of 278 messages/minute per unit

(400,000 messages/day per unit)

S2 Up to 16 MB/minute per unit

(22.8 GB/day/unit)

Average of 4167 messages/minute per unit

(6 million messages/day per unit)

S3 Up to 814 MB/minute per unit

(1144.4 GB/day/unit)

Average of 208,333 messages/minute per unit

(300 million messages/day per unit)

Problem #2: Security
The challenge

While the IoT peer-to-peer model (i.e. multiple connected devices working together) is essential for the solution and similar large-scale IoT use cases, this approach introduces an important security issue.

The overall security profile is only as strong as the weakest device that is part of it. If the security on a particular vendor’s outdoor sensors is weak, and a number of the other vendors’ sensors and devices depend on the data from those potentially compromised sensors, the possibility of a critical “indirect” impact is strong. For example, a breached sensor could deliver the wrong outdoor temperature to the system, resulting in a system decision to adjust a zone temperature in a way that ruins the food in that zone.


What we need to solve

To solve this problem, the IoT peer-to-peer model must be implemented in a way that enables the system to double-check a particular sensor’s reading by checking with other physically co-located sensors to confirm that reading. For example, if one outdoor sensor is reading particularly high while its neighboring sensors uniformly read a lower temperature, then the system should not make an immediate decision to adjust the relevant zone temperature. Instead, the system should issue an alert to validate the functionality of that sensor and to check the physical area around the sensor.

While checking with neighboring sensors is an important technique to help detect a possible breach, the security profile of the solution could be further strengthened by adding the capability to look at historical data to see if an anomalous reading has a precedent, based on, for example, weather conditions, time of year, time of day, amount of inventory, etc., that would account for the aberration.


The existing solution and the debts

Our customer already built their own in-house security token solution to secure communication channel between devices and controlling system using shared certificate. The certificate is installed into device in factory and will be used to communicate with IoT ecosystem.

Any device with certificate can be automatically registered to send data to IoT ecosystem without being actually authorized. The device itself or the certificate can be cloned to create a malicious device to attack IoT ecosystem.

And the last but not least, maintaining certificate in millions of devices require huge effort and cost.

How to solve the debts

To resolve this challenge, the following solution was applied.

  • Unique certificate is installed per device at the time of manufacturing.
  • Develop device registration process with 2 ways trusted certificate.
  • Develop authorization service to issue private key for each device.
  • Enable two gate of encryption with certificate and private key
  • Replace in-house components by Azure IoT Hub to
  • Gather and delegate request/ event to respective component
  • Enable monitoring of device connectivity operations
  • Develop certificate management API/ device management service to enhance devices management process.


Figure 3 – IoT Secured Communication Channel

Azure IoT Hub addresses the device-connectivity challenges in the following ways

  • Per-device authentication and secured connectivity. You can provision each device with its own security key to enable it to connect to IoT Hub.
  • Manage device identities via IoT Hub identity registry which support to create device identity, update device identity, retrieve device identity by ID, delete device identity, export and import all identities.
Problem #3: Not understanding the data
The challenge

Having a lot of data doesn’t mean you can understand and use it. Because of the range and diversity of IoT use cases, it is unlikely that a single vendor can create a comprehensive solution for an environment of the scale of this warehouse. Even if one were created, it would likely force a rip-and-replace approach that would be cost prohibitive.


What we need to solve

Instead, creating a fully functional, secure, and robust IoT environment requires a complete peer-to-peer solution in which devices from one vendor can translate the information from the devices of the other vendors. This, however, isn’t feasible, given the possible involvement of perhaps hundreds of vendors, including legacy devices. Without a better solution, the warehouse solution designer is facing a modern day Tower of Babel.

A more practical solution to ensuring all data can be understood and fully utilized across the deployment is to create a hub model in which one or more IoT gateways and IoT central servers are constantly receiving data from all the devices and sensors. A rules engine can analyze the incoming data, and the hub can then pass on appropriate commands to a receiving controller, such as turn the refrigeration up in Zone 2 where the sun is heating up the southwest outer wall.

Such a hub needs to be able to translate different data types or units of measure, such as Fahrenheit to Celsius. It also requires a common data model to make it possible to compare and integrate data from any vendor’s devices, thus making it possible for the system to “understand the data.”


How to solve the challenge

To leverage with Azure services, the solution to

  • Identify quickly abnormal events.
  • Store and query huge data sets.
  • Visualize both real-time and historical data.
  • Integrate with back-office system.
  • Scaling fast to ensure system performance.
  • Provide data analytics platform to analyze and find hidden information from data.


Figure 4 – Real-time Analytics & Big Data on Azure

Microsoft Azure provides many services to support data analytics, including

  • Stream processing (Complex Event Processing) with Stream Analytics.
  • Storage hierarchy with Azure Storage, Azure SQL Database, Azure SQL Data Warehouse, Azure Redis Cache, Azure DocumentDB.
  • Analytics & machine learning with Azure Machine Learning, Azure HDInsight, Azure Batch Service, Azure Analysis Service (preview).
  • Visualization with Power BI, or building application with Azure App Services.


Using Microsoft Azure is one of good option to solve not only three challenges above but also other challenges of IoT world. In this article, I already shared with you the following challenges and how to overcome with Microsoft Azure

  • Data ingestion with IoT Hub – scalable, reliability and secured.
  • Remote controlling – sending commands from cloud and control device remotely.
  • Real-time monitoring and analytics.

I’m working with some very interesting IoT projects and I will share with you more about them in other articles.


Son Nguyen

Son Nguyen

Son Nguyen is a Cloud Consultant working for FPT Software’s Cloud Innovation team. With deep knowledge in AWS and MS Azure, Son acts as a cloud consultant in various areas, ranging from assessment to architecture design, supporting customers from Japan, EU to US.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *