Difference between Nifi and Mini NiFi (MiNiFi)

Apache NiFi is a robust and secure framework for routing, transforming, and delivering data across a multitude of systems. NiFi can run in parallel with other applications, but it performs best when the entire system (or multiple systems in a cluster) are dedicated to it. It often uses SAN or RAID storage at the TB level for the massive amounts of content it ingests and the provenance it generates. The UI allows multiple users to quickly modify flows simultaneously on the same machine or across a cluster. The latest release candidate of NiFi (1.1.0 RC1) includes over 170 processors for custom integration with various systems and operations, and is 762 MB when compressed for download. In other words, NiFi is a server-class application.

Apache MiNiFi was developed out of a recognized need to bring the capabilities of NiFi to the "edge" as "agents" -- accessing data from IoT and desktop-level devices, and applying primary features of NiFi at the earliest possible stage. Now data can be collected from a variety of protocols, have data provenance generated immediately for more holistic governance and transparency, have light transformations applied at source, be encrypted, be prioritized, and be redundantly routed back to the more powerful transformations done in the cloud or data center.

Now, all of these behaviors can be performed with custom scripts, but then the problem of command & control (C2) is encountered. With hundreds, thousands, or even millions of these devices existing, how can each be monitored and exfilled, and what happens when the flow needs to change? It could be to report back to a new endpoint, to update the frequency at which it is collected or transmitted, or to handle new metrics or metadata from the device. This manual process does not scale. With MiNiFi's integration with NiFi, a flow can be developed using the UI in NiFi and transparently translated to a MiNiFi flow and pushed out to classes of agents across the world.

With manual modification to remove unnecessary processors and features, NiFi can be trimmed to fit on a Raspberry Pi. But it still requires the JVM, and there are plenty of devices that won't support it. MiNiFi is offered in Java and C++, and the footprint is on a completely different scale -- 39 MB for the Java agent (tar) and 310K for the C++ agent (tar).

A great example of the power and usefulness of MiNiFi is a recent demo at the TU-Automotive Detroit exhibition, where MiNiFi was loaded onto a custom Qualcomm modem located in a "connected car". As the car drives, massive amounts of data are generated by components throughout the car and routed via the CANBUS to be processed. Some data is important to stream back to a remote processing center in real-time -- this data is transmitted via an LTE connection. LTE is widely available, but bandwidth is expensive. Meanwhile, data that was much larger but less time-relevant (system diagnostics, etc.) could be batched and compressed, and then sent in bursts over WiFi when the car was in range of a known hotspot. MiNiFi coordinated all of the flow decisions and routing via geo-enrichment and control plane feedback. Here is a short video of Joe Niemiec explaining the process and showing the flow.

You can extrapolate that demo to many other use cases. It is helpful to think of MiNiFi as a "good guest" -- a lightweight agent that runs on hardware that is probably dedicated to a different primary purpose. Whether this is IoT, a cash register/point of sale system, a car modem, physical sensors, etc., is irrelevant to MiNiFi -- its job is to process and exfil this data while not taking unnecessary resources from the primary function. Contrast this with NiFi, which again, can run simultaneously with other applications, but ideally it has dedicated resources which it can maximize for its own performance.

NiFi :It has more no. of predefined processor, has User Interface where you can monitor, configure anything at run time, you can write your own processor.

MiNiFi :It has less no. of processor(light weight) compared to NIFI. Easy to deploy. But it has no User Interface. You can integrate it with NIFI.