Is there a way to do layer 7 filtering in Linux?

You must be talking of (the former) project Application Layer Packet Classifier for Linux, which was implemented as patches, for the 2.4 and the 2.6 kernels.

The major problem with this project, is that the technology which it proposed to control, quickly outpaced the usefulness and efficacy of the implementation.

The members of the project, also had no time (and money) to further invest in outpacing some advancements of the technology, as far as I remember, and then sold the rights to the implementation, which killed for good an already problematic project.

The challenges this project/technology has faced over the years are, by no particular order:

• adapting the patches to the 3.x/4.x kernel versions;
• scarcity of processing power - in several countries, nowadays the speed of even domestic gigabit broad will demand ASICs to do efficient layer 7 traffic-shapping;
• bittorrent started using heavy obfuscation;
• HTTPS started being used heavily to encapsulate several protocols and/or to avoid detection;
• peer-to-peer protocols stopped using fixed ports, and started trying to get their way by any open/allowed port;
• the rise of ubiquitous voIP and video in real time, that makes traffic very sensitive to even small time delays:
• the widespread use of VPN connections.

Heavy R&D was then invested heavily, into professional traffic shaping products.

The state of the art ten years ago, involved already specific ASICs and (heavy use) of heuristics, for detecting encrypted/obfuscated traffic.

At the present, besides of more than a decade of experience in advanced heuristics, with the advancement of global broadband, traffic-shapping (and firewall) vendors, are also using peer-2-peer sharing in real-time, of global data, to enhance the efficacy of their solutions.

They are combining advanced heuristics, with real time profiling / sharing data from thousands of locations in the world.

It would be very difficult, to put together a open source product, that will work as efficiently as an Allot NetEnforcer.

Using open source solutions, for the purpose of infra-structure bandwidth health, it is not so usual, anymore, trying to traffic shape by the type/nature of traffic that IP address is using at the network level.

Nowadays, for generic traffic control and protecting the bandwidth capacity of the infra-structure, the usual strategy is (besides firewalling), without using advanced traffic shaping hardware, allocating a small part of the bandwidth per IP address.