Going full functional (Haskell style)

Currying

I don't know if it is possible to make all functions work in the Currying form (h[x1][x2][..]) but it is at least possible to extend Hold behavior to all arguments which natively that pattern will not have. I will copy my favorite method which I learned from this post by Grisha Kirilin:

SetAttributes[f, HoldAllComplete]

f[a_, b_, c_] := Hold[a, b, c]
f[a__] := Function[x, f[a, x], HoldAll]

Now:

f[2 + 2][8/4][7^0]

Hold[2 + 2, 8/4, 7^0]

I think conceivably most other Attribute behaviors could be implemented, but the pattern-matching changes of e.g. Orderless might be prohibitively difficult.

Formatting

Following jVincent's lead, if we would like to format the intermediate expressions cleanly we could use:

MakeBoxes[Function[x$, h_[a__, x$], HoldAll], _] := ToBoxes @ HoldForm[h[a]]

Now:

f[2 + 2][8/4]

f[2 + 2, 8/4]

This seems appropriate as f is already defined such that this form can be given as input. However, if one prefers the visual form f[2 + 2][8/4] then we can use:

MakeBoxes[Function[x$, h_[a__, x$], HoldAll], _] := 
  ToBoxes[HeadCompose @@ HoldForm /@ Unevaluated[{h, a}]]

f[2 + 2][8/4]

f[2 + 2][8/4]

(HeadCompose is a deprecated function but still quite useful.)
Note that in these rules I used x$ which is the automatic renaming of x that occurs within the Function. This code could be made more robust by using a more unique symbol name.

Low level Box forms

I cannot recall the limits of the Notation package in this regard, but you can determine how Mathematica parses an expression, and therefore what is accessible with $PreRead or CellEvaluationFunction, using a method from John Fultz (learned here):

parseString[s_String, prep : (True | False) : True] := 
  FrontEndExecute[UndocumentedTestFEParserPacket[s, prep]]

The default True should be used in our application as this is the form that will be seen by $PreRead, etc. Testing your example expressions:

parseString @ "<+1>"
parseString @ "<1+>"

{BoxData[RowBox[{"<", RowBox[{"+", "1"}], ">"}]], StandardForm}

{BoxData[RowBox[{"<", RowBox[{"1", "+"}], ">"}]], StandardForm}

One can see that in isolation these parse to distinct forms.

New operators

I described how to create a new operator in: How can one define an infix operator with an arbitrary unicode character?

And a more mild example in: Prefix operator with low precedence

I recommend that you do not attempt to redefine the $ symbol itself as your operator as this is extensively used internally in temporary symbol names (Module, Unique, etc.).

That's how I finally defined haskell operators:

rapply[x_] := x
rapply[x_, y__] := x[rapply[y]]
InfixNotation[ParsedBoxWrapper["|"], rapply]

lapply[x_] := x
lapply[x__, y_] := lapply[x][y]
InfixNotation[ParsedBoxWrapper["∘"], lapply]

InfixNotation[ParsedBoxWrapper["·"], Composition]

Now $\circ$, $\dot{}{}$ and | act exactly like haskell's space, . and $ respectfully. Also if we have only single left application then @ is still helpful and it can be hidden with escape characters :@:.

And beautiful code like $\bf{show\cdot take\ 10\cdot map\ (\lambda\ x\to x{}^{\wedge}2)\cdot range | \infty }$ is possible. There are invisible @'s between take and 10, map and ($\lambda\ x\to x{}^{\wedge}2)$. In haskell the same would look like $\bf{show . take\ 10.map(\backslash x->x{}^{\wedge}2)$[1..]}$.

With double left application, $\circ$ is necessary: $\bf{map\circ(\lambda\ x\to x+1)\circ \{1,2,3\}}$

UPDATE: I made this TextCell hack to make partial infix operators:

infix[f_String] := Block[{x,y},Head[ToExpression["x" <> f <> "y"]]]

〈TextCell[s_][x_]〉 := infix[s][#, x] &
〈x_[TextCell[s_]]〉 := infix[s][x, #] &

We again can use invisible @ for application. Aliases can be made with TextCell on the left and on the right within AngleBrackets to enter them conveniently. Now stuff like <~Mod~10>, <2^>, <^3> also works.