Is a concave border radius possible?

You can give the impression of a concave border using radial gradients on the background. For example, something like this:

#test {
    width: 200px;
    height: 200px;
    background: #888888;
    background:
      radial-gradient(circle 20px at -20% 50%,transparent,transparent 100px,#888888 100px),
      radial-gradient(circle 20px at 120% 50%,transparent,transparent 100px,#888888 100px);
    background-size:100px 200px, 100px 200px;
    background-position:0 0,100% 0;
    background-repeat:no-repeat;
}

Note that most webkit browsers still require prefixes for radial-gradients, and if you want to fully support older browsers you may need to implement the older gradient syntax too.

With clever use of the :before and :after pseudo classes, we can simulate a concave form:

    #test{
        width: 100px;
        height: 300px;
        background: green;
        position: relative;
        display: inline-block;
    }
    
    #test:before{
        background: white;
        height: 300px;
        width: 30px;
        border-radius: 0 60px 60px 0 / 0 300px 300px 0;
        display: inline-block;
        content: '';
    }
    
    #test:after{
        background: white;
        height: 300px;
        width: 30px;
        border-radius: 60px 0 0 60px / 300px 0 0  300px;
        display: inline-block;
        content: '';
        position: relative;
        left: 40px;
    }

<div id="test"></div>

The #test div is a plain rectangle. But its :before and :after elements are half-side concave filled with the background color (white in this case).

See this jsfiddle.