Hexagon-In or Hexagon-Out?

JavaScript (ES6) 77 83

(a,b,l,h=Math.sqrt(3)*l,x=a<0?-a:a,y=b<0?-b:b)=>y|x!=l?2*y<h&x/l+y/h<1?1:-1:0

Test

f=(a,b,l,h=Math.sqrt(3)*l,x=a<0?-a:a,y=b<0?-b:b)=>y|x!=l?2*y<h&x/l+y/h<1?1:-1:0

// TEST

function go() {
  C.width=400;C.height=300;
  var l=+I.value, x,y, cols={0:'#ff0',1:'#0f0','-1':'#888'},
  ctx = C.getContext("2d")
  ctx.translate(200,150)
  ctx.strokeStyle='#000'
  ctx.lineWidth=1;
  ctx.beginPath();
  ctx.moveTo(0,-150);ctx.lineTo(0,150);ctx.moveTo(-200,0);ctx.lineTo(200,0);
  ctx.stroke();
  ctx.strokeStyle='#f00'
  ctx.beginPath();
  ctx.moveTo(l*10,0);ctx.lineTo(l*5,l*Math.sqrt(3)*5);ctx.lineTo(-l*5,l*Math.sqrt(3)*5)
  ctx.lineTo(-l*10,0);ctx.lineTo(-l*5,-l*Math.sqrt(3)*5);ctx.lineTo(l*5,-l*Math.sqrt(3)*5)
  ctx.closePath();
  ctx.stroke();

  for(y=-14;y<15;y++)
    for(x=-19;x<20;x++) {
      ctx.beginPath();
      ctx.moveTo(x*10,y*10-3);ctx.lineTo(x*10,y*10+3);
      ctx.moveTo(x*10-3,y*10);ctx.lineTo(x*10+3,y*10);
      ctx.strokeStyle=cols[f(x,y,l)]
      ctx.stroke()
    }
}

go()

#C {
  border: 1px solid #000
}

<b>L</b> <input id=I value=15><button onclick="go()">GO</button><br>
<canvas id=C width=400 height=300></canvas>

Ruby, 150 145 137 127 125 106 88 76 bytes

76 bytes

->(x,y,l){x,y,t=x.abs,y.abs,3**0.5;d=t*l;z=d-t*x-y;2*y>d ?-1:2*x<l ?1:z<=>0}

Changed triple comparison to a rocket.

88 bytes

->(x,y,l){x,y,t=x.abs,y.abs,3**0.5;d=t*l;z=d-t*x-y;2*y>d ?-1:2*x<l ?1:z==0 ?0:0<z ?1:-1}

Remove the y equal to apothem test for points on the hexagon, because for integers, that can never be true.

106 bytes:

->(x,y,l){x,y,t=x.abs,y.abs,3**0.5;d=t*l;z=d-t*x-y;2*y==d&&2*x<=l ?0:2*y>d ?-1:2*x<l ?1:z==0 ?0:0<z ?1:-1}

Poster suggested not using epsilon, so replaced epsilon with zero and rearranged, removed an abs, etc.

125 bytes:

->(x,y,l){x,y,t,e=x.abs,y.abs,3**0.5,1e-9;d=t*l;z=d-t*x-y;(2*y-d).abs<=e&&2*x<=l ?0:2*y>d ?-1:2*x<l ?1:z.abs<=e ?0:0<z ?1:-1}

Incorporate y into definition of z and remove some parentheses.

127 bytes:

->(x,y,l){x,y,t,e=x.abs,y.abs,3**0.5,1e-9;d=t*l;z=d-t*x;(2*y-d).abs<=e&&2*x<=l ?0:2*y>d ?-1:2*x<l ?1:(z-y).abs<=e ?0:y<z ?1:-1}

Rearranged terms to avoid necessity of to_f cast. Use d (double the apothem) instead of a (the apothem). Combine multiple assignments.

137 bytes:

->(x,y,l){x=x.abs.to_f;y=y.abs.to_f;a=3**0.5*l/2;e=1e-9;z=2*a*(1-x/l);(y-a).abs<=e&&2*x<=l ?0:y>a ?-1:2*x<l ?1:(z-y).abs<=e ?0:y<z ?1:-1}

Inlined 'c'.

150 bytes:

->(x,y,l){c=l/2.0;x=x.abs.to_f;y=y.abs.to_f;a=3**0.5*l/2;e=1e-10;z=2*a*(1-x/l);(y-a).abs<=e&&x<=c ?0:(y>a ?-1:(x<c ?1:((z-y).abs<=e ?0:(y<z ?1:-1))))}

This works for integers or floats! The epsilon test is so that points within round off error of being on the edge are correctly identified.

The absolute values move everything into quadrant one.

The value 'a' is the apothem distance (the y-intercept of the hexagon).

The value 'c' is the x-value of the upper right corner of the hexagon.

The value 'z' is to see if the point is above or below the slant line from the corner to the x-intercept.

Ungolfed:

hex = ->(x,y,l){ 
    c = l/2.0;
    x = x.abs.to_f;
    y = y.abs.to_f;
    a = 3**0.5 * l / 2;
    e = 1e-10;
    z = 2*a*(1 - x/l);
    if (y-a).abs <= e && x <= c then 0
    elsif (y>a) then -1
    elsif (x<c) then 1
    elsif (z-y).abs <= e then 0
    elsif y < z then 1
    else -1
    end
}

Test

hex = ->(x,y,l){x,y,t=x.abs,y.abs,3**0.5;d=t*l;z=d-t*x-y;2*y>d ?-1:2*x<l ?1:z<=>0}

cases = [
    [0,0,1,1],
    [0,1,1,-1],
    [0,-1,1,-1],
    [1,0,1,0],
    [-1,0,1,0],
    [-1,-1,1,-1],
    [1,1,1,-1],
    [-2,-3,4,1],
    [32,45,58,1],
    [99,97,155,-1],
    [123,135,201,1]
]

cases.each { |test| 
  expected = test[3]
  actual = hex.call(test[0],test[1],test[2])
  status = expected == actual ? "PASS" : "FAIL";
  p "#{status}. #(x,y) L = (#{test[0]},#{test[1]}) #{test[2]} Expects #{expected}. Actual #{actual}"
}
"Done!"