CAShapeLayer with border and fill color and rounding

If you set the layer's fillColor property to something other than nil or transparent, the layer will fill its path.

If you set the layer's lineWidth to a number larger than zero and you set its strokeColor to something other than nil or transparent, the layer will stroke its path.

If you set all of those properties, the layer will fill and stroke its path. It draws the stroke after the fill.

The layer's path must actually enclose some area in order for it to fill anything. In your post, you set the path like this:

UIBezierPath* path = [UIBezierPath bezierPath];
[path moveToPoint:self.lineStart];
[path addLineToPoint:self.lineEnd];
self.lineShape.path = path.CGPath;

That path contains a single line segment. It doesn't enclose any area, so the layer has nothing to fill.

Swift 3+ extension method (combined from AechoLiu answer below):

// Usage:
self.btnGroup.roundCorner([.topRight, .bottomRight], radius: 4.0, borderColor: UIColor.red, borderWidth: 1.0)

// Apply round corner and border. An extension method of UIView.
public func roundCorner(_ corners: UIRectCorner, radius: CGFloat, borderColor: UIColor, borderWidth: CGFloat) {

let path = UIBezierPath.init(roundedRect: self.bounds, byRoundingCorners: corners, cornerRadii: CGSize(width: radius, height: radius))

let mask = CAShapeLayer()
mask.path = path.cgPath
self.layer.mask = mask

let borderPath = UIBezierPath.init(roundedRect: self.bounds, byRoundingCorners: corners, cornerRadii: CGSize(width: radius, height: radius))
let borderLayer = CAShapeLayer()
borderLayer.path = borderPath.cgPath
borderLayer.lineWidth = borderWidth
borderLayer.strokeColor = borderColor.cgColor
borderLayer.fillColor = UIColor.clear.cgColor
borderLayer.frame = self.bounds
self.layer.addSublayer(borderLayer)

}

Objective-C:

self.lineShapeBorder = [[CAShapeLayer alloc] init];
self.lineShapeBorder.zPosition = 0.0f;
self.lineShapeBorder.strokeColor = [UIColor blueColor].CGColor;
self.lineShapeBorder.lineWidth = 25;
self.lineShapeBorder.lineCap = kCALineCapRound;
self.lineShapeBorder.lineJoin = kCALineJoinRound;
     
self.lineShapeFill = [[CAShapeLayer alloc] init];
[self.lineShapeBorder addSublayer:self.lineShapeFill];
self.lineShapeFill.zPosition = 0.0f;
self.lineShapeFill.strokeColor = [UIColor greenColor].CGColor;
self.lineShapeFill.lineWidth = 20.0f;
self.lineShapeFill.lineCap = kCALineCapRound;
self.lineShapeFill.lineJoin = kCALineJoinRound;
  
// ...
    
UIBezierPath* path = [UIBezierPath bezierPath];
[path moveToPoint:self.lineStart];
[path addLineToPoint:self.lineEnd];
// call addLineToPoint over and over to create the outline for the shape
self.lineShapeBorder.path = self.lineShapeFill.path = path.CGPath;

Also calling out @FarrasDoko comment to try changing borderWidth to lineWidth and borderColor to strokeColor.