how to design a custom close, minimize and maximize button in windows form application?

Yes, it's possible without additional libraries.

First, hide the window's original border.

public Form1()
{
    InitializeComponent();

    FormBorderStyle = FormBorderStyle.None;
}

Next, create a panel, or whatever you want really, with your three buttons (I know it's ugly, for demo purposes):

Then, assign the correct action to each of them, using the WindowState:

private void minimizeButton_Click(object sender, System.EventArgs e)
{
    WindowState = FormWindowState.Minimized;
}

private void maximizeButton_Click(object sender, System.EventArgs e)
{
    WindowState = FormWindowState.Maximized;
}

private void closeButton_Click(object sender, System.EventArgs e)
{
    Close();
}

Finally, make the form draggable with our panel. Add those at the class level:

public const int WM_NCLBUTTONDOWN = 0xA1;
public const int HTCAPTION = 0x2;
[DllImport("User32.dll")]
public static extern bool ReleaseCapture();
[DllImport("User32.dll")]
public static extern int SendMessage(IntPtr hWnd, int Msg, int wParam, int lParam);

and plug them in a MouseDown event of the panel:

private void OnMouseDown(object sender, MouseEventArgs e)
{
    if (e.Button == MouseButtons.Left)
    {
        ReleaseCapture();
        SendMessage(Handle, WM_NCLBUTTONDOWN, HTCAPTION, 0);
    }
}

And now you have a draggable form, with your own bar at the top.

If you want it to be resizable, as @PhilWright mentionned, you can trap the WM_NCHITTEST message from WndProc and return HTBOTTOMRIGHT to trigger the resizing :

protected override void WndProc(ref Message m)
{
    if (m.Msg == 0x84)
    { 
        const int resizeArea = 10;
        Point cursorPosition = PointToClient(new Point(m.LParam.ToInt32() & 0xffff, m.LParam.ToInt32() >> 16));
        if (cursorPosition.X >= ClientSize.Width - resizeArea && cursorPosition.Y >= ClientSize.Height - resizeArea )
        {
            m.Result = (IntPtr)17;
            return;
        }
    }

    base.WndProc(ref m);
}

As @BenVoigt mentionned, you can use the Dock and Anchor properties on the buttons/panel so they can resize properly. If you don't, they will not follow the form on resize.

Just to complete the useful solution of @Pierre-Luc. When the window is maximized, how to resize it to its normal position if we click again to the maximize button. Here is the code:

private static bool MAXIMIZED = false;
private void maximizeButton_Click(object sender, System.EventArgs e)
{
    if(MAXIMIZED)
    {
        WindowState = FormWindowState.Normal;
        MAXIMIZED = false;
    }
    else
    {
        WindowState = FormWindowState.Maximized;
        MAXIMIZED = true;
    }
}

EDIT: As suggested by @LarsTech in the comment

private void maximizeButton_Click(object sender, System.EventArgs e)
{
    if (this.WindowState != FormWindowState.Maximized)
        this.WindowState = FormWindowState.Maximized;
    else
        this.WindowState = FormWindowState.Normal;
}