How do I draw simple graphics in C#?

Here's a simple code sample that will get you started (assumes you have a PictureBox named pictureBox1):

Bitmap bmp = new Bitmap(pictureBox1.Width, pictureBox1.Height);
using (Graphics g = Graphics.FromImage(bmp))
{
    g.DrawLine(new Pen(Color.Red), 0, 0, 10, 10);
}
pictureBox1.Image = bmp;

The graphics object has a bunch of other drawing methods, and Intellisense will show you how to call them.

As others have said, check out System.Drawing. (I'm only repeating that for completeness.) System.Drawing exposes the GDI+ Windows drawing library to your application.

A good tutorial to get you jump-started with System.Drawing and GDI+ can be found at C# Corner.

Some important items to note:

1. Many GDI+ objects implement the IDisposable interface, and therefore should be wrapped in using blocks. Be sure you follow the appropriate disposal conventions; failing to dispose GDI+ objects can result in really nasty side effects for your app. (GDI+ objects in .NET correspond to their underlying Windows API equivalents.)
2. APIs such as DirectX are extremely complex, and for good reason. They're designed not for simple shapes, but rather for complex, highly-performant and highly-interactive multimedia applications. (In other words, games, typically.) You can access DirectX through the Managed DirectX interfaces, but again, it's probably overkill for your direct purposes.
3. If you are interested in an easier way to work with DirectX, XNA is the way to go. However, this is very much a gaming-specific library, and again is likely to be overkill. I'm a bit late to the party, but according to the comments below, this is no longer supported at all. (This makes sense; I haven't heard anything about it in years.)

Read about GDI, GDI+, System.Drawing namespace, for example here.
DirectX is not something you would use to draw simple shapes, rather render complicated 3D stuff, also, using DX Api under C# is a bit trickier (although not that hard).