What exactly is a current source?

A current source is the dual of a voltage source. An ideal voltage source has zero output impedance, so that the voltage doesn't drop under load. It shouldn't be shorted, because in theory there would flow an infinite current.
An ideal current source has infinite output impedance. This means that the load's impedance is negligible and won't influence the current flowing. Like voltage sources shouldn't be shorted, current sources shouldn't be left open. An open current source will still try to source the set current, and the theoretical current source will go to infinite voltage.

edit (following your comment)
Here you can read impedance as resistance. If the current source would have a limited resistance changes in load would change the current, because the total resistance would change. You don't want that. So if the current source's resistance is infinite the load can be ignored and the resistance always remains the same (infinite). Therefore the current will as well.

A practical current source may be constructed as follows:

One diode has the same voltage drop as the base-emitter junction, so the other diode sets the transistor's emitter to about 0.7V. A fixed voltage across a fixed resistor gives a fixed emitter current, which is about the same as the collector current if the transistor's \$H_{FE}\$ is high enough. (Strictly speaking this is a current sink rather than a current source, but the principle remains the same.)

Another current sink uses an opamp as control element:

The main thing you need to know about opamps in this configuration is that they will try to keep the voltage on both inputs equal. So suppose you set \$V_{SET}\$ to 1V, then the opamp will try to make the - input also 1V. It does so by inserting current into the transistor's base. This will cause a current through the load \$I_{LOAD}\$ which is (almost) equal to \$I_{SET}\$. And \$I_{SET}\$ is constant to get the 1V across \$R_{SET}\$, according to Ohm's Law:

\$ I_{SET} = \dfrac{V_{SET}}{R_{SET}} \$

Since \$V_{SET}\$ and \$R_{SET}\$ are constant, so will \$I_{SET}\$ be. QED.

After reading your comments, I'm going to make a bit different answer to this question.

What exactly is a current source? It's nothing, or to put it a bit better it's just a mathematical model. The one you're describing does not exist, just as a voltage source doesn't exist.

I think that the main problem here is cased by this statement: for example a battery which has a constant potential difference across its ends irrespective of the changes in the circuit it is connected to which is incorrect. That it the behavior of ideal battery which is real as ideal current source and just like ideal current source it doesn't exist. The output (and internal state) of every real battery is affected by the circuit it is connected to.

So why do we have voltage and current sources? Well the idea is that engineer's job is to basically construct a device which does something pretty well and as it turns out for that complete understanding of how each component used in the device isn't needed. That is why we have things such as ideal current and voltage sources.

Let's get back to the battery example once again. Here's a simple experiment I did with a lithium polymer battery I have: First I completely charged the battery. Since it's a two cell battery, its voltage was 8.4 V when fully charged, even though its nominal voltage is 7.4 V. Then I connected a \$ 100 \mbox{ } k\Omega\$ resistor to the battery. It's voltage remained 8.4 V and from that I could perhaps conclude that the battery is indeed ideal voltage source since I connected the load to it, but its voltage did not change. Then I took an electrical motor I have and connected it to the battery and measured battery's voltage again. This time, it was 8.2 V. Clearly the motor affected the battery and it is no longer an ideal voltage source, even though it's the same battery as before. So I disconnected the motor and connected the resistor again and again the voltage at the battery was 8.4 V.

So what's happening here? Is the battery an ideal voltage source or not? Well we know it isn't because cause I said so at the beginning of the answer, but here I'll explain why it sometimes seems that it is and sometimes it seems that it isn't. As I said, the voltage source is a mathematical model. When the external circuit doesn't make a big impact on the operation of the battery, I can use it an when the external circuit does make a big impact on the battery, I can't use it. So we're using a simple model to represent behavior of a real circuit. Another model would be to use an ideal voltage source with a resistor in series at its output. When I connect and external load to that circuit, some voltage will be dropped at the internal resistor and the external resistor will see lower voltage at the output. This allows me to once again use the ideal voltage source to represent the battery and since I'm using the internal resistor together with the ideal voltage source, the output will more closely represent the behavior of a real battery. Should I want more precision, I could decide to use a more complex model and get more accurate results.

An important point of electrical engineering is to learn when to use right model to represent an extremely complex real-life circuit component (and even the humble resistor, when analyzed in detail, is a masterpiece of modern science). But to be able to do that, we start with simple circuits so that we can get to know how the simplest mathematical models actually work.

When we start analyses of more complicated circuit components, like transistor or diode for example, we'll break them down into a simple circuit consisting of things such as resistors and ideal current and voltage sources. This will allow us to simplify the behavior of more complex component and avoid analyzing in detail how it works, if the simple model is sufficient for our needs.

Completely same story works for current sources, but I decided not to tell it here since, as you can see from the other answers, circuits which can be modeled as ideal current sources are too complicated for you to understand at this point.

So to sum this up: There are no real-life objects that can be used to represent ideal voltage and current sources, but there are some objects that can be (in some cases quite closely) represented with ideal voltage and current sources. The best thing you can do now is to memorize correctly the definitions of ideal voltage and current sources and not to confuse them with real objects. This way you won't be surprised if a battery doesn't provide its nominal voltage or if a circuit labeled ideal current source starts smoking at one point although it should be completely immune to external changes in the circuit.

As a side-note consider what happens to ideal voltage source when its outputs are shorted and what happens to ideal current source when its outputs are open? And what happens when you short a battery an why all batteries have the warning not to short the output pins?

Perhaps this answer will help. I'm saying pretty much the same thing as AndrejaKo, but my post will be shorter.

Much like voltage sources, current sources are only a theoretical construct. A battery can be a resonably close approximation to a voltage source, but it is not exact.

Howeer unlike voltage sources, which are approximated by batteries, there is no simple component that approximates a general current source particulkarly well. That does not mean the concept is not useful though, as many real world circuits can be modeled using the concept.

I've seen lab power supplies that have two knobs, one which adjusts voltage, the other adjusts current. To use these supplies as a voltage source, you simply set the current to the maximum, and dial in the voltage you want. As long as the circuit does not require more than the maximum current, the supply will supply your chosen voltage. To use it as a current source, dial the voltage to maximum, and set your desired current. The supply will provide that current as long doing so does not require a voltage greater than the maximum.