bundler vs RVM vs gems vs RubyGems vs gemsets vs system ruby
You're asking for more information in one question than is in-scope for Stack Overflow. To cover it all would take a book.
On Ubuntu it's easy to install and remove gems to the "system" version of Ruby, so get used to installing and removing regular gems via sudo. (On Mac OS I'd give different advice because Apple bundles Ruby for their own use and it's not a great idea to mess with it.) Then, when you have an idea how the whole gem idea works, and you know you want multiple Ruby versions on your system, try "rbenv" or "RVM" and install a version or two in your sandbox.
Linux makes it easy to add/remove Ruby via a distribution, but we're limited to the versions the distro maintainers have packaged, so I usually install from source. But, that's a pain when managing several versions of Ruby for development, test and production systems, which is why rbenv and RVM were invented -- they handle the dirty detail allowing us to concentrate on programming.
I've used both rbenv and RVM, and have been using rbenv for the last six months or so, with good results. It's less complicated than RVM which I like. In either case they make it easy to have different versions installed, with separate sets of Gems. You can have different Ruby versions open in different terminal windows if you want, making it easy to test for compatibility.
Rule one when debugging is to make changes one at a time, which is true for learning to program or learning a new language. Don't be distracted, just keep it simple.
As per the previous answer, this is quite a lot to cover, so consider this a short introduction.
gems are the way Ruby libraries are packaged. They are to Ruby what jars are to Java. Inside a gem file, you find Ruby code (.rb files), but also tests, and a special file giving information on the gem itself, such as its name, dependencies and version (gemspec). Any Ruby project can define the gems it needs via a Gemfile that just need to declare dependencies. Rubygems is the name of the package manager - the tool used to install the packages (while the gems are the packages themselves). Rubygems is now part of Ruby.
Bundler is what makes managing gems bearable. Based on your Gemfile, a simple call to bundler using bundle install will download and install all the required gems. Using standard gem command, you would have to install each of them manually, using gem install <gem_name>. Bundler is not part of Ruby (it is itself packaged as a gem), but it a "de facto standard" for most applications (you will not find many people not using it, and no good reasons not to use it, actually).
RVM is a tool allowing you to install multiple versions of Ruby on a machine, switching between them when needed. This can be used to install both a Ruby 1.8 and 1.9, or even a "MRI" (Matz's Ruby, the default implementation) and alternatives (such as JRuby or Rubinius). Note that RVM is not alone in this field, see for instance rbenv.
A gemset in RVM is a set of gems specific to a given context, typically a project. This is useful if you are for example developing different applications, each with its own sets of gems, and want to keep them separate.
system Ruby is, when using RVM, the Ruby version installed on the machine (ie, not via RVM).
If you are just starting, gems and bundler are of interest to you. You can let RVM and gemsets aside for now.