How to avoid bit torrent to degrade my connection performance without limiting bandwidth usage?

1-Choose a good ISP

The bottleneck could be at your ISP's DSLAM / CMTS, and if your ISP doesn't handle congestion avoidance and traffic shaping properly, except changing or asking less from it, there's nothing you can do.

2-On your side

However, usually what's degrade the network performance is not the download bandwidth used alone (for a single connection, a large bandwidth usage is well handled natively by ISP, router, OS and other pieces of the network) but the number of concurrent connections used and/or bad allocation of your (limited) upload bandwidth

2.1-Your router is an essential piece of your network

So a modem/router that handle well many simultaneous connections and congestion could really help. My trust would go to linksys-cisco or netgear. But this link SmallNetBuilder.com Router Charts could be interesting

Additionally, you could get many advanced features (QoS, alternative congestion algorithm, iptable, ...) on some home routers with an alternative firmware: dd-wrt (supported routers list) (tomato and openwrt are popular alternative too, but they support less devices and are less user friendly).
Note: if your router have low spec (memory/cpu) don't be too greedy...

2.2-How to setup µTorrent

2.2.1-As it was said by Fishwalker, it's a good idea to start with the µTorrent speed guide. Additionally:

• Select the listing closest to your upload speed result (rounding down as necessary)
• Higher settings will not give you better speed, and may in fact make download speeds worse. Too low of an upload speed will give the same result. So don't try to be greedy.

2.2.2-Now that the basic setup is done, if you still have connections issues, decrease slightly the numbers, step by step with some tests. As I said, I would start by the number of connections. And depending on your connection, upload bandwidth usage could be critical too, but for the good health of the torrent network and the reward system, you should keep it at the highest possible...

• All the settings are in Options > Preferences > Bandwidth
• Except Half-open connections: Options > Preferences > Advanced > net.max_halfopen

2.2.3-For indication, here are some recommend settings from torrentfreak.com to optimize µTorrent download speed, so you should decrease them to avoid to degrade the other connections :

• Upload bandwidth: approximately 85% of your upload rate
• Download bandwidth: approximately 95% of your download rate (Slower connections (upload <60 kB/s) need a little more space so download rate * 85%)
• Maximum connected peers per torrent: Not too high, not too low: try upload rate * 1.3
• Maximum upload slots: 1 + (upload rate / 6)
• Maximum half-open tcp connections: upload rate * 2 (& never higher than 1000) and more for slower connections (upload <60 kB/s): upload rate * 4

2.3-Scheduling

As it was said by Pangea A simple trick but maybe the most effective way to deal with it: If you keep your computer running when you don't use it, or use it locally only, set µtorrent to download only at these times: Options > Preferences > Scheduler

2.4-Sources, trackers & seedbox

If you find sources that have a good upload bandwidth you could use less simultaneous sources/connections and however achieve a good speed:

• A good tracker with generous users will help you.
• A good ratio tracker with many users that have a seedbox will help you more.
• Using a seedbox yourself beside give you a high ratio, could enable you to download at the maximum speed of your connection while avoiding to deal with the requiring bittorrent protocol on your own computer. But cost you some bucks every month...

2.5-How to setup QoS

As it was also said by Pangea, QoS could be used on your side for traffic shaping. (only for outbound data)

An overview: What's QoS?

2.5.1-Usually QoS is set on the router to be beneficial to the entire network:

You could just let µTorrent at the default priority and up the other connections...

But to create a QoS rule specifically for µTorrent on your home router, you must choose a port (outside of the well-know ports range, and outside of the 6881-6999 range to avoid limitations of some providers) so that µTorrent will be recognized by the router

• µTorrent: Options > Preferences > Advanced > net.outgoing_port
• set a router QoS rule accordingly.

For more resources: QoS tutorial linksysinfo.org

2.5.2-To setup QoS on your computer:

• Win + R
• gpedit.msc
• Computer Configuration (or User Configuration) > Windows Settings > right-click Policy-based QoS, and then click Create new Policy.
  • DSCP: A higher value indicates a higher priority for the packet. By default, the Specify DSCP Value check box is selected and its value is 0.
  • Throttle Rate: will limit the outgoing network traffic to a specified rate
  • Create your rule based on Application / URL / IP / Protocol / Port

For more resources - Microsoft Technet: Working with QoS policies / Policy-based Quality of Service (QoS) / Creating and Editing a QoS Policy / Advanced QoS Settings / Quality of Service / Policy-based QoS Architecture in Windows Server 2008 and Windows Vista

2.6-Additional softwares that could help

I don't have used them myself, but I have read some good reviews about:

cFosSpeed

• NDIS driver for Windows 7
• Traffic Shaping
• Traffic limits, speed limits
• Prioritization of Programs
• RTP/VoIP Detection
• ...

Cost $ 19.90

Netlimiter

Grant is a feature in NetLimiter 2 (pro version only). It could seem similar to limits, but it works differently. When you set grant for application or connection, then it means that you grant specified bandwidth for it. If other application/connections take too much bandwidth, then it is taken from them and is given to application/connection with granted bandwidth.

Cost $ 29.95

Your router might have options for traffic shaping and QoS, this will give higher priority to traffic you designate like HTTP and lower the priority for your torrents. Also some torrent clients allow for scheduling pretty so you can designate your torrents to upload and download during off hours when everyone is asleep.

Since you're on Windows 7, you can use QoS without a router.

See the Microsoft article at : URL-based QoS
as well as QoS program manager's blog: QoS in Windows 7
See also this Cable Guy article: QoS Support in Windows

A step-by-step description of the process is here : How URL QoS Works in Windows 7: