ArcGIS Server vs Open Source software

I personally favor Open Source Software, and lucky enough to work for an Open Source based company. In the past year, I've worked with a large veriety of open source GIS tools (mainly PostgreSQL, PostGIS, QGis. Ubuntu, Lighttpd and Python), and I favor them over any propietry tool in the market. Anyway, here's my 50 cents:

Open Source GIS Infrastructure

Pros

• Easy to start with. If you're starting a small company, a private venture, or even a project within a large company, you'll appreciate the ability to be able to freely experiment with technologies without paying any royalties.
• Community support. Perhaps the greatest FOSS advantage. There's virtually no question regardinga popular open source project that haven't got a profound answer in the web. For the undocumented questions, you'll probably get an answer within 24 hours in a professional forum.
• Scalability. See DavidF's answer.
• Trying before implementing. If you wnat to convert a software component to a different infrastructure, technology or environment, you can have a free sandbox to play with before converting, and you can always go back. This allowes priceless experience with cutting-edge technologies without the financial risks involved with trying new pricy products.
• Easy to port.. When your data is kept in open formats, translating from one data stype to another is straight forward, and there is probably a piece of software that does exactly that. Figuring out closed format is a truely dantesque experience.
• Maximal Control. Open source software allowes extensive configurability, which means that you can fine-tune the product to your exact needs. For niche demands, hiring a software developer to change the product will be considerably cheaper than paying a software company for changing the product (and they probably JUST DON'T DO this kind of things).
• Attracts better developers. Without starting a flame war, I think that in general, Open Source software developers seem to better perform, be more indpependent, productive and curious than developers under propietry software infrastructure.
• Great web tools. There's a plethora of web-oriented open source tools: mapping, tiles, databases, webservers, web framework and web authoring tools. Buidling your first GIS website will be very easy.

Cons

• Too many projects. There are many open source projects for every niche, and you'll have to spend some tome picking the best one, because other will be abandoned.
• Probably no tech support or SLA, Unless you pay a consultant.
• GUI tools are of inferior quality. This is probably because software developers are familiar with command-line tools, and there are not enough open source GUI designers. The resulting GUI (e.g., GRASS GIS) is often slow, ugly, and have counterintuitive interface.
• Mostly Linux based software. Windows Open Source software tools have gone a long way in the past years, but the best-of-breed software is almost always in the Linux world.
• GPL\License restrictions. This is hardly a con, because propietry software can't be distributed at all. But note that in most cases, modifying and distributing Open Source product forces your to redistribute the new source code.

Propietry GIS Infrastructure

Pros

• Spec. Propietry software tend to come with an well-written spec and SLA papers.
• ArcGIS - De Facto standard. ArcGIS, the most popular non-free GIS infrastructure, tend to be an industry standard. It's not the only standard there is, and there are many conversion tools, by most of your clients wll ask for data that can be opened with ArcGIS tools.
• Better GUI. Nearly always.
• MS-Windows friendly. Most of the users, and many developers, will consider this as an advantage.
• Support. If you're going to pay for it, you'll probably get good telephone support from the vendor.

Cons

Apart of the aforementioned disadvantages (cost, difficult scalability, restrictions, allows little or no trial period and uncustomizable):

• Cost tend to pile up. When you need an extra software component that would fit to your existing infrastructure, it's probably going to cost you a lot more.

I think that your question is too vague to start talking about individual capabilities, but there are two things that come to my mind in general:

Scalability - If you write that 'killer app' and a lot of people start using it, all of a sudden, your initial server/database setup is overloaded. If you are using OpenSource software, you can switch over to a more powerful server or add a second server behind a load balancer. If you are paying licenses based on the number of cores or the number of users hitting the app, you are looking at a significant increase in license costs. Ever hear of anyone buying less capable servers because their licensing only allowed two cores?...

Controlling your own fate - Ever had a software bug that you couldn't get the company to acknowledge and then when the did verify it, they kept promising to fix it in the next release? If you discover a bug in most OpenSource products, you can fix it yourself or pay someone else to fix it if it is a priority. As soon as the bug is fixed, you can use the new version instead of having to wait for an official release.

I have mixed opinions our IT people have some reservations about using opensource products as some (not all) can appear one day and be gone the next.

I have played with both Opensource (MapGuide) and Proprietry (Bentley Geo Web Publisher) while I have found the Proprietry software to be a great start point and quick to get your project up and running, I found developing for that particular software a nightmare, slow to get help, small community.... and you'll pay through the teeth to get something fixed.

On the other hand MapGuide took a while to get my data into a format that it would recognize (bad data management on our side prevented some export operations).

But once in MapGuide the community is very active and very large and the api/sdk is easy to write applications for. I wouldn't look back and I'm sure many other opensource projects are similar.

Dunno If this helps but it's my personal experience.