How do I get the base URL in the formula editor?

This will give you the Base URL

LEFT($Api.Partner_Server_URL_260, FIND( '/services', $Api.Partner_Server_URL_260))

In almost all instances you shouldn't need the instance base url to make a custom button. APEX has very recently added functions to get this programmatically. And as @techtrekker points out you can also get it in the formula field.

Let's say you're on the na9 instance and you want a custom button to create a task from an account for the subject and comments. You'd end up at this url when you create a task using the regular new task button for an account:

https://na9.salesforce.com/00T/e?what_id=001E000000XXXvC&retURL=%2F001E000000XXXvC
&tsk6=Account+Name&tsk5=Account+Name

To make this into a custom button you could code hard code the instance like this:

https://na9.salesforce.com/00T/e?what_id={!Account.Id}&retURL=%2F{!Account.Id}
&tsk6={!Account.name}&tsk5={!Account.name}

Or use the api partner url to add the instance dynamically:

LEFT($Api.Partner_Server_URL_260, FIND( '/services', $Api.Partner_Server_URL_260))/00T/e
?what_id={!Account.Id}&retURL=%2F{!Account.Id}&tsk6={!Account.name}&tsk5={!Account.name}

But why not just avoid the issue and not specify the instance entirely:

/00T/e?what_id={!Account.Id}&retURL=%2F{!Account.id}
&tsk6={!Account.name}&tsk5={!Account.name}