How do I get the base URL in the formula editor?
This will give you the Base URL
LEFT($Api.Partner_Server_URL_260, FIND( '/services', $Api.Partner_Server_URL_260))
In almost all instances you shouldn't need the instance base url to make a custom button. APEX has very recently added functions to get this programmatically. And as @techtrekker points out you can also get it in the formula field.
Let's say you're on the na9 instance and you want a custom button to create a task from an account for the subject and comments. You'd end up at this url when you create a task using the regular new task button for an account:
https://na9.salesforce.com/00T/e?what_id=001E000000XXXvC&retURL=%2F001E000000XXXvC
&tsk6=Account+Name&tsk5=Account+Name
To make this into a custom button you could code hard code the instance like this:
https://na9.salesforce.com/00T/e?what_id={!Account.Id}&retURL=%2F{!Account.Id}
&tsk6={!Account.name}&tsk5={!Account.name}
Or use the api partner url to add the instance dynamically:
LEFT($Api.Partner_Server_URL_260, FIND( '/services', $Api.Partner_Server_URL_260))/00T/e
?what_id={!Account.Id}&retURL=%2F{!Account.Id}&tsk6={!Account.name}&tsk5={!Account.name}
But why not just avoid the issue and not specify the instance entirely:
/00T/e?what_id={!Account.Id}&retURL=%2F{!Account.id}
&tsk6={!Account.name}&tsk5={!Account.name}