Programmatically edit/update metadata in ArcGIS?

We had a big need for a similar capability and ended up building a general, free, open source Python library for the purpose. You can find it at https://github.com/ucd-cws/arcpy_metadata or by running a "pip install arcpy_metadata". There is some documentation of features and how to use it, with some additional contributions from the World Resources Institute. We tried to keep things relatively Pythonic so that it integrates well and can be learned quickly. Here's an example:

import arcpy_metadata as md
import datetime

metadata = md.MetadataEditor(path_to_some_feature_class)  # also has a feature_layer parameter if you're working with one, but edits get saved back to the source feature class
metadata.title = "The metadata title!"

generated_time = "This layer was generated on {0:s}".format(datetime.datetime.now().strftime("%m/%d/%Y %I:%M %p"))

metadata.purpose = "Layer represents locations of the rare Snipe."

metadata.abstract.append("generated by ___ software")
metadata.abstract.append(generated_time)  # .prepend also exists
metadata.tags.add(["foo", "bar", "baz"])  # tags.extend is equivalent to maintain list semantics

metadata.finish()  # save the metadata back to the original source feature class and cleanup. Without calling finish(), your edits are NOT saved!

It still has plenty that could be added, but is pretty extensible if you subclass the items that are already there, or configure them correctly. It's still about alpha quality software, but it works and we're happy with it.

---

For anyone looking for this capability within ArcGIS Pro, as of version 2.5, they now include a metadata API from Python. There are more details in the Metadata class documentation.

The easiest way to do this from ArcPy is to create an XML file using Python and then invoking Import Metadata (Conversion). However, this will overwrite everything.

An alternative is to use ArcObjects to obtain an IName to the dataset, cast to IMetadata, and edit the IPropertyset.

We have been struggling with this and solved it by having a simple python library built around the arcobjects metadata functions. It's based on a xml-file where the metadata values have been replaced with keywords. You can use these keywords in your python code to get or set the value of a metadata field. Code woud look something like this:

import dataprocessing
DP = dataprocessing.create()

md = DP.OpenMetadata("C:/MyShapefile.shp") # Works also on geodatabases
myKeyword = "%FGDC.abstract" # Keywords have to start with '%' but the rest is up to you

if md.GetValue(myKeyword) == "Not what I want":
    md.SetValue(myKeyword, "New value that suits me better")

The tool can be found here: http://www.aris.nl/dataprocessing_arcgis and is freely available. The version on the website is for arcgis 9.3. An arcgis 10 version will be available but it will no longer be free.

(Full disclosure: I work for a dutch government agency where we had the current version of this tool built partly from taxpayers money and partly from time donated by the programmer. That's why it's available for free but not open source. As of 2013 we will not be funding any new versions. I'm not making any money out of it in any way. I just hope it gets put to more use.)