How can I build documentation using docbuild.xml outside of the Workbench?

Alternatively, is it possible to build the documentation directly from Mathematica, without having to use Ant and docbuild.xml? The Workbench includes a DocumentationBuild package.

Yes this is possible, but it requires tracing what docbuild.xml is doing. It's deleting build directory at beginning, and copying PacletInfo.m and everything, except */Guides/**, */ReferencePages/**, and */Tutorials/** files, from package directory to build directory at the end. Besides above trivial tasks it calls DocumentationBuild/SystemFiles/ant/Build/notebook.xml file which builds documentation notebooks, and /DocumentationBuild/SystemFiles/ant/Build/html.xml file which builds HTML version of documentation.

notebook.xml and html.xml contain over five hundred lines with most of it being blocks of Mathematica code. Fortunately most of this code is related to logging and exchanging data with Ant and parts are only related with building built-in Mathematica documentation, not user created packages. After trowing out those irrelevant parts following code can be assembled.

We start with some helper functions, mostly file-system related:

throwOnFailed // ClearAll
throwOnFailed[tag_]@$Failed := Throw[$Failed, tag]
throwOnFailed[tag_]@expr_ := expr

(* OptionValue reports OptionValue::nodef warning when options,
   from which value is taken, contain options not listed in used defaults.
   We could pass all relevant defaults as first argument of OptionValue,
   or in OptionsPattern, but doc-building functions usually pass options
   to called functions and defaults (or symbols on which they are set)
   are scattered all around, so it's easier to just supress this warrning
   and allow passing of arbitrary options. *)
nodefQuiet = Function[Null, Quiet[#, OptionValue::nodef], HoldFirst];

cleanDir // ClearAll
cleanDir::unexpectedFiles = "\
Directory `1` exists and contains files `2` that don't match expected \
patterns `3`. Directory was not cleaned.";
cleanDir::file = "\
File `1` already exists, but is not a directory. \
Clean directory not created.";
cleanDir[dirName_String?DirectoryQ, All | "*" | Verbatim@__] := (
    DeleteDirectory[dirName, DeleteContents -> True];
    CreateDirectory[dirName]
)
cleanDir[dirName_String?DirectoryQ, deletableFileNames_] :=
    With[
        {
            undeletable = Select[
                Complement[
                    FileNames[All, dirName, Infinity],
                    FileNames[deletableFileNames, dirName, Infinity]
                ],
                Not@*DirectoryQ
            ]
        },
        If[undeletable === {},
            DeleteDirectory[dirName, DeleteContents -> True];
            CreateDirectory[dirName]
        (* else *),
            Message[cleanDir::unexpectedFiles,
                HoldForm@dirName,
                HoldForm@undeletable,
                HoldForm@deletableFileNames
            ];
            $Failed
        ]
    ]
cleanDir[dirName_String?FileExistsQ, _] := (
    Message[cleanDir::file, HoldForm@dirName];
    $Failed
)
cleanDir[dirName_String, _] := CreateDirectory[dirName]
cleanDir[dirNames : {___String}, deletableFileNames_] :=
    cleanDir[#, deletableFileNames]& /@ dirNames

ensureDir // ClearAll
ensureDir[dir_String] := (
    Quiet[DeleteFile@dir, {DeleteFile::fdir, DeleteFile::fdnfnd}];
    Quiet[CreateDirectory@dir, CreateDirectory::filex]
)

forcedCopyFile // ClearAll
forcedCopyFile[from_String, to_String] := (
    Quiet[DeleteDirectory[to, DeleteContents -> True], DeleteDirectory::nodir];
    Quiet[CreateDirectory@DirectoryName@to, CreateDirectory::filex];
    CopyFile[from, to, OverwriteTarget -> True]
)

copyFiles // ClearAll
copyFiles[
    from_String, to_String, forms_ : All, n_ : Infinity,
    OptionsPattern@FileNames
] :=
    With[{fromDepth = FileNameDepth@from},
        Cases[FileNames[forms, from, n, IgnoreCase -> OptionValue@IgnoreCase],
            f_ /; FileType@f === File :>
                forcedCopyFile[f,
                    FileNameJoin@{to, FileNameDrop[f, fromDepth]}
                ]
        ]
    ]

withDirectory // ClearAll
withDirectory[Inherited] = Identity;
withDirectory[dir_String?DirectoryQ] := Function[Null,
    With[{oldCWD = Directory[]},
        CheckAll[
            SetDirectory@dir;
            #
            ,
            (
                SetDirectory@oldCWD;
                ReleaseHold@#2;
                #1
            )&
        ]
    ],
    HoldFirst
];

(* Documentation building leaves bunch of open links,
   this environment will close all links created during
   evaluation of its body. *)
linkBlock = Function[Null,
    With[{oldLinks = Links[]},
        CheckAll[#,
            (
                LinkClose /@ Complement[Links[],
                    oldLinks, {$ParentLink, $CurrentLink}
                ];
                ReleaseHold@#2;
                #1
            )&
        ]
    ],
    HoldFirst
];

Now functions assembled from code from notebook.xml and html.xml files, together with some helper functions solving common problems like customization of web URLs or switching off cell rasterization. We start by loading DocumentationBuild` package, path to Eclipse module containing it should be adapted to your installation.

If[Not@MemberQ[$Path, #], PrependTo[$Path, #]]&@
   "path/to/eclipse/configuration/org.eclipse.osgi/679/0/.cp/MathematicaSource";
Needs@"DocumentationBuild`"

$languages = <|
    "English" -> "en",
    "Japanese" -> "ja",
    "ChineseSimplified" -> "zh",
    "Spanish" -> "es"
|>;

validLangQ = KeyExistsQ[$languages, #]&;

$notebookInformationMetaDataNames = <|
    "Title" -> "windowtitle",
    "TitleModifier" -> "titlemodifier",
    "HistoryData" -> "history",
    "EntityType" -> "type",
    "PacletName" -> "paclet",
    "Context" -> "context",
    "URI" -> "uri",
    "Keywords" -> "keywords",
    "Underdevelopment" -> None,
    "TutorialCollectionLinks" -> "tutorialcollectionlinks",
    "ForeignFunctionKeywords" -> None,
    "Synonyms" -> "synonyms",
    "Summary" -> "summary",
    "Flag" -> "status",
    "IndexQ" -> "index"
|>;

metaDataToNotebookInformation::missingInfo = "\
Key `1` not present in MetaData: `2`";
metaDataToNotebookInformation =
    With[
        {
            info =
                Replace@Dispatch@Join[#, {
                    None -> None,
                    k_ :> (
                        Message[metaDataToNotebookInformation::missingInfo,
                            HoldForm@k, HoldForm@#
                        ];
                        Return[$Failed, With]
                    )
                }] /@
                    $notebookInformationMetaDataNames
        },
        Normal@ReplacePart[info, {
            "HistoryData" -> ({
                "New" -> #1, "Modified" -> #2,
                "Obsolete" -> #3, "Excised" -> #4
            }& @@ info@"HistoryData"),
            "Underdevelopment" -> False,
            "ForeignFunctionKeywords" -> {},
            "Flag" -> Replace[info@"Flag", "None" -> None]
        }]
    ]&;

fullSourceInformation // ClearAll
fullSourceInformation[
    inputDir_String, lang:(_String?validLangQ):"English", appName_String
] :=
    Module[{builtinSrcInfo, packageSrcInfo},
        builtinSrcInfo = Get@FileNameJoin@{
            $DocumentationBuildDirectory, "Internal", "data",
            "SourceInformation." <> Replace[lang, $languages] <> ".m"
        };
        If[builtinSrcInfo === $Failed, Return[$Failed, Module]];
        packageSrcInfo = DocumentationBuild`Info`CreateSourceInformation[
            inputDir, lang, appName
        ];
        Join[FilterRules[builtinSrcInfo, Except@packageSrcInfo], packageSrcInfo]
    ]

originalDotHTML // ClearAll
originalDotHTML // DownValues =
    DownValues@DotHTML /. DotHTML -> originalDotHTML;

withFixedHTMLExportLinks // ClearAll
withFixedHTMLExportLinks@appName_String := Function[Null,
    Internal`InheritedBlock[{DotHTML},
        DotHTML[s_String] := originalDotHTML@StringReplace[s,
            StartOfString ~~ "http://reference.wolfram.com" | ""
                ~~ "/language/" <> appName <> "/" ~~ rest___ :>
                    "../" <> rest
            ];
        #
    ],
    HoldFirst
]

withNoGraphicsToBitmapsConversion = Function[Null,
    Internal`InheritedBlock[{ApplyConvertGraphicsToBitmapsToNotebook},
        ApplyConvertGraphicsToBitmapsToNotebook = #1&;
        #
    ],
    HoldFirst
];

withCustomWebURL // ClearAll
withCustomWebURL@None := Function[Null,
    Internal`InheritedBlock[{DocumentationBuild`Make`Private`MakeAnchorBarCell},
        DocumentationBuild`Make`Private`MakeAnchorBarCell =
            Internal`InheritedBlock[
                {
                    DocumentationBuild`Make`Private`MakeAnchorBarCell,
                    StringMatchQ
                }
                ,
                DocumentationBuild`Make`Private`MakeAnchorBarCell =.;
                Unprotect@StringMatchQ;
                (* This StringMatchQ test determines
                   whether web links are added.  *)
                StringMatchQ[_, "*ref/message/*"] = True;

                DocumentationBuild`Make`Private`MakeAnchorBarCell@##
            ]&;
        #
    ],
    HoldFirst
]
withCustomWebURL@webURL_String := Function[Null,
    Internal`InheritedBlock[{DocumentationBuild`Make`Private`MakeAnchorBarCell},
        DownValues@DocumentationBuild`Make`Private`MakeAnchorBarCell =
            DownValues@DocumentationBuild`Make`Private`MakeAnchorBarCell /.
                "http://reference.wolfram.com/language/" -> webURL;
        #
    ],
    HoldFirst
]

withCustomNonSplittableCamelCase // ClearAll
withCustomNonSplittableCamelCase@Automatic = Identity;
withCustomNonSplittableCamelCase@strPatt_ := Function[Null,
    Internal`InheritedBlock[{AddSpaceToCamelCase},
        (* https://mathematica.stackexchange.com/a/132294/14303 *)
        DownValues@AddSpaceToCamelCase =
            DownValues@AddSpaceToCamelCase /.
                aut : HoldPattern[Alternatives][__String] :>
                    (strPatt /. Automatic -> aut);
        #
    ],
    HoldFirst
]

docBuildEnv // ClearAll
docBuildEnv // Options = {
    "CoreInformation" -> Automatic,
    "Debug" -> False,
    "LogFunction" -> Print,
    "FrontEndOptions" ->
        {RenderingOptions -> {"HardwareAntialiasingQuality" -> 1}},
    "CloseLinks" -> True
};
docBuildEnv[
    appName_String, inputDir_String, lang:(_String?validLangQ):"English",
    OptionsPattern[]
] := Function[Null,
    If[TrueQ@OptionValue@"CloseLinks", linkBlock, Identity]@Internal`InheritedBlock[
        {
            Print, Global`AntLog, DocumentationBuild`Info`Private`LogIt,
            DocumentationBuild`Make`Private`LogIt, DocumentationBuild`Utils`Private`LogIt
            ,
            $DocumentationExportDebug, $DocumentationBuildDebug,
            DocumentationBuild`Transform`debug,
            DocumentationBuild`Make`Private`$DocumentationExportDebug
            ,
            $CoreInformation, $Pages, $Graph, $NameToNumberRules,
            $NumberToNameRules, DocumentationBuild`indexer,
            URIsWithoutCloudSupport, URIsWithoutDesktopSupport,
            URIsWithoutPublicCloudSupport,
            $ShowStringCharacters, $XMLTransformPath
            ,
            DocumentationBuild`Utils`Private`appear, DocumentationBuild`Utils`Private`bb,
            DocumentationBuild`Info`Private`links, DocumentationBuild`Info`Private`linktypes,
            DocumentationBuild`Make`Private`GetRootCaptionList, DocumentationBuild`Export`Private`modifiedNotesQ,
            DocumentationBuild`Export`Private`ModifyRawCharacterList,
            DocumentationBuild`Export`Private`ModifyRawCharacterListForSolutions, DocumentationBuild`Make`Private`nb,
            DocumentationBuild`Make`Private`newSections, DocumentationBuild`Common`Private`oldids,
            DocumentationBuild`Make`Private`PlaceTutorialCollectionGroup,
            DocumentationBuild`Make`Private`ShortFormTitle, DocumentationBuild`Info`Private`str,
            DocumentationBuild`Utils`Private`TextCharacterReplacementRules,
            DocumentationBuild`Make`Private`tutorialCollection, DocumentationBuild`Export`Private`$InputCount,
            DocumentationBuild`Export`Private`$InputQueue, DocumentationBuild`Utils`Private`$Localization,
            DocumentationBuild`Common`Private`$PubsFELink
            ,
            Transmogrify`Private`atomval, Transmogrify`Private`btiDefaultImageReturn,
            Transmogrify`ImportCache`Private`CacheData, Transmogrify`Private`dims,
            Transmogrify`Private`filelist, Transmogrify`Private`funcs, Transmogrify`Private`GetNodeFunction,
            Transmogrify`ImportCache`Private`impopts, Transmogrify`Private`OUTPUT, Transmogrify`Private`params,
            Transmogrify`Private`rdp, Transmogrify`ImportCache`Private`time, Transmogrify`Layout`Private`vals,
            Transmogrify`Internal`$AtomReturnValue, Transmogrify`Internal`$AutoRecurse,
            Transmogrify`Private`$GroupOpen, Transmogrify`ImportCache`$ImportCache,
            Transmogrify`Private`$LastRequestedTransform, Transmogrify`Private`$Parameters,
            Transmogrify`Private`$Rules, Transmogrify`Private`$XMLTransformList
        },
        (* DocumentationBuild subpackages use different global variables
           as debug flags, set them all to single value. *)
        $DocumentationBuildDebug = $DocumentationExportDebug =
            DocumentationBuild`Transform`debug =
            DocumentationBuild`Make`Private`$DocumentationExportDebug =
            OptionValue@"Debug";

        (* Doc-building functions use different mechanisms for logging,
           some use Print or AntLog directly.
           Some use BuildLog which logs depending on $DocumentationBuildDebug,
           and calls Print or AntLog depending on value of Global`$AntBuildQ.
           Some use private LogIt function which is not defined at all. *)
        Unprotect@Print;
        Print = Global`AntLog = OptionValue@"LogFunction";
        Protect@Print;
        DocumentationBuild`Info`Private`LogIt =
            DocumentationBuild`Make`Private`LogIt =
            DocumentationBuild`Utils`Private`LogIt =
            BuildLog;

        (* Some building/exporting functions use FrontEnd
           and need specific $FrontEnd options. *)
        PubsEvaluateWithFE@SetOptions[$FrontEnd, OptionValue@"FrontEndOptions"];

        (* By default Automatic value of "CoreInformation" results
           in builtin-only info being used. Here, for Automatic,
           we combine builtin info with package specific info.
           Although many functions accept "CoreInformation" option,
           they cache it, on first usage, in $CoreInformation[lang] global variable,
           and subsequent functions always use this catched value,
           regardless of possible different explicit "CoreInformation" option setting.
           So just set it here once, for whole environment.
           If different core informations are really needed inside environment
           manually clear or reset $CoreInformation. Make definition delayed,
           so that if, in e.g. incremental build, there's nothong to do,
           then huge list with built-in core information will not be loaded at all. *)
        $CoreInformation[lang] := $CoreInformation[lang] = LoadCoreInformation[
            Replace[OptionValue@"CoreInformation",
                Automatic :> fullSourceInformation[inputDir, lang, appName]
            ],
            lang
        ];

        $Pages := $Pages = ComputeLinkTrails@GetGuidesDirectory[inputDir, lang];

        #
    ],
    HoldFirst
]

Inspired by Gulp.js, file processing will be composed of small easily chainable functions accepting as argument, and returning as result, some form of unified representation of a file. As this unified file representation I've chosen an Association with elements corresponding to file path, some representation of contents and any additional required metadata. We start with general functions useful in this approach:

virtualFileDate = Quiet[Check[
    FileDate@#, DateObject@0, FileDate::fdnfnd
], FileDate::fdnfnd]&;

toFileObject // ClearAll
toFileObject[] = <|
    "Path" -> ExpandFileName@#,
    "ModTime" -> virtualFileDate@#,
    "History" -> {}
|>&;

renameTo // ClearAll
renameTo[newDir_String] := renameTo[FileNameJoin@{newDir, FileNameTake@#Path}&];
renameTo[pathFunc_] := Append[#,
    {"Path" -> ExpandFileName@pathFunc@#, "History" -> {#History, #Path}}
]&;

exportUsing // ClearAll
exportUsing::failed = "Exporting to file `1` failed.";
exportUsing[exportFunc_] := (
    ensureDir@DirectoryName@#Path;
    Module[{result = exportFunc@#},
        If[StringQ@result && ExpandFileName@result === #Path &&
            FileType@result === File && FileByteCount@result >= 1
        (* then *),
            Append[#, "ModTime" -> FileDate@result]
        (* else *),
            Message[exportUsing::failed, #Path];
            #
        ]
    ]
)&

functionalIf // ClearAll
functionalIf[condFunc_, tFunc_, fFunc_ : Identity] :=
    If[condFunc@#, tFunc@#, fFunc@#]&
functionalIf[condFunc_, tFunc_, fFunc_, uFunct_] :=
    If[condFunc@#, tFunc@#, fFunc@#, uFunct@#]&

ifNewerThan // ClearAll
ifNewerThan[refPathFunc_, tFunc_, fFunc_ : Identity] :=
    If[#ModTime > virtualFileDate@refPathFunc@#, tFunc@#, fFunc@#]&

Now wrappers for documentation building functions adapting them to above scheme:

getDocPath // ClearAll
getDocPath::notSubPath = "\
File `1` is not inside directory `2`. Using its full path as DocPath.";
getDocPath[langDocDir: _String | Automatic : Automatic] :=
    With[
        {
            docBaseDirSeq = Sequence @@ If[langDocDir === Automatic,
                {Longest@___, "Documentation", _?validLangQ}
            (* else *),
                FileNameSplit@ExpandFileName@langDocDir
            ],
            msg = If[langDocDir === Automatic,
                "containing Documentation"<> $PathnameSeparator <>
                "(" <> ToString[Alternatives @@ Keys@$languages] <>
                ") sub-directory"
            (* else *),
                ExpandFileName@langDocDir
            ]
        },
        Append[#,
            "DocPath" -> FileNameJoin@Replace[FileNameSplit@#Path, {
                {docBaseDirSeq, docDirPath___, fileName_} :>
                    {docDirPath, FileBaseName@fileName},
                {docDirPath___, fileName_} :> (
                    Message[getDocPath::notSubPath,
                        HoldForm@#Path, HoldForm@msg
                    ];
                    {docDirPath, FileBaseName@fileName}
                )
            }]
        ]&
    ]

getDocNotebook // ClearAll
getDocNotebook::notANotebook = "\
Last expression from `1` file is not a Notebook expression.";
getDocNotebook::invNbInfo = "\
Could not extract notebook information from notebook `1`";
getDocNotebook[isSource : True | False : True, opts : OptionsPattern[]] :=
    With[{optsSeq = Sequence @@ Flatten@{opts}},
        Module[{nb, info, meta = {}},
            nb = Quiet[Get@#Path, General::newl];
            If[Head@nb === Notebook,
                If[isSource,
                    info = GetNotebookInformation[nb, optsSeq]
                (* else *),
                    meta = GetSearchMetaDataList[nb, optsSeq];
                    info = metaDataToNotebookInformation@meta
                ];
                If[info === $Failed, Message[getDocNotebook::invNbInfo, #Path]]
            (* else *),
                Message[getDocNotebook::notANotebook, #Path];
                nb = info = $Failed
            ];
            Append[#, {
                "Notebook" -> nb,
                "NotebookInformation" -> info,
                "NotebookMetaData" -> meta
            }]
        ]&
    ]

buildDocNotebook // ClearAll
buildDocNotebook // Options = {
    "WebURL" -> None,
    "NonSplittableCamelCase" -> Automatic,
    "ConvertGraphicsToBitmaps" -> True,
    "ExcludedDocFlagsPatt" ->
        "ExcisedFlag" | "FutureFlag" | "InternalFlag" | "TemporaryFlag"
};
buildDocNotebook[opts : OptionsPattern[]] :=
    With[
        {
            optsSeq = Sequence @@ Flatten@{opts},
            exlFlags = nodefQuiet@OptionValue@"ExcludedDocFlagsPatt",
            webUrlWrapper = withCustomWebURL@nodefQuiet@OptionValue@"WebURL",
            camelWrapper = withCustomNonSplittableCamelCase@
                nodefQuiet@OptionValue@"NonSplittableCamelCase",
            bitmapsWrapper =
                If[nodefQuiet@OptionValue@"ConvertGraphicsToBitmaps",
                    Identity
                (* else *),
                    withNoGraphicsToBitmapsConversion
                ]
        },
        If[#NotebookInformation === $Failed,
            #
        (* else *),
            Append[#,
                If[MatchQ[OptionValue[#NotebookInformation, "Flag"], exlFlags],
                    "Notebook" -> None
                (* else *),
                    With[
                        {nb = bitmapsWrapper@camelWrapper@webUrlWrapper@
                            MakeNotebook[
                                #Notebook, #NotebookInformation,
                                "FilePath" -> #Path, optsSeq
                            ]
                        },
                        {
                            "Notebook" -> nb,
                            "NotebookMetaData" ->
                                GetSearchMetaDataList[nb, optsSeq]
                        }
                    ]
                ]
            ]
        ]&
    ]

Needs@"DocumentationSearch`"
indexDocNotebook // ClearAll
indexDocNotebook[indexer_DocumentationNotebookIndexer] := (
    If[Not@MatchQ[#Notebook, None | $Failed],
        AddDocumentationNotebook[
            indexer,
            Import[#Path, {"NB", "Plaintext"}],
            #NotebookMetaData,
            #Path
        ]
    ];
    #
)&

builtNBPath // ClearAll
builtNBPath@outputDir_String := FileNameJoin@{outputDir, #DocPath <> ".nb"}&

builtHTMLPath // ClearAll
builtHTMLPath@outputDir_String := FileNameJoin@{
    outputDir,
    FilePathToDocURI[$PathnameSeparator <> #DocPath, $PathnameSeparator] <>
        ".html"
}&

A basic build, that only builds documentation notebooks, might look as follows.

inputDir = "path/to/MyApplication/Documentation/English";
outputDir = "path/to/build/MyApplication/Documentation/English";
docBuildEnv["MyApplication", inputDir][
    cleanDir[outputDir, "*.nb"];
    FileNames["*.nb", inputDir, Infinity] // Map[
        toFileObject[] /*
        getDocPath[] /*
        getDocNotebook[] /*
        buildDocNotebook[] /*
        renameTo[builtNBPath@outputDir] /*
        exportUsing[Export[#Path, #Notebook]&] /*
        (Last@#History -> #Path &)
    ]
]

Incremental build, building notebooks, HTML pages, and search index, might look like this:

appName = "MyApplication";
inputDir = "path/to/MyApplication/Documentation/English";
buildDir = "path/to/build";

docBuildEnv[appName, inputDir]@Module[
    {nbOutputDir, indexDir, indexSpellDir, htmlOutputDir, indexer, result}
    ,
    nbOutputDir = FileNameJoin@{buildDir, appName, "Documentation", "English"};
    indexDir = FileNameJoin@{nbOutputDir, "Index"};
    indexSpellDir = FileNameJoin@{nbOutputDir, "SpellIndex"};
    htmlOutputDir = FileNameJoin@{buildDir, "HTML", "English"};

    cleanDir[{indexDir, indexSpellDir},
        {"_*.cfs", "segments_*", "segments.gen"}
    ];
    indexer =
        NewDocumentationNotebookIndexer[indexDir, "Language" -> "English"];

    result = FileNames["*.nb", inputDir, Infinity] // Map[
        toFileObject[] /*
        getDocPath[] /*
        ifNewerThan[builtNBPath@nbOutputDir,
            getDocNotebook[] /*
            buildDocNotebook[] /*
            renameTo[builtNBPath@nbOutputDir] /*
            exportUsing[Export[#Path, #Notebook]&]
        (* else *),
            renameTo[builtNBPath@nbOutputDir] /*
            getDocNotebook[False]
        ] /*
        indexDocNotebook[indexer] /*
        ifNewerThan[builtHTMLPath@htmlOutputDir,
            renameTo[builtHTMLPath@htmlOutputDir] /*
            exportUsing[
                withFixedHTMLExportLinks[appName]@ExportWebPage[
                    #Path, #Notebook, #NotebookInformation,
                    "CompleteHTMLQ" -> True
                ]&
            ]
        (* else *),
            renameTo[builtHTMLPath@htmlOutputDir]
        ] /*
        (Flatten@{#History, #Path}&)
    ];

    CloseDocumentationNotebookIndexer[indexer];
    CreateSpellIndex[indexDir, indexSpellDir];

    result
]

If needed, it should be easy to inject some additional processing between build steps.

Continued in second answer.

Putting together all pieces from first answer, for general, commonest case, gives final buildDocumentation function:

nonStrOptRule // ClearAll
nonStrOptRule // Attributes = HoldAll;
nonStrOptRule[msg_MessageName, optName_, onError_] :=
    wrong : "" | Except@_String :> (
        Message[msg, HoldForm@optName, HoldForm@wrong];
        onError
    )

useProcessDirectory // ClearAll
useProcessDirectory // Attributes = HoldFirst;
useProcessDirectory[_, appDir_, Automatic] := withDirectory@appDir
useProcessDirectory[_, _, procDir_ : (_String?DirectoryQ | Inherited)] :=
    withDirectory@procDir
useProcessDirectory[onError_, _, procDir_] := (
    Message[buildDocumentation::optAutInhDir,
        HoldForm@"ProcessDirectory", HoldForm@procDir
    ];
    onError
)

buildDocumentation // ClearAll
buildDocumentation::optStr = "\
Value of option `1` -> `2` should be a non-empty string.";
buildDocumentation::optAutStr = "\
Value of option `1` -> `2` should be Automatic, or a non-empty string.";
buildDocumentation::optAutInhDir = "\
Value of option `1` -> `2` should be Automatic, Inherited, \
or a non-empty string with valid path to a directory.";
buildDocumentation // Options = {
    "ApplicationName" -> Automatic,
    "SourceDirectory" -> "Documentation",
    "BuildDirectory" -> FileNameJoin@{"..", "build"},
    "NBOutputDirectory" -> Automatic,
    "HTMLOutputDirectory" -> Automatic,
    "ProcessDirectory" -> Automatic,
    "Incremental" -> True,
    "BuildIndex" -> True,
    "NBBuilder" -> Function[{nbOutputDir, fullOpts},
        getDocNotebook[fullOpts] /*
        buildDocNotebook[fullOpts] /*
        renameTo[builtNBPath@nbOutputDir] /*
        exportUsing[Export[#Path, #Notebook]&]
    ],
    "HTMLExporter" -> Function[{appName, htmlOutputDir, fullOpts},
        renameTo[builtHTMLPath@htmlOutputDir] /*
        exportUsing[withFixedHTMLExportLinks[appName]@ExportWebPage[
            #Path, #Notebook, #NotebookInformation, Sequence @@ fullOpts
        ]&]
    ],
    "CompleteHTMLQ" -> True,
    "WebAssets" :> {
        {
            FileNameJoin@{$DocumentationBuildDirectory,
                "Internal", "web", "html", "2014", "standard"
            },
            "",
            {
                "*.js", "*.json", "*.css", "*.png", "*.gif", "*.jpg",
                "*.svg", "*.eot", "*.otf", "*.ttf", "*.woff"
            }
        },
        {
            FileNameJoin@{$DocumentationBuildDirectory,
                "Internal", "web", "html", "2014", "minimal", "javascript"
            },
            "javascript",
            "faster-page-load.js"
        },
        {
            FileNameJoin@{$DocumentationBuildDirectory,
                "Internal", "web", "html", "images", "mathematicaImages"
            },
            FileNameJoin@{"images", "mathematicaImages"},
            "bullet.gif"
        }
    },
    "DeletableFileNames" -> {
        "*.nb",
        "_*.cfs", "segments_*", "segments.gen",
        "*.html", "*.txt", "*.js", "*.json", "*.css", "*.png", "*.gif", "*.jpg",
        "*.svg", "*.eot", "*.otf", "*.ttf", "*.woff"
    }
};
buildDocumentation[appDir_String?DirectoryQ, opts : OptionsPattern[]] :=
    With[
        {
            appName = Replace[nodefQuiet@OptionValue@"ApplicationName", {
                Automatic :> FileNameTake@appDir,
                nonStrOptRule[buildDocumentation::optAutStr,
                    "ApplicationName", Return[$Failed, With]
                ]
            }],
            srcDir = useProcessDirectory[
                Return[$Failed, With],
                appDir,
                nodefQuiet@OptionValue@"ProcessDirectory"
            ]@ExpandFileName@Replace[nodefQuiet@OptionValue@"SourceDirectory",
                nonStrOptRule[buildDocumentation::optStr,
                    "SourceDirectory", Return[$Failed, With]
                ]
            ]
        },
        (* Call buildDocumentation for all language specific
           documentation directories. *)
        Cases[FileNames[Keys@$languages, srcDir],
            inputDir_?DirectoryQ :> With[{lang = FileNameTake@inputDir},
                lang -> buildDocumentation[
                    appDir, inputDir, appName, lang, opts
                ]
            ]
        ]
    ]
buildDocumentation[
    appDir_String?DirectoryQ, inputDir_String?DirectoryQ,
    appName : Except["", _String], lang:(_String?validLangQ):"English",
    opts : OptionsPattern[]
] :=
    Module[
        {
            buildDir, nbOutputDir, htmlOutputDir, incremental, buildIndex,
            completeHTMLQ, fullOpts, nbBuilder, htmlExporter, results, tag
        },
        useProcessDirectory[
            Return[$Failed, Module], appDir,
            nodefQuiet@OptionValue@"ProcessDirectory"
        ]@docBuildEnv[
            appName, inputDir, lang, FilterRules[{opts}, Options@docBuildEnv]
        ]@Catch[
            buildDir = ExpandFileName@Replace[
                nodefQuiet@OptionValue@"BuildDirectory",
                nonStrOptRule[buildDocumentation::optStr,
                    "BuildDirectory", Throw[$Failed, tag]
                ]
            ];
            nbOutputDir = ExpandFileName@Replace[
                nodefQuiet@OptionValue@"NBOutputDirectory", {
                Automatic :> FileNameJoin@
                    {buildDir, appName, "Documentation", lang},
                nonStrOptRule[buildDocumentation::optAutStr,
                    "NBOutputDirectory", Throw[$Failed, tag]
                ]
            }];
            htmlOutputDir = ExpandFileName@Replace[
                nodefQuiet@OptionValue@"HTMLOutputDirectory", {
                Automatic :> FileNameJoin@{buildDir, appName <> "-HTML", lang},
                nonStrOptRule[buildDocumentation::optAutStr,
                    "HTMLOutputDirectory", Throw[$Failed, tag]
                ]
            }];
            {incremental, buildIndex, completeHTMLQ} = MapThread[
                If[BooleanQ@#2,
                    #2
                (* else *),
                    Message[buildDocumentation::opttf,
                        HoldForm@#1, HoldForm@#2
                    ];
                    Throw[$Failed, tag]
                ]&
                ,
                {#, nodefQuiet@OptionValue@#}&@
                    {"Incremental", "BuildIndex", "CompleteHTMLQ"}
            ];
            fullOpts = Flatten@
                {"Language" -> lang, opts, Options@buildDocumentation};
            nbBuilder = nodefQuiet[OptionValue@"NBBuilder"][
                nbOutputDir, fullOpts
            ];
            htmlExporter = nodefQuiet[OptionValue@"HTMLExporter"][
                appName, htmlOutputDir, fullOpts
            ];

            If[Not@incremental,
                (* Clean all output directories. *)
                Scan[
                    throwOnFailed[tag]@cleanDir[#,
                        nodefQuiet@OptionValue@"DeletableFileNames"
                    ]&,
                    Union@{buildDir, nbOutputDir, htmlOutputDir}
                ]
            ];

            results = Select[
                notebookFileNames[inputDir, Sequence @@ fullOpts],
                StringMatchQ["*.nb"]
            ] // Map[
                toFileObject[] /*
                getDocPath[inputDir] /*
                If[incremental,
                    ifNewerThan[builtNBPath@nbOutputDir,
                        nbBuilder
                    (* else *),
                        renameTo[builtNBPath@nbOutputDir]
                    ] /*
                    ifNewerThan[builtHTMLPath@htmlOutputDir,
                        functionalIf[Not @* KeyExistsQ["Notebook"],
                            getDocNotebook[False, fullOpts]
                        ] /*
                        htmlExporter
                    (* else *),
                        renameTo[builtHTMLPath@htmlOutputDir]
                    ]
                (* else *),
                    nbBuilder /*
                    htmlExporter
                ] /*
                (Flatten@{#History, #Path}&)
            ];

            If[buildIndex,
                DocumentationBuild`FunctionPaclets`Private`indexNotebooks[
                    Cases[results, {_, builtNB_String, ___} :> builtNB],
                    nbOutputDir,
                    lang
                ]
            ];

            If[completeHTMLQ,
                Cases[nodefQuiet@OptionValue@"WebAssets",
                    {fromDir_String, toDir_String, rest___} :>
                        copyFiles[
                            fromDir,
                            FileNameJoin@{htmlOutputDir, toDir},
                            rest
                        ],
                    {0, 1}
                ]
            ];

            results
            ,
            tag
        ]
    ]

Building documentation for MyApplication application, with standard directory structure, can be performed in following way:

buildDocumentation["path/to/MyApplication"]

It will build all documentation source notebooks from path/to/MyApplication/Documentation/<language> directories and save result together with search index in path/to/build/MyApplication/Documentation directory and HTML version together with web assets in path/to/build/MyApplication-HTML.