What is the recommended workflow using Liquibase and Git?
TL;DR: Add /path/to/changelogfile.xml merge=union into .gitattributes file.
I've seen some discussions on having separate files. There are two ways of having separate files in liquibase:
- Use the
includeAlltag and let keep pushing new files to the folder. - Use the
includetag for each separate file in the master file to dictate the order.
Both ways will still cause problems: (1) includeAll will just pick files on lexicographic order of file names, which works fine if changelog is executed every time there a new change is submitted, but this will fail if you want to deploy to a brand new file. Unless you have a naming strategy to dictate the order. But this will still be a problem in case of Michael and Jacob working at the same time: they will probably use the same name, which results in conflicts. (2) Both Michael and Jacob need to edit the last few lines of the master file. In other words: merge conflict!
The actual solution is not to change your liquibase strategy. Just keep the same strategy you've been following so far. The root of the problem here is the source control and merging mechanism. A liquibase changelog file is essentially a history. Think of it as a release notes file where developers just keep adding their work to the last line of the file. In that case, doesn't matter if two developers add two lines add the end, both should go regardless of the order. So, the solution is to create a .gitattributes file in the root folder of your git project (if you don't already have one) and add the following line to it:
/path/to/changelogfile.xml merge=union
This is still not fail-prof. To address your advanced example scenario, I can't think of any automatic strategy or workflow. The developers should communicate when they are touching the same table (or same db objects). This scenario is not only a "file" conflict, but also a semantic conflict. That's when a good CI comes handy. The developer who commits and merge the code last will get the error, and needs to get in touch with the one who won the race at first.
Note, maybe your Git portal does not support merge strategy and you still get merge conflicts on pull requests (aka merge requests). But still, when you see that and try to merge back or rebase to resolve conflicts, the conflict is already solved. I know GitLab does support it. Not sure about Github or BitBucket.
At my company, the way we use liquibase prevents these situations from occurring. Basically, you create a separate liquibase file for each change. We name the files after the JIRA ticket that originated the change with a little descriptive text. Each of these files, we put in a folder for the version of the system they are for; if the next release is 1.22 then that folder is created when we start making database changes and we put each liquibase file in there along with an update.xml script that just includes them. That update.xml file winds up being the only place where conflicts can really happen, and they're trivial to resolve.
To illustrate, this is the src/main/liquibase folder:
├── install
│ ├── projectauthor.xml
│ ├── project_obspriorities.xml
│ ├── project_priorities.xml
│ ├── project_udv.xml
│ ├── project.xml
│ ├── roles.xml
│ ├── scan.xml
│ ├── (the other table definitions in the system go here)
│
├── install.xml <-- this reads all the files in ./install
│
├── local.properties <--
├── prod.properties <-- these are database credentials (boo, hiss)
├── staging.properties <--
├── test.properties <--
│
├── update.xml <-- reads each version/master.xml file
│
├── v1.16
│ ├── 2013-06-06_EVL-2240.xml
│ ├── 2013-07-01_EVL-2286-remove-invalid-name-characters.xml
│ ├── 2013-07-02_defer-coauthor-projectauthor-unique-constraint.xml
│ └── master.xml
├── v1.17
│ ├── 2013-07-19_EVL-2295.xml
│ ├── 2013-09-11_EVL-2370_otf-mosaicking.xml
│ └── master.xml
├── v1.18
│ ├── 2014-05-05_EVL-2326-remove-prerequisite-construct.xml
│ ├── 2014-06-03_EVL-2750_fix-p-band-polarizations.xml
│ └── master.xml
The install.xml file is just a bunch of file inclusions:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">
<include file="src/main/liquibase/project/install/projectauthor.xml"/>
<include file="src/main/liquibase/project/install/project_obspriorities.xml"/>
...
</databaseChangeLog>
The update.xml file is the same story:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">
<include file="src/main/liquibase/project/v1.18/master.xml"/>
</databaseChangeLog>
The one aspect of the workflow I am not in love with is that the install/*.xml are supposed to create the database as it is right before the current version, but we usually don't remember to do that.
Anyway, this approach will save you from a lot of grief with merging. We're using Subversion and not having any merge difficulties with this approach.
There are always edge cases that need to be manually handled, but they generally happen very infrequently. Git generally handles the merging of changes at the text level just fine, so the merged file will have both changeSets in it, one after the other.
Since liquibase tracks changeSets by id/author/filename, the fact that Jacob's changeSet happens to end up before Michaels' in the final changeSet doesn't matter. When both devs run the final changeSet, Liquibase will run the other dev's changeSet only because theirs has been marked as ran but the other has not. For all other environments, both changeSets will run.
Your advanced case run into problems because both developers are making changes that are contradictory to each other. You could also run into similar problems if both developers drop a column, or add a new column with the same name. It's also not always simply one developer vs. another, sometimes conflicting changeSets come from two separate feature branches being merged in. There is no problem physically with the merged changeSet itself, the problem is that the new changelog is not logically correct. It's not really a git problem, it's a logic problem.
In practice, this type conflict happens rarely because different developers and different branches are usually working on separate areas of the codebase and when there is potential for conflict, they handle it through communication and planning.
If you do run into a conflict, there are several ways to resolve it. Usually that is handled (like in your example) by deleting incorrect or duplicate changeSets but can also be handled by creating a brand new changeSet that is a combination of both. In either case, you need to handle databases that have ran the "wrong" changeSet. How to best handle that depends on how many systems have ran it.
If it is a single developer, it is sometimes easiest to simply run liquibase changeLogSync to mark the new changeSet as ran and manually make the change in the database. If the bad changeSet was ran lately, they could even run liquibase rollbackCount X to revert their bad change and then remove the changeSet and then liquibase update
If there were multiple conflicts and/or multiple systems that have ran problem changeSets, the easiest approach is usually to use <preConditions onFail="MARK_RAN"><changeSetExecuted id=....></preConditions> tags. You can remove the bad changeSet and add a new changeSet that only runs if the old changeSet was executed and puts the database back in the state expected by later changeSets. In your example, it would rename first_name back to name so that the name to last_name changeSet works just fine.