Two SFDX project folder structure questions

Before answering Keith's specific questions, I'd like to set the stage by describing the fundamentals of Salesforce DX "Projects", "Package Directories", and the "Default Package Directory".

IMPORTANT: If you don't need (or want) to read the "fundamentals" info, just scroll down to the bottom third of this answer. I've got Keith's questions blockquoted down there with the answers just below them.

What is a Salesforce DX "Project"?

In general, a Salesforce DX Project is a new local file structure that collects your org's metadata (code and config), org templates, sample data, and tests. The project root is often the repository root of a version control system (VCS) as well.

Specifically, an SFDX Project exists when you have a local directory that contains the following:

1. A project configuration file. This file is always named sfdx-project.json and the directory where it's located becomes the "root" of that Salesforce DX project.
2. One or more "Salesforce DX Package Directories", which contain SFDX source. These directories live within the SFDX project root, contain SFDX source, and must be explicitly declared as "Package Directories" by adding their path(s) to the packageDirectories array inside the project's sfdx-project.json file.
3. A hidden directory named .sfdx where the Salesforce CLI keeps a variety of files and directories that support the internal operation of the CLI for that specific Salesforce DX project.

There are a couple more files that you get after running the sfdx force:project:create command, but the above list describes the minimum set of files and directories that are required to have a functional Salesforce DX project.

What is a Salesforce DX "Package Directory"?

The Salesforce CLI works by scanning all of the Package Directories declared in sfdx-project.json for locally added or modified SFDX source metadata. It attempts to synchronize that source with any scratch orgs you point the CLI at with sfdx force:source:push or sfdx force:source:pull.

Any metadata (Custom Objects, Apex Code, Profiles, etc.) created outside of your SFDX project (like in a scratch org) will be new to the CLI's internal map of your project's metadata. When this happens, the CLI needs to have a place to put newly discovered metadata. This is where the Default Package Directory comes in.

What is the "Default Package Directory" and why is it special?

Defined as part of a project's sfdx-project.json file, the Default Package Directory is the CLI's go-to location for storing "Remotely Added" metadata.

For example, if you add a new Custom Object named MyObject__c to your scratch org using the Setup UI and then run sfdx force:source:pull, the CLI is going to save the SFDX source for your new object locally in your Default Package Directory.

The path to the SFDX source for MyObject__c will look something like this:

sfdx-project-dir
└─ sfdx-package-dir
   └─ main
      └─ default
         └─ objects
            └─ MyObject__c
               ├─ fields
               └─ MyObject__c.object-meta.xml

The names for sfdx-project-dir and sfdx-package-dir will be different for your project, but everything else would look exactly like this. The Salesforce CLI will always use the path main/default/<metadata-type> inside of your default package directory when storing remotely added metadata. This also happens when you use the sfdx force:mdapi:convert command to convert MDAPI source to SFDX source.

Without a consistent, known location for the CLI to put remotely added SFDX source, developers would need to pre-define locations for all metadata types. Even if a developer thought of "everything" and defined complex rules for what metadata goes where on a force:source:pull, you'd still need a generic default just in case they missed something.

In other words, having a clean, simple, consistent default location for remotely added metadata is a feature, not a bug. :-)

And now, time to address Keith's original questions (finally!)

Now that we've covered the fundamentals of SFDX Projects and Package Directories, I'll tackle each of Keith's original questions, one at a time.

Question One: Two levels of folder (e.g. "main" and "default") seem redundant. Why have the "default" level?

I look at main as a module inside of your Package Directory. This is where the core set of your org customizations would go if you're a customer. If you're building a managed package, main is where the "shared" code that your app's features depend on lives.

The SFDX-Falcon Template uses this concept of "core" and "feature" metadata quite extensively. The idea is that code/metadata that you put in a feature module can depend on what's in the main module, but not the other way around.

If you're an ISV, logical separation of code/metadata like this doesn't just make it easier to understand and organize your packaged code. It's also a fantastic way to get ready for Second-Generation Packaging (Packaging 2).

So, what about the "redundant" default directory inside of main?

I don't think it's fair to call the default directory redundant. It's literally the default location for the CLI to put remotely added metadata. You might implement your own code organization scheme inside of main, but the CLI is guaranteed to get a known, consistent location to put any new source because it's not interfering with anything "non-default" you might be doing.

Finally, if you're wondering why/how you might further organize SFDX source inside of your main module, one suggestion is to implement a design pattern based on "separation of concerns". Here's how SFDX-Falcon approaches this:

Source: Salesforce DX 201 - Advanced Implementation for ISVs (Slide 33)

Question Two: I started out thinking that (for a largish project) I should have > more than one package directory e.g. "force-lts" as well as "force-app" but in sfdx-project.json only one can be nominated as the default which is where sfdx force:source:pull pulls changes to. So I'm now thinking it would be better to just have multiple folders inside the single "force-app" package directory folder that is marked as or is implicitly the default so as to avoid a pull dumping files in the wrong directory. What folder structure works best?

The best answer here depends on whether you're an ISV Partner building a managed package or a customer working on customizations to your production org.

If you're a Customer

If you're a customer with a complex, convoluted, monolithic codebase (aka the "Happy Soup") it can be helpful to try to start breaking things up into many small, independent unmanaged packages wherever possible. Better yet, take a look at Developer Controlled Packages (DCPs) which are part of the Packaging 2 Beta in Spring '18.

Either way, you're using separate SFDX Package Directories with the expectation that you'll want to deploy packages independently of each other as part of a regular agile delivery process. Build fast, build small, deploy often.

If you're an ISV Partner

If you're an ISV building a first-generation managed package, all of your metadata is already in a single package (unless you're using extension packages). When you deploy updates to your packaging org, you're typically going to send everything at once.

The CLI makes it easy to do this because sfdx force:source:convert operates on only one package directory during the conversion from SFDX to MDAPI source. This makes it easier for ISVs to focus on whole-package deployments when they are ready to push code to their packaging org.

The good news is that you can easily organize source by module inside a single Package Directory anyway. All your core code goes into main, as I described above. For the rest of your app (ie. your features), you add one or more feature module directories and segment your code, similar to the following.

Source: Salesforce DX 201 - Advanced Implementation for ISVs (Slide 34)

This is essentially what Keith C is suggesting when he says "it would be better to just have multiple folders inside the single 'force-app' package directory folder". All that SFDX-Falcon does is formalize a basic template for how one might set up those additional folders.

There is one inconvenience that comes up when organizing your SFDX source, however. The fact that doing an sfdx force:source:pull on remotely added metadata results in the CLI creating SFDX source files in <default-package-dir>/main/default/.

I personally don't feel like this is a dealbreaker when it comes to choosing to organize SFDX source outside of main/default. Yes, it's inconvenient to move the source files when you do it but the benefits of having well organized (and hopefully well segmented) code is worth the investment. In fact, the bigger and more complex your code base is, the more you'll benefit over time by investing in getting things organized now.

Conclusion

Salesforce DX brings us new flexibility when organizing the metadata we use to build apps or customize our production orgs. Understanding how the Salesforce CLI uses SFDX Projects and Package Directories to synchronize metadata with scratch orgs can help you get the most out of your Salesforce DX project.

The use of default/main helps the CLI know where to save new metadata. Organizing your own customizations into modules that are siblings to main inside of your default Package Directory can make managing large projects easier. Community templates like SFDX-Falcon provide a model for how you can make this happen in your own project, and the investments made now can pay dividends later by having a codebase that's easier to understand and (hopefully!) update.

Based on the webinar mentioned in the comment, breaking the components out into multiple folders that have meaningful names is the way to go. For example, they suggest using the managed package namespace prefix as one of the root folders in the nominated package directory.

Didn't see any explicit mention of single or multiple package directories: probably one is assumed.

PS

The work discussed in the webinar is this https://github.com/sfdx-isv/sfdx-falcon-template.

PPS

Be aware of this current problem How to use "sfdx force:source:pull" with folders other than "main/default" where components are added in the scratch org?.