Protocol Buffer Package Name and File Name mapping

[xfbs]

Problem

What guide do we give for naming protocol buffer files? The overarching goals are:

• ensuring consistency (it is obvious how files are named without needing to read the code)
• preventing code breakage (allow flexibility in implementation without breaking consumers)
• make it easy to do the right thing (make it obvious, use tooling to detect invalid uses)

The concrete goals are:

1. From knowing a package name, it should be obvious which file needs to be imported
2. From knowing a file name, it should be obvious which package this file would declare
3. Allow flexibility in implementation (be able to split things into subfiles)

How Protocol Buffers work

A protocol buffer project consists of multiple files:

proto/file.proto
proto/other.proto
proto/subfolder/this.proto
proto/subfolder/that.proto

Every single file declares a package name, which might be like my_package.sub_package. When you import the file, any types defined in this file are then prefixed with the package path.

package my_package.sub_package;

message MyMessage {
    string field = 1;
}

In this example, the full name of the MyMessage type is my_package.sub_package.MyMessage.

Usually, when you use a protocol buffer type, you know the full path and you need to import it. To import it, you need to know which file it is defined in. By default, there is no requirement that the package name inside the file (my_package.sub_package) has anything to do with the file name. So you could have a file named proto/physics/temperature.rs which declares that it is the package named music_library.jazz.

What we want to achieve is having some kind of correspondence between file names and package names to fulfil goals 1 and 2.

Previous Ideas

We have had some ideas previously, with varying levels of success. The challenge is that we need to choose a path that allows people to keep using patterns that they are currently using.

Strict mapping

Previously, an idea I suggested is to just have a one-to-one correspondence between package names and file paths. This means that every package name can appear in only a single file, and all types must be defined in that file. For example:

Package	File
mypackage	mypackage/root.proto
mypackage.sub	mypackage/sub.proto
mypackage.sub.way	mypackage/sub/way.proto

Downsides

• This fulfills the goals (1) and (2)
• It does not fulfil goals (3): when you split up your protocol buffer files into sub-files, they need their own package names, which are part of the public API.

For example, if you want to split your mypackage.sub into multiple files, you could do this:

proto/mypackage/sub/this.proto
proto/mypackage/sub/that.proto

But these two files would not be able to be in the mypackage.sub package, due to their path, but would be in mypackage.sub.this and mypackage.sub.that respectively, which might not be what you want. Also, these "internal" modules would still be part of the public API.

Non-strict mapping

The next idea we have (and currently try to implement) is to have a non-strict mapping. This means that multiple files may declare the same package name.

Example:

proto
├── root.proto         -> package fancylib
└── sub
    ├── one.proto      -> package fancylib.sub
    └── two.proto      -> package fancylib.sub

Downsides

What makes this approach suboptimal is a few reasons:

• Doing it like this means there is not a simple mapping of package name to file name. The package name fancylib.sub is defined in both proto/sub/one.proto and proto/sub/two.proto. Which one do you import? You don't know, unless you read the code. This breaks goals (1) and (2)
• The other issue is when you rename a file: what you name your internal files should not matter (implementation detail). But because this file name is the one other users need to import, it is part of the public API. Therefore it is not possible to to rename files without breaking things. This breaks goal (3).

My Suggestion

I feel like I have come up with an approach that eliminates the downsides of both previous approaches, while preserving the upsides of both.

We have a strict mapping of package name to file name.

Package	File
mypackage	mypackage/root.proto
mypackage.sub	mypackage/sub.proto
mypackage.sub.way	mypackage/sub/way.proto

but patrick, what about splitting things into multiple files?

We allow for having project-internal "hidden" files. These are prefixed with an underscore!

im-test-lib
├── Proto.toml
└── proto
    ├── root.proto
    ├── _my_stuff.proto
    ├── mass.proto
    ├── mass
    │   └── _this.proto
    │   └── _that.proto
    ├── temperature.proto
    │   └── _kelvin.proto
    │   └── _fahrenheit.proto
    ├── units.proto
    └── vendor

The idea here is that we have a public API, whereby there is a simple mapping between package name and file name:

• im_test_lib in root.proto
• im_test_lib.mass in mass.proto
• im_test_lib.temperature in temperature.proto
• im_test_lib.units in units.proto

But additionally, there are also implementation details, which is files that start with an underscore character:

• mass/_this.proto, which is re-exported by mass.proto
• mass/_that.proto, which is re-exported by mass.proto
• temperature/_kelvin.proto, which is re-exported by temperature.proto
• temperature/_fahrenheit.proto, which is re-exported by temperature.proto

These should not be used externally, they should only be used by the current package to re-export types from into the proper file. The idea here is:

• Anything prefixed with an underscore is like mod xyz in Rust: private module (implementation detail, not usable from outside).
• We only lint the public API.
• If our package depends on other packages, we forbid from importing hidden files from other packages (to enforce that you do not use the non-public API).
• Buffrs projects are allowed at any point to reorganize their "hidden" files without it being a breaking API change, so long as the types exported by the public API does not change.

This accomplishes:

1. Freedom to split up files into multiple modules without breaking API, goal (3)
2. Simple mapping of package name to file name. No need to look into the sources to figure out which file exactly defined something. AKA: if you want mypackage.sub.ject, you know you need to import mypackage/sub/ject.proto, you do not have to look it up to find out that the specific type you need is defined in mypackage/sub/ject/somefile.proto. Goal (1) and (2).
3. Simpler for us to lint, too (we basically only need to lint the public API)

What do you think about this idea, @JoeHut and @mara-schulke?

[mara-schulke]

I agree that we need to talk about this. Sorry for the delay! Merging #167 for now – we can revisit this!

[xfbs]

Sounds good! This is not really a priority, so no worries 😄