I think that ideally, the application file format would internally be a directory but I want the directory to be hidden from the user.
Ideally there would be something like cross platform macOS bundles (https://en.wikipedia.org/wiki/Bundle_(macOS))
Sketch does this by zipping up the directory but I would like to avoid zipping things.
I don't want to use sqlite.
PDF is an interesting case study, so are the old Microsoft Office formats (e.g. a "doc" file)
The new Microsoft Office (e.g "docx") format does what almost anybody would do today which is pack everything into a ZIP file, they just don't put a ZIP extension in it.
It is not hard at all to treat a ZIP file as a filesystem or create it dynamically (look at the stdlib ZIP support in Python or Java) so you don't need to "zip up a directory". In fact the ZIP file format was designed to be editable in place, with the slight awkwardness that you have to rewrite the directory at the end of the file if you extend the file.
You can definitely make a "hole" at the end of the file that you can write new items at the beginning of and write new directory entries at the end of. You can "delete" a file by trashing its directory entries and at some point "garbage collect" the file removing all the holes.
Why not? If you read this page here:
https://sqlite.org/appfileformat.html
You'll find that your requirement of "internally be a directory" can be done using Sqlite with this one, single, simple table:
CREATE TABLE files(filename TEXT PRIMARY KEY, content BLOB);
I.e., you do not have to create a traditional relational DB schema if you do not want to do so. Of course, if you later do want to add additional relational data, if you are already using Sqlite then doing so becomes trivial.
Plus you mention "cross platform macOS bundles", Sqlite provides cross platform compatibility automatically (even to Linux or Windows platforms).
And, presumably, your focus is on the desktop app rather than a file format. Using Sqlite then removes your need to distract your focus to a file format.