Remote page

[johnnychen94]

Let's take JuliaImages as an example.

Images.jl in JuliaImages consists of different packages, some of which (e.g., ImageFiltering) already have their own documentation. In the meantime, there is a centralized documentation in https://juliaimages.org. It is unclear whether a centralized version or a distributed version is good.

This proposal aims to collect the distributed documentation and rebuild it in a centralized way.

This solution includes two steps:

First, configure things in config.json. For example,

src
├── index.md
└── pkg
    ├── axes
    ├── config.json
    └── segmentation

with docs/src/pkg/config.json filled with following items:

{
    "remote":{
        "filter": [
            "git",
            "https://github.com/JuliaImages/ImageFiltering.jl.git",
            "docs/src"
        ],
        "binarization": "https://github.com/zygmuntszpak/ImageBinarization.jl.git/[docs/src]"
    }
}

Two valid syntaxes can be supported here:

• the first item is a little bit verbose, it says: use git protocol to clone https://github.com/JuliaImages/ImageFiltering.jl.git and collect the whole docs/src folder as one section filter in src/pkg.
• the second item is a more compact version for git, which follows the subdir syntax in Pkg.jl. i.e., pkg> add /tmp/testproject/dev/FooBar.git/[otherfolder/subfolder/Bar]

The type hierarchy for this is:

abstract AbstractRemotePath end

struct GitRemote{T} <: AbstractRemotePath
    path::String
    item::T
end

This type models how remote (git) repo is cloned into local folders during deployment.

Remote folders are not cloned directly into docs/src/pkg folder because that's not friendly to folder arrangement. Instead, all remote repos are cloned into .democards folder. This also enables repo sharing.

Because generally, DemoCards does not allow indirect paths. Not cloneing remote folders into docs/src/pkg requires a new type LocalRemote to break such requirement and models runtime folder link/move/copy.

---

The workflow is:

1. when DemoPage("src/pkg") notices the existence of remote pages/sections, it will immediately clone those repo into docs/.democards folder, and then generate a LocalRemote placeholder for it.
2. generate(::LocalRemote) would copy the contents at runtime, doing normal generation, and deleting the copied contents to restore the original folder structure.

[timholy]

So the idea here is that we'd continue to have both, but they would be identical? If we do keep both then having them be the same seems ideal.

However, I'd also be quite happy with the idea of only hosting docs at juliaimages. That said, it would probably be nicer to maintain the docs in each package's own repo, and maybe that's what you're really driving at here. Bottom line: 👍

[johnnychen94]

Yes, that's pretty much the case. What I have in mind is to develop the package-level documentation at the repo of each packages, and then collect them together and build the centralized juliaimages. The separation of code-dev and docs-dev does not look right to me.

With this done, we could, of course, only host one juliaimages doc. But since what's needed for package-level documentation setup is only docs CI, we could definitely provide both; the only reason I can foresee from not providing package-level documentation is gh-pages might take a lot of storage and thus makes clone/pull slower. That said, I'm not sure if multiple docs are really needed, though.

This issue only tries to make this operation and maintenance as flexible and easy as it can, whether package-level documentation is needed can be decided in other issues.

My previous words are not very illustrative. Let me try to explain it again with a concrete example:

It's worth noting that, this remote feature is not a site-level solution; it only links one DemoPage/DemoSection/DemoCard for each remote configuration.

This is the folder structure in ImageFiltering (after JuliaImages/ImageFiltering.jl#185):

docs
├── Manifest.toml
├── Project.toml
├── demos
│   ├── config.json
│   └── filters
├── make.jl
└── src
    ├── democards
    ├── filterintro.png
    ├── function_reference.md
    └── index.md

This is the folder structure in JuliaImages.github.io :

docs/
├── examples
│   ├── color_channels
│   ├── config.json
│   ├── contours
│   ├── image_quality_and_benchmarks
│   ├── index.md
│   └── spatial_transformation
├── make.jl
└── src
    ├── api_comparison.md
    ├── assets
    ├── function_reference.md
    ├── index.md
    ├── install.md
    └── tutorials

Both documentation defines a DemoPage: docs/demos in ImageFiltering and docs/examples in juliaimages.github.io.

This design enables us to break the documentation of ImageFiltering, and then re-compose it in juliaimages.github.io. Say we only want to copy the demos from ImageFiltering to juliaimages.github.io, what's needed is to modify docs/examples/config.json in juliaimages.github.io:

`docs/examples/config.json` in juliaimages.github.io:
{
-    "theme": "grid"
+    "theme": "grid",
+    "remote": {
+        "filters": "https://github.com/JuliaImages/ImageFiltering.jl.git/[docs/demos]"
+    }
}

so that only the demos Page from ImageFiltering is used in juliaimages.github.io. None of the others will be used.

If we also need to remote-sync the index page of ImageFiltering.jl, what's needed is:

• build pkgs part with DemoCards (The contents are placed in docs/pkgs [WIP] build pkgs documentation with DemoCards JuliaImages/juliaimages.github.io#159)
• build index page of ImageFiltering.jl with DemoCards: (not yet done, let's assume the contents are placed in docs/index)
• configure docs/pkgs/config.json in juliaimages.github.io:

docs/pkgs/config.json in juliaimages.github.io
{
+    "remote":{
+        "ImageFiltering": "https://github.com/JuliaImages/ImageFiltering.jl.git/[docs/index]"
+    },
     "order":[
         "introduction",
         "axes",
         "metadata",
         "features",
+        "ImageFiltering",
         "segmentation",
         "transformations"
     ]
}

This way we break the ImageFiltering documentation into two parts, and re-compose them in the "pkgs" and "examples" page in juliaimages.github.io, respectively.

[timholy]

With this done, we could, of course, only host one juliaimages doc. But since what's needed for package-level documentation setup is only docs CI, we could definitely provide both; the only reason I can foresee from not providing package-level documentation is gh-pages might take a lot of storage and thus makes clone/pull slower. That said, I'm not sure if multiple docs are really needed, though.

I'm just not sure I'd see the point of having it hosted in two places. After all, many of us get to the docs via the badge; couldn't the badge just point to a page that's hosted in the overall juliaimages docs? We could have some standard headline, "Documentation for this package is integrated into the overall JuliaImages documentation: "

If there are two separate hosted docs, I could imagine one of them having build failures and the other not, thus getting desynchronized. Or even just someone spending time looking at both wondering if there are any differences.

[johnnychen94]

Good point, I agree with you on this.