Blog post proposal: "Modern Julia workflows"

[gdalle]

EDIT: the blog post is being created!

• Preview: https://modernjuliaworkflows.github.io/
• Repo: https://github.com/modernjuliaworkflows/modernjuliaworkflows.github.io
• Progress: https://github.com/modernjuliaworkflows/modernjuliaworkflows.github.io/issues

---

Is your feature request related to a problem? Please describe.

AFAIK, the typical tools and packages a Julia user needs daily are not documented in a single place.

• In the documentation, the section on "workflow tips" only mentions Revise and the REPL (https://docs.julialang.org/en/v1/manual/workflow-tips/)
• On the front page, the section "Essential tools" is very short and quite opinionated, with no descriptions (https://julialang.org/)

Describe the solution you'd like

A lengthy blog post detailing the typical workflow for using, developing and testing packages. See structure proposal below.

Describe alternatives you've considered

• Adding things to the documentation (but where?)
• Redesign the front page (but how long would it take?)

Additional context

Related issues:

• Frontpage re-design #800

Possible contributors:

• @jacobusmmsmit

[gdalle]

Possible structure of the post:

Step 1: writing code

Installation

• official downloads
• juliaup

Code loading

• startup file
• Revise.jl

Development environments

• VSCode / VSCodium
• emacs, vim
• Jupyter
• Pluto.jl

Package management

• Pkg.jl
• stacking environments
• VSCode extension

Debugging

• Debugger.jl
• VSCode extension

Esthetics

• Term.jl
• OhMyREPL.jl

Calling other languages

• PythonCall.jl
• RCall.jl
• JuliaInterOp

Experiments

• ProgressMeter.jl
• DrWatson.jl

Step 2: sharing code

Setup

• Git(Hub)
• PkgTemplates.jl

Formatting

• style guides (BlueStyle, SciMLStyle)
• JuliaFormatter.jl

Testing

• unit tests
• Aqua.jl
• TestEnv.jl
• VSCode extension (TestItemRunner.jl)

Documentation

• docstrings
• Documenter.jl
• LiveServer.jl
• Literate.jl
• Quarto

Compatibility

• package extensions
• Requires.jl
• PkgCompatUI.jl

Publishing

• Discourse post
• general registry
• LocalRegistry.jl

Step 3: speeding up code

Speed measurements

• BenchmarkTools.jl

Profiling

• built-in (time + memory)
• VSCode extension
• ProfileView.jl / ProfileSVG.jl

Type stability

• JET.jl
• Cthulhu.jl

Precompilation

• PrecompileTools.jl
• PackageCompiler.jl

Parallelism?

[gdalle]

What would you add?

[fredrikekre]

LiveServer.jl, in particular the live-building of documentation, is a game changer for writing documentation. The instant reward of seeing the rendered page is making it so much more fun.

[albheim]

Something I've found really useful since I first found it was project environments and global environments, and how they stack. So the global environment can contain tools that are commonly useful but you might not always want to add to a project env, and can be seen a bit like an extended stdlib with what you personally think should always be available :)

I think this is a great feature and use it often both as a user and package developer.

[nilshg]

What about PackageCompiler? It should probably be mentioned for users (eliminate ttfx) and for developers (deploying apps)

[jacobusmmsmit]

Thanks for opening this discussion, Guillaume, I think it's an important collection of information that we can point newer users to! I really believe that centralising information like this is one of the best things that we can do as a community. One of the biggest problem we want to avoid is the feeling that there are some Julia wizards that know these magical incantations to make the language 100x better but the knowledge is stored in their head or their cryptic documentation.

I do believe that the post shouldn't be split up into three levels. We want to present a curated list of workflow solutions to common problems or questions, as in my video I think it's better to present all of the solutions together and qualify them with how powerful or useful they are to whom and when to use them. Also, Julia really blurs the line between scripter, package developer, and optimiser such that I don't think we should hint that they are separate things.

I do like how each package is part of a subheading talking about a specific topic, but I think some of the most important (commonly used according to my survey) packages are under headings that most people wouldn't think to read unless they were more experienced with Julia. For example, not knowing about JET and Cthulhu would be a real shame! Perhaps we should present certain tools as helping you write idiomatic Julia code as everyone wants to do that.

I think there's also a danger of presenting Julia as some sort of monolithic beast of packages that are required to be used to have a nice developer experience. This may be the impression that some newer folks take from such a post.

[gdalle]

LiveServer.jl, in particular the live-building of documentation, is a game changer for writing documentation.

Thanks, I just added it!

Something I've found really useful since I first found it was project environments and global environments, and how they stack.

Good point, also in the list now!

What about PackageCompiler? It should probably be mentioned for users (eliminate ttfx) and for developers (deploying apps)

Indeed, but I tried to structure the list by difficulty, and in my mind it is a rather advanced tool. Probably cause I don't use it myself 🤷

[nilshg]

I don't use it either, but it always felt like something that might actually be useful to me - 90% of my time on Julia is spent data wrangling and doing statistical modelling, simulation or optimization using the same 10 packages, so theoretically I think having a custom sysimage could be great but I could never be bothered to try it.

As far as I know it's one line of code to create a sysimage, maybe I should try it and see how practical it is before suggesting it 😂

[gdalle]

One of the biggest problem we want to avoid is the feeling that there are some Julia wizards that know these magical incantations to make the language 100x better but the knowledge is stored in their head or their cryptic documentation.

You read my mind.

I do believe that the post shouldn't be split up into three levels. [...] Julia really blurs the line between scripter, package developer, and optimiser such that I don't think we should hint that they are separate things.

That is a valid remark, and this user progression is something I definitely want to encourage. However, even with access to a blog post like this, it has to happen gradually. Beginners will need to master Revise and Pkg long before they even look at Cthulhu or PrecompileTools. And it seems a bit daunting to give them everything at once, especially with a very flat hierarchical structure (maybe we could find other natural headings?).

I think some of the most important (commonly used according to my survey) packages are under headings that most people wouldn't think to read unless they were more experienced with Julia. For example, not knowing about JET and Cthulhu would be a real shame!

I would actually love to know the results of your survey in terms of package use statistics, maybe you could share them here?
My putting JET and Cthulhu at the end is a subjective choice, mainly due to the order in which I discovered things myself. Both have greatly improved in usability recently (Cthulhu mapping directly to source code is a game changer), so there is a case to be made for mentioning them earlier. However, my guess is that Julia beginners take things in the following order:

1. Write code
2. Share code
3. Improve code

I'm open to being proven wrong, but if I'm not, then JET and Cthulhu belong in part 3

I think there's also a danger of presenting Julia as some sort of monolithic beast of packages that are required to be used to have a nice developer experience. This may be the impression that some newer folks take from such a post.

Agreed, when I wrote it down this morning I thought "boy that list is scary long".
That's part of why I think we need a sense of progression and increasing difficulty. I am also considering a series of 3 blog posts for that very reason.

[jacobusmmsmit]

As far as I know it's one line of code to create a sysimage, maybe I should try it and see how practical it is before suggesting it 😂

It's not hard persay, in my video I show how to use it. But it is very fickle: some things won't compile no matter how hard you try, and knowing about incremental sysimages is a game changer. There's also vscode's built in functionality for generating sysimages, but I don't like it so much as it both didn't work for me when I tried to use it and doesn't allow for much control nor understanding of how the process works (useful for debugging it).

I would actually love to know the results of your survey in terms of package use statistics, maybe you could share them here?

Here you go: https://discourse.julialang.org/t/survey-on-how-you-use-julia/99807/6

I did a small writeup. The part that surprised me was just how used LocalRegistry was, more so than JET, Debugger, and Cthulhu.

my guess is that Julia beginners take things in the following order

I think that's a reasonable way of putting it, I also like the framing of "Write, Share, Improve". It might be useful if we made it clear that the first section/post is to get you up and writing/running whatever code you want in a structured way, and the third is about making the code itself better.

The main reason I want JET (and to a lesser extent Cthulhu as I see it as more advanced) to be made more prominent is because many "senior" (for lack of a better word) members of the community have expressed a desire to see it adopted more or that it may one day be a more integrated part of the language.

By making this blog post we have a lot of influence over how people learn Julia and think about writing Julia code. I want the future of Julia to be statically checked by JET! I want compile times to go down because instabilities and piracy are caught!

[gdalle]

By making this blog post we have a lot of influence over how people learn Julia and think about writing Julia code. I want the future of Julia to be statically checked by JET! I want compile times to go down because instabilities and piracy are caught!

Maybe we could frame them as debugging tools instead of performance optimization. That way we mention them earlier and trick beginners into believing JET and Cthulhu are already the standard for tracking problems in your code. Then type-stable Julia becomes a self-realizing prophecy

[jacobusmmsmit]

Well, they kinda are debugging tools. Python has PEP8 checkers as a standard part of the workflow as well as things like isort. To me they come even before debugging tools, they should always be there in the background reminding you of small changes to your code to improve it.

I wish JET could be used as a passive static analyser/linter as opposed to having to be called actively to find errors. That's the part that holds it back imo.

[gdalle]

That makes sense, I use JET and Cthulhu much more than the standard debugger anyway (yes I'm a println("here") kinda guy)

[jacobusmmsmit]

Oh wait it can and is used in that way as a linter.

[gdalle]

True but you still need to run your functions with a macro