Making rgl widget

[dmurdoch]

I'd like to make an rgl widget, but I'm having trouble getting started, because the examples I've seen start with a Javascript library, rather than starting with R code. Here's what I'm thinking of doing:

1. Put the rglClass definition that writeWebGL() creates into a Javascript file, and call it the Javascript library.
2. Add functions to it to support the kinds of modifications people would want to do when using an rgl widget. These would essentially replace the writeWebGL function.

Does this sound like the right approach? Are there any examples to work from that are similar?

Duncan Murdoch

[ramnathv]

@dmurdoch Great to know that you want to make an rgl widget. The approach you have indicated is broadly correct. To make things concrete, here are some suggestions.

1. Refactor writeWebGL separating the javascript and the R code. The js code essentially would be a large js function that accepts a json object as its argument. You can put this inside the renderValue method in the htmlwidgets/rgl.js folder.
2. The R code would then work to create the R object that then gets passed on to this javascript function.

You can use htmlwidgets::scaffoldWidget('rgl') to set up a scaffold of all the components required for this widget. I would be happy to help with this widget. So let me know if you have any questions or run into any sort of troubles.

[ramnathv]

The one challenge I see here is that writeWebGL seems to be writing directly to script tags. For example, I pulled this off one of the examples.

<!-- ****** spheres object 6 ****** -->
<script id="vshader6" type="x-shader/x-vertex">
 attribute vec3 aPos;
 attribute vec4 aCol;
 uniform mat4 mvMatrix;
 uniform mat4 prMatrix;
 varying vec4 vCol;
 varying vec4 vPosition;
 attribute vec3 aNorm;
 uniform mat4 normMatrix;
 varying vec3 vNormal;
 void main(void) {
   vPosition = mvMatrix * vec4(aPos, 1.);
   gl_Position = prMatrix * vPosition;
   vCol = aCol;
   vNormal = normalize((normMatrix * vec4(aNorm, 1.)).xyz);
 }
</script>

Injecting such a script tag is not easy in htmlwidgets. @jcheng5 any thoughts on what mechanisms will help this use case. One approach that came to my mind is to save the generated script tag in a temporary javascript file and then use the attachment mechanism to dynamically load it.

[dmurdoch]

That's not really necessary, it's just the way the model code did it that I based writeWebGL on. I was planning to make it pure Javascript anyway: I just need to put that shader code in a string.

[ramnathv]

Good to know @dmurdoch. If you hit any roadblocks along the way of implementing this widget, do let us know.

[timelyportfolio]

I would be happy to help also but with @dmurdoch and @ramnathv not sure I can add much.

[dmurdoch]

I have some questions about details and philosophy of the implementation of htmlwidgets. Some background first:

Currently, writeWebGL is monolithic, not distinguishing between initialization and rendering. So I need to decide where the dividing line goes.

My plan for the first pass is to mimic writeWebGL from the user point of view. The user will call a series of rgl functions creating a scene within R, then make a single function call to translate everything into WebGL for display in the widget.

Next I'll add functions to allow modifications to an existing scene, but not adding or deleting elements. These will be like the existing propertySetter() and related functions.

Finally I'll make it general, so the htmlwidget display in RStudio could be used as the main display while developing the scene.

So, what's initialization, and what's rendering? One idea is a very minimal initialization: just create an empty rglClass object. When renderValue is called, argument "x" will be the whole scene to be rendered. The other extreme is that the initialization does most of what writeWebGL does now, and the renderValue function hardly does anything.

Which makes sense?

And one other question: in RMarkdown documents, it makes sense that each new display can start with the previous one, and just make some modifications. How do I do this in htmlwidgets?

[jcheng5]

(Sorry for brevity, typing this from my phone while on vacation)

Initialize is called once, renderValue can be called many times (not in an RStudio or R Markdown context, but when used with Shiny). If new data should always cause your widget to be reset then essentially everything in your monolithic approach can go into renderValue and you can leave initialize empty. However if you want to, say, preserve the camera angle (or whatever it's called in WebGL) or you want to somehow diff the old and new data and only change the parts of the scene that are necessary, then you might need to do a different initialize/renderValue split.

For the second question, we do this by having the R representation of the widget be a list that you modify using accessor functions, with pass by value semantics. The dygraph and leaflet packages both use this approach.

[jjallaire]

Examples of how progressively building up a leaflet visualization are here: http://rstudio.github.io/leaflet/.

You'll notice two functionally equivalent syntaxes. Here's the traditional one passing the map as the first argument to various mutator functions:

m <- leaflet()
m #print empty map

m <- addTiles(m)
m # print map with tiles

m <- addMarkers(m, lng=174.768, lat=-36.852, popup="The birthplace of R")
m # print map with markers

The same transformations can also be done via the %>% operator in a chain if you don't want to print the intermediate renderings:

leaflet() %>% 
   addTiles() %>% 
   addMarkers(lng=174.768, lat=-36.852, popup="The birthplace of R")

As Joe described there are different philosophies on whether to rebuild the world in renderValue verses doing smart "diffing" of representations and providing smooth transformation. Part of this depends on the ability of the underlying library to handle the "diffs" and part depends on whether you expect the component to be used frequently in Shiny applications where smooth transformations based on varying inputs will be required. In dygraphs we rebuild the world every time largely based on the fact that dygraphs doesn't handle incremental updates very well. In leaflet things are handled incrementally because this is often very useful for exploring spatial data.

[dmurdoch]

Thanks Joe and JJ. Since rgl scenes are so big, I think it makes sense to make the initialize method fairly heavy, and then each of the renderValue calls won't need to send much.

Now I have a more detailed question. (Remember that I don't really know what I'm doing in Javascript.)

I need to transfer 4 by 4 matrices from R into Javascript "CanvasMatrix4" objects. I have been doing it by generating code like

m = new CanvasMatrix4();
m.load([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]);

Can this happen automatically in JSON (e.g. with a custom serializer), or should I just pass an array, and convert it to a CanvasMatrix4 object with Javascript code like the above?

[jcheng5]

Can this happen automatically in JSON (e.g. with a custom serializer), or should I just pass an array, and convert it to a CanvasMatrix4 object with Javascript code like the above?

The latter. htmlwidgets will only give you generic JS scalars, arrays, and objects--it's entirely on the author to turn those into domain specific objects (if necessary).

[jjallaire]

One more thing to keep in mind: the distinction between initialize and renderValue only matters for use within Shiny (at the R console and in R Markdown both are called only once). You can still of course progressively display / build up a widget payload with successive calls that mutate the x object but except for within Shiny every time you print the widget it's going to run both initialize and renderValue.

In the Shiny case however the widget is updated in place via the call to renderValue. This gives you the opportunity to do intelligent diffing between current and previous states of x and to only render the new options / data (and perhaps even do a nice visual transition from old to new state). However it should be emphasized that this sort of smart transition is going to be quite complex to code correctly. D3 provides a nice start for data transitions via it's enter/update/exit handlers however for many other libraries you'll need to do this from first principles. In some cases the library already supports this sort of dynamic update (in my case for dygraphs they advertised that they could do dynamic updates however in practice there were a bunch of exceptions to this so I ended up just re-rendering the world on every renderValue).

[dmurdoch]

I've looked at it in more detail, and now I have an approach in mind. The initialize call will just set up an empty scene. When I call renderValue, it will either send the whole scene, or just some updates, according to a flag in x. (Essentially the flag will say whether to clear the scene or not.)

I don't want to send the whole scene every time because that can be a lot of data, and I'm worried it will be too slow. My first attempt at a Shiny implementation (back in December, long before I'd heard of htmlwidgets) resent the whole scene, and it took a couple of seconds to make a change.

[jjallaire]

Not to complicate things too much further, but I think the direction you are going is very similar to what we ended up with for leaflet. In that case for the Shiny scenario we created a leafletProxy function which provides a lightweight handle to an existing leaflet instance. All functions that work against leaflet also work against leafletProxy. See the docs here for usage examples: http://rstudio.github.io/leaflet/shiny.html

If the idea is to avoid the transfer of data from R to JavaScript then I think you'll need to do something like leafletProxy to avoid re-sending all of x every time. If however the main issue is rendering time then the flag within x is probably okay (that said, I think the proxy pattern is a nice way for the user to express the desire for incremental updates within Shiny). We ended up doing the same thing for DataTables (see the code at the bottom of this example: https://yihui.shinyapps.io/DT-proxy).

@jcheng5 and @yihui Know more about the particulars of the proxy pattern so may want to add something here.

[dmurdoch]

Thanks for the pointer. It's the transfer of data that's the issue; WebGL is quite fast at rendering once things are in the browser. I'll follow the leafletProxy model when I get to that part of the implementation.

[dmurdoch]

I'm making some progress on this, but now my lack of Javascript knowledge is slowing me down. These are probably simple questions:

1. If I have a matrix in R, it gets passed to Javascript as an array of arrays (essentially row-major order). Are there Javascript functions defined for operations like cbind() (to concatenate rows of an n x p and an n x q matrix to form an n x (p+q) matrix?
2. Is there a simple way to simplify my array of arrays to an array of values (e.g. change [[1,2], [3,4]] to [1,2,3,4])?
3. Are there standard functions like nrow() and ncol(), or do I just use m.length for nrow(m), and m[0].length for ncol(m)?