Add example notebook

[SarahAlidoost]

Here, I added an example notebook that shows the complete workflow: model training, data processing and prediction.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[SarahAlidoost]

@QianqianHan96 here I added an example notebook mainly to show how to optimize the emulator workflow using dask, dask-ml and xarray. There might be small differences in handling data for example, LAI interpolation using data from the previous year. I didn't run the example notebook on snellius with a large dataset. I created this pull request, just in case you would like to add the example to the repository.

[QianqianHan96]

Hi Sarah, thanks for adding the example notebook. I will have a look at it probably at the end of this month when I start producing fluxes dataset. Now I am still working on cleaning the training data.

I will let you know after I go through your optimized script.

[QianqianHan96]

Hi Sarah,

I started working on this pull request this week. Your script is way better organized than the script I shared with you in May.
However, I found that for LAI "preprocessing", for short period and just Europe, there is no problem. But if I do it for one year and global scale, it is super slow (I tried to use same chunk as you "500", but there are too many tasks: 370000 tasks). So maybe it's better to do interpolations first and then export to zarr for LAI, in this way we decrease 81 times data volume from 1km to 9km.

[QianqianHan96]

Hi Sarah,

I started working on this pull request this week. Your script is way better organized than the script I shared with you in May. However, I found that for LAI "preprocessing", for short period and just Europe, there is no problem. But if I do it for one year and global scale, it is super slow (I tried to use same chunk as you "500", but there are too many tasks: 370000 tasks). So maybe it's better to do interpolations first and then export to zarr for LAI, in this way we decrease 81 times data volume from 1km to 9km.

I found that the problem is not about data volume without interpolation. The real reason is chunk size. If we change "chunks=500" to "chunks ={'longitude': 250, 'latitude': 250,'time':750}", there is no problem for global one year. For small data, "chunks=500" is okay, but big data we need to specify chunks for each dimension.