Skip to content

Conversation

@deepgabani8
Copy link
Contributor

@deepgabani8 deepgabani8 commented Sep 21, 2022

Fixes #206 .

@alxmrs
Copy link
Contributor

alxmrs commented Sep 21, 2022

This is a good start. However, the abstractions we provide in the Pipeline object should make adding variations to the pipeline easy (where we expect there to be changes). Our internal pipelines should be a guide for what kinds of changes we can expect from users. For example, users should be able to:

  • choose custom file openers (for their own file types) / ways users can create EE assets
  • create filters or combine URIs into groups to suite specific pipeline needs
  • specify if they want a batch or streaming pipeline

These only reflect our initial variants -- and only for the EE destination. I think that we could benefit from implementing this feature later, as we've still only seen a handful of variations of our pipeline.

@deepgabani8
Copy link
Contributor Author

Yes, It is an abstraction of what weather-mv pipeline is doing which the user can inherit and override pipeline-level functions.

Like this going one step deeper, the user should be able to override bq, ee, rg level functions. For that we can import them in weather_mv's init and make extendable classes.
Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Factor weather-tools as a library that users can extend.

3 participants