Warning overkill? {providers_manager.py:293} WARNING - Exception when importing

[dstandish]

With a fresh install of airflow 2.0, with no dags, when starting up webserver you are bombarded with warnings about import errors.

For example:

[2020-12-29 22:26:24,847] {providers_manager.py:293} WARNING - Exception when importing 'airflow.providers.yandex.hooks.yandex.YandexCloudBaseHook' from 'apache-airflow-providers-yandex' package: No module named 'yandexcloud'
[2020-12-29 22:26:24,848] {providers_manager.py:293} WARNING - Exception when importing 'airflow.providers.redis.hooks.redis.RedisHook' from 'apache-airflow-providers-redis' package: No module named 'redis'
[2020-12-29 22:26:24,849] {providers_manager.py:293} WARNING - Exception when importing 'airflow.providers.apache.spark.hooks.spark_jdbc.SparkJDBCHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined
[2020-12-29 22:26:24,852] {providers_manager.py:293} WARNING - Exception when importing 'airflow.providers.apache.spark.hooks.spark_submit.SparkSubmitHook' from 'apache-airflow-providers-apache-spark' package: name 'client' is not defined
[2020-12-29 22:26:24,853] {providers_manager.py:293} WARNING - Exception when importing 'airflow.providers.apache.cassandra.hooks.cassandra.CassandraHook' from 'apache-airflow-providers-apache-cassandra' package: No module named 'cassandra'
...

The full list is much longer.

And it repeats over and over again.

This does not seem desirable.

@potiuk you have any thoughts?

Interestingly, not an issue with scheduler.

[potiuk]

That is an interesting one. How did you install airflow on the webserver? How did you install provider packages there?

Was it from sources or PIP packages? Or maybe some local, modified version of airflow? That could happen in case you installed providers without dependencies or installed airlfow from sources but without adding [extras] when installing it.

[dstandish]

ok yeah you're right i was mistaken... i thought it was vanilla install but in actuality it was some screwy image i built

cancel that :)

thanks

[dstandish]

ok @potiuk i figured out how to repro

mkvirtualenv airflow
pip install -e '.[devel]'

[potiuk]

ok @potiuk i figured out how to repro

mkvirtualenv airflow
pip install -e '.[devel]'

You need to set INSTALL_PROVIDERS_FROM_SOURCES variable when runing pip install -e.

BTW. If you find some rock-solid way of knowing in setup.py that we are just running pip install -e that would be great to automate it. https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#provider-packages

[potiuk]

Otherwise I might want to make it stand out much more

[potiuk]

Actually - I found it :) https://stackoverflow.com/questions/51292333/how-to-tell-from-setup-py-if-the-module-is-being-installed-in-editable-mode

[potiuk]

Will be solved in #13439

[dstandish]

Whoa, nice :)