Why is 1.10.12 much slower than 1.10.10?

[doowhtron]

Recently I have my airflow upgraded from 1.10.10 to 1.10.12 and find my dags are scheduled much slower than before. One of my dag is as follows（attachment） and it has 400 Tasks of DummyOperator. It took only 4+mins with 1.10.10 while it took more than 16mins with 1.10.12.

Can someone tell me what the problem may be？I tried to compare the code diff between these two versions with https://github.com/apache/airflow/compare/1.10.10...1.10.12 and failed to draw a conclusion.

huge_dag_demo.py.py.zip

[turbaszek]

@doowhtron can you tell us more about your deployment?

[doowhtron]

Thanks for your attention. My deployment is as follows:

Host: 2 TencentCloud VM, 8 cores, 16GB
OS: centos7
kernel: 3.10.0
Mysql: 5.7.17

Host1 runs: webserver, scheduler, mysqld, worker
Host2 runs: webserver, worker, RabbitMQ

I found two suspicious phenomena:

1. In logs/scheduler/latest/huge_demo3.py.log, Process file took more seconds in 1.10.12 than in 1.10.10.
2. Mysqld's CPU is higher in 1.10.12 than in 1.10.10.

If you need more infomation, please let me know.

[doowhtron]

Problem solved by upgrading to 1.10.13

#11010