Fix an issue that batch predict in mltoolbox fails. (#695)

* Fix an issue that batch predict in mltoolbox fails.

In local prediction run, it is a dataflow local run job. One DataFlow operator, the TF graph runner, calls TF's "bundle_shim.load_session_bundle_or_saved_model_bundle_from_path" API in its "start_bundle" function. This API not only creates a session, but also populate default TF graph with operators loaded from the saved model.

If this operator is called multiple times, multiple operators (even with the same name) will be added to the default graph, which is incorrect and causes unexpected behavior.

The fix resets default graph in the operator's "finish_bundle" call to properly do cleanup.

* Pin dill's version.

Author: qimingj

solutionbox/structured_data/mltoolbox/_structured_data/prediction/predict.py

@@ -157,7 +157,10 @@ def start_bundle(self, element=None):
     self._aliases, self._tensor_names = zip(*self._output_alias_map.items())
 
   def finish_bundle(self, element=None):
+    import tensorflow as tf
+
     self._session.close()
+    tf.reset_default_graph()
 
   def process(self, element):
     """Run batch prediciton on a TF graph.

tox.ini

@@ -14,6 +14,7 @@ skip_missing_interpreters = true
 # For pandas-profiling, we have to install 1.4.0 as 1.4.1 triggers division-by-zero errors.
 deps = pandas-profiling==1.4.0
        apache-airflow==1.9.0
+       dill==0.2.6
        tensorflow==1.8.0
        lime==0.1.1.23
        xgboost==0.6a2