diff --git a/notebooks/VISDOM/CSV Data to MySQL for use in VISDOM.ipynb b/notebooks/VISDOM/CSV Data to MySQL for use in VISDOM.ipynb
index d858b08..2d0586c 100644
--- a/notebooks/VISDOM/CSV Data to MySQL for use in VISDOM.ipynb	
+++ b/notebooks/VISDOM/CSV Data to MySQL for use in VISDOM.ipynb	
@@ -542,7 +542,7 @@
     {
      "data": {
       "text/plain": [
-       "<sqlalchemy.engine.result.ResultProxy at 0x10a349550>"
+       "<sqlalchemy.engine.result.ResultProxy at 0x1062e1210>"
       ]
      },
      "execution_count": 13,
@@ -812,11 +812,22 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 18,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<sqlalchemy.engine.result.ResultProxy at 0x1022d35d0>"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
     "accounts_df['site_pair'] = accounts_df['school_city'] + \"_\" + accounts_df['school_site_name']\n",
     "\n",
@@ -856,16 +867,9 @@
     "### Creating the meter_data table in the desired format"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Note: this is currently treating the meter_uuid, account_uuid, date and zip5 as integers, but they should more likely be treated as varchar, varchar, datetime and varchar, respectively."
-   ]
-  },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 19,
    "metadata": {
     "collapsed": false
    },
@@ -873,10 +877,10 @@
     {
      "data": {
       "text/plain": [
-       "<sqlalchemy.engine.result.ResultProxy at 0x10a4521d0>"
+       "<sqlalchemy.engine.result.ResultProxy at 0x106309410>"
       ]
      },
-     "execution_count": 18,
+     "execution_count": 19,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -915,7 +919,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 20,
    "metadata": {
     "collapsed": true
    },
@@ -928,7 +932,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 21,
    "metadata": {
     "collapsed": true
    },
@@ -941,7 +945,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 22,
    "metadata": {
     "collapsed": false
    },
@@ -1354,6 +1358,1230 @@
     "                    print \"failed sql insert. meter_uuid:\" + str(df['meter_uuid'][0]) + \", filename: \" + f.split(\"_Pacific\")[0] + \"...\""
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Causes of sql insert errors"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Spot checks of these insert errors suggest that they (or at least many of them) are caused by attempting to insert rows that replicate the table's primary key, which is a combination of meter_uuid and date. Looking at the csv files for some of these errors do show that they are replicated data in some pairs of csv files. Below is an example, which covers the first sql insert error noted above:\n",
+    "\n",
+    "```\n",
+    "failed sql insert. meter_uuid:8496493494, filename: 01100176106751_20122013...\n",
+    "```\n",
+    "\n",
+    "This is caused by these two files being identical (this can also be verified with a text diff tool):\n",
+    "\n",
+    "```\n",
+    "PGE_csv/2012-2013/Electricity/01100176106751_20122013_PacificGasElectric_ELECTRIC_20151104.xml_INTERVAL.csv\n",
+    "PGE_csv/2012-2013/Electricity/01100170130419_20122013_PacificGasElectric_ELECTRIC_20151104.xml_INTERVAL.csv\n",
+    "```\n",
+    "\n",
+    "Digging further to see if the original xml files are different suggests that the files are not exactly the same (a text diff tool says they are different, but since they treat the xml as all one line it is difficult to tell how they are different), but inspecting their data trees manually suggests that they may indeed contain the same data. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 52,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>agreement</th>\n",
+       "      <th>start</th>\n",
+       "      <th>d1</th>\n",
+       "      <th>d2</th>\n",
+       "      <th>d3</th>\n",
+       "      <th>d4</th>\n",
+       "      <th>d5</th>\n",
+       "      <th>d6</th>\n",
+       "      <th>d7</th>\n",
+       "      <th>d8</th>\n",
+       "      <th>...</th>\n",
+       "      <th>d87</th>\n",
+       "      <th>d88</th>\n",
+       "      <th>d89</th>\n",
+       "      <th>d90</th>\n",
+       "      <th>d91</th>\n",
+       "      <th>d92</th>\n",
+       "      <th>d93</th>\n",
+       "      <th>d94</th>\n",
+       "      <th>d95</th>\n",
+       "      <th>d96</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341126000</td>\n",
+       "      <td>29.940</td>\n",
+       "      <td>29.244</td>\n",
+       "      <td>28.584</td>\n",
+       "      <td>28.464</td>\n",
+       "      <td>28.332</td>\n",
+       "      <td>27.744</td>\n",
+       "      <td>28.140</td>\n",
+       "      <td>29.460</td>\n",
+       "      <td>...</td>\n",
+       "      <td>32.856</td>\n",
+       "      <td>32.508</td>\n",
+       "      <td>30.324</td>\n",
+       "      <td>30.360</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.800</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.300</td>\n",
+       "      <td>30.000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341212400</td>\n",
+       "      <td>30.924</td>\n",
+       "      <td>29.724</td>\n",
+       "      <td>30.984</td>\n",
+       "      <td>28.788</td>\n",
+       "      <td>29.640</td>\n",
+       "      <td>30.420</td>\n",
+       "      <td>30.156</td>\n",
+       "      <td>31.092</td>\n",
+       "      <td>...</td>\n",
+       "      <td>31.812</td>\n",
+       "      <td>31.968</td>\n",
+       "      <td>29.052</td>\n",
+       "      <td>29.040</td>\n",
+       "      <td>30.120</td>\n",
+       "      <td>29.712</td>\n",
+       "      <td>30.444</td>\n",
+       "      <td>29.136</td>\n",
+       "      <td>23.712</td>\n",
+       "      <td>22.128</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341298800</td>\n",
+       "      <td>21.504</td>\n",
+       "      <td>21.756</td>\n",
+       "      <td>23.112</td>\n",
+       "      <td>22.032</td>\n",
+       "      <td>21.576</td>\n",
+       "      <td>21.600</td>\n",
+       "      <td>21.840</td>\n",
+       "      <td>21.660</td>\n",
+       "      <td>...</td>\n",
+       "      <td>25.776</td>\n",
+       "      <td>24.672</td>\n",
+       "      <td>23.520</td>\n",
+       "      <td>24.732</td>\n",
+       "      <td>24.204</td>\n",
+       "      <td>19.752</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341385200</td>\n",
+       "      <td>5.700</td>\n",
+       "      <td>34.416</td>\n",
+       "      <td>26.676</td>\n",
+       "      <td>24.336</td>\n",
+       "      <td>23.856</td>\n",
+       "      <td>23.880</td>\n",
+       "      <td>23.604</td>\n",
+       "      <td>23.628</td>\n",
+       "      <td>...</td>\n",
+       "      <td>24.732</td>\n",
+       "      <td>24.240</td>\n",
+       "      <td>22.596</td>\n",
+       "      <td>22.704</td>\n",
+       "      <td>22.356</td>\n",
+       "      <td>22.740</td>\n",
+       "      <td>22.512</td>\n",
+       "      <td>23.544</td>\n",
+       "      <td>22.068</td>\n",
+       "      <td>21.780</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341471600</td>\n",
+       "      <td>21.036</td>\n",
+       "      <td>20.676</td>\n",
+       "      <td>21.420</td>\n",
+       "      <td>22.488</td>\n",
+       "      <td>20.976</td>\n",
+       "      <td>21.276</td>\n",
+       "      <td>20.832</td>\n",
+       "      <td>20.508</td>\n",
+       "      <td>...</td>\n",
+       "      <td>30.072</td>\n",
+       "      <td>35.196</td>\n",
+       "      <td>25.692</td>\n",
+       "      <td>24.360</td>\n",
+       "      <td>25.008</td>\n",
+       "      <td>30.636</td>\n",
+       "      <td>22.560</td>\n",
+       "      <td>23.076</td>\n",
+       "      <td>24.720</td>\n",
+       "      <td>22.440</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>5 rows × 98 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "    agreement       start      d1      d2      d3      d4      d5      d6  \\\n",
+       "0  8496493494  1341126000  29.940  29.244  28.584  28.464  28.332  27.744   \n",
+       "1  8496493494  1341212400  30.924  29.724  30.984  28.788  29.640  30.420   \n",
+       "2  8496493494  1341298800  21.504  21.756  23.112  22.032  21.576  21.600   \n",
+       "3  8496493494  1341385200   5.700  34.416  26.676  24.336  23.856  23.880   \n",
+       "4  8496493494  1341471600  21.036  20.676  21.420  22.488  20.976  21.276   \n",
+       "\n",
+       "       d7      d8   ...       d87     d88     d89     d90     d91     d92  \\\n",
+       "0  28.140  29.460   ...    32.856  32.508  30.324  30.360  31.236  32.556   \n",
+       "1  30.156  31.092   ...    31.812  31.968  29.052  29.040  30.120  29.712   \n",
+       "2  21.840  21.660   ...    25.776  24.672  23.520  24.732  24.204  19.752   \n",
+       "3  23.604  23.628   ...    24.732  24.240  22.596  22.704  22.356  22.740   \n",
+       "4  20.832  20.508   ...    30.072  35.196  25.692  24.360  25.008  30.636   \n",
+       "\n",
+       "      d93     d94     d95     d96  \n",
+       "0  31.800  30.696  30.300  30.000  \n",
+       "1  30.444  29.136  23.712  22.128  \n",
+       "2   0.000   0.000   0.000   0.000  \n",
+       "3  22.512  23.544  22.068  21.780  \n",
+       "4  22.560  23.076  24.720  22.440  \n",
+       "\n",
+       "[5 rows x 98 columns]"
+      ]
+     },
+     "execution_count": 52,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "root = \"PGE_csv/2012-2013/Electricity\"\n",
+    "f = \"01100176106751_20122013_PacificGasElectric_ELECTRIC_20151104.xml_INTERVAL.csv\"\n",
+    "df = pd.read_csv(os.path.join(root,f), usecols=usecols)\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 53,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>agreement</th>\n",
+       "      <th>start</th>\n",
+       "      <th>d1</th>\n",
+       "      <th>d2</th>\n",
+       "      <th>d3</th>\n",
+       "      <th>d4</th>\n",
+       "      <th>d5</th>\n",
+       "      <th>d6</th>\n",
+       "      <th>d7</th>\n",
+       "      <th>d8</th>\n",
+       "      <th>...</th>\n",
+       "      <th>d87</th>\n",
+       "      <th>d88</th>\n",
+       "      <th>d89</th>\n",
+       "      <th>d90</th>\n",
+       "      <th>d91</th>\n",
+       "      <th>d92</th>\n",
+       "      <th>d93</th>\n",
+       "      <th>d94</th>\n",
+       "      <th>d95</th>\n",
+       "      <th>d96</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341126000</td>\n",
+       "      <td>29.940</td>\n",
+       "      <td>29.244</td>\n",
+       "      <td>28.584</td>\n",
+       "      <td>28.464</td>\n",
+       "      <td>28.332</td>\n",
+       "      <td>27.744</td>\n",
+       "      <td>28.140</td>\n",
+       "      <td>29.460</td>\n",
+       "      <td>...</td>\n",
+       "      <td>32.856</td>\n",
+       "      <td>32.508</td>\n",
+       "      <td>30.324</td>\n",
+       "      <td>30.360</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.800</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.300</td>\n",
+       "      <td>30.000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341212400</td>\n",
+       "      <td>30.924</td>\n",
+       "      <td>29.724</td>\n",
+       "      <td>30.984</td>\n",
+       "      <td>28.788</td>\n",
+       "      <td>29.640</td>\n",
+       "      <td>30.420</td>\n",
+       "      <td>30.156</td>\n",
+       "      <td>31.092</td>\n",
+       "      <td>...</td>\n",
+       "      <td>31.812</td>\n",
+       "      <td>31.968</td>\n",
+       "      <td>29.052</td>\n",
+       "      <td>29.040</td>\n",
+       "      <td>30.120</td>\n",
+       "      <td>29.712</td>\n",
+       "      <td>30.444</td>\n",
+       "      <td>29.136</td>\n",
+       "      <td>23.712</td>\n",
+       "      <td>22.128</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341298800</td>\n",
+       "      <td>21.504</td>\n",
+       "      <td>21.756</td>\n",
+       "      <td>23.112</td>\n",
+       "      <td>22.032</td>\n",
+       "      <td>21.576</td>\n",
+       "      <td>21.600</td>\n",
+       "      <td>21.840</td>\n",
+       "      <td>21.660</td>\n",
+       "      <td>...</td>\n",
+       "      <td>25.776</td>\n",
+       "      <td>24.672</td>\n",
+       "      <td>23.520</td>\n",
+       "      <td>24.732</td>\n",
+       "      <td>24.204</td>\n",
+       "      <td>19.752</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341385200</td>\n",
+       "      <td>5.700</td>\n",
+       "      <td>34.416</td>\n",
+       "      <td>26.676</td>\n",
+       "      <td>24.336</td>\n",
+       "      <td>23.856</td>\n",
+       "      <td>23.880</td>\n",
+       "      <td>23.604</td>\n",
+       "      <td>23.628</td>\n",
+       "      <td>...</td>\n",
+       "      <td>24.732</td>\n",
+       "      <td>24.240</td>\n",
+       "      <td>22.596</td>\n",
+       "      <td>22.704</td>\n",
+       "      <td>22.356</td>\n",
+       "      <td>22.740</td>\n",
+       "      <td>22.512</td>\n",
+       "      <td>23.544</td>\n",
+       "      <td>22.068</td>\n",
+       "      <td>21.780</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>8496493494</td>\n",
+       "      <td>1341471600</td>\n",
+       "      <td>21.036</td>\n",
+       "      <td>20.676</td>\n",
+       "      <td>21.420</td>\n",
+       "      <td>22.488</td>\n",
+       "      <td>20.976</td>\n",
+       "      <td>21.276</td>\n",
+       "      <td>20.832</td>\n",
+       "      <td>20.508</td>\n",
+       "      <td>...</td>\n",
+       "      <td>30.072</td>\n",
+       "      <td>35.196</td>\n",
+       "      <td>25.692</td>\n",
+       "      <td>24.360</td>\n",
+       "      <td>25.008</td>\n",
+       "      <td>30.636</td>\n",
+       "      <td>22.560</td>\n",
+       "      <td>23.076</td>\n",
+       "      <td>24.720</td>\n",
+       "      <td>22.440</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>5 rows × 98 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "    agreement       start      d1      d2      d3      d4      d5      d6  \\\n",
+       "0  8496493494  1341126000  29.940  29.244  28.584  28.464  28.332  27.744   \n",
+       "1  8496493494  1341212400  30.924  29.724  30.984  28.788  29.640  30.420   \n",
+       "2  8496493494  1341298800  21.504  21.756  23.112  22.032  21.576  21.600   \n",
+       "3  8496493494  1341385200   5.700  34.416  26.676  24.336  23.856  23.880   \n",
+       "4  8496493494  1341471600  21.036  20.676  21.420  22.488  20.976  21.276   \n",
+       "\n",
+       "       d7      d8   ...       d87     d88     d89     d90     d91     d92  \\\n",
+       "0  28.140  29.460   ...    32.856  32.508  30.324  30.360  31.236  32.556   \n",
+       "1  30.156  31.092   ...    31.812  31.968  29.052  29.040  30.120  29.712   \n",
+       "2  21.840  21.660   ...    25.776  24.672  23.520  24.732  24.204  19.752   \n",
+       "3  23.604  23.628   ...    24.732  24.240  22.596  22.704  22.356  22.740   \n",
+       "4  20.832  20.508   ...    30.072  35.196  25.692  24.360  25.008  30.636   \n",
+       "\n",
+       "      d93     d94     d95     d96  \n",
+       "0  31.800  30.696  30.300  30.000  \n",
+       "1  30.444  29.136  23.712  22.128  \n",
+       "2   0.000   0.000   0.000   0.000  \n",
+       "3  22.512  23.544  22.068  21.780  \n",
+       "4  22.560  23.076  24.720  22.440  \n",
+       "\n",
+       "[5 rows x 98 columns]"
+      ]
+     },
+     "execution_count": 53,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "root = \"PGE_csv/2012-2013/Electricity\"\n",
+    "f = \"01100170130419_20122013_PacificGasElectric_ELECTRIC_20151104.xml_INTERVAL.csv\"\n",
+    "df = pd.read_csv(os.path.join(root,f), usecols=usecols)\n",
+    "df.head()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To see how widespread this duplication in the underlying csv data is in causing the sql insert errors, consider comparisons of all of their top lines via pandas and identifying duplicates as shown below. This suggests that 353 of the 383 sql insert errors may indeed be caused by duplication in the csv data. The identified duplicates pertain to 165 different meter_uuids."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "top_lines = []\n",
+    "file_names = []\n",
+    "for root, dirs, files in os.walk(csv_dir):\n",
+    "    for f in files:\n",
+    "        if f.endswith('_INTERVAL.csv'):\n",
+    "            df = pd.read_csv(os.path.join(root,f), usecols=usecols)\n",
+    "            if len(df) > 0:\n",
+    "                file_names.append(f)\n",
+    "                top_lines.append(df.to_dict(orient='records')[0])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>agreement</th>\n",
+       "      <th>d1</th>\n",
+       "      <th>d10</th>\n",
+       "      <th>d11</th>\n",
+       "      <th>d12</th>\n",
+       "      <th>d13</th>\n",
+       "      <th>d14</th>\n",
+       "      <th>d15</th>\n",
+       "      <th>d16</th>\n",
+       "      <th>d17</th>\n",
+       "      <th>...</th>\n",
+       "      <th>d89</th>\n",
+       "      <th>d9</th>\n",
+       "      <th>d90</th>\n",
+       "      <th>d91</th>\n",
+       "      <th>d92</th>\n",
+       "      <th>d93</th>\n",
+       "      <th>d94</th>\n",
+       "      <th>d95</th>\n",
+       "      <th>d96</th>\n",
+       "      <th>start</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>29.9400</td>\n",
+       "      <td>28.992</td>\n",
+       "      <td>27.24</td>\n",
+       "      <td>28.776</td>\n",
+       "      <td>28.344</td>\n",
+       "      <td>29.1720</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>28.8720</td>\n",
+       "      <td>27.8640</td>\n",
+       "      <td>...</td>\n",
+       "      <td>30.324</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>30.36</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.80</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.30</td>\n",
+       "      <td>30.00</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>29.9400</td>\n",
+       "      <td>28.992</td>\n",
+       "      <td>27.24</td>\n",
+       "      <td>28.776</td>\n",
+       "      <td>28.344</td>\n",
+       "      <td>29.1720</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>28.8720</td>\n",
+       "      <td>27.8640</td>\n",
+       "      <td>...</td>\n",
+       "      <td>30.324</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>30.36</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.80</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.30</td>\n",
+       "      <td>30.00</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>5.637200e+09</td>\n",
+       "      <td>5.5600</td>\n",
+       "      <td>5.160</td>\n",
+       "      <td>5.08</td>\n",
+       "      <td>5.000</td>\n",
+       "      <td>5.200</td>\n",
+       "      <td>5.0800</td>\n",
+       "      <td>5.1200</td>\n",
+       "      <td>5.0800</td>\n",
+       "      <td>5.0800</td>\n",
+       "      <td>...</td>\n",
+       "      <td>5.440</td>\n",
+       "      <td>5.1600</td>\n",
+       "      <td>5.48</td>\n",
+       "      <td>5.400</td>\n",
+       "      <td>5.400</td>\n",
+       "      <td>5.44</td>\n",
+       "      <td>5.400</td>\n",
+       "      <td>5.32</td>\n",
+       "      <td>5.36</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>5.637200e+09</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.24</td>\n",
+       "      <td>0.360</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>0.3600</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>...</td>\n",
+       "      <td>0.360</td>\n",
+       "      <td>0.3600</td>\n",
+       "      <td>0.24</td>\n",
+       "      <td>0.360</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.36</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.24</td>\n",
+       "      <td>0.36</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>5.637199e+09</td>\n",
+       "      <td>7.2336</td>\n",
+       "      <td>7.320</td>\n",
+       "      <td>7.32</td>\n",
+       "      <td>7.128</td>\n",
+       "      <td>7.128</td>\n",
+       "      <td>7.0464</td>\n",
+       "      <td>7.0464</td>\n",
+       "      <td>6.9568</td>\n",
+       "      <td>6.9568</td>\n",
+       "      <td>...</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>7.4144</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>1.355990e+09</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>5 rows × 98 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "      agreement       d1     d10    d11     d12     d13      d14      d15  \\\n",
+       "0  8.496493e+09  29.9400  28.992  27.24  28.776  28.344  29.1720  27.6600   \n",
+       "1  8.496493e+09  29.9400  28.992  27.24  28.776  28.344  29.1720  27.6600   \n",
+       "2  5.637200e+09   5.5600   5.160   5.08   5.000   5.200   5.0800   5.1200   \n",
+       "3  5.637200e+09   0.2400   0.240   0.24   0.360   0.240   0.2400   0.3600   \n",
+       "4  5.637199e+09   7.2336   7.320   7.32   7.128   7.128   7.0464   7.0464   \n",
+       "\n",
+       "       d16      d17      ...          d89       d9    d90     d91     d92  \\\n",
+       "0  28.8720  27.8640      ...       30.324  27.6600  30.36  31.236  32.556   \n",
+       "1  28.8720  27.8640      ...       30.324  27.6600  30.36  31.236  32.556   \n",
+       "2   5.0800   5.0800      ...        5.440   5.1600   5.48   5.400   5.400   \n",
+       "3   0.2400   0.2400      ...        0.360   0.3600   0.24   0.360   0.240   \n",
+       "4   6.9568   6.9568      ...        0.000   7.4144   0.00   0.000   0.000   \n",
+       "\n",
+       "     d93     d94    d95    d96         start  \n",
+       "0  31.80  30.696  30.30  30.00  1.341126e+09  \n",
+       "1  31.80  30.696  30.30  30.00  1.341126e+09  \n",
+       "2   5.44   5.400   5.32   5.36  1.341126e+09  \n",
+       "3   0.36   0.240   0.24   0.36  1.341126e+09  \n",
+       "4   0.00   0.000   0.00   0.00  1.355990e+09  \n",
+       "\n",
+       "[5 rows x 98 columns]"
+      ]
+     },
+     "execution_count": 41,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "top_lines_df = pd.DataFrame.from_records(top_lines)\n",
+    "top_lines_df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>agreement</th>\n",
+       "      <th>d1</th>\n",
+       "      <th>d10</th>\n",
+       "      <th>d11</th>\n",
+       "      <th>d12</th>\n",
+       "      <th>d13</th>\n",
+       "      <th>d14</th>\n",
+       "      <th>d15</th>\n",
+       "      <th>d16</th>\n",
+       "      <th>d17</th>\n",
+       "      <th>...</th>\n",
+       "      <th>d9</th>\n",
+       "      <th>d90</th>\n",
+       "      <th>d91</th>\n",
+       "      <th>d92</th>\n",
+       "      <th>d93</th>\n",
+       "      <th>d94</th>\n",
+       "      <th>d95</th>\n",
+       "      <th>d96</th>\n",
+       "      <th>start</th>\n",
+       "      <th>filename</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>29.9400</td>\n",
+       "      <td>28.992</td>\n",
+       "      <td>27.24</td>\n",
+       "      <td>28.776</td>\n",
+       "      <td>28.344</td>\n",
+       "      <td>29.1720</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>28.8720</td>\n",
+       "      <td>27.8640</td>\n",
+       "      <td>...</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>30.36</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.80</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.30</td>\n",
+       "      <td>30.00</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01100170130419_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>29.9400</td>\n",
+       "      <td>28.992</td>\n",
+       "      <td>27.24</td>\n",
+       "      <td>28.776</td>\n",
+       "      <td>28.344</td>\n",
+       "      <td>29.1720</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>28.8720</td>\n",
+       "      <td>27.8640</td>\n",
+       "      <td>...</td>\n",
+       "      <td>27.6600</td>\n",
+       "      <td>30.36</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.80</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.30</td>\n",
+       "      <td>30.00</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01100176106751_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>5.637200e+09</td>\n",
+       "      <td>5.5600</td>\n",
+       "      <td>5.160</td>\n",
+       "      <td>5.08</td>\n",
+       "      <td>5.000</td>\n",
+       "      <td>5.200</td>\n",
+       "      <td>5.0800</td>\n",
+       "      <td>5.1200</td>\n",
+       "      <td>5.0800</td>\n",
+       "      <td>5.0800</td>\n",
+       "      <td>...</td>\n",
+       "      <td>5.1600</td>\n",
+       "      <td>5.48</td>\n",
+       "      <td>5.400</td>\n",
+       "      <td>5.400</td>\n",
+       "      <td>5.44</td>\n",
+       "      <td>5.400</td>\n",
+       "      <td>5.32</td>\n",
+       "      <td>5.36</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01612340000000_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>5.637200e+09</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.24</td>\n",
+       "      <td>0.360</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>0.3600</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>0.2400</td>\n",
+       "      <td>...</td>\n",
+       "      <td>0.3600</td>\n",
+       "      <td>0.24</td>\n",
+       "      <td>0.360</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.36</td>\n",
+       "      <td>0.240</td>\n",
+       "      <td>0.24</td>\n",
+       "      <td>0.36</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01612340116301_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>5.637199e+09</td>\n",
+       "      <td>7.2336</td>\n",
+       "      <td>7.320</td>\n",
+       "      <td>7.32</td>\n",
+       "      <td>7.128</td>\n",
+       "      <td>7.128</td>\n",
+       "      <td>7.0464</td>\n",
+       "      <td>7.0464</td>\n",
+       "      <td>6.9568</td>\n",
+       "      <td>6.9568</td>\n",
+       "      <td>...</td>\n",
+       "      <td>7.4144</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>0.000</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>1.355990e+09</td>\n",
+       "      <td>01612340130054_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>5 rows × 99 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "      agreement       d1     d10    d11     d12     d13      d14      d15  \\\n",
+       "0  8.496493e+09  29.9400  28.992  27.24  28.776  28.344  29.1720  27.6600   \n",
+       "1  8.496493e+09  29.9400  28.992  27.24  28.776  28.344  29.1720  27.6600   \n",
+       "2  5.637200e+09   5.5600   5.160   5.08   5.000   5.200   5.0800   5.1200   \n",
+       "3  5.637200e+09   0.2400   0.240   0.24   0.360   0.240   0.2400   0.3600   \n",
+       "4  5.637199e+09   7.2336   7.320   7.32   7.128   7.128   7.0464   7.0464   \n",
+       "\n",
+       "       d16      d17                        ...                          \\\n",
+       "0  28.8720  27.8640                        ...                           \n",
+       "1  28.8720  27.8640                        ...                           \n",
+       "2   5.0800   5.0800                        ...                           \n",
+       "3   0.2400   0.2400                        ...                           \n",
+       "4   6.9568   6.9568                        ...                           \n",
+       "\n",
+       "        d9    d90     d91     d92    d93     d94    d95    d96         start  \\\n",
+       "0  27.6600  30.36  31.236  32.556  31.80  30.696  30.30  30.00  1.341126e+09   \n",
+       "1  27.6600  30.36  31.236  32.556  31.80  30.696  30.30  30.00  1.341126e+09   \n",
+       "2   5.1600   5.48   5.400   5.400   5.44   5.400   5.32   5.36  1.341126e+09   \n",
+       "3   0.3600   0.24   0.360   0.240   0.36   0.240   0.24   0.36  1.341126e+09   \n",
+       "4   7.4144   0.00   0.000   0.000   0.00   0.000   0.00   0.00  1.355990e+09   \n",
+       "\n",
+       "                                            filename  \n",
+       "0  01100170130419_20122013_PacificGasElectric_ELE...  \n",
+       "1  01100176106751_20122013_PacificGasElectric_ELE...  \n",
+       "2  01612340000000_20122013_PacificGasElectric_ELE...  \n",
+       "3  01612340116301_20122013_PacificGasElectric_ELE...  \n",
+       "4  01612340130054_20122013_PacificGasElectric_ELE...  \n",
+       "\n",
+       "[5 rows x 99 columns]"
+      ]
+     },
+     "execution_count": 43,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "top_lines_df['filename'] = file_names\n",
+    "top_lines_df.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 44,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>agreement</th>\n",
+       "      <th>d1</th>\n",
+       "      <th>d10</th>\n",
+       "      <th>d11</th>\n",
+       "      <th>d12</th>\n",
+       "      <th>d13</th>\n",
+       "      <th>d14</th>\n",
+       "      <th>d15</th>\n",
+       "      <th>d16</th>\n",
+       "      <th>d17</th>\n",
+       "      <th>...</th>\n",
+       "      <th>d9</th>\n",
+       "      <th>d90</th>\n",
+       "      <th>d91</th>\n",
+       "      <th>d92</th>\n",
+       "      <th>d93</th>\n",
+       "      <th>d94</th>\n",
+       "      <th>d95</th>\n",
+       "      <th>d96</th>\n",
+       "      <th>start</th>\n",
+       "      <th>filename</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>29.94</td>\n",
+       "      <td>28.992</td>\n",
+       "      <td>27.240</td>\n",
+       "      <td>28.776</td>\n",
+       "      <td>28.344</td>\n",
+       "      <td>29.172</td>\n",
+       "      <td>27.660</td>\n",
+       "      <td>28.872</td>\n",
+       "      <td>27.864</td>\n",
+       "      <td>...</td>\n",
+       "      <td>27.66</td>\n",
+       "      <td>30.36</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.80</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.30</td>\n",
+       "      <td>30.00</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01100170130419_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>29.94</td>\n",
+       "      <td>28.992</td>\n",
+       "      <td>27.240</td>\n",
+       "      <td>28.776</td>\n",
+       "      <td>28.344</td>\n",
+       "      <td>29.172</td>\n",
+       "      <td>27.660</td>\n",
+       "      <td>28.872</td>\n",
+       "      <td>27.864</td>\n",
+       "      <td>...</td>\n",
+       "      <td>27.66</td>\n",
+       "      <td>30.36</td>\n",
+       "      <td>31.236</td>\n",
+       "      <td>32.556</td>\n",
+       "      <td>31.80</td>\n",
+       "      <td>30.696</td>\n",
+       "      <td>30.30</td>\n",
+       "      <td>30.00</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01100176106751_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>5.637199e+09</td>\n",
+       "      <td>4.96</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.680</td>\n",
+       "      <td>5.440</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.520</td>\n",
+       "      <td>5.680</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>...</td>\n",
+       "      <td>5.52</td>\n",
+       "      <td>5.44</td>\n",
+       "      <td>5.760</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.36</td>\n",
+       "      <td>4.720</td>\n",
+       "      <td>4.32</td>\n",
+       "      <td>4.32</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01612340130484_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>5.637199e+09</td>\n",
+       "      <td>4.96</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.680</td>\n",
+       "      <td>5.440</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.520</td>\n",
+       "      <td>5.680</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>...</td>\n",
+       "      <td>5.52</td>\n",
+       "      <td>5.44</td>\n",
+       "      <td>5.760</td>\n",
+       "      <td>5.600</td>\n",
+       "      <td>5.36</td>\n",
+       "      <td>4.720</td>\n",
+       "      <td>4.32</td>\n",
+       "      <td>4.32</td>\n",
+       "      <td>1.341126e+09</td>\n",
+       "      <td>01612340135426_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>18</th>\n",
+       "      <td>2.657846e+09</td>\n",
+       "      <td>0.00</td>\n",
+       "      <td>4.368</td>\n",
+       "      <td>3.816</td>\n",
+       "      <td>3.640</td>\n",
+       "      <td>3.152</td>\n",
+       "      <td>3.544</td>\n",
+       "      <td>4.664</td>\n",
+       "      <td>4.536</td>\n",
+       "      <td>3.616</td>\n",
+       "      <td>...</td>\n",
+       "      <td>4.08</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>1.365664e+09</td>\n",
+       "      <td>03739810000000_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "<p>5 rows × 99 columns</p>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "       agreement     d1     d10     d11     d12     d13     d14     d15  \\\n",
+       "0   8.496493e+09  29.94  28.992  27.240  28.776  28.344  29.172  27.660   \n",
+       "1   8.496493e+09  29.94  28.992  27.240  28.776  28.344  29.172  27.660   \n",
+       "5   5.637199e+09   4.96   5.600   5.600   5.680   5.440   5.600   5.520   \n",
+       "6   5.637199e+09   4.96   5.600   5.600   5.680   5.440   5.600   5.520   \n",
+       "18  2.657846e+09   0.00   4.368   3.816   3.640   3.152   3.544   4.664   \n",
+       "\n",
+       "       d16     d17                        ...                             d9  \\\n",
+       "0   28.872  27.864                        ...                          27.66   \n",
+       "1   28.872  27.864                        ...                          27.66   \n",
+       "5    5.680   5.600                        ...                           5.52   \n",
+       "6    5.680   5.600                        ...                           5.52   \n",
+       "18   4.536   3.616                        ...                           4.08   \n",
+       "\n",
+       "      d90     d91     d92    d93     d94    d95    d96         start  \\\n",
+       "0   30.36  31.236  32.556  31.80  30.696  30.30  30.00  1.341126e+09   \n",
+       "1   30.36  31.236  32.556  31.80  30.696  30.30  30.00  1.341126e+09   \n",
+       "5    5.44   5.760   5.600   5.36   4.720   4.32   4.32  1.341126e+09   \n",
+       "6    5.44   5.760   5.600   5.36   4.720   4.32   4.32  1.341126e+09   \n",
+       "18    NaN     NaN     NaN    NaN     NaN    NaN    NaN  1.365664e+09   \n",
+       "\n",
+       "                                             filename  \n",
+       "0   01100170130419_20122013_PacificGasElectric_ELE...  \n",
+       "1   01100176106751_20122013_PacificGasElectric_ELE...  \n",
+       "5   01612340130484_20122013_PacificGasElectric_ELE...  \n",
+       "6   01612340135426_20122013_PacificGasElectric_ELE...  \n",
+       "18  03739810000000_20122013_PacificGasElectric_ELE...  \n",
+       "\n",
+       "[5 rows x 99 columns]"
+      ]
+     },
+     "execution_count": 44,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "duplicated_top_lines = top_lines_df[top_lines_df.duplicated(subset=usecols, keep=False)]\n",
+    "duplicated_top_lines.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 56,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(645, 292, 165)"
+      ]
+     },
+     "execution_count": 56,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(duplicated_top_lines), len(duplicated_top_lines[['agreement','start']].drop_duplicates()), len(duplicated_top_lines[['agreement']].drop_duplicates()), "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 51,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "353"
+      ]
+     },
+     "execution_count": 51,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(duplicated_top_lines) - len(duplicated_top_lines[['agreement','start']].drop_duplicates())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "note that there are 383 failed inserts above, so this duplication of 353 top-rows could explain most of them\n",
+    "\n",
+    "the first 5 fails are:\n",
+    "\n",
+    "```\n",
+    "failed sql insert. meter_uuid:8496493494, filename: 01100176106751_20122013...\n",
+    "failed sql insert. meter_uuid:5637199274, filename: 01612340135426_20122013...\n",
+    "failed sql insert. meter_uuid:5741361055, filename: 04615230123687_20122013...\n",
+    "failed sql insert. meter_uuid:1626781530, filename: 06616220630038_20122013...\n",
+    "failed sql insert. meter_uuid:6449693283, filename: 10739651030402_20122013...\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 48,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>agreement</th>\n",
+       "      <th>filename</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>01100170130419_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>8.496493e+09</td>\n",
+       "      <td>01100176106751_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>5.637199e+09</td>\n",
+       "      <td>01612340130484_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>5.637199e+09</td>\n",
+       "      <td>01612340135426_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>18</th>\n",
+       "      <td>2.657846e+09</td>\n",
+       "      <td>03739810000000_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>26</th>\n",
+       "      <td>2.657846e+09</td>\n",
+       "      <td>03739816107395_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>27</th>\n",
+       "      <td>5.741361e+09</td>\n",
+       "      <td>04615230000000_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>29</th>\n",
+       "      <td>5.741361e+09</td>\n",
+       "      <td>04615230123687_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>32</th>\n",
+       "      <td>2.084624e+09</td>\n",
+       "      <td>06616220118729_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>33</th>\n",
+       "      <td>2.084624e+09</td>\n",
+       "      <td>06616220630038_20122013_PacificGasElectric_ELE...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "       agreement                                           filename\n",
+       "0   8.496493e+09  01100170130419_20122013_PacificGasElectric_ELE...\n",
+       "1   8.496493e+09  01100176106751_20122013_PacificGasElectric_ELE...\n",
+       "5   5.637199e+09  01612340130484_20122013_PacificGasElectric_ELE...\n",
+       "6   5.637199e+09  01612340135426_20122013_PacificGasElectric_ELE...\n",
+       "18  2.657846e+09  03739810000000_20122013_PacificGasElectric_ELE...\n",
+       "26  2.657846e+09  03739816107395_20122013_PacificGasElectric_ELE...\n",
+       "27  5.741361e+09  04615230000000_20122013_PacificGasElectric_ELE...\n",
+       "29  5.741361e+09  04615230123687_20122013_PacificGasElectric_ELE...\n",
+       "32  2.084624e+09  06616220118729_20122013_PacificGasElectric_ELE...\n",
+       "33  2.084624e+09  06616220630038_20122013_PacificGasElectric_ELE..."
+      ]
+     },
+     "execution_count": 48,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "duplicated_top_lines[['agreement','filename']].head(10)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -1363,7 +2591,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 23,
    "metadata": {
     "collapsed": false
    },
@@ -1496,7 +2724,7 @@
        "[3 rows x 100 columns]"
       ]
      },
-     "execution_count": 22,
+     "execution_count": 23,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1507,7 +2735,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 24,
    "metadata": {
     "collapsed": false
    },
@@ -1537,7 +2765,7 @@
        "0   1286757"
       ]
      },
-     "execution_count": 23,
+     "execution_count": 24,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1557,7 +2785,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 25,
    "metadata": {
     "collapsed": false
    },
@@ -1565,10 +2793,10 @@
     {
      "data": {
       "text/plain": [
-       "<sqlalchemy.engine.result.ResultProxy at 0x10a1268d0>"
+       "<sqlalchemy.engine.result.ResultProxy at 0x107010b90>"
       ]
      },
-     "execution_count": 24,
+     "execution_count": 25,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1642,7 +2870,7 @@
     {
      "data": {
       "text/plain": [
-       "<sqlalchemy.engine.result.ResultProxy at 0x10a126890>"
+       "<sqlalchemy.engine.result.ResultProxy at 0x10860b210>"
       ]
      },
      "execution_count": 27,