Dataset format is not currently documented.

[brentwalther]

Howdy Lab41,

In each of the provided ipython notebooks, there is a reference to "post_etl_datasets" however I didn't see any documentation explaining the format of these datasets so this piece of information is not very useful at the moment.

What format should the datasets be in?

[ymt123]

Great point. The post_etl_data is the raw Twitter JSON data saved as parquet and partitioned by month (we also reparation to reduce the number of files). Here is the code we actually use to ETL the data:

sc.addPyFile ('/path/to/soft-boiled.zip') 
from src.utils.schema import get_twitter_schema
month = 8
data_path = 'hdfs:///datasets/twitter/2015/%d/*/*/*'%month
json_path = '/path/to/twitter/schema/twitter_format.json'
schema = get_twitter_schema(json_path)
all_tweets = sqlCtx.jsonFile(data_path, schema)
all_tweets.repartition(1000).save("hdfs:///post_etl_datasets/twitter/year=2015/month=%d"%month)

Note 1: Using a saved schema ensures that all the data is the same format, and avoids sqlCtx.jsonFile doing two passes over the data
Note 2: I know the preferred syntax for reading/writing dataframes has changed a few times in the last couple Spark releases. I'll check that and then add the ETL code to the repo