NodeBB crawler

[sic2]

GDG CT forum is such an example: forum.gdgcatania.org

[vigliag]

The task can be divided this way:

• Figuring out what is the best way to get contents from NodeBB, (json, rss). Can we get the latest posts? Do we get their unique ID? Can we get the individual replies? Can we get the number of replies and participants? Is there some image to show alongside the summary?
• Figuring out if we can also get Events from the forum, or if it's better to keep getting them from the facebook page.
• Figuring out if we need additional database columns (refer to the migration file to see the existing ones) to represent these posts, their categories (which we can maybe convert to tags) and maybe also the threaded conversations.
• Figure out if there are some sections of the forum we need to exclude from crawling and how to address that.
• Writing the actual crawler as a ruby library
• Writing the script that uses the library and adds/updates the posts in the database
• Add the crawler to the "favara" rake task
• done :)