Communication Protocol

[nrbrd]

PR Overview

The communication protocol is being developed in this branch (it's based in #3, implementing the communication itself).

Testing

You can run example spiders from #4
(I'll be adding more examples, and all spiders in #4 are working).

Run with:

scrapy streaming <path_of_executable>

[codecov-io]

Current coverage is 100% (diff: 100%)

Merging #5 into master will increase coverage by 11.78%

@@             master    #5   diff @@
=====================================
  Files            11    12     +1   
  Lines           246   387   +141   
  Methods           0     0          
  Messages          0     0          
  Branches          0     0          
=====================================
+ Hits            217   387   +170   
+ Misses           29     0    -29   
  Partials          0     0          

Powered by Codecov. Last update e3f8e33...2ce2b4d

[nrbrd]

@eLRuLL @redapple
This PR is ready to be reviewed.

• Refactored the project structure to keep it cleaner
• Implemented the communication validators (this classes checks the received message, checking required fields, required data type, and default values)
• spider, request, response, error, close, log, and form_request messages
• request with base64 encoding, to request binary data
• Error handling (message validating)
• Getting exceptions from requests
• Getting raised exceptions

[eLRuLL]

let's improve this name, as a scrapy FormRequest is just a POST request, while I think you are trying to do here a from_response_request

[nrbrd]

@eLRuLL
Yeah, from_response_request is really a better name. I renamed it in the last commit.
Also, I added a new test spider (https://github.com/aron-bordin/scrapy-streaming/blob/bfae10172f4f8a08b44a6339a8a8db51c7eaa890/tests/spiders/multiple_data.py) that sends a long message with 1000 log messages to check if the scrapy-streaming buffering is getting everything.