Allowing comments to be in statemenet blocks

[AsherGlick]

This PR allows for comments to exist within module statement blocks when the parse is configured with include_comments=True

It feels like I kinda hacked in the second parser into the testcase. I wanted to avoid refactoring the harness entirely for such a small change.

[revarbat]

I think this also includes comments from modules when include_comments=false

[AsherGlick]

You might be right. I will refactor the tests to validate the ast not just that the parse succeeded.

[AsherGlick]

I think that I have confirmed that the comments are not included when include_comments=False.

here are the arpeggio token trees for each parse result

module | test | ( | ) | { | sphere | ( | 5 | ) | ; | cube | ( | 10 | ) | ; | } | 
module | test | ( | ) | { | sphere | ( | 5 | ) | ; | // comment | cube | ( | 10 | ) | ; | } | 

The comment is only present in the one that has include_comments=True

I also experimented a bit with validating the parsed tree in the unit tests, but the tree is very deep, especially when parsing numbers, so I dont think validating the entire tree is desirable as it will create really noisy tests that need to be adjusted whenever any part of the parser changes. I could instead validate all the tokens that were parsed, without their parent nodes. otherwise we get something very large and unweildly like this:

[
    [
        [
            [
                [
                    "KWD_MODULE 'module' [0]"
                ],
                [
                    [
                        "TOK_ID 'test' [7]"
                    ]
                ],
                [
                    [
                        "TOK_PAREN '(' [11]"
                    ],
                    [
                        "TOK_ENDPAREN ')' [12]"
                    ]
                ],
                [
                    [
                        [
                            "TOK_BRACE '{' [14]"
                        ],
                        [
                            [
                                [
                                    [
                                        [
                                            [
                                                "TOK_ID 'sphere' [16]"
                                            ]
                                        ],
                                        [
                                            "TOK_PAREN '(' [22]"
                                        ],
                                        [
                                            [
                                                [
                                                    [
                                                        [
                                                            [
                                                                [
                                                                    [
                                                                        [
                                                                            [
                                                                                [
                                                                                    [
                                                                                        [
                                                                                            [
                                                                                                [
                                                                                                    [
                                                                                                        [
                                                                                                            [
                                                                                                                "TOK_NUMBER '5' [23]"
                                                                                                            ]
                                                                                                        ]
                                                                                                    ]
                                                                                                ]
                                                                                            ]
                                                                                        ]
                                                                                    ]
                                                                                ]
                                                                            ]
                                                                        ]
                                                                    ]
                                                                ]
                                                            ]
                                                        ]
                                                    ]
                                                ]
                                            ]
                                        ],
                                        [
                                            "TOK_ENDPAREN ')' [24]"
                                        ],
                                        [
                                            [
                                                "empty_statement ';' [25]"
                                            ]
                                        ]
                                    ]
                                ]
                            ]
                        ],
                        [
                            [
                                "comment_line '// comment' [27]"
                            ]
                        ],
                        [
                            [
                                [
                                    [
                                        [
                                            [
                                                "TOK_ID 'cube' [38]"
                                            ]
                                        ],
                                        [
                                            "TOK_PAREN '(' [42]"
                                        ],
                                        [
                                            [
                                                [
                                                    [
                                                        [
                                                            [
                                                                [
                                                                    [
                                                                        [
                                                                            [
                                                                                [
                                                                                    [
                                                                                        [
                                                                                            [
                                                                                                [
                                                                                                    [
                                                                                                        [
                                                                                                            [
                                                                                                                "TOK_NUMBER '10' [43]"
                                                                                                            ]
                                                                                                        ]
                                                                                                    ]
                                                                                                ]
                                                                                            ]
                                                                                        ]
                                                                                    ]
                                                                                ]
                                                                            ]
                                                                        ]
                                                                    ]
                                                                ]
                                                            ]
                                                        ]
                                                    ]
                                                ]
                                            ]
                                        ],
                                        [
                                            "TOK_ENDPAREN ')' [45]"
                                        ],
                                        [
                                            [
                                                "empty_statement ';' [46]"
                                            ]
                                        ]
                                    ]
                                ]
                            ]
                        ],
                        [
                            "TOK_ENDBRACE '}' [47]"
                        ]
                    ]
                ]
            ]
        ]
    ],
    [
        "EOF [48]"
    ]
]

What were some of the decisions made when writing these tests cases originally as to what should vs should not be tested in each module?