Foreign characters cause failure in python3

[cortlandstarrett]

The following line is in a marking file that gets included during gen_erate translation.

.// 以下 does not exit になるやつ

On Linux, gen_erate deals with this just fine. On Windows, I see a character mapping error.

q.main.arc: 19:  ERROR:  'charmap' codec can't decode byte 0x81 in position 26: character maps to 
Traceback  (most recent call last):
    File "C:\Users\cort\workspace2016_2\MicrowaveOven\gen\code_generation\arc\sys.arc", line 73
    File "C:\Users\cort\workspace2016_2\MicrowaveOven\gen\code_generation\arc\q.main.arc", line 19
'charmap' codec can't decode byte 0x81 in position 26: character maps to 

Do you have advice?

[cortlandstarrett]

http://stackoverflow.com/questions/9233027/unicodedecodeerror-charmap-codec-cant-decode-byte-x-in-position-y-character

[cortlandstarrett]

The encoding in the example above seems to be cp932.

[cortlandstarrett]

I am learning way to recognize the encoding, coerce it, etc. Nothing I have found looks clean, yet.

I wonder about discarding comments early on somehow.

[cortlandstarrett]

narrowing further
The problem happens only with the Windows-packaged gen_erate.exe. With Python installed, the encoding seems to be fine.

[cortlandstarrett]

Notice the simple command line. The datatype.mark file consists of just 1 line (having the line shown in a previous comment).

gen_erate.exe -arch datatype.mark
Traceback (most recent call last):
  File "gen_erate.py", line 255, in 
  File "gen_erate.py", line 236, in main
  File "C:\projects\pyrsl\rsl\parse.py", line 1352, in parse_file
  File "C:\projects\pyrsl\rsl\parse.py", line 137, in filename_input
  File "C:\Python34\lib\encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 26: character maps to <undefined>

[cortlandstarrett]

A few more data points:
Test case now has 2 lines to better see successful execution:

.// 以下 does not exit になるやつ
.print "hello"

From the Cygwin command line, I run:
gen_erate.exe -arch datatype.mark <--- fails
gen_erate.pyz -arch datatype.mark <--- succeeds (and prints hello)

[john-tornblom]

could you try to change the mode in which files are opened for parsing to binary mode:

In rsl/parse.py

    def filename_input(self, filename):
        with open(filename, 'rb') as f:
            return self.text_input(f.read(), filename)

[cortlandstarrett]

I will do this. I am still setting up my gen_erate.exe packaging environment.

[cortlandstarrett]

This has been working. And the input archetypes clearly have wide characters.

However, I ran into one more case:
fx_domain_debug_utility.arc: 676: INFO: File '....\myGenDir\3020_src\IOTsc_dbg_utility_full.c' CREATED
fx_domain_debug_utility.arc: 676: ERROR: 'charmap' codec can't encode characters in position 4751-4753: character maps to
Traceback (most recent call last):
File "Z:\gitf\fxmc\models\IOTsc\gen\code_generation\arc\sys.arc", line 411
File "Z:\gitf\fxmc\models\IOTsc\gen\code_generation\arc\fx_domain_debug_utility.arc", line 676
'charmap' codec can't encode characters in position 4751-4753: character maps to
Code generation complete.
Sun Mar 20 05:30:47 2016

I wonder if there is another place where encoding needs to be coerced or forced? (in the emit operation)?

[john-tornblom]

Possibly. I've temporarily enabled the python stack trace on a feature branch at https://github.com/john-tornblom/pyrsl/tree/win32_py34_encoding This version should reveal more information on where the exception is encountered in the python code.
You'll see two stack traces printed to stdout, one with the rsl stack trace, and one from the python stack trace.

A binary is available at https://ci.appveyor.com/api/buildjobs/7ypg64cxpghrudiu/artifacts/dist/gen_erate.exe

[cortlandstarrett]

...re-running my test case now. I will report in about 5 hours. :)

[cortlandstarrett]

fx_domain_debug_utility.arc: 676: INFO: File '....\myGenDir\3020_src\IOTsc_dbg_utility_full.c' REPLACED
fx_domain_debug_utility.arc: 676: ERROR: 'charmap' codec can't encode characters in position 4751-4753: character maps to
Traceback (most recent call last):
File "Z:\gitf\fxmc\models\IOTsc\gen\code_generation\arc\sys.arc", line 411
File "Z:\gitf\fxmc\models\IOTsc\gen\code_generation\arc\fx_domain_debug_utility.arc", line 676
Traceback (most recent call last):
File "C:\projects\pyrsl\rsl\eval.py", line 45, in accept
File "C:\Python34\lib\site-packages\xtuml\tools.py", line 65, in accept
File "C:\projects\pyrsl\rsl\eval.py", line 360, in accept_EmitNode
File "C:\projects\pyrsl\rsl\runtime.py", line 249, in emit_buffer
File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 4751-4753: character maps to

'charmap' codec can't encode characters in position 4751-4753: character maps to
Code generation complete.
Sun Mar 20 14:52:13 2016

[john-tornblom]

Yepp, the emit method compares with existing data on disk, and thus may read non-utf8.

Updated binary available at https://ci.appveyor.com/api/buildjobs/409tpah04nvt2a6w/artifacts/dist/gen_erate.exe

[cortlandstarrett]

O.K. Thanks.
downloading, will have more results soon.

[cortlandstarrett]

Hmm, it seems to have behaved the same.

fx_domain_debug_utility.arc: 676: INFO: File '....\myGenDir\3020_src\IOTsc_dbg_utility_full.c' REPLACED
fx_domain_debug_utility.arc: 676: ERROR: 'charmap' codec can't encode characters in position 4751-4753: character maps to
Traceback (most recent call last):
File "Z:\gitf\fxmc\models\IOTsc\gen\code_generation\arc\sys.arc", line 411
File "Z:\gitf\fxmc\models\IOTsc\gen\code_generation\arc\fx_domain_debug_utility.arc", line 676
Traceback (most recent call last):
File "C:\projects\pyrsl\rsl\eval.py", line 45, in accept
File "C:\Python34\lib\site-packages\xtuml\tools.py", line 65, in accept
File "C:\projects\pyrsl\rsl\eval.py", line 360, in accept_EmitNode
File "C:\projects\pyrsl\rsl\runtime.py", line 249, in emit_buffer
File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 4751-4753: character maps to

'charmap' codec can't encode characters in position 4751-4753: character maps to
Code generation complete.
Sun Mar 20 20:22:32 2016

My gen_erate.exe binary has 5502877 bytes.