ODBC Error: Invalid character value for cast specification

[RickL18393]

I am using ODBC driver version 8.15.1-windows-x86_64 connecting to an ElasticSearch server running version 8.15.1. When I attempt to query data from an ElasticSearch table I am getting this ODBC error: [Elastic][EsODBC 8.15.1(v8.15.0-1-g43209d3,u,r) Driver]Invalid character value for cast specification;

When I turn on ODBC error logging I get this message: Mon Nov 18 12:12:53 2024 - [ERROR] sql2c_string()@convert.c:3489 [STMT@0x0000017FEFACF0B0] can't convert Soham Sarkar to double.

Is there a way to determine which column the bad data is in from the ODBC error message?

Let me know if you need any more information.

Thanks,
Rick L.

[bpintea]

You should get a message like column #<number> is bound, copying data. before the failure message, which should help you id the column.
Please feel free to use https://discuss.elastic.co/ for getting help, or open issues here if you encounter defects. Thanks!

[RickL18393]

Which logging level should I use? The error message I sent was the only line in the log file when I had the logging level set to Error. Thanks, Rick L. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Bogdan Pintea ***@***.***> Sent: Tuesday, November 19, 2024 10:57:12 AM To: elastic/elasticsearch-sql-odbc ***@***.***> Cc: Leasure, Rick ***@***.***>; Author ***@***.***> Subject: Re: [elastic/elasticsearch-sql-odbc] ODBC Error: Invalid character value for cast specification (Issue #327) Caution: External email. Do not open attachments or click on links if you do not recognize the sender. Closed #327<https://urldefense.com/v3/__https://github.com/elastic/elasticsearch-sql-odbc/issues/327__;!!IqUcNYopQPk7!O20clEdM5-rfuBda2DbpMOLJ-vFzpJDkHOfzeaPK9FH7-SaMqDARIuzZSMe5OVXWtMeHUsLRF6ipJWWyQU2zy8jl9xPdZg$> as not planned. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/elastic/elasticsearch-sql-odbc/issues/327*event-15351772563__;Iw!!IqUcNYopQPk7!O20clEdM5-rfuBda2DbpMOLJ-vFzpJDkHOfzeaPK9FH7-SaMqDARIuzZSMe5OVXWtMeHUsLRF6ipJWWyQU2zy8i51V3YIg$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BNCAB2C2KIUUCBVOTF34RTD2BNNVRAVCNFSM6AAAAABSCHMK2GVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJVGM2TCNZXGI2TMMY__;!!IqUcNYopQPk7!O20clEdM5-rfuBda2DbpMOLJ-vFzpJDkHOfzeaPK9FH7-SaMqDARIuzZSMe5OVXWtMeHUsLRF6ipJWWyQU2zy8hfDMs7PQ$>. You are receiving this because you authored the thread.Message ID: ***@***.***> This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or intended recipient’s authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.

[bpintea]

Which logging level should I use?

Debug

[RickL18393]

In the attached log I found the error message, Wed Nov 20 07:20:57 2024 - [ERROR] sql2c_string()@convert.c:3489 [STMT@0x00000230C2E6C6F0] can't convert Soham Sarkar to double, at row 7594. Looking at the previous rows, it appears that the #3 column contains the invalid data. However, I have checked the third and fourth (In case this is a zero based index) columns and neither of them are defined as a numeric data type so I do not understand why the ElasticSearch ODBC driver is attempting to convert a character string into a double.

A little more background, I am not pulling all the columns available in the table. I am assuming that the column # is assigned to the third column I selected and not the actual third column in the table. Is this correct? I have checked the actual third column in the table, which is a REAL/float data type, but it does not contain any character string data that I can see.

Am I reading this log correctly?

Thanks,
Rick L.
esodbc_20241120072056_8c37f8d3110e45509651c681c5b9d344.ece.optum.com_9243_20420-1.log

[bpintea]

The logs contain the following relevant entries:

Wed Nov 20 07:20:57 2024 - [DEBUG] SQLGetData()@odbc.c:947 ENTER: *StatementHandle=@0x00000230C2E6C6F0, ColumnNumber=3, TargetType=7, *TargetValuePtr=@0x00000230C3A60534, BufferLength=51, *StrLen_or_IndPtr=@0x000000BA493FDF48.
...
Wed Nov 20 07:20:57 2024 - [DEBUG] get_rec_c_type()@convert.c:581 [DESC@0x000000BA493F5470] AxD data C type: 7.
Wed Nov 20 07:20:57 2024 - [DEBUG] sql2c_string()@convert.c:3486 [STMT@0x00000230C2E6C6F0] string `Soham Sarkar` converted to dbl=0.000000e+00.
Wed Nov 20 07:20:57 2024 - [ERROR] sql2c_string()@convert.c:3489 [STMT@0x00000230C2E6C6F0] can't convert `Soham Sarkar` to double.

The first line logs the parameters received from the application, with no manipulation (i.e. before copying anything in internal structures). The key element there is TargetType=7, which means the application wants to have the result provided as an SQL real type (SQL_C_FLOAT, which is same as SQL_REAL).

The ColumnNumber=3 means the 3rd column in the result set (not the source table), 1-based. (The function's documented here.)

[RickL18393]

I am using Microsoft SQL Server SSIS (also known as Microsoft SQL Server Data Tools) 2019 to pull the data from ElasticSearch. Looking at the ODBC Source task, the third result set column (called Output Column in SSIS) is set to data type text stream [DT_TEXT]: ***@***.*** However, when I look at the External Columns, the third column is amountExpected, which is data type float [DT_R4]: ***@***.*** I am not trying to pull amountExpected. Is it possible that the ODBC driver is actually looking at the External Columns, which are all the columns available in the index, instead of just the columns I selected which are represented by the Output Columns? Let me know if you need any more information. Thanks, Rick L. From: Bogdan Pintea ***@***.***> Sent: Thursday, December 12, 2024 12:29 PM To: elastic/elasticsearch-sql-odbc ***@***.***> Cc: Leasure, Rick ***@***.***>; Author ***@***.***> Subject: Re: [elastic/elasticsearch-sql-odbc] ODBC Error: Invalid character value for cast specification (Issue #327) Caution: External email. Do not open attachments or click on links if you do not recognize the sender. The logs contain the following relevant entries: Wed Nov 20 07:20:57 2024 - [DEBUG] ***@***.***:947 ENTER: ***@***.***, ColumnNumber=3, TargetType=7, ***@***.***, BufferLength=51, ***@***.*** ... Wed Nov 20 07:20:57 2024 - [DEBUG] ***@***.***:581 ***@***.*** AxD data C type: 7. Wed Nov 20 07:20:57 2024 - [DEBUG] ***@***.***:3486 ***@***.*** string `Soham Sarkar` converted to dbl=0.000000e+00. Wed Nov 20 07:20:57 2024 - [ERROR] ***@***.***:3489 ***@***.*** can't convert `Soham Sarkar` to double. The first line logs the parameters received from the application, with no manipulation (i.e. before copying anything in internal structures). The key element there is TargetType=7, which means the application wants to have the result provided as an SQL real type (SQL_C_FLOAT<https://urldefense.com/v3/__https:/github.com/microsoft/ODBC-Specification/blob/4dda95986bda5d3b55d7749315d3e5a0951c1e50/Windows/inc/sqlext.h*L553C9-L553C20__;Iw!!IqUcNYopQPk7!MrdqNX7hR5aTqmHxGPUaNIsbWPlE8ONTSKjZDlE2bxZO1BAgXm2jAujHPvgrXfH3mVGzCCPu9vh0_lD99Le3ca7mCT__aA$>, which is same as SQL_REAL<https://urldefense.com/v3/__https:/github.com/microsoft/ODBC-Specification/blob/4dda95986bda5d3b55d7749315d3e5a0951c1e50/Windows/inc/sql.h*L206__;Iw!!IqUcNYopQPk7!MrdqNX7hR5aTqmHxGPUaNIsbWPlE8ONTSKjZDlE2bxZO1BAgXm2jAujHPvgrXfH3mVGzCCPu9vh0_lD99Le3ca6ScOqw1g$>). The ColumnNumber=3 means the 3rd column in the result set (not the source table), 1-based. (The function's documented here<https://urldefense.com/v3/__https:/learn.microsoft.com/en-us/sql/odbc/reference/syntax/sqlgetdata-function__;!!IqUcNYopQPk7!MrdqNX7hR5aTqmHxGPUaNIsbWPlE8ONTSKjZDlE2bxZO1BAgXm2jAujHPvgrXfH3mVGzCCPu9vh0_lD99Le3ca7_GaT4ag$>.) — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/elastic/elasticsearch-sql-odbc/issues/327*issuecomment-2539575390__;Iw!!IqUcNYopQPk7!MrdqNX7hR5aTqmHxGPUaNIsbWPlE8ONTSKjZDlE2bxZO1BAgXm2jAujHPvgrXfH3mVGzCCPu9vh0_lD99Le3ca7ZDFrmRQ$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BNCAB2BTTAARQ7R2MAAWC2D2FHBVJAVCNFSM6AAAAABSCHMK2GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZZGU3TKMZZGA__;!!IqUcNYopQPk7!MrdqNX7hR5aTqmHxGPUaNIsbWPlE8ONTSKjZDlE2bxZO1BAgXm2jAujHPvgrXfH3mVGzCCPu9vh0_lD99Le3ca6IiHty9A$>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> This e-mail, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If the reader of this e-mail is not the intended recipient or intended recipient’s authorized agent, the reader is hereby notified that any dissemination, distribution or copying of this e-mail is prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and delete this e-mail immediately.

[bpintea]

@RickL18393, your email didn't make it through as you might have sent it. Have a look over the web, please.

[RickL18393]

I am using Microsoft SQL Server SSIS (also known as Microsoft SQL Server Data Tools) 2019 to pull the data from ElasticSearch. Looking at the ODBC Source task, the third result set column (called Output Column in SSIS) is set to data type text stream [DT_TEXT]:

However, when I look at the External Columns, the third column is amountExpected, which is data type float [DT_R4]:

I am not trying to pull amountExpected. Is it possible that the ODBC driver is actually looking at the External Columns, which are all the columns available in the index, instead of just the columns I selected which are represented by the Output Columns?

I also checked the amountExpected column using DBVis and the string that is causing the error is not in any of the rows for that column.

Let me know if you need any more information.

Thanks,
Rick L.

[bpintea]

the third result set column (called Output Column in SSIS) is set to data type text stream [DT_TEXT]:

Is that autodetected, or does the user set that?

Is it possible that the ODBC driver is actually looking at the External Columns, which are all the columns available in the index

No, the driver doesn't have access to any of that, it's just a proxy between the ODBC API / application and Elasticsearch'es SQL REST API and doesn't hold state more than connection and results-related.

The query is contained in the log (truncated excerpt):

Wed Nov 20 07:20:56 2024 - [WARN] EsSQLExecute()@queries.c:3723 [STMT@0x00000230C2E6C6F0] query `select "addnlDetails" ,"amountExpected" ,"analyst" , 

and so are the columns of the response (0-based indexing):

Wed Nov 20 07:20:57 2024 - [INFO] attach_columns_cbor()@queries.c:629 [STMT@0x00000230C2E6C6F0] columns received: 331.
Wed Nov 20 07:20:57 2024 - [DEBUG] update_rec_count()@handles.c:1470 [DESC@0x00000230C2E6CFF8] recs array is growing 0 -> 331.
Wed Nov 20 07:20:57 2024 - [DEBUG] attach_one_column()@queries.c:329 [STMT@0x00000230C2E6C6F0] column #0: name=`addnlDetails`, type=-9 (`text`).
Wed Nov 20 07:20:57 2024 - [DEBUG] attach_one_column()@queries.c:329 [STMT@0x00000230C2E6C6F0] column #1: name=`amountExpected`, type=7 (`float`).
Wed Nov 20 07:20:57 2024 - [DEBUG] attach_one_column()@queries.c:329 [STMT@0x00000230C2E6C6F0] column #2: name=`analyst`, type=-9 (`text`).

But as written above, SSIS requests for some reason the output to be converted to a floating point number (1-based):

Wed Nov 20 07:20:57 2024 - [DEBUG] SQLGetData()@odbc.c:947 ENTER: *StatementHandle=@0x00000230C2E6C6F0, ColumnNumber=3, TargetType=7, *TargetValuePtr=@0x00000230C3A60534, BufferLength=51, *StrLen_or_IndPtr=@0x000000BA493FDF48.

I also checked the amountExpected column using DBVis and the string that is causing the error is not in any of the rows for that column.

Indeed, the string that can't be converted to a number is sourced from the analyst field/column.

Does it work loading this data in other apps (like Excel or some other SQL editor, like DbVisualizer)?
What happens in SSIS if you limit the loaded fields/columns to only string ones, like just the first and third?
You could also try to set the Varchar limit to a non-0 value and observe if SSIS handles the field differently.

[RickL18393]

Is that autodetected, or does the user set that?

That column is autodetected as Unicode text stream [DT_NTEXT]. We do not need Unicode, so I changed it to text stream [DT_TEXT]

But as written above, SSIS requests for some reason the output to be converted to a floating point number (1-based):

I would suggest that this is the bug we are chasing, whether it be in the ODBC driver or SSIS.

Indeed, the string that can't be converted to a number is sourced from the analyst field/column.

Thank you for the clarification. That column does contain the string in question.

Does it work loading this data in other apps (like Excel or some other SQL editor, like DbVisualizer)?

I have not tried to pull the data with any other applications. My DbVis connection is using the JDBC driver, so I don't think that
would be a good comparison.

What happens in SSIS if you limit the loaded fields/columns to only string ones, like just the first and third?

I created a new Data Flow task and only selected the addnl and anaylst columns. I am now getting a different set of error
messages:

[ODBC Source [2]] Error: SQLSTATE: HY000, Message: [Elastic][EsODBC 8.15.1(v8.15.0-1-g43209d3,u,r) Driver]Invalid server answer;

[ODBC Source [2]] Error: SQLSTATE: HY000, Message: [Elastic][EsODBC 8.15.1(v8.15.0-1-g43209d3,u,r) Driver]Invalid server answer;

[ODBC Source [2]] Error: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "ODBC Source.Outputs[ODBC
Source Output]" failed because error code 0xC020F450 occurred, and the error row disposition on "ODBC Source" specifies failure
on error. An error occurred on the specified object of the specified component. There may be error messages posted before this
with more information about the failure.

[SSIS.Pipeline] Error: SSIS Error Code DTS_E_PRIMEOUTPUTFAILED. The PrimeOutput method on ODBC Source returned error
code 0xC0209029. The component returned a failure code when the pipeline engine called PrimeOutput(). The meaning of the
failure code is defined by the component, but the error is fatal and the pipeline stopped executing. There may be error messages
posted before this with more information about the failure.

You could also try to set the Varchar limit to a non-0 value and observe if SSIS handles the field differently.

Varchar Limit is already set to 8000.

[bpintea]

I changed it to text stream [DT_TEXT]

Is DT_WSTR/DT_STR making any difference? (Not very familiar with SSIS though.)

this is the bug we are chasing

Could be a bug, could also be a misconfiguration.

I am now getting a different set of error messages

Could you share the log files of the driver in this case too?