SQL-864: Json datasets for TDVT

[nbagnard]

No description provided.

[mattChiaravalloti]

Big fan of the step-by-step detail in the README 🙏 There are a couple small things to clean up there, though!

[bucaojit]

Thanks for adding a tdvt readme!

A few suggestions.

[bucaojit]

Let's add a bit more detail to this line: such as for the local ADF the configuration change is in the file ./testdata/config/mongodb_local/tenant-config.json. For remote ADF ensure the database and collection names match the below configuration.

[bucaojit]

In this tdvt Readme it would be helpful to add the LogicalQueryFormat we use and the command to run with unsigned taco files:

[Datasource]
LogicalQueryFormat = simple
CommandLineOverride = -DDisableVerifyConnectorPluginSignature=true

[mattChiaravalloti]

And can we provide some context on what these are?

[bucaojit]

Suggested change

db.runCommand({ sqlSetSchema: "Staples", schema: { version: Long("1"), jsonSchema: { bsonType: ['object'], properties: { PID: { bsonType: ['int'] }, 'Ship Priority': { bsonType: ['string'] }, 'Total Cycle Time': { bsonType: ['double'] }, 'Employee Salary': { bsonType: ['decimal'] }, 'Ship Mode': { bsonType: ['string'] }, 'Product ID': { bsonType: ['string'] }, 'Gross Profit': { bsonType: ['double'] }, 'Product Name': { bsonType: ['string'] }, 'Fill Time': { bsonType: ['double'] }, 'Market Segment': { bsonType: ['string'] }, 'Ship Handle Cost': { bsonType: ['decimal'] }, 'Supplier Name': { bsonType: ['string'] }, 'Order Quantity': { bsonType: ['double'] }, 'Order Priority': { bsonType: ['string'] }, 'Supplier Balance': { bsonType: ['double'] }, 'Ship Date': { bsonType: ['date'] }, 'Supplier State': { bsonType: ['string'] }, 'Prod Type4': { bsonType: ['string'] }, 'Customer State': { bsonType: ['string'] }, 'Employee Yrs Exp': { bsonType: ['double'] }, 'Employee Dept': { bsonType: ['string'] }, 'Manager Name': { bsonType: ['string'] }, 'Product In Stock': { bsonType: ['string'] }, 'Supplier Region': { bsonType: ['string'] }, 'Receive Time': { bsonType: ['double'] }, 'Order Date': { bsonType: ['date'] }, 'Prod Type1': { bsonType: ['string'] }, 'Product Base Margin': { bsonType: ['double'] }, 'Order Quarter': { bsonType: ['string'] }, 'Customer Balance': { bsonType: ['double'] }, 'Ship Promo': { bsonType: ['string'] }, '': { bsonType: ['int'] }, 'Item Count': { bsonType: ['int'] }, 'Customer Name': { bsonType: ['string'] }, 'Sales Total': { bsonType: ['double'] }, 'Prod Type2': { bsonType: ['string'] }, Discount: { bsonType: ['double'] }, 'Received Date': { bsonType: ['date'] }, 'Order Year': { bsonType: ['int'] }, 'Customer Segment': { bsonType: ['string'] }, 'Ship Charge': { bsonType: ['decimal'] }, 'Tax Rate': { bsonType: ['double'] }, 'Employee Name': { bsonType: ['string'] }, 'Prod Type3': { bsonType: ['string'] }, 'Order ID': { bsonType: ['string'] }, _id: { bsonType: ['objectId'] }, 'Order Status': { bsonType: ['string'] }, Price: { bsonType: ['decimal'] }, 'Call Center Region': { bsonType: ['string'] }, 'Order Month': { bsonType: ['int'] }, 'Product Container': { bsonType: ['string'] } } } } })

db.runCommand({ sqlSetSchema: "Staples", schema: { version: NumberLong("1"), jsonSchema: { bsonType: ['object'], properties: { PID: { bsonType: ['int'] }, 'Ship Priority': { bsonType: ['string'] }, 'Total Cycle Time': { bsonType: ['double'] }, 'Employee Salary': { bsonType: ['decimal'] }, 'Ship Mode': { bsonType: ['string'] }, 'Product ID': { bsonType: ['string'] }, 'Gross Profit': { bsonType: ['double'] }, 'Product Name': { bsonType: ['string'] }, 'Fill Time': { bsonType: ['double'] }, 'Market Segment': { bsonType: ['string'] }, 'Ship Handle Cost': { bsonType: ['decimal'] }, 'Supplier Name': { bsonType: ['string'] }, 'Order Quantity': { bsonType: ['double'] }, 'Order Priority': { bsonType: ['string'] }, 'Supplier Balance': { bsonType: ['double'] }, 'Ship Date': { bsonType: ['date'] }, 'Supplier State': { bsonType: ['string'] }, 'Prod Type4': { bsonType: ['string'] }, 'Customer State': { bsonType: ['string'] }, 'Employee Yrs Exp': { bsonType: ['double'] }, 'Employee Dept': { bsonType: ['string'] }, 'Manager Name': { bsonType: ['string'] }, 'Product In Stock': { bsonType: ['string'] }, 'Supplier Region': { bsonType: ['string'] }, 'Receive Time': { bsonType: ['double'] }, 'Order Date': { bsonType: ['date'] }, 'Prod Type1': { bsonType: ['string'] }, 'Product Base Margin': { bsonType: ['double'] }, 'Order Quarter': { bsonType: ['string'] }, 'Customer Balance': { bsonType: ['double'] }, 'Ship Promo': { bsonType: ['string'] }, '': { bsonType: ['int'] }, 'Item Count': { bsonType: ['int'] }, 'Customer Name': { bsonType: ['string'] }, 'Sales Total': { bsonType: ['double'] }, 'Prod Type2': { bsonType: ['string'] }, Discount: { bsonType: ['double'] }, 'Received Date': { bsonType: ['date'] }, 'Order Year': { bsonType: ['int'] }, 'Customer Segment': { bsonType: ['string'] }, 'Ship Charge': { bsonType: ['decimal'] }, 'Tax Rate': { bsonType: ['double'] }, 'Employee Name': { bsonType: ['string'] }, 'Prod Type3': { bsonType: ['string'] }, 'Order ID': { bsonType: ['string'] }, _id: { bsonType: ['objectId'] }, 'Order Status': { bsonType: ['string'] }, Price: { bsonType: ['decimal'] }, 'Call Center Region': { bsonType: ['string'] }, 'Order Month': { bsonType: ['int'] }, 'Product Container': { bsonType: ['string'] } } } } })

Changing Long to NumberLong will allow the command to run correctly with mongo and mongosh.

[mattChiaravalloti]

Do we even need to wrap it in NumberLong? The sqlSetSchema command above for Calcs just does version: 1 without a wrapper at all

[bucaojit]

Oh that's a good point 😄. This does work with version: 1, no need to use NumberLong as I suggested.

[mattChiaravalloti]

almost there! I'll check back one more time to see what the new section Oliver proposed looks like.

[mattChiaravalloti]

And can we provide some context on what these are?

[mattChiaravalloti]

Do we even need to wrap it in NumberLong? The sqlSetSchema command above for Calcs just does version: 1 without a wrapper at all

[terakilobyte]

evergreen retry