Skip to content

Commit 23bcde9

Browse files
nmemondnmemond
authored
Add documentation for how to run integration tests locally (GoogleCloudPlatform#1228)
* Add documentation for how to run integration tests locally * Fix lint and integration test workflows * Allow Go version 1.24 for lint tool (latest staticcheck requires at least 1.24)
1 parent dedefca commit 23bcde9

3 files changed

Lines changed: 127 additions & 2 deletions

File tree

‎.github/workflows/lint.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ jobs:
2525
- uses: actions/checkout@v3
2626
- uses: actions/setup-go@v2
2727
with:
28-
go-version: '^1.22.7'
28+
# Need to use a version less than 1.25 due to https://github.com/golang/go/issues/74462.
29+
go-version: '>=1.23.0 <1.25.0'
2930
- uses: actions/setup-node@v2
3031
with:
3132
node-version: 18

‎docs/testing.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
# Running Integration Tests Locally
2+
3+
The integration tests involve performing actual migrations from a variety of sources (including actual databases),
4+
targeted towards a local emulated instance of spanner. The following describes the steps needed to set up all the
5+
resources required for running the integration tests.
6+
7+
## Setup Source Databases
8+
9+
The following integration tests expect an actually running database to use as the source:
10+
- `testing/dynamodb/streaming`
11+
- `testing/mysql`
12+
- `testing/oracle`
13+
- `testing/postgres`
14+
- `testing/sqlserver`
15+
16+
The schema and data for these source databases can be found in the `test_data` directory, as described below.
17+
18+
### DynamoDB
19+
20+
TODO
21+
22+
### MySQL
23+
24+
Start up a MySQL database server (either installed locally, via docker, an external server, etc.).
25+
26+
Initialize the database server with the following test data SQL scripts:
27+
- `test_data/mysqldump.test.out`: A database must already exist (no specific DB name required); populates existing
28+
database
29+
- `test_data/mysql_interleave_dump.test.out`: Creates and populates a database named `test_interleave_table_data`
30+
- `test_data/mysql_foreignkeyaction_dump.test.out`: Creates and populates a database named `test_foreign_key_action_data`
31+
- `test_data/mysql_checkconstraint_dump.test.out`: Creates and populates a database named `test_mysql_checkconstraint`
32+
33+
In the terminal from which you'll be running the tests, set the following environment variables:
34+
```sh
35+
export MYSQLHOST=<MySQL DB server host>
36+
export MYSQLPORT=<MySQL DB server port> # not required if default MySQL port of 3306 is being used
37+
export MYSQLUSER=<MySQL DB server username>
38+
export MYSQLPWD=<MySQL DB server password>
39+
export MYSQLDATABASE=test_interleave_table_data
40+
export MYSQLDB_FKACTION=test_foreign_key_action_data
41+
export MYSQLDB_CHECK_CONSTRAINT=test_mysql_checkconstraint
42+
```
43+
44+
### Oracle
45+
46+
Start up an Oracle Express Edition database server (either installed locally, via docker, an external server, etc.).
47+
48+
Initialize the database server with the following test data SQL scripts:
49+
- `test_data/oracle.test.out`: Creates user "sti" with password "test1" and populates tables for that user
50+
51+
### Postgres
52+
53+
Start up a PostgreSQL database server (either installed locally, via docker, an external server, etc.).
54+
55+
Initialize the database server with the following test data SQL scripts:
56+
- `test_data/pg_dump.test.out`: A database must already exist (initial postgres database is acceptable); populates
57+
existing database
58+
59+
In the terminal from which you'll be running the tests, set the following environment variables:
60+
```sh
61+
export PGHOST=<Postgres DB server host>
62+
export PGPORT=<Postgres DB server port>
63+
export PGUSER=<Postgres DB server user>
64+
export PGPASSWORD=<Postgres DB server password>
65+
export PGDATABASE=<Postgres DB server database name>
66+
```
67+
68+
### SQLServer
69+
70+
Start up a SQLServer database server (either installed locally, via docker, etc.). It must be available via hostname
71+
`localhost` and user "sa" must exist (these are hard-coded in the test).
72+
73+
Initialize the database server with the following test data SQL scripts:
74+
- `test_data/sqlserver.test.out`: Creates and populates a database named `SqlServer_IntTest`
75+
76+
In the terminal from which you'll be running the tests, set the following environment variables:
77+
```sh
78+
export MSSQL_SA_PASSWORD=<Password for user "sa">
79+
```
80+
81+
82+
## Setup the Spanner emulator
83+
84+
Install and setup the spanner emulator as described
85+
[here](https://docs.cloud.google.com/spanner/docs/emulator#emulator-for-gcloud).
86+
87+
Start the emulator:
88+
```sh
89+
gcloud emulators spanner start
90+
```
91+
92+
In another terminal, create a Spanner instance to use for testing (here we use the name test-instance, but this may be
93+
changed to any other name):
94+
```sh
95+
gcloud spanner instances create <instance name> --config=emulator-config --description=<brief instance description> --nodes=1
96+
```
97+
98+
In the terminal from which you'll be running the tests, set the following environment variables (note, the instance name
99+
is the same as the one created above):
100+
```sh
101+
export SPANNER_EMULATOR_HOST=localhost:9010
102+
export SPANNER_MIGRATION_TOOL_TESTS_GCLOUD_INSTANCE_ID=<instance name>
103+
export SPANNER_MIGRATION_TOOL_TESTS_GCLOUD_PROJECT_ID=<project id>
104+
```
105+
106+
## Running the Tests
107+
108+
The integration tests are all found in the `testing` directory. Prior to running the integration tests, ensure you have
109+
all required components (databases, emulated spanner, etc) up and running, and the relevant environment variables set as
110+
described above.
111+
112+
Note that if you are only running certain unit tests, you only need to perform the setup for those specific tests (for
113+
example, if you only want to run the the tests in `testing/mysql`, you only need to have a running MySQL instance with
114+
the relevant environment variables set, there's no need to have any other database running).
115+
116+
Once you have everything setup as required, run the integration tests as follows:
117+
```sh
118+
go test -v ./testing/...
119+
```
120+
121+
Note that running all tests (using `go test -v ./...`) will also run the integration tests; if the
122+
`SPANNER_EMULATOR_HOST` environment variable is **not** set, the integration tests will simply be skipped.

‎testing/import_cmd/integration_test.go‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,9 @@ type testStruct struct {
2828
sourceFormat string
2929
}
3030

31-
var expectedMysqlDumpDDL = "CREATE INDEX fk_author_id_11 ON Books(author_id)|CREATE INDEX fk_book_authors_author_id_5 ON BookAuthors(author_id)|CREATE INDEX fk_customer_id_21 ON Orders(customer_id)|CREATE INDEX fk_genre_id_13 ON Books(genre_id)|CREATE INDEX fk_publisher_id_12 ON Books(publisher_id)|CREATE INDEX fk_reviews_customer_id_29 ON Reviews(customer_id)|CREATE TABLE Authors ( author_id INT64 NOT NULL, first_name STRING(50) NOT NULL, last_name STRING(50) NOT NULL, biography STRING(MAX),) PRIMARY KEY(author_id)|CREATE TABLE BookAuthors ( book_author_id INT64 NOT NULL, book_id INT64 NOT NULL, author_id INT64 NOT NULL,) PRIMARY KEY(book_author_id)|CREATE TABLE Books ( book_id INT64 NOT NULL, title STRING(255) NOT NULL, isbn STRING(20) NOT NULL, publication_date DATE, price NUMERIC NOT NULL, stock_quantity INT64, author_id INT64, publisher_id INT64, genre_id INT64, is_featured BOOL, is_available BOOL, edition_number BOOL, page_count INT64, cover_image_binary BYTES(100), cover_image_varbinary BYTES(200), abstract_blob BYTES(MAX), sample_chapter_mediumblob BYTES(MAX), notes_tinyblob BYTES(MAX), full_text_longblob BYTES(MAX), flags BYTES(MAX), series_code STRING(10), volume_code STRING(5), last_updated TIMESTAMP, discount_rate NUMERIC, special_price NUMERIC, average_rating FLOAT64, binding_type STRING(MAX), weight_grams FLOAT32, internal_id INT64, catalog_id INT64, shelf_number INT64, inventory_code INT64, metadata JSON, keywords STRING(MAX), description_text STRING(MAX), long_description_mediumtext STRING(MAX), short_summary_tinytext STRING(MAX), full_review_longtext STRING(MAX), last_modified TIMESTAMP, product_code STRING(50), ean_code STRING(13),) PRIMARY KEY(book_id)|CREATE TABLE Customers ( customer_id INT64 NOT NULL, first_name STRING(50) NOT NULL, last_name STRING(50) NOT NULL, email STRING(100) NOT NULL, phone_number STRING(20), address STRING(MAX), registration_date TIMESTAMP,) PRIMARY KEY(customer_id)|CREATE TABLE Genres ( genre_id INT64 NOT NULL, name STRING(50) NOT NULL,) PRIMARY KEY(genre_id)|CREATE TABLE OrderItems ( order_item_id INT64 NOT NULL, order_id INT64 NOT NULL, book_id INT64 NOT NULL, quantity INT64 NOT NULL, price_at_purchase NUMERIC NOT NULL,) PRIMARY KEY(order_item_id)|CREATE TABLE Orders ( order_id INT64 NOT NULL, customer_id INT64 NOT NULL, order_date TIMESTAMP, shipping_address STRING(MAX) NOT NULL, order_status STRING(MAX), total_amount NUMERIC NOT NULL,) PRIMARY KEY(order_id)|CREATE TABLE Publishers ( publisher_id INT64 NOT NULL, name STRING(100) NOT NULL, address STRING(MAX),) PRIMARY KEY(publisher_id)|CREATE TABLE Reviews ( review_id INT64 NOT NULL, book_id INT64 NOT NULL, customer_id INT64 NOT NULL, rating INT64, comment STRING(MAX), review_date TIMESTAMP, CONSTRAINT reviews_chk_1 CHECK((rating>=1) AND (rating<=5)),) PRIMARY KEY(review_id)|CREATE UNIQUE INDEX book_id ON BookAuthors(book_id, author_id)|CREATE UNIQUE INDEX book_id_28 ON Reviews(book_id, customer_id)|CREATE UNIQUE INDEX email ON Customers(email)|CREATE UNIQUE INDEX isbn ON Books(isbn)|CREATE UNIQUE INDEX name ON Genres(name)|CREATE UNIQUE INDEX name_23 ON Publishers(name)"
31+
// TODO review expected values of sample_chapter_mediumblob and full_text_longblob. They both have value 10 * 2^20
32+
// instead of the implemented values (mediumblob 2^24 - 1, longblob 2^32 -1)
33+
var expectedMysqlDumpDDL = "CREATE INDEX fk_author_id_11 ON Books(author_id)|CREATE INDEX fk_book_authors_author_id_5 ON BookAuthors(author_id)|CREATE INDEX fk_customer_id_21 ON Orders(customer_id)|CREATE INDEX fk_genre_id_13 ON Books(genre_id)|CREATE INDEX fk_publisher_id_12 ON Books(publisher_id)|CREATE INDEX fk_reviews_customer_id_29 ON Reviews(customer_id)|CREATE TABLE Authors ( author_id INT64 NOT NULL, first_name STRING(50) NOT NULL, last_name STRING(50) NOT NULL, biography STRING(MAX),) PRIMARY KEY(author_id)|CREATE TABLE BookAuthors ( book_author_id INT64 NOT NULL, book_id INT64 NOT NULL, author_id INT64 NOT NULL,) PRIMARY KEY(book_author_id)|CREATE TABLE Books ( book_id INT64 NOT NULL, title STRING(255) NOT NULL, isbn STRING(20) NOT NULL, publication_date DATE, price NUMERIC NOT NULL, stock_quantity INT64, author_id INT64, publisher_id INT64, genre_id INT64, is_featured BOOL, is_available BOOL, edition_number BOOL, page_count INT64, cover_image_binary BYTES(100), cover_image_varbinary BYTES(200), abstract_blob BYTES(65535), sample_chapter_mediumblob BYTES(10485760), notes_tinyblob BYTES(255), full_text_longblob BYTES(10485760), flags BYTES(MAX), series_code STRING(10), volume_code STRING(5), last_updated TIMESTAMP, discount_rate NUMERIC, special_price NUMERIC, average_rating FLOAT64, binding_type STRING(MAX), weight_grams FLOAT32, internal_id INT64, catalog_id INT64, shelf_number INT64, inventory_code INT64, metadata JSON, keywords STRING(MAX), description_text STRING(MAX), long_description_mediumtext STRING(MAX), short_summary_tinytext STRING(MAX), full_review_longtext STRING(MAX), last_modified TIMESTAMP, product_code STRING(50), ean_code STRING(13),) PRIMARY KEY(book_id)|CREATE TABLE Customers ( customer_id INT64 NOT NULL, first_name STRING(50) NOT NULL, last_name STRING(50) NOT NULL, email STRING(100) NOT NULL, phone_number STRING(20), address STRING(MAX), registration_date TIMESTAMP,) PRIMARY KEY(customer_id)|CREATE TABLE Genres ( genre_id INT64 NOT NULL, name STRING(50) NOT NULL,) PRIMARY KEY(genre_id)|CREATE TABLE OrderItems ( order_item_id INT64 NOT NULL, order_id INT64 NOT NULL, book_id INT64 NOT NULL, quantity INT64 NOT NULL, price_at_purchase NUMERIC NOT NULL,) PRIMARY KEY(order_item_id)|CREATE TABLE Orders ( order_id INT64 NOT NULL, customer_id INT64 NOT NULL, order_date TIMESTAMP, shipping_address STRING(MAX) NOT NULL, order_status STRING(MAX), total_amount NUMERIC NOT NULL,) PRIMARY KEY(order_id)|CREATE TABLE Publishers ( publisher_id INT64 NOT NULL, name STRING(100) NOT NULL, address STRING(MAX),) PRIMARY KEY(publisher_id)|CREATE TABLE Reviews ( review_id INT64 NOT NULL, book_id INT64 NOT NULL, customer_id INT64 NOT NULL, rating INT64, comment STRING(MAX), review_date TIMESTAMP, CONSTRAINT reviews_chk_1 CHECK((rating>=1) AND (rating<=5)),) PRIMARY KEY(review_id)|CREATE UNIQUE INDEX book_id ON BookAuthors(book_id, author_id)|CREATE UNIQUE INDEX book_id_28 ON Reviews(book_id, customer_id)|CREATE UNIQUE INDEX email ON Customers(email)|CREATE UNIQUE INDEX isbn ON Books(isbn)|CREATE UNIQUE INDEX name ON Genres(name)|CREATE UNIQUE INDEX name_23 ON Publishers(name)"
3234
var expectedMysqlDumpCustomerRow = "string_value:\"1\"string_value:\"Christopher\"string_value:\"Miller\"string_value:\"christopher.miller0@example.com\"string_value:\"123-456-4205\"string_value:\"57 Main St, Anytown, CA 18404\"string_value:\"2025-04-08T08:40:54Z\""
3335

3436
var (

0 commit comments

Comments
 (0)