Skip to content

[Improvement] optimize structure migration statements #518

Description

@gnolong

Get-tables statements using information_schema has low performace,It is better to use pg_catalog.
this is just for one schema whith 1000 tables:

EXPLAIN (ANALYZE, BUFFERS)
SELECT c.table_schema,
       c.table_name,
       c.column_name,
       c.data_type,
       c.udt_name,
       c.character_maximum_length,
       c.is_nullable,
       c.column_default,
       c.numeric_precision,
       c.numeric_scale,
       c.is_identity,
       c.identity_generation,
       c.ordinal_position
FROM information_schema.columns c
JOIN information_schema.tables t
    ON c.table_schema = t.table_schema
   AND c.table_name = t.table_name
WHERE c.table_schema IN ('mock_db_1')
  AND t.table_type = 'BASE TABLE'
ORDER BY c.table_schema, c.table_name, c.ordinal_position;
                                                                                                                                                                      QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=1509.10..1509.11 rows=1 width=432) (actual time=128792.260..128796.189 rows=5304 loops=1)
   Sort Key: ((c.relname)::information_schema.sql_identifier), ((a.attnum)::information_schema.cardinal_number)
   Sort Method: quicksort  Memory: 1770kB
   Buffers: shared hit=1710447
   ->  Nested Loop Left Join  (cost=146.42..1509.09 rows=1 width=432) (actual time=22.900..128641.952 rows=5304 loops=1)
         Buffers: shared hit=1710447
         ->  Nested Loop Left Join  (cost=146.01..1506.30 rows=1 width=1060) (actual time=20.211..126137.271 rows=5304 loops=1)
               Join Filter: (dep.refobjsubid = a.attnum)
               Buffers: shared hit=1697967
               ->  Nested Loop  (cost=145.73..1504.92 rows=1 width=1064) (actual time=18.951..125574.356 rows=5304 loops=1)
                     Join Filter: (c.relname = c_1.relname)
                     Rows Removed by Join Filter: 3813576
                     Buffers: shared hit=1681697
                     ->  Nested Loop Left Join  (cost=1.48..259.97 rows=1 width=128) (actual time=0.402..167.673 rows=720 loops=1)
                           Buffers: shared hit=1533
                           ->  Hash Join  (cost=1.07..258.89 rows=1 width=132) (actual time=0.311..60.444 rows=720 loops=1)
                                 Hash Cond: (c_1.relnamespace = nc_1.oid)
                                 Join Filter: ((CASE WHEN (nc_1.oid = pg_my_temp_schema()) THEN 'LOCAL TEMPORARY'::text WHEN (c_1.relkind = ANY ('{r,p}'::"char"[])) THEN 'BASE TABLE'::text WHEN (c_1.relkind = 'v'::"char") THEN 'VIEW'::text WHEN (c_1.relkind = 'f'::"char") THEN 'FOREIGN'::text ELSE NULL::text END)::text = 'BASE TABLE'::text)
                                 Buffers: shared hit=93
                                 ->  Seq Scan on pg_class c_1  (cost=0.00..256.07 rows=655 width=73) (actual time=0.156..20.250 rows=931 loops=1)
                                       Filter: ((relkind = ANY ('{r,v,f,p}'::"char"[])) AND (pg_has_role(relowner, 'USAGE'::text) OR has_table_privilege(oid, 'SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER'::text) OR has_any_column_privilege(oid, 'SELECT, INSERT, UPDATE, REFERENCES'::text)))
                                       Rows Removed by Filter: 2715
                                       Buffers: shared hit=92
                                 ->  Hash  (cost=1.06..1.06 rows=1 width=68) (actual time=0.077..0.691 rows=1 loops=1)
                                       Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                       Buffers: shared hit=1
                                       ->  Seq Scan on pg_namespace nc_1  (cost=0.00..1.06 rows=1 width=68) (actual time=0.048..0.053 rows=1 loops=1)
                                             Filter: ((NOT pg_is_other_temp_schema(oid)) AND (nspname = 'mock_db_1'::name))
                                             Rows Removed by Filter: 4
                                             Buffers: shared hit=1
                           ->  Nested Loop  (cost=0.41..1.07 rows=1 width=4) (actual time=0.111..0.111 rows=0 loops=720)
                                 Buffers: shared hit=1440
                                 ->  Index Scan using pg_type_oid_index on pg_type t_1  (cost=0.28..0.91 rows=1 width=8) (actual time=0.087..0.087 rows=0 loops=720)
                                       Index Cond: (oid = c_1.reloftype)
                                       Buffers: shared hit=1440
                                 ->  Index Only Scan using pg_namespace_oid_index on pg_namespace nt_1  (cost=0.13..0.15 rows=1 width=4) (never executed)
                                       Index Cond: (oid = t_1.typnamespace)
                                       Heap Fetches: 0
                     ->  Hash Join  (cost=144.25..1239.30 rows=452 width=1064) (actual time=0.218..168.553 rows=5304 loops=720)
                           Hash Cond: (t.typnamespace = nt.oid)
                           Buffers: shared hit=1680164
                           ->  Hash Left Join  (cost=143.16..1235.77 rows=452 width=1004) (actual time=0.202..154.947 rows=5304 loops=720)
                                 Hash Cond: ((a.attrelid = ad.adrelid) AND (a.attnum = ad.adnum))
                                 Buffers: shared hit=1680163
                                 ->  Hash Left Join  (cost=143.14..1233.38 rows=452 width=972) (actual time=0.198..143.430 rows=5304 loops=720)
                                       Hash Cond: (t.typbasetype = bt.oid)
                                       Join Filter: (t.typtype = 'd'::"char")
                                       Buffers: shared hit=1680163
                                       ->  Nested Loop  (cost=1.65..1084.54 rows=452 width=838) (actual time=0.171..131.892 rows=5304 loops=720)
                                             Buffers: shared hit=1680079
                                             ->  Nested Loop  (cost=1.36..1030.92 rows=452 width=475) (actual time=0.159..100.086 rows=5304 loops=720)
                                                   Buffers: shared hit=1679761
                                                   ->  Hash Join  (cost=1.07..152.80 rows=233 width=136) (actual time=0.059..12.043 rows=720 loops=720)
                                                         Hash Cond: (c.relnamespace = nc.oid)
                                                         Buffers: shared hit=66241
                                                         ->  Seq Scan on pg_class c  (cost=0.00..146.69 rows=931 width=76) (actual time=0.050..9.690 rows=931 loops=720)
                                                               Filter: (relkind = ANY ('{r,v,f,p}'::"char"[]))
                                                               Rows Removed by Filter: 2715
                                                               Buffers: shared hit=66240
                                                         ->  Hash  (cost=1.06..1.06 rows=1 width=68) (actual time=0.245..0.248 rows=1 loops=1)
                                                               Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                                               Buffers: shared hit=1
                                                               ->  Seq Scan on pg_namespace nc  (cost=0.00..1.06 rows=1 width=68) (actual time=0.016..0.018 rows=1 loops=1)
                                                                     Filter: ((NOT pg_is_other_temp_schema(oid)) AND (nspname = 'mock_db_1'::name))
                                                                     Rows Removed by Filter: 4
                                                                     Buffers: shared hit=1
                                                   ->  Index Scan using pg_attribute_relid_attnum_index on pg_attribute a  (cost=0.29..3.75 rows=2 width=343) (actual time=0.044..0.107 rows=7 loops=518400)
                                                         Index Cond: ((attrelid = c.oid) AND (attnum > 0))
                                                         Filter: ((NOT attisdropped) AND (pg_has_role(c.relowner, 'USAGE'::text) OR has_column_privilege(c.oid, attnum, 'SELECT, INSERT, UPDATE, REFERENCES'::text)))
                                                         Buffers: shared hit=1613520
                                             ->  Memoize  (cost=0.29..0.34 rows=1 width=367) (actual time=0.002..0.002 rows=1 loops=3818880)
                                                   Cache Key: a.atttypid
                                                   Cache Mode: logical
                                                   Hits: 3818774  Misses: 106  Evictions: 0  Overflows: 0  Memory Usage: 39kB
                                                   Buffers: shared hit=318
                                                   ->  Index Scan using pg_type_oid_index on pg_type t  (cost=0.28..0.33 rows=1 width=367) (actual time=0.016..0.016 rows=1 loops=106)
                                                         Index Cond: (oid = a.atttypid)
                                                         Buffers: shared hit=318
                                       ->  Hash  (cost=115.78..115.78 rows=2057 width=138) (actual time=16.900..16.904 rows=2057 loops=1)
                                             Buckets: 4096  Batches: 1  Memory Usage: 378kB
                                             Buffers: shared hit=84
                                             ->  Hash Join  (cost=1.09..115.78 rows=2057 width=138) (actual time=0.155..12.995 rows=2057 loops=1)
                                                   Hash Cond: (bt.typnamespace = nbt.oid)
                                                   Buffers: shared hit=84
                                                   ->  Seq Scan on pg_type bt  (cost=0.00..103.57 rows=2057 width=78) (actual time=0.047..7.497 rows=2057 loops=1)
                                                         Buffers: shared hit=83
                                                   ->  Hash  (cost=1.04..1.04 rows=4 width=68) (actual time=0.037..0.038 rows=5 loops=1)
                                                         Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                                         Buffers: shared hit=1
                                                         ->  Seq Scan on pg_namespace nbt  (cost=0.00..1.04 rows=4 width=68) (actual time=0.012..0.018 rows=5 loops=1)
                                                               Buffers: shared hit=1
                                 ->  Hash  (cost=0.00..0.00 rows=1 width=38) (actual time=0.022..0.024 rows=0 loops=1)
                                       Buckets: 1024  Batches: 1  Memory Usage: 8kB
                                       ->  Seq Scan on pg_attrdef ad  (cost=0.00..0.00 rows=1 width=38) (actual time=0.021..0.021 rows=0 loops=1)
                           ->  Hash  (cost=1.04..1.04 rows=4 width=68) (actual time=0.032..0.033 rows=5 loops=1)
                                 Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                 Buffers: shared hit=1
                                 ->  Seq Scan on pg_namespace nt  (cost=0.00..1.04 rows=4 width=68) (actual time=0.010..0.015 rows=5 loops=1)
                                       Buffers: shared hit=1
               ->  Nested Loop  (cost=0.29..1.37 rows=1 width=8) (actual time=0.094..0.094 rows=0 loops=5304)
                     Join Filter: (dep.objid = seq.seqrelid)
                     Buffers: shared hit=16270
                     ->  Index Scan using pg_depend_reference_index on pg_depend dep  (cost=0.29..1.35 rows=1 width=12) (actual time=0.055..0.061 rows=1 loops=5304)
                           Index Cond: ((refclassid = '1259'::oid) AND (refobjid = c.oid))
                           Filter: ((classid = '1259'::oid) AND (deptype = 'i'::"char"))
                           Rows Removed by Filter: 3
                           Buffers: shared hit=16270
                     ->  Seq Scan on pg_sequence seq  (cost=0.00..0.00 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=5004)
         ->  Nested Loop  (cost=0.41..0.48 rows=1 width=4) (actual time=0.039..0.039 rows=0 loops=5304)
               Buffers: shared hit=12480
               ->  Index Scan using pg_collation_oid_index on pg_collation co  (cost=0.28..0.30 rows=1 width=72) (actual time=0.019..0.019 rows=0 loops=5304)
                     Index Cond: (oid = a.attcollation)
                     Buffers: shared hit=11232
               ->  Index Scan using pg_namespace_oid_index on pg_namespace nco  (cost=0.13..0.17 rows=1 width=68) (actual time=0.032..0.032 rows=0 loops=624)
                     Index Cond: (oid = co.collnamespace)
                     Filter: ((nspname <> 'pg_catalog'::name) OR (co.collname <> 'default'::name))
                     Rows Removed by Filter: 1
                     Buffers: shared hit=1248
 Planning:
   Buffers: shared hit=84
 Planning Time: 47.428 ms
 Execution Time: 128801.257 ms
(122 rows)

Time: 129003.590 ms (02:09.004)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions