CrazyCasta CrazyCasta - 5 months ago 16
SQL Question

Fast query to do normalization on SQL data

I have some data that I want to normalize. Specifically I'm normalizing it so I can process the portions getting normalized without having to worry about duplicates. What I'm doing is:

INSERT INTO new_table (a, b, c)
SELECT DISTINCT a,b,c
FROM old_table;

UPDATE old_table
SET abc_id = new_table.id
FROM new_table
WHERE new_table.a = old_table.a
AND new_table.b = old_table.b
AND new_table.c = old_table.c;


First off, it seems as if there should be a better way of doing this. It seems that the inherent process of finding the distinct data could produce a list of the members that belong to it. Second, and more important, the INSERT takes a couple and the UPDATE takes FOREVER (I don't actually have a value for how long it takes yet because it's still running). I'm using postgresql. Is there a better way of doing this (perhaps all in one query).

Answer

This is my other answer, extended to three columns:

        -- Some test data
CREATE TABLE the_table
        ( id SERIAL NOT NULL PRIMARY KEY
        , name varchar
        , a INTEGER
        , b varchar
        , c varchar
        );
INSERT INTO the_table(name, a,b,c) VALUES
 ( 'Chimpanzee' , 1, 'mammals', 'apes' )
,( 'Urang Utang' , 1, 'mammals', 'apes' )
,( 'Homo Sapiens' , 1, 'mammals', 'apes' )
,( 'Mouse' , 2, 'mammals', 'rodents' )
,( 'Rat' , 2, 'mammals', 'rodents' )
,( 'Cat' , 3, 'mammals', 'felix' )
,( 'Dog' , 3, 'mammals', 'canae' )
        ;

        -- [empty] table to contain the "squeezed out" domain {a,b,c}
CREATE TABLE abc_table
        ( id SERIAL NOT NULL PRIMARY KEY
        , a INTEGER
        , b varchar
        , c varchar
        , UNIQUE (a,b,c)
        );

        -- The original table needs a "link" to the new table
ALTER TABLE the_table
        ADD column abc_id INTEGER -- NOT NULL
        REFERENCES abc_table(id)
        ;
        -- FK constraints are helped a lot by a supportive index.
CREATE INDEX abc_table_fk ON the_table (abc_id);

        -- Chained query to:
        -- * populate the domain table
        -- * initialize the FK column in the original table
WITH ins AS (
        INSERT INTO abc_table(a,b,c)
        SELECT DISTINCT a,b,c
        FROM the_table a
        RETURNING *
        )
UPDATE the_table ani
SET abc_id = ins.id
FROM ins
WHERE ins.a = ani.a
AND ins.b = ani.b
AND ins.c = ani.c
        ;

        -- Now that we have the FK pointing to the new table,
        -- we can drop the redundant columns.
ALTER TABLE the_table DROP COLUMN a, DROP COLUMN b, DROP COLUMN c;

SELECT * FROM the_table;
SELECT * FROM abc_table;

        -- show it to the world
SELECT a.*
        , c.a, c,b, c.c
FROM the_table a
JOIN abc_table c ON c.id = a.abc_id
        ;

Results:

CREATE TABLE
INSERT 0 7
CREATE TABLE
ALTER TABLE
CREATE INDEX
UPDATE 7
ALTER TABLE
 id |     name     | abc_id 
----+--------------+--------
  1 | Chimpanzee   |      4
  2 | Urang Utang  |      4
  3 | Homo Sapiens |      4
  4 | Mouse        |      3
  5 | Rat          |      3
  6 | Cat          |      1
  7 | Dog          |      2
(7 rows)

 id | a |    b    |    c    
----+---+---------+---------
  1 | 3 | mammals | felix
  2 | 3 | mammals | canae
  3 | 2 | mammals | rodents
  4 | 1 | mammals | apes
(4 rows)

 id |     name     | abc_id | a |    c    |    b    |    c    
----+--------------+--------+---+---------+---------+---------
  1 | Chimpanzee   |      4 | 1 | apes    | mammals | apes
  2 | Urang Utang  |      4 | 1 | apes    | mammals | apes
  3 | Homo Sapiens |      4 | 1 | apes    | mammals | apes
  4 | Mouse        |      3 | 2 | rodents | mammals | rodents
  5 | Rat          |      3 | 2 | rodents | mammals | rodents
  6 | Cat          |      1 | 3 | felix   | mammals | felix
  7 | Dog          |      2 | 3 | canae   | mammals | canae
(7 rows)

Edit: This seems to work well enough and I hate to see the down-vote I put there, so useless edit (CrazyCasta).

Comments