Compact or renumber IDs for all tables, and reset sequences to max(id)?

The question is old, but we got a new question from a desperate user on dba.SE after trying to apply what is suggested here. Find an answer with more details and explanation over there:

• Compacting a sequence in PostgreSQL

The currently accepted answer will fail for most cases.

• Typically, you have a PRIMARY KEY or UNIQUE constraint on an id column, which is NOT DEFERRABLE by default. (OP mentions references and constraints.) Such constraints are checked after each row, so you most likely get unique violation errors trying. Details:

  • Constraint defined DEFERRABLE INITIALLY IMMEDIATE is still DEFERRED?

• Typically, one wants to retain the original order of rows while closing gaps. But the order in which rows are updated is arbitrary, leading to arbitrary numbers. The demonstrated example seems to retain the original sequence because physical storage still coincides with the desired order (inserted rows in desired order just a moment earlier), which is almost never the case in real world applications and completely unreliable.

The matter is more complicated than it might seem at first. One solution (among others) if you can afford to remove the PK / UNIQUE constraint (and related FK constraints) temporarily:

BEGIN;

LOCK tbl;

-- remove all FK constraints to the column

ALTER TABLE tbl DROP CONSTRAINT tbl_pkey;  -- remove PK

-- for the simple case without FK references - or see below:    
UPDATE tbl t  -- intermediate unique violations are ignored now
SET    id = t1.new_id
FROM  (SELECT id, row_number() OVER (ORDER BY id) AS new_id FROM tbl) t1
WHERE  t.id = t1.id;

-- Update referencing value in FK columns at the same time (if any)

SELECT setval('tbl_id_seq', max(id)) FROM tbl;  -- reset sequence

ALTER TABLE tbl ADD CONSTRAINT tbl_pkey PRIMARY KEY(id); -- add PK back

-- add all FK constraints to the column back

COMMIT;

This is also much faster for big tables, because checking PK (and FK) constraint(s) for every row costs a lot more than removing the constraint(s) and adding it (them) back.

If there are FK columns in other tables referencing tbl.id, use data-modifying CTEs to update all of them.

Example for a table fk_tbl and a FK column fk_id:

WITH u1 AS (
   UPDATE tbl t
   SET    id = t1.new_id
   FROM  (SELECT id, row_number() OVER (ORDER BY id) AS new_id FROM tbl) t1
   WHERE  t.id = t1.id
   RETURNING t.id, t1.new_id  -- return old and new ID
   )
UPDATE fk_tbl f
SET    fk_id = u1.new_id      -- set to new ID
FROM   u1
WHERE  f.fk_id = u1.id;       -- match on old ID

More in the referenced answer on dba.SE.

Assuming your ids are generated from a bignum sequence, just RESTART the sequence and update the table with idcolumn = DEFAULT.

CAVEAT: If this id column is used as a foreign key by other tables, make sure you have the on update cascade modifier turned on.

For example:

Create the table, put some data in, and remove a middle value:

db=# create sequence xseq;
CREATE SEQUENCE
db=# create table foo ( id bigint default nextval('xseq') not null, data text );
CREATE TABLE
db=# insert into foo (data) values ('hello'), ('world'), ('how'), ('are'), ('you');
INSERT 0 5
db=# delete from foo where data = 'how';
DELETE 1
db=# select * from foo;
 id | data  
----+-------
  1 | hello
  2 | world
  4 | are
  5 | you
(4 rows)

Reset your sequence:

db=# ALTER SEQUENCE xseq RESTART;
ALTER SEQUENCE

Update your data:

db=# update foo set id = DEFAULT;
UPDATE 4
db=# select * from foo;
 id | data  
----+-------
  1 | hello
  2 | world
  3 | are
  4 | you
(4 rows)

new id column and Foreign Key(s) while the old ones are still in use. With some (quick) renaming, applications do not have to be aware. (But applications should be inactive during the final renaming step)

\i tmp.sql
    -- the test tables
CREATE TABLE one (
    id serial NOT NULL PRIMARY KEY
    , payload text
    );
CREATE TABLE two (
    id serial NOT NULL PRIMARY KEY
    , the_fk INTEGER REFERENCES one(id)
            ON UPDATE CASCADE ON DELETE CASCADE
    );
    -- And the supporting index for the FK ...
CREATE INDEX ON two(the_fk);

    -- populate
INSERT INTO one(payload)
SELECT x::text FROM generate_series(1,1000) x;

INSERT INTO two(the_fk)
SELECT id FROM one WHERE random() < 0.3;

    -- make some gaps
DELETE FROM one WHERE id % 13 > 0;

-- SELECT * FROM two;

    -- Add new keycolumns to one and two
ALTER TABLE one
    ADD COLUMN new_id SERIAL NOT NULL UNIQUE
    ;

    -- UPDATE:
    -- This could need DEFERRABLE
    -- Note since the update is only a permutation of the
    -- existing values, we dont need to reset the sequence.
UPDATE one SET new_id = self.new_id
FROM ( SELECT id, row_number() OVER(ORDER BY id) AS new_id FROM one ) self
WHERE one.id = self.id;

ALTER TABLE two
    ADD COLUMN new_fk INTEGER REFERENCES one(new_id)
    ;

    -- update the new FK
UPDATE two t
SET new_fk = o.new_id
FROM one o
WHERE t.the_fk = o.id
    ;

SELECT * FROM two;

    -- The crucial part: the final renaming
    -- (at this point it would be better not to allow other sessions
    -- messing with the {one,two} tables ...
    -- --------------------------------------------------------------
ALTER TABLE one DROP COLUMN id CASCADE;
ALTER TABLE one rename COLUMN new_id TO id;
ALTER TABLE one ADD PRIMARY KEY(id);

ALTER TABLE two DROP COLUMN the_fk CASCADE;
ALTER TABLE two rename COLUMN new_fk TO the_fk;
CREATE INDEX ON two(the_fk);

    -- Some checks.
    -- (the automatically generated names for the indexes
    -- and the sequence still contain the "new" names.)
SELECT * FROM two;
\d one
\d two

---

UPDATE: added the permutation of new_id (after creating it as a serial) Funny thing is: it doesn't seem to need 'DEFERRABLE'.