Postgresql and BLOBs - maximum size of bytea?

If you need to ensure that you don't upload the same image twice, you can create a unique index on the md5 (or some other hash) of the bytea:

create table a(a bytea);
create unique index a_bytea_unique_hash on a (md5(a));
insert into a values ('abc');
INSERT 0 1
insert into a values ('abc');
ERROR:  duplicate key value violates unique constraint "a_bytea_unique_hash"
DETAIL:  Key (md5(a))=(900150983cd24fb0d6963f7d28e17f72) already exists.

Apparently you have an index on that column (to be honest I'm surprised that you could create it - I would have expected Postgres to reject that).

An index on a bytea column does not really make sense. If you remove that index, you should be fine.

The real question is: why did you create an index on a column that stores binary data?

Max size of bytea

According to this old thread, maximum size for a field in Postgres is 1 GB.

The PostgreSQL version 12 protocol limits row size to 2 GiB minus message header when it is sent to the client (SELECTed). (The protocol uses 32-bit signed integers to denote message size.) No other limits found (another topic).

But largeobjects are stored as multiple bytea records so they not limited on such way. See this docs for them.