Postgres / SQL pattern for atomic SELECT and UPDATE or INSERT within a transaction

How to properly handle dummy values? The state column is not nullable

Don't use them. If you do, make it nullable and denote lack of user-group relation with a state is null.

Is this the right way to handle this scenario?
Is it "safe"?
Is there a better / easier solution?

If it works, it works. It's not unsafe but magic/dummy/flag/trip/rogue/signal/sentinel values aren't very elegant.

Make the selection, validation and upsert a single operation:
_{demo at db<>fiddle}

prepare find_validate_apply(int,int,text) as
with find as(
  select state
  from user_group_membership as f
  where $1=user_id and $2=group_id
  for update of f
  --limit 1--unncessary given the uniqueness and non-nullability
),empty_as_null as(--`coalesce()` for rows
 (select state from find)
  union(select null)
  order by 1 nulls last limit 1
),validate as(
  select exists(select from allowed_transitions as t
                where t.source_state is not distinct from found.state
                and t.target_state is not distinct from $3) 
         as is_transition_allowed
  from empty_as_null as found
),apply as(
  insert into user_group_membership
  select $1,$2,$3
  from validate 
  where is_transition_allowed
  on conflict(user_id,group_id)do update
  set state=$3
  returning *)
select*from apply;

Now each worker can wait for others to finish their whole thing before they attempt to apply their desired transition. Note that this doesn't require a dummy value or sentinel row - they lock each other out either in the initial select or in the final upsert (concurrent writes to the unique pk index), but the whole thing is one atomic operation so it doesn't matter which one exactly.

The demo uses an audit trigger to track operations on the table and dblink to run parallel workers - at the end you can see how they queued up and applied their changes one after another:

create function trg1_f()returns trigger as $f$
begin
  create table if not exists user_group_membership_history
  as select TG_OP,transaction_timestamp(),clock_timestamp()
           ,OLD.user_id as OLD_user_id
           ,OLD.group_id as OLD_group_id
           ,OLD.state as OLD_state
           ,NEW.* 
  limit 0;

  insert into user_group_membership_history
  select TG_OP,now(),clock_timestamp(),OLD.*,NEW.*;
  
  return new;
end $f$ language plpgsql;

create trigger trg1 
after insert or update or delete
on user_group_membership
for each row execute function trg1_f();

create extension dblink;
select dblink_connect('another_worker1','');--defaults to localhost here
select dblink_connect('another_worker2','');
select dblink_send_query('another_worker1',
  $q$ begin;
      select pg_sleep(0.5);
      select find_validate_apply(9,9,'invited');
      select pg_sleep(1);
      commit;
  $q$);

select dblink_send_query('another_worker2',
  $q$ begin;
      select pg_sleep(0.7);
      select find_validate_apply(9,9,'banned');
      select pg_sleep(1);
      commit;
  $q$);
select pg_sleep(4);

table user_group_membership_history;

The allowed_transitions idea is self-explanatory:

create table allowed_transitions(source_state,target_state)
as values
 (null,'invited'),(null, 'banned')
,('invited','joined'),('invited','banned')
,('joined','left'),('joined','banned')
,('left','invited'),('left','banned')
,('banned','invited'),('banned','left');
alter table allowed_transitions 
  add constraint uniq unique(source_state,target_state);