EDIT: I know, my database isn't normalized, but I can't change it, thanks.
I have a table called
id, user_id, dialog_id, type, status, created_at
user_id IN(1,2) and type='PRIVATE'
My understanding is that you want to find dialog_id's shared between user 1 and user 2 with type='PRIVATE' for both.
One solution is to search for all dialog_id's for the user 1 with type = 'PRIVATE', then search dialog_id's for user 2 with type = 'PRIVATE' which are in common with user 1
SELECT ud.dialog_id FROM user_dialogs ud WHERE type='PRIVATE' AND user_id = 2 AND ud.dialog_id IN ( SELECT dialog_id FROM user_dialogs WHERE type='PRIVATE' AND user_id = 1)