How to create a modal JDialog on top of another modal JDialog

It is now 2021 and so about 8.5 years after the question was posted, but back in 2012 and still in these days by far and away the easiest solution is to simply create the JDialog with a Dialog as owner. Java has 5 constructors with a Dialog owner to offer, next to 5 similar ones with a Frame owner and 5 with a Window owner.

The issue that the OP was probably faced with was the default template for a JDialog in Netbeans or some other IDE. Netbeans (12.2) offers this as a template for a new JDialog Form...:

public class NewJDialog extends javax.swing.JDialog {

    /**
     * Creates new form NewJDialog
     */
    public NewJDialog(java.awt.Frame parent, boolean modal) {
        super(parent, modal);
        initComponents();
    }

You can simply change the signature of the constructor to use a Dialog as owner instead of the default option of a Frame:

public class NewJDialog extends javax.swing.JDialog {

    /**
     * Creates new form NewJDialog
     */
    public NewJDialog(java.awt.Dialog parent, boolean modal) {
        super(parent, modal);
        initComponents();
    }

And away you go.

Not sure what exactly you have as a problem, but here is an example on how you can have multiple modal dialogs:

import java.awt.BorderLayout;
import java.awt.Window;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;

import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;

public class TestDialog {

    protected static void initUI() {
        JPanel pane = newPane("Label in frame");
        JFrame frame = new JFrame("Title");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.add(pane);
        frame.pack();
        frame.setVisible(true);

    }

    public static JPanel newPane(String labelText) {
        JPanel pane = new JPanel(new BorderLayout());
        pane.add(newLabel(labelText));
        pane.add(newButton("Open dialog"), BorderLayout.SOUTH);
        return pane;
    }

    private static JButton newButton(String label) {
        final JButton button = new JButton(label);
        button.addActionListener(new ActionListener() {

            @Override
            public void actionPerformed(ActionEvent e) {
                Window parentWindow = SwingUtilities.windowForComponent(button);
                JDialog dialog = new JDialog(parentWindow);
                dialog.setLocationRelativeTo(button);
                dialog.setModal(true);
                dialog.add(newPane("Label in dialog"));
                dialog.pack();
                dialog.setVisible(true);
            }
        });
        return button;
    }

    private static JLabel newLabel(String label) {
        JLabel l = new JLabel(label);
        l.setFont(l.getFont().deriveFont(24.0f));
        return l;
    }

    public static void main(String[] args) {
        SwingUtilities.invokeLater(new Runnable() {

            @Override
            public void run() {
                initUI();
            }
        });
    }
}

1.please read The New Modality API in Java SE 6

2.Is there a way to get the (J)Frame of a JDialog?

Window ancestor = SwingUtilities.getWindowAncestor(this);

or

Window ancestor = (Window) this.getTopLevelAncestor();