Creating menus with options that select other menus, without causing stack overflow due to recursion

I've implemented a different method because, due to my inability, I can't find a way to modify your code and make it work.

First observation: when using a menu, selecting an item calls a method, making it very difficult to add items to it, which is why I'm disregarding this option.

First we create an enum with the names of the actions:

public enum Item {
   OPEN,
   CLOSE,
   EXIT,
   CLEAR,
   ETC, 
   SELECT,
   DELETE,
   CHANGE, 
   UP
}

This interface is used to group objects of the Menu class and the Action, since a menu can have both a Menu and an Action as an item.

public interface SubList {
   public void action();
   public String getName();
   public int getLen();   
}

The Action class does little more than implement the interface methods, within the contructor, if it has not already been created, it creates the Select object and instantiates action.

public class Action implements SubList {

   static Select select;
   Item action;
   
   public Action( Item act ) {
      action = act;
      if( select == null ) select = new Select();
   }
 
   @Override
   public void action() {
      select.selection( action );
   }

   @Override
   public String getName() {
      return action.toString().toLowerCase();
   }

   @Override
   public int getLen() {
      return 1;
   }
}

The Menu class has three attributes: an array of SubList items, an array of String with their names, and the name itself. All of these are instantiated in the constructor. It implements the interface methods and adds getSubItem() to retrieve a specific item from the list.

public class Menu implements SubList {

   SubList[] list;
   String names[];
   String name;
   
   public Menu( SubList sub[], String nams[], String nam ) {
      names = nams;
      list = sub;
      name = nam; 
   } 

   public SubList getSubItem( int position ) {
      return list[ position ];
   }

   @Override
   public void action() {
      System.out.println( "Chose your option: " );
      for( int i = 0; i < list.length; i++ ) {
         System.out.println((i + 1) + " -> " + list[ i ].getName() );
      }
   }

   @Override
   public String getName() {
      return name;
   }

   @Override
   public int getLen() {
      return names.length;
   }
}

The Select class, whose existence is due to my "mania" of having many small classes instead of few large ones, has the selection method, which receives an Item and consequently calls the corresponding method.

public class Select {

   public String selection( Item name ) {
      switch( name ) {
         case OPEN:
            return open();
         case CLEAR:
            return clear();
         case EXIT:
            return "exit";

         ... // others
         default:
            throw new AssertionError();
      }
   }

   boolean up() {
      System.out.println( "-> up" );
      return false;
   }

   boolean open() {
      System.out.println( "-> open" );
      return false;
   }

     // innecesary
     // boolean exit() {}

   ... // others
}

Finally, the Main has only two attributes, the Scanner object and a boolean, which are shared by all the show methods (I say "all the methods" because we are doing recursion), which allows us to finish the execution of all of them.

public class Main {

   Scanner sca = new Scanner( System.in );
   boolean toBeContinued = true;

   void init() {
      String namesMenu[] = { "Options", "Archive", "exit" };
      String namesSubMenuA[] = { "close", "open", "exit" };
      String namesSubMenuB[] = { "options", "clear", "etc", "exit" };
      String namesSubMenuC[] = { "select", "delete", "change", "exit" };
      Action[] acts = new Action[]{ new Action( CLOSE ), 
         new Action( OPEN ), new Action( CLEAR ), new Action( ETC ), 
         new Action( EXIT ), new Action( SELECT ), new Action( CHANGE ), 
         new Action( DELETE ), new Action( UP ) };

      SubList[] subOptions = { acts[ 5 ], acts[ 7 ], acts[ 6 ], acts[ 8 ], acts[ 4 ] };
      Menu opt = new Menu(
              subOptions,
              namesSubMenuB,
              "subOptions" );
      SubList[] actions = { acts[ 0 ], acts[ 1 ], acts[ 8 ], acts[ 4 ] };
      SubList[] options = { opt, acts[ 2 ], acts[ 3 ], acts[ 8 ], acts[ 4 ] };

      Menu menu = new Menu(
                      new SubList[]{
                         new Menu(
                                 actions,
                                 namesSubMenuA,
                                 namesMenu[ 0 ] ),
                         new Menu(
                                 options,
                                 namesSubMenuB,
                                 namesMenu[ 1 ] ),
                         new Menu(
                                 new SubList[]{
                                    acts[ 4 ] },
                                 new String[]{ "exit" },
                                 namesMenu[ 2 ] )
                      },
                      namesMenu,
                      "Menu Principal"
      );
      show( menu );
   }

   void show( Menu menu ) {
      boolean localContinue = true;
      int option;
      while( toBeContinued && localContinue ) {
         menu.action();
         option = sca.nextInt() - 1;
         SubList aux = menu.getSubItem( option );

         if( aux.getLen() == 1 ) {
            Action aux2;
            if( aux instanceof Menu ) {
               aux2 = (Action) ( (Menu) aux ).getSubItem( 0 );
            }
            else {
               aux2 = ( (Action) aux );
            }
            if( aux2.getName().equals( "exit" ) ) {
               toBeContinued = false;
            }
            if( aux2.getName().equals( "up" ) ) {
               localContinue = false;
            }
            aux2.action();
         }
         else {
            if( aux instanceof Action ) {
               ( (Action) aux ).action();
            }
            else {
               show( (Menu) aux );
            }
         }
      }
   }

   public static void main( String[] args ) {
      new Main().start();
   }
}

The init method first creates the arrays with the names of the items, then creates another array, which contains all the Item objects that exist, that way, we reuse them, instead of creating the menu object directly, I preferred to create the Menu opt that will end up being a submenu, and the different SubList that will be part of the different Menu (I think it's much clearer this way, at the cost of writing more code).

The show method instantiates localContinue as "true", to control the execution of each instance of it, within the while, we call the action of the received object, which will show us the list of options, we instantiate option with the user's response, and we instantiate aux with the corresponding item.

If the size of aux is "1", we instantiate aux2, determining if it is an object of the Menu or Action class, then we check if the Action name is "exit" we pass toBeContinued to "false" so that the execution ends, and if it is "up" we pass localContinued to false so that the current method ends.

Finally, if we do not enter the first if, we check if aux is an instance of Action, in which case we call its action() method, and if not, we invoke show passing it aux as a parameter.

PS: I'm sure this implementation can be improved, I just hope it helps you get ideas and build your own solution.