How does Google's custom iOS keyboard, Gboard, programmatically dismiss the frontmost app?

Question

Google's custom iOS app, Gboard, has an interesting feature that can't be accomplished using public APIs for in the iOS SDK (as of iOS 10). I'd like to know exactly how Google accomplishes the task of programmatically popping back one app in the App Switching stack in Gboard.

Custom iOS keyboards have two major components: the container app and the keyboard app extension. The keyboard app extension runs in a separate OS process that is started up whenever a user is in any app on their phone that requires text input.

These are the approximate steps that can be followed, using Gboard, to see the effect of programmatically returning to a previous app:

A user starts the Apple Messages app on their iPhone and taps a text field to begin entering text.
The Gboard keyboard extension is launched and the users sees the Gboard custom keyboard (while they are still in the Apple Messages app).
The user taps the microphone key inside the Gboard keyboard extension to do voice-to-text input.
Gboard uses a custom url scheme to launch the Gboard container app. The Gboard keyboard and Apple messages app are pushed down one layer in the App stack and the Gboard container app is now the frontmost app in the App stack. The Gboard container app uses the microphone to listen to the user's speech and translates it into text which it places onto the screen.
The user taps the "Done" button when they are satisfied with the text input they see on the screen.
This is where the magic happens… as the text input screen is dismissed, the Gboard container app is also dismissed automatically. The Gboard container app goes away and is replaced by the Apple Messages app (sometimes the Gboard keyboard extension process is still alive, sometimes it is relaunched, and sometimes it needs to be re-launched manually by tapping inside a text field.) . How does Google accomplish this?
Finally, the user sees the text that was just translated inserted automatically inside the text input field. Presumably Google accomplishes this by sharing data between the Gboard container app and the keyboard extension.

I would assume that Google is using private APIs by exploring the status bar's view hierarchy using Objective-C runtime introspection and somehow synthesizing tap events or calling an exposed target / action. I've explored this a very little and have been able to find interesting UIView subclasses inside the status bar, like UIStatusBarBreadcrumbItemView which contains an array of UISystemNavigationActions. I'm continuing to explore these classes in the hope that I can find some way of replicating the user interaction.

I understand that using private APIs is a good way to get your app submission rejected from the App Store - this isn't a concern that I'd like to be addressed in the answer. I'm looking primarily for specific answers about how exactly how Google accomplishes the task of programmatically popping back one app in the App Switching stack in Gboard.

Does this happen with any arbitrary application? In case of the system Messages app, it is quite easy - there is a scheme to launch it : developer.apple.com/library/content/featuredarticles/…. If this works also with other apps (especially less popular ones) it would be indeed curious. — Losiowaty
– Losiowaty, Commented Feb 28, 2017 at 19:13
@Losiowaty It appears to happen with any arbitrary application. — 2snacc
– 2snacc, Commented Feb 28, 2017 at 20:34
About the step 4, I can't open the container/host app, how do you do that? Thanks — Changwei
– Changwei, Commented Jan 25, 2018 at 23:51

kennytm · Accepted Answer · 2017-03-29 19:06:10Z

58

+500

Your guess is correct — Gboard is using private API to do it.

… though not through exploring view hierarchy or event injection.

When the voice-to-text action is done, we can check the syslog from Xcode or Console that it calls the -[AVAudioSession setActive:withOptions:error:] method. So I've reverse-engineered the Gboard app and look for the stack trace related to this.

Climbing up the call stack we can find the -[GKBVoiceRecognitionViewController navigateBackToPreviousApp] method, and…

…_systemNavigationAction? Yep, definitely private API.

Since class_getInstanceVariable is a public API and "_systemNavigationAction" is a string literal, the automatic checker is not able to note the private API usage, and the human reviewers probably don't see anything wrong with the "jump back to the previous app" behavior. Or probably because they are Google and you are not…

The actual code that performs the "jump back to previous app" action is like this:

@import UIKit;
@import ObjectiveC.runtime;

@interface UISystemNavigationAction : NSObject
@property(nonatomic, readonly, nonnull) NSArray<NSNumber*>* destinations;
-(BOOL)sendResponseForDestination:(NSUInteger)destination;
@end

inline BOOL jumpBackToPreviousApp() {
    Ivar sysNavIvar = class_getInstanceVariable(UIApplication.class, "_systemNavigationAction");
    UIApplication* app = UIApplication.sharedApplication;
    UISystemNavigationAction* action = object_getIvar(app, sysNavIvar);
    if (!action) {
        return NO;
    }
    NSUInteger destination = action.destinations.firstObject.unsignedIntegerValue;
    return [action sendResponseForDestination:destination];
}

In particular, the -sendResponseForDestination: method performs the actual "go back" action.

(Since the API is undocumented, Gboard is actually using the API incorrectly. They used the wrong signature -(void)sendResponseForDestination:(id)destination. But it happens that all numbers other than 1 will work the same, so the Google developers are lucky this time)

answered Mar 29, 2017 at 19:06

kennytm

526k110 gold badges1.1k silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

19 Comments

Segev Over a year ago

Great answer! I doubt that Google intentionally tried to hide this, though. I guess that private apis are permitted to some extent between the big corporations.

Ahmet Akkök Over a year ago

Simply impressive! Thank you for shedding light on this. Just to make sure that I get it right: is the code provided supposed to work us as well or is it still private API?

kennytm Over a year ago

@AhmetAkkök It's still private to Apple.

Ahmet Akkök Over a year ago

@kennytm 😬 I think I will give it shot. I appreciate very much if you can add the Swift version of the solution...

Starwave Over a year ago

Swift version would be highly appreciated. Currently, after successful bridging, build fails is I call "jumpBackToPreviousApp()" from a ViewController with error: ":-1: linker command failed with exit code 1 (use -v to see invocation)"

|

Phil Dukhov · Accepted Answer · 2023-11-01 03:44:13Z

6

Update: this hack stopped working on iOS 17, Gboard also can't do it anymore

Swift version of @kennytm answer:

@objc private protocol PrivateSelectors: NSObjectProtocol {
    var destinations: [NSNumber] { get }
    func sendResponseForDestination(_ destination: NSNumber)
}

func jumpBackToPreviousApp() -> Bool {
    guard
        let sysNavIvar = class_getInstanceVariable(UIApplication.self, "_systemNavigationAction"),
        let action = object_getIvar(UIApplication.shared, sysNavIvar) as? NSObject,
        let destinations = action.perform(#selector(getter: PrivateSelectors.destinations)).takeUnretainedValue() as? [NSNumber],
        let firstDestination = destinations.first
    else {
        return false
    }
    action.perform(#selector(PrivateSelectors.sendResponseForDestination), with: firstDestination)
    return true
}

edited Nov 1, 2023 at 3:44

answered Aug 29, 2021 at 13:28

Phil Dukhov

91k22 gold badges276 silver badges302 bronze badges

6 Comments

Daniel Saidi Over a year ago

This unfortunately stopped working in iOS 17. object_getIvar now returns nil.

Phil Dukhov Over a year ago

@DanielSaidi can Google keyboard still do it?

Daniel Saidi Over a year ago

Not the last time I checked, using the last public beta.

strawnut Over a year ago

@PhilDukhov Nop. Gboard is same. not works

CH Wing Over a year ago

Are there other feasible ways to achieve the same effect in iOS 17?

|

nutclump · Accepted Answer · 2023-12-05 21:19:31Z

For iOS17: Use a private API to figure out what app is currently open whenever the keyboard appears, and store that in sharedDefaults. Then use that stored bundle_id to open it again from the keyboard app. Probably going to get your app rejected though!

Find out current open app using KeyboardKit or another SO answer

Store bundleID on keyboard appear, or when they press the mic button:

    .onAppear() {
        sharedDefaults?.set(bundleId, forKey: "last_app_bundle_id")
    }

Open last_app_bundle_id using LSApplicationWorkspace from your keyboard app if iOS17, else use Phil's code:

static func jumpBackToPreviousApp() -> Bool {
    if #available(iOS 17.0, *) {
        let sharedDefaults = UserDefaults(suiteName: "group.lexia")
        let lastAppBundleId = sharedDefaults?.string(forKey: "last_app_bundle_id")
        guard let obj = objc_getClass("LSApplicationWorkspace") as? NSObject else { return false }
        let workspace = obj.perform(Selector(("defaultWorkspace")))?.takeUnretainedValue() as? NSObject
        let open = workspace?.perform(Selector(("openApplicationWithBundleID:")), with: lastAppBundleId) != nil
        return open
    } else {
        guard
            let sysNavIvar = class_getInstanceVariable(UIApplication.self, "_systemNavigationAction"),
            let action = object_getIvar(UIApplication.shared, sysNavIvar) as? NSObject,
            let destinations = action.perform(#selector(getter: PrivateSelectors.destinations)).takeUnretainedValue() as? [NSNumber],
            let firstDestination = destinations.first
        else {
            return false
        }
        action.perform(#selector(PrivateSelectors.sendResponseForDestination), with: firstDestination)
        return true
    }
}

Hmm... It works. But, as you know, hostBundleId of KeyboardKit is really dangerous code because it is using private APIs and XPC framework as low-level intercommunication framework.

Collectives™ on Stack Overflow

How does Google's custom iOS keyboard, Gboard, programmatically dismiss the frontmost app?

3 Answers 3

Your guess is correct — Gboard is using private API to do it.

19 Comments

6 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Your guess is correct — Gboard is using private API to do it.

19 Comments

6 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related