What Gemini and Google AI features are we waiting for?

In the past year or so, Google has previewed a number of Gemini-branded AI and other features across its consumer-facing apps. Here’s everything that’s been announced and when it might be available.

pixel

At the end of Made by Google 2023, the Pixel 8 Pro was teased a Zoom Enhance feature that “intelligently fills gaps between pixels and predicts fine details.” Leveraging the device’s “custom generative AI image model,” Google pitched this as useful when you forget to zoom in.

It’s an amazing application of generative AI, opening up a range of possibilities for framing and editing your photos. So the kind of zoom enhancement you’re used to seeing in science fiction — right in your phone.

In October, Google said the feature was “coming later.” After launching three new Pixel features, the feature has yet to arrive. It’s not clear if the model Google is referring to is the Gemini Nano with Multimodality. At this point, the feature could debut with the Pixel 9 Pro as the phone’s main photography feature.

Google House

In the Google Home app, generative AI will be used to summarize events in a “simplified view of what happened recently.” This “quick and easy summary” will use bullet points, while you’ll also be able to “ask about your home” conversationally to find historical videos and get automations. The “experimental features” will be available to Nest Aware subscribers in 2024.

Fitbit

Fitbit Labs will allow Fitbit Premium users to test experimental AI capabilities and provide feedback on them.

One such feature is a chatbot that lets you ask questions about your Fitbit data in a natural, conversational way. This “personal coaching” that takes fitness goals into account aims to create “actionable messages and guidance,” with responses that can include personalized charts.

• “For example, you can dig deeper into how many active zone minutes (AZMs) you get and how they relate to how well you recover from sleep.”
• “…This model may be able to analyze differences in your sleep patterns and sleep quality, and then suggest recommendations on how to change the intensity of your exercise based on those insights.”

Behind the scenes, this is supported by a new LLM in Personal Health from Fitbit and Google Research built on Gemini. Starting in March, it’s coming “later this year” to “a limited number of Android users enrolled in the Fitbit Labs program in the Fitbit mobile app.”

Google Images

Ask Photos lets you ask questions about the photos and videos in your library. Beyond searching for images, it can pull up information and give you a text-based answer. Gemini-powered queries include “Show me the best photo from every national park you’ve visited” and “What were the themes of our birthday parties?” and can be used to “suggest the best photos” and create annotations for them. Ask Photos is an “experimental feature” that will be rolling out soon, with Google already hinting at more possibilities in the future.

Gmail + Google Workspace

In the Gmail app for Android and iOS, you’ll find a Gemini button in the top right corner that lets you bring up the mobile equivalent of a sidebar for entering full instructions. Gmail is also getting contextual smart replies that offer more personalized, detailed, and accurate suggestions. This feature will roll out to Workspace Labs in July.

At its Cloud Next 2024 conference in April, Google also showed off the ability to direct voice commands to the Help me write in mobile Gmail app. Meanwhile, an “instant polish” feature will “turn raw notes into a full email with a single tap.”

On the web and desktop, the sidebar is available in Gmail, Google Drive, and Docs/Sheets/Slide. Gemini will then come to Google Chat to summarize conversations and answer questions.

Google Maps

Last February, Google announced that its Maps would use LLMs to power its “Ask About” chatbot. You can use it to find places that match your prompt with follow-up questions. It’s powered by details about 250 million places, photos, videos, and reviews submitted by users.

Chrom

Gemini Nano comes to the Chrome desktop browser to power browser features like Help Me Typing. It should be available on most modern laptops and desktops.

Google search

In addition to launching AI Overviews, Google previewed a number of upcoming features that will arrive first in Search Labs:

• You will be able to take an original AI overview and make it “simpler” (just a few sentences) or “break it down” (a longer response).
• Multi-step reasoning capabilities will allow you to ask a complex question in one step rather than breaking it down into multiple queries.
• Planning meals and trips
• Search results page organized by artificial intelligence
• Video Searches: Record a video and ask a question about it.

Android

The Gemini Nano multimedia app will launch on the Pixel “later this year” and will power features like on-device/offline TalkBack descriptions, and real-time fraud alerts that listen to calls for revealing patterns. Google will share more details later this year.

At I/O 2024, Google also showed off how Gemini on Android will soon become an overlay panel instead of opening up a full-screen UI to display results. Along with preserving context, this will let you drag and drop the generated image into the conversation. For Gemini Advanced subscribers, the “Ask this video” and “Ask this PDF” buttons will see Gemini videos and documents, respectively. This will roll out “over the next few months.” Additionally, Gemini Nano’s dynamic suggestions will use multimedia to understand what’s on your screen:

For example, if you activate Gemini in a conversation about pickleball, suggestions might include “Find pickleball clubs near me” and “Pickleball rules for beginners.”

Another addition that will be particularly useful on mobile is the Gemini extension for Google Calendar, Google Tasks, and Keep. This extension will let you take a photo of a page with multiple upcoming dates, and Gemini will be able to turn them into calendar events. In the coming months, the Gemini mobile app’s “Utilities” feature will bring access to the Clock app on Android.

We’re also looking forward to the Gemini mobile app arriving on the Pixel Tablet this summer.

twin

Live will let you have a two-way conversation with Gemini. To make the experience more natural, Gemini will send you brief responses that you can interrupt to add new information or ask for clarification. You can choose from 10 different voices, and Google imagines Gemini Live could be useful for preparing for an interview or practicing a speech. It will be available to Gemini Advanced members in the “coming months.”

“Later this year,” Gemini Live will let you turn on live camera mode. Simply point to something in the real world and ask a question about it. This is supported by Project Astra.

Gems are customized versions of Gemini that let you have a “gym buddy, sous chef, coding partner, or creative writing guide.” Gemini Advanced members will be able to create custom gems, while all users will have access to pre-made gems, such as the Learning Coach.

Simply describe what you want your gem to do and how you want it to respond – such as “You are my running coach, give me a daily running plan and be positive, upbeat and motivating.” Gemini will take those instructions and, with one click, refine them to create a gem that meets your specific needs.

Gemini Advanced users will also get a “comprehensive planner” that goes beyond just suggesting activities but actually takes into account travel times and stops, as well as people’s interests, to create a detailed itinerary. Gemini will use your Gmail flight/travel details, Google Maps recommendations for foods and museums near your hotel, and search for other activities.