How do you prompt dialogue and sound in an AI-generated video?

What is Veo 3.1 and why use it for audio?

The Gemini API offers two models for generating video: Gemini Omni Flash and Veo. While Gemini Omni Flash is the default model for video generation, offering superior video coherence and character consistency, Veo 3.1 is the model to use when specific capabilities like native audio generation are required. Veo 3.1 is stable and generally available for production on Vertex AI, providing professional-grade creative controls and support for multiple aspect ratios.

Veo 3.1 builds upon its predecessor, Veo 3, by offering stronger prompt adherence and improved audiovisual quality, especially when turning images into videos. It integrates rich, synchronous audio with existing video generation capabilities, helping users craft detailed scenes. This model supports features such as video extension, frame-specific generation, and image-based direction through the generateContent API.

How do you prompt dialogue in Veo 3.1?

Veo 3.1 excels at generating realistic, synchronized sound, including multi-person conversations. When you want to include specific speech from characters in your video, Google Cloud Blog documents that you should use quotation marks around the dialogue. For example, you might write, "A woman says, 'We have to leave now.'" This clear formatting guides the AI to produce the exact spoken words.

This capability makes Veo 3.1 ideal for creating multi-shot scenes where consistent characters are engaged in conversation. The model's ability to craft dialogue ensures that the spoken words align with the visual actions and character consistency across the video.

How do you prompt sound effects in Veo 3.1?

Beyond dialogue, Veo 3.1 can generate precisely timed sound effects. To prompt these, Google Cloud Blog suggests describing the sounds with clarity. A common practice is to use "SFX:" followed by the description of the sound. An example provided is, "SFX: thunder cracks in the distance." This method helps the model understand and integrate specific sound events into your video.

The model's ability to generate synchronized sound effects means that the audio will align with the visual elements of your video, creating a more immersive and professional-grade output. This precision in sound effect generation is a key feature of Veo 3.1.

How do you prompt ambient noise in Veo 3.1?

Setting the background soundscape is crucial for creating a complete audio experience in your video. Veo 3.1 allows you to define ambient noise through your prompt. Google Cloud Blog recommends using "Ambient noise:" followed by a description of the desired background sound. For instance, you could write, "Ambient noise: the quiet hum of a starship bridge."

This feature enables you to establish the environment and mood of your scene with subtle, continuous sounds. By guiding the model with specific descriptions for ambient noise, you can ensure the generated video has a rich and realistic audio layer that complements the visuals.

How do you ensure sound is included in Google Vids?

Google Vids allows you to create and modify video clips using AI, including generating new clips from text prompts, reference images, or existing videos. When you use the "Generate video" function in Google Vids, it is important to include comprehensive details in your prompt. This includes specifying elements like the subject, location, action, camera, lighting, dialogue, sounds, and tone to guide the AI in creating the desired output.

If you generate a video clip in Google Vids and find that it has no sound, there are a few steps to troubleshoot. First, verify that your computer speakers are unmuted and functioning correctly with other applications. Next, within the Google Vids timeline, select the object track of your generated content. Ensure that the speaker icon for that specific generated object track is not muted.

Common questions

Which Google AI model should I use for video generation with audio?

For video generation that includes native audio, you should use Google's Veo 3.1 model. It is specifically designed to generate video with synchronized sound, including dialogue, sound effects, and ambient noise.

What is the default Google AI model for video generation if I don't need specific audio features?

The default model for video generation in the Gemini API is Gemini Omni Flash. It is recommended for general video generation due to its superior video coherence, character consistency, and multi-input reasoning capabilities.

Can Veo 3.1 generate multi-person conversations?

Yes, Veo 3.1 excels at generating realistic, synchronized sound, including multi-person conversations. You can guide these conversations by using quotation marks for specific dialogue within your prompt.

What details should I include in a prompt for Google Vids to ensure sound?

When generating a video clip in Google Vids, you should include details such as the subject, location, action, camera, lighting, dialogue, sounds, and tone in your text prompt to help the AI generate the desired audio elements.

What should I do if my Google Vids clip has no sound?

If your Google Vids clip generates with no sound, first check that your computer speakers are unmuted. Then, in the Google Vids timeline, select the object track of your generated content and ensure its speaker icon is not muted.