How do you prompt an AI video model for specific props or backgrounds?

How do you describe the scene and context?

The "scene or context" element of your prompt tells the AI model about the environment for your video. This is where you specify the background, setting, and any particular props you want to appear. This environment grounds the main subject of your video and helps establish the overall mood and atmosphere. For example, you might describe a "bustling city street at dusk" or a "quiet library with antique books on wooden shelves" to guide the AI in generating the background and props.

The subject of your video is the "who" or "what" that the action revolves around. Actions describe the "verb" of your video, detailing what is happening and bringing the subject to life through movements, interactions, and subtle expressions. By clearly defining the subject and its actions within a detailed scene, you provide the AI with a comprehensive picture of what to generate, including the specific props and backgrounds that support the narrative.

How do you use negative prompts to control unwanted elements?

To refine the output and ensure specific props or backgrounds are excluded, you can use negative prompts. These prompts specify "things that shouldn't appear in the generated videos." For instance, if you want to avoid a cluttered background, you might include terms like "low quality," "ugly," or "deformed" in your negative prompt to prevent undesirable visual elements. This helps guide the AI away from generating visuals that do not align with your creative vision.

Which AI models are best for controlling video elements?

The Gemini API provides two models for video generation: Gemini Omni Flash and Veo. Gemini Omni Flash is recommended as the default model for video generation. It is a fast, multimodal model that excels at quickly turning text prompts and images into short videos. Gemini Omni Flash offers superior video coherence, character consistency, and multi-turn conversational editing, allowing for element replacement or perspective changes to refine your video's props and backgrounds. It also supports multi-input reasoning, processing text, images, audio, and video inputs simultaneously.

Veo 3.1 is another model available through the Gemini API, specifically useful for certain capabilities. While Gemini Omni Flash is the general recommendation, you would use Veo 3.1 if you require features like scene extension, last-frame control, or integration with legacy pipelines. Veo 3.1 is also a model for generating video with native audio and supports frame-specific generation and image-based direction through the generateContent API. Both Gemini Omni Flash and Veo offer extensive customization through textual prompts, allowing you to specify details for your video.

What technical parameters control the video's appearance?

When generating videos, you can specify several technical parameters that influence the final output. The aspect ratio defines the shape of your video. For example, Veo 3.1 allows you to create landscape videos with a 16:9 aspect ratio, which is the default setting, or portrait videos with a 9:16 aspect ratio. This choice is made using the `aspect_ratio` parameter. The Gemini Enterprise Agent Platform also allows you to specify the expected aspect ratio for output videos, whether generating from text or an image.

You can also control the resolution of your generated videos. Veo 3.1 can directly generate videos in 720p, 1080p, or 4k, though 4k is not available for Veo 3.1 Lite. Higher resolutions, such as 4k, typically result in higher latency and are more expensive. The Gemini Enterprise Agent Platform defaults to 720p if no resolution is provided, but you can specify 720p or 1080p for generated videos. Note that video extension capabilities are limited to 720p videos.

How do you generate videos from text or an image?

The Gemini Enterprise Agent Platform allows you to generate new videos from a text prompt using either Gemini Omni Flash or Veo. This process is guided by the text prompt you provide, which describes the desired video content, including props and backgrounds. Supported interfaces for this include the Google Cloud console and the Agent Platform API. You can also specify an optional Cloud Storage bucket to store the output videos, the expected aspect ratio, and the video output resolution, which defaults to 720p if not provided. The length of these generated video files can be set between 3 and 10 seconds.

Alternatively, you can generate new videos using an existing image as the first frame. This capability is available through the Gemini Enterprise Agent Platform Media Studio or using the Gemini Enterprise Agent Platform video generation API, utilizing Gemini Omni or Veo. For this method, you provide a text prompt to guide the video generation from the initial image. You also specify the Cloud Storage bucket containing the input image and can optionally provide a bucket for output videos, the expected aspect ratio, and the video output resolution, which defaults to 720p. The length of these generated videos can also be set between 3 and 10 seconds.

What are other useful generation parameters?

Beyond visual specifications, you can control other aspects of video generation. You can specify the target duration of the generated videos in seconds, though the actual duration may vary slightly. If not specified, the model will generate 1 video by default, but you can request more. To ensure consistent video generation results when using the same input parameters, you can provide a seed for random number generation.

For content control, you can use a parameter to prevent the generation of videos with people, based on age appearance. Setting this to `dont_allow` will prevent such videos from being generated. Finally, you must specify a Google Cloud Storage URI, starting with `gs://`, where the generated videos will be saved. This URI should point to a bucket or folder for storage.

Common questions

What is the maximum length for generated videos?

The length of generated video files can be set as integers between 3 and 10 seconds, followed by "s" for seconds.

Can I prevent the AI from generating videos with people?

Yes, you can control whether videos of people can be generated based on age appearance. By setting the appropriate parameter to `dont_allow`, you can prevent the generation of videos that include people.

How can I ensure consistent video results across multiple generations?

To produce consistent video generation results with the same input parameters, you can provide a seed for random number generation.

What are the supported aspect ratios for generated videos?

Supported aspect ratios for generated videos include 16:9 for landscape orientation and 9:16 for portrait orientation.

What are the available video resolutions?

Supported video resolutions are 720p and 1080p. Veo 3.1 can also generate 4k videos, but 4k is not available for Veo 3.1 Lite, and video extension is limited to 720p.