Amazon Bedrock Advanced Prompt Optimization: Streamline Model Migration and Performance Tuning

Introduction to Advanced Prompt Optimization

Amazon Bedrock has unveiled a powerful new capability: Advanced Prompt Optimization. This tool is designed to help users refine their prompts for any model hosted on Bedrock, enabling seamless model migration and performance enhancement. By automating the iterative process of prompt testing and optimization, it saves time and ensures consistent quality across different AI models.

Key Features of the Optimization Tool

Simultaneous Multi-Model Comparison

One standout feature is the ability to compare your original prompt with optimized versions across up to five models at once. This is particularly useful when you're considering a model change or simply want to gauge improvements. You can set your current model as a baseline and test up to four alternatives, or just evaluate optimizations on your existing model.

Metric-Driven Optimization Loop

The optimizer operates on a feedback loop guided by your chosen evaluation metric. It takes in your prompt template, sample user inputs, ground truth answers, and a metric definition—then iteratively refines the prompt to maximize performance. You can provide the evaluation criteria as a Lambda function, an LLM-as-a-judge rubric, or a simple natural language description.

Multimodal Input Support

Unlike many optimization tools, this one supports multimodal prompts. You can include images in PNG, JPG, or PDF format within your prompt templates, making it ideal for tasks like document analysis or image-based reasoning.

How to Get Started

Step 1: Access the Tool

Navigate to the Advanced Prompt Optimization page in the Amazon Bedrock console and click Create prompt optimization.

Step 2: Select Models

Choose up to five inference models for which you want to optimize. If you're migrating, include your current model as a baseline alongside new candidates. For performance tuning alone, just select your current model.

Step 3: Prepare Your Data

You'll need a JSONL file containing your prompt templates, example inputs, ground truth, and evaluation criteria. Each line of the file is a single JSON object. Below is a simplified example of the structure:

{
    "version": "bedrock-2026-05-14",
    "templateId": "my-template",
    "promptTemplate": "Answer the following question: ",
    "steeringCriteria": ["Provide factual answers"],
    "customEvaluationMetricLabel": "accuracy",
    "customLLMJConfig": {
        "customLLMJPrompt": "Rate the answer's accuracy from 1 to 5",
        "customLLMJModelId": "anthropic.claude-v2"
    },
    "evaluationSamples": [
        {
            "inputVariables": {
                "question": "What is the capital of France?"
            },
            "referenceResp": "Paris"
        }
    ]
}

Real-World Applications

Model Migration

When switching to a new model, you often face regression risks. The optimization tool helps you test optimized prompts across both old and new models simultaneously, ensuring no drop in performance on known tasks while improving weaker areas.

Performance Improvement

Even without changing models, you can boost effectiveness. The optimizer identifies underperforming tasks and adjusts prompt phrasing, structure, or context to align better with the model's capabilities.

Custom Evaluation Options

You're not limited to built-in metrics. The tool allows for three types of custom evaluation:

• Lambda function – Run a custom scoring algorithm.
• LLM-as-a-judge rubric – Have another LLM evaluate responses based on criteria you define.
• Natural language description – Provide plain-English guidance for the optimiser.

Understanding the Output

After optimization, you receive both the original and final prompt templates, along with evaluation scores, cost estimates, and latency comparisons. This transparency helps you make informed decisions about which prompt—and which model—to deploy.

Conclusion

Amazon Bedrock's Advanced Prompt Optimization is a versatile tool for anyone working with large language models. Whether you're migrating to a more cost-effective model or squeezing extra accuracy out of your current one, it provides a structured, data-driven approach to prompt engineering. By leveraging multimodal support and custom evaluation methods, it addresses a wide range of real-world AI tasks.