Saying Amazon SageMaker Inference for customized Amazon Nova fashions

Since we launched Amazon Nova customization in Amazon SageMaker AI at AWS NY Summit 2025, prospects have been asking for a similar capabilities with Amazon Nova as they do once they customise open weights fashions in Amazon SageMaker Inference. Additionally they needed have extra management and adaptability in customized mannequin inference over occasion sorts, auto-scaling insurance policies, context size, and concurrency settings that manufacturing workloads demand.

At the moment, we’re saying the overall availability of customized Nova mannequin help in Amazon SageMaker Inference, a production-grade, configurable, and cost-efficient managed inference service to deploy and scale full-rank personalized Nova fashions. Now you can expertise an end-to-end customization journey to coach Nova Micro, Nova Lite, and Nova 2 Lite fashions with reasoning capabilities utilizing Amazon SageMaker Coaching Jobs or Amazon HyperPod and seamlessly deploy them with managed inference infrastructure of Amazon SageMaker AI.

With Amazon SageMaker Inference for customized Nova fashions, you’ll be able to cut back inference value by optimized GPU utilization utilizing Amazon Elastic Compute Cloud (Amazon EC2) G5 and G6 situations over P5 situations, auto-scaling based mostly on 5-minute utilization patterns, and configurable inference parameters. This characteristic permits deployment of personalized Nova fashions with continued pre-training, supervised fine-tuning, or reinforcement fine-tuning to your use circumstances. It’s also possible to set superior configurations about context size, concurrency, and batch dimension for optimizing the latency-cost-accuracy tradeoff to your particular workloads.

Let’s see the right way to deploy personalized Nova fashions on SageMaker AI real-time endpoints, configure inference parameters, and invoke your fashions for testing.

Deploy customized Nova fashions in SageMaker Inference

At AWS re:Invent 2025, we launched new serverless customization in Amazon SageMaker AI for fashionable AI fashions together with Nova fashions. With a couple of clicks, you’ll be able to seamlessly choose a mannequin and customization method, and deal with mannequin analysis and deployment. If you have already got a educated customized Nova mannequin artifact, you’ll be able to deploy the fashions on SageMaker Inference by the SageMaker Studio or SageMaker AI SDK.

Within the SageMaker Studio, select a educated Nova mannequin in Fashions in your fashions within the Fashions menu. You’ll be able to deploy the mannequin by selecting Deploy button, SageMaker AI and Create new endpoint.

Select the endpoint identify, occasion kind, and superior choices resembling occasion rely, max occasion rely, permission and networking, and Deploy button. At GA launch, you need to use g5.12xlarge, g5.24xlarge, g5.48xlarge, g6.12xlarge, g6.24xlarge, g6.48xlarge, and p5.48xlarge occasion sorts for the Nova Micro mannequin, g5.48xlarge, g6.48xlarge, and p5.48xlarge for the Nova Lite mannequin, and p5.48xlarge for the Nova 2 Lite mannequin.

Creating your endpoint requires time to provision the infrastructure, obtain your mannequin artifacts, and initialize the inference container.

After mannequin deployment completes and the endpoint standing exhibits InService, you’ll be able to carry out real-time inference utilizing the brand new endpoint. To check the mannequin, select the Playground tab and enter your immediate within the Chat mode.

It’s also possible to use the SageMaker AI SDK to create two sources: a SageMaker AI mannequin object that references your Nova mannequin artifacts, and an endpoint configuration that defines how the mannequin can be deployed.

The next code pattern creates a SageMaker AI mannequin that references your Nova mannequin artifacts. For supported container photos by Area, refer desk lists the container picture URIs:

# Create a SageMaker AI mannequin
    model_response = sagemaker.create_model(
        ModelName="Nova-micro-ml-g5-12xlarge",
        PrimaryContainer={
            'Picture': '708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-inference-repo:v1.0.0',
            'ModelDataSource': {
                'S3DataSource': {
                   'S3Uri': 's3://your-bucket-name/path/to/mannequin/artifacts/',
                   'S3DataType': 'S3Prefix',
                   'CompressionType': 'None'
                }
            },
            # Mannequin Parameters
            'Setting': {
                'CONTEXT_LENGTH': 8000,
                'MAX_CONCURRENCY': 16,
                'DEFAULT_TEMPERATURE': 0.0,
                'DEFAULT_TOP_P': 1.0
            }
        },
        ExecutionRoleArn=SAGEMAKER_EXECUTION_ROLE_ARN,
        EnableNetworkIsolation=True
    )
    print("Mannequin created efficiently!")

Subsequent, create an endpoint configuration that defines your deployment infrastructure and deploy your Nova mannequin by making a SageMaker AI real-time endpoint. This endpoint will host your mannequin and supply a safe HTTPS endpoint for making inference requests.

# Create Endpoint Configuration
    production_variant = {
        'VariantName': 'main',
        'ModelName': 'Nova-micro-ml-g5-12xlarge',
        'InitialInstanceCount': 1,
        'InstanceType': 'ml.g5.12xlarge',
    }
    
    config_response = sagemaker.create_endpoint_config(
        EndpointConfigName="Nova-micro-ml-g5-12xlarge-Config",
        ProductionVariants= production_variant
    )
    print("Endpoint configuration created efficiently!")
    
# Deploy your Noval mannequin
    endpoint_response = sagemaker.create_endpoint(
        EndpointName="Nova-micro-ml-g5-12xlarge-endpoint",
        EndpointConfigName="Nova-micro-ml-g5-12xlarge-Config"
    )
    print("Endpoint creation initiated efficiently!")

After the endpoint is created, you’ll be able to ship inference requests to generate predictions out of your customized Nova mannequin. Amazon SageMaker AI helps synchronous endpoints for real-time with streaming/non-streaming modes and asynchronous endpoints for batch processing.

For instance, the next code creates streaming completion format for textual content technology:

# Streaming chat request with complete parameters
streaming_request = {
"messages": [
        {"role": "user", "content": "Compare our Q4 2025 actual spend against budget across all departments and highlight variances exceeding 10%"}
    ],
    "max_tokens": 512,
    "stream": True,
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 40,
    "logprobs": True,
    "top_logprobs": 2,
    "reasoning_effort": "low",  # Choices: "low", "excessive"
    "stream_options": {"include_usage": True}
}

invoke_nova_endpoint(streaming_request)

def invoke_nova_endpoint(request_body):
"""
    Invoke Nova endpoint with computerized streaming detection.
    
    Args:
        request_body (dict): Request payload containing immediate and parameters
    
    Returns:
        dict: Response from the mannequin (for non-streaming requests)
        None: For streaming requests (prints output immediately)
    """
    physique = json.dumps(request_body)
    is_streaming = request_body.get("stream", False)
    
    attempt:
        print(f"Invoking endpoint ({'streaming' if is_streaming else 'non-streaming'})...")
        
        if is_streaming:
            response = runtime_client.invoke_endpoint_with_response_stream(
                EndpointName=ENDPOINT_NAME,
                ContentType="utility/json",
                Physique=physique
            )
            
            event_stream = response['Body']
            for occasion in event_stream:
                if 'PayloadPart' in occasion:
                    chunk = occasion['PayloadPart']
                    if 'Bytes' in chunk:
                        information = chunk['Bytes'].decode()
                        print("Chunk:", information)
        else:
            # Non-streaming inference
            response = runtime_client.invoke_endpoint(
                EndpointName=ENDPOINT_NAME,
                ContentType="utility/json",
                Settle for="utility/json",
                Physique=physique
            )
            
            response_body = response['Body'].learn().decode('utf-8')
            outcome = json.hundreds(response_body)
            print("✅ Response obtained efficiently")
            return outcome
    
    besides ClientError as e:
        error_code = e.response['Error']['Code']
        error_message = e.response['Error']['Message']
        print(f"❌ AWS Error: {error_code} - {error_message}")
    besides Exception as e:
        print(f"❌ Sudden error: {str(e)}")

To make use of full code examples, go to Getting began with customizing Nova fashions on SageMaker AI. To be taught extra about greatest practices on deploying and managing fashions, go to Finest practices for SageMaker AI.

Now obtainable

Amazon SageMaker Inference for customized Nova fashions is on the market immediately in US East (N. Virginia) and US West (Oregon) AWS Areas. For Regional availability and a future roadmap, go to the AWS Capabilities by Area.

The characteristic helps Nova Micro, Nova Lite, and Nova 2 Lite fashions with reasoning capabilities, working on EC2 G5, G6, and P5 situations with auto-scaling help. You pay just for the compute situations you utilize, with per-hour billing and no minimal commitments. For extra data, go to Amazon SageMaker AI Pricing web page.

Give it a attempt in Amazon SageMaker AI console and ship suggestions to AWS re:Publish for SageMaker or by your normal AWS Assist contacts.

— Channy