Introducing Apache Spark Join help in AWS Glue interactive periods

Once we constructed AWS Glue interactive periods, our aim was to make AWS Glue as interactive as working native Python from a pocket book. We principally succeeded. With an easy Python bundle and a Jupyter pocket book, you can execute remotely towards the AWS Glue ephemeral Spark backend. The Livy-based strategy was forward of its time, nevertheless it had limitations from its REST-based protocol. Operating native PySpark unlocked highly effective built-in improvement surroundings (IDE) options corresponding to debugging and linting, so your surroundings might perceive the code and make it easier to develop Spark purposes extra shortly. Clients would typically cut up their improvement work. They used native Spark (or Docker containers) to develop in an IDE on a small quantity of information, then switched to AWS Glue interactive periods to validate scaling and tuning towards the total dataset.

With trendy PySpark releases got here a brand new protocol: Apache Spark Join. Spark Join bridges the hole between these two worlds: you develop in native Python, however execute on AWS Glue towards precise information. Right this moment, AWS Glue interactive periods help Spark Join natively. You may join from any surroundings that helps the PySpark distant() API, together with VS Code, PyCharm, Amazon SageMaker Unified Studio notebooks, and standalone Python purposes. You don’t want to put in specialised kernels or handle cluster infrastructure.

What Spark Join modifications

Spark Join, launched in Spark 3.4, decouples the Spark consumer from the server by a light-weight gRPC protocol. As an alternative of working your driver program on the cluster, your IDE communicates with a distant Spark server by a skinny consumer layer. This structure unlocks the important thing workflow enchancment: you develop regionally and execute remotely.

Spark Join structure — skinny consumer with the total energy of Apache Spark

With Spark Join help in AWS Glue interactive periods, you get:

• IDE freedom – Use VS Code, PyCharm, JupyterLab, or any Python surroundings. No kernel set up required.
• Programmatic entry – Construct Spark into your Python purposes and automation scripts with an ordinary SparkSession.builder.distant() name.
• Serverless execution – AWS Glue provisions and manages the Spark cluster. You pay just for the info processing models (DPUs) consumed whereas your session is lively.
• Spark Join monitoring – The Spark Dwell UI now features a devoted Join tab displaying lively Spark Join periods and operations alongside the present Jobs, Phases, and Executors views.

Getting began with SageMaker Unified Studio

Amazon SageMaker Unified Studio supplies probably the most direct path to Spark Join on AWS Glue. The pocket book surroundings handles session creation, endpoint retrieval, and token refresh robotically, so no connection boilerplate is required.

Prerequisite: You want an Amazon SageMaker Unified Studio venture to make use of this workflow. For those who don’t have one, create a venture in your SageMaker Unified Studio area first.

To hook up with an AWS Glue Spark Join session:

1. Check in to SageMaker Unified Studio, select your venture, and create or open a Pocket book.

A pocket book open in SageMaker Unified Studio

2. Select the compute icon within the left toolbar to open the Compute surroundings panel. Develop the Spark part.

The Compute surroundings panel with the Spark dropdown checklist

3. Choose a Glue Spark connection. Relying in your SageMaker area configuration, you will notice both default.spark or named connections corresponding to venture.spark.compatibility. Choose the suitable Glue (Spark) connection and select Apply.

Related to Glue Spark Join — working spark.model returns ‘3.5.6-amzn-1’

After you make your choice, you’re related. The spark session object is on the market natively. No imports or configuration are wanted. Begin working PySpark instantly:

spark.sql("SHOW DATABASES").present()

The session manages itself within the background, together with computerized token refresh.

Utilizing the sagemaker_studio SDK

The sagemaker-studio Python bundle extends the Spark Join expertise past SageMaker Unified Studio notebooks into native IDEs, steady integration and steady supply (CI/CD) pipelines, and any Python surroundings. The sparkutils module handles session initialization and connection configuration in a single name. You get the identical streamlined expertise as within the pocket book, wherever you run Python:

from sagemaker_studio import sparkutils

# Initialize a Glue Spark Join session utilizing your venture connection
spark = sparkutils.init(connection_name="default.spark")

# Run queries instantly
spark.sql("SHOW DATABASES").present()

You can even use sparkutils.get_spark_options() to retrieve pre-configured Java Database Connectivity (JDBC) choices for studying and writing to information sources by your venture connections. Supported sources embody Amazon Redshift, Amazon Aurora, and Amazon DocumentDB (with MongoDB compatibility):

# Get connection choices for a Redshift connection in your venture
choices = sparkutils.get_spark_options("my_redshift_connection")

# Learn from Redshift through Spark Join
df = spark.learn.format("jdbc").choices(**choices).choice("dbtable", "analytics.orders").load()
df.present()

Inside SageMaker Unified Studio, the sagemaker-studio SDK is native to the surroundings. The spark session and sparkutils can be found with out set up. For native IDE use, set up it with pip set up sagemaker-studio and configure credentials by an AWS named profile or boto3 session.

The way it works

Spark Join periods in AWS Glue use a three-step workflow:

1. Create a session – Name the CreateSession API with SessionType set to SPARK_CONNECT. The session provisions in roughly 30 seconds.
2. Retrieve the endpoint – Name GetSessionEndpoint to obtain a sc:// gRPC endpoint URL and a time-limited authentication token.
3. Join with PySpark – Move the endpoint and token to SparkSession.builder.distant() and begin working Spark operations.

Spark Join protocol movement — DataFrame API translated to logical plan, despatched through gRPC/protobuf, outcomes streamed again through gRPC/Arrow

Connecting with the low-level API

Some environments don’t have the sagemaker-studio SDK, corresponding to customized containers, AWS Lambda capabilities, or non-Python toolchains. In these environments, or if you happen to’re not utilizing SageMaker Unified Studio, you need to use the AWS SDK (Boto3) to handle periods immediately. The next instance demonstrates the total workflow:

import time, boto3, urllib.parse
from pyspark.sql import SparkSession

glue = boto3.consumer("glue", region_name="us-east-1")

# 1. Create a Spark Join session
session_id = "my-spark-connect-session"
glue.create_session(
    Id=session_id,
    Position="arn:aws:iam::123456789012:function/GlueServiceRole",
    Command={"Title": "glueetl"},
    GlueVersion="5.1",
    SessionType="SPARK_CONNECT",
    DefaultArguments={"--enable-spark-live-ui": "true"},
)

# 2. Await the session to achieve READY
whereas True:
    standing = glue.get_session(Id=session_id)["Session"]["Status"]
    if standing == "READY":
        break
    time.sleep(5)

# 3. Get the Spark Join endpoint
sc = glue.get_session_endpoint(SessionId=session_id)["SparkConnect"]
endpoint_url = sc["Url"]
auth_token = sc["AuthToken"]

# 4. Join with PySpark
encoded_token = urllib.parse.quote(auth_token, secure="")
connection_string = f"{endpoint_url}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
spark = SparkSession.builder.distant(connection_string).getOrCreate()
spark.sql("SELECT 1 + 1 AS consequence").present()

Monitoring with Spark Dwell UI

If you allow the Spark Dwell UI at session creation, you achieve entry to a real-time dashboard displaying:

• Jobs and Phases – Observe lively, accomplished, and failed jobs with stage-level metrics.
• Executors – Monitor reminiscence utilization, shuffle information, and executor well being.
• SQL – Examine question plans and execution particulars.
• Join tab – View lively Spark Join periods and operations (particular to Spark Join).

Entry the dashboard by the GetDashboardUrl API or immediately from the AWS Glue console.

import boto3, webbrowser

glue = boto3.consumer("glue", region_name="us-east-1")
dashboard = glue.get_dashboard_url(
    ResourceId="my-spark-connect-session",
    ResourceType="SESSION",
)
webbrowser.open(dashboard["Url"])

In SageMaker Unified Studio, no API name is required. Select Prepared within the pocket book standing bar to open the kernel information popover. From there, open the Spark UI hyperlink for the stay dashboard or Spark Driver Logs for real-time log output.

Picture displaying “Prepared” within the standing bar to entry Spark UI and Driver Logs immediately from the pocket book

Token refresh

Authentication tokens expire after half-hour. In SageMaker Unified Studio, that is dealt with robotically. For programmatic use, you need to use a background thread to maintain the connection alive. The next helper reconnects transparently earlier than the token expires:

import threading, time, boto3, urllib.parse
from pyspark.sql import SparkSession

class GlueSparkConnect:
    """Maintains a SparkSession with computerized token refresh."""

    def __init__(self, session_id, area="us-east-1", refresh_margin=300):
        self.session_id = session_id
        self.glue = boto3.consumer("glue", region_name=area)
        self.refresh_margin = refresh_margin  # seconds earlier than expiry to refresh
        self._lock = threading.Lock()
        self.spark = self._connect()
        self._start_refresh_loop()

    def _connect(self):
        sc = self.glue.get_session_endpoint(SessionId=self.session_id)["SparkConnect"]
        encoded_token = urllib.parse.quote(sc["AuthToken"], secure="")
        remote_url = f"{sc['Url']}:443/;use_ssl=true;x-aws-proxy-auth={encoded_token}"
        self._token_expiry = sc["AuthTokenExpirationTime"].timestamp()
        return SparkSession.builder.distant(remote_url).getOrCreate()

    def _start_refresh_loop(self):
        def _loop():
            whereas True:
                sleep_for = max(self._token_expiry - time.time() - self.refresh_margin, 30)
                time.sleep(sleep_for)
                with self._lock:
                    self.spark = self._connect()
        t = threading.Thread(goal=_loop, daemon=True)
        t.begin()

# Utilization
session = GlueSparkConnect("my-spark-connect-session")
session.spark.sql("SELECT 1 + 1 AS consequence").present()

The background thread sleeps till 5 minutes earlier than token expiry, then transparently reconnects. As a result of the daemon thread exits when your script ends, there isn’t any cleanup required.

Getting began

To start out utilizing Spark Join with AWS Glue interactive periods:

1. Use AWS Glue model 5.1 (Apache Spark 3.5.6).
2. Set up PySpark 3.5.6 regionally: pip set up pyspark==3.5.6.
3. Grant your AWS Identification and Entry Administration (IAM) identification permissions for glue:CreateSession, glue:GetSession, and glue:GetSessionEndpoint.
4. Create a session with --session-type SPARK_CONNECT and join out of your most popular surroundings.

VPC word: For those who connect with AWS Glue interactive periods by a digital personal cloud (VPC) endpoint, add the brand new Spark Join endpoint (com.amazonaws.{area}.glue.periods) to your VPC configuration. Current AWS Glue VPC endpoints don’t cowl Spark Join site visitors.

For detailed directions, see Connecting to a Spark Join session within the AWS Glue Developer Information.

Concerning the authors