Combining Atlan, Snowflake, and Sigma to Democratize Entry to Effectively-governed Information
At a Look
- North, a number one funds answer supplier processing over $100 billion in annual transactions, aimed to enhance knowledge discovery and governance atop their Snowflake- and Sigma-powered trendy knowledge stack
- Adopting Atlan as their energetic metadata platform, North has since achieved end-to-end visibility of 225,000 knowledge belongings, securing delicate knowledge, and growing adoption of self-service analytics
- In lower than a 12 months, their group is projecting $1.4mm in annual effectivity features and a 200% development in Sigma adoption whereas realizing a 700% improve in tagged Snowflake belongings which has contributed to mitigating tens of millions of {dollars} in danger
For over 30 years, North has established itself as a number one funds answer supplier, now processing over $100 billion in transactions yearly. With a various portfolio of over 20 manufacturers and subsidiaries, they help a broad vary of cost wants, supporting brick-and-mortar and on-line shops by means of cost gateways, cellular options, and extra. Dedicated to serving industries that span eating places to authorities entities, North goals to be the simplest end-to-end funds supplier to do enterprise with.
With an enormous knowledge property comprising 225,000 belongings, totaling 41 terabytes on Snowflake, the accountability of managing and governing North’s knowledge belongs to Daniel Dowdy, Vice President of Information Analytics & Governance, serving for 13 years on the firm. Becoming a member of Atlan at Snowflake Summit 2024, Daniel shared his group’s journey, constructing and governing a contemporary knowledge stack that helps each inner knowledge shoppers and their flagship merchandise.
What do 20 subsidiaries and types, and 100,000+ retailers, imply for our group? It means we’ve got plenty of knowledge. We use that knowledge to help a few of our hottest merchandise.”
Daniel Dowdy, Vice President, Information Analytics & Governance
North’s flagship choices, Payanywhere and Funds Hub, equip retailers with something from {hardware} and software program, to safe on-line portals, simplifying the complexities of funds for retailers.
Funds Hub, a complete funds and operations platform, serves as a command heart for all the things from transaction monitoring and reporting to stock administration and fraud safety. To ship important, transaction-level reporting to their clients, North depends on Snowflake.
“We make the most of Snowflake for each of those on the backend to offer close to real-time reporting and transaction-level reporting to our clients,” Daniel shared. “With out Snowflake, this simply wouldn’t be attainable.”
Earlier than Snowflake, Information Silos
Earlier than Snowflake, North struggled with unchecked and unclear knowledge. Easy duties like pulling uncooked knowledge or producing reviews might take days and even weeks, hindered by ETL bottlenecks and knowledge silos scattered throughout a number of servers and databases, or Excel recordsdata circulated throughout the group.
As soon as a file goes out, that knowledge is already stale. To not point out that after that file leaves you, customers can then modify it and make their very own supply of fact. So, you’re combating a number of sources of the reality.”
Daniel Dowdy, Vice President, Information Analytics & Governance
The dearth of a single supply of fact sophisticated efforts to drive belief in North’s knowledge, and with out clear lineage and governance, controlling that knowledge as soon as it left their arms proved troublesome.
After Snowflake + Sigma, Correct, Actual-time Information
In 2019, North adopted Snowflake as their enterprise knowledge lake, considerably enhancing their knowledge science, analytics, and reporting capabilities. By shifting from outdated batch processing to real-time CDC options, their group traded cumbersome ETL workflows and embraced the effectivity of contemporary ELT processes.
“We went from a ‘Information Desert,’ the place retrieving data took days, to an oasis the place knowledge was out there in actual time, and we might question it in seconds,” mentioned Daniel.
Whereas Snowflake represented a major leap ahead within the availability and reliability of knowledge, Daniel’s group have been nonetheless chargeable for fielding easy requests for knowledge entry. To scale back this workload, and to make knowledge much more accessible to a rising variety of shoppers, their group launched Sigma, a enterprise intelligence instrument that empowers customers to independently entry, analyze, and visualize knowledge, providing essential data in a self-service method.
With its acquainted, Excel-like interface, enterprise customers rapidly embraced Sigma, leading to a skyrocketing adoption charge, with annual consumer development now exceeding 200%.
Starting with 10 customers in 2019, we now have over 350 customers in Sigma, with an annual development charge of greater than 200% year-over-year. Now, Sigma is our self-service analytics platform, and it’s taken plenty of burden off our knowledge group.”
Daniel Dowdy, Vice President, Information Analytics & Governance
North has since engineered a contemporary knowledge stack, organising strong pipelines that seamlessly ingest and replicate knowledge from various sources like Oracle, MySQL, Salesforce, and S3. This wealth of knowledge is then channeled into highly effective instruments akin to Qlik, Snowpipe, and Kafka, all converging inside Snowflake.
As the quantity and number of knowledge grew, so did the demand for entry, shifting the problem from mere knowledge accessibility to the extra complicated points of knowledge governance and discovery.
“What turned the issue was easy methods to discover the information, and easy methods to govern it properly,” Daniel shared. “Atlan turned the central level addressing all of those issues.”
Unlocking Snowflake and Sigma with Atlan
Connecting to over 225,000 knowledge belongings inside Snowflake and Sigma took simply hours with Atlan, driving unprecedented visibility on a big scale.
“After partnering with Atlan, it took most likely a few hours earlier than we have been built-in with Snowflake and Sigma, connecting to over 225,000 belongings. We’re so proud of the outcomes,” Daniel shared.
This near-instant connection reworked knowledge discovery at North, as soon as a chaotic expertise of leaping by means of a number of silos, all of a sudden making it easy to navigate and perceive their knowledge property.
Atlan began getting us this fast worth by means of knowledge discovery. The best way I hold fascinated about it was the early days of the web. You had entry to this huge quantity of data, but it surely was actually laborious to search out. Then, search engines like google and yahoo made it simple to run a search and discover related content material. For us, Atlan turned Google for Information.”
Daniel Dowdy, Vice President, Information Analytics & Governance
Transferring past easy knowledge discovery, Daniel’s group launched knowledge glossaries, definitions, and complete phrases, enabling deeper dives into knowledge, and using lineage to grasp upstream and downstream dependencies with ease.
With this helpful context unlocked, their subsequent step was to ship it on to enterprise customers, who primarily devour knowledge throughout the Sigma, with out switching to different instruments.
“That’s the place Atlan’s Chrome Extension turned a game-changer,” Daniel shared.
Atlan’s Chrome Extension reworked knowledge consumption inside Sigma. With context about their knowledge belongings—dashboard descriptions, knowledge definitions, tags, lineage, notifications, and upstream high quality alerts—now out there straight in Sigma, customers now not needed to bounce between instruments to perform their duties.
Obtainable in a sidebar, Atlan’s Chrome Extension made this data immediately accessible, with out coaching or change administration.
“The initiative required no coaching,” Daniel defined. “As a result of it’s a sidebar and intuitively exhibits this data, we’d simply add it, then customers have been robotically up and operating, enhancing their effectivity, and instantly seeing the profit.”
Subject Decision for Engineers and Shoppers, Alike
With context about their knowledge belongings made out there in Sigma, Daniel and his group started work to enhance the method of troubleshooting and speaking points associated to those belongings. Upon receiving a ticket, North’s engineering group as soon as spent important time looking by means of Sigma to establish the belongings referenced within the request. With out a direct hyperlink between tickets and these belongings, decision would typically be delayed, and work could be duplicated.
“It was actually inefficient,” Daniel shared. “Generally even the identical engineers could be engaged on the identical mission, and so they didn’t even realize it as a result of a number of tickets have been submitted by a number of folks.”
By integrating the Atlan Chrome Extension with Jira and Slack, North’s knowledge shoppers might elevate points straight of their instrument of selection, with a hyperlink to particular belongings included robotically, permitting Information Engineers to grasp when and the place points occurred.
For knowledge shoppers in Sigma, this integration supplied a complete historical past of all Jira tickets and Slack conversations referring to particular knowledge belongings, and for engineers, a method of stopping duplicate tickets or a number of engineers engaged on the identical subject, and time financial savings by higher understanding comparable, previous points.
Eliminating duplicate work, or eliminating your engineers engaged on the identical mission with out realizing about it? These effectivity features are big, and the financial savings add up actually rapidly.”
Daniel Dowdy, Vice President, Information Analytics & Governance
Introducing Snowflake Dynamic Information Masking
As a funds answer supplier, North handles huge quantities of delicate monetary knowledge. Whereas enhancements in knowledge discovery and enrichment are essential, Daniel’s group realized the worth of those enhancements could be restricted to a small variety of trusted customers with out strong knowledge governance and documentation securing these belongings.
To safe delicate knowledge and implement stringent entry insurance policies, North’s knowledge group selected to make the most of Dynamic Information Masking with Snowflake, providing complete column-level masking for delicate knowledge, and policy-based management for customers.
With a purpose to make the most of Dynamic Information Masking, Daniel and his group started the method of figuring out particular knowledge belongings that these insurance policies would apply to. Given the size and complexity of their knowledge panorama, they acknowledged {that a} granular, asset-by-asset method could be not solely time-consuming, however doubtlessly error-prone. Trying to find a approach to establish and safe these belongings programmatically, Daniel turned to Atlan.
“How do you discover and tag 1000’s of belongings? You don’t Google it, you Atlan it. It’s actually easy,” Daniel defined.
Governing 1000’s of Belongings, on the Click on of a Button
Fairly than painstakingly finding these belongings, then tagging them one-by-one, their group utilized Atlan Playbooks, rules-based bulk automations that might establish 1000’s of belongings primarily based on chosen standards, then robotically apply actions, akin to including chosen tags and insurance policies.
By the point I added a filter, Atlan scanned our 225,000 belongings and virtually instantly discovered 45 matches. As I modified the filters so as to add new ones, it always up to date to point out me what number of belongings meet that standards. Hours of guide analysis and testing, earlier than Atlan, was transformed into a pair seconds.”
Daniel Dowdy, Vice President, Information Analytics & Governance
Rapidly figuring out 1000’s of belongings to better-secure, Daniel’s group used Playbooks to construct workflows that may robotically apply the proper tag to the proper asset, then propagate that tag upstream, on the click on of a button. Then, with Atlan’s Two-way Tag Sync, these tags would function the inspiration for Dynamic Information Masking at North, robotically making use of masking insurance policies primarily based on how and from whom knowledge was queried.
By combining Atlan Playbooks, Two-way Tag Sync, and Automated Lineage with Snowflake Dynamic Information Masking, North efficiently ruled 1000’s of knowledge belongings in report time with a programmatic method, avoiding the painstaking work of figuring out, tagging, and recording insurance policies one-by-one throughout a 225,000-asset panorama.
With out Atlan tracing upstream and downstream, serving to us perceive potential publicity and implement insurance policies, this course of would have been sophisticated and really guide. However with Atlan, you can’t solely hint knowledge lineage and arrange the complexity, however you possibly can see recognition and utilization metrics that assist prioritize the place efforts are wanted probably the most, or the place the most important influence could possibly be made.”
Daniel Dowdy, Vice President, Information Analytics & Governance
By this programmatic method, Daniel’s group affected a 700% improve in tagged belongings, considerably enhancing their safety and compliance, and paving the way in which for a major improve within the variety of their colleagues trusted to make use of knowledge.
“We’ve used Atlan to spice up the pace and accuracy of our asset tagging by greater than 700% since implementation,” Daniel shared. “With the flexibility to find knowledge quicker, apply tags, and reverse sync that knowledge again into Snowflake, we added greater than 2,000 governance belongings in beneath a 12 months. That’s simply the tip of the iceberg.”
Tens of millions in Worth, in Months
In lower than one 12 months of being outfitted with Atlan, Daniel and his group reworked the way in which North makes use of and governs its knowledge, unlocking the potential for tens of millions in worth by means of effectivity features, value reductions, and enhanced danger mitigation.
By thoughtfully converging essential context about Snowflake and Sigma right into a user-friendly information base in Atlan, North’s technical and enterprise customers are projecting financial savings of $1.4 million per 12 months in effort by means of accelerated knowledge discovery and diminished effort understanding lineage.
“That’s $1.4 million of duplicate work, guide and tedious analysis. That’s time we will now spend on altering issues, versus attempting to navigate knowledge prefer it’s the web in 1997,” Daniel shared.
And with built-in insights in Atlan, close to–immediately figuring out pointless spend of their Snowflake atmosphere, Daniel and his group estimate an annual value financial savings of greater than $20,000 by deprecating greater than 10,000 idle belongings and inefficient queries.
“That financial savings was principally handed to us on a silver platter with no further work,” Daniel remarked.
Past value financial savings and effectivity features, and existential to an organization that processes delicate monetary knowledge, lies important danger mitigation. By growing their tagged belongings by 700%, rapidly and in bulk, North’s governance and safety improved virtually in a single day, to the tune of tens of millions of {dollars} in diminished danger.
We’ve been capable of cut back our danger and publicity by tens of millions and tens of millions of {dollars}.”
Daniel Dowdy, Vice President, Information Analytics & Governance
A 200% Enhance in Self-service Analytics, with Room to Develop
Whereas these enhancements are spectacular, whether or not in knowledge discovery, with context about their knowledge belongings supporting a 200% year-over-year improve in self-service analytics adoption, or in Information Governance, driving tens of millions in danger mitigation, it’s the mixture of those packages that’s set to rework the way in which North makes use of knowledge.
By affecting a leap ahead in the way in which that North manages entry management and ensures safety throughout a large knowledge property, the way in which Daniel’s group defines a trusted consumer of knowledge is evolving and increasing considerably.
“Information Governance gave us the arrogance to securely and securely share extra knowledge in Sigma,” Daniel defined.Now assured that probably the most delicate knowledge is safe, appropriately tagged, and ruled by the proper entry coverage, North can confidently supply knowledge and self-service analytics capabilities to extra enterprise customers than ever earlier than, enhancing the already important worth they’ve yielded from combining Atlan, Snowflake, and Sigma.
We have been in a state of near-paralysis attempting to manually decide who ought to see what. Using a mix of improved governance insurance policies, tags, and dynamic knowledge masking, we have been in a position to make use of knowledge with confidence. Because of the enhanced consumer expertise, mixed with the elevated knowledge availability, we anticipate to proceed or exceed our self-service atmosphere development charges.”
Daniel Dowdy, Vice President, Information Analytics & Governance
Mere years after adopting Snowflake as the inspiration of their Trendy Information Stack, and fewer than a 12 months after adopting Atlan, North’s knowledge property has reworked from, as Daniel described, a “Information Desert” into an “Oasis.”
“For the reason that knowledge by no means leaves our umbrella, there’s no extra flat recordsdata which can be shared,” Daniel defined. “There’s no extra a number of variations of the reality, and our knowledge has a stage of confidence, leading to quicker and extra knowledgeable selections.”
Regardless of the unimaginable accomplishments of North’s knowledge group, constructing self-service analytics and Information Governance capabilities from scratch in report time, Daniel believes there’s nonetheless a lot extra to perform.
Within the months and years to return, their knowledge group is keen to make use of Atlan AI to simplify complicated queries into pure language, and speed up the enrichment of their knowledge belongings. And by supporting a rising variety of Information Stewards to independently share and doc information, a extra knowledgeable and empowered group of enterprise customers is rising.
Reflecting on their journey with Atlan, Daniel emphasised that past the proper expertise, group, and technique lies an important ingredient: The correct partnership.
One of the vital essential issues about Atlan has been the folks. They’re knowledge people who have labored within the trade, perceive what you’re going by means of, and perceive what you’re speaking about from the second you begin the dialog.”
Daniel Dowdy, Vice President, Information Analytics & Governance
Picture by Jonas Leupe on Unsplash
