Unifying a Knowledge Stack and Leveraging Self-Serve Analytics with Atlan
The Energetic Metadata Pioneers sequence options Atlan clients who’ve accomplished a radical analysis of the Energetic Metadata Administration market. Paying ahead what you’ve realized to the following information chief is the true spirit of the Atlan neighborhood! In order that they’re right here to share their hard-earned perspective on an evolving market, what makes up their trendy information stack, revolutionary use circumstances for metadata, and extra.
On this installment of the sequence, we meet Daniel Ferguson, Knowledge and Analytics Director at PHMG, an audio branding firm that helps over 36,000 shoppers throughout 56 nations sound their greatest. Daniel shares how PHMG remodeled their information stack from fragmented to unified, and the way Atlan has been a vital piece in by monitoring lineage, managing stories, and easing workforce onboarding.
This interview has been edited for brevity and readability.
May you inform us a bit about your self, your background, and what drew you to Knowledge & Analytics?
I was a DJ after which labored as a sound engineer constructing recording studios. Throughout my time as a sound engineer, I discovered myself within the technical and analytical aspect of issues. After beginning a household, I needed a change. After promoting my recording studio, my mom, who managed a council workplace, provided me a job and I began out within the name middle unit, dealing with calls. It shortly turned obvious that I may do extra than simply calls, so I moved to the database workforce.
I began learning for a level in Economics and Mathematical Science at The Open College whereas working on the council. Utilizing the abilities learnt on my diploma I began to construct logistic regression fashions to focus on contacts within the name middle I had beforehand labored in. I proposed that, with only one individual, I may obtain the identical outcomes as all the workforce. My work generated 300% extra outcomes than the workforce’s mixed efforts by optimizing information assortment, addressing lacking info, and cherry selecting one of the best contacts. After that I used to be hooked on the facility of information & analytics.
I then constructed an organization offering information providers to different Native Authorities. Close to the tip of my diploma, a consultancy in Scotland, Aquila Insights, provided me a place. They labored with shoppers like Sony, Workplace Depot, and RBS, which gave me early publicity to the info career. From there, I superior within the discipline and ultimately joined PHMG. My journey into information was considerably unintentional, but it surely introduced me to the place I’m at the moment.
Would you thoughts describing PHMG?
We concentrate on audio branding. Consider logos like Netflix or Disney Plus, by way of sound alone, these manufacturers are immediately recognizable as business leaders in leisure and streaming even when their visible logos should not in sight.
We additionally transcend conventional audio branding by growing customized music tailor-made to every group. We’ll take Atlan for example: What’s Atlan about? What do you signify? What’s the kind of rhythm that it needs to carry?
This connection between music and identification is what attracted me to the corporate. We have now been extremely profitable, working in 56 nations with 36,000 shoppers.
May you describe your information stack, and the way it got here collectively?
After I obtained right here, we had been utilizing SQL servers with Excel spreadsheets. There have been restricted to no interactive stories, and each information request needed to be raised to the info workforce.
There was a have to modernize the knowledge flowing into the corporate and implement the best know-how to attain this effectively and reliably. I targeted on discovering know-how options that might streamline operations and cut back the necessity for extra engineers.
I used to be actually cautious with know-how choice, avoiding options for the sake of it, and never constructing from scratch. Whereas Azure Material provides a complete resolution, for instance, it’s nonetheless new and that comes with extra dangers, however is one thing I’m conserving my eye on. It’s essential to decide on one of the best instruments for the job and guarantee they work nicely collectively. Investing in a seamless course of with these instruments lets you begin robust and exhibit worth shortly, with room to evolve as you scale.
In my board proposal, I highlighted two important instruments: Atlan and ThoughtSpot. I defined that whereas we may handle with out them, they might make a big distinction. I needed governance to change into embedded in our processes, and that as a substitute of assigning information stewards with out clear course, we offered actionable stories and comprehensible information. With correctly organized information, governance turns into simple, and Atlan streamlines this course of.
I chosen Snowflake for its robustness and affordable pricing, and Fivetran for its dependable pipeline efficiency, which successfully handles our information integration wants.
I carried out PowerBI for government stories, and ThoughtSpot for our self-serve information wants. I’m a giant fan of ThoughtSpot, as a result of it permits customers to regulate their very own stories, lowering the necessity for fixed modifications from the info workforce.
For orchestration, I take advantage of Airflow to handle pipelines, and DBT with GitLab for our code repository and CI/CD processes.
Why was Atlan a very good match? Did something stand out throughout your analysis course of?
In my earlier group, I attempted utilizing open-source with DataHub, however its upkeep and improvement required vital funding. Atlan stood out as a result of it’s plug-and-play, robotically constructing out miners that reveal beforehand unknown insights. It identifies and explains scripts we weren’t conscious of, saving time and lowering technical debt from having to manually evaluation intensive code.
Atlan lets us observe and monitor what we’ve constructed, together with information lineage and property. It’s invaluable for reviewing stories without having to ask for code particulars—simply navigate by way of Atlan to see the report’s historical past. New workforce members also can perceive report development by way of Atlan.
For me, Atlan was a key piece of the puzzle.
I researched Collibra, Alation, and Atlan extensively, and Atlan was the clear alternative. It felt designed for medium-sized enterprises and required minimal engineering effort. Given our state of affairs, it was essential to combine it from the beginning, moderately than as an afterthought. This allowed us to study and develop Atlan alongside our present techniques, moderately than attempting to drive it into our pre-built setup.
I at all times make it a degree to fulfill with management groups at occasions to gauge their angle and willpower, and I don’t know of another gamers which might be doing it in addition to Atlan. I used to be genuinely impressed by Atlan’s management workforce — not solely their ardour for the product but in addition their dedication to addressing my challenges and bettering our state of affairs.
How are you planning to harness Atlan to reinforce your information stack? What thrilling use circumstances and objectives do you take into consideration?
We’ve invested in a knowledge vault mannequin for our information warehouse, which feeds into an operational information retailer, what I name the info mart. All our stories and metrics are constructed from this information mart. In Atlan, we outline the right way to assemble all the pieces, so as soon as a metric is outlined, we will write the SQL to extract it from the mart.
We then create curated tables for consumer providers and gross sales organizations, enabling them to self-serve by way of ThoughtSpot. For detailed insights into the development and rationale of those metrics, we retailer that info in Atlan, which turns into our catalog.
As new folks come on board, I be certain that there’s no want for a handover. By default, we doc our processes as we go and construct techniques that go away clear breadcrumbs for others to observe. Atlan performs a vital function on this. We direct new workforce members to Atlan to assist them perceive how all the pieces is constructed and what it’s constructed from. Atlan doesn’t simply spill out the code, it highlights the important thing objects, their utilization, and their significance.
One other main mission entails making a complete glossary inside Atlan, serving as our single supply of fact. This setting permits enterprise customers to entry all company metrics and consider stories from Salesforce, PowerBI, and ThoughtSpot, all linked round key KPIs.
We’re additionally at the moment refining our information lineage and mannequin descriptions. As we create new information fashions, we replace descriptions incrementally moderately than in bulk. This ongoing effort helps be certain that our information fashions are well-documented and simply comprehensible.
Do you might have any recommendation to share along with your friends who’re beginning out in managing and organizing their information property successfully?
Companies at all times speak about being information pushed, however they don’t discuss in regards to the property that truly drive the info. We wish info to movement in our group, however info can’t movement if it’s not organized persistently. And for me, instruments like Atlan are making it considerably simpler for us to prepare and talk what information issues.
Don’t get me incorrect, Atlan isn’t a silver bullet. It gained’t repair poor group inside your information warehouse. Nevertheless, it does present a centralized place to outline and assess your processes, serving to you determine which of them are efficient and which of them want enchancment.
Atlan helped us decide the place to start out by figuring out our most important tables and specializing in what was essential. As an example, we discovered one desk essential for all the pieces we constructed, permitting us to prioritize it. We then assessed our stories and found that some we thought had been essential had been related solely to particular stories, not the broader context.
As we get delicate information, we will additionally instantly flag it. If we get audited, we will merely pull up Atlan and say, “Hey, that is what we’ve got. That is how we handle our information. That is what our information property are.” So, for these dedicated to being data-driven, they should take care of their information property and perceive what their information property are.
Photograph by Adi Goldstein on Unsplash
