<div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png" width="1280" height="960" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:960,&quot;width&quot;:1280,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2444376,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://truestate.substack.com/i/159511159?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png&quot;,&quot;isProcessing&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F71896b04-eafd-431d-a028-0cd970bf5151_1280x960.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><div class="pencraft pc-reset icon-container restack-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-refresh-cw"><path d="M3 12a9 9 0 0 1 9-9 9.75 9.75 0 0 1 6.74 2.74L21 8"></path><path d="M21 3v5h-5"></path><path d="M21 12a9 9 0 0 1-9 9 9.75 9.75 0 0 1-6.74-2.74L3 16"></path><path d="M8 16H3v5"></path></svg></div><div class="pencraft pc-reset icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></div></div></a></figure></div><p>One thing to think about in artificial intelligence is what we're actually talking about when we're talking about AI. "We're building an AI agent to transform your business" doesn't tell you much, in the same way that "we're bringing you Web3" didn't either. The claim itself carries almost no information content, though it does signal membership in a specific tribe of technology enthusiasts.</p><p>I've previously compared the AI cycle to the crypto cycle (oops), but there's an important difference, which is that AI systems actually do things. They write code and analyse data and create images and run call centres. They just might not do the specific thing that someone is promising they'll do, and distinguishing between what's real and what's aspirational requires a taxonomy.</p><p>If you're a normal person with a job, you periodically get pitched by vendors who want to sell you an "AI agent" to revolutionise your workflow. "Our AI agent will optimise your supply chain," they'll say, or "our AI agent will predict customer churn," or "our AI agent will solve your data analytics challenges." Every demo looks roughly the same &#8212; a chatbot interface where you type instructions, and magic happens.</p><p>But the magic ranges from "pretty basic automation that's existed for years" to "genuine intelligence that can transform business processes." How do you tell the difference? And more importantly, how do you know what you actually need?</p><h4><strong>The Levels Framework</strong></h4><p>At my company, TrueState, we encountered this problem as we tried to automate the implementation (aka coding-side) of data science consulting. We had proven market demand for our services but our potential was limited by our scalability. We were asking ourself, how do you grow a traditionally high-touch, expert-driven business without simply hiring an army of expensive data scientists? The obvious answer was to use AI to augment and eventually automate parts of our workflow.</p><p>But this created a new problem. We couldn't clearly articulate what our AI systems should be doing at different stages of development. Without a shared vocabulary and clear taxonomy, we couldn't plan our technology roadmap or have coherent conversations about what we were building. Are we creating glorified dashboards? Decision support systems? Fully autonomous analytical engines? The terms "AI" and "agent" were too vague to be useful for actual planning.</p><p>So we developed a framework inspired by the levels used to classify self-driving cars &#8212; a practical ontology that helped us navigate the transition from human-centric consulting to AI-augmented delivery. It gave us a way to communicate clearly about capabilities, limitations, and development priorities, both internally and with clients. I'll give examples to the analytics ecosystem throughout as this is where we this framework was born.</p><p>Here's how it works:</p><h4><strong>L0: The Rule Follower</strong></h4><p>At Level 0, you have systems that do exactly what they're programmed to do, with zero adaptability. It's glorified automation. In analytics, this is a scheduled report that runs the same query every day and emails the results to stakeholders. Nothing wrong with this! Most business processes can benefit from basic automation. Just don't call it AI.</p><h4><strong>L1: The Helpful Assistant</strong></h4><p>Level 1 systems make suggestions to help you work better, but you make all the decisions. This is where most "AI assistants" actually operate today. They're pattern-matching systems that can recognize context and offer relevant recommendations.</p><p>In analytics, it's like having a programming copilot that works with you to write data science code.</p><p>Imagine a smart intern who's constantly looking over your shoulder with helpful suggestions, but who you'd never allow to push changes to production. That's Level 1.</p><h4><strong>L2: The Specialist</strong></h4><p>At Level 2, systems can independently handle complex tasks within one domain. They can work independently but can't collaborate effectively with other systems or handle tasks that span multiple domains. It's the equivalent of hiring a specialist with deep expertise in a single area - they're brilliant at what they do but useless outside their lane.</p><p>In analytics, think of a system can write an SQL query based on a technical brief.</p><h4><strong>L3: The Coordinator</strong></h4><p>Level 3 is where things get interesting. These systems can break down complex requests into component parts and coordinate across subservient agents (i.e. down to L2 systems). They understand different capabilities and constraints and can orchestrate work across them. Interestingly a fractal pattern starts to emerge. L3 systems are, at the top, an L2 agent that calls other L2 agents. The coordinator itself has a restricted domain - it only knows how to delegate information to other specialised systems, not do the actual work. We start to see that doing "stuff" is just work briefs and specialised tasks all the way down.</p><p>In analytics, this is like having a competent project manager who's not really proactively engaged. They do what you ask them within the confines of project work, but they're not going to suggest anything new for you, or even work with you to clarify information. They take the brief and go. Really convenient, but you're kind of left wanting more.</p><h4><strong>L4: The Fixer</strong></h4><p>Level 4 systems can take a business objective and figure out how to achieve it. You don't tell them what analysis to run; you tell them what problem to solve.</p><p>L4 systems then work with you to clarify the brief, understand any issues before going off into the L3/2 fractal maze of delegated work and getting it done.</p><p>In analytics, you might say, "We need to reduce customer churn," and the system come back asking "what represents churn for you?" (often more complex than you'd think...) "what data can I rely on for this?". This info goes into a predictive modelling brief to build a solution that identifies likely churners every week and your next-best-action to prevent that churn.</p><p>This is your VP of Analytics who understands the business context, can translate objectives into concrete plans, and can execute those plans independently. They still operate within your strategic framework, but they have significant autonomy in how they achieve objectives.</p><h4><strong>L5: The Strategic Partner</strong></h4><p>Level 5 systems can actually help determine which goals to pursue before going off an getting it done. These systems don't just do work - suggest new work that's relevant to your organisation.</p><p>It's like having a Chief Intelligence Officer who not only executes your data strategy but helps shape it based on their understanding of your business and the competitive landscape.</p><h4><strong>Where Are We Now?</strong></h4><p>Here's the thing: most "AI agents" being sold today operate at Levels 0-2. A small number of cutting-edge applications are reaching Level 3. Level 4 exists primarily in research labs and specialised applications. Level 5 remains largely theoretical.</p><p>That's not necessarily a problem. Most business processes don't need Level 5 intelligence. Basic automation and augmentation (Levels 0-2) already deliver tremendous value. The key is matching the right level of capability to each function in your organisation.</p><p>When vendors pitch you an "AI agent," the first question should be: "Which level is it really operating at?" This immediately cuts through marketing hype and sets realistic expectations about what the technology can actually do.</p><p>It also helps your internal planning. Where do you need full autonomy (Levels 3-4) versus augmented human capabilities (Levels 1-2)? Which processes can be fully automated with rule-based systems (Level 0), and which require more sophisticated intelligence?</p><p>As for TrueState, we're currently working on an L4 enterprise data science system. We've built the end-to-end infrastructure to support the use-cases (data ingestion, petabyte scale ETL, ML training and inference across multiple modalities, LLM workflows, logging, security, blah blah) and now we're building the "data team in a box" to build bespoke data science solutions for our customers. This is our unlock to take high-value data science consulting and scale it.</p>

One thing to think about in artificial intelligence is what we're actually talking about when we're talking about AI.

There's levels to AI 

<p>TrueState is a technology company focused on empowering organisations to solve their most important problems with machine learning and artificial intelligence.</p><p>Our developer platform provides teams with access to cutting-edge algorithms, high-leverage APIs and an easy-to-use interface, while our implementation solutions help organisations take their first steps with AI with confidence and peace of mind.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.truestate.io&quot;,&quot;text&quot;:&quot;Learn More&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.truestate.io"><span>Learn More</span></a></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg" width="1440" height="1076" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1076,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:526270,&quot;alt&quot;:&quot;background pattern&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false}" class="sizing-normal" alt="background pattern" title="background pattern" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66688376-3284-4088-a4b8-18133936b323_1440x1076.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><div class="pencraft pc-reset icon-container restack-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-refresh-cw"><path d="M3 12a9 9 0 0 1 9-9 9.75 9.75 0 0 1 6.74 2.74L21 8"></path><path d="M21 3v5h-5"></path><path d="M21 12a9 9 0 0 1-9 9 9.75 9.75 0 0 1-6.74-2.74L3 16"></path><path d="M8 16H3v5"></path></svg></div><div class="pencraft pc-reset icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></div></div></a><figcaption class="image-caption">Photo by <a href="true">Maita Cayson</a> on <a href="https://unsplash.com">Unsplash</a></figcaption></figure></div><p>In late 2019 our team was huddled around a whiteboard in a conference room at one of our Fortune 50 clients. The company had engaged QuantumBlack to help optimise their sales process, which spanned across multiple regions and product lines. </p><p>We built a sophisticated machine learning pipeline that could predict the likelihood that a lead was going to convert in the next 14 days and recommend targeted strategies to maximise the likelihood of conversion. The model worked well and everyone was happy with the initial results. <br><br>In that room we faced a critical challenge: the initial scaling from one product to two had taken 16 weeks. With dozens of products across EMEA, North America, and APAC regions awaiting similar solutions, we needed a fundamentally different approach to scale the solution. <br><br>Our ROI was directly related to how many new geographies and products we could serve, but our technology stack was holding us back. Rather painfully, we created a scalable solution focusing on high-degrees of software reuse and developer productivity. There were maintenance challenges and teething pains along the way but we ultimately reached the scale we needed and delivered enormous value. <br><br>In a rather black swan occurrence for the data science industry, 5 years on, our solution is still in production, delivering value across all of the products and geographies.</p><p>This scenario, which played out during my time at McKinsey's AI division, highlighted a fundamental truth about data science in enterprise settings: the ability to scale your impact is often more important than improving the performance of individual models from a pure ROI perspective. <br><br>In this post we will explore the role of speed and scalability in driving impact for data science teams as well as how you can set your data science team up to maximise their ROI.</p><h2>The Economics of Data Science ROI</h2><p>In accounting, one of the primary metrics for the &#8220;value&#8221; of a team or division is their return on investment (ROI). In other words, given how much money is required to keep this team running, how much value does the organisation generate. Let&#8217;s get specific on the economics ROI of data science teams.</p><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;ROI_{DS} = (\\sum_{i=1}^{n} \\Delta P_i - C_i ) / C_{fixed} &quot;,&quot;id&quot;:&quot;VXKPLPQGKQ&quot;}" data-component-name="LatexBlockToDOM"></div><p> This equation tells us that the ROI for a data science team is the sum (&#931;) of all of the net benefits (&#916;Pi - Ci) of all the n data science initiatives, divided by the fixed costs of running that team. This is generally calculated on a yearly-basis. </p><p>Let&#8217;s break down the different parts of this equation.</p><h3>ROI</h3><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;ROI_{DS}&quot;,&quot;id&quot;:&quot;GDFIZOFJBR&quot;}" data-component-name="LatexBlockToDOM"></div><p>ROI is the return on investment of your data science function - in other words, how much value does your data science function generate vs how much it costs to run. ROI should be the &#8220;North Star&#8221; for all data science teams and is expressed as a percentage.</p><h3>Initiative Performance</h3><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot; \\Delta P_i&quot;,&quot;id&quot;:&quot;QYJIZRGGCZ&quot;}" data-component-name="LatexBlockToDOM"></div><p>This is the incremental impact a single data science initiative (i) has on the organisation. It could be the result of increased conversions, prevented churn, or cost savings from automations. These should always be calculated in dollar equivalents.</p><p>This performance improvement is made up of two factors:</p><ol><li><p>How good the models are - derived from the data, methods and general know-how of the team.</p></li><li><p>How well the models are integrated into operations - derived from solution design and change management.</p></li></ol><p>Most data science teams are focused on maximising model performance and neglect the other parts of the ROI equation. </p><h4>Variable Costs</h4><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;C_i&quot;,&quot;id&quot;:&quot;SVDUOIRZZV&quot;}" data-component-name="LatexBlockToDOM"></div><p>These are the costs of running each initiative (i). Variable costs commonly take the form of:</p><ol><li><p>Processing costs (e.g., compute, data storage).</p></li><li><p>Variable licence costs (e.g., licences for third-party data).</p></li></ol><h4>Fixed Costs</h4><div class="latex-rendered" data-attrs="{&quot;persistentExpression&quot;:&quot;C_{fixed}&quot;,&quot;id&quot;:&quot;FQENWZJFQQ&quot;}" data-component-name="LatexBlockToDOM"></div><p>These are the fixed costs of your data science team. Fixed costs are primarily broken down into two factors:</p><ol><li><p>Staff.</p></li><li><p>Any fixed licence costs (e.g., software).</p></li></ol><h2>Prioritising data science speed and scalability as a driver of ROI</h2><p>If we reflect on the ROI formula for data science, a theme that emerges; given decent model performance, ROI is dependent on the number of initiatives you can build and maintain. </p><p>But what does this mean? It means your impact is correlated with your scale.<br><br>Is this approach of balancing scale and individual initiative performance reflected in the way most data science teams operate? Hardly.</p><p>As an industry we&#8217;re overwhelmingly focused on model improvements in individual use-cases, to our detriment. We want our models to be great. We want our stakeholders to be amazed by their predictive power. We want to knock it out of the park. <br><br>The reality is that perfect is the enemy of good in data science. In other words, if the model is good-enough and you&#8217;re able to effectively communicate the benefits of your models, your stakeholders are probably already going to be amazed.  It&#8217;s much better to make things stable and move on to another use-case. <br><br>In our example at the start of this article, we were building a sales optimisation solution where we were classifying leads in a pipeline as likely or unlikely to convert in the next 14 days and recommending actions to convert these leads. Our core model reached the mid-80% mark for precision and recall which, given there was no existing ML solution supporting this function, blew the past performance out of the water. <br><br>Management was thrilled and wanted to see these results across the whole business. Would it have been nice to get the performance to the low-90% range? Absolutely. Was it a better use of our time than scaling the impact to rest of the business? Absolutely not. <br><br>For data science teams looking to boost their ROI, there&#8217;s first a mindset shift that&#8217;s required. It&#8217;s one that appreciates the value of scalability as well as model performance. <br><br>Then there&#8217;s a review of ways of working. </p><p>Unfortunately many teams are unable to go beyond the 1 or 2 use-cases they&#8217;re in their current backlog because their setup is holding them back. They&#8217;re stuck in a very special kind of hell integrating an alphabet soup of technologies while also trying to build a great solution and communicate its effectiveness to an often apathetic group of stakeholders. </p><p>This affects the build of new use-cases and the maintenance of existing ones. It also affects morale. Fighting day-in, day-out to make complex data science solutions performant and stable is crippling for morale. It burns data teams out. </p><p>So how do we improve the productivity and scalability of our data science teams? How do we make these teams kick goals while also remaining sane?<br><br>We&#8217;ll dive into this in more detail in a future post but TL;DR here&#8217;s a few ways;</p><ol><li><p>Deliberately consider your data science technology stack: </p><p></p><p>Are you building notebooks from scratch every time? Are you falling over an alphabet soup of niche tools in pursuit of a feature-complete framework? If this is you, I&#8217;d recommend checking out <a href="http://www.truestate.io">TrueState</a>, our fully-integrated data science platform focused on making data scientists as productive as possible.<br><br>In any case, make sure you&#8217;re not fighting against your tech stack.</p><p></p></li><li><p>Practice concise, accessible communication with stakeholders: </p><p></p><p>Meetings and communications are some of the biggest time-sinks for data scientists. </p><p></p><p>Prepare communications relevant to your target audience and ensure you keep the conversations focused on impact to their job, not the technical nuances of your solution. Hard truth - they don&#8217;t care and you&#8217;re only going to lose them trying to explain it.</p><p></p></li><li><p>Focus on reusability:</p><p></p><p>It takes precious time to write code. If you&#8217;re re-writing the same modelling code every time, you&#8217;re shooting yourself in the foot. Create reusable assets instead and create mechanisms to share those assets easily between projects.</p></li></ol><h2><br>Wrap up</h2><p>In this post we&#8217;ve explored the importance of scaling your impact as a data science team. It&#8217;s not easy but if you can shift your mindset from focusing just on individual model performance to also prioritising the scale of your impact, you&#8217;re off to a great start. <br><br>If you can follow that up with selecting the right technology stack, practicing effective and efficient stakeholder communications and prioritising reusability, you&#8217;ll unlock new levels of impact for your data science team and your organisation. <br><br></p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://truestate.substack.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading TrueState Blog! Subscribe for free to receive new posts in your inbox.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>

Data science teams often obsess over model improvements. While this is important, focusing on scaling your impact can dramatically improve the value data science brings to your organisation.

Improving the ROI of your data science team with scale

<p>Hi there,<br><br>I&#8217;m Will, the Founder &amp; CEO of TrueState, a Sydney-based start-up focused on simplifying the way companies build their own AI solutions to enable the next wave of AI adoption. Today we announced our A$1.5m pre-seed raise, led by <a href="https://www.airtree.vc/">Airtree Ventures</a> and supported by <a href="https://www.sidestage.vc/">Side Stage</a>, <a href="https://galileo.ventures/">Galileo</a>, <a href="http://imprint-partners.com/">Imprint Capital Partners</a> and a number of angel investors, so I thought I&#8217;d take a moment to introduce myself and TrueState.</p><p>Prior to starting TrueState I worked at McKinsey &amp; Company&#8217;s AI division QuantumBlack, in Private Equity portfolio operations and as a freelance consultant enabling companies to identify high-impact opportunities for AI then rapidly scale solutions to create real value.</p><p>Across all of the clients I supported, I saw the same problem time and time again. Organisations, sometimes with strong reputations for technical prowess, were stumped by the process of building appropriate, stable solutions to capture value, caught off-guard by the technical complexity of building in this space. </p><p>The extent of the problem is massive. Despite businesses pouring hundreds of billions of dollars being into AI, the average initial implementation time is 10-20 months. This lead time is due to a mismatch between the expertise required to build these systems and the teams that have been put together to build them.</p><p>To aid them, companies typically hire costly consultants to implement bespoke solutions for them and hire large, expensive teams to maintain the software over the long term.</p><p>Our approach is to radically simplify the way people build AI, combining all of the disparate technologies required into a single platform suitable that removes the requirement for a PhD or years of experience in the technology. Instead of building foundational infrastructure, users configure AI workflows on our no-code platform.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png" width="1456" height="1118" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/93690539-a718-42fd-baef-4a288ed96933_2138x1642.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1118,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:370553,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93690539-a718-42fd-baef-4a288ed96933_2138x1642.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><div class="pencraft pc-reset icon-container restack-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-refresh-cw"><path d="M3 12a9 9 0 0 1 9-9 9.75 9.75 0 0 1 6.74 2.74L21 8"></path><path d="M21 3v5h-5"></path><path d="M21 12a9 9 0 0 1-9 9 9.75 9.75 0 0 1-6.74-2.74L3 16"></path><path d="M8 16H3v5"></path></svg></div><div class="pencraft pc-reset icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></div></div></a><figcaption class="image-caption">A predictive maintenance AI workflow</figcaption></figure></div><p>Easily customisable workflows can be selected from a library of proven AI use-cases, like supply chain optimisation, predictive maintenance, and personalised customer retention recommendations - further accelerating the process. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png" width="1456" height="799" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:799,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1355664,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 424w, https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 848w, https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 1272w, https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F43106b46-39e5-4da8-a117-27849f136f8d_2912x1598.png 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><div class="pencraft pc-reset icon-container restack-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-refresh-cw"><path d="M3 12a9 9 0 0 1 9-9 9.75 9.75 0 0 1 6.74 2.74L21 8"></path><path d="M21 3v5h-5"></path><path d="M21 12a9 9 0 0 1-9 9 9.75 9.75 0 0 1-6.74-2.74L3 16"></path><path d="M8 16H3v5"></path></svg></div><div class="pencraft pc-reset icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></div></div></div></div></a><figcaption class="image-caption">Our first set of workflow templates</figcaption></figure></div><p>For those that get stuck along the way, our on-platform AI data scientist is available 24/7, complete with a growing knowledge of high-performance use-cases, shortcuts to better performance and infinite time and patience to explain the underlying technology to curious users.</p><p>The platform is not just built for beginners - more experienced users can flex their knowledge through a variety of advanced settings to further push the envelope of performance.</p><p>We&#8217;re excited for the future of TrueState and grateful to our investors for supporting us in this ambitious journey. From here we are focused on continuing to improve the platform for our beta customers, expand the suite of AI technologies available on platform and engage with new customers across Australia, the UK and the US.</p><p></p><p></p><p></p><p></p><p></p>

Enabling us all to drive the next wave of AI adoption

Introducing TrueState

<p>TrueState is a technology company focused on empowering organisations to solve their most important problems with machine learning and artificial intelligence.</p><p>Our developer platform provides teams with access to cutting-edge algorithms, high-leverage APIs and an easy-to-use interface, while our implementation solutions help organisations take their first steps with AI with confidence and peace of mind.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.truestate.io&quot;,&quot;text&quot;:&quot;Learn More&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.truestate.io"><span>Learn More</span></a></p><p>In the ever-evolving landscape of AI, businesses often struggle to move beyond toy demos to tangible results. In this post, I&#8217;ll cover some recipes I&#8217;ve used to create hundreds of millions of dollars in value for clients at McKinsey &amp; Company&#8217;s AI division QuantumBlack, in Private Equity operations, and more recently for customers at TrueState.</p><p>Creating value with AI first requires the right frame of mind, so let&#8217;s step away from specific AI systems, like ChatGPT, and define intelligence as the competence of a system (either biological or artificial) over a broad range of task types (e.g., creative writing, forecasting, categorisation, optimal decision making). A system is generally considered more intelligent if it can competently perform a wide range of tasks.</p><p>While there is a broad range of tasks in business, any one individual algorithm is unlikely to have state-of-the-art performance across all of these tasks, especially those tasks typically aligned closest to business impact.</p><p>This is because different AI systems have different strengths and weaknesses across different domains. This is clear from model benchmarks, where you can see certain conversational systems performing better in math, problem-solving, or writing tasks. </p><p>Stepping away from conversational/multimodal models, there are also a swathe of models far more capable of event prediction, time series forecasting, record classification, search, and more. These alternative systems include tree-based classification and regression models, deep recurrent neural nets, as well as embedding, classification, and scoring models for natural language and many more.</p><p>By deploying a range of AI algorithms, each suited to their specific use-case, executives give themselves the best chance at creating real impact. The disillusionment many are currently experiencing with AI is due to the opposite case: deploying the wrong type of AI to solve low-impact problems, leading to failure and lacklustre results.</p><p>To effectively harness AI, it&#8217;s crucial to ask the right questions. For executives grappling with how to navigate this landscape, I encourage you to reframe your thinking with the following questions:</p><ol><li><p>What are the decisions in your organisation that impact the bottom line the most?</p></li><li><p>What would be the most helpful information for you or others to make those decisions better?</p></li><li><p>Given your data, what is the best algorithm/combination of algorithms to use to achieve state-of-the-art results?</p></li></ol><p>By getting specific on these questions, you can set yourself up to select the right type of AI for your use-case and begin creating real impact with AI.</p><p>Let&#8217;s explore these questions.</p><p><strong>Question 1 &#8211; What are your high-impact decisions?</strong><br>Here, you need to have a clear view of your value chain and what decisions are critical. Where do you make money? Where do you lose money? If you could have a crystal ball, where would you use it in your organisation?</p><p>Decisions that are made at high frequency and are directly tied to earnings are a great fit for AI. Here are a few examples and how you might think of the impact AI could have:</p><ul><li><p><strong>Sales engagement:</strong> If I could predict the best way to engage with my customers in a personalised way, I could improve conversions and net profit.</p></li><li><p><strong>Retention:</strong> If I could predict which customers were likely to churn and what I could do to stop them churning, I could improve retention and net profit.</p></li><li><p><strong>Supply chain optimisation:</strong> If I could predict demand for my products and understand the impact of my operations team&#8217;s actions on available stock, I could prevent excessive capital outlay and improve cash flow.</p></li><li><p><strong>Pricing:</strong> If I could set the right price for all of my products, I could sustainably maximise profit over the long term.</p></li></ul><p>If you&#8217;re stuck, look at your income statement and dive deep. Look for areas of high cost, sources of revenue, and consider the key processes surrounding those items.</p><p><strong>Question 2 &#8211; What kind of insights do you need?</strong><br>If you could wave a magic wand, what information would your dream AI system provide you with? What would the format of that information be? For example, would you just want to predict key events, or would you also want actionable recommendations to achieve the desired result?</p><p>For best results, consider conducting workshops with key stakeholders to brainstorm and prioritise the types of insights that would be most helpful.</p><p>Answering this question correctly is critical in identifying the right AI systems to build.</p><p><strong>Question 3 &#8211; Given your data and the type of output you want, what type of AI should you use?</strong></p><p><br>You&#8217;ve identified several high-impact areas and the type of outputs you want. Now it&#8217;s time to see if some form of AI can help. Assuming you&#8217;ve got at least 6-12 months of operational data related to those areas, you&#8217;ll likely have enough data to train your own AI solutions that can beat human-level performance, if not get close to state-of-the-art.</p><p>Here are a few algorithm combinations associated with some different formats of decision support:</p><ul><li><p><strong>Predict key events before they happen</strong> <strong>(e.g., which customers am I likely to convert this month? Will I run out of stock for a given product next week? Which of my machines are most likely to require maintenance this week?):</strong></p><p></p><p>For numerical data, tabular classification algorithms such as decision trees are a solid choice. For text-heavy data, natural language classifiers are a good option. Ensure these algorithms are configured to predict the occurrence of the event in question in a short-term timeframe. For example, train a model to predict if each customer in your pipeline will convert in the next 30 days based on historical conversion behaviour.</p><p></p></li><li><p><strong>Recommend the next best action to maximise performance (e.g., how can I prevent high-risk customers from churning? What price should I set for my products to maximise volume?):</strong></p><p></p><p>Here, great results come first by building an event prediction algorithm (as above) that understands the impact of your organisation&#8217;s actions on the key event. For example, did calling customers improve their chance of conversion? Did certain conversation topics work well?</p><p></p><p>Once you have that model trained, you can synthetically generate a large number of future scenarios where you might, for example, engage your customers in different ways. Apply your prediction model to identify the likely outcome of your actions in each case and run an optimisation algorithm to choose the best actions. </p><p></p><p>There are more complex approaches, like reinforcement learning; however, it&#8217;s rare to find a business problem constrained enough and with enough data for RL to work well.</p><p></p></li><li><p><strong>Needle-in-a-haystack style search</strong> <strong>(e.g., picking the best founders/companies to invest in, identifying the best candidates for a new role):</strong></p><p></p><p>Here you&#8217;ve got two solid options. </p><p></p><p>First, if you know what good looks like and you have detailed data about the entities you&#8217;re searching over (e.g., detailed professional profiles), your best bet is a class of models known as embeddings models. These take natural language (e.g., your ideal founder profile and your database of founder profiles) and convert it into a format where you can compare different records numerically to get similarity scores. Combine this with some natural language classification to give you filters for each entity to further refine your results.</p><p></p><p>If your records are too similar to one another and the results from the first option aren&#8217;t meeting expectations, you can train a natural language scoring algorithm to assign quality scores to each record based on expert-generated preferences (i.e., a hiring manager first ranks a small set of candidates which your model learns from). This takes more time given the requirement for data labelling but can produce good results.</p></li></ul><p>We&#8217;ll explore more use-cases in future posts, but for now, I hope this demonstrates that there are different solutions for different task types.</p><p><strong>Key takeaways</strong></p><p><br>In this post, we&#8217;ve explored a three-step approach to delivering impact with AI.</p><p><strong>Step 1:</strong> Identify high-impact opportunities directly tied to bottom-line value.</p><p><strong>Step 2:</strong> Be clear about the decision support you need. Brainstorm with your team and identify the kind of insights that will be helpful.</p><p><strong>Step 3:</strong> Pick the right tools for the job. We&#8217;ve explored a small number of the potential options, however if you need support on this, feel free to reach out to us at <a href="mailto:info@truestate.io">info@truestate.io</a>.</p><p>Remember, the key to creating real impact with AI lies in targeting high-value decisions with the right algorithms. Stay focused, stay specific, and the value will follow.</p>

Creating value with AI requires identifying high-impact decisions and applying the right algorithms to make those decisions better. Odds are a conversational AI like ChatGPT isn&#8217;t the best choice