Implementing Machine Learning Projects

Machine learning provides tactical business insights for marketing teams who want to maximize impact on their spend. In particular when teams require deep analysis of their user's behaviors. For example, an online advertising machine learning model that allocates the ad spend to users who have a higher probability of conversion.

According to the Gartner Data Science Team survey (2017), less than half of Artificial Intelligence (A.I.) projects end up being fully deployed. Not surprisingly, the greatest barrier is the difficulty in deploying A.I. products into existing marketing operations.

Data Science Project Deployed

It's critical to embed these models inside appropriate marketing campaigns that deliver value. For organizations that have a full stack data science teams, the return on investment has been significant.

However, most marketing teams lack the technical skills or access to their data science teams to unveil the opportunities within their own datasets. The focus of data science teams has traditionally been on developing analytical assets, while dealing with the implementing A.I. has been merely a science lab. Many reasons underline that oversight:

  • Lack of a process-centric understanding from the data science team
  • Unwillingness to deal with the operational aspect of advanced analytics, along with a misunderstanding of their practical business impact
  • Lack of a formal operationalization methodology
  • A deficit of formal communication channels between the data science team, the marketing team, and the key executives.

  • Implementing Machine learning projects is a critical step in aligning analytics investments with strategic marketing objectives to achieve organizational goals and values.

    The traditional data science model is a niche experiment, conducted in vitro. Data science teams must ensure Artificial Intelligence models can be traced, monitored, and secured, which requires discipline and management capabilities. Deploying analytical assets within operational processes in a repeatable, manageable, secure, and traceable manner requires more than a set of Application Programming Interface (APIs) and a cloud service.

    The current analytical open-source movement is producing a wide range of analytical assets that will eventually have to be consumed. However, current open-source deployment techniques provide "dissemination means" not "managed production means." From that perspective, containerization techniques are efficient and effective in pushing models into production, but they do not relieve the organization from adopting a formal implementation process.

    But establishing best practices within the marketing department requires a commitment from key stakeholders. It starts with clearly identifying the organization's mission, objectives, and goals. Then systematically applying a model life cycle discipline. This takes a commitment from upper management to invest the necessary time, talent, and resources necessary to stay competitive.

    Implementing Machine Learning Stage 1: Model Development Cycle

    Bonding the Organization.

    No machine learning project should be started without understanding its mission, objectives, and goals. Key stakeholders (e.g. namely, the CEO, CTO, and CMO) should be able to clearly articulate what the tangible business benefits they are expecting from the cross polination of teams. Questions to be considered include:

  • What is the market problem that the business is solve?
  • Who is the primary beneficiary of the objective?
  • What is the business process that will host that model?
  • How is success measured?
  • Gartner developed a Business Value Model that can be used to establish the appropriate set of metrics. This Business Value Model is a structured framework and definition of nonaccounting metrics that can be applied generically to help organizations identify how their marketing activities will impact financial performance. Ultimately, the goals should be specific and measurable. For example, increasing online conversions by 10%, or reducing churn by 7%, or improving cross-selling on item X and item Y by 15%.

    In addition to defining the marketing goals of Artificial Intelligence projects, the exercise (not experiment) has a symbiotic function: to bond the data science team with the marketing department, which align with the organization's mission, objectives, and goals mandated by the executive suite. This alignment is critical during the implementing phase and increase the probability of repeat success.

    It is important to note that A.I. has often baffled data scientists with seemingly irrational decisions. In many cases, these irrational decisions often demonstrate A.I.'s ability to think intelligently (or unintelligently). In both circumstances, model interpretability is gaining importance as more organizations implement machine learning into marketing campaigns. Model interpretability is critical to understand why a particular A.I. model succeeded, or failed. It is essential to explain conclusions when results deviate from expectations. More simply, A.I. trainers need to be able to explain how an insight was derived. However, attempting to understand why a conclusion occurred, is as complex as the human brain and its impact on our behavior (outside the scope of this piece).

    Recommendations:

  • Identify critical team members, including data science/A.I. trainers, marketing stakeholders, and key executives.
  • Develop use cases, while clearly defining specifica and measurable marketing outcomes.
  • Prioritize objectives based on their technical feasibility and measurable economic viability.
  • Create a Systematic Operationalization Process

    The model development cycle is a variation on the Cross-Industry Standard Process for Data Mining (CRISP-DM) methodology that has been used by data science practitioners for two decades. This methodology is simple, powerful, and imposes a development system that promotes robustness. The development cycle has always been the glamorous part of the full analytical process, which developers tend to gravitate towards.

    The model operationalization cycle, on the other hand, is less glamorous but it's where business value is realized. Neglecting the operationalization cycle has often prevented organizations from fully realizing the business value of their A.I. projects, while generating distrustful attitudes toward data analytics.

    Operationalization Cycle Functionality

    Successful marketing teams have adopted various technologies to provide the foundation to manage operational decision services in a reliable, repeatable, scalable and secured way. That foundation should provide:

  • Catalog An agile method to store and secure data to make it easier for analysts to collaborate. This allows data scientists to access, reuse or swap models or datasets quickly and easily.
  • Governance Protocols to ensure regulatory compliance. This allows teams to scale as data aggregates, reducing technical debt.
  • Capabilities Automated versioning, fine-grained traceable model scoring and change management capabilities (including champion/challenging features) to closely test, monitor and audit analytical asset life cycles from a technical as well as a business performance perspective (through Key Performance Indicators).
  • Coherence Establish simple procedures to provide a fluid transition between the development and operationalization cycles. This strengthens communication between development and operationalization teams, while provide efficient "translation" services between data science and the marketing team.
  • Operationalization Cycle Process

    Once a model has been released from the development cycle, it enters into the operationalization cycle. This takes place in two main phases: the release phase and the activation phase. The goal of the release phase is to run a model in a live sandbox to test the model with real world data. Once the model has been tested and proven in real world samples, it is ready for full deployment in the activation phase. The goal of the activation phase is to launch the model into existing business operations identified (and validated) in the release phase of the production process.

    Depending on their project scope and complexity, not all organizations undergo two phases. It is possible to integrate the release phase as part of the development cycle.

    It is possible to launch a live single phase model to provide insights within a isolated campaign. However, with less testing and with greater variables, the possibilities of errors will increase exponentially.

    Release Phase: Live Testing

    Implementing Machine Learning Stage 2: Release Phase
  • Management & Governance Once a model is ready for activation, it should be cataloged, documented, and versioned for quality control.
  • Model Activation The production-ready models transitions from operationalization-process-validated models to the production team.
  • Model Deployment Whether the models are run on-premises or in the cloud, measures need to be taken to smooth the transactions leveraging the model.
  • Application Integration Insights are delivered through decision milestones with the large majority applied to existing applications. This stage should reveal early insights, which address the project objectives.
  • Production Audit Procedures Production needs to be monitored for accuracy, confidence levels, infrastructure load, etc. These data points are used for baselining or comparison with existing production models. Other tests, such A/B testing, multivariate testing, Multi-Armed Bandits (MABs), and shadow models can be implemented compare the efficacy of the A.I. project.
  • Model Tracking Behavior Alerts and monitoring methods have to be established to track the performance of models in production. Performance thresholds and notification mechanisms are implemented in this step to systematically detect anomalies.
  • Key Performance Indicators KPIs measures the effectiveness of the A.I. models in production. The purpose is to measure the business value that can be attributed to various model so tactical decisions (e.g. budget and resource allocation) can be made.

  • Operationalization Phase

    Implementing Machine Learning Stage 3: Operationalization Phase

    The operational cycle will continue to revolve as long as the performing, meeting attribution thresholds, and aligning with the campaigns objectives. As long as models are performing, they keep their place in the production cycle. Should the models require re-evaluation, the models are then recycled back into the development cycle process.

    Machine learning systems carry a significant technical debt that has to be addressed upfront. To confront the problem early, the data scientists should:

    1. Monitor the KPIs that determine the success of analytical models in production
    2. Organize and manage the performance of machine learning models deployed across the marketing campaigns
    3. Maintain data governance to ensure regulatory compliance and data custodianship

    Teamwork

    Typically, data science teams work in silos. When it comes to operationalizing Machine Learning into marketing campaigns, orchestration is required between cross functional teams. These include:
    System Experts At the application level, system experts have intimate knowledge of the data flows and sources.
    System Architects Architects have intimate knowledge of the infrastructure supporting the processes, systems, and applications that will integrate machine learning models.
    System Administrators Maintain IT operations and enforce coherence to regulations.
    Application Developers The builders of product applications along with their User Interface/Experience (UIUX) counterparts.
    Process Engineers Process Engineers understand the complexity and middleware relationships of the various business processes leveraging analytical assets.

    Beyond these roles, a business translator orchestrates and communicates the overall goals of machine learning projects. The business translator could be a business savvy data scientist or inversely, a technical marketer. Business translators have a strong understanding of the technical workings and requirements as well as the high level business structure. This individual typically emerges as the common denominators to the executive stakeholders.

    Recommendations

  • Work closely with core talents from the IT department and get corporate buy-in from the executive suite.
  • Recruit business translators (this could be you) who will become the trusted leader of the machine learning project.
  • Provide the data science team with a clear understanding of the marketing mission, goals, and objectives.
  • Establish a production committee, which may include the head of data science, an I.T. operationalization leader, the business translator, and the application product manager. This team oversees the full machine learning project cycle from model creation to reployment.

  • Monitor and Modify Machine Learning Models in Production

    The Key Performance Indicators (KPIs) should to be revalidated while the model is in production. Market conditions can sift and consumer preferences may change, shifting the mission, goals and objectives of the organization. Although these variables can degradate machine learning models, others factors to consider are:

  • Concept Drift where seasonality, decision impacts, local data particularities, and dynamic data transitions can induce a change in class distribution
  • Bias Reinforcements where decisions are increasingly subjective through models' self-fulfilling recommendations
  • Feedback Loops where models influence their own behavior as they evolve over time

  • KPIs should also be subjected to a higher level of scrutiny across the business processes that are impacted by the machine learning decisions or recommendations. This will avoid the local minimum problem where a few decisions are optimized for marginal issues, ignoring more critical problems. Through continuous machine learning audits, refinements can be made of the model to glean insights and actionable tactics.

    Recommendations

  • Secure the commitment of the executive stakeholders at the problem identification stage. This provides continuous validation of business KPIs while the model is in production.
  • Track the models degradation signals by constantly monitoring the organization's mission, goals, and objectives influenced by productized models.
  • Measure and refine the overall business impact of integrating machine learning models by comparing baseline KPIs to quarterly results.

  • Summary of Challenges:

  • Organizations often start their machine learning efforts as science projects, without first understanding the mission, goals, and objectives.
  • Marketing teams often struggle with integrating machine learning processes, as the analysis is tedious, time consuming, with varied results.
  • Data scientists and A.I. trainers tend to spend most of their time collecting and cleaning data. It is a necessary evil, but can take away from developing models
  • Systematically productizing machine learning operations can be a struggle, as the production process is either overlooked or left solely to the DevOps team.
  • Even if successfully productized, machine learning efforts still face significant maintainability and scalability challenges.

  • Lessons to learn:

    To launch your organization's machine learning projects, marketing teams should:

  • Start by leveraging your current datasets (e.g. Google Analytics) to find quick wins independently (e.g. determining attribution using Markov models)
  • Work closely with data science teams. They will be busy (we all are), but demonstrate your proficiency with your independent findings before graduating to complex challenges, which require larger datasets.
  • Ensure the integrity (technical and economic), transparency and sustainability of deployed analytical models by establishing a systematic operationalization process.
  • Securing the help of domain experts in Machine Learning (both internally and externally). Remember, this is a young field and everyone is learning and exploring different tools and techniques.

  • Ask your doctor about Morfene.

    We can help you discover deep insights by integrating Machine Learning processes into your marketing campaigns.