Here’s how Companies can reduce the friction between Data Scientists and Product Developers

Alright, picture this: You’re a month deep in building a product, under pressure to hit the deploy & sales. You’re continuously appending & rectifying. Amidst the chaos, you ask yourself: “Why are we even building these features?”.  As the team shifts from data to design and development, it’s inevitable that the focus too shifts from practical details to inclusive details.

In this data-driven age, can any digital product or service company establish itself in the market without the optimum use of its data? The answer is a big No. Digital companies like Facebook and LinkedIn are as popular for their service offerings as they are for the innovative ways that they use data to improve their products. For example, Facebook uses product teams comprising of both product developers and data scientists to measure the positive (or negative) impact of a new product feature before releasing it to over 2 billion Facebook users.

With the growing importance of Big data in product development, software engineering teams (or product developers) are required to increasingly collaborate with data specialists including data scientists and data analysts. 

Spotify, the audio streaming company, is driving cross-disciplinary collaboration between user researchers and data science teams to drive better product decisions. Similarly, food tech company, Zomato is creating a data-driven culture with a 150-member engineering team that includes both data scientists and product managers.

So, how do product companies ensure a productive collaboration between these 2 teams for the purpose of developing & delivering an ingenious product?

Through this article, we shall understand the critical role played by product developers and data scientists in product development along with useful tips on ensuring a fruitful collaboration between the two.

Data Science vs Product Development

In simple terms, product development (or software engineering) is defined as the structured approach towards the development and maintenance of a successful software product. Using the Software Development Lifecycle (or SDLC), product development teams use a variety of tools in product designing, database management, or web application to develop a robust solution.

On the other hand, data science is a discipline aimed at effective analysis and management of data in order to derive useful business insights and enable high-quality decision making. A data scientist uses a variety of data analytics and visualization tools to extract a deeper business understanding of data from sources such as social media platforms, business apps, and public data.

How do Data Science functions differ from Product Development? Here are a few pointers:

	Product Development	Data Science
Role	Build a robust product.	Analyze and convert business data into useful knowledge.
Data sources	The end-user requirement, desired product features	Big Data sources including Social media data, business transactions, public data, business app data
Enables the	Development of new products or features with a structured approach	Informed decision-making process based on the analysed data
Driving factor	End-user requirements	Data availability
Building blocks	Product design, Writing code, Testing	Data modeling, Data algorithms, Machine learning, Business Intelligence
Approach	Based on project management frameworks like Design Sprints, Agile, lean or Waterfall methodologies.	Based on process-oriented approaches such as data pattern recognition and data algorithms.
Roles	Software developers, Product Managers, Product Testers, UI-UX designers.	Data scientists, Data analysts, Big Data specialists, Data engineers
Required skillset	Core programming skills, Testing tools, Product build tools, Prototyping, Running Design Sprints.	Product domain knowledge, Data mining, Machine learning, fundamentals of AI, Unstructured data processing, Data algorithms.

Despite the obvious differences between the 2 disciplines, product companies stand to gain business benefit through their efficient collaboration and teamwork. In the following sections, we will explore the major challenges faced by both product development and data science functions (from each other) along with practical tips to overcome these challenges.

Common Complaints from Product Managers

Here are the common complaints that Product Managers have about Data science specialists:

i) Product features

Executive decision-makers decide on new product features or product lines based on useful data insights. Product development managers have the rationale that these stakeholders do not actually use the product hence cannot be in any position to determine the value of new features. The data-centric approach is a violation of the more dependable customer-centric approach. Product data that does not offer useful answers to common concerns related to product usage, user experience, and competitive product features are simply not relevant enough.

ii) Event Data or Customer Feedback

According to product managers, no event-based data can compare with the insights gained by directly talking to a user and taking their feedback. Observational event data (taken from the product itself) can be an effective starting point but needs to be combined with user feedback data for finding a good product-market fit.

iii) Cost factor

This relates to product management concerns on whether data sciences can justify its high cost and can effectively improve the customer revenue (and not just make for happier customers). For example, is the latest Machine learning project driving growth in ROI, or did implementing product upsells (using Artificial intelligence) increase customer purchases?

Common Complaints from Data Scientists

Among the reasons for the high failure rate of data analytics projects, data scientists complain of being treated as technical resources rather than being valued for their business relevance. Some companies lack the focus of defining business goals and opportunities that need to be addressed by data science specialists. 

While skills like Python programming, data mining, and statistical analysisremain among the leading skills for data scientists, it is equally important to involve them in the business process, learn how work is conducted, and also connect with other company functions.

How to enable collaboration between Product Development and Data Science

Listed below are 4 useful tips on how to bridge the divide between these 2 important functions and encouraging a collaborative environment:

i) Building a data culture in the company

Product managers collaborate efficiently with data science function when business data is part of the daily decision-making process at each and every level. Implementing and building a data culture where data-driven decision-making combines data from across organizational functions ensuring the market success of your software product even before it has been released.

Image Source:  https://i0.wp.com/www.wholewhale.com/wp-content/uploads/2013/12/how-to-build-data-culture1.png?fit=1000%2C486&ssl=1

Collaborative frameworks that utilize metrics such as the customer journey map, user persona, and lifetime value can build and guide effective collaboration between product development and data science teams.

ii) Implementing a Data Lake storage repository

The question that product managers face is how much access to production data must be given to data scientists. “Too less access” means that data scientists have to wait long for data inputs, while “too much access” typically result in more-than-necessary access to the production database resulting in production delays. 

Image Source:  https://d1.awsstatic.com/Data%20Lake/320x320-what-is-a-data-lake.b32634fa96e91bb5670b885be9428a2c0c40c76d.png

A data lake enables the sharing of raw business data with data scientists that is separate from the production data. While the product development team can provide the application data, data scientists must ensure that this data is stored in raw format in the data lake. This requires minimum levels of involvement from the production team, while the data sciences team takes the call on the type of data format and data schema to be used.

iii) Data interpretation

Though effective, data interpretation can be a problem for most project managers and even for experienced data scientists. Incorrect data interpretation can occur from incomplete data and even from accurate data.

Hiring a data scientist as part of the product development team can be valuable when it comes to data interpretation. Alternatively, a collaborative team comprising of multiple professionals (with varying skill sets) can be the right answer to effective interpretation.

iv) Building a data science toolbox

Building a data science toolbox with high-level abstractions can lead to a deeper understanding of the tools that data scientists require for data exploration and derive business value.

Companies can include a software engineer in their data science team so as to review existing code and append new functionalities into the toolbox. Software engineers can utilize their knowledge of building modular software to define the data science toolbox requirements gradually over time.

v) Data Science and Agile

The use of Agile methodology and design sprints have been widely adopted by software teams for the purpose of designing and prototyping a product (or product feature). The question is whether Agile can work successfully with data science? 

The answer is that yes, Agile can be incorporated into the following aspects of data science:

1) Planning and prioritizing data science-related tasks before each design sprint (that typically lasts around 1-2 weeks).
2) Defining the data science task (or problem) along with setting the deadline (or timeline) to mitigate the problem.
3) Conducting a retrospective session (after every sprint) to reflect on the achievements (or failures) of the data science team.

A collaborative environment comprising of product engineering, Agile specialists, and data science specialists can help in seamless project management and delivery.

Conclusion

While data science focuses on deriving deeper business insights from user data that can lead to a more productive and accurate decision-making process, product development focuses their efforts into building more feature-rich and customer-centric product solutions. Multidisciplinary teams comprising of data analysts, software coders, digital marketing professionals, and UI designers are now critical for making really cool product decisions. 

This article presents an outline of how these functional domains can collaborate more efficiently to build software products that bring more market success and revenues to the company.