Why data scientists are leaving your company

The company I work for has been actively recruiting for a new data science hire since January. During this process I spoke to a lot of candidates with diverse backgrounds and work histories. They each had their own relevant projects to discuss, and in general very few people gave the same answer about why they were interested in working at a startup like ours.

But almost everyone said the same thing about why they were considering new opportunities in the first place. It's because they stopped learning.

Very few candidates said this explicitly, but instead talked about situations where this was a second-order consequence. Here are some examples, paraphrased from our conversation:

1. Candidate A works at a mid-size public company. The executive team at this company tells investors that they are embracing machine learning, but internally are skeptical about its value. This leads them to halt work on modeling efforts in favor of more traditional methods, and very few projects make it past POC.
2. Candidate B works at a large and succesful startup. After a structural reorganization of the business they were moved to the online experiments team. The new role largely involves running A/B tests.
3. Candidate C works at a Fortune 100 corporation. Their first role was working on a greenfield project with a large scope. As time went on, their projects got smaller and more focused on one particular subject matter area.
4. Candidate D works at small private company. They are the only data scientist on staff.
5. Candidate E works at a mid-size public company. They were hired on as a data scientist after grad school, and were quickly promoted several times. Their work now is largely focused on human resource management.
6. Candidate F works at a mid-size company which is, for good reasons, regulated by the federal government. The regulations impact their work directly by restricting how they are allowed to approach modeling problems.

The stories themselves are quite different, but what they have in common is that their current positions have removed them from the ability to be challenged at learn at work.

In candidate B's case, the proximal cause of this is a mismatch between the job that they applied for and the job that they ended up in. The ultimate cause could be the reorg, or it could be performance issues, or it could be office politics -- it's hard to be sure. But the end result is a data scientist who didn't find their work to be challenging, and isn't on a team that is trying to address online experimentation in new or more efficient ways.

Candidate C looks a lot like candidate B, but in this case the issue is not the lack of a challenge but the reduction in scope. This company has found one subset of one particular kind of modeling problem that they believe is important for the future of their business and is focusing several teams on that. While the business outcomes might be very useful, the team working on this project runs the risk of getting too specialized in this one domain.

Candidate E is a similar story but in the other direction. They were a little too successful and were very quickly promoted up the organization, which in this case meant the management track. As a mid-size, non-technology company, it's possible that they don't have a separate ladder or leveling system for technical leaders and no equivalent of a staff or staff+ role. The only way for them to stay technical is to look for opportunities at other companies.

Candidate A is probably the most interesting story here as it's not a mismatch between the candidate and the role but instead a mismatch between the data science org and leadership. There is a lot to say about this for other reasons, but the key outcome for us here is that the data science org is hesitant to try new things. Candidate F has the same problem, but in their case this restriction is imposed externally through regulation instead of internally through lack of alignment.

It's easiest to see the direct link between candidate D's situation and the lack of personal growth. There are always blog posts to read, videos to watch, and MeetUps to attend, but the link between what the candidate can learn via those avenues and the business value they deliver can only exist in their own head (because the details would be proprietary). In general and for a lot of other reasons, it's not a great idea to have any teams of size 1 and that function probably should have been supplied by some product or service firm.

I don't know for sure, but I would guess that some of the motivation here is an concern abound future employability. E.g. if you stayed in your current role for the next two years, how hard would it be to find a good role at a different company afterward? Some fraction of your future employability is going to depend on experience using modern tools and methods and so there is some implicit Bellman calculation here that includes whatever your next job might be.

Some other fraction of the motivation is about being challenged appropriately. Some candidates mentioned this explicitly -- that they found their current work to be too formulaic, or that it involved an area of data science that they personally did not find interesting. There were probably opportunities to make the work more engaging by encouraging these employees to think about how to automate the boring parts or to link this technical domain to problems they care about, but this requires active and attentive leadership.

What can you do if you want to retain your data science talent?

1. Don't put someone on a team by themselves. If you need less than 1 FTE of a data scientist you can hire a services firm.
2. Ensure their job description either matches what they were hired to do, or what they have asked to be moved into. If this is not the case, it's time for a candid conversation about whether they enjoy their current responsibilities.
3. Data science team members should have a mix of task difficulties. Everyone will have to do things that are easy or tedious, but everyone should also have: opportunities to show their talents where they are strongest; and, opportunities to stretch their capabilities even at a risk of failure or delay.
4. Talk a lot about the throughline from data science projects to business value. This is important not just for the data scientists and their motivation, but for the rest of the business to see the value of good R&D, analytics, and data-powered products.

Also, and as a final thought, it never hurts to start a journal club.

data science recruitment and retention hiring