Row 8043

Row ID: 8043 | Dataset Entry | Axioma AXP Content Repository

Content Data

This page contains data entry 8043 from the Axioma AXP content repository. The structured data below represents the complete record for this entry.

I joined a beautiful ML/DL RnD project entering its product phase. I'm a research scientist hired to unstuck the project. I'm supposed to turn the work of 10ish data scientists into a deployed solution.

Turns out another team has a senior Cpp SWE who got his hands on all of the projects critical components: embedded software control, data storage and format, architecture, pipeline orchestration... He's the only one working in Cpp, everybody else works in Python, me included.

Because he sprayed Cpp everywhere, and built the servers everything has to go through him. And he won't to work with anything that's not in Cpp. He thinks Python is too slow and nothing ever fits our "specific needs" (without any proof whatsoever).

So he's been developing dashboards in Cpp, he created a binary format to store matrix files (the standard in our field is hdf5), doesn't have CI/CD in place, never heard of MLOps, he even uses his personal GitHub because our company's Gitlab does not fit his needs...

He's creeping into the DS-team perimeter by constantly imposing his own Cpp code with Python binding: He created a torch-style Dataset, reinvented the Dataclass. Last I heard he wanted to create a library to perform matrix computations "because numpy arrays can't store their own metadata" (wtf). At some point he even mentioned writing his own GPU SDK (so writing CUDA basically...).

Basically everything MUST be custom made, by him, in Cpp. If you're not managing L1 cache yourself, your code is garbage, regardless of the requirements.

His next move is now to forbid us from deploying/orchestrating our ML as we see fit. Instead he wants to call our code inside the Cpp process. This is a move that allows him to write his own orchestration software when so many open source solutions already exist.

My opinion is that this guy doesn't care about the project and just wants to have fun solving algorithmic problems that are already solved by a pip install.

The result being that it's impossible for the team and myself to contribute and upskill. The DS team work-quality remains abismal because they have no clue about production constraints. They do a for loop : he rewrite to Cpp. The project can't move forward.

I'm stuck playing politics when I was told I'd be doing deep learning on petabytes of data.

I'm 4 months in and got opportunities to go elsewhere... Anyone here been in a similar situation? Did things get better after a while ? Should I just ditch this project ? This is obviously a rant but I'm genuinely curious to hear about your stories ...

Edit: Wow, I didn't expect so many responses, thank you all. My plan was to convince the SWE of Python's and Docker quality. I understand management is the new target (he will always respond "I can do this myself").

From what's been suggested my current plan is the following:

1- wait and see if the team meets deadlines and milestones I've set after my arrival.

2- if not, talk to the managers, explain the situation and request that this SWE be focused on his perimeter: embedded software, sysadmin and optimisation upon request. He should let DS do their job for the following reasons:

a) Upskilling: Cpp refacto and SWE scope creep prevents DS to upskill and enrollment of future staff.

b) Maintability: our ML codebase must be in a format that uses standard tools (Python vs Cpp, docker vs Cpp, hdf5 vs custom, numpy vs Cpp, cloud vs on-prem...).

c) Velocity: 10 upskilled DS will write code and train models faster than the SWE can refacto in Cpp

d) Quality: DS know better what features are needed. If we need parallel computing and L1 cache management they'll ask. The SWE should be supportive instead of imposing his solutions.

e) Flexibility: DS must own and understand the stack if they want to try new things.

f) Security: this SWE creates security risks by not complying with the company policies/tools.

g) Independence: the current workflow and architecture are putting us at risk in case the SWE leaves.

Meanwhile I'll find project examples and codebases that meet our requirements using standard industry tools and languages.

4- if things don't improve fast I'll leave.

FieldValue
text I joined a beautiful ML/DL RnD project entering its product phase. I'm a research scientist hired to unstuck the project. I'm supposed to turn the work of 10ish data scientists into a deployed solution. Turns out another team has a senior Cpp SWE who got his hands on all of the projects critical components: embedded software control, data storage and format, architecture, pipeline orchestration... He's the only one working in Cpp, everybody else works in Python, me included. Because he spray…
label r/datascience
dataType post
communityName r/datascience
datetime 2024-05-18
username_encoded Z0FBQUFBQm5LakwzZk5hb0dhUU83Q2Y0ejFYZ2JValZ2MkhKZ2pVQlh4ZVZoYkdLckFEYUdCRmpSSTVKa3FwTzMzeW1TeDBSemxtQWV5T1MxUk5uWjIydk9DXzlwR09WOW9yVkhPS2xNcEo2WWQ2LVVPSW1LT3c9
url_encoded Z0FBQUFBQm5Lak9IenM5R2tTWUVfZjkzQ1p4cTJHSVpZVkNpQkI2bjV3ZWl2cWs4VHp0UVNnOVBaeXdFWWJmcUZLN3RDZHEyaEw4ckExQ0V3dkNkclgzUnd5X0VMb3pPb2VFbXB4bDZjLVh1c3Q3WmhhRy05eEFkMndNWHRVRXJ0QWk2UnEySU0xXy1obUw2VlF0Njh5c1M0Tmo5TVQwTS1KOFB6eTdpdDhoalZuY1JNSVlUUzJhOTZhLVpFLU9jejBBN2lCakI1MENw

Raw Record

{
  "text": "I joined a beautiful ML/DL RnD project entering its product phase. I'm a research scientist hired to unstuck the project. I'm supposed to turn the work of 10ish data scientists into a deployed solution.\n\n\nTurns out another team has a senior Cpp SWE who got his hands on all of the projects critical components: embedded software control, data storage and format, architecture, pipeline orchestration... He's the only one working in Cpp, everybody else works in Python, me included.\n\n\nBecause he sprayed Cpp everywhere, and built the servers everything has to go through him. And he won't to work with anything that's not in Cpp. He thinks Python is too slow and nothing ever fits our \"specific needs\" (without any proof whatsoever).\n\n\nSo he's been developing dashboards in Cpp, he created a binary format to store matrix files (the standard in our field is hdf5), doesn't have CI/CD in place, never heard of MLOps, he even uses his personal GitHub because our company's Gitlab does not fit his needs...\n\n\nHe's creeping into the DS-team perimeter by constantly imposing his own Cpp code with Python binding:\nHe created a torch-style Dataset, reinvented the Dataclass. Last I heard he wanted to create a library to perform matrix computations \"because numpy arrays can't store their own metadata\" (wtf). At some point he even mentioned writing his own GPU SDK (so writing CUDA basically...).\n\nBasically everything MUST be custom made, by him, in Cpp. If you're not managing L1 cache yourself, your code is garbage, regardless of the requirements.\n\nHis next move is now to forbid us from deploying/orchestrating our ML as we see fit. Instead he wants to call our code inside the Cpp process. This is a move that allows him to write his own orchestration software when so many open source solutions already exist.\n\n\nMy opinion is that this guy doesn't care about the project and just wants to have fun solving algorithmic problems that are already solved by a pip install. \n\n\nThe result being that it's impossible for the team and myself to contribute and upskill. The DS team work-quality remains abismal because they have no clue about production constraints. They do a for loop : he rewrite to Cpp. The project can't move forward.\n\n\nI'm stuck playing politics when I was told I'd be doing deep learning on petabytes of data.\n\n\nI'm 4 months in and got opportunities to go elsewhere... Anyone here been in a similar situation? Did things get better after a while ? Should I just ditch this project ? This is obviously a rant but I'm genuinely curious to hear about your stories ...\n\nEdit:\nWow, I didn't expect so many responses, thank you all. My plan was to convince the SWE of Python's and Docker quality. I understand management is the new target (he will always respond \"I can do this myself\").\n\nFrom what's been suggested my current plan is the following:\n\n1- wait and see if the team meets deadlines and milestones I've set after my arrival.\n\n2- if not, talk to the managers, explain the situation and request that this SWE be focused on his perimeter: embedded software, sysadmin and optimisation upon request. He should let DS do their job for the following reasons:\n\na) Upskilling: Cpp refacto and SWE scope creep prevents DS to upskill and enrollment of future staff.\n\nb) Maintability: our ML codebase must be in a format that uses standard tools (Python vs Cpp, docker vs Cpp, hdf5 vs custom, numpy vs Cpp, cloud vs on-prem...).\n\nc) Velocity: 10 upskilled DS will write code and train models faster than the SWE can refacto in Cpp\n\nd) Quality: DS know better what features are needed. If we need parallel computing and L1 cache management they'll ask. The SWE should be supportive instead of imposing his solutions.\n\ne) Flexibility: DS must own and understand the stack if they want to try new things.\n\nf) Security: this SWE creates security risks by not complying with the company policies/tools.\n\ng) Independence: the current workflow and architecture are putting us at risk in case the SWE leaves.\n\n\nMeanwhile I'll find project examples and codebases that meet our requirements using standard industry tools and languages.\n\n\n4- if things don't improve fast I'll leave.",
  "label": "r/datascience",
  "dataType": "post",
  "communityName": "r/datascience",
  "datetime": "2024-05-18",
  "username_encoded": "Z0FBQUFBQm5LakwzZk5hb0dhUU83Q2Y0ejFYZ2JValZ2MkhKZ2pVQlh4ZVZoYkdLckFEYUdCRmpSSTVKa3FwTzMzeW1TeDBSemxtQWV5T1MxUk5uWjIydk9DXzlwR09WOW9yVkhPS2xNcEo2WWQ2LVVPSW1LT3c9",
  "url_encoded": "Z0FBQUFBQm5Lak9IenM5R2tTWUVfZjkzQ1p4cTJHSVpZVkNpQkI2bjV3ZWl2cWs4VHp0UVNnOVBaeXdFWWJmcUZLN3RDZHEyaEw4ckExQ0V3dkNkclgzUnd5X0VMb3pPb2VFbXB4bDZjLVh1c3Q3WmhhRy05eEFkMndNWHRVRXJ0QWk2UnEySU0xXy1obUw2VlF0Njh5c1M0Tmo5TVQwTS1KOFB6eTdpdDhoalZuY1JNSVlUUzJhOTZhLVpFLU9jejBBN2lCakI1MENw"
}
Explore Dataset Explore Row

Entry Information