Diptanu Choudhury, a technological visionary, has contributed enormously to the building of the infrastructure on which the world today is connected. Diptanu has a career spanning across leading tech companies like LinkedIn and Facebook and has led his own venture, Tensorlake, that deals with problems related to large scale data management.
His work in distributed computing, most recently in Nomad and Titus, has revolutionized industry standards for efficiency, reliability, and scalability in Internet systems. These projects showcase both technical ingenuity, as well as strategic foresight on optimizing systems for performance and utilizing AI in a smart way. As a dear leader in the tech industry, Diptanu has significantly catalyzed the wave of innovation that has contributed firmly to his place as one of the leaders who have helped in the evolution of global digital connectivity.
Diptanu Choudhury
A Catalyst for Change
Diptanu’s first foray into large scale systems was during his university days when he co founded a mobile social networking platform targeted at connecting people across India. This early began his love for system design at scale and set the stage for what was to come. He continued his innovation at Netflix, where he created Titan, a critical cluster scheduler which is still an integral part of the company’s cloud infrastructure.
This is an industry that is constantly changing, so in order to stay ahead of the crowd you need to be learning and doing it constantly. Diptanu recalls, “I’m always looking at new technologies and new trends, I attend workshops, I’m always in touch with academic institutions.” He is very involved in conferences, technical forums and industry discussions to grow professionally, and to keep his projects on the cutting edge of innovation.
Breaking Language Barriers
Diptanu played a key role in dismantling language barriers in technology at Facebook. Realizing that speech recognition tools can’t be used by non-English speakers, he worked to develop advanced language identification models. Diptanu explains: “We expanded our speech recognition capabilities to support underrepresented languages, bringing communities that were not served by Facebook’s platforms, to Facebook’s platforms.”
Based on this progress, Diptanu’s team was directed to create tailored models for these newly identified languages. ‘Then we created speech recognition systems tailored to these languages, and expanded Facebook’s accessibility,’ he remembers. This initiative has greatly increased global connectivity, enabling a wide range of users whose native languages were not previously included into much more natural interaction with Facebook’s services in their own languages.
Lessons from Experience: Resilient Infrastructure
Diptanu especially learned a lot from his work on critical projects like Nomad and Titus: designing infrastructure that is both robust and scalable. According to him, “Systems can break unexpectedly — there can be sudden spikes in traffic, human error, or unforeseen events.” His philosophy for embedding reliability into the foundation of system design was based on this firsthand understanding.
“Reliability is something you have to design into your product,” says Diptanu, “and you really need to work on it from the beginning.” His strategies go beyond the design phase to long term system management. “We kept the fault domains to a minimum and we simplified the architecture so that it would be resilient under operational stress,” he says. In addition to increasing performance in critical moments, this focus on structural efficiency also provided stability during times of demand surge.
System Optimization and AI: A Strategic Approach
Diptanu’s approach to integrating AI in large scale systems is deliberate and deliberate, with technical solutions being engineered to achieve business objectives. He starts by saying that he defines the business metrics that will help measure the success of the AI implementation, and highlights the need to keep technical efforts in sync with the broader organizational objectives.
His teams do small scale experiments with small amounts of data. “The good thing about this is we can test ideas without it being too costly to us,” he notes. When these tests are promising they expand the scope to refine the models to make them better learners. Diptanu adds that, “Once the model performs as per the benchmarks we set for it, we optimize it for inference to trade off effectiveness with the computation power we have available to us.”
Mentoring the Next Generation of Innovators
Diptanu understands that nurturing talent is crucial, he therefore takes time to mentor his teams and to create a learning culture. “I’ve opened the door to thought leaders in industry and academia to work with my teams throughout my career,” he says. This exposure also helps his teams get out of the box and think beyond the immediate tasks and to explore new ideas that break the conventional boundaries.
Diptanu organizes reading groups with the focus on academic research, as well as industry trends to cultivate collaborative learning. “With these discussions, my teammates get a chance to go deep on complicated things together,” he says. Not only does this structured learning environment expand knowledge of individuals, but it promotes teamwork, which is necessary for driving innovation in large scale systems.
The Challenges of Digital Connectivity and Navigating It
Energy consumption is becoming a major challenge in the development of digital infrastructure, and Diptanu points out that this will only become more apparent as AI technology becomes more prominent. The biggest challenge that lies ahead is power, how much we can make and deliver to data centers, he adds, calling for energy efficient ways to make it all work.
Energy challenges are not the only things that Diptanu mentions, there is also the need for regulatory frameworks. He notes we need to rethink how information access is regulated as connectivity develops. These are challenges, he acknowledges, that more closely reflect the reality of global digital expansion, in which technological progress must be in harmony with ethical and regulatory concerns.
The Future of Technology Shaping
Diptanu believes that technology can solve some of the biggest challenges of our society. He asserts that for centuries engineering and technology have propelled human progress, and underlined their role in addressing the global challenges of today, from climate change to food security. He champions solutions that are accessible, energy efficient and able to scale. Diptanu’s key is to develop technologies that need to be affordable and sustainable, he says, in practice, a responsible innovation.
From a technology point of view, he believes in the potential for technology to improve quality of life and support environmental sustainability. Diptanu was one of the pioneers of large scale Internet systems and his example is a strong reminder of the disruptive potential technology can be. But his story is also about personal achievement, and the vision to connect and include, innovatively.
