Samsung Research

Seoul, South Korea
Computer Vision Engineer, Next Generation Digital Appliances Research Division
2023 - Current

Key Achievements:

• AI Model Development:
  Developed and deployed a vision-based AI model for millions of Samsung’s next-gen smart refrigerators using TensorFlow, PyTorch, and OpenCV, enhancing user experience by automating food identification and inventory management.

• Core Expertise:
  Worked on:
  • Object detection
  • Instance detection
  • Feature matching
  • Classification
  • Temporal trajectory detection
  • Position detection
  • Gaze prediction
  • Human activity recognition
  • Label detection, transformation, and matching

• Out-Of-Distribution (OOD) Detection:
  Addressed OOD detection issues and improved model accuracy by 4% through the use of openset loss, contrastive loss, and ensemble methods, significantly reducing misclassifications and improving reliability in diverse conditions.

• NPU Optimization:
  Optimized AI models for NPU deployment, achieving performance metrics of:
  • <40MB size
  • <70ms latency
  • >90% accuracy
    Ensured seamless integration into refrigerator systems without compromising performance.

Samsung AI Inside Refrigerator

Bespoke 4 door refrigerator

Link 1

Link 2

Excerpt shared

Lessons Learned from Applying On-Device AI Models and Services in Refrigerators

1. Analyzing Service Suitability for On-Device AI from the Planning Stage

The project began by applying food recognition in refrigerators to provide immediate feedback to users, identifying which food was added or removed within just a few milliseconds. Initially, large models were used, but to optimize the system for on-device performance, we separated the AI models by function and abandoned the initial structure, redesigning the entire system from scratch.

As a result, the costs incurred was saved, enabling the offering of additional services. This approach not only enhanced usability by providing real-time (ms-level) feedback but also helped differentiate the service by leveraging real-time capabilities.

2. The Importance of “NPU”

We spent considerable time understanding the characteristics of the NPU (Neural Processing Unit) and, through numerous setbacks, gained valuable insights. We repeatedly tuned the model using techniques such as Quantization and Knowledge Distillation to optimize it for the specific NPU being used. When the limits of the system were reached, we revised the functional design, rechecked user scenarios, collaborated on UX/UI, and adjusted performance accordingly.

For layers not supported by the NPU, modifications were made to ensure optimal performance. Without the NPU, I would not have delved deeply into each network, but in the process, I gained significant knowledge and experience.

3. AI Data in the “Refrigerator”

It became clear that the model’s performance significantly improved when we began processing data directly. Despite limited resources (especially time and money), careful planning was implemented to maximize data collection efficiency, but there were still many shortcomings. In previous projects, extensive data collection and annotation had been carried out, but the change in circumstances led to unexpected challenges. Despite this, we worked diligently to develop the system within the given time and environment. Data collection remains an ongoing task, with continuous improvements being made.

4. Home Appliances = Team Play

The development of home appliances is not something that can be done in isolation. Hardware (mechanics, circuits), software (apps, servers), UX, design, planning, services, quality, marketing, sales, purchasing, legal, patents, and support teams (special thanks to those who helped with the camera and data) all played essential roles. Each team worked efficiently and effectively in their respective areas to ensure the project’s completion. Though initial opinions didn’t always align, I found great satisfaction in resolving issues through marathon meetings and collaboration. By working with the business unit , I not only developed technical skills but also slightly improved project management (PM) capabilities.

5. Why Is There a Camera in the Refrigerator?

The real-time On-Device Food Recognition feature is still in its first release and is a pioneering effort not pursued by other companies, leading to many trial and error processes. However, the Daily Active Users have been steadily increasing. We monitor refrigerators worldwide, analyzing issues, improving the service, updating, and observing consumer reactions.

The goal is to make cameras a standard feature in all refrigerators and create an AI service that users will embrace and it will make their lives convenient.

Software Engineer, Data Service Lab

January 2022 - January 2023

Key Achievements:

• Samsung Health Research Application Deployment:
  Deployed the Samsung Health Research Application, facilitating participant life-log data analysis at Samsung Medical Center and other hospitals. Designed the end-to-end system architecture using Clean Architecture to enhance scalability and maintainability.

• Client-Side and Server-Side Development:
  • Developed the client-side application using Kotlin and the Android Framework to optimize user experience.
  • Engineered the server-side infrastructure using Node.js to ensure secure and efficient data handling and integration.
• Data Visualization Dashboard:
  Implemented a data visualization dashboard using Apache Superset, allowing for easy interaction and insight derivation from complex data. Collaborated closely with medical researchers to streamline digital solutions for medical research processes.

Press Release Press Release 2 Press Release 3

Samsung Health SDK suite

Data Science Engineer, Data Intelligence Lab

October 2020 - January 2022

Key Achievements:

• Data Analysis on User Database:
  Conducted extensive data analysis on a 200 million user database within Samsung Health, modeling weight, BMI, sleep, exercise, and step habits to unveil critical health trends and behaviors using AWS EMR, Athena, Spark, Glue, and BigQuery.

• Data Decryption and Analysis:
  Performed decryption, pipelining, and data analysis on 300TB of anonymized health data stored on AWS S3, analyzing trends by demographics and external factors such as COVID-19 and lockdowns, leading to key insights into patterns and correlations.

• Health Markers Analysis:
  Established causal relationships between physical and mental health markers using statistical tools and time series forecasting, laying the groundwork for personalized health interventions based on factors like sleep, exercise, steps, age, screen time, and country.

• Core Proficiencies:

  • SQL
  • AWS EMR, Glue, S3, Athena, Spark, Hadoop
  • AWS CLI, IAM
  • Google BigQuery, Cloud Storage
  • PySpark
  • Visualization tools (Matlab, Matplotlib, Seaborn)
  • Statistical significance analysis
  • Time series forecasting, A/B testing
  • Explainable models, Deep models for time series analysis

Press Release

Press Release 2

News Feature

Interview on how technology can help improve sleep

180bpm

Anonymizing faces

Federated Learning

Ranking, re-ranking algorithm

Kotlin

AWS

Overview

AWS

• AWS ML Pipeline
• AWS Security
• AWS Technical Essentials
• Samsung Health Data Decryption
  • EMR, S3, Glue
• Shealth Data Analysis
  • Schema, Avro, Parquet
  • Glue, Athena, EMR
  • Data Analysis and Visualization, Pivoting, SQL

Federated Learning

• Federated_nntrainer_pyrec_ondevice
  • Federated on-device training
  • NNTrainer, NNStreamer
  • PYREC package: Developing state-of-the-art (SOTA) recommendation algorithms

Machine Learning & Reinforcement Learning

• Edge NNTrainer
  • Experiments run on Google Cloud images
• Reinforcement Learning
  • Pacman CS188 (Berkeley)
  • Coursera Alberta Course

Software Development

• Android Development
  • Android SDK and Studio
  • Kotlin, Jetpack Compose
  • Firebase Cloud Messaging and Notification System
  • Medicine Research App
  • Development for Wearables (e.g., Samsung Watch)
  • “Now in Android” resources
  • OCMS, Privilege SDK, Shealth SDK
  • RoomDB for database management
  • Background App Communication
• Google Android Courses
  • Multiple courses and tutorials completed
• FLEX Framework
  • Admin and Client Development

Recommender Systems

• PYREC Package
  • State-of-the-art recommendation algorithms for system development

FreeBSD

• Kernel Management
• FIRE Time Graphs

Samsung Research

• Shealth Analysis
  • SBPA Paper and Big Data Analysis
  • Nature Papers
  • Sleep Expertise and Custom Metrics
  • User and Distribution Modeling in Time
  • Analysis and Prediction of User Behavior and Churn
  • Sleep Stages, Obesity, Weight, Exercise, and Steps Analysis
  • COVID Pre/Post Analysis
  • A/B Testing
• Applications
  • Research and Development on Health Apps

Unbinding Bodes

Unbinding Bodies swiftly captures the fissures in the process where the human body, technology, consciousness, life, perception, and existence are mutually excessive and deficient. This fissure is understood as a transition from a closed to an open world, surpassing the self-contained characteristics of Western metaphysics.

Visitors to the exhibition will experience their own senses and perceptions oscillating between excess and lack as they themselves bind and separate from technology through the work (2024). They can also track the relational encounters of various heterogeneous subjects through the performance work <≈> (2024).

Main Artist: Hwia Kim

Website: Unbinding Bodies

Hosted by: Ministry of Culture, Sports, and Tourism

Organized by: Korea Arts Management Service, Art Korea Lab

Technology

I designed the end to end Computer Vision software for the exhibition. The software was used to track the faces of visitors and move the robot arms accordingly. Another algorithm was used to display what an AI sees when it looks at the humans.

Exhibit

Process

Harvard University, School of Engineering and Applied Sciences

Massachusetts, USA (Remote)
Research Intern, Visual Computing Group
March 2020 - August 2020

• Deep Learning Optimization: Enhanced deep learning in computer graphics by optimizing the PyRedner library to improve inverse rendering techniques, advancing cutting-edge integration research in graphics simulation.
• Monte Carlo Simulation Tool Development: Developed a Monte Carlo differentiable graphics simulation tool with GPU acceleration, enabling more efficient simulations for real-time applications in visual computing.

Image 1,2, 3

Key Skills & Tools:

• Deep Learning

  Inverse Rendering

  Monte Carlo Simulation

• PyRedner

  Computer Graphics

  Robotics

  ROS

  Baxter Robot

Samsung Research, Data Analytics Lab

Seoul, South Korea
Research Intern
May 2019 - July 2019

• Semantic Information Extraction: Extracted semantic information from Business-to-Consumer (B2C) email data, enabling actionable insights across various business verticals.
• Gmail Juicer System Analysis: Analyzed and replicated Gmail’s state-of-the-art Juicer system, focusing on its data constraints and generalizability.
• Multilingual Email Processing: Designed an architecture to process emails in multiple languages and formats, handling both structured data and informal text.
• Natural Language Processing (NLP) Tools: Utilized advanced NLP tools like SpaCy, KlonPY, NLTK, OpenIE, and StanfordIE to build an adaptive model for multiple data segments.
• Custom Post-Processing Heuristics: Developed custom post-processing heuristics to enhance field extraction and semantic analysis, tailored to specific business verticals.

Image 1,2, 3

Key Skills & Tools:

• Natural Language Processing (NLP)

  SpaCy

  KlonPY

  NLTK

  OpenIE

  StanfordIE

• Data Analytics

  Semantic Information Extraction

  Post-Processing Heuristics

Kyutech Institute of Technology, Shibata Labs

Fukuoka, Japan
Research Intern
Brain and Human Intelligence Systems
May 2018 - July 2018

• Cloth Segmentation Enhancement: Enhanced the Mask R-CNN model for cloth segmentation by augmenting it with a custom dataset, improving the accuracy of object recognition.
• Robotic Arm Manipulation: Designed an algorithm for determining optimal picking points from Kinect point cloud data (PCD) for Baxter robotic arm manipulation, optimizing robotic efficiency.
• Modular Software Interface: Developed a modular software interface using the QT library to integrate Kinect data streams, robotic motion control, ROI detection, and segmentation results into a seamless workflow.

• Heuristic-based Dataset Generation: Generated an extended labeled dataset using heuristic-based algorithms, improving the training process for deep learning models.

• Elderly Care Solution: Created an innovative elderly care solution integrating deep learning and robotic systems using ROS to enable the Baxter robot to assist in tasks like clothing identification, segregation, and dressing. This technology greatly enhanced the quality of life for Japan’s aging population.
• Custom Hardware Solution for Robotic Systems: Engineered a custom hardware solution, including a specialized gripper and deep learning models for object detection and segmentation, resulting in the project’s feature at RoboMech 2018 and recognition on Japanese National Television.

Image 1, 2, 3

Key Skills & Tools:

• Object Detection

  Mask R-CNN

  Kinect

  Robotic Manipulation

  QT Library

Awards and Scholastic Achievements

Research and Development Projects

July 2018 – April 2019

** Recovery-Aware File Allocation APEX, Prof. Rajkumar Buyya, University of Melbourne, Australia.**
Supervisor: Prof. Rajkumar Buyya, University of Melbourne, Australia

• Designed and implemented a FUSE-based file system with custom File I/O operations for optimized data allocation and recovery.
• Developed a Q-learning-based reinforcement learning module to simulate file system processes on disks.
• Focused on integrity, space efficiency, and computational constraints for deployment on edge computing devices.

January – April 2019

Operating Systems Design
Supervisor: Prof. S. R. Sarangi, IIT Delhi, India

• Implemented inter-process communication, multicast, and distributed algorithms on the XV6 virtual OS.
• Deployed virtualized containers with features such as syscall independence, resource isolation, and memory paging.
• Developed a parallel implementation of Jacobi’s function for Maekawa’s mutual exclusion algorithm, integrating kernel modifications for unicast and multicast system calls.

January – April 2019

Kernel Hacking FreeBSD
Supervisor: Prof. S. R. Sarangi, IIT Delhi, India

• Optimized page replacement using Minimal Counting Bloom filters, reducing memory overhead and page laundry times.
• Conducted a comparative analysis of Linux’s Completely Fair Scheduler (CFS) and FreeBSD’s ULE scheduler using benchmarks such as Phoronix and Sysbench.

July – November 2019

AlphaGo Zero – Deep Reinforcement Learning
Supervisor: Prof. Parag Singla, IIT Delhi, India

• Built AlphaGo Zero from scratch using deep reinforcement learning and Monte Carlo Tree Search, enabling competitive performance in tournaments.
• Developed a trading card game AI bot, accounting for probabilistic shuffling, card strengths, and strategic gameplay across varying difficulty levels.

November 2017 – May 2018

NIR Skin Detection
Supervisor: Prof. Prathosh AP, IIT Delhi, India

• Developed a model for skin detection and segmentation using near-infrared images, training on manually labeled datasets.
• Integrated this model into IoT systems for applications in heart rate and biological signal detection.

December 2019 – Present

Disjunctive Rado Numbers
Supervisor: Prof. Amitabha Tripathi, IIT Delhi, India

• Investigated bounds for Disjunctive Rado numbers across varying systems of equations.
• Designed algorithms to traverse exponential search spaces, leveraging mathematical constraints and memory optimization techniques.

Software Development Projects

January – April 2018

Software Development, Development Club

• Developed a universal anonymized peer review system with Django and modified RSA encryption.
• Built an internal file-sharing platform with peer-to-peer WebRTC backend for secure file exchanges.

January – April 2019

Machine Learning Algorithms
Supervisor: Prof. Parag Singla, IIT Delhi, India

• Implemented machine learning algorithms, including weighted regression, Gaussian discriminant analysis, SVMs, and decision trees.
• Applied deep reinforcement learning techniques to the ATARI game Breakout.

August – November 2018

Artificial Intelligence Principles
Supervisor: Prof. Mausam, IIT Delhi, India

• Developed advanced game-playing bots for Yinsh and Blackjack using reinforcement learning and probabilistic Markov Chains.

August – November 2018

Digital Image Analysis
Supervisor: Prof. P. Kalra, IIT Delhi, India

• Implemented Floyd-Steinberg dithering and image in-painting techniques for enhanced visualizations and image transformations.

February – April 2018

Computer Architecture and System Design
Supervisor: Prof. Anshul Kumar, IIT Delhi, India

• Designed a two-player game in ARM Assembly language with positional predictions.
• Implemented and tested a pipelined processor with ARM ISA on Basys 3 FPGA.

January – April 2018

Programming Languages
Supervisor: Prof. Sanjeeva Prasad, IIT Delhi, India

• Developed Krivine and SECD machines for low-level languages, supporting both lazy and eager operational semantics.
• Created a Prolog interpreter, including Lexer, Parser, and Horn Clause Solver.

Jan 2018

Analytic Chat Bot
Microsoft CodeFunDo, IIT Delhi

• Built a chatbot for retrieving related articles, news, and summaries. Integrated OCR for extracting text from images. GitHub.

June - July 2017

Machine Learning Projects

• Recommendation System: Predicted user preferences using collaborative filtering.
• Spam Classifier: Built a spam detection model using SVM and classifiers.
• Digit Recognition: Recognized handwritten digits using deep neural networks.

June - July 2017

Robotics and Automation Projects
Prof. P.V.M. Rao, Robotics Club, IIT Delhi

• Smart Switch Board: Created a Bluetooth-enabled home automation system.
• Bomb Disposal Robot: Designed a functional robot for obstacle evasion and object retrieval.

Extra-Curricular Activities

• Leadership & Training:
  Participated as a Crew Member in the Samsung 2024 Spring Global Newcomers Program at Samsung University (April 2024).
  • Delivered engaging lectures and panel talks to facilitate learning and professional development among 32 international participants.
  • Led intensive networking and team-building programs across Samsung’s global offices, including SDS, DX, Cheil, and E&A.

Feb - Apr 2019

Literary Analysis for IFE Understanding
Prof. Rukmini Bhaya Nair, IIT-Delhi, New Delhi

• Conducted an in-depth study of Indian Fiction in English by exploring various classic literary works and novels.
• Analyzed the essence of stories by award-winning literary authors, appreciating the subtle distinctions between fiction and reality-based narratives.

Jan 2017 - Feb 2019

Technical Development Executive
Development Club, IIT-Delhi, New Delhi

• Served as a Software and Web Developer in the technical team.
• Designed and developed projects including the File Send Service, a review system, and miscellaneous institute websites.
• Contributed to the Development Club Website.

Aug 2017 - Present

Microsoft Student Partner (MSP)
IIT-Delhi, New Delhi, India

• Organized lectures, hackathons, and hands-on workshops on various technical topics.
• Represented Microsoft at IIT-Delhi to promote technical education and engagement.

April - Sept 2017

Activity Head
Rendezvous 2017, Technical Team, IIT-Delhi

• Contributed as a Web Developer for the technical team of Rendezvous, IIT-Delhi’s cultural festival.
• Developed back-end and front-end features for the multi-platform Rendezvous website using technologies such as Node.js, AngularJS, DynamoDB, Bootstrap, and Django.
• Supported the Rendezvous Website.

May - July 2017

Summer Intern
Anhad Music, Hauz-Khas, New Delhi

• Developed a platform to connect artists and audiences, enabling them to showcase their skills and get hired by event managers.
• Built the platform’s back-end systems using Python technologies and the Django framework.