About

Retrieval-Augmented Generation (RAG) Chatbot for Real Estate Client

• Built a production RAG pipeline for a real estate and property client: starting from an Azure OpenAI chat-app quickstart as an initial proof-of-concept, developed a custom document chunking and indexing pipeline in Azure AI Search and connected it to GPT-3.5 to power a company-knowledge chat service.
• Evaluated the system against a ground-truth dataset curated by the customer’s team from their knowledge base, measuring retrieval accuracy against known-correct source documents and using an LLM-as-judge to score generated answer quality against human-written response summaries.
• Implemented CI/CD and infrastructure deployment best practices.

Migration of Forecast Model to Snowflake for Food Retail Client

• Migrated the forecasting model’s source code and model artifacts from AWS SageMaker to Snowflake-native technologies, including Snowflake’s model registry, using Python and Snowpark.
• Reduced ML pipeline execution time from 2 hours to 30 minutes by pushing feature engineering into Snowpark DataFrame operations, right-sizing the Snowflake warehouse, and materializing intermediate tables to avoid recomputation.
• Engineered lag and lead features and seasonality signals, and integrated marketing and events data into the model, improving forecast accuracy from 82% to 87%.
• Introduced code reviews via GitHub, CI/CD with GitHub Actions, and experiment tracking for the Data Science team, improving development workflow and productivity.
• Served as Tech Lead for the project, responsible for setting weekly milestones, steering the technical direction, and leading customer interactions for requirement gathering and alignment.

SQLServer to Snowflake Migration for Industrial and IoT Technology Client

• Proficiently migrated 13TB of customer data to Snowflake, utilizing Airflow for streamlined job orchestration and ensuring a seamless data transition
• Employed BCP for data export and tuned Snowflake stage file format, compression, batch size, and parallel loading to speed up data ingestion and storage.
• Integrated AWS S3 to facilitate secure and scalable data transfers, ensuring data integrity and accessibility during the migration process
• Implemented a dynamic data pipeline for ongoing synchronization, enabling continuous updates of SQL Server changes into Snowflake through an incremental copy approach. This real-time data availability enhances analytical capabilities and reporting accuracy

Data Engineering

• Led a team of two engineers, focusing on their career development through regular 1:1s, meaningful work allocation, and training. Strengthened leadership skills by managing cross-cultural teams and driving key internal projects.
• Contributed to developing and maintaining the data analysis platform, which collected and analyzed data from various internal departments. Designed the system architecture of core components such as Trino, HDFS, authentication workflow, and Hadoop ecosystem.
• Collaborated with the data analysis team to improve the performance of BI dashboards by tuning queries and optimizing underlying physical schema.
• Developed data pipeline to ingest network traffic flow data from Apache Kafka to HDFS using PySpark. This pipeline processed over a million records per second and was used for real-time anomaly detection to mitigate DDoS attacks. Published at Japan Society of Artificial Intelligence 2020.
• Integrated Trino with internal data platform components such as BI tools, authentication systems, and data warehouses. Developed custom connectors for Trino using Java and contributed to the open-source community.
• Developed YouTube playback statistics collector using Java, Docker, and JavaScript. Analyzed QoE over QUIC vs HTTP using data from 1000+ nodes. Visualization with Python/Plotly. Presented at ACM CoNEXT 2018, Greece.

Software Engineering

• Developed data catalog metadata platform using ReactJS to increase the productivity of the team by providing features such as advanced search for Japanese text, table search, schema information, and metadata from over 10 different internal systems and tools.
• Developed JP-EN and EN-JP translation feature and Furigana (reading in Hiragana for Japanese Kanji) in the open source markdown documentation tool called CodiMD using DeepL API, JavaScript, and NodeJS. This feature enhanced the communication and collaboration between non-Japanese speaking engineers and Japanese engineers.
• Presented NTT Communication’s Trino usage and architecture design at Trino Japan Virtual Meetup 2021.
• Trained new team members and helped them to understand the overall architecture of the data platform. Created useful tutorials and guides which improved team productivity.