Sosuke Kobayashi

/ 小林颯介
I’m a researcher at Preferred Networks, Inc. in Tokyo and a Specially Appointed Associate Professor (Visiting) at the Center for Language AI Research, Tohoku University (history). I hold a PhD in Information Science, and have worked in areas like machine learning, natural language processing, and 2D/3D/4D computer vision. I'm broadly interested in exploring surprising and useful findings and applications across various AI fields.

high-quality 3D/4D reconstruction (see YouTube or page)
neural scene representations with semantic or language features (NeurIPS 2022)
tractable training methods of large language models (Higuchi et al. COLM 2025) (Yano et al. COLM 2025) (Takase et al. COLM 2025) (Takase et al. ACL Findings 2023)
analyzing influence of training by deterministic dropout (SustaiNLP 2020, JNLP 2021)
subnetworks for building diverse models at once (BigScience 2022, SustaiNLP 2020, JNLP 2021)
random parameters in neural networks (SustaiNLP 2020, JNLP 2021, Takase and me. NeurIPS 2020)
controlling robots/agents with language or other interactions (ICRA 2018 [Best Paper in HRI], Fu et al. ViGIL 2019, Arakawa et al. RT-DUNE 2019, GTC 2019)
language-model-based conditional data augmentation (NAACL 2018)
entity-centric representations in NLP (NAACL 2016, IJCNLP 2017)

Other key collaborations with my colleagues and collaborators are as follows:
machine translation for language learning (Arakawa et al. CHI 2022), shift invariance in Transformers (Kiyono et al. EMNLP 2021), instance-based learning (Ouchi et al. ACL 2020, Ouchi et al. TACL 2021), video generation (Saito et al. IJCV 2020), test-time data augmentation (Shimada et al. LLD 2019), co-occurrence estimation (Yokoi et al. EMNLP 2018), controlling text styles (Akama et al. IJCNLP 2017, Akama et al. ACL 2018).

Refereed Publications and Preprints

(or see Google Scholaroogle Scholar)

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars

Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa, Kazusato Oko, Shoichiro Yamaguchi, Sosuke Kobayashi, Seiya Tokui, Kohei Hayashi, Daisuke Okanohara, Taiji Suzuki

In Proceedings of Conference on Language Modeling (COLM) 2025, Oct. 2025. [arxiv]

Efficient Construction of Model Family through Progressive Training Using Model Expansion

Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki

In Proceedings of Conference on Language Modeling (COLM) 2025, Oct. 2025. [arxiv]

User-Guided Correction of Reconstruction Errors in Structure-from-Motion

Sotaro Kanazawa, Jinyao Zhou, Yuta Kikuchi, Sosuke Kobayashi, Chunyu Li, Fabrice Matulic, Takeo Igarashi, Keita Higuchi

ACM Conference on Intelligent User Interfaces (IUI), Mar. 2025. [paper]

Spike No More: Stabilizing the Pre-training of Large Language Models

Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki

In Proceedings of Conference on Language Modeling (COLM) 2025, Oct. 2025. (Preprint 2023.) [arxiv]

B2T Connection: Serving Stability and Performance in Deep Transformers

Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki

In Proceedings of Findings of ACL 2023, July 2023. [paper] [arxiv] [code]

Decomposing NeRF for Editing via Feature Field Distillation

Sosuke Kobayashi, Eiichi Matsumoto, Vincent Sitzmann

In Proceedings of NeurIPS 2022, Nov. 2022. [paper] [arxiv] [code] [project]

Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, Kentaro Inui

In Proceedings of Workshop on Challenges & Perspectives in Creating Large Language Models (BigScience), May 2022. [paper] [arxiv] [bib]

VocabEncounter: NMT-powered Vocabulary Learning by Presenting Computer-Generated Usages of Foreign Words into Users' Daily Lives

Riku Arakawa, Hiromu Yakura, Sosuke Kobayashi

In Proceedings of CHI 2022, Apr. 2022. [paper] [bib]

Instance-Based Neural Dependency Parsing

Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Masashi Yoshikawa, Kentaro Inui

TACL (Transactions of the Association for Computational Linguistics), Vol. 9, Dec. 2021. [paper] [arxiv] [bib]

SHAPE: Shifted Absolute Position Embedding for Transformers

Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, Kentaro Inui

In Proceedings of EMNLP 2021, Nov. 2021. [paper] [arxiv] [bib]

*Efficient Estimation of Influence of a Training Instance

Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, Kentaro Inui

Journal of Natural Language Processing (*written in Japanese, extended version of the paper at SustaiNLP 2020), Vol. 28-2, June 2021. (論文賞; Best Paper Award) [paper] [arxiv] [code] [bib]

Efficient Estimation of Influence of a Training Instance

Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, Kentaro Inui

In Proceedings of First Workshop on Simple and Efficient Natural Language Processing (SustaiNLP 2020), Nov. 2020. [paper] [arxiv] [code] [bib]

All Word Embeddings from One Embedding

Sho Takase, Sosuke Kobayashi

In Proceedings of NeurIPS 2020, Dec. 2020. [paper] [arxiv] [code] [bib]

Instance-based Learning of Span Representations: A Case Study through Named Entity Recognition

Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, Kentaro Inui

In Proceedings of ACL 2020, July 2020. [paper] [arxiv] [bib]

Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN

Masaki Saito, Shunta Saito, Masanori Koyama, Sosuke Kobayashi

International Journal of Computer Vision, May 2020. [paper] [arxiv] [code]

Learning from Observation-Only Demonstration for Task-Oriented Language Grounding via Self-Examination

Tsu-Jui Fu, Yuta Tsuboi, Sosuke Kobayashi, Yuta Kikuchi

In Proceedings of 3rd Workshop on Visually Grounded Interaction and Language (ViGIL), Dec. 2019. [paper]

Data Interpolating Prediction: Alternative Interpretation of Mixup

Takuya Shimada, Shoichiro Yamaguchi, Kohei Hayashi, Sosuke Kobayashi

In Proceedings of 2nd Workshop on Learning with Limited Labeled Data: Weak Supervision and Beyond (LLD), May 2019. [paper] [arxiv]

DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, Shin-ichi Maeda

In Proceedings of 2nd Workshop on Human-Robot Teaming Beyond Human Operational Speeds and Robot Teammates Operating in Dynamic, Unstructured Environments (RT-DUNE), May 2019. [arxiv]

Pointwise HSIC: A Linear-Time Kernelized Co-occurrence Norm for Sparse Linguistic Expressions

Sho Yokoi, Sosuke Kobayashi, Kenji Fukumizu, Jun Suzuki, Kentaro Inui

In Proceedings of EMNLP 2018, Oct. 2018. [paper] [arxiv] [slide] [bib]

Unsupervised Learning of Style-sensitive Word Vectors

Reina Akama, Kento Watanabe, Sho Yokoi, Sosuke Kobayashi, Kentaro Inui

In Proceedings of ACL 2018, July 2018. [paper] [arxiv] [code] [bib] [project]

Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations

Sosuke Kobayashi

In Proceedings of NAACL HLT 2018, June 2018. [paper] [arxiv] [code] [bib]

Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

Jun Hatori*, Yuta Kikuchi*, Sosuke Kobayashi*, Kuniyuki Takahashi*, Yuta Tsuboi*, Yuya Unno*, Wilson Ko, Jethro Tan (* equally contributed)

In Proceedings of International Conference on Robotics and Automation (ICRA) 2018, June 2018. (Best Paper Award on Human-Robot Interaction (HRI)) [paper] [arxiv] [dataset] [bib] [project] [video]

A subsequent work of this is on an autonomous robot system demo (WSJ, BBC, NVIDIA GTC 2019)

A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse

Sosuke Kobayashi, Naoaki Okazaki, Kentaro Inui

In Proceedings of IJCNLP 2017, Nov. 2017. [paper] [arxiv] [slide] [bib] [video]

Generating Stylistically Consistent Dialog Responses with Transfer Learning

Reina Akama, Kazuaki Inada, Naoya Inoue, Sosuke Kobayashi, Kentaro Inui

In Proceedings of IJCNLP 2017, Nov. 2017. [paper] [bib]

An RNN-based Binary Classifier for the Story Cloze Test

Melissa Roemmele, Sosuke Kobayashi, Naoya Inoue, Andrew Gordon

In Proceedings of 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem), Apr. 2017. [paper] [bib]

Explaining Potential Risks in Traffic Scenes by Combining Logical Inference and Physics Simulation

Ryo Takahashi, Naoya Inoue, Yasutaka Kuriya, Sosuke Kobayashi, Kentaro Inui

International Journal of Machine Learning and Computing (IJMLC), Oct. 2016. [paper]

Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection

Yuki Igarashi, Hiroya Komatsu, Sosuke Kobayashi, Naoaki Okazaki, Kentaro Inui

In Proceedings of 10th International Workshop on Semantic Evaluation (SemEval 2016), June 2016. [paper] [bib]

Dynamic Entity Representation with Max-pooling Improves Machine Reading

Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, Kentaro Inui

In Proceedings of NAACL HLT 2016, June 2016. [paper] [code] [bib]

Recognizing Potential Traffic Risks through Logic-based Deep Scene Understanding

Naoya Inoue, Yasutaka Kuriya, Sosuke Kobayashi, Kentaro Inui

In Proceedings of 22nd ITS World Congress, Oct. 2015. (Best of the Rest) [paper]

Other Publications (mainly in Japanese)

新規視点画像生成による車載3D物体検出器のカメラ高データ増強

髙橋友紀子, 徐天涵, 小林颯介, 加藤大晴, 髙濱敦, 加藤雄大, 横山美優, 光森卓, 高椋佐和

In Proceedings of 画像の認識・理解シンポジウム MIRU2025, July 2025.

モデル拡張を用いた段階的事前学習によるモデル系列の効率的な構築

矢野一樹, 高瀬翔, 小林颯介, 清野舜, 鈴木潤

In Proceedings of 言語処理学会第31回年次大会, Mar. 2025. [paper]

ユーザ教示による Structure-from-Motion 再構成エラーの修正

金澤爽太郎, 周矜瑶, 菊池悠太, 小林颯介, 李淳雨, マートリッチファブリス, 五十嵐健夫, 樋口啓太

In Proceedings of INTERACTION2025, Mar. 2025. [abstract]

四次元再構成視聴におけるインタラクティブなカメラサーフェスの提案

元村愛美, 菊池悠太, 小林颯介, Fabrice Matulic, 五十嵐健夫, 樋口啓太

In Proceedings of INTERACTION2025, Mar. 2025. [abstract]

大規模言語モデル事前学習の安定化

高瀬翔, 清野舜, 小林颯介, 鈴木潤

In Proceedings of 言語処理学会第30回年次大会, Mar. 2024. (委員特別賞; Special Committee Award) [paper]

Transformer を多層にする際の勾配消失問題と解決法について

高瀬翔, 清野舜, 小林颯介, 鈴木潤

In Proceedings of 言語処理学会第28回年次大会, Mar. 2022. (優秀賞; Outstanding Paper Award) [paper]

シフト付き絶対位置埋め込み

清野舜, 小林颯介, 鈴木潤, 乾健太郎

In Proceedings of 言語処理学会第28回年次大会, Mar. 2022. (若手奨励賞; Young Researcher Award) [paper]

事例ベース依存構造解析のための依存関係表現学習

大内啓樹, 鈴木潤, 小林颯介, 横井祥, 栗林樹生, 吉川将司, 乾健太郎

In Proceedings of 言語処理学会第27回年次大会, Mar. 2021. [paper]

単一事例エキスパートの統合によるドメイン適応

清野舜, 小林颯介, 鈴木潤, 乾健太郎

In Proceedings of 言語処理学会第27回年次大会, Mar. 2021. [paper]

スパン間の類似性に基づく事例ベース構造予測

大内啓樹, 鈴木潤, 小林颯介, 横井祥, 栗林樹生, 乾健太郎

In Proceedings of 言語処理学会第26回年次大会, Mar. 2020. [paper]

実世界での話し言葉指示による物体移動：深層学習による画像・言語理解

高橋城志*, 羽鳥潤*, 菊池悠太*, 小林颯介*, 坪井祐太*, 海野裕也*, 中島統太郎, 福田昌昭, Wilson Ko, Jethro Tan (* equally contributed)

In Proceedings of 第36回日本ロボット学会学術講演会, Sep. 2018. (研究奨励賞; Young Investigation Excellence Award) [paper]

カーネル法に基づく疎な言語表現のための高速計算可能な共起尺度

横井祥, 小林颯介, 福水健次, 乾健太郎

In Proceedings of 第32回人工知能学会全国大会, June 2018. (全国大会優秀賞; JSAI Annual Conference Award) [paper]

スタイルの類似性を捉えた単語ベクトルの教師なし学習

赤間怜奈, 渡邉研斗, 横井祥, 小林颯介, 乾健太郎

In Proceedings of 第32回人工知能学会全国大会, June 2018. (全国大会優秀賞; JSAI Annual Conference Award) [paper]

実世界におけるインタラクティブな物体指示

羽鳥潤*, 菊池悠太*, 小林颯介*, 高橋城志*, 坪井祐太*, 海野裕也*, Wilson Ko, Jethro Tan (* equally contributed)

In Proceedings of 言語処理学会第24回年次大会, Mar. 2018. [paper]

カーネル法に基づく疎な言語表現のための共起尺度

横井祥, 小林颯介, 福水健次, 乾健太郎

In Proceedings of 言語処理学会第24回年次大会, Mar. 2018. [paper]

サンプリング戦略に基づく単語ベクトルの意味成分とスタイル成分の分離

赤間怜奈, 横井祥, 渡邉研斗, 小林颯介, 田然, 乾健太郎

In Proceedings of 言語処理学会第24回年次大会, Mar. 2018. [paper]

カーネル法に基づく共起尺度

横井祥, 福水健次, 小林颯介, 乾健太郎

In Proceedings of 第20回情報論的学習理論ワークショップ (IBIS), Nov. 2017. [slide]

対話返答生成における個性の追加反映

濱田晃一, 藤川和樹, 小林颯介, 菊池悠太, 海野裕也, 土田正明

In Proceedings of 第232回情報処理学会自然言語処理研究会, July 2017. [paper] [slide] [bib]

転移学習を用いた対話応答のスタイル制御

赤間怜奈, 稲田和明, 小林颯介, 佐藤祥多, 乾健太郎

In Proceedings of 言語処理学会第23回年次大会, Mar. 2017. [paper]

Recognizing Potential Traffic Risks through Logic-based Deep Scene Understanding

Shunya Maruta, Yasutaka Kuriya, Naoya Inoue, Sosuke Kobayashi, Kentaro Inui

In Proceedings of Denso Technical Review, 2016. (Modified version of the paper of 22nd ITSWC) [paper]

共参照関係に基づく分散表現の共有と動的更新

小林颯介, 岡崎直観, 乾健太郎

In Proceedings of NLP若手の会 (YANS) 第11回シンポジウム, Aug. 2016. (奨励賞; Encouragement Award)

談話内における局所文脈の動的分散表現

小林颯介, 田然, 岡崎直観, 乾健太郎

In Proceedings of 言語処理学会第22回年次大会, Mar. 2016. (優秀賞; Outstanding Paper Award) [paper]

再帰型ニューラルネットワークを用いた対話破綻検出と言語モデルのマルチタスク学習

小林颯介, 海野裕也, 福田昌昭

In Proceedings of 第75回人工知能学会言語・音声理解と対話処理研究会, Oct. 2015. [paper]

物理モデルと論理推論の統合による運転シーンの潜在的危険の予測

小林颯介, 井之上直也, 栗谷康隆, 近藤敏之, 安部克則, 奥野英一, 乾健太郎

In Proceedings of 自動車技術会 2015年春季大会学術講演会講演予稿集, May 2015.

Other Activities

Education and Experience

2024.10 - present: Specially Appointed Associate Professor (Visiting) at the Center for Language AI Research, Tohoku University, Sendai.
2021.11 - 2024.10: Researcher (part-time) at FaiLab in Tohoku University, Sendai.
2018.10 - 2021.9: Doctor of Information Science at Inui-Suzuki Lab in Graduate School of Information Sciences, Tohoku University, Sendai.
2016.10 - present: Researcher at Preferred Networks, Inc., Tokyo. Research and development on machine learning, 2D/3D computer vision, NLP, robotics, etc.
2014.10 - 2016.9: Master of Information Science at Inui-Okazaki Lab in Graduate School of Information Sciences, Tohoku University, Sendai.
2011.4 - 2014.9: Department of Information and Intelligent Systems, Tohoku University, Sendai. (Early graduation)
2008.4 - 2011.3: Suzaka High School, Nagano.

Internship and Part-time Work

Part-time Work at Preferred Infrastructure, Inc., Neural Dialogue Response Generation and Multimodal NLP, Oct. 2015, Feb. and Mar. 2016.
Internship at Preferred Infrastructure, Inc., Neural Dialogue Response Generation, Aug. and Sep. 2015.
Part-time Work at Recruit Jobs Co., Ltd., Data Analysis and Development for Web Services, 2015.
Internship at Recruit Holdings Co., Ltd., Data Analysis and Development for Web Services, Feb. 2015. (最優秀賞; Best Team Award)
Collaborative Research with DENSO Corporation, Recognizing Potential Risks in Traffic Scenes, 2014--2015

Competition

20th/946, solo silver medal, at chaii - Hindi and Tamil Question Answering in Kaggle, Nov. 2021. (First time participating)

Review

Paper review of NLP Conferences/Journals: COLM (2024), ACL Rolling Review (2021-2022, 2024-2025), ACL (2018-2021), EMNLP (2018-2020), NAACL (2019), COLING [Area Chair] (2022), 論文誌自然言語処理 (2017, 2020-2022), [Editor] (2023-2024), 情報処理学会NL研 [Award Committee] (2023-2024)
Paper review of ML Conferences: ICLR (2024), NeurIPS (2022-2023), ICML (2022, 2024)
Paper review of Robotics Conference: ICRA (2019)
Review and revision of Deep Learning Tutorial with Chainer, 2019.
Review of Kaggleに挑む深層学習プログラミングの極意 (Essentials of Deep Learning Programming for Kaggle Challenge), 小嵜耕平, 秋葉拓哉, 林孝紀, 石原祥太郎, 講談社, 2023. [amazon]
Review of 深層学習による自然言語処理 (機械学習プロフェッショナルシリーズ) (Natural Language Processing by Deep Learning, A Series of Machine Learning Professionals), 坪井祐太, 海野裕也, 鈴木潤, 講談社, 2017. [amazon]
Review of ウェブデータの機械学習 (機械学習プロフェッショナルシリーズ) (Web Data and Machine Learning, A Series of Machine Learning Professionals), ダヌシカボレガラ, 岡﨑直観, 前原貴憲, 講談社, 2016. [amazon]
Review of 深層学習 Deep Learning, 神嶌敏弘 (編), 麻生英樹, 安田宗樹, 前田新一, 岡野原大輔, 岡谷貴之, 久保陽太郎, ボレガラダヌシカ, 人工知能学会監修, 近代科学社, 2015. [amazon]

Research/Teaching Assistant

Research Assistant of JST, CREST, 構造理解に基づく大規模文献情報からの知識発見, Dec. 2015--Sep. 2016.
Teaching Assistant (Research Mentor) of Team-based Engineering Design Course (for underguraduates) of Tohoku University, Vector Representations of Emoji (Emoticons) on Microblogs, Autumn--Winter 2015--2016.
Teaching Assistant of Information Communication Theory and Natural Language Processing (for guraduates) of Tohoku University, Autumn--Winter 2015--2016.
Teaching Assistant of Programming Practice A (for undergraduates) of Tohoku University, Spring--Summer 2015.
Teaching Assistant (Research Mentor) of Team-based Engineering Design Course (for underguraduates) of Tohoku University, Knowledge Extraction from Web for Question Answering, Autumn--Winter 2014--2015.

Talk

Talk about Machine Learning for Language and the World at NLP若手の会 (YANS) 第18回シンポジウム (18th Symposium of Young Researcher Association for NLP Studies), Aug. 2023.
Talk about Efficient Estimation of Influence of a Training Instance (best paper invited talk of JNLP) at the 28th Annual Meeting of the Association for Natural Language Processing (remote), Mar. 2022.
Talk about an efficient estimation of machine learning models trained without an instance at the 22nd Doctoral Student Presentations of GSIS, Tohoku University, Dec. 2019. (Best Presentation Award)
Talk about ビジョン＆ランゲージによる意図理解と曖昧性解消 (Vision and Language Help Intent Understanding and its Disambiguation) at 第25回画像センシングシンポジウム (SSII2019) オーガナイズドセッション『ビジョン & ランゲージ「意図」をどのようにモデリングするか？』, Jun. 2019
Talk about Learning Communication with Neural Networks at PFN Seminar, Mar. 2017.
Talk about NAACL-HLT 2016 at NLP若手の会 (YANS) 第11回シンポジウム (11th Symposium of Young Researcher Association for NLP Studies), Aug. 2016.
Talk about Recent Progress in RNN and NLP at NLP-DL Study Group, Jun. 2016.

Organizer

Editorial Comittee of 論文誌自然言語処理 (Journal of Natural Language Processing), 2023-2025.
Comittee of 情報処理学会自然言語処理(NL)研究会 (Special Interest Group on Natural Language Processing in IPSJ), 2023-2025.
Organizer of 言語処理技術セミナー2021 (NLP Technology Seminar 2021), Oct. 2021.
Organizer of 第11回最先端NLP勉強会 (11th NLP Paper Reading Circle), Sep. 2019.
Organizer of 第10回最先端NLP勉強会 (10th NLP Paper Reading Circle), Aug. 2018.
Organizer of 第9回最先端NLP勉強会 (9th NLP Paper Reading Circle), Sep. 2017.
Organizer of 第8回最先端NLP勉強会 (8th NLP Paper Reading Circle), Sep. 2016.

Github

Research: Reproduction
Research: Original
- Turn-over Dropout - (My work at SustaiNLP 2020)
- Contextual Augmentation - (My work at NAACL 2018)
- PFN Picking Instructions for Commodities Dataset (PFN-PIC) - (Our work at ICRA 2018)
- Emergence of Language Using Discrete Sequences with Autoencoder - (My work not published, in 2017)
- Next Event Predictor - (Related to our work at LSDSem 2017)
- Sentence Generater Using Deep Convolutional Generative Adversarial Network (DCGAN) - (My work not published, in 2016)
- Dynamic Entity Representation Network - (Our work at NAACL 2016)
- Dynamic Neural Text Model - (Our work at IJCNLP 2017)
Others
- ArxivLeaks - Read secret comment of TeX in arXiv papers
- Contributions for Chainer, a python-based Deep Learning framework. e.g., Layer normalization, Tree LSTM, seq2seq example and text classification example.