论文

138 篇 · 按建议阅读顺序排列 · 自学路线

筛选

共 138 篇

  1. 基础与评估

  2. A Survey of Direct Preference Optimization

    001
    偏好优化基础已核验摘要2025

  3. A Survey on Progress in LLM Alignment from the Perspective of Reward Design

    002
    奖励建模与评估 / 偏好优化基础已核验摘要2025

  4. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

    003
    偏好优化基础 / 数据与主动学习已核验摘要2024

  5. Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

    004
    偏好优化基础 / 奖励建模与评估已核验摘要2024

  6. Rethinking reward modeling in preference-based large language model alignment

    005
    奖励建模与评估 / 偏好优化基础已核验摘要2025

  7. How to Evaluate Reward Models for RLHF

    006
    奖励建模与评估 / 偏好优化基础已核验摘要2025

  8. Learning LLM-as-a-Judge for Preference Alignment

    007
    奖励建模与评估 / 偏好优化基础已核验摘要2025

  9. Post-hoc Alignment of LLM-judges to Human Judgment Distribution

    008
    奖励建模与评估 / 多目标与个性化已核验摘要2026

  10. Discretizing Reward Models

    009
    奖励建模与评估 / 噪声与鲁棒性已核验摘要2026

  11. Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output

    010
    奖励建模与评估 / 优化动力学已核验摘要2026

  12. 数据与偏好结构

  13. What Matters in Data for DPO?

    011
    数据与主动学习 / 偏好优化基础已核验摘要2025

  14. Less is More: Improving LLM Alignment via Preference Data Selection

    012
    数据与主动学习 / 噪声与鲁棒性已核验摘要2025

  15. Which Pairs to Compare for LLM Post-Training?

    013
    数据与主动学习 / 偏好优化基础已核验摘要2026

  16. Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?

    014
    数据与主动学习 / 优化动力学已核验摘要2025

  17. Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap

    015
    数据与主动学习已核验摘要2025

  18. On Preference Learning Based on Sequential Bayesian Optimization with Pairwise Comparison

    016
    数据与主动学习 / 多目标与个性化已核验摘要2021

  19. Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment

    017
    数据与主动学习 / 奖励建模与评估已核验摘要2025

  20. ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment

    018
    数据与主动学习 / 偏好优化基础已核验摘要2026

  21. Online Preference Alignment for Language Models via Count-based Exploration

    019
    数据与主动学习 / 偏好优化基础已核验摘要2025

  22. Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

    020
    数据与主动学习 / 噪声与鲁棒性已核验摘要2025

  23. Diverse AI Feedback For Large Language Model Alignment

    021
    数据与主动学习 / 偏好优化基础已核验摘要2025

  24. Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains

    022
    偏好优化基础 / 奖励建模与评估已核验摘要2026

  25. TODO: Enhancing LLM Alignment with Ternary Preferences

    023
    偏好优化基础 / 噪声与鲁棒性已查阅全文2024

  26. Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

    024
    多目标与个性化 / 偏好优化基础已查阅全文2025

  27. Mind the Gap: Structure-Aware Consistency in Preference Learning

    025
    偏好优化基础 / 优化动力学已核验摘要2026

  28. Preference Ranking Optimization for Human Alignment

    026
    偏好优化基础 / 数据与主动学习已核验摘要2024

  29. Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling

    027
    偏好优化基础 / 多目标与个性化已核验摘要2025

  30. Mult-DPO: Multinomial Direct Preference Optimization for Recommender Systems

    028
    偏好优化基础 / 安全与应用已查阅全文2026

  31. GroupDPO: Memory efficient Group-wise Direct Preference Optimization

    029
    偏好优化基础 / 优化动力学已核验摘要2026

  32. Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

    030
    偏好优化基础 / 数据与主动学习已核验摘要2024

  33. 优化动力学

  34. Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models

    031
    优化动力学已查阅全文2026

  35. AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

    032
    优化动力学已核验摘要2026

  36. Small-Margin Preferences Still Matter-If You Train Them Right

    033
    数据与主动学习 / 优化动力学已核验摘要2026

  37. Towards Disentangled Preference Optimization Dynamics Beyond Likelihood Displacement

    034
    优化动力学 / 偏好优化基础已核验摘要2026

  38. SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization

    035
    优化动力学 / 偏好优化基础已核验摘要2026

  39. Linear Preference Optimization: Decoupled Gradient Control via Absolute Regularization

    036
    优化动力学 / 偏好优化基础已核验摘要2025

  40. BPO: Revisiting Preference Modeling in Direct Preference Optimization

    037
    优化动力学 / 偏好优化基础已核验摘要2025

  41. A Stable and Principled Loss Function for Direct Language Model Alignment

    038
    偏好优化基础 / 优化动力学已核验摘要2025

  42. SPO: Self Preference Optimization with Self Regularization

    039
    优化动力学 / 偏好优化基础已核验摘要2025

  43. Boosting Direct Preference Optimization with Penalization

    040
    偏好优化基础 / 优化动力学已核验摘要2026

  44. Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

    041
    数据与主动学习 / 优化动力学已核验摘要2025

  45. Explicit Preference Optimization: No Need for an Implicit Reward Model

    042
    偏好优化基础 / 优化动力学已核验摘要2025

  46. WHY DPO IS A MISSPECIFIED ESTIMATOR AND HOW TO FIX IT

    043
    偏好优化基础 / 优化动力学已核验摘要2025

  47. Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

    044
    偏好优化基础 / 优化动力学已核验摘要2026

  48. Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO

    045
    偏好优化基础 / 优化动力学已核验摘要2025

  49. Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization

    046
    优化动力学已核验摘要2026

  50. Convex Optimization for Alignment and Preference Learning on a Single GPU

    047
    偏好优化基础 / 推理时对齐已核验摘要2026

  51. InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

    048
    偏好优化基础 / 多目标与个性化已核验摘要2025

  52. Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL

    049
    奖励建模与评估 / 数据与主动学习已核验摘要2025

  53. Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization

    050
    偏好优化基础 / 噪声与鲁棒性已核验摘要2024

  54. DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment

    051
    数据与主动学习 / 偏好优化基础已核验摘要2024

  55. 噪声与鲁棒性

  56. When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

    052
    奖励建模与评估 / 噪声与鲁棒性已核验摘要2026

  57. Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases

    053
    安全与应用 / 奖励建模与评估已核验摘要2026

  58. Uncertainty-Aware Reward Modeling for Stable RLHF

    054
    奖励建模与评估 / 噪声与鲁棒性已核验摘要2026

  59. PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration

    055
    奖励建模与评估 / 多目标与个性化已核验摘要2026

  60. Unbiased Alignment for Large Language Models with Noisy Preferences

    056
    噪声与鲁棒性 / 偏好优化基础已核验摘要2026

  61. When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF

    057
    噪声与鲁棒性 / 偏好优化基础已核验摘要2026

  62. PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

    058
    噪声与鲁棒性 / 偏好优化基础已核验摘要2026

  63. Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

    059
    噪声与鲁棒性 / 优化动力学已核验摘要2026

  64. RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment

    060
    噪声与鲁棒性 / 偏好优化基础已核验摘要2025

  65. wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment

    061
    噪声与鲁棒性 / 优化动力学已核验摘要2026

  66. Robust LLM Alignment via Distributionally Robust Direct Preference Optimization

    062
    噪声与鲁棒性 / 偏好优化基础已核验摘要2025

  67. Distributionally Robust Listwise Preference Optimization

    063
    噪声与鲁棒性 / 偏好优化基础已核验摘要2026

  68. Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment

    064
    噪声与鲁棒性 / 优化动力学已核验摘要2026

  69. GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning

    065
    数据与主动学习 / 噪声与鲁棒性已核验摘要2026

  70. Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models

    066
    优化动力学 / 噪声与鲁棒性已核验摘要2026

  71. Robust Preference Alignment via Directional Neighborhood Consensus

    067
    推理时对齐 / 多目标与个性化已核验摘要2025

  72. Gradient Regularization Prevents Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

    068
    优化动力学 / 奖励建模与评估已核验摘要2026

  73. Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking

    069
    奖励建模与评估 / 噪声与鲁棒性已核验摘要2025

  74. Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance

    070
    奖励建模与评估 / 噪声与鲁棒性已核验摘要2025

  75. Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

    071
    奖励建模与评估 / 噪声与鲁棒性已核验摘要2026

  76. Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback

    072
    噪声与鲁棒性 / 奖励建模与评估已核验摘要2026

  77. Primal-Dual Direct Preference Optimization for Constrained LLM Alignment

    073
    安全与应用 / 偏好优化基础已核验摘要2025

  78. Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game

    074
    偏好优化基础 / 奖励建模与评估已核验摘要2024

  79. Adversarial Preference Learning for Robust LLM Alignment

    075
    安全与应用 / 数据与主动学习已核验摘要2025

  80. Improving LLM Safety Alignment with Dual-Objective Optimization

    076
    安全与应用 / 优化动力学已核验摘要2025

  81. DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment

    077
    数据与主动学习 / 安全与应用已核验摘要2026

  82. Causal Direct Preference Optimization for Language Model Alignment

    078
    噪声与鲁棒性 / 数据与主动学习已核验摘要2026

  83. Robust Preference Optimization through Reward Model Distillation

    079
    噪声与鲁棒性 / 奖励建模与评估已核验摘要2024

  84. 多目标与推理时对齐

  85. Nash Learning from Human Feedback

    080
    偏好优化基础 / 多目标与个性化已核验摘要2024

  86. Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

    081
    偏好优化基础 / 优化动力学已核验摘要2025

  87. Efficient Exploration for Iterative Nash Preference Optimization

    082
    偏好优化基础 / 数据与主动学习已核验摘要2026

  88. Panacea: Pareto Alignment via Preference Adaptation for LLMs

    083
    多目标与个性化已核验摘要2024

  89. PolyAlign: Conditional Human-Distribution Alignment

    084
    多目标与个性化已核验摘要2026

  90. Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization

    085
    奖励建模与评估 / 多目标与个性化已核验摘要2026

  91. MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

    086
    偏好优化基础 / 奖励建模与评估已核验摘要2025

  92. Robust Multi-Objective Preference Alignment with Online DPO

    087
    多目标与个性化 / 数据与主动学习已核验摘要2025

  93. Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment

    088
    多目标与个性化 / 优化动力学已核验摘要2026

  94. Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixing

    089
    多目标与个性化 / 偏好优化基础已核验摘要2025

  95. Multi-Objective Exploration and Preference Optimization via Mutual Information

    090
    多目标与个性化 / 数据与主动学习已核验摘要2026

  96. PAPA: Online Personalized Active Preference Alignment

    091
    多目标与个性化 / 数据与主动学习 / 扩散与流匹配已核验摘要2026

  97. NextQuill: Causal Preference Modeling for Enhancing LLM Personalization

    092
    多目标与个性化 / 数据与主动学习已核验摘要2025

  98. LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization

    093
    多目标与个性化 / 优化动力学已核验摘要2026

  99. MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

    094
    推理时对齐 / 多目标与个性化已核验摘要2026

  100. Aligner: Efficient Alignment by Learning to Correct

    095
    推理时对齐 / 安全与应用已核验摘要2024

  101. Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

    096
    推理时对齐 / 偏好优化基础已核验摘要2025

  102. PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

    097
    推理时对齐 / 多目标与个性化已核验摘要2025

  103. Transfer Q Star: Principled Decoding for LLM Alignment

    098
    推理时对齐 / 偏好优化基础已核验摘要2024

  104. Alignment-Aware Decoding

    099
    推理时对齐 / 偏好优化基础已核验摘要2025

  105. Gradient-Guided Reward Optimization for Inference-time Alignment

    100
    推理时对齐 / 奖励建模与评估已核验摘要2026

  106. REAR: Test-time Preference Realignment through Reward Decomposition

    101
    推理时对齐 / 多目标与个性化已核验摘要2026

  107. DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

    102
    推理时对齐 / 多目标与个性化 / 噪声与鲁棒性已核验摘要2026

  108. Spectral Editing of Activations for Large Language Model Alignment

    103
    推理时对齐 / 优化动力学已核验摘要2024

  109. Aligning Large Language Models with Human Preferences through Representation Engineering

    104
    推理时对齐 / 多目标与个性化已核验摘要2024

  110. D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe - Beneath the Surface, DPO as Steering Vector Perturbation in Activation Space

    105
    优化动力学 / 推理时对齐已核验摘要2025

  111. On the Limits of Steering Vectors for Preference-Aligned Generation

    106
    推理时对齐 / 多目标与个性化已核验摘要2026

  112. When Metropolis and Hastings Meet Bradley and Terry: Exact MCMC From Preference Voting

    107
    推理时对齐 / 密度比估计已核验摘要2026

  113. 密度比基础

  114. Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation

    108
    密度比估计 / 偏好优化基础已核验摘要2012

  115. Generative Modeling with Flow-Guided Density Ratio Learning

    109
    密度比估计 / 扩散与流匹配已核验摘要2024

  116. Don't Forget Its Variance! The Minimum Path Variance Principle for Accurate and Stable Score-Based Density Ratio Estimation

    110
    密度比估计 / 优化动力学已核验摘要2026

  117. Preference Optimization by Estimating the Ratio of the Data Distribution

    111
    密度比估计 / 偏好优化基础已核验摘要2025

  118. Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models

    112
    密度比估计 / 扩散与流匹配已核验摘要2023

  119. 扩散、流匹配与应用

  120. Beyond Optimal Transport: Model-Aligned Coupling for Flow Matching

    113
    扩散与流匹配 / 数据与主动学习已核验摘要2025

  121. Distributional Preference Alignment of LLMs via Optimal Transport

    114
    偏好优化基础 / 多目标与个性化已查阅全文2024

  122. Direct Distributional Optimization for Provable Alignment of Diffusion Models

    115
    扩散与流匹配 / 密度比估计已核验摘要2025

  123. Diffusion Policy Optimization without Drifting Apart

    116
    扩散与流匹配 / 优化动力学已核验摘要2026

  124. Beyond Reward Margin: Rethinking and Resolving Likelihood Displacement in Diffusion Models via Video Generation

    117
    优化动力学 / 扩散与流匹配已核验摘要2025

  125. Stitched Value Model for Diffusion Alignment

    118
    扩散与流匹配 / 奖励建模与评估已核验摘要2026

  126. DiffRGD: An Inference-Time Diffusion Guidance Through Riemannian Gradient Descent

    119
    推理时对齐 / 扩散与流匹配已核验摘要2026

  127. PG-MAP: Joint MAP Optimization for Inference-Time Alignment of Diffusion and Flow-Matching Models

    120
    推理时对齐 / 扩散与流匹配已核验摘要2026

  128. Threshold-Guided Optimization for Visual Generative Models

    121
    扩散与流匹配 / 偏好优化基础已核验摘要2026

  129. STAR: SpatioTemporal Adaptive Reward Allocation for Text-to-Image RL Post-Training

    122
    扩散与流匹配 / 优化动力学已核验摘要2026

  130. NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning

    123
    扩散与流匹配 / 优化动力学已核验摘要2026

  131. Optimizing Visual Generative Models via Distribution-wise Rewards

    124
    扩散与流匹配 / 奖励建模与评估已核验摘要2026

  132. Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

    125
    扩散与流匹配 / 优化动力学已核验摘要2026

  133. ReNFT: Repairing Mode Collapse in Reward Post-Training via Internal Probability-Mass Recalibration

    126
    扩散与流匹配 / 优化动力学已核验摘要2026

  134. Reward-guided Fine-Tuning of One-Step Generative Models via Wasserstein Gradient Flow

    127
    扩散与流匹配 / 奖励建模与评估已核验摘要2026

  135. One-step Diffusion Models with Bregman Density Ratio Matching

    128
    密度比估计 / 扩散与流匹配已核验摘要2025

  136. Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion

    129
    密度比估计 / 扩散与流匹配已核验摘要2025

  137. Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

    130
    扩散与流匹配 / 多目标与个性化已核验摘要2026

  138. Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence

    131
    多目标与个性化 / 安全与应用已核验摘要2025

  139. PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

    132
    安全与应用 / 偏好优化基础已核验摘要2026

  140. LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

    133
    优化动力学 / 安全与应用已核验摘要2025

  141. Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

    134
    偏好优化基础 / 安全与应用已核验摘要2024

  142. Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

    135
    偏好优化基础 / 数据与主动学习已核验摘要2025

  143. DynamicPO: Dynamic Preference Optimization for Recommendation

    136
    数据与主动学习 / 安全与应用已核验摘要2026

  144. Align³GR: Unified Multi-Level Alignment for LLM-based Generative Recommendation

    137
    安全与应用 / 多目标与个性化已核验摘要2026

  145. MARD: Mirror-Augmented Reasoning Distillation for Mechanism-Level Drug-Drug Interaction Prediction

    138
    安全与应用 / 奖励建模与评估已核验摘要2026

148 条原始记录已合并去重。核验范围见各篇笔记。

阅读进度仅保存在当前浏览器。