基于细粒度令牌竞争注意力的文生图方法

刘一; 周静; 袁婷; 汪茜

doi:10.3969/j.issn.0258-2724.20250670

基于细粒度令牌竞争注意力的文生图方法

doi: 10.3969/j.issn.0258-2724.20250670

刘一^{1, 2,},
周静³,
袁婷^{1, 2},
汪茜^{1, 2, ,}

1.
民航局数据治理与决策优化重点实验室，北京 100102
2.
中国民航管理干部学院，北京 100102
3.
西南交通大学计算机与人工智能学院，四川成都 611752

基金项目: 国家自然科学基金青年项目（61802053）视频问答中的机器学习与推理方法研究

详细信息

作者简介:
刘一（1981—），男，教授，博士，研究方向为大数据与人工智能、智慧民航，E-mail：liuyi@camic.cn

通讯作者:
汪茜（1997—），女，讲师，博士，研究方向为大数据与人工智能、智慧民航，E-mail：wangxi@camic.cn

中图分类号: 文献标识码:A
计量
- 文章访问数: 3
- HTML全文浏览量: 2
- PDF下载量: 2
- 被引次数: 0
出版历程
- 收稿日期: 2025-12-29
- 修回日期: 2026-04-07
- 网络出版日期: 2026-10-21

Text-to-Image Generation Method Based on Fine-Grained Token Competitive Attention

LIU Yi^{1, 2
,},
ZHOU Jing³,
YUAN Ting^{1, 2},
WANG Qian^{1, 2
, ,}

1.
Key Laboratory of Civil Aviation Data Governance and Decision Optimization, CAAC, Beijing 100102, China
2.
Civil Aviation Management Institute of China, Beijing 100102, China
3.
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611752, China

摘要

摘要:
在多属性对象的文本到图像生成任务中，复杂场景下兼顾对象完整性与语义一致性仍具挑战，主要源于多属性间的竞争与干扰易导致结构缺失与语义错配. 现有扩散生成模型在处理多对象及其属性描述时，易受提示信息不足、语义纠缠及注意力分配不均的影响，从而引发对象缺失、属性混淆和关系错位等问题. 为此，本文提出一种基于细粒度令牌竞争注意力正则化的多阶段生成方法，通过提示增强、语义解耦与注意力优化实现精细控制. 首先，在提示阶段，利用大语言模型对输入文本进行语义增强与结构化表达，以缓解歧义；其次，在编码阶段，设计属性-对象级语义解耦机制，构建 Token 编码模块，引入正反语义向量并结合语义邻居策略，提升嵌入的稳定性与区分度，降低语义干扰；最后，在生成阶段引入双重竞争注意力正则化：既增强属性对象在跨模态注意力中的响应，又抑制不同对象间的注意力重叠，以提高对象独立性并减少语义干扰. 在 T2I-benchmark 上的实验表明，该方法在颜色、形状和纹理属性绑定任务中均取得显著提升，BLIP-VQA（Bootstrapping Language-Image Pre-training Visual Question Answering）得分分别为 0.7392、0.6205 和 0.5013；可视化结果进一步验证，其在复杂多对象场景下能有效减少属性混合与语义错位，同时保持对象完整性与语义一致性，展现出良好的鲁棒性与可控性.
- 文本到图像生成 /
- 提示增强 /
- 多属性对象 /
- 扩散模型 /
- 注意力机制
Abstract:
In multi-attribute text-to-image generation, maintaining object integrity and semantic consistency in complex scenes remains challenging, primarily due to competition and interference among attributes that can lead to structural incompleteness and semantic mismatches. Existing diffusion models struggle with multiple objects and their attribute descriptions due to insufficient prompt information, semantic entanglement, and uneven attention allocation, leading to missing objects, attribute confusion, and misaligned relations. To address this, a multi-stage generation method based on fine-grained token-level competitive attention regularization is proposed, enabling precise control via prompt enhancement, semantic decoupling, and attention optimization. First, in the prompt stage, an LLM enhances the semantics of the input text and structures its representation to reduce ambiguity. Next, in the encoding stage, an attribute-object-level semantic decoupling mechanism is designed, integrating a token encoding module, positive and negative semantic vectors, and a semantic neighbor strategy to improve the stability and discriminability of embeddings while reducing semantic interference. Finally, in the generation stage, dual competitive attention regularization enhances the responses of attributes and objects in cross-modal attention while suppressing attention overlap between different objects, thus improving object independence and reducing semantic interference. Experiments on the T2I-benchmark show significant improvements in color, shape, and texture attribute binding, with bootstrapping language-image pre-training visual question answering (BLIP-VQA) scores of 0.7392, 0.6205, and 0.5013, respectively. Visualizations further confirm that the method effectively reduces attribute mixing and semantic misalignment in complex multi-object scenes while preserving object integrity and semantic consistency, demonstrating good robustness and controllability.
- text-to-image generation /
- prompt enhancement /
- multi-attribute objects /
- diffusion model /
- attention mechanism

HTML全文

图 1 模型整体框架

Figure 1. Overall model framework

下载: 全尺寸图片幻灯片

图 2 Token编码处理模块架构图

Figure 2. Architecture of token encoding module

下载: 全尺寸图片幻灯片

图 3 内部损失（局部注意力控制）解释

Figure 3. Inner loss （local attention control）

下载: 全尺寸图片幻灯片

图 4 外部损失（全局注意力控制）解释图

Figure 4. Outer loss （global attention control）

下载: 全尺寸图片幻灯片

算法1：算法流程

输入：原始文本提示 $ P $、扩散模型 $ D $、文本编码器 Enc、大语言模型 LLM、Token库 $ S $
输出：生成图像 $ {I}_{{\mathrm{out}}} $.
1）语义增强： 利用 LLM 对进行扩展，得到增强提示，并提取属性–对象对 $ {A}_{i}\& {E}_{i} $；
2）文本编码：采用 Enc 对增强提示进行编码，获得文本嵌入表示c；
3）Token语义解耦：对每个对象 Token $ {E}_{i} $，构建正向吸附向量v_null与负向排斥向量$ {v}_{\text{push}} $，更新嵌入 $ {E}_{i} $.
4）扩散生成：初始化噪声分布$ {Z}_{{\mathrm{T}}}\sim N(0,1) $，在反向去噪过程前 20% 步骤内提取目标 Token 的注意力图$ {A}_{i} $；
5）竞争注意力约束：在前若干扩散步骤中，构建总损失$ {\mathcal{L}}_{\text{total}} $更新 Token 特征和噪声分布 $ {Z}_{{\mathrm{T}}} $.
6）反向去噪：根据更新后的Token 特征和噪声分布完成去噪过程，得到去噪分布$ {Z}_{{\mathrm{T}}-1} $.
7）图像解码：基于$ {Z}_{0} $解码出最终生成图像 $ {I}_{{\mathrm{out}}} $.

下载: 导出CSV

表 1 T2I-CompBench 属性绑定性能比较

Table 1. Comparison of attribute binding performance on T2I-CompBench

方法	颜色	纹理	形状
SD2.1	0.5001	0.4900	0.4201
SDXL	0.6369	0.5637	0.4975
Composal	0.4001	0.3609	0.3202
TokenCompose	0.5516	0.6160	0.4785
Ranni	0.5446	0.5970	0.4732
GORS	0.6400	0.5900	0.4500
CONFORM	0.6800	0.6100	0.4900
InitNO	0.6900	0.6100	0.4900
SynGen	0.7010	0.6044	0.5069
StructureDiff	0.5092	0.4996	0.4416
Attn-Exct	0.6309	0.5550	0.5075
MAGNET	0.6839	0.5608	0.4974
本文方法（Magnet）	0.7392	0.6205	0.5013
IterComp	0.7418	0.6410	0.5419
本文方法（IterComp）	0.7822	0.6501	0.5480

下载: 导出CSV

表 2 航空场景下不同方法的生成结果对比

Table 2. Comparison of generation results from different methods in aviation scenes

提示词	本文方法	SDXL	MAGNET	Attn-Exct
a silver runway and a blue passenger plane
a red terminal and a white jet
a golden control tower and a green cargo
a silver gate and a blue drone
a white hangar and a purple private jet
a black runway and a yellow biplane

下载: 导出CSV

表 3 不同生成配置下的多属性对象结果

Table 3. Generation results for multi-attribute objects under different configurations

提示词	Config1	Config2	Config3	Config4	Config5
a silver runway and a blue passenger plane
a black apple and a green backpack
a round bagel and a square toaster
a fabric pants and a fluffy rug

下载: 导出CSV

表 4 各模块组合的消融实验结果

Table 4. Ablation results for different module combinations

方法	颜色	纹理	形状
基础模型	0.6369	0.5637	0.4975
语义增强	0.6407	0.5724	0.4977
语义增强 + Token编码处理	0.6850	0.5738	0.4982
语义增强 + Token编码处理 + 内部损失	0.7286	0.6185	0.4992
Token编码处理 + 内部损失 + 外部损失	0.7362	0.6190	0.5001
语义增强 + Token编码处理 + 内部损失 + 外部损失	0.7392	0.6205	0.5013

下载: 导出CSV

表 5 正向/反向组合在T2I-benchmark上的性能比较

Table 5. Performance comparison of positive and negative combinations on T2I-Benchmark

方法	颜色	纹理	形状
反向注入 + 多邻居模块	0.7001	0.6020	0.4899
正向注入 + 多邻居模块	0.7311	0.6160	0.5001
正向注入 + 反向注入	0.7308	0.6149	0.5987
正向注入 + 反向注入 + 多邻居模块	0.7392	0.6205	0.5013

下载: 导出CSV

参考文献(36)

[1]	Li B W, Qi X J, Lukasiewicz T, et al. Controllable text-to-image generation[C]//Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019). San Diego: Neural Information Processing Systems Foundation, Inc. , 2019: 2065-2075.
[2]	张旭龙, 瞿晓阳, 谢骏飞, 等. 人工智能生成内容技术综述[J]. 大数据, 2026, 12(1): 146-173. doi: 10.11959/j.issn.2096-0271.2025071 Zhang Xulong, Qu Xiaoyang, Xie Junfei, et al. AIGC: a survey of AI-generated content technologies[J]. Big Data Research, 2026, 12(1): 146-173. doi: 10.11959/j.issn.2096-0271.2025071
[3]	Croitoru F A, Hondru V, Ionescu R T, et al. Diffusion models in vision: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10850-10869. doi: 10.1109/TPAMI.2023.3261988
[4]	Huang K Y, Sun K Y, Xie E Z, et al. T2I-CompBench: a comprehensive benchmark for open-world compositional text-to-image generation[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 78723-78747.
[5]	高欣宇, 杜方, 宋丽娟. 基于扩散模型的文本图像生成对比研究综述[J]. 计算机工程与应用, 2024, 60(24): 44-64. doi: 10.3778/j.issn.1002-8331.2405-0048 Gao Xinyu, Du Fang, Song Lijuan. Comparative review of text-to-image generation techniques based on diffusion models[J]. Computer Engineering and Applications, 2024, 60(24): 44-64. doi: 10.3778/j.issn.1002-8331.2405-0048
[6]	Cho J, Zala A, Bansal M. Visual programming for step-by-step text-to-image generation and evaluation[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 6048-6069.
[7]	Guo X F, Liu J L, Cui M M, et al. Initno: boosting text-to-image diffusion models via initial noise optimization[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 9380-9389.
[8]	Salih Meral T H, Simsar E, Tombari F, et al. CONFORM: contrast is all you need for high-fidelity text-to-image diffusion models[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 9005-9014.
[9]	Wang Z R, Sha Z Z, Ding Z, et al. TokenCompose: text-to-image diffusion with token-level supervision[PP/OL]. V2. arXiv (2024-06-23)[2025-02-12]. https://doi.org/10.48550/arXiv.2312.03626.
[10]	Balaji Y, Nah S, Huang X, et al. eDiff-I: text-to-image diffusion models with an ensemble of expert denoisers[PP/OL]. V5. arXiv (2023-03-14)[2025-02-03]. https://doi.org/10.48550/arXiv.2211.01324.
[11]	童至慧, 孙立洋, 董雪, 等. 一种改进扩散Transformer的文本到图像生成方法[J]. 计算机与现代化, 2026(1): 76-82. doi: 10.3969/j.issn.1006-2475.2026.01.011 Tong Zhihui, Sun Liyang, Dong Xue, et al. A method of improving diffusion transformer for text-to-image generation[J]. Computer and Modernization, 2026(1): 76-82. doi: 10.3969/j.issn.1006-2475.2026.01.011
[12]	Chen M H, Laina I, Vedaldi A. Training-free layout control with cross-attention guidance[C]//2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE, 2024: 5331-5341.
[13]	Wang X D, Darrell T, Rambhatla S S, et al. InstanceDiffusion: instance-level control for image generation[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 6232-6242.
[14]	Mou C, Wang X T, Xie L B, et al. T2I-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(5): 4296-4304. doi: 10.1609/aaai.v38i5.28226
[15]	Yang L, Yu Z C, Meng C L, et al. Mastering text-to-image diffusion: recaptioning, planning, and generating with multimodal LLMs[PP/OL]. V3. arXiv (2024-05-05)[2025-02-06]. https://doi.org/10.48550/arXiv.2401.11708.
[16]	Li S, Wang R C, Hsieh C J, et al. MuLan: multimodal-LLM agent for progressive and interactive multi-object diffusion[PP/OL]. V2. arXiv (2024-05-24)[2025-02-25]. https://doi.org/10.48550/arXiv.2402.12741.
[17]	Zhou D W, Li Y, Ma F, et al. MIGC: multi-instance generation controller for text-to-image synthesis[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 6818-6828.
[18]	胡洪峰, 周越. 基于LoRA的多主体个性化图像生成方法[J]. 中国体视学与图像分析, 2024, 29(4): 320-332. doi: 10.13505/j.1007-1482.2024.29.04.006 Hu Hongfeng, Zhou Yue. Multi-subject personalized image generation method based on LoRA[J]. Chinese Journal of Stereology and Image Analysis, 2024, 29(4): 320-332. doi: 10.13505/j.1007-1482.2024.29.04.006
[19]	Jiang D Z, Song G L, Wu X S, et al. CoMat: aligning text-to-image diffusion model with image-to-text concept matching[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024: 76177-76209.
[20]	Feng Y T, Gong B, Chen D, et al. Ranni: taming text-to-image diffusion for accurate instruction following[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 4744-4753.
[21]	Hu X W, Wang R, Fang Y X, et al. ELLA: equip diffusion models with LLM for enhanced semantic alignment[PP/OL]. V1. arXiv (2024-03-08)[2025-03-08]. https://doi.org/10.48550/arXiv.2403.05135.
[22]	Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. San Diego: International Machine Learning Society, 2021: 8748-8763.
[23]	Lai Z Q, Zhu X Z, Dai J F, et al. Mini-DALLE3: interactive text to image by prompting large language models[PP/OL]. V2. arXiv (2023-10-12)[2025-02-13]. https://doi.org/10.48550/arXiv.2310.07653.
[24]	Zhuang C Y, Hu Y, Gao P. Magnet: we never know how text-to-image diffusion models work, until we learn how vision-language models function[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024: 57115-57149.
[25]	Qi P, Zhang Y H, Zhang Y H, et al. Stanza: a Python natural language processing toolkit for many human languages[PP/OL]. V2. arXiv (2020-04-23)[2025-03-16]. https://doi.org/10.48550/arXiv.2003.07082.
[26]	Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 10674-10685.
[27]	Podell D, English Z, Lacey K, et al. SDXL: improving latent diffusion models for high-resolution image synthesis[PP/OL]. V1. arXiv (2023-07-04)[2025-02-05]. https://doi.org/10.48550/arXiv.2307.01952.
[28]	Tang Z N, Yang Z Y, Zhu C G, et al. Any-to-any generation via composable diffusion[PP/OL]. V1. arXiv (2023-05-19)[2025-02-22]. https://doi.org/10.48550/arXiv.2305.11846.
[29]	Wang Z R, Sha Z Z, Ding Z, et al. TokenCompose: text-to-image diffusion with token-level supervision[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 8553-8564.
[30]	Chefer H, Alaluf Y, Vinker Y, et al. Attend-and-excite: attention-based semantic guidance for text-to-image diffusion models[J]. ACM Transactions on Graphics, 2023, 42(4): 1-10. doi: 10.1145/3592116
[31]	Feng W X, He X H, Fu T J, et al. Training-free structured diffusion guidance for compositional text-to-image synthesis[PP/OL]. V3. arXiv (2023-02-28)[2025-02-12]. https://doi.org/10.48550/arXiv.2212.05032.
[32]	Zhang X C, Yang L, Li G H, et al. IterComp: iterative composition-aware feedback learning from model gallery for text-to-image generation[PP/OL]. V2. arXiv (2025-02-05)[2025-05-10]. https://doi.org/10.48550/arXiv.2410.07171.
[33]	马庆禄, 张丽, 马恋, 等. 面向自动驾驶的隧道车道线AI自适应识别方法[J]. 西南交通大学学报, 2026, 61(2): 452-465. doi: 10.3969/j.issn.0258-2724.20240070 Ma Qinglu, Zhang Li, Ma Lian, et al. Artificial intelligence adaptive recognition method for tunnel lane lines in autonomous driving[J]. Journal of Southwest Jiaotong University, 2026, 61(2): 452-465. doi: 10.3969/j.issn.0258-2724.20240070
[34]	王庞伟, 徐京辉, 何昕泽, 等. 基于多模态语义认知的城市自主式交通系统态势推演模型研究[]. 西南交通大学学报, 2026, 61(5): 1743-1756. Wang Pangwei, Xu Jinghui, He Xinze, et al. Urban autonomous traffic system situation evolution modeling based on multimodal semantic cognition[J/OL]. Journal of Southwest Jiaotong University, 2026, 61(5): 1743-1756.
[35]	夏英, 刘敏. 基于时空注意力卷积神经网络的交通流量预测[J]. 西南交通大学学报, 2023, 58(2): 340-347. doi: 10.3969/j.issn.0258-2724.20210526 Xia Ying, Liu Min. Traffic flow prediction based on spatial-temporal attention convolutional neural network[J]. Journal of Southwest Jiaotong University, 2023, 58(2): 340-347. doi: 10.3969/j.issn.0258-2724.20210526
[36]	Rassin R, Hirsch E, Glickman D, et al. Linguistic binding in diffusion models: enhancing attribute correspondence through attention map alignment[C]//Advances in Neural Information Processing Systems 36. New Orleans, Louisiana, USA. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 3536-3559.