- ISSN 0258-2724
- CN 51-1277/U
- EI Compendex
- Scopus
- Indexed by Core Journals of China, Chinese S&T Journal Citation Reports
- Chinese S&T Journal Citation Reports
- Chinese Science Citation Database
| Citation: | LIU Yi, ZHOU Jing, YUAN Ting, WANG Qian. Text-to-Image Generation Method Based on Fine-Grained Token Competitive Attention[J]. Journal of Southwest Jiaotong University. doi: 10.3969/j.issn.0258-2724.20250670
|
In multi-attribute text-to-image generation, maintaining object integrity and semantic consistency in complex scenes remains challenging, primarily due to competition and interference among attributes that can lead to structural incompleteness and semantic mismatches. Existing diffusion models struggle with multiple objects and their attribute descriptions due to insufficient prompt information, semantic entanglement, and uneven attention allocation, leading to missing objects, attribute confusion, and misaligned relations. To address this, a multi-stage generation method based on fine-grained token-level competitive attention regularization is proposed, enabling precise control via prompt enhancement, semantic decoupling, and attention optimization. First, in the prompt stage, an LLM enhances the semantics of the input text and structures its representation to reduce ambiguity. Next, in the encoding stage, an attribute-object-level semantic decoupling mechanism is designed, integrating a token encoding module, positive and negative semantic vectors, and a semantic neighbor strategy to improve the stability and discriminability of embeddings while reducing semantic interference. Finally, in the generation stage, dual competitive attention regularization enhances the responses of attributes and objects in cross-modal attention while suppressing attention overlap between different objects, thus improving object independence and reducing semantic interference. Experiments on the T2I-benchmark show significant improvements in color, shape, and texture attribute binding, with bootstrapping language-image pre-training visual question answering (BLIP-VQA) scores of
| [1] |
Li B W, Qi X J, Lukasiewicz T, et al. Controllable text-to-image generation[C]//Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019). San Diego: Neural Information Processing Systems Foundation, Inc. , 2019: 2065-2075.
|
| [2] |
张旭龙, 瞿晓阳, 谢骏飞, 等. 人工智能生成内容技术综述[J]. 大数据, 2026, 12(1): 146-173. doi: 10.11959/j.issn.2096-0271.2025071
Zhang Xulong, Qu Xiaoyang, Xie Junfei, et al. AIGC: a survey of AI-generated content technologies[J]. Big Data Research, 2026, 12(1): 146-173. doi: 10.11959/j.issn.2096-0271.2025071
|
| [3] |
Croitoru F A, Hondru V, Ionescu R T, et al. Diffusion models in vision: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10850-10869. doi: 10.1109/TPAMI.2023.3261988
|
| [4] |
Huang K Y, Sun K Y, Xie E Z, et al. T2I-CompBench: a comprehensive benchmark for open-world compositional text-to-image generation[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 78723-78747.
|
| [5] |
高欣宇, 杜方, 宋丽娟. 基于扩散模型的文本图像生成对比研究综述[J]. 计算机工程与应用, 2024, 60(24): 44-64. doi: 10.3778/j.issn.1002-8331.2405-0048
Gao Xinyu, Du Fang, Song Lijuan. Comparative review of text-to-image generation techniques based on diffusion models[J]. Computer Engineering and Applications, 2024, 60(24): 44-64. doi: 10.3778/j.issn.1002-8331.2405-0048
|
| [6] |
Cho J, Zala A, Bansal M. Visual programming for step-by-step text-to-image generation and evaluation[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 6048-6069.
|
| [7] |
Guo X F, Liu J L, Cui M M, et al. Initno: boosting text-to-image diffusion models via initial noise optimization[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 9380-9389.
|
| [8] |
Salih Meral T H, Simsar E, Tombari F, et al. CONFORM: contrast is all you need for high-fidelity text-to-image diffusion models[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 9005-9014.
|
| [9] |
Wang Z R, Sha Z Z, Ding Z, et al. TokenCompose: text-to-image diffusion with token-level supervision[PP/OL]. V2. arXiv (2024-06-23)[2025-02-12]. https://doi.org/10.48550/arXiv.2312.03626.
|
| [10] |
Balaji Y, Nah S, Huang X, et al. eDiff-I: text-to-image diffusion models with an ensemble of expert denoisers[PP/OL]. V5. arXiv (2023-03-14)[2025-02-03]. https://doi.org/10.48550/arXiv.2211.01324.
|
| [11] |
童至慧, 孙立洋, 董雪, 等. 一种改进扩散Transformer的文本到图像生成方法[J]. 计算机与现代化, 2026(1): 76-82. doi: 10.3969/j.issn.1006-2475.2026.01.011
Tong Zhihui, Sun Liyang, Dong Xue, et al. A method of improving diffusion transformer for text-to-image generation[J]. Computer and Modernization, 2026(1): 76-82. doi: 10.3969/j.issn.1006-2475.2026.01.011
|
| [12] |
Chen M H, Laina I, Vedaldi A. Training-free layout control with cross-attention guidance[C]//2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE, 2024: 5331-5341.
|
| [13] |
Wang X D, Darrell T, Rambhatla S S, et al. InstanceDiffusion: instance-level control for image generation[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 6232-6242.
|
| [14] |
Mou C, Wang X T, Xie L B, et al. T2I-adapter: learning adapters to dig out more controllable ability for text-to-image diffusion models[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(5): 4296-4304. doi: 10.1609/aaai.v38i5.28226
|
| [15] |
Yang L, Yu Z C, Meng C L, et al. Mastering text-to-image diffusion: recaptioning, planning, and generating with multimodal LLMs[PP/OL]. V3. arXiv (2024-05-05)[2025-02-06]. https://doi.org/10.48550/arXiv.2401.11708.
|
| [16] |
Li S, Wang R C, Hsieh C J, et al. MuLan: multimodal-LLM agent for progressive and interactive multi-object diffusion[PP/OL]. V2. arXiv (2024-05-24)[2025-02-25]. https://doi.org/10.48550/arXiv.2402.12741.
|
| [17] |
Zhou D W, Li Y, Ma F, et al. MIGC: multi-instance generation controller for text-to-image synthesis[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 6818-6828.
|
| [18] |
胡洪峰, 周越. 基于LoRA的多主体个性化图像生成方法[J]. 中国体视学与图像分析, 2024, 29(4): 320-332. doi: 10.13505/j.1007-1482.2024.29.04.006
Hu Hongfeng, Zhou Yue. Multi-subject personalized image generation method based on LoRA[J]. Chinese Journal of Stereology and Image Analysis, 2024, 29(4): 320-332. doi: 10.13505/j.1007-1482.2024.29.04.006
|
| [19] |
Jiang D Z, Song G L, Wu X S, et al. CoMat: aligning text-to-image diffusion model with image-to-text concept matching[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024: 76177-76209.
|
| [20] |
Feng Y T, Gong B, Chen D, et al. Ranni: taming text-to-image diffusion for accurate instruction following[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 4744-4753.
|
| [21] |
Hu X W, Wang R, Fang Y X, et al. ELLA: equip diffusion models with LLM for enhanced semantic alignment[PP/OL]. V1. arXiv (2024-03-08)[2025-03-08]. https://doi.org/10.48550/arXiv.2403.05135.
|
| [22] |
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. San Diego: International Machine Learning Society, 2021: 8748-8763.
|
| [23] |
Lai Z Q, Zhu X Z, Dai J F, et al. Mini-DALLE3: interactive text to image by prompting large language models[PP/OL]. V2. arXiv (2023-10-12)[2025-02-13]. https://doi.org/10.48550/arXiv.2310.07653.
|
| [24] |
Zhuang C Y, Hu Y, Gao P. Magnet: we never know how text-to-image diffusion models work, until we learn how vision-language models function[C]//Advances in Neural Information Processing Systems. San Diego: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024: 57115-57149.
|
| [25] |
Qi P, Zhang Y H, Zhang Y H, et al. Stanza: a Python natural language processing toolkit for many human languages[PP/OL]. V2. arXiv (2020-04-23)[2025-03-16]. https://doi.org/10.48550/arXiv.2003.07082.
|
| [26] |
Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2022: 10674-10685.
|
| [27] |
Podell D, English Z, Lacey K, et al. SDXL: improving latent diffusion models for high-resolution image synthesis[PP/OL]. V1. arXiv (2023-07-04)[2025-02-05]. https://doi.org/10.48550/arXiv.2307.01952.
|
| [28] |
Tang Z N, Yang Z Y, Zhu C G, et al. Any-to-any generation via composable diffusion[PP/OL]. V1. arXiv (2023-05-19)[2025-02-22]. https://doi.org/10.48550/arXiv.2305.11846.
|
| [29] |
Wang Z R, Sha Z Z, Ding Z, et al. TokenCompose: text-to-image diffusion with token-level supervision[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2024: 8553-8564.
|
| [30] |
Chefer H, Alaluf Y, Vinker Y, et al. Attend-and-excite: attention-based semantic guidance for text-to-image diffusion models[J]. ACM Transactions on Graphics, 2023, 42(4): 1-10. doi: 10.1145/3592116
|
| [31] |
Feng W X, He X H, Fu T J, et al. Training-free structured diffusion guidance for compositional text-to-image synthesis[PP/OL]. V3. arXiv (2023-02-28)[2025-02-12]. https://doi.org/10.48550/arXiv.2212.05032.
|
| [32] |
Zhang X C, Yang L, Li G H, et al. IterComp: iterative composition-aware feedback learning from model gallery for text-to-image generation[PP/OL]. V2. arXiv (2025-02-05)[2025-05-10]. https://doi.org/10.48550/arXiv.2410.07171.
|
| [33] |
马庆禄, 张丽, 马恋, 等. 面向自动驾驶的隧道车道线AI自适应识别方法[J]. 西南交通大学学报, 2026, 61(2): 452-465. doi: 10.3969/j.issn.0258-2724.20240070
Ma Qinglu, Zhang Li, Ma Lian, et al. Artificial intelligence adaptive recognition method for tunnel lane lines in autonomous driving[J]. Journal of Southwest Jiaotong University, 2026, 61(2): 452-465. doi: 10.3969/j.issn.0258-2724.20240070
|
| [34] |
王庞伟, 徐京辉, 何昕泽, 等. 基于多模态语义认知的城市自主式交通系统态势推演模型研究[]. 西南交通大学学报, 2026, 61(5): 1743-1756.
Wang Pangwei, Xu Jinghui, He Xinze, et al. Urban autonomous traffic system situation evolution modeling based on multimodal semantic cognition[J/OL]. Journal of Southwest Jiaotong University, 2026, 61(5): 1743-1756.
|
| [35] |
夏英, 刘敏. 基于时空注意力卷积神经网络的交通流量预测[J]. 西南交通大学学报, 2023, 58(2): 340-347. doi: 10.3969/j.issn.0258-2724.20210526
Xia Ying, Liu Min. Traffic flow prediction based on spatial-temporal attention convolutional neural network[J]. Journal of Southwest Jiaotong University, 2023, 58(2): 340-347. doi: 10.3969/j.issn.0258-2724.20210526
|
| [36] |
Rassin R, Hirsch E, Glickman D, et al. Linguistic binding in diffusion models: enhancing attribute correspondence through attention map alignment[C]//Advances in Neural Information Processing Systems 36. New Orleans, Louisiana, USA. Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 3536-3559.
|
Figures(4) / Tables(6)
DownLoad:
