[Seminar] Layout Generation using Retrieval Augmented Generation Framework

Speaker: SeongRae Noh, github.com/Noh...
Target Paper: Retrieval-Augmented Layout Transformer for Content-Aware Layout Generation, 2024, CVPR

[@IIIXRLab]

[Overall] Your seminar flow is good and presenting background knowledge is truly helpful to understand the paper architecture. However, the presentation covers a complex topic with a significant amount of detail, but it struggles to effectively communicate these details to the audience. The visual and textual elements are not well-aligned, which may confuse viewers and detract from the overall message. [Feedback] F1: Simplification and clarity are needed! Simplify the content of the figures or provide more detailed explanations that clearly indicate which part of the figure is being discussed. This will help prevent the audience from feeling overwhelmed by the complexity. For example, on the 7th topic "What is retrieval augmented generation?", a bunch of figures without clear explanations overwhelm audiences. F2: Do audience-friendly design! Redesign the presentation to be more audience-friendly. The presentation should guide the viewer through the content in a way that is easy to follow, ensuring that key points are highlighted and clearly explained. F3: Need enhanced alignment. Ensure that the text and visuals are better aligned. The audience should be able to understand the overall concept from the visuals, even if they miss part of the spoken explanation. Additionally, the detailed aspects should be clearly marked and explained in relation to the visuals. F4: Be aware of the page number. [Question] Q1: Please clarify whether the conditional inputs are provided by the user and If there are specific constraints on these inputs. Q2: On topic 9 "Input", are the "Relations" like "text2 at the bottom of the canvas" created automatically? or do they require manual formatting to convey specific meanings? Q3: May I ask you to provide a clearer explanation of how conditional inputs interact with the bidirectional transformer encoder? Specifically, clarify what is represented by the image that follows the token-is it an image description, or is there a mistake in the presentation's depiction of the input image? Q4: Please explain why saliency is used instead of segmentation, especially if saliency is meant to express the area to be preserved. Is the use of saliency indeed superior to segmentation in this context? Q5: Please let me know how the pre-trained transformer model extracts features from the retrieved layout. I need rationale behind the model focusing on the layout feature and potentially ignoring the image area.

[@노성래99]

Thank you for your feedback and questions. Q1: Similar to other conditional generation models, the conditional input must be provided by the user. In this paper, the constraint on conditional input is that only a limited set of words can be used, as the model is not trained to accept free-form natural language input like an LLM or CLIP’s text encoder. This restriction on the form of the conditional input is also detailed in the "constraint serialization" section of the Input page. Q2: These must carry specific meanings and be manually crafted to fit the required format. Q3: The constraint serialization technique is a module that formats the relation constraints of a layout, specified by the conditional input, to start with and end with . The image following the token is distinct from the input image; it refers to an image that appears as a layout element. I'm sorry for not providing a sufficient explanation of the differences between the two images. Q4: Segmentation returns the boundaries of all detailed objects in a given image, whereas a saliency map provides a heatmap highlighting the areas that are most attention-grabbing to humans. In layout design, saliency maps are used based on the assumption that no other elements will be placed in the most attention-grabbing areas of an image. While this assumption is often correct, there are exceptions in real-world data, which is a limitation of this approach. Q5: In an image and layout pair, the image feature is used to find the most similar content image through similarity-based search with the input query. The image feature is then discarded, leaving only the layout feature in use, based on the assumption that similar content images will have similar layout distributions. This assumption works strongly in the fixed domain, but it may reveal limitations when the purpose or type of layout changes.

[@김병민_dm]

Thank you for presentation. I have two questions: I'm curious about specific methods for providing a guide to a generative model. For example, does it automatically recognize an image you input and identify the positions of text and pictures, or is there another method? I understand that input modalities include text, code, and images, but what do "Knowledge" and "Science" mean in this context? Does it refer to giving input related to fields like medicine, or does it mean that it can generate content based on actual chemical formulas provided as input?

[@노성래99]

Thank you for your questions. Q1: The model presented in the paper is a deep learning-based generative model, which automates the processing of various layout features using transformers, eliminating the need for manual processing. The image encoder automatically interprets the image, while the decoder generates the layout, making the entire process largely automatic. Q2: Indeed, RAG-based approaches can be applied to molecular generation and drug discovery, and there is an entire domain dedicated to this area. In terms of knowledge, it is typically addressed from the perspectives of knowledge discovery and data mining. For example, it is possible to predict credit card approval rates based on a person's social media followers/following and the content of their posts.

[@박주현-123]

Thank you for your presentation Q1. The model seems to return the bounding boxes within the input that designate where the layouts should be placed. Has there been any works on providing more high-level details for the layouts, such as the fonts or the style of expressions for each bounding box? Q2. Do you find the quantitative evaluation for a poster's layout appropriate? From my perspective, the concept of the "best design" can be subjective and should differ depending on individual preferences.

[@노성래99]

Thank you for your questions. Q1: While some end-to-end frameworks generate layouts using existing layout generation methods and then proceed with content generation based on those layouts, most papers treat layout generation and layout-based content generation as separate tasks. Q2: Evaluating based on the "best design" has the inherent limitation that it can vary significantly due to individual preferences. Among the evaluation metrics mentioned, FID (Fréchet Inception Distance) is relatively agnostic to personal preference, as it quantitatively measures the similarity between the generated images and real images without relying on subjective judgment.

[@LRH_iiixrlab]

Thank you for your presentation, but I have some questions. It wasn't very clear to me why using RAG leads to better performance compared to other layout generation models. What specific aspects of RAG do you think contribute to this improvement? If there are points I might have missed during the presentation, please let me know.

[@노성래99]

Thank you for your question. Q1: As mentioned on slide 5 of the presentation, layout generation often involves multiple possible layouts for a given input, and the model may struggle to determine which one to generate. In such cases, a layout retrieved from a database based on the existing content can serve as a guideline for the model, helping it make more informed decisions during the generation process.

[@정승재_teclados078]

Thank you for your presentation. Q1. Why was the latent-based method chosen among the four methods described on slide 7? The main difference between the other methodologies seems to be at which level the retrieved data is combined. It is unclear why combining at the image level or encoding each retrieved data individually would be less advantageous. Q2. Using simpler modules or processes is generally thought to increase processing time, model complexity, and model reusability. This paper uses various inputs, constraints, and pre-trained models to achieve high scores across multiple metrics. However, there seem to be too few ablation studies presented, making it unclear what role each module plays. Are there any ablation studies not covered in the presentation? For example, showing the performance difference between a model trained with only the image and retrieved data versus one that also uses a saliency map would help justify the use of the saliency map. Q3. Finally, if the model does not generate what content should be placed inside the layout, what is the meaning or practical use of layout generation? Are there fields that generate both the layout and the content that will fill the layout together?

[@노성래99]

Thank you for your questions. Q1: It's unclear what exactly is meant by combining data at the image level, but from what I know, query-based RAG is typically used when the input and retrieved results can be seamlessly integrated (e.g., input: a short text condition, retrieved results: a detailed document, combined results: the text condition followed by the detailed document). While it might be possible to combine images without using an encoder, I am not aware of this being practiced in the field. The paper does not mention encoding individual data separately. Personally, I speculate that when multiple data points are retrieved, conditioning on the features that encompass these data points might provide a richer representation. Q2: As you mentioned, additional ablation studies were conducted, including an investigation into the fusion of features before passing them through the layout encoder. Some other minor ablation studies are also included in the supplemental material. Q3: Layout-based content generation (input: well-defined layout, output: image) is an area of active research. Additionally, image or image-based content generation conditioned on layout generally yields higher quality results compared to text-based or other forms of conditional generation.

[@MisongKim-ki9nh]

Thank you for your presentation. 1. Is the model described in the paper also generating text, or is it focused solely on layout generation? If it is only generating layouts, could you explain why the metric for evaluating the non-flatness of text elements is included in the evaluation? 2. Design preferences can vary greatly depending on individual tastes. While the paper focuses on quantitative evaluation, I'm curious to know if any user studies were conducted to complement these evaluations.

[@노성래99]

Thank you for your questions. Q1: The model does not generate text directly; however, it does generate the category of each layout element (e.g., logo, text, underline). Non-flatness is measured specifically for layout elements predicted to belong to the text category. Q2: The paper did not conduct a user study. Additionally, design preference is a crucial factor when using generative models to create content where layout and planning play significant roles. While the evaluation of whether the model adheres to constraints in conditional generation can partially address this aspect, it is not yet fully comprehensive.

[@SeungWonSeo-q3s]

Thank you for your presentation. I have two questions. Q1. I have a question about the Retriever Module. From what I understand, it seems that BASNet can be used to easily convert images into saliency maps. Since a saliency map represents regions that attract human attention, I think that retrieving based on the saliency map would reduce the chances of retrieving layouts that obscure or distract from the image. However, in the materials you presented, it seems that the retriever module only uses the image itself for retrieval. Is there a specific reason for this? Q2. I'm curious about why the cross-attention values, called feature fuse, are included in the Generator Module. Since f_I and f_L are already inputted, what hidden insights might the authors have had for also concatenating their cross-attention values (feature fuse) as input?

[@노성래99]

Thank you for your questions. Q1: Saliency-based retrieval showed lower performance compared to image-based retrieval in the ablation study. I suspect this is because a saliency map can be considered a subset of the total features of a content image. Using the richer set of features from the full image likely leads to better results, as it captures more comprehensive information about the content. Q2: This approach is actually a conventional methodology in the deep learning field, primarily used for extracting and computing features from multiple perspectives. While it's clear that ​ image features and layout features will be processed through attention in the transformer decoder, introducing cross-attention during intermediate steps and then concatenating the results could have experimentally proven beneficial. This additional step may enhance the model's ability to integrate and leverage different types of features more effectively.

[@황주영-j7y]

Thank you for the seminar. I have two questions. 1. What happens when the conditional input contradicts the ground truth? 2. Can the layout suitable for images be expanded to symbols (like pictograms) instead of text, or do more factors need to be considered?

[@노성래99]

Thank you for your questions. Q1: During the training process, if there are issues with the dataset or if the conditional input constraints are intentionally made inconsistent with the ground truth (GT) data, the model's performance will degrade. Q2: While it is possible to use a single model to predict detailed content for each layout starting from an input image, a 2-stage approach is more practical. The problem with using a single model is that it would make the data excessively high-dimensional, leading to increased complexity. In a 2-stage approach, the process is divided into two steps: first, generating the layout, and then predicting the detailed content for each layout element. This approach simplifies the task and improves manageability.

[@홍성은-iiixr]

Thank you for your great presentation. I have two questions. On page 9, in the example image | text ..., is the '|' symbol actually used to represent a special token for separating different modalities, or is it just a placeholder symbol? The presentation mentioned several evaluation metrics, but it did not include WD, which is commonly used in this research area. Was there a specific reason why WD was not evaluated?

[@노성래99]

Thank you for your questions. Q1: The target paper states that it follows the implementation of Layoutformer++, but does not provide detailed explanations. Based on the original Layoutformer++ paper, it appears to have been used as a placeholder. I apologize for any confusion this may have caused. Q2: I am not entirely sure why WD is not used. One possible hypothesis is that the poster layout domain exhibits a long-tail distribution among layouts, making it more practical to use a manually designed metric based on widely recognized characteristics of posters rather than calculating WD based on specific features.

[@포로뤼]

Thank you for the excellent presentation. I have a few questions: 1. Does the conditional input consist of text? I'm not quite sure how the conditional input is reflected in the learning process. Could you clarify that? 2. What kind of loss function is used to train the Generator module? 3. What data is required for training? To my understanding, poster images, layouts, and conditional input are needed. Is that correct? Also, I'm curious to know if there are many datasets like this available. 4. Why is DreamSim considered a more human perception-aligned metric compared to other similarity metrics?"

[@노성래99]

Thank you for your questions. Q1: The conditional input is converted into standardized text through constraint serialization, which is then encoded into a latent variable by a bidirectional transformer. This latent variable is subsequently concatenated with other latent variables and fed into the layout decoder. Q2: As outlined in the objective function provided on page 11, this paper generates layouts autoregressively based on a transformer architecture, comparing the generated individual layout elements' c, x, y, w, and h values with the ground truth (GT) values. A key implementation detail is the use of teacher forcing during training. Q3: Training requires three types of data: input image content without a layout, ground truth (GT) data containing the same image content with layout elements, and a layout dataset for retrieval. The conditional input is optional. For poster datasets, the PKU dataset and CGL dataset are commonly used. Compared to other fields, poster layout datasets are relatively limited in both scale and variety. Q4: I recommend referring to the original DreamSim paper for more details on this. From my understanding, DreamSim is designed to be a similarity function based on features that humans perceive as significantly different between images. Unlike traditional similarity functions, DreamSim is supported by a large-scale, detailed user study, which validates its effectiveness in capturing these perceptual differences.

[@tjswodud-c6c]

Thank you for your good presentation. I have 4 questions: 1. (In p.11) I was wondering what authors mean by token that the layout decoder predicts every frame. As I understand it, they mean each layout component (image, text, etc...), am I understanding you correctly? 2. It seems that the features that pass the layout encoder in the Retriever module enter the Generator module as input as f_L, where f_L is a concatenation of the features of “all” the layouts retrieved from the database? Or is it entering the Generator module in some other way? 3. The final output of the model seems to be of the form (c, x, y, w, h), what do c, x, and y stand for respectively? (I assume that w, h stand for the width and height of the layout). 4. I understand that you are working on a study on city generation. City generation and poster layout generation are perceived as two very different domains, do you have any prior studies of applying layout generation to city generation? Thank you.

[@노성래99]

Thank you for your questions. Q1: Correct. More precisely, the model autoregressively predicts individual elements of the ground truth (GT) layout data (e.g., c1, x1, y1, …). This means that each element of the layout is generated one after the other in sequence, based on the previously generated elements. Q2: All layouts retrieved by the retriever module are passed through the layout encoder, and then concatenated. This concatenated representation is subsequently fused with the input image latent variable (as described on page 11). Q3: As described on page 2 of the presentation, x and y refer to the coordinates in a 2D space, representing the location of elements. C denotes the category of the layout element (e.g., Logo, text, underline, etc.). Q4: While the characteristics of the two domains are quite different, city generation fundamentally requires a city layout. Directly generating a 3D city remains challenging outside of procedural generation methods. In fact, published city layout generation papers share many similarities with layout generation. While layout generation involves determining the position and shape of rectangular layout elements with specific categories on a standardized rectangular canvas, city layout generation can be seen as an extended problem. It involves handling the positions and shapes of various building forms on a non-standardized canvas.

[@RounLee0927]

I have a few questions. Q1. I'm curious about how the research could be used and why it's needed. Is it out of necessity to create a poster, or on a larger scale, where can creating a layout be utilized? Q2. The generated layout seems to reflect categories such as text and logo, but does it also reflect elements such as the content of the text, font, and font size?

[@노성래99]

Thank you for your question. Q1: Layout-based content generation remains a promising and actively researched field. While this paper primarily focuses on poster layouts, the concepts are also applicable to scene layouts and 3D mesh layouts. Moreover, layout-based content generation tends to produce higher-quality results compared to simple text-based or unconditional generation, and it better adheres to complex conditions or constraints. Q2: No, it does not create actual content.

[@한동현Han]

Thank you for the presentation. I have two questions. Q1: How can this generated layout be used? I can't quite imagine how this layout output alone could be applied to real content. How does this research field connect to final content generation, such as incorporating text, logos, or shapes into the layout? Q2: How large of a dataset is generally needed for high-quality layout generation?

[@노성래99]

Thank you for your question. Q1: Layout-based content generation remains a promising and actively researched field. While this paper primarily focuses on poster layouts, the concepts are also applicable to scene layouts and 3D mesh layouts. Moreover, layout-based content generation tends to produce higher-quality results compared to simple text-based or unconditional generation, and it better adheres to complex conditions or constraints. Q2: While it's challenging to determine exact numbers, one can estimate the potential number of realistic layouts for a given content image by considering the size of the existing dataset multiplied by the number of feasible layouts per content image. This provides a rough approximation of the diversity and variety of layouts that the model might encounter or generate.

[@critbear]

It feels unnatural to create a layout just by looking at the image. Wouldn't it be better to create a layout by considering the content of the layout? Or are there already such researches?