UI Element Collector

This page explains the purpose of Curator's UI Element Collector, how to use it while building an automation, and how the Markdown file it generates is leveraged inside code assistants such as Cursor, Claude Code, and Codex. The content is aimed at automation developers, analysts, and teams building robots with the help of artificial intelligence.

Overview

Every interface automation needs to know where the elements it will interact with are located: fields, buttons, menus, tables, and windows. The slowest and most fragile part of this work is usually discovering and validating the selectors for these elements, one by one, by manually inspecting the screen.

The Element Collector solves this step. It lets you capture elements directly on screen, both in web pages in Chrome and in Windows desktop applications, give each one a friendly name, annotate the desired action, and export everything into a single Markdown file, called the selector dossier.

Real screen -> Element Collector -> seletores-automacao.md -> Code assistant -> Automation

The dossier works as a bridge between the person who knows the business process and the AI tool that will write the code. Instead of describing the screen in free text, you hand the assistant a precise list of elements, with already-validated selectors and the intended use of each one.

Where the collector fits in automation production

A typical automation creation flow in Curator follows these steps:

1. Create the automation project in Curator, for example a Performer that consumes items from a queue.
2. Open the Element Collector and capture the elements of the screens involved in the process, in the order the automation will use them.
3. Save the dossier in the project folder.
4. Open the project in a code assistant, such as Cursor.
5. Ask the assistant to implement the automation logic using the dossier's selectors.
6. Test, adjust, and publish the automation.

Without the collector, the code assistant would have to guess the selectors, or you would have to inspect each element manually and transcribe the results. With the collector, this information arrives ready, validated, and organized.

What the collector captures

The collector works with two sources at the same time:

• Web pages open in Chrome.
• Windows desktop applications.

The capture order is preserved across both sources. This allows documenting a process that alternates between browser and desktop system, in the actual execution sequence.

For each captured element, the collector records:

• A friendly name, defined by you. For example: CPF Field, Login Button, Results grid.
• The screen where the element is, such as the page title or the program name.
• The desired action, annotated by you. For example: click, type, collect information.
• Selectors to locate the element, with the most robust one recommended first.
• In tables and grids, the columns relevant to the automation.

For web elements, selectors follow the pattern used by browser automation tools, prioritizing stable locators such as test identifiers, element role, and visible text, before falling back to more fragile paths. For desktop elements, the selector describes the process, the window, and the control's automation properties.

How to use the collector

1. Select the project in Curator.
2. Open the Tools menu and click Element Collector (UI).
3. Start web collection to capture in Chrome pages, or desktop collection to capture in Windows applications. You can switch between the two in the same session.
4. Hover the mouse over the screen: the element under the cursor is highlighted.
5. Use Ctrl+Click to capture the highlighted element. On desktop, normal clicking keeps working, so you can navigate through the system while collecting.
6. In the list of captured elements, rename each item with a name that makes sense for the process and annotate the desired action.
7. If you capture a table, choose the columns the automation will use.
8. Use the watch option to confirm that the selector still locates the correct element on screen.
9. When finished, stop the collection and save the dossier.

Capture tips

• Capture elements in the order the automation will interact with them. The dossier order helps the code assistant understand the flow.
• Prefer names that describe the element's role in the process, not its appearance. Account holder CPF field is better than input 3.
• Annotate the action with the business intent. Type the CPF before continuing is more useful than just type.
• Remove accidental captures before saving. A clean dossier produces better code.

The generated file

When saving, the collector suggests writing the file seletores-automacao.md in the root of the project folder. Keeping this name and location makes it easier for code assistants, which read the project structure, to discover the file.

The file is a simple, lean Markdown. Each element becomes a section with name, screen, action, columns when it is a table, and up to two selectors, from the most robust to the alternative one.

Dossier example:

# Elementos coletados

## 1. Campo CPF

**Tela:** Portal do Cliente

**Acao:** Digitar o CPF do titular antes de clicar em Continuar

**Seletor 1**

    page.get_by_role("textbox", name="CPF")

**Seletor 2**

    page.locator("input[name=\"cpf\"]")

## 2. Botao Continuar

**Tela:** Portal do Cliente

**Acao:** Clicar apos preencher o CPF

**Seletor 1**

    page.get_by_role("button", name="Continuar")

## 3. Grid de resultados

**Tela:** Sistema ERP

**Acao:** Extrair as linhas visiveis da consulta

**Colunas:** Codigo, Descricao, Valor

**Seletor 1**

    process="erp.exe" window="Consulta" > type="DataGridControl" automation_id="gridResultados"

The format is intentionally minimalist. Internal capture details, such as screenshots and complete technical metadata, stay only in the collector session and do not go into the file. A short, objective dossier takes up less of the AI model's context and produces more precise answers.

Using the dossier in code assistants

The dossier was designed to be consumed by AI tools that generate code, such as Cursor, Claude Code, and Codex. The recommended flow is:

1. Save seletores-automacao.md in the project root.
2. Open the project in the code assistant. In Curator, the Tools menu has the option to open the project directly in Cursor.
3. Reference the file in your request. In Cursor, for example, use @seletores-automacao.md in the chat. In other tools, ask the assistant to read the file or paste the content into the prompt.
4. Describe the automation's goal and ask for the implementation using the dossier's selectors.

Request example:

Read the seletores-automacao.md file. Implement the processing of each
queue item: fill the CPF Field with the item's data, click the Continue
Button, and extract the Results grid columns as output data.
Use exactly the selectors from the dossier, always preferring Selector 1.

Why this works well:

• The assistant does not need to guess selectors. AI-invented selectors are one of the main causes of automations that fail on the first run.
• The friendly name and annotated action provide business context. The model understands what each element represents in the process, not just where it is.
• The order of elements suggests the automation flow.
• The selectors were already validated on the real screen during collection.

Use with queue projects

The collector complements queue-based automation models. A common design is:

1. Create the queue and a Performer project in Curator.
2. Collect the elements of the screens the Performer will operate on.
3. Save the dossier in the Performer project folder.
4. Ask the code assistant to implement the item processing step using the dossier's selectors.

This way, the project structure handles the queue cycle, such as fetching the next item and recording success or failure, and the dossier handles the screen interaction. The code assistant connects the two parts.

Limitations and precautions

• Desktop collection works with Windows applications. Web collection works with Chrome.
• The dossier describes elements, not business rules. Validations, decisions, and error handling remain the responsibility of the developer and the request made to the assistant.
• Selectors can break when the target system changes. Treat the dossier as a living artifact: re-collect when the screen changes.

Common mistakes

• ❌ Saving the dossier outside the project folder so the assistant cannot find the file.
• ❌ Leaving generic names, such as element 1, which do not help the model understand the process.
• ❌ Capturing elements out of order and expecting the assistant to deduce the correct flow.
• ❌ Manually editing selectors in the code without updating the dossier, creating two diverging sources of truth.
• ❌ Asking the AI to create new selectors when the right thing to do would be to re-collect on the real screen.

Next steps

• Read Queue System to understand how to structure automations with Dispatcher and Performer.
• Read Dispatcher and Performer to see where screen interaction fits into the item processing cycle.