A peek into rabbit’s progress with LAM playground

Web agents share the same basic problems to solve as more general cross-platform agents, and they approach these problems by performing two main kinds of work: planning and grounding [11][12]. Planning is predicting how to use a particular website, step by step. Grounding is knowing how to find the interactive elements on the rendered page, and knowing how to interact with them effectively. Both planning and grounding are difficult for even the most state-of-the-art LLM-powered web agents.

Planning

Let’s walk through a hypothetical task in order to see its subparts. The task is to find an Airbnb in LA. Your first step is to enter “Los Angeles” into the location bar. Your second step is to add the dates of your trip. Your third step is to add your guests. Then you need to go into the filters and select the ones you care about. Then comes the actual search. You start looking through all the displayed listings, and one-by-one you decide: do I like this or not? This process of searching might not seem very planned to us, but it does actually follow a pattern. You have an internal set of preferences, and for each listing you measure how it fares against those preferences. The listing that fits best, wins.

LLMs have a hard time with this kind of planning. They have to reason over both the structure of a website as well as the user’s set of preferences (which may or may not be explicit). Most people visit the same websites over and over again, and at some level they have memorized the flow of the application. You might even be able to close your eyes and visualize the layout of a website you use often. Web agents built on LLMs can’t do that out of the box. Reasoning over the structure of an application is much harder if you don’t know the website ahead of time.

Out of the box, every time an LLM “sees” a new website, it’s seeing it for the first time. Try going to a website you’ve never visited, and perform a task. You’ll find that even for us humans, new websites can be unintuitive. So, when an agent comes to a new website, it needs to plan on the fly, with no prior knowledge – just like a human would if we’ve never seen the site before. This often will lead to sequences of actions that result in a dead end. This is where planning blends with search, and depending on the website this can become a major time sink. This is one of the problems we’ve been working on at rabbit, and you can see in LAM playground the agent may go down the wrong path. Ideally, it will recognize the error and find its way back to a better state.

The fact that LLMs don’t have dependable, baked-in prior knowledge about web interfaces means that they can’t complete tasks just like a human would. A plan for an agent must be under-specified from the outset, and then updated as the agent continues its task. The result is a plan that should have a high-level functional step-by-step outline. Once the agent sees a new page load, then it can start to map its rough plan onto actionable items (e.g. click the “Location” text box). Every new page yields new information (e.g. menus expand to reveal new information), and as such a good agent should update and adjust its plan constantly. As you can see, simple tasks for humans are not so simple for machines. So far we’ve only talked about planning, but the story continues when we get to grounding.

Grounding

Grounding is where the rubber hits the road. It’s where the agent’s plan turns into specific, concrete actions on a rendered webpage. Let’s say the task is: “Buy me a 12-pack of Diet Coke on Amazon.” The agent needs to make a plan without dependable prior knowledge of Amazon.com’s web interface. The agent can make a plan that turns the task conditions into a logical sequence: 1) go to amazon.com 2) search for a 12-pack of Diet Coke 3) add it to the cart 4) proceed to checkout 5) finalize the payment. When the agent sees Amazon.com’s homepage, it must translate “search for a 12-pack of Diet Coke” to “click on the search bar, type in ‘12-pack of Diet Coke’, hit enter”. The grounding problem involves finding the search bar, and engaging it with the webdriver.

Again, this is such a trivial task for humans that we don’t even think about it. When you click an element on a webpage, you just move the cursor over the element and click it. This is a basic motor task, but it involves a tight feedback loop between perception and movement. It’s not practical to implement web agents with this same approach. Rather, we aim to find the location of the search bar, and directly engage it. This can be done by either referencing the location of the search bar in pixels as it’s presented in the browser, or we can find the search bar in the underlying source code of the webpage. Both of these approaches are difficult, and this “finding” is the grounding problem. Researchers have investigated both these approaches. Combining the visual input (screenshot) with the underlying HTML structure has shown promising results. This is what we’ve also found works best, and is currently running in LAM playground.

As you can see, web-based tasks that are simple for humans are not so simple for machines. The agent must understand what a user wants, break the task down into conditions, turn those conditions into a plan, update the plan over time, and ground all actions on the webpage. The latest advances in LLMs have made great progress in planning and grounding, and at rabbit our goal is to utilize the best of what’s available and productionize a useful AI agent system and make it accessible to our users as early as possible.