This explains how to use the Authoritas keyword ranking API to capture AI Overviews
As you are no doubt aware, Google has now started to roll-out its generative AI powered search results called AI Overviews (AIOs) to more users.
Please note, the layout, design and queries that AIOs show for are very different to what we see in Google's Search Generative Experience opt-in beta. Whilst, we are keeping a watching brief on this beta experiment, we are currently tracking AIOs in the markets where Google has fully rolled them out to all users including non-logged in anonymised users.
Update: Latest news (Aug 15th 2024) - Google has finally rolled out AI Overviews to all users in the United States (e.g. Users no longer have to be logged in to their Google account to see AI results) and has announced it is rolling it out in 6 major markets.
So, AIOS are not fully rolled out to all users worldwide yet, but when it is you will see an AI generated summary and related links appearing at the top of the SERPs.
It's like a mini-SERP at the top of the SERPs and it's going to have a big impact on SEO visibility for everyone.
In my opinion, the introduction of AI Overviews will be the biggest shake up in the SEO world in the past two decades.
Here's some of our latest research to prove my point:
SGE Research on eCommerce terms.
SGE Research on Brand terms.
Needless to say, SEOs will definitely need to know:
AIO penetration rates - which keywords generate AI Overview citations
AIO penetration by keyword intent or category - what types of keywords/categories are AIOs more prevalent
AIO performance by domain and/or page - which domains and pages are ranking in AIOs, where are they ranking and how many times are they being cited
AIO snippets - what does the AI generated text snippet cover and how is this changing
Over the past few months we have been developing our APIs and platform to ensure we are ready to give our clients the most up-to-date view of the SERPs the moment AIO rolls out globally for all users. This tracking ability is now available for queries against Google in the United States and will eventually support all markets that Google supports.
If you just want to continue rank tracking as usual, then there are no breaking changes to the structure of the JSON and you can ignore the rest of this help page.
If you want to understand what the SERP looks like after Google injects the generative AI snippet and URLs into the top of the SERP then read on.
N/B:
Please bear in mind, Google is adopting a phased rollout and AIOs may only be available in certain languages and markets in the future - so, if you are unsure, please do check with us before wasting your queries by sending them to currently unsupported countries.
New Parameter to request AIO Jobs
There is a new option to request that we expand the SGE element and extract the generative snippet and ranking pages.
To request an AIO-enabled SERPs Job, simply add 'include_sge' in your SERPs API request JSON.
"include_sge": true
If this is requested, we make a SERPs request as usual and get the initial page of results ('universal_initial').
We then detect whether an AIO trigger (call-to-action) is present or whether the AIO is auto-generated (see image below).
If there is no AIO button or link to click, or, the AIO results are not auto-generated, then the SERP results will be paginated and parsed as usual and the new JSON objects will be empty.
If we detect an AIO trigger, we will click on it and wait until Google has injected the new generative AI snippet and URLs.
We will then extract and parse these new results into the JSON objects set out below and paginate accordingly until we reach the maximum number of results you have specified.
New JSON Objects for AIOs in the SERPs API Response
If we find generative results in the SERPs then the following objects will be populated, otherwise they will be empty.
These examples cover the main types of results we have seen so far in both the AIO and SGE deployments. It is highly likely Google will introduce new features and these objects may change and new ones will be added. We will endeavour to make any changes non-breaking wherever possible.
'generative_trigger' - This shows the type of AIO response we found. This is useful to know, as there are three different types (so far) and they take up differing amounts of space on the SERP and so will have different click rates.
'sub_type' will be set to 'auto', 'generate_button' or 'show_more'.
"generative_trigger": {"1": {"page_number": 1, "visible": true, "markup": 1, "sub_type": "generate_button", "above_the_fold": true, "description": null, "top_left": "230,301", "bottom_right": "1905,2013", "type": "generative_trigger", "title": null, "url": null, "content": "Chicago, Illinois has many places to get pizza, including casual spots, places with Chicago-style pizza, and other pizza places."}}
'generative' - This shows the pages ranking in the generative results. We try and identify different types of AIO results and sub_types and parse them according to how they are found on the SERPs page.
The default text links and citations are given a 'sub_type' of 'NULL'.
"generative": {"1": {"page_number": 1, "visible": true, "markup": 1, "sub_type": null, "above_the_fold": true, "description": null, "top_left": "1204,369", "bottom_right": "1330,587", "position": "right", "type": "generative", "title": "THE 10 BEST PIZZA DELIVERY in Chicago 2024 | Order Pizza Near ...", "url": "https://www.ubereats.com/category/chicago-il/pizza#:~:text=Pizza%20Delivery%20in%20Chicago%20%C2%B7%20Domino's%20Pizza,Grill%20(Clark%20St)%20%C2%B7%20Sbarro%20(333%20S."}}Here is a list of the current 'generative' sub_types we parse.
Below this list, we provide an example screenshot for clarity:
Carousels have a
'sub_type'of'carousels'.Citations have a
'sub_type'of'citations'.Listings have a
'sub_type'of'Listings'.Local AIO results have a
'sub_type'of'place'.Product listings have a
'sub_type'of'product'.Overlaid have a
'sub_type'of'overlaid'.
Carousels have a 'sub_type' of 'carousels' - see example below. This is usually a list of ~10 sites and only appears on mobile devices. These URLs seem to exactly match those displayed in the overlay.
Citations have a 'sub_type' of 'citations'.
Google has recently introduced small 'link' icons next to each paragraph of the AI Overview text snippet which show the pages that Google's Gemini model has cited as sources for the text. Clicking this icon will give you an overlay on mobile and desktop with the pages listed in a scrollable container. We capture the URLs that are cited in the generative AI snippet. You can see an example icon after the word 'soil' in the screenshot below.
Listings have a 'sub_type' of 'Listings' - they are the ranking pages that appear in the scrollable right-rail on Desktop devices. Note, this right-rail does not always appear.
The ranking pages are the same as the ones that appear on mobile devices in the overlay if you click on the mobile favicons (see example below).
Local AIO results have a 'sub_type' of 'place' - see examples below.
Please note, this example has only been seen in the wild in SGE and is not currently live in AIOs.
The JSON lists up to 5 (this is the most we've seen so far) local business listings, places or locations. Note, Google is not showing the URL of the business, but you can build a lookup table for known locations using Google's unique identifier for each local entity - see 'cid'. Of course, this will likely change when Google rolls out local AI Overviews.
Product listings have a 'sub_type' of 'product' - see examples below.
There are a variety of product lists, grids and carousels that appear within the main AIO snippet. These lists vary in size from 3 rows and 1 column, to a 1 x 3 (carousel pack), 3 x 3 grid, 1 x 10 grid and 5 x 5 grids.
As they are part of the main AIO snippet and are often topped and tailed with text and citations, you will find them in the 'generative' object with a 'sub_type' of 'product' (see examples below).
This is likely to change as Google iterates its designs for integrating generative AI into Search.
Overlaid have a 'sub_type' of 'overlaid' - see examples below. At the moment, these appear only on mobile, where click the Favicons produces an overlay of the top ~10 or so pages ranking in AIOs.
Clicking on the Favicons generates this overlay:
'generative_follow_up_question' - contains a list of up to 3 frequently asked questions.
"generative_follow_up_question": {"1": {"page_number": 1, "visible": true, "markup": 1, "above_the_fold": false, "description": null, "top_left": "230,1868", "bottom_right": "1330,1914", "type": "generative_follow_up_question", "title": null, "items": {"1": {"title": "Which pizza place offers the best deals?"}, "2": {"title": "Which pizza place has the best Chicago-style pizza?"}, "3": {"title": "What are some affordable pizza places in Chicago?"}}, "url": null}}
'univeral_initial' - this reflects the state of the SERP prior to the expansion of the AIO element and only appears for AIO-enabled jobs.
'universal' - this reflects the state of the SERP for standard non AIO-enabled jobs and after the expansion of the AIO element for AIO-enabled jobs. The JSON is structured the same way as the 'universal_initial' pictured above.
All other objects - reflect the state of the SERP for standard non AIO-enabled jobs and after the expansion of the AIO element for AIO-enabled jobs.
We have also added a few extra data points that will be helpful for analysis and optimisation:
N/B: 'sub_type' will be set to 'null' (for standard text listings and unidentified sub-types) or 'product' or 'place'.
'position' will be set to 'left', 'right'.
Whilst not new, these key value pairs will be useful for further SEO analysis:
'visible' will be set to 'true', 'false'.
'top_left' shows the x,y coordinates for the item and it will be useful to understand how big the SGE results are and how far down the page your organic results have been pushed.
Order of AIO Results for Ranking Purposes
For years we have utilised the x,y coordinates ('top_left', 'bottom_right') to determine which organic results are ranking highest with the URL closest to the to left (in left-to-right reading languages) ranking in position 1.
As a general rule this does work well, and where possible we follow this rule when considering the ranking position of a particular URL for a particular SERP feature type.
Please note, that Google's current designs for AIOs makes this harder to calculate as the designs on mobile and desktop both have interactions which can trigger elements to overlap existing elements in the same position on the page!
So, when calculating a URLs appearance in the generative AIO results, we first count the top right-hand-rail results from 1 to 10 (if there are 10). So far, these are always unique pages and seem to be identical and in the same order on mobile and desktop results. We then count the pages being cited in order as we navigate down the page with AIO elements closer to the top and left ranked highest.
Hopefully, you agree with this approach and our ranking model. If you see this differently then you should be easily able to calculate your own custom AIO ranking model. With the combination of generative sub_types and position (right or left) you should have no problem determining which pages you want to consider as the top ranking sites for a given keyword and then excluding or including certain results and renumbering accordingly.
We certainly welcome any thoughts or feedback you have on our ranking model and recognise that Google is likely to change their designs again which may cause us to review this further in the future.
SERPs API Billing & Reporting Changes
The new AIO results require more effort and resources to get, as such they will be charged at 2x the cost of a standard query. This will be reflected in your usage and billing emails and invoices.
We will review the costs associated with providing AIO results in response to demand and supply considerations (like reliability and consistency of results) and in light of how Google eventually rolls this out to all users in different markets and languages.
It's possible that Google may eventually roll out AIO automatically in the SERPs for all queries, so in time the need for this additional parameter and the additional work and costs may eventually be unnecessary.
AIO - Initially only available on Standard Priority
Initially, AIO results will only be available on the 'Standard' priority jobs. We do plan to add support for higher priority jobs post-launch and will advise you of this in due course.
Tips:
We have noticed that search engines treats requests for 100 results with greater suspicion these days, as it seems only SEOs paginate or scroll this far. We expect this will continue to be the case for the new generative SERPs. We therefore recommend setting the maximum number of results to a number between 30 and 50 for all SERPs API requests. This is likely to make your requests look less like a bot and more like a human - and decreases the chance of your requests bots getting blocked by the search engines.
When an AIO is clicked the organic results are just pushed down the page. So far we have not observed any changes in the make-up or order of the ranking organic URLs. So, rather than make two separate requests for the same keyword with AIO enabled and disabled. You could make just one request with AIO enabled and then compare the
'universal_intial'with type'organic'against the'universal'results with type'organic'to see how far the top ranking organic results have fallen using differences in the y position extracted from the'top_left'x, y coordinates. Please note, the'universal_initial'only contains the first page Google shows us, so it will not have a complete list of all organic results. To see the complete list use'universal'with type set to'organic'.
FAQs:
Q. What happens if AIO is auto-generated for a keyword and the client has not set the parameter for us to click on the AIO trigger?
A. There will be no 'universal_initial', we will not expand the AIO elements and will only capture generative content that's visible without user interaction on the initial page or in the HTML.
Footnote:
Please note that AIO is being rolled out in phases and the types of results and content displayed is changing quite often. We will update this page regularly to reflect changes we observe. If any of the data here is out of date then please do let us know.
Comments
0 comments
Article is closed for comments.