Moving from our Legacy Big Query Integration to our new Big Query Integration

A few years ago, we started looking at how we could export data from the platform to Google Big Query, to support clients looking to mash-up data and visualise their SEO data in Google Data Studio, PowerBI, Tableau and similar tools.

A number of SEO teams from forward-thinking SEO agencies and enterprises were the early guinea pigs, and helped us as we started building out a new database schema and sending keyword ranking and 'Share of Search' data to Big Query.

If you are one of these early adopters, then this article explains how you can now upgrade your existing Big Query platform data feed to our new and improved feed.

(If you are looking to integrate the Authoritas SEO platform with Big Query for the first time then STOP reading and please read this help guide instead).

What's new with our Big Query Integration? Why do I need to upgrade?

Well, here are four excellent reasons!
​
​1. New Data Sources

We have added a number of new data sources and tables. We now send the following SEO platform data across to Big Query, which makes it easier than ever to produce comprehensive SEO Reports.

• Keyword Ranking data (including tags)

• Share of Search data

• Google Search Console data (if you have added the integration)

• Google Analytics data (if you've added the integration)

• Website crawl data

• Backlink data
  ​
  ​Click here for more details of the datasets created.

   

2. Real-time data synchronisation between the platform and Big Query

This means that as you make changes in the platform they are queued and streamed to Big Query as quickly as possible. This means, you shouldn't have to keep flicking between each tool.

3. New Google Data Studio templates

To help you create insightful and compelling SEO reports, we've created a set of Google Data Studio templates that you can customise. We'll be releasing these to all our paying subscribers in June 2021.

4. This is our new standard Big Query data feed

We have settled on this as our new standard schema and as such any new developments or data sources we add in the future will be added here first.

For this reason, we'd like to migrate away from these early custom integrations and align everyone on the same standard feed. This will also allow us to move more quickly, as it requires less operational overhead going forward.

So, we'd be very grateful if you can move to the new feed as soon as possible.
(Eventually, we will deprecate the existing feeds).

How to migrate your existing Big Query setup to the new Big Query integration

It's relatively simple, and as long as you don't upload the same Big Query Project and Service Key to the platform, then you should be able to have a short period of parallel running before moving to the new structure.

​Step 1: Let us know that you are planning to move to the new structure.
​

Step 2: Create a separate Big Query project and Service Key.

Step 3: Upload this new key to the platform.
​
(Instructions on Steps 2 & 3 are here).

Step 4: Wait whilst we create the new tables in your new project and start the migration. The process will start automatically and how long it takes will depend on how many projects you have and how long you have had them live on the platform.
​
We will send across all data for all projects. Expect it to take up to 24 hours for the first time; thereafter the data should be kept in sync.

Step 5: Enable Billing
​

You must enable billing on your Google Big Query account for the selected project.
​
The integration will not work correctly or at all without billing enabled. We stream data to Big Query and this is not a feature on the free Big Query plan. Furthermore, data is deleted automatically by Google after 60 days on free tier accounts.
​
Please note, this does not necessarily mean you will start incurring additional costs straightaway as Google offers a reasonably generous free allowance.