Using BigQuery with Workers AI

Last updated Jan 29, 2026 The easiest way to get started with Workers AI is to try it out in the Multi-modal Playground ↗︎ and the LLM playground ↗︎. If you decide that you want to integrate your code with Workers AI, you may then decide to use its REST API endpoints or a Worker binding.

But what about the data? What if you want these models to ingest data that is stored outside Cloudflare?

In this tutorial, you will learn how to bring data from Google BigQuery to a Cloudflare Worker so that it can be used as input for Workers AI models.

Prerequisites

You will need:

1. Set up your Cloudflare Worker

To ingest the data into Cloudflare and feed it into Workers AI, you will be using a Cloudflare Worker. If you have not created one yet, please review our tutorial on how to get started.

After following the steps to create a Worker, you should have the following code in your new Worker project:

export default {
    async fetch(request, env, ctx) {
        return new Response("Hello World!");
    },
};

If the Worker project has successfully been created, you should also be able to run npx wrangler dev in a console to run the Worker locally:

[wrangler:inf] Ready on http://localhost:8787

Open a browser tab at http://localhost:8787/ to see your deployed Worker. Please note that the port 8787 may be a different one in your case.

You should be seeing Hello World! in your browser:

Hello World!

If you run into any issues during this step, please review the Worker's Get Started Guide.

2. Import GCP Service key into the Worker as Secrets

Now that you have verified that the Worker has been created successfully, you will need to reference the Google Cloud Platform service key created in the Prerequisites section of this tutorial.

Your downloaded key JSON file from Google Cloud Platform should have the following format:

{
    "type": "service_account",
    "project_id": "<your_project_id>",
    "private_key_id": "<your_private_key_id>",
    "private_key": "<your_private_key>",
    "client_email": "<your_service_account_id>@<your_project_id>.iam.gserviceaccount.com",
    "client_id": "<your_oauth2_client_id>",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/<your_service_account_id>%40<your_project_id>.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
}

For this tutorial, you will only need the values of the following fields: client_email, private_key, private_key_id, and project_id.

Instead of storing this information in plain text in the Worker, you will use Secrets to make sure its unencrypted content is only accessible via the Worker itself.

Import those three values from the JSON file into Secrets, starting with the field from the JSON key file called client_email, which we will now call BQ_CLIENT_EMAIL (you can use another variable name):

npx wrangler secret put BQ_CLIENT_EMAIL

You will be asked to enter a secret value, which will be the value of the field client_email in the JSON key file.

Note

Do not include any double quotes in the secret that you store, as it will already be interpreted as a string.

If the secret was uploaded successfully, the following message will be displayed:

✨ Success! Uploaded secret BQ_CLIENT_EMAIL

Now import the secrets for the three remaining fields; private_key, private_key_id, and project_id as BQ_PRIVATE_KEY, BQ_PRIVATE_KEY_ID, and BQ_PROJECT_ID respectively:

npx wrangler secret put BQ_PRIVATE_KEY

npx wrangler secret put BQ_PRIVATE_KEY_ID

npx wrangler secret put BQ_PROJECT_ID

At this point, you have successfully imported three fields from the JSON key file downloaded from Google Cloud Platform into Cloudflare Secrets to be used in a Worker.

Secrets are only made available to Workers once they are deployed. To make them available during development, create a .dev.vars file to locally store these credentials and reference them as environment variables.

Your dev.vars file should look like the following:

BQ_CLIENT_EMAIL="<your_service_account_id>@<your_project_id>.iam.gserviceaccount.com"
BQ_CLIENT_KEY="-----BEGIN PRIVATE KEY-----<content_of_your_private_key>-----END PRIVATE KEY-----\n"
BQ_PRIVATE_KEY_ID="<your_private_key_id>"
BQ_PROJECT_ID="<your_project_id>"

Make sure to include .dev.vars in your project .gitignore file to prevent your credentials being uploaded to a repository when using version control.

Check the secrets are loaded correctly in src/index.js by logging their values into a console output, as follows:

export default {
    async fetch(request, env, ctx) {
        console.log("BQ_CLIENT_EMAIL: ", env.BQ_CLIENT_EMAIL);
        console.log("BQ_PRIVATE_KEY: ", env.BQ_PRIVATE_KEY);
        console.log("BQ_PRIVATE_KEY_ID: ", env.BQ_PRIVATE_KEY_ID);
        console.log("BQ_PROJECT_ID: ", env.BQ_PROJECT_ID);
        return new Response("Hello World!");
    },
};

Restart the Worker and run npx wrangler dev. You should see that the server now mentions the newly added variables:

Using vars defined in .dev.vars
Your worker has access to the following bindings:
- Vars:
  - BQ_CLIENT_EMAIL: "(hidden)"
  - BQ_PRIVATE_KEY: "(hidden)"
  - BQ_PRIVATE_KEY_ID: "(hidden)"
  - BQ_PROJECT_ID: "(hidden)"
[wrangler:inf] Ready on http://localhost:8787

If you open http://localhost:8787 in your browser, you should see the values of the variables show up in your console where the npx wrangler dev command is running, while still seeing only the Hello World! text in the browser window.

You now have access to the GCP credentials from a Worker. Next, you will install a library to help with the creation of the JSON Web Token needed to interact with GCP's API.

3. Install library to handle JWT operations

To interact with BigQuery's REST API, you will need to generate a JSON Web Token ↗︎ to authenticate your requests using the credentials that you have loaded into Worker secrets in the previous step.

For this tutorial, you will be using the jose ↗︎ library for JWT-related operations. Install it by running the following command in a console:

npm i jose

To verify that the installation succeeded, you can run npm list, which lists all the installed packages, to check if the jose dependency has been added:

<project_name>@0.0.0
/<path_to_your_project>/<project_name>
├── @cloudflare/vitest-pool-workers@0.4.29
├── jose@5.9.2
├── vitest@1.5.0
└── wrangler@3.75.0

4. Generate JSON web token

Now that you have installed the jose library, it is time to import it and add a function to your code that generates a signed JSON Web Token (JWT):

import * as jose from 'jose';
...
const generateBQJWT = async (aCryptoKey, env) => {
const algorithm = "RS256";
const audience = "https://bigquery.googleapis.com/";
const expiryAt = (new Date().valueOf() / 1000);
    const privateKey = await jose.importPKCS8(env.BQ_PRIVATE_KEY, algorithm);

// Generate signed JSON Web Token (JWT)
    return new jose.SignJWT()
        .setProtectedHeader({
            typ: 'JWT',
            alg: algorithm,
            kid: env.BQ_PRIVATE_KEY_ID
        })
        .setIssuer(env.BQ_CLIENT_EMAIL)
        .setSubject(env.BQ_CLIENT_EMAIL)
        .setAudience(audience)
        .setExpirationTime(expiryAt)
        .setIssuedAt()
        .sign(privateKey)
}

export default {
    async fetch(request, env, ctx) {
       ...
// Create JWT to authenticate the BigQuery API call
        let bqJWT;
        try {
            bqJWT = await generateBQJWT(env);
        } catch (e) {
            return new Response('An error has occurred while generating the JWT', { status: 500 })
        }
    },
       ...
};

Now that you have created a JWT, it is time to do an API call to BigQuery to fetch some data.

5. Make authenticated requests to Google BigQuery

With the JWT token created in the previous step, issue an API request to BigQuery's API to retrieve data from a table.

You will now query the table that you created in BigQuery earlier in this tutorial. This example uses a sampled version of the Hacker News Corpus ↗︎ that was used under its MIT licence and uploaded to BigQuery.

const queryBQ = async (bqJWT, path) => {
    const bqEndpoint = `https://bigquery.googleapis.com${path}`
    // In this example, text is a field in the BigQuery table that is being queried (hn.news_sampled)
    const query = 'SELECT text FROM hn.news_sampled LIMIT 3';
    const response = await fetch(bqEndpoint, {
        method: "POST",
        body: JSON.stringify({
            "query": query
        }),
        headers: {
            Authorization: `Bearer ${bqJWT}`
        }
    })
    return response.json()
}
...
export default {
    async fetch(request, env, ctx) {
        ...
            let ticketInfo;
            try {
            ticketInfo = await queryBQ(bqJWT);
        } catch (e) {
            return new Response('An error has occurred while querying BQ', { status: 500 });
        }
    ...
    },
};

Having the raw row data from BigQuery means that you can now format it in a JSON-like style next.

6. Format results from the query

Now that you have retrieved the data from BigQuery, your BigQuery API response should look something like this:

{
    ...
    "schema": {
        "fields": [
            {
                "name": "title",
                "type": "STRING",
                "mode": "NULLABLE"
            },
            {
                "name": "text",
                "type": "STRING",
                "mode": "NULLABLE"
            }
        ]
    },
    ...
    "rows": [
        {
            "f": [
                {
                    "v": "<some_value>"
                },
                {
                    "v": "<some_value>"
                }
            ]
        },
        {
            "f": [
                {
                    "v": "<some_value>"
                },
                {
                    "v": "<some_value>"
                }
            ]
        },
        {
            "f": [
                {
                    "v": "<some_value>"
                },
                {
                    "v": "<some_value>"
                }
            ]
        }
    ],
    ...
}

This format may be difficult to read and work with when iterating through results. So you will now implement a function that maps the schema into each individual value, and the resulting output will be easier to read, as shown below. Each row corresponds to an object within an array.

[
    {
        title: "<some_value>",
        text: "<some_value>",
    },
    {
        title: "<some_value>",
        text: "<some_value>",
    },
    {
        title: "<some_value>",
        text: "<some_value>",
    },
];

Create a formatRows function that takes a number of rows and fields returned from the BigQuery response body and returns an array of results as objects with named fields.

```javascript const formatRows = (rowsWithoutFieldNames, fields) => { // Index to fieldName const fieldsByIndex = new Map();

// Load all fields by name and have their index in the array result as their key fields.forEach((field, index) => { fieldsByIndex.set(index, field.name) })

// Iterate through rows const rowsWithFieldNames = rowsWithoutFieldNames.map(row => { // Per each row represented by an array f, iterate through the unnamed values and find their field names by searching them in the fieldsByIndex. let newRow = {} row.f.forEach((field, index) => { const fieldName = fieldsByIndex.get(index); if (fieldName) { // For every field in a row, add them to newRow newRow = ({ ...newRow, [fieldName]: field.v }); } }) return newRow })