Introduction to data exports

In this section, we will cover how to export data from Botify.
To know when you should use exports instead of queries, see the related guide.

Definition

A data export corresponds to a BQL query that is encapsulated in an export job. This means you can export any data linked to a certain project using exports. The job adds metadata around the BQL query, that specifies the expected format, the target location, the number of lines to export, etc.

Querying the API interactively will return a limited number of rows, the export allows you to fetch a much larger number of rows, potentially your entire dataset.
When you have a working BQL query, adapting it for an export does not require a lot of manipulations. What you will need to do is:

• Change the URL to https://api.botify.com/v1/jobs
• Encapsulate the BQL query inside another payload
• Choose the format
• Choose the export location

A typical use case could be: at the end of each recurrent crawl, export metrics from your new crawl directly into your S3 bucket, that allows you manipulate it freely.

Available backends

• Direct download - Botify stores the exported file and provides you with a link to access it
• AWS S3
• AWS Redshift
• Google Cloud Storage
• Google BigQuery

Details and the usage of each backend can be found in the Backends and connectors section.

Available formats

• CSV
• JSON
• XML
• Sitemap

Details and available options for each format can be found in the Formatters section.

Available data

Any data you can query through a BQL query!