Format	File extensions	Mime Types
PDF Documents	`.pdf`	`application/pdf`
Images ¹	`.jpeg`, `.jpg`, `.png`, `.webp`, `.svg`	`image/jpeg`, `image/png`, `image/webp`, `image/svg+xml`
HTML Documents	`.html`	`text/html`
XML Documents	`.xml`	`application/xml`
Microsoft Office Documents	`.xlsx`, `.xlsm`, `.xlsb`, `.xls`, `.et`	`application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`, `application/vnd.ms-excel.sheet.macroenabled.12`, `application/vnd.ms-excel.sheet.binary.macroenabled.12`, `application/vnd.ms-excel`, `application/vnd.ms-excel`
Open Document Format	`.ods`	`application/vnd.oasis.opendocument.spreadsheet`
CSV	`.csv`	`text/csv`
Apple Documents	`.numbers`	`application/vnd.apple.numbers`

Operation	Description
CreateDatabase	Creation of a new database.
DeleteDatabase	Deletion of an existing database.
TimeTravel	Restoration of a past database version.

; return (

{post.title}

{post.text}

Published {new Date(post.published_at).toLocaleString()}

Go back

); }; export default Post; ``` ## 2. Build your API You will now create a Pages Functions that stores your blog content and retrieves it via a JSON API. ### Write your Pages Function To create the Pages Function that will act as your JSON API: 1. Create a `functions` directory in your `blog-frontend` directory. 2. In `functions`, create a directory named `api`. 3. In `api`, create a `posts.js` file in the `api` directory. 4. Populate `posts.js` with the following code: ```js import posts from "./post/data"; export function onRequestGet() { return Response.json(posts); } ``` This code gets blog data (from `data.js`, which you will make in step 8) and returns it as a JSON response from the path `/api/posts`. 5. In the `api` directory, create a directory named `post`. 6. In the `post` directory, create a `data.js` file. 7. Populate `data.js` with the following code. This is where your blog content, blog title, and other information about your blog lives. ```js const posts = [ { id: 1, title: "My first blog post", text: "Hello world! This is my first blog post on my new Cloudflare Workers + Pages blog.", published_at: new Date("2020-10-23"), }, { id: 2, title: "Updating my blog", text: "It's my second blog post! I'm still writing and publishing using Cloudflare Workers + Pages :)", published_at: new Date("2020-10-26"), }, ]; export default posts; ``` 8. In the `post` directory, create an `[[id]].js` file. 9. Populate `[[id]].js` with the following code: ```js title="[[id]].js" import posts from "./data"; export function onRequestGet(context) { const id = context.params.id; if (!id) { return new Response("Not found", { status: 404 }); } const post = posts.find((post) => post.id === Number(id)); if (!post) { return new Response("Not found", { status: 404 }); } return Response.json(post); } ``` `[[id]].js` is a [dynamic route](/pages/functions/routing#dynamic-routes) which is used to accept a blog post `id`. ## 3. Deploy After you have configured your Pages application and Pages Function, deploy your project using the Wrangler or via the dashboard. ### Deploy with Wrangler In your `blog-frontend` directory, run [`wrangler pages deploy`](/workers/wrangler/commands/#deploy-1) to deploy your project to the Cloudflare dashboard. ```sh wrangler pages deploy blog-frontend ``` ### Deploy via the dashboard To deploy via the Cloudflare dashboard, you will need to create a new Git repository for your Pages project and connect your Git repository to Cloudflare. This tutorial uses GitHub as its Git provider. #### Create a new repository Create a new GitHub repository by visiting [repo.new](https://repo.new). After creating a new repository, prepare and push your local application to GitHub by running the following commands in your terminal: ```sh git init git remote add origin https://github.com// git add . git commit -m "Initial commit" git branch -M main git push -u origin main ``` #### Deploy with Cloudflare Pages Deploy your application to Pages: 1. Log in to the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account. 2. In Account Home, select **Workers & Pages** > **Create application** > **Pages** > **Connect to Git**. 3. Select the new GitHub repository that you created and, in the **Set up builds and deployments** section, provide the following information:

| Configuration option | Value | | -------------------- | --------------- | | Production branch | `main` | | Build command | `npm run build` | | Build directory | `build` |

After configuring your site, begin your first deploy. You should see Cloudflare Pages installing `blog-frontend`, your project dependencies, and building your site. By completing this tutorial, you have created a full-stack Pages application. ## Related resources - Learn about [Pages Functions routing](/pages/functions/routing) --- # Create a HTML form URL: https://developers.cloudflare.com/pages/tutorials/forms/ In this tutorial, you will create a simple `

` using plain HTML and CSS and deploy it to Cloudflare Pages. While doing so, you will learn about some of the HTML form attributes and how to collect submitted data within a Worker. :::note[MDN Introductory Series] This tutorial will briefly touch upon the basics of HTML forms. For a more in-depth overview, refer to MDN's [Web Forms – Working with user data](https://developer.mozilla.org/en-US/docs/Learn/Forms) introductory series. ::: This tutorial will make heavy use of Cloudflare Pages and [its Workers integration](/pages/functions/). Refer to the [Get started guide](/pages/get-started/) guide to familiarize yourself with the platform. ## Overview On the web, forms are a common point of interaction between the user and the web document. They allow a user to enter data and, generally, submit their data to a server. A form is comprised of at least one form input, which can vary from text fields to dropdowns to checkboxes and more. Each input should be named – using the [`name`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#attr-name) attribute – so that the input's value has an identifiable name when received by the server. Additionally, with the advancement of HTML5, form elements may declare additional attributes to opt into automatic form validation. The available validations vary by input type; for example, a text input that accepts emails (via [`type=email`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#input_types)) can ensure that the value looks like a valid email address, a number input (via `type=number`) will only accept integers or decimal values (if allowed), and generic text inputs can define a custom [`pattern`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#attr-pattern) to allow. However, all inputs can declare whether or not a value is [`required`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input#attr-required). Below is an example HTML5 form with a few inputs and their validation rules defined: ```html

``` If an HTML5 form has validation rules defined, browsers will automatically check all rules when the user attempts to submit the form. Should there be any errors, the submission is prevented and the browser displays the error message(s) to the user for correction. The `

` will only `POST` data to the `/submit` endpoint when there are no outstanding validation errors. This entire process is native to HTML5 and only requires the appropriate form and input attributes to exist — no JavaScript is required. Form elements may also have a [``](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/label) element associated with them, allowing you to clearly describe each input. This is great for visual clarity, of course, but it also allows for more accessible user experiences since the HTML markup is more well-defined. Assistive technologies directly benefit from this; for example, screen readers can announce which `` is focused. And when a `` is clicked, its assigned form input is focused instead, increasing the activation area for the input. To enable this, you must create a `` element for each input and assign each `` element and unique `id` attribute value. The `` must also possess a [`for`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/label#attr-for) attribute that reflects its input's unique `id` value. Amending the previous snippet should produce the following: ```html Full Name Email Address Your Age ``` :::note Your `for` and `id` values do not need to exactly match the values shown above. You may use any `id` values so long as they are unique to the HTML document. A `` can only be linked with an `` if the `for` and `id` attributes match. ::: When this `

` is submitted with valid data, its data contents are sent to the server. You may customize how and where this data is sent by declaring attributes on the form itself. If you do not provide these details, the `` will GET the data to the current URL address, which is rarely the desired behavior. To fix this, at minimum, you need to define an [`action`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form#attr-action) attribute with the target URL address, but declaring a [`method`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form#attr-method) is often recommended too, even if you are redeclaring the default `GET` value. By default, HTML forms send their contents in the `application/x-www-form-urlencoded` MIME type. This value will be reflected in the `Content-Type` HTTP header, which the receiving server must read to determine how to parse the data contents. You may customize the MIME type through the [`enctype`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/form#attr-enctype) attribute. For example, to accept files (via `type=file`), you must change the `enctype` to the `multipart/form-data` value: ```html Full Name Email Address Your Age Profile Picture

``` Because the `enctype` changed, the browser changes how it sends data to the server too. The `Content-Type` HTTP header will reflect the new approach and the HTTP request's body will conform to the new MIME type. The receiving server must accommodate the new format and adjust its request parsing method. ## Live example The rest of this tutorial will focus on building an HTML form on Pages, including a Worker to receive and parse the form submissions. :::note[GitHub Repository] The source code for this example is [available on GitHub](https://github.com/cloudflare/submit.pages.dev). It is a live Pages application with a [live demo](https://submit.pages.dev/) available, too. ::: ### Setup To begin, create a [new GitHub repository](https://repo.new/). Then create a new local directory on your machine, initialize git, and attach the GitHub location as a remote destination: ```sh # create new directory mkdir new-project # enter new directory cd new-project # initialize git git init # attach remote git remote add origin git@github.com:/.git # change default branch name git branch -M main ``` You may now begin working in the `new-project` directory you created. ### Markup The form for this example is fairly straightforward. It includes an array of different input types, including checkboxes for selecting multiple values. The form also does not include any validations so that you may see how empty and/or missing values are interpreted on the server. You will only be using plain HTML for this example project. You may use your preferred JavaScript framework, but raw languages have been chosen for simplicity and familiarity – all frameworks are abstracting and/or producing a similar result. Create a `public/index.html` in your project directory. All front-end assets will exist within this `public` directory and this `index.html` file will serve as the home page for the website. Copy and paste the following content into your `public/index.html` file: ```html Form Demo ``` This HTML document will contain a form with a few fields for the user to fill out. Because there is no validation rules within the form, all fields are optional and the user is able to submit an empty form. For this example, this is intended behavior. :::note[Optional content] Technically, only the `

` and its child elements are necessary. The `` and the enclosing `` and `` tags are optional and not strictly necessary for a valid HTML document. The HTML page is also completely unstyled at this point, relying on the browsers' default UI and color palettes. Styling the page is entirely optional and not necessary for the form to function. If you would like to attach a CSS stylesheet, you may [add a `` element](https://developer.mozilla.org/en-US/docs/Learn/CSS/First_steps/Getting_started#adding_css_to_our_document). Refer to the finished tutorial's [source code](https://github.com/cloudflare/submit.pages.dev/blob/8c0594f48681935c268987f2f08bcf3726a74c57/public/index.html#L11) for an example or any inspiration – the only requirement is that your CSS stylesheet also resides within the `public` directory. ::: ### Worker The HTML form is complete and ready for deployment. When the user submits this form, all data will be sent in a `POST` request to the `/api/submit` URL. This is due to the form's `method` and `action` attributes. However, there is currently no request handler at the `/api/submit` address. You will now create it. Cloudflare Pages offers a [Functions](/pages/functions/) feature, which allows you to define and deploy Workers for dynamic behaviors. Functions are linked to the `functions` directory and conveniently construct URL request handlers in relation to the `functions` file structure. For example, the `functions/about.js` file will map to the `/about` URL and `functions/hello/[name].js` will handle the `/hello/:name` URL pattern, where `:name` is any matching URL segment. Refer to the [Functions routing](/pages/functions/routing/) documentation for more information. To define a handler for `/api/submit`, you must create a `functions/api/submit.js` file. This means that your `functions` and `public` directories should be siblings, with a total project structure similar to the following: ```txt ├── functions │ └── api │ └── submit.js └── public └── index.html ``` The `` will send `POST` requests, which means that the `functions/api/submit.js` file needs to export an `onRequestPost` handler: ```js /** * POST /api/submit */ export async function onRequestPost(context) { // TODO: Handle the form submission } ``` The `context` parameter is an object filled with several values of potential interest. For this example, you only need the [`Request`](/workers/runtime-apis/request/) object, which can be accessed through the `context.request` key. As mentioned, a `` defaults to the `application/x-www-form-urlencoded` MIME type when submitting. And, for more advanced scenarios, the `enctype="multipart/form-data"` attribute is needed. Luckily, both MIME types can be parsed and treated as [`FormData`](https://developer.mozilla.org/en-US/docs/Web/API/FormData). This means that with Workers – which includes Pages Functions – you are able to use the native [`Request.formData`](https://developer.mozilla.org/en-US/docs/Web/API/Request/formData) parser. For illustrative purposes, the example application's form handler will reply with all values it received. A `Response` must always be returned by the handler, too: ```js /** * POST /api/submit */ export async function onRequestPost(context) { try { let input = await context.request.formData(); let pretty = JSON.stringify([...input], null, 2); return new Response(pretty, { headers: { "Content-Type": "application/json;charset=utf-8", }, }); } catch (err) { return new Response("Error parsing JSON content", { status: 400 }); } } ``` With this handler in place, the example is now fully functional. When a submission is received, the Worker will reply with a JSON list of the `FormData` key-value pairs. However, if you want to reply with a JSON object instead of the key-value pairs (an Array of Arrays), then you must do so manually. Recently, JavaScript added the [`Object.fromEntries`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/fromEntries) utility. This works well in some cases; however, the example `` includes a `movies` checklist that allows for multiple values. If using `Object.fromEntries`, the generated object would only keep one of the `movies` values, discarding the rest. To avoid this, you must write your own `FormData` to `Object` utility instead: ```js /** * POST /api/submit */ export async function onRequestPost(context) { try { let input = await context.request.formData(); // Convert FormData to JSON // NOTE: Allows multiple values per key let output = {}; for (let [key, value] of input) { let tmp = output[key]; if (tmp === undefined) { output[key] = value; } else { output[key] = [].concat(tmp, value); } } let pretty = JSON.stringify(output, null, 2); return new Response(pretty, { headers: { "Content-Type": "application/json;charset=utf-8", }, }); } catch (err) { return new Response("Error parsing JSON content", { status: 400 }); } } ``` The final snippet (above) allows the Worker to retain all values, returning a JSON response with an accurate representation of the `` submission. ### Deployment You are now ready to deploy your project. If you have not already done so, save your progress within `git` and then push the commit(s) to the GitHub repository: ```sh # Add all files git add -A # Commit w/ message git commit -m "working example" # Push commit(s) to remote git push -u origin main ``` Your work now resides within the GitHub repository, which means that Pages is able to access it too. If this is your first Cloudflare Pages project, refer to the [Get started guide](/pages/get-started/) for a complete walkthrough. After selecting the appropriate GitHub repository, you must configure your project with the following build settings: - **Project name** – Your choice - **Production branch** – `main` - **Framework preset** – None - **Build command** – None / Empty - **Build output directory** – `public` After clicking the **Save and Deploy** button, your Pages project will begin its first deployment. When successful, you will be presented with a unique `*.pages.dev` subdomain and a link to your live demo. In this tutorial, you built and deployed a website and its back-end logic using Cloudflare Pages with its Workers integration. You created a static HTML document with a form that communicates with a Worker handler to parse the submission request(s). If you would like to review the full source code for this application, you can find it on [GitHub](https://github.com/cloudflare/submit.pages.dev). ## Related resources - [Build an API for your front end using Cloudflare Workers](/pages/tutorials/build-an-api-with-pages-functions/) - [Handle form submissions with Airtable](/workers/tutorials/handle-form-submissions-with-airtable/) --- # Localize a website with HTMLRewriter URL: https://developers.cloudflare.com/pages/tutorials/localize-a-website/ import { Render, PackageManagers, WranglerConfig } from "~/components"; In this tutorial, you will build an example internationalization and localization engine (commonly referred to as **i18n** and **l10n**) for your application, serve the content of your site, and automatically translate the content based on your visitors’ location in the world. This tutorial uses the [`HTMLRewriter`](/workers/runtime-apis/html-rewriter/) class built into the Cloudflare Workers runtime, which allows for parsing and rewriting of HTML on the Cloudflare global network. This gives developers the ability to efficiently and transparently customize their Workers applications. ![An example site that has been successfully localized in Japanese, German and English](~/assets/images/workers/tutorials/localize-website/i18n.jpg) --- ## Prerequisites This tutorial is designed to use an existing website. To simplify this process, you will use a free HTML5 template from [HTML5 UP](https://html5up.net). With this website as the base, you will use the `HTMLRewriter` functionality in the Workers platform to overlay an i18n layer, automatically translating the site based on the user’s language. If you would like to deploy your own version of the site, you can find the source [on GitHub](https://github.com/lauragift21/i18n-example-workers). Instructions on how to deploy this application can be found in the project’s README. ## Create a new application Create a new application using the [`create-cloudflare`](/pages/get-started/c3), a CLI for creating and deploying new applications to Cloudflare. For setup, select the following options: - For _What would you like to start with_?, select `Framework Starter`. - For _Which development framework do you want to use?_, select `React`. - For, _Do you want to deploy your application?_, select `No`. The newly generated `i18n-example` project will contain two folders: `public` and `src` these contain files for a React application: ```sh cd i18n-example ls ``` ```sh output public src package.json ``` We have to make a few adjustments to the generated project, first we want to the replace the content inside of the `public` directory, with the default generated HTML code for the HTML5 UP template seen in the demo screenshot: download a [release](https://github.com/signalnerve/i18n-example-workers/archive/v1.0.zip) (ZIP file) of the code for this project and copy the `public` folder to your own project to get started. Next, let's create a functions directory with an `index.js` file, this will be where the logic of the application will be written. ```sh mkdir functions cd functions touch index.js ``` Additionally, we'll remove the `src/` directory since its content isn't necessary for this project. With the static HTML for this project updated, you can focus on the script inside of the `functions` folder, at `index.js`. ## Understanding `data-i18n-key` The `HTMLRewriter` class provided in the Workers runtime allows developers to parse HTML and write JavaScript to query and transform every element of the page. The example website in this tutorial is a basic single-page HTML project that lives in the `public` directory. It includes an `h1` element with the text `Example Site` and a number of `p` elements with different text: ![Demo code shown in Chrome DevTools with the elements described above](~/assets/images/workers/tutorials/localize-website/code-example.png) What is unique about this page is the addition of [data attributes](https://developer.mozilla.org/en-US/docs/Learn/HTML/Howto/Use_data_attributes) in the HTML – custom attributes defined on a number of elements on this page. The `data-i18n-key` on the `h1` tag on this page, as well as many of the `p` tags, indicates that there is a corresponding internationalization key, which should be used to look up a translation for this text: ```html

Example Site

This is my example site. Depending o...

Disclaimer: the initial translations...

``` Using `HTMLRewriter`, you will parse the HTML within the `./public/index.html` page. When a `data-i18n-key` attribute is found, you should use the attribute's value to retrieve a matching translation from the `strings` object. With `HTMLRewriter`, you can query elements to accomplish tasks like finding a data attribute. However, as the name suggests, you can also rewrite elements by taking a translated string and directly inserting it into the HTML. Another feature of this project is based on the `Accept-Language` header, which exists on incoming requests. You can set the translation language per request, allowing users from around the world to see a locally relevant and translated page. ## Using the HTML Rewriter API Begin with the `functions/index.js` file. Your application in this tutorial will live entirely in this file. Inside of this file, start by adding the default code for running a [Pages Function](/pages/functions/get-started/#create-a-function). ```js export function onRequest(context) { return new Response("Hello, world!"); } ``` The important part of the code lives in the `onRequest` function. To implement translations on the site, take the HTML response retrieved from `env.ASSETS.fetch(request)` this allows you to fetch a static asset from your Pages project and pass it into a new instance of `HTMLRewriter`. When instantiating `HTMLRewriter`, you can attach handlers using the `on` function. For this tutorial, you will use the `[data-i18n-key]` selector (refer to the [HTMLRewriter documentation](/workers/runtime-apis/html-rewriter/) for more advanced usage) to locate all elements with the `data-i18n-key` attribute, which means that they must be translated. Any matching element will be passed to an instance of your `ElementHandler` class, which will contain the translation logic. With the created instance of `HTMLRewriter`, the `transform` function takes a `response` and can be returned to the client: ```js export async function onRequest(context) { const { request, env } = context; const response = await env.ASSETS.fetch(request); return new HTMLRewriter() .on("[data-i18n-key]", new ElementHandler(countryStrings)) .transform(response); } ``` ## Transforming HTML Your `ElementHandler` will receive every element parsed by the `HTMLRewriter` instance, and due to the expressive API, you can query each incoming element for information. In [How it works](#understanding-data-i18n-key), the documentation describes `data-i18n-key`, a custom data attribute that could be used to find a corresponding translated string for the website’s user interface. In `ElementHandler`, you can define an `element` function, which will be called as each element is parsed. Inside of the `element` function, you can query for the custom data attribute using `getAttribute`: ```js class ElementHandler { element(element) { const i18nKey = element.getAttribute("data-i18n-key"); } } ``` With `i18nKey` defined, you can use it to search for a corresponding translated string. You will now set up a `strings` object with key-value pairs corresponding to the `data-i18n-key` value. For now, you will define a single example string, `headline`, with a German `string`, `"Beispielseite"` (`"Example Site"`), and retrieve it in the `element` function: ```js null {1,2,3,4,5,10} const strings = { headline: "Beispielseite", }; class ElementHandler { element(element) { const i18nKey = element.getAttribute("data-i18n-key"); const string = strings[i18nKey]; } } ``` Take your translated `string` and insert it into the original element, using the `setInnerContent` function: ```js null {11,12,13} const strings = { headline: "Beispielseite", }; class ElementHandler { element(element) { const i18nKey = element.getAttribute("data-i18n-key"); const string = strings[i18nKey]; if (string) { element.setInnerContent(string); } } } ``` To review that everything looks as expected, use the preview functionality built into Wrangler. Call [`wrangler pages dev ./public`](/workers/wrangler/commands/#dev) to open up a live preview of your project. The command is refreshed after every code change that you make. You can expand on this translation functionality to provide country-specific translations, based on the incoming request’s `Accept-Language` header. By taking this header, parsing it, and passing the parsed language into your `ElementHandler`, you can retrieve a translated string in your user’s home language, provided that it is defined in `strings`. To implement this: 1. Update the `strings` object, adding a second layer of key-value pairs and allowing strings to be looked up in the format `strings[country][key]`. 2. Pass a `countryStrings` object into our `ElementHandler`, so that it can be used during the parsing process. 3. Grab the `Accept-Language` header from an incoming request, parse it, and pass the parsed language to `ElementHandler`. To parse the `Accept-Language` header, install the [`accept-language-parser`](https://www.npmjs.com/package/accept-language-parser) npm package: ```sh npm i accept-language-parser ``` Once imported into your code, use the package to parse the most relevant language for a client based on `Accept-Language` header, and pass it to `ElementHandler`. Your final code for the project, with an included sample translation for Germany and Japan (using Google Translate) looks like this: ```js null {32,33,34,39,62,63,64,65} import parser from "accept-language-parser"; // do not set to true in production! const DEBUG = false; const strings = { de: { title: "Beispielseite", headline: "Beispielseite", subtitle: "Dies ist meine Beispielseite. Abhängig davon, wo auf der Welt Sie diese Site besuchen, wird dieser Text in die entsprechende Sprache übersetzt.", disclaimer: "Haftungsausschluss: Die anfänglichen Übersetzungen stammen von Google Translate, daher sind sie möglicherweise nicht perfekt!", tutorial: "Das Tutorial für dieses Projekt finden Sie in der Cloudflare Workers-Dokumentation.", copyright: "Design von HTML5 UP.", }, ja: { title: "サンプルサイト", headline: "サンプルサイト", subtitle: "これは私の例のサイトです。このサイトにアクセスする世界の場所に応じて、このテキストは対応する言語に翻訳されます。", disclaimer: "免責事項：最初の翻訳はGoogle翻訳からのものですので、完璧ではないかもしれません！", tutorial: "Cloudflare Workersのドキュメントでこのプロジェクトのチュートリアルを見つけてください。", copyright: "HTML5 UPによる設計。", }, }; class ElementHandler { constructor(countryStrings) { this.countryStrings = countryStrings; } element(element) { const i18nKey = element.getAttribute("data-i18n-key"); if (i18nKey) { const translation = this.countryStrings[i18nKey]; if (translation) { element.setInnerContent(translation); } } } } export async function onRequest(context) { const { request, env } = context; try { let options = {}; if (DEBUG) { options = { cacheControl: { bypassCache: true, }, }; } const languageHeader = request.headers.get("Accept-Language"); const language = parser.pick(["de", "ja"], languageHeader); const countryStrings = strings[language] || {}; const response = await env.ASSETS.fetch(request); return new HTMLRewriter() .on("[data-i18n-key]", new ElementHandler(countryStrings)) .transform(response); } catch (e) { if (DEBUG) { return new Response(e.message || e.toString(), { status: 404, }); } else { return env.ASSETS.fetch(request); } } } ``` ## Deploy Your i18n tool built on Cloudflare Pages is complete and it is time to deploy it to your domain. To deploy your application to a `*.pages.dev` subdomain, you need to specify a directory of static assets to serve, configure the `pages_build_output_dir` in your project’s Wrangler file and set the value to `./public`: ```toml null {2} name = "i18n-example" pages_build_output_dir = "./public" compatibility_date = "2024-01-29" ``` Next, you need to configure a deploy script in `package.json` file in your project. Add a deploy script with the value `wrangler pages deploy`: ```json null {3} "scripts": { "dev": "wrangler pages dev", "deploy": "wrangler pages deploy" } ``` Using `wrangler`, deploy to Cloudflare’s network, using the `deploy` command: ```sh npm run deploy ``` ![An example site that has been successfully localized in Japanese, German and English](~/assets/images/workers/tutorials/localize-website/i18n.jpg) ## Related resources In this tutorial, you built and deployed an i18n tool using `HTMLRewriter`. To review the full source code for this application, refer to the [repository on GitHub](https://github.com/lauragift21/i18n-example-workers). If you want to get started building your own projects, review the existing list of [Quickstart templates](/workers/get-started/quickstarts/). --- # Use R2 as static asset storage with Cloudflare Pages URL: https://developers.cloudflare.com/pages/tutorials/use-r2-as-static-asset-storage-for-pages/ import { WranglerConfig } from "~/components"; This tutorial will teach you how to use [R2](/r2/) as a static asset storage bucket for your [Pages](/pages/) app. This is especially helpful if you're hitting the [file limit](/pages/platform/limits/#files) or the [max file size limit](/pages/platform/limits/#file-size) on Pages. To illustrate how this is done, we will use R2 as a static asset storage for a fictional cat blog. ## The Cat blog Imagine you run a static cat blog containing funny cat videos and helpful tips for cat owners. Your blog is growing and you need to add more content with cat images and videos. The blog is hosted on Pages and currently has the following directory structure: ``` . ├── public │ ├── index.html │ ├── static │ │ ├── favicon.ico │ │ └── logo.png │ └── style.css └── wrangler.toml ``` Adding more videos and images to the blog would be great, but our asset size is above the [file limit on Pages](/pages/platform/limits/#file-size). Let us fix this with R2. ## Create an R2 bucket The first step is creating an R2 bucket to store the static assets. A new bucket can be created with the dashboard or via Wrangler. Using the dashboard, navigate to the R2 tab, then click on *Create bucket.* We will name the bucket for our blog _cat-media_. Always remember to give your buckets descriptive names: ![Dashboard](~/assets/images/pages/tutorials/pages-r2/dash.png) With the bucket created, we can upload media files to R2. I’ll drag and drop two folders with a few cat images and videos into the R2 bucket: ![Upload](/images/pages/tutorials/pages-r2/upload.gif) Alternatively, an R2 bucket can be created with Wrangler from the command line by running: ```sh npx wrangler r2 bucket create # i.e # npx wrangler r2 bucket create cat-media ``` Files can be uploaded to the bucket with the following command: ```sh npx wrangler r2 object put / -f # i.e # npx wrangler r2 object put cat-media/videos/video1.mp4 -f ~/Downloads/videos/video1.mp4 ``` ## Bind R2 to Pages To bind the R2 bucket we have created to the cat blog, we need to update the Wrangler configuration. Open the [Wrangler configuration file](/pages/functions/wrangler-configuration/), and add the following binding to the file. `bucket_name` should be the exact name of the bucket created earlier, while `binding` can be any custom name referring to the R2 resource: ```toml [[r2_buckets]] binding = "MEDIA" bucket_name = "cat-media" ``` :::note Note: The keyword `ASSETS` is reserved and cannot be used as a resource binding. ::: Save the [Wrangler configuration file](/pages/functions/wrangler-configuration/), and we are ready to move on to the last step. Alternatively, you can add a binding to your Pages project on the dashboard by navigating to the project’s _Settings_ tab > _Functions_ > _R2 bucket bindings_. ## Serve R2 Assets From Pages The last step involves serving media assets from R2 on the blog. To do that, we will create a function to handle requests for media files. In the project folder, create a _functions_ directory. Then, create a _media_ subdirectory and a file named `[[all]].js` in it. All HTTP requests to `/media` will be routed to this file. After creating the folders and JavaScript file, the blog directory structure should look like: ``` . ├── functions │ └── media │ └── [[all]].js ├── public │ ├── index.html │ ├── static │ │ ├── favicon.ico │ │ └── icon.png │ └── style.css └── wrangler.toml ``` Finally, we will add a handler function to `[[all]].js`. This function receives all media requests, and returns the corresponding file asset from R2: ```js export async function onRequestGet(ctx) { const path = new URL(ctx.request.url).pathname.replace("/media/", ""); const file = await ctx.env.MEDIA.get(path); if (!file) return new Response(null, { status: 404 }); return new Response(file.body, { headers: { "Content-Type": file.httpMetadata.contentType }, }); } ``` ## Deploy the blog Before deploying the changes made so far to our cat blog, let us add a few new posts to `index.html`. These posts depend on media assets served from R2: ```html

Awesome Cat Blog! 😺

Today's post:

Yesterday's post:

``` With all the files saved, open a new terminal window to deploy the app: ```sh npm run deploy ``` Once deployed, media assets are fetched and served from the R2 bucket. ![Deployed App](/images/pages/tutorials/pages-r2/deployed.gif) ## **Related resources** - [Learn how function routing works in Pages.](/pages/functions/routing/) - [Learn how to create public R2 buckets](/r2/buckets/public-buckets/). - [Learn how to use R2 from Workers](/r2/api/workers/workers-api-usage/). --- # Handle rate limits of external APIs URL: https://developers.cloudflare.com/queues/tutorials/handle-rate-limits/ import { Render, PackageManagers, WranglerConfig } from "~/components"; This tutorial explains how to use Queues to handle rate limits of external APIs by building an application that sends email notifications using [Resend](https://www.resend.com/). However, you can use this pattern to handle rate limits of any external API. Resend is a service that allows you to send emails from your application via an API. Resend has a default [rate limit](https://resend.com/docs/api-reference/introduction#rate-limit) of two requests per second. You will use Queues to handle the rate limit of Resend. ## Prerequisites 4. Sign up for [Resend](https://resend.com/) and generate an API key by following the guide on the [Resend documentation](https://resend.com/docs/dashboard/api-keys/introduction). 5. Additionally, you will need access to Cloudflare Queues. ## 1. Create a new Workers application To get started, create a Worker application using the [`create-cloudflare` CLI](https://github.com/cloudflare/workers-sdk/tree/main/packages/create-cloudflare). Open a terminal window and run the following command: Then, go to your newly created directory: ```sh frame="none" cd resend-rate-limit-queue ``` ## 2. Set up a Queue You need to create a Queue and a binding to your Worker. Run the following command to create a Queue named `rate-limit-queue`: ```sh title="Create a Queue" npx wrangler queues create rate-limit-queue ``` ```sh output Creating queue rate-limit-queue. Created queue rate-limit-queue. ``` ### Add Queue bindings to your [Wrangler configuration file](/workers/wrangler/configuration/) In your Wrangler file, add the following: ```toml [[queues.producers]] binding = "EMAIL_QUEUE" queue = "rate-limit-queue" [[queues.consumers]] queue = "rate-limit-queue" max_batch_size = 2 max_batch_timeout = 10 max_retries = 3 ``` It is important to include the `max_batch_size` of two to the consumer queue is important because the Resend API has a default rate limit of two requests per second. This batch size allows the queue to process the message in the batch size of two. If the batch size is less than two, the queue will wait for 10 seconds to collect the next message. If no more messages are available, the queue will process the message in the batch. For more information, refer to the [Batching, Retries and Delays documentation](/queues/configuration/batching-retries) Your final Wrangler file should look similar to the example below. ```toml title="wrangler.toml" #:schema node_modules/wrangler/config-schema.json name = "resend-rate-limit-queue" main = "src/index.ts" compatibility_date = "2024-09-09" compatibility_flags = ["nodejs_compat"] [[queues.producers]] binding = "EMAIL_QUEUE" queue = "rate-limit-queue" [[queues.consumers]] queue = "rate-limit-queue" max_batch_size = 2 max_batch_timeout = 10 max_retries = 3 ``` ## 3. Add bindings to environment Add the bindings to the environment interface in `worker-configuration.d.ts`, so TypeScript correctly types the bindings. Type the queue as `Queue`. Refer to the following step for instructions on how to change this type. ```ts title="worker-configuration.d.ts" interface Env { EMAIL_QUEUE: Queue; } ``` ## 4. Send message to the queue The application will send a message to the queue when the Worker receives a request. For simplicity, you will send the email address as a message to the queue. A new message will be sent to the queue with a delay of one second. ```ts title="src/index.ts" export default { async fetch(req: Request, env: Env): Promise { try { await env.EMAIL_QUEUE.send( { email: await req.text() }, { delaySeconds: 1 }, ); return new Response("Success!"); } catch (e) { return new Response("Error!", { status: 500 }); } }, }; ``` This will accept requests to any subpath and forwards the request's body. It expects that the request body to contain only an email. In production, you should check that the request was a `POST` request. You should also avoid sending such sensitive information (email) directly to the queue. Instead, you can send a message to the queue that contains a unique identifier for the user. Then, your consumer queue can use the unique identifier to look up the email address in a database and use that to send the email. ## 5. Process the messages in the queue After the message is sent to the queue, it will be processed by the consumer Worker. The consumer Worker will process the message and send the email. Since you have not configured Resend yet, you will log the message to the console. After you configure Resend, you will use it to send the email. Add the `queue()` handler as shown below: ```ts title="src/index.ts" ins={1-3,17-28} interface Message { email: string; } export default { async fetch(req: Request, env: Env): Promise { try { await env.EMAIL_QUEUE.send( { email: await req.text() }, { delaySeconds: 1 }, ); return new Response("Success!"); } catch (e) { return new Response("Error!", { status: 500 }); } }, async queue(batch: MessageBatch, env: Env): Promise { for (const message of batch.messages) { try { console.log(message.body.email); // After configuring Resend, you can send email message.ack(); } catch (e) { console.error(e); message.retry({ delaySeconds: 5 }); } } }, }; ``` The above `queue()` handler will log the email address to the console and send the email. It will also retry the message if sending the email fails. The `delaySeconds` is set to five seconds to avoid sending the email too quickly. To test the application, run the following command: ```sh title="Start the development server" npm run dev ``` Use the following cURL command to send a request to the application: ```sh title="Test with a cURL request" curl -X POST -d "test@example.com" http://localhost:8787/ ``` ```sh output [wrangler:inf] POST / 200 OK (2ms) QueueMessage { attempts: 1, body: { email: 'test@example.com' }, timestamp: 2024-09-12T13:48:07.236Z, id: '72a25ff18dd441f5acb6086b9ce87c8c' } ``` ## 6. Set up Resend To call the Resend API, you need to configure the Resend API key. Create a `.dev.vars` file in the root of your project and add the following: ```txt title=".dev.vars" RESEND_API_KEY='your-resend-api-key' ``` Replace `your-resend-api-key` with your actual Resend API key. Next, update the `Env` interface in `worker-configuration.d.ts` to include the `RESEND_API_KEY` variable. ```ts title="worker-configuration.d.ts" ins={3} interface Env { EMAIL_QUEUE: Queue; RESEND_API_KEY: string; } ``` Lastly, install the [`resend` package](https://www.npmjs.com/package/resend) using the following command: ```sh title="Install Resend" npm install resend ``` You can now use the `RESEND_API_KEY` variable in your code. ## 7. Send email with Resend In your `src/index.ts` file, import the Resend package and update the `queue()` handler to send the email. ```ts title="src/index.ts" ins={1,21,26-40} del={24,41} import { Resend } from "resend"; interface Message { email: string; } export default { async fetch(req: Request, env: Env): Promise { try { await env.EMAIL_QUEUE.send( { email: await req.text() }, { delaySeconds: 1 }, ); return new Response("Success!"); } catch (e) { return new Response("Error!", { status: 500 }); } }, async queue(batch: MessageBatch, env: Env): Promise { // Initialize Resend const resend = new Resend(env.RESEND_API_KEY); for (const message of batch.messages) { try { console.log(message.body.email); // send email const sendEmail = await resend.emails.send({ from: "onboarding@resend.dev", to: [message.body.email], subject: "Hello World", html: "Sending an email from Worker!", }); // check if the email failed if (sendEmail.error) { console.error(sendEmail.error); message.retry({ delaySeconds: 5 }); } else { // if success, ack the message message.ack(); } message.ack(); } catch (e) { console.error(e); message.retry({ delaySeconds: 5 }); } } }, }; ``` The `queue()` handler will now send the email using the Resend API. It also checks if sending the email failed and will retry the message. The final script is included below: ```ts title="src/index.ts" import { Resend } from "resend"; interface Message { email: string; } export default { async fetch(req: Request, env: Env): Promise { try { await env.EMAIL_QUEUE.send( { email: await req.text() }, { delaySeconds: 1 }, ); return new Response("Success!"); } catch (e) { return new Response("Error!", { status: 500 }); } }, async queue(batch: MessageBatch, env: Env): Promise { // Initialize Resend const resend = new Resend(env.RESEND_API_KEY); for (const message of batch.messages) { try { // send email const sendEmail = await resend.emails.send({ from: "onboarding@resend.dev", to: [message.body.email], subject: "Hello World", html: "Sending an email from Worker!", }); // check if the email failed if (sendEmail.error) { console.error(sendEmail.error); message.retry({ delaySeconds: 5 }); } else { // if success, ack the message message.ack(); } } catch (e) { console.error(e); message.retry({ delaySeconds: 5 }); } } }, }; ``` To test the application, start the development server using the following command: ```sh title="Start the development server" npm run dev ``` Use the following cURL command to send a request to the application: ```sh title="Test with a cURL request" curl -X POST -d "delivered@resend.dev" http://localhost:8787/ ``` On the Resend dashboard, you should see that the email was sent to the provided email address. ## 8. Deploy your Worker To deploy your Worker, run the following command: ```sh title="Deploy your Worker" npx wrangler deploy ``` Lastly, add the Resend API key using the following command: ```sh title="Add the Resend API key" npx wrangler secret put RESEND_API_KEY ``` Enter the value of your API key. Your API key will get added to your project. You can now use the `RESEND_API_KEY` variable in your code. You have successfully created a Worker which can send emails using the Resend API respecting rate limits. To test your Worker, you could use the following cURL request. Replace `` with the URL of your deployed Worker. ```bash title="Test with a cURL request" curl -X POST -d "delivered@resend.dev" ``` Refer to the [GitHub repository](https://github.com/harshil1712/queues-rate-limit) for the complete code for this tutorial. If you are using [Hono](https://hono.dev/), you can refer to the [Hono example](https://github.com/harshil1712/resend-rate-limit-demo). ## Related resources - [How Queues works](/queues/reference/how-queues-works/) - [Queues Batching and Retries](/queues/configuration/batching-retries/) - [Resend](https://resend.com/docs/) --- # Build a web crawler with Queues and Browser Rendering URL: https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-rendering/ import { Render, PackageManagers, WranglerConfig } from "~/components"; This tutorial explains how to build and deploy a web crawler with Queues, [Browser Rendering](/browser-rendering/), and [Puppeteer](/browser-rendering/platform/puppeteer/). Puppeteer is a high-level library used to automate interactions with Chrome/Chromium browsers. On each submitted page, the crawler will find the number of links to `cloudflare.com` and take a screenshot of the site, saving results to [Workers KV](/kv/). You can use Puppeteer to request all images on a page, save the colors used on a site, and more. ## Prerequisites ## 1. Create new Workers application To get started, create a Worker application using the [`create-cloudflare` CLI](https://github.com/cloudflare/workers-sdk/tree/main/packages/create-cloudflare). Open a terminal window and run the following command: Then, move into your newly created directory: ```sh cd queues-web-crawler ``` ## 2. Create KV namespace We need to create a KV store. This can be done through the Cloudflare dashboard or the Wrangler CLI. For this tutorial, we will use the Wrangler CLI. ```sh npx wrangler kv namespace create crawler_links npx wrangler kv namespace create crawler_screenshots ``` ```sh output 🌀 Creating namespace with title "web-crawler-crawler-links" ✨ Success! Add the following to your configuration file in your kv_namespaces array: [[kv_namespaces]] binding = "crawler_links" id = "" 🌀 Creating namespace with title "web-crawler-crawler-screenshots" ✨ Success! Add the following to your configuration file in your kv_namespaces array: [[kv_namespaces]] binding = "crawler_screenshots" id = "" ``` ### Add KV bindings to the [Wrangler configuration file](/workers/wrangler/configuration/) Then, in your Wrangler file, add the following with the values generated in the terminal: ```toml kv_namespaces = [ { binding = "CRAWLER_SCREENSHOTS_KV", id = "" }, { binding = "CRAWLER_LINKS_KV", id = "" } ] ``` ## 3. Set up Browser Rendering Now, you need to set up your Worker for Browser Rendering. In your current directory, install Cloudflare’s [fork of Puppeteer](/browser-rendering/platform/puppeteer/) and also [robots-parser](https://www.npmjs.com/package/robots-parser): ```sh npm install @cloudflare/puppeteer --save-dev npm install robots-parser ``` Then, add a Browser Rendering binding. Adding a Browser Rendering binding gives the Worker access to a headless Chromium instance you will control with Puppeteer. ```toml browser = { binding = "CRAWLER_BROWSER" } ``` ## 4. Set up a Queue Now, we need to set up the Queue. ```sh npx wrangler queues create queues-web-crawler ``` ```txt title="Output" Creating queue queues-web-crawler. Created queue queues-web-crawler. ``` ### Add Queue bindings to wrangler.toml Then, in your Wrangler file, add the following: ```toml [[queues.consumers]] queue = "queues-web-crawler" max_batch_timeout = 60 [[queues.producers]] queue = "queues-web-crawler" binding = "CRAWLER_QUEUE" ``` Adding the `max_batch_timeout` of 60 seconds to the consumer queue is important because Browser Rendering has a limit of two new browsers per minute per account. This timeout waits up to a minute before collecting queue messages into a batch. The Worker will then remain under this browser invocation limit. Your final Wrangler file should look similar to the one below. ```toml #:schema node_modules/wrangler/config-schema.json name = "web-crawler" main = "src/index.ts" compatibility_date = "2024-07-25" compatibility_flags = ["nodejs_compat"] kv_namespaces = [ { binding = "CRAWLER_SCREENSHOTS_KV", id = "" }, { binding = "CRAWLER_LINKS_KV", id = "" } ] browser = { binding = "CRAWLER_BROWSER" } [[queues.consumers]] queue = "queues-web-crawler" max_batch_timeout = 60 [[queues.producers]] queue = "queues-web-crawler" binding = "CRAWLER_QUEUE" ``` ## 5. Add bindings to environment Add the bindings to the environment interface in `src/index.ts`, so TypeScript correctly types the bindings. Type the queue as `Queue`. The following step will show you how to change this type. ```ts import { BrowserWorker } from "@cloudflare/puppeteer"; export interface Env { CRAWLER_QUEUE: Queue; CRAWLER_SCREENSHOTS_KV: KVNamespace; CRAWLER_LINKS_KV: KVNamespace; CRAWLER_BROWSER: BrowserWorker; } ``` ## 6. Submit links to crawl Add a `fetch()` handler to the Worker to submit links to crawl. ```ts type Message = { url: string; }; export interface Env { CRAWLER_QUEUE: Queue; // ... etc. } export default { async fetch(req, env): Promise { await env.CRAWLER_QUEUE.send({ url: await req.text() }); return new Response("Success!"); }, } satisfies ExportedHandler; ``` This will accept requests to any subpath and forwards the request's body to be crawled. It expects that the request body only contains a URL. In production, you should check that the request was a `POST` request and contains a well-formed URL in its body. This has been omitted for simplicity. ## 7. Crawl with Puppeteer Add a `queue()` handler to the Worker to process the links you send. ```ts import puppeteer from "@cloudflare/puppeteer"; import robotsParser from "robots-parser"; async queue(batch: MessageBatch, env: Env): Promise { let browser: puppeteer.Browser | null = null; try { browser = await puppeteer.launch(env.CRAWLER_BROWSER); } catch { batch.retryAll(); return; } for (const message of batch.messages) { const { url } = message.body; let isAllowed = true; try { const robotsTextPath = new URL(url).origin + "/robots.txt"; const response = await fetch(robotsTextPath); const robots = robotsParser(robotsTextPath, await response.text()); isAllowed = robots.isAllowed(url) ?? true; // respect robots.txt! } catch {} if (!isAllowed) { message.ack(); continue; } // TODO: crawl! message.ack(); } await browser.close(); }, ``` This is a skeleton for the crawler. It launches the Puppeteer browser and iterates through the Queue's received messages. It fetches the site's `robots.txt` and uses `robots-parser` to check that this site allows crawling. If crawling is not allowed, the message is `ack`'ed, removing it from the Queue. If crawling is allowed, you can continue to crawl the site. The `puppeteer.launch()` is wrapped in a `try...catch` to allow the whole batch to be retried if the browser launch fails. The browser launch may fail due to going over the limit for number of browsers per account. ```ts type Result = { numCloudflareLinks: number; screenshot: ArrayBuffer; }; const crawlPage = async (url: string): Promise => { const page = await (browser as puppeteer.Browser).newPage(); await page.goto(url, { waitUntil: "load", }); const numCloudflareLinks = await page.$$eval("a", (links) => { links = links.filter((link) => { try { return new URL(link.href).hostname.includes("cloudflare.com"); } catch { return false; } }); return links.length; }); await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1, }); return { numCloudflareLinks, screenshot: ((await page.screenshot({ fullPage: true })) as Buffer).buffer, }; }; ``` This helper function opens a new page in Puppeteer and navigates to the provided URL. `numCloudflareLinks` uses Puppeteer's `$$eval` (equivalent to `document.querySelectorAll`) to find the number of links to a `cloudflare.com` page. Checking if the link's `href` is to a `cloudflare.com` page is wrapped in a `try...catch` to handle cases where `href`s may not be URLs. Then, the function sets the browser viewport size and takes a screenshot of the full page. The screenshot is returned as a `Buffer` so it can be converted to an `ArrayBuffer` and written to KV. To enable recursively crawling links, add a snippet after checking the number of Cloudflare links to send messages recursively from the queue consumer to the queue itself. Recursing too deep, as is possible with crawling, will cause a Durable Object `Subrequest depth limit exceeded.` error. If one occurs, it is caught, but the links are not retried. ```ts null {3-14} // const numCloudflareLinks = await page.$$eval("a", (links) => { ... await page.$$eval("a", async (links) => { const urls: MessageSendRequest[] = links.map((link) => { return { body: { url: link.href, }, }; }); try { await env.CRAWLER_QUEUE.sendBatch(urls); } catch {} // do nothing, likely hit subrequest limit }); // await page.setViewport({ ... ``` Then, in the `queue` handler, call `crawlPage` on the URL. ```ts null {8-23} // in the `queue` handler: // ... if (!isAllowed) { message.ack(); continue; } try { const { numCloudflareLinks, screenshot } = await crawlPage(url); const timestamp = new Date().getTime(); const resultKey = `${encodeURIComponent(url)}-${timestamp}`; await env.CRAWLER_LINKS_KV.put(resultKey, numCloudflareLinks.toString(), { metadata: { date: timestamp }, }); await env.CRAWLER_SCREENSHOTS_KV.put(resultKey, screenshot, { metadata: { date: timestamp }, }); message.ack(); } catch { message.retry(); } // ... ``` This snippet saves the results from `crawlPage` into the appropriate KV namespaces. If an unexpected error occurred, the URL will be retried and resent to the queue again. Saving the timestamp of the crawl in KV helps you avoid crawling too frequently. Add a snippet before checking `robots.txt` to check KV for a crawl within the last hour. This lists all KV keys beginning with the same URL (crawls of the same page), and check if any crawls have been done within the last hour. If any crawls have been done within the last hour, the message is `ack`'ed and not retried. ```ts null {12-23} type KeyMetadata = { date: number; }; // in the `queue` handler: // ... for (const message of batch.messages) { const sameUrlCrawls = await env.CRAWLER_LINKS_KV.list({ prefix: `${encodeURIComponent(url)}`, }); let shouldSkip = false; for (const key of sameUrlCrawls.keys) { if (timestamp - (key.metadata as KeyMetadata)?.date < 60 * 60 * 1000) { // if crawled in last hour, skip message.ack(); shouldSkip = true; break; } } if (shouldSkip) { continue; } let isAllowed = true; // ... ``` The final script is included below. ```ts import puppeteer, { BrowserWorker } from "@cloudflare/puppeteer"; import robotsParser from "robots-parser"; type Message = { url: string; }; export interface Env { CRAWLER_QUEUE: Queue; CRAWLER_SCREENSHOTS_KV: KVNamespace; CRAWLER_LINKS_KV: KVNamespace; CRAWLER_BROWSER: BrowserWorker; } type Result = { numCloudflareLinks: number; screenshot: ArrayBuffer; }; type KeyMetadata = { date: number; }; export default { async fetch(req: Request, env: Env): Promise { // util endpoint for testing purposes await env.CRAWLER_QUEUE.send({ url: await req.text() }); return new Response("Success!"); }, async queue(batch: MessageBatch, env: Env): Promise { const crawlPage = async (url: string): Promise => { const page = await (browser as puppeteer.Browser).newPage(); await page.goto(url, { waitUntil: "load", }); const numCloudflareLinks = await page.$$eval("a", (links) => { links = links.filter((link) => { try { return new URL(link.href).hostname.includes("cloudflare.com"); } catch { return false; } }); return links.length; }); // to crawl recursively - uncomment this! /*await page.$$eval("a", async (links) => { const urls: MessageSendRequest[] = links.map((link) => { return { body: { url: link.href, }, }; }); try { await env.CRAWLER_QUEUE.sendBatch(urls); } catch {} // do nothing, might've hit subrequest limit });*/ await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1, }); return { numCloudflareLinks, screenshot: ((await page.screenshot({ fullPage: true })) as Buffer) .buffer, }; }; let browser: puppeteer.Browser | null = null; try { browser = await puppeteer.launch(env.CRAWLER_BROWSER); } catch { batch.retryAll(); return; } for (const message of batch.messages) { const { url } = message.body; const timestamp = new Date().getTime(); const resultKey = `${encodeURIComponent(url)}-${timestamp}`; const sameUrlCrawls = await env.CRAWLER_LINKS_KV.list({ prefix: `${encodeURIComponent(url)}`, }); let shouldSkip = false; for (const key of sameUrlCrawls.keys) { if (timestamp - (key.metadata as KeyMetadata)?.date < 60 * 60 * 1000) { // if crawled in last hour, skip message.ack(); shouldSkip = true; break; } } if (shouldSkip) { continue; } let isAllowed = true; try { const robotsTextPath = new URL(url).origin + "/robots.txt"; const response = await fetch(robotsTextPath); const robots = robotsParser(robotsTextPath, await response.text()); isAllowed = robots.isAllowed(url) ?? true; // respect robots.txt! } catch {} if (!isAllowed) { message.ack(); continue; } try { const { numCloudflareLinks, screenshot } = await crawlPage(url); await env.CRAWLER_LINKS_KV.put( resultKey, numCloudflareLinks.toString(), { metadata: { date: timestamp } }, ); await env.CRAWLER_SCREENSHOTS_KV.put(resultKey, screenshot, { metadata: { date: timestamp }, }); message.ack(); } catch { message.retry(); } } await browser.close(); }, }; ``` ## 8. Deploy your Worker To deploy your Worker, run the following command: ```sh npx wrangler deploy ``` You have successfully created a Worker which can submit URLs to a queue for crawling and save results to Workers KV. To test your Worker, you could use the following cURL request to take a screenshot of this documentation page. ```bash title="Test with a cURL request" curl \ -H "Content-Type: application/json" \ -d 'https://developers.cloudflare.com/queues/tutorials/web-crawler-with-browser-rendering/' ``` Refer to the [GitHub repository for the complete tutorial](https://github.com/cloudflare/queues-web-crawler), including a front end deployed with Pages to submit URLs and view crawler results. ## Related resources - [How Queues works](/queues/reference/how-queues-works/) - [Queues Batching and Retries](/queues/configuration/batching-retries/) - [Browser Rendering](/browser-rendering/) - [Puppeteer Examples](https://github.com/puppeteer/puppeteer/tree/main/examples) --- # S3 API compatibility URL: https://developers.cloudflare.com/r2/api/s3/api/ import { Details } from "~/components"; R2 implements the S3 API to allow users and their applications to migrate with ease. When comparing to AWS S3, Cloudflare has removed some API operations' features and added others. The S3 API operations are listed below with their current implementation status. Feature implementation is currently in progress. Refer back to this page for updates. The API is available via the `https://.r2.cloudflarestorage.com` endpoint. Find your [account ID in the Cloudflare dashboard](/fundamentals/setup/find-account-and-zone-ids/). ## How to read this page This page has two sections: bucket-level operations and object-level operations. Each section will have two tables: a table of implemented APIs and a table of unimplemented APIs. Refer the feature column of each table to review which features of an API have been implemented and which have not. ✅ Feature Implemented
🚧 Feature Implemented (Experimental)
❌ Feature Not Implemented ## Bucket region When using the S3 API, the region for an R2 bucket is `auto`. For compatibility with tools that do not allow you to specify a region, an empty value and `us-east-1` will alias to the `auto` region. This also applies to the `LocationConstraint` for the `CreateBucket` API. ## Bucket-level operations The following tables are related to bucket-level operations. ### Implemented bucket-level operations Below is a list of implemented bucket-level operations. Refer to the Feature column to review which features have been implemented (✅) and have not been implemented (❌). | API Name | Feature | | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ✅ [ListBuckets](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html) | | | ✅ [HeadBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [CreateBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html) | ❌ ACL:
❌ x-amz-acl
❌ x-amz-grant-full-control
❌ x-amz-grant-read
❌ x-amz-grant-read-acp
❌ x-amz-grant-write
❌ x-amz-grant-write-acp
❌ Object Locking:
❌ x-amz-bucket-object-lock-enabled
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [DeleteBucket](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [DeleteBucketCors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [GetBucketCors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [GetBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLifecycleConfiguration.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [GetBucketLocation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [GetBucketEncryption](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketEncryption.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [PutBucketCors](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html) | ❌ Checksums:
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [PutBucketLifecycleConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html) | ❌ Checksums:
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | ### Unimplemented bucket-level operations

| API Name | Feature | | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ❌ [GetBucketAccelerateConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAccelerateConfiguration.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketAcl](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketAnalyticsConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAnalyticsConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketIntelligentTieringConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketIntelligentTieringConfiguration.html) | ❌ id | | ❌ [GetBucketInventoryConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketInventoryConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketLifecycle](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLifecycle.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketLogging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLogging.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketMetricsConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketMetricsConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketNotification](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketNotification.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketNotificationConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketNotificationConfiguration.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketOwnershipControls](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketOwnershipControls.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketPolicy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketPolicyStatus](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicyStatus.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketReplication](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketReplication.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketRequestPayment](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketRequestPayment.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketTagging.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketVersioning](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetBucketWebsite](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketWebsite.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetObjectLockConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectLockConfiguration.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [GetPublicAccessBlock](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetPublicAccessBlock.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [ListBucketAnalyticsConfigurations](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBucketAnalyticsConfigurations.html) | ❌ Query Parameters:
❌ continuation-token
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [ListBucketIntelligentTieringConfigurations](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBucketIntelligentTieringConfigurations.html) | ❌ Query Parameters:
❌ continuation-token
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [ListBucketInventoryConfigurations](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBucketInventoryConfigurations.html) | ❌ Query Parameters:
❌ continuation-token
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [ListBucketMetricsConfigurations](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBucketMetricsConfigurations.html) | ❌ Query Parameters:
❌ continuation-token
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketAccelerateConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAccelerateConfiguration.html) | ❌ Checksums:
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketAcl](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html) | ❌ Permissions:
❌ x-amz-grant-full-control
❌ x-amz-grant-read
❌ x-amz-grant-read-acp
❌ x-amz-grant-write
❌ x-amz-grant-write-acp
❌ Checksums:
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketAnalyticsConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAnalyticsConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketEncryption](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketEncryption.html) | ❌ Checksums:
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketIntelligentTieringConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketIntelligentTieringConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketInventoryConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketInventoryConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketLifecycle](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycle.html) | ❌ Checksums:
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketLogging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycle.html) | ❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketMetricsConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketMetricsConfiguration.html) | ❌ id
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketNotification](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketNotification.html) | ❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketNotificationConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketNotificationConfiguration.html) | ❌ Validation:
❌ x-amz-skip-destination-validation
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketOwnershipControls](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketOwnershipControls.html) | ❌ Checksums:
❌ Content-MD5
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketPolicy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html) | ❌ Validation:
❌ x-amz-confirm-remove-self-bucket-access
❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketReplication](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketReplication.html) | ❌ Object Locking:
❌ x-amz-bucket-object-lock-token
❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketRequestPayment](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketRequestPayment.html) | ❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketTagging.html) | ❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketVersioning](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketVersioning.html) | ❌ Multi-factor authentication:
❌ x-amz-mfa
❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutBucketWebsite](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketWebsite.html) | ❌ Checksums:
❌ Content-MD5
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutObjectLockConfiguration](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectLockConfiguration.html) | ❌ Object Locking:
❌ x-amz-bucket-object-lock-token
❌ Checksums:
❌ Content-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ❌ [PutPublicAccessBlock](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutPublicAccessBlock.html) | ❌ Checksums:
❌ Content-MD5
❌ x-amz-sdk-checksum-algorithm
❌ x-amz-checksum-algorithm
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner |

## Object-level operations The following tables are related to object-level operations. ### Implemented object-level operations Below is a list of implemented object-level operations. Refer to the Feature column to review which features have been implemented (✅) and have not been implemented (❌). | API Name | Feature | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ✅ [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) | ✅ Conditional Operations:
✅ If-Match
✅ If-Modified-Since
✅ If-None-Match
✅ If-Unmodified-Since
✅ Range:
✅ Range (has no effect in HeadObject)
✅ partNumber
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [ListObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html) | Query Parameters:
✅ delimiter
✅ encoding-type
✅ marker
✅ max-keys
✅ prefix
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) | Query Parameters:
✅ list-type
✅ continuation-token
✅ delimiter
✅ encoding-type
✅ fetch-owner
✅ max-keys
✅ prefix
✅ start-after
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) | ✅ Conditional Operations:
✅ If-Match
✅ If-Modified-Since
✅ If-None-Match
✅ If-Unmodified-Since
✅ Range:
✅ Range
✅ PartNumber
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) | ✅ System Metadata:
✅ Content-Type
✅ Cache-Control
✅ Content-Disposition
✅ Content-Encoding
✅ Content-Language
✅ Expires
✅ Content-MD5
✅ Storage Class:
✅ x-amz-storage-class
✅ STANDARD
✅ STANDARD_IA
❌ Object Lifecycle
❌ Website:
❌ x-amz-website-redirect-location
❌ SSE:
❌ x-amz-server-side-encryption-aws-kms-key-id
❌ x-amz-server-side-encryption
❌ x-amz-server-side-encryption-context
❌ x-amz-server-side-encryption-bucket-key-enabled
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Tagging:
❌ x-amz-tagging
❌ Object Locking:
❌ x-amz-object-lock-mode
❌ x-amz-object-lock-retain-until-date
❌ x-amz-object-lock-legal-hold
❌ ACL:
❌ x-amz-acl
❌ x-amz-grant-full-control
❌ x-amz-grant-read
❌ x-amz-grant-read-acp
❌ x-amz-grant-write-acp
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) | ❌ Multi-factor authentication:
❌ x-amz-mfa
❌ Object Locking:
❌ x-amz-bypass-governance-retention
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) | ❌ Multi-factor authentication:
❌ x-amz-mfa
❌ Object Locking:
❌ x-amz-bypass-governance-retention
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [ListMultipartUploads](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html) | ✅ Query Parameters:
✅ delimiter
✅ encoding-type
✅ key-marker
✅️ max-uploads
✅ prefix
✅ upload-id-marker | | ✅ [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) | ✅ System Metadata:
✅ Content-Type
✅ Cache-Control
✅ Content-Disposition
✅ Content-Encoding
✅ Content-Language
✅ Expires
✅ Content-MD5
✅ Storage Class:
✅ x-amz-storage-class
✅ STANDARD
✅ STANDARD_IA
❌ Website:
❌ x-amz-website-redirect-location
❌ SSE:
❌ x-amz-server-side-encryption-aws-kms-key-id
❌ x-amz-server-side-encryption
❌ x-amz-server-side-encryption-context
❌ x-amz-server-side-encryption-bucket-key-enabled
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Tagging:
❌ x-amz-tagging
❌ Object Locking:
❌ x-amz-object-lock-mode
❌ x-amz-object-lock-retain-until-date
❌ x-amz-object-lock-legal-hold
❌ ACL:
❌ x-amz-acl
❌ x-amz-grant-full-control
❌ x-amz-grant-read
❌ x-amz-grant-read-acp
❌ x-amz-grant-write-acp
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner
❌ Request Payer:
❌ x-amz-request-payer | | ✅ [AbortMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html) | ❌ Request Payer:
❌ x-amz-request-payer | | ✅ [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html) | ✅ Operation Metadata:
✅ x-amz-metadata-directive
✅ System Metadata:
✅ Content-Type
✅ Cache-Control
✅ Content-Disposition
✅ Content-Encoding
✅ Content-Language
✅ Expires
✅ Conditional Operations:
✅ x-amz-copy-source
✅ x-amz-copy-source-if-match
✅ x-amz-copy-source-if-modified-since
✅ x-amz-copy-source-if-none-match
✅ x-amz-copy-source-if-unmodified-since
✅ Storage Class:
✅ x-amz-storage-class
✅ STANDARD
✅ STANDARD_IA
❌ ACL:
❌ x-amz-acl
❌ x-amz-grant-full-control
❌ x-amz-grant-read
❌ x-amz-grant-read-acp
❌ x-amz-grant-write-acp
❌ Website:
❌ x-amz-website-redirect-location
❌ SSE:
❌ x-amz-server-side-encryption
❌ x-amz-server-side-encryption-aws-kms-key-id
❌ x-amz-server-side-encryption-context
❌ x-amz-server-side-encryption-bucket-key-enabled
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
✅ x-amz-copy-source-server-side-encryption-customer-algorithm
✅ x-amz-copy-source-server-side-encryption-customer-key
✅ x-amz-copy-source-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Tagging:
❌ x-amz-tagging
❌ x-amz-tagging-directive
❌ Object Locking:
❌ x-amz-object-lock-mode
❌ x-amz-object-lock-retain-until-date
❌ x-amz-object-lock-legal-hold
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner
❌ x-amz-source-expected-bucket-owner
❌ Checksums:
❌ x-amz-checksum-algorithm | | ✅ [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) | ✅ System Metadata:
✅ Content-MD5
❌ SSE:
❌ x-amz-server-side-encryption
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | | ✅ [UploadPartCopy](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html) | ❌ Conditional Operations:
❌ x-amz-copy-source
❌ x-amz-copy-source-if-match
❌ x-amz-copy-source-if-modified-since
❌ x-amz-copy-source-if-none-match
❌ x-amz-copy-source-if-unmodified-since
✅ Range:
✅ x-amz-copy-source-range
✅ SSE-C:
✅ x-amz-server-side-encryption-customer-algorithm
✅ x-amz-server-side-encryption-customer-key
✅ x-amz-server-side-encryption-customer-key-MD5
✅ x-amz-copy-source-server-side-encryption-customer-algorithm
✅ x-amz-copy-source-server-side-encryption-customer-key
✅ x-amz-copy-source-server-side-encryption-customer-key-MD5
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner
❌ x-amz-source-expected-bucket-owner | | ✅ [ListParts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html) | Query Parameters:
✅ max-parts
✅ part-number-marker
❌ Request Payer:
❌ x-amz-request-payer
❌ Bucket Owner:
❌ x-amz-expected-bucket-owner | :::caution Even though `ListObjects` is a supported operation, it is recommended that you use `ListObjectsV2` instead when developing applications. For more information, refer to [ListObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html). ::: ### Unimplemented object-level operations

| API Name | Feature | | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ❌ [GetObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner
❌ Request Payer:
❌ x-amz-request-payer | | ❌ [PutObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner
❌ Request Payer:
❌ x-amz-request-payer
❌ Checksums:
❌ x-amz-sdk-checksum-algorithm | | ❌ [DeleteObjectTagging](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html) | ❌ Bucket Owner:
❌ x-amz-expected-bucket-owner |

--- # Extensions URL: https://developers.cloudflare.com/r2/api/s3/extensions/ R2 implements some extensions on top of the basic S3 API. This page outlines these additional, available features. Some of the functionality described in this page requires setting a custom header. For examples on how to do so, refer to [Configure custom headers](/r2/examples/aws/custom-header). ## Extended metadata using Unicode The [Workers R2 API](/r2/api/workers/workers-api-reference/) supports Unicode in keys and values natively without requiring any additional encoding or decoding for the `customMetadata` field. These fields map to the `x-amz-meta-`-prefixed headers used within the R2 S3-compatible API endpoint. HTTP header names and values may only contain ASCII characters, which is a small subset of the Unicode character library. To easily accommodate users, R2 adheres to [RFC 2047](https://datatracker.ietf.org/doc/html/rfc2047) and automatically decodes all `x-amz-meta-*` header values before storage. On retrieval, any metadata values with unicode are RFC 2047-encoded before rendering the response. The length limit for metadata values is applied to the decoded Unicode value. :::caution[Metadata variance] Be mindful when using both Workers and S3 API endpoints to access the same data. If the R2 metadata keys contain Unicode, they are stripped when accessed through the S3 API and the `x-amz-missing-meta` header is set to the number of keys that were omitted. ::: These headers map to the `httpMetadata` field in the [R2 bindings](/workers/runtime-apis/bindings/): | HTTP Header | Property Name | | --------------------- | --------------------------------- | | `Content-Encoding` | `httpMetadata.contentEncoding` | | `Content-Type` | `httpMetadata.contentType` | | `Content-Language` | `httpMetadata.contentLanguage` | | `Content-Disposition` | `httpMetadata.contentDisposition` | | `Cache-Control` | `httpMetadata.cacheControl` | | `Expires` | `httpMetadata.expires` | | | | If using Unicode in object key names, refer to [Unicode Interoperability](/r2/reference/unicode-interoperability/). ## Auto-creating buckets on upload If you are creating buckets on demand, you might initiate an upload with the assumption that a target bucket exists. In this situation, if you received a `NoSuchBucket` error, you would probably issue a `CreateBucket` operation. However, following this approach can cause issues: if the body has already been partially consumed, the upload will need to be aborted. A common solution to this issue, followed by other object storage providers, is to use the [HTTP `100`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/100) response to detect whether the body should be sent, or if the bucket must be created before retrying the upload. However, Cloudflare does not support the HTTP `100` response. Even if the HTTP `100` response was supported, you would still have additional latency due to the round trips involved. To support sending an upload with a streaming body to a bucket that may not exist yet, upload operations such as `PutObject` or `CreateMultipartUpload` allow you to specify a header that will ensure the `NoSuchBucket` error is not returned. If the bucket does not exist at the time of upload, it is implicitly instantiated with the following `CreateBucket` request: ```txt PUT / HTTP/1.1 Host: bucket.account.r2.cloudflarestorage.com auto ``` This is only useful if you are creating buckets on demand because you do not know the name of the bucket or the preferred access location ahead of time. For example, you have one bucket per one of your customers and the bucket is created on first upload to the bucket and not during account registration. In these cases, the [`ListBuckets` extension](#listbuckets), which supports accounts with more than 1,000 buckets, may also be useful. ## PutObject and CreateMultipartUpload ### cf-create-bucket-if-missing Add a `cf-create-bucket-if-missing` header with the value `true` to implicitly create the bucket if it does not exist yet. Refer to [Auto-creating buckets on upload](#auto-creating-buckets-on-upload) for a more detailed explanation of when to add this header. ## PutObject ### Conditional operations in `PutObject` `PutObject` supports [conditional uploads](https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests) via the [`If-Match`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Match), [`If-None-Match`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match), [`If-Modified-Since`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since), and [`If-Unmodified-Since`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Unmodified-Since) headers. These headers will cause the `PutObject` operation to be rejected with `412 PreconditionFailed` error codes when the preceding state of the object that is being written to does not match the specified conditions. ## CopyObject ### MERGE metadata directive The `x-amz-metadata-directive` allows a `MERGE` value, in addition to the standard `COPY` and `REPLACE` options. When used, `MERGE` is a combination of `COPY` and `REPLACE`, which will `COPY` any metadata keys from the source object and `REPLACE` those that are specified in the request with the new value. You cannot use `MERGE` to remove existing metadata keys from the source — use `REPLACE` instead. ## `ListBuckets` `ListBuckets` supports all the same search parameters as `ListObjectsV2` in R2 because some customers may have more than 1,000 buckets. Because tooling, like existing S3 libraries, may not expose a way to set these search parameters, these values may also be sent in via headers. Values in headers take precedence over the search parameters. | Search parameter | HTTP Header | Meaning | | -------------------- | ----------------------- | ----------------------------------------------------------------- | | `prefix` | `cf-prefix` | Show buckets with this prefix only. | | `start-after` | `cf-start-after` | Show buckets whose name appears lexicographically in the account. | | `continuation-token` | `cf-continuation-token` | Resume listing from a previously returned continuation token. | | `max-keys` | `cf-max-keys` | Return this maximum number of buckets. Default and max is `1000`. | | | | | The XML response contains a `NextContinuationToken` and `IsTruncated` elements as appropriate. Since these may not be accessible from existing S3 APIs, these are also available in response headers: | XML Response Element | HTTP Response Header | Meaning | | ----------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------- | | `IsTruncated` | `cf-is-truncated` | This is set to `true` if the list of buckets returned is not all the buckets on the account. | | `NextContinuationToken` | `cf-next-continuation-token` | This is set to continuation token to pass on a subsequent `ListBuckets` to resume the listing. | | `StartAfter` | | This is the start-after value that was passed in on the request. | | `KeyCount` | | The number of buckets returned. | | `ContinuationToken` | | The continuation token that was supplied in the request. | | `MaxKeys` | | The max keys that were specified in the request. | | | | | ### Conditional operations in `CopyObject` for the destination object :::note This feature is currently in beta. If you have feedback, reach out to us on the [Cloudflare Developer Discord](https://discord.cloudflare.com) in the #r2-storage channel or open a thread on the [Community Forum](https://community.cloudflare.com/c/developers/storage/81). ::: `CopyObject` already supports conditions that relate to the source object through the `x-amz-copy-source-if-...` headers as part of our compliance with the S3 API. In addition to this, R2 supports an R2 specific set of headers that allow the `CopyObject` operation to be conditional on the target object: * `cf-copy-destination-if-match` * `cf-copy-destination-if-none-match` * `cf-copy-destination-if-modified-since` * `cf-copy-destination-if-unmodified-since` These headers work akin to the similarly named conditional headers supported on `PutObject`. When the preceding state of the destination object to does not match the specified conditions the `CopyObject` operation will be rejected with a `412 PreconditionFailed` error code. #### Non-atomicity relative to `x-amz-copy-source-if` The `x-amz-copy-source-if-...` headers are guaranteed to be checked when the source object for the copy operation is selected, and the `cf-copy-destination-if-...` headers are guaranteed to be checked when the object is committed to the bucket state. However, the time at which the source object is selected for copying, and the point in time when the destination object is committed to the bucket state are not necessarily the same. This means that the `cf-copy-destination-if-...` headers are not atomic in relation to the `x-amz-copy-source-if...` headers. --- # S3 URL: https://developers.cloudflare.com/r2/api/s3/ import { DirectoryListing } from "~/components"; --- # Presigned URLs URL: https://developers.cloudflare.com/r2/api/s3/presigned-urls/ Presigned URLs are an [S3 concept](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html) for sharing direct access to your bucket without revealing your token secret. A presigned URL authorizes anyone with the URL to perform an action to the S3 compatibility endpoint for an R2 bucket. By default, the S3 endpoint requires an `AUTHORIZATION` header signed by your token. Every presigned URL has S3 parameters and search parameters containing the signature information that would be present in an `AUTHORIZATION` header. The performable action is restricted to a specific resource, an [operation](/r2/api/s3/api/), and has an associated timeout. There are three kinds of resources in R2: 1. **Account**: For account-level operations (such as `CreateBucket`, `ListBuckets`, `DeleteBucket`) the identifier is the account ID. 2. **Bucket**: For bucket-level operations (such as `ListObjects`, `PutBucketCors`) the identifier is the account ID, and bucket name. 3. **Object**: For object-level operations (such as `GetObject`, `PutObject`, `CreateMultipartUpload`) the identifier is the account ID, bucket name, and object path. All parts of the identifier are part of the presigned URL. You cannot change the resource being accessed after the request is signed. For example, trying to change the bucket name to access the same object in a different bucket will return a `403` with an error code of `SignatureDoesNotMatch`. Presigned URLs must have a defined expiry. You can set a timeout from one second to 7 days (604,800 seconds) into the future. The URL will contain the time when the URL was generated (`X-Amz-Date`) and the timeout (`X-Amz-Expires`) as search parameters. These search parameters are signed and tampering with them will result in `403` with an error code of `SignatureDoesNotMatch`. Presigned URLs are generated with no communication with R2 and must be generated by an application with access to your R2 bucket's credentials. ## Presigned URL use cases There are three ways to grant an application access to R2: 1. The application has its own copy of an [R2 API token](/r2/api/tokens/). 2. The application requests a copy of an R2 API token from a vault application and promises to not permanently store that token locally. 3. The application requests a central application to give it a presigned URL it can use to perform an action. In scenarios 1 and 2, if the application or vault application is compromised, the holder of the token can perform arbitrary actions. Scenario 3 keeps the credential secret. If the application making a presigned URL request to the central application leaks that URL, but the central application does not have its key storage system compromised, the impact is limited to one operation on the specific resource that was signed. Additionally, the central application can perform monitoring, auditing, logging tasks so you can review when a request was made to perform an operation on a specific resource. In the event of a security incident, you can use a central application's logging functionality to review details of the incident. The central application can also perform policy enforcement. For example, if you have an application responsible for uploading resources, you can restrict the upload to a specific bucket or folder within a bucket. The requesting application can obtain a JSON Web Token (JWT) from your authorization service to sign a request to the central application. The central application then uses the information contained in the JWT to validate the inbound request parameters. The central application can be, for example, a Cloudflare Worker. Worker secrets are cryptographically impossible to obtain outside of your script running on the Workers runtime. If you do not store a copy of the secret elsewhere and do not have your code log the secret somewhere, your Worker secret will remain secure. However, as previously mentioned, presigned URLs are generated outside of R2 and all that's required is the secret + an implementation of the signing algorithm, so you can generate them anywhere. Another potential use case for presigned URLs is debugging. For example, if you are debugging your application and want to grant temporary access to a specific test object in a production environment, you can do this without needing to share the underlying token and remembering to revoke it. ## Supported HTTP methods R2 currently supports the following methods when generating a presigned URL: - `GET`: allows a user to fetch an object from a bucket - `HEAD`: allows a user to fetch an object's metadata from a bucket - `PUT`: allows a user to upload an object to a bucket - `DELETE`: allows a user to delete an object from a bucket `POST`, which performs uploads via native HTML forms, is not currently supported. ## Generate presigned URLs Generate a presigned URL by referring to the following examples: - [AWS SDK for Go](/r2/examples/aws/aws-sdk-go/#generate-presigned-urls) - [AWS SDK for JS v3](/r2/examples/aws/aws-sdk-js-v3/#generate-presigned-urls) - [AWS SDK for JS](/r2/examples/aws/aws-sdk-js/#generate-presigned-urls) - [AWS SDK for PHP](/r2/examples/aws/aws-sdk-php/#generate-presigned-urls) - [AWS CLI](/r2/examples/aws/aws-cli/#generate-presigned-urls) ## Presigned URL alternative with Workers A valid alternative design to presigned URLs is to use a Worker with a [binding](/workers/runtime-apis/bindings/) that implements your security policy. :::note[Bindings] A binding is how your Worker interacts with external resources such as [KV Namespaces](/kv/concepts/kv-namespaces/), [Durable Objects](/durable-objects/), or [R2 Buckets](/r2/buckets/). A binding is a runtime variable that the Workers runtime provides to your code. You can declare a variable name in your Wrangler file that will be bound to these resources at runtime, and interact with them through this variable. Every binding's variable name and behavior is determined by you when deploying the Worker. Refer to [Environment Variables](/workers/configuration/environment-variables/) for more information. A binding is defined in the Wrangler file of your Worker project's directory. ::: A possible use case may be restricting an application to only be able to upload to a specific URL. With presigned URLs, your central signing application might look like the following JavaScript code running on Cloudflare Workers, workerd, or another platform. If the Worker received a request for `https://example.com/uploads/dog.png`, it would respond with a presigned URL allowing a user to upload to your R2 bucket at the `/uploads/dog.png` path. ```ts import { AwsClient } from "aws4fetch"; const r2 = new AwsClient({ accessKeyId: "", secretAccessKey: "", }); export default { async fetch(req): Promise { // This is just an example to demonstrating using aws4fetch to generate a presigned URL. // This Worker should not be used as-is as it does not authenticate the request, meaning // that anyone can upload to your bucket. // // Consider implementing authorization, such as a preshared secret in a request header. const requestPath = new URL(req.url).pathname; // Cannot upload to the root of a bucket if (requestPath === "/") { return new Response("Missing a filepath", { status: 400 }); } const bucketName = ""; const accountId = ""; const url = new URL( `https://${bucketName}.${accountId}.r2.cloudflarestorage.com`, ); // preserve the original path url.pathname = requestPath; // Specify a custom expiry for the presigned URL, in seconds url.searchParams.set("X-Amz-Expires", "3600"); const signed = await r2.sign( new Request(url, { method: "PUT", }), { aws: { signQuery: true }, }, ); // Caller can now use this URL to upload to that object. return new Response(signed.url, { status: 200 }); }, // ... handle other kinds of requests } satisfies ExportedHandler; ``` Notice the total absence of any configuration or token secrets present in the Worker code. Instead, in your [Wrangler configuration file](/workers/wrangler/configuration/), you would create a [binding](/r2/api/workers/workers-api-usage/#3-bind-your-bucket-to-a-worker) to whatever bucket represents the bucket you will upload to. Additionally, authorization is handled in-line with the upload which can reduce latency. In some cases, Workers lets you implement certain functionality more easily. For example, if you wanted to offer a write-once guarantee so that users can only upload to a path once, with pre-signed URLs, you would need to sign specific headers and require the sender to send them. You can modify the previous Worker to sign additional headers: ```ts const signed = await r2.sign( new Request(url, { method: "PUT", }), { aws: { signQuery: true }, headers: { "If-Unmodified-Since": "Tue, 28 Sep 2021 16:00:00 GMT", }, }, ); ``` Note that the caller has to add the same `If-Unmodified-Since` header to use the URL. The caller cannot omit the header or use a different header. If the caller uses a different header, the presigned URL signature would not match, and they would receive a `403/SignatureDoesNotMatch`. In a Worker, you would change your upload to: ```ts const existingObject = await env.DROP_BOX_BUCKET.put( url.toString().substring(1), request.body, { onlyIf: { // No objects will have been uploaded before September 28th, 2021 which // is the initial R2 announcement. uploadedBefore: new Date(1632844800000), }, }, ); if (existingObject?.etag !== request.headers.get("etag")) { return new Response("attempt to overwrite object", { status: 400 }); } ``` Cloudflare Workers currently have some limitations that you may need to consider: - You cannot upload more than 100 MiB (200 MiB for Business customers) to a Worker. - Enterprise customers can upload 500 MiB by default and can ask their account team to raise this limit. - Detecting [precondition failures](/r2/api/s3/extensions/#conditional-operations-in-putobject) is currently easier with presigned URLs as compared with R2 bindings. Note that these limitations depends on R2's extension for conditional uploads. Amazon's S3 service does not offer such functionality at this time. ## Differences between presigned URLs and public buckets Presigned URLs share some superficial similarity with public buckets. If you give out presigned URLs only for `GET`/`HEAD` operations on specific objects in a bucket, then your presigned URL functionality is mostly similar to public buckets. The notable exception is that any custom metadata associated with the object is rendered in headers with the `x-amz-meta-` prefix. Any error responses are returned as XML documents, as they would with normal non-presigned S3 access. Presigned URLs can be generated for any S3 operation. After a presigned URL is generated it can be reused as many times as the holder of the URL wants until the signed expiry date. [Public buckets](/r2/buckets/public-buckets/) are available on a regular HTTP endpoint. By default, there is no authorization or access controls associated with a public bucket. Anyone with a public bucket URL can access an object in that public bucket. If you are using a custom domain to expose the R2 bucket, you can manage authorization and access controls as you would for a Cloudflare zone. Public buckets only provide `GET`/`HEAD` on a known object path. Public bucket errors are rendered as HTML pages. Choosing between presigned URLs and public buckets is dependent on your specific use case. You can also use both if your architecture should use public buckets in one situation and presigned URLs in another. It is useful to note that presigned URLs will expose your account ID and bucket name to whoever gets a copy of the URL. Public bucket URLs do not contain the account ID or bucket name. Typically, you will not share presigned URLs directly with end users or browsers, as presigned URLs are used more for internal applications. ## Limitations Presigned URLs can only be used with the `.r2.cloudflarestorage.com` S3 API domain and cannot be used with custom domains. Instead, you can use the [general purpose HMAC validation feature of the WAF](/ruleset-engine/rules-language/functions/#hmac-validation), which requires a Pro plan or above. ## Related resources - [Create a public bucket](/r2/buckets/public-buckets/) - [Storing user generated content](/reference-architecture/diagrams/storage/storing-user-generated-content/) --- # Workers API URL: https://developers.cloudflare.com/r2/api/workers/ import { DirectoryListing } from "~/components"; --- # Workers API reference URL: https://developers.cloudflare.com/r2/api/workers/workers-api-reference/ import { Type, MetaInfo, WranglerConfig } from "~/components"; The in-Worker R2 API is accessed by binding an R2 bucket to a [Worker](/workers). The Worker you write can expose external access to buckets via a route or manipulate R2 objects internally. The R2 API includes some extensions and semantic differences from the S3 API. If you need S3 compatibility, consider using the [S3-compatible API](/r2/api/s3/). ## Concepts R2 organizes the data you store, called objects, into containers, called buckets. Buckets are the fundamental unit of performance, scaling, and access within R2. ## Create a binding :::note[Bindings] A binding is how your Worker interacts with external resources such as [KV Namespaces](/kv/concepts/kv-namespaces/), [Durable Objects](/durable-objects/), or [R2 Buckets](/r2/buckets/). A binding is a runtime variable that the Workers runtime provides to your code. You can declare a variable name in your Wrangler file that will be bound to these resources at runtime, and interact with them through this variable. Every binding's variable name and behavior is determined by you when deploying the Worker. Refer to [Environment Variables](/workers/configuration/environment-variables/) for more information. A binding is defined in the Wrangler file of your Worker project's directory. ::: To bind your R2 bucket to your Worker, add the following to your Wrangler file. Update the `binding` property to a valid JavaScript variable identifier and `bucket_name` to the name of your R2 bucket: ```toml [[r2_buckets]] binding = 'MY_BUCKET' # <~ valid JavaScript variable name bucket_name = '' ``` Within your Worker, your bucket binding is now available under the `MY_BUCKET` variable and you can begin interacting with it using the [bucket methods](#bucket-method-definitions) described below. ## Bucket method definitions The following methods are available on the bucket binding object injected into your code. For example, to issue a `PUT` object request using the binding above: ```js export default { async fetch(request, env) { const url = new URL(request.url); const key = url.pathname.slice(1); switch (request.method) { case "PUT": await env.MY_BUCKET.put(key, request.body); return new Response(`Put ${key} successfully!`); default: return new Response(`${request.method} is not allowed.`, { status: 405, headers: { Allow: "PUT", }, }); } }, }; ``` - `head` - Retrieves the `R2Object` for the given key containing only object metadata, if the key exists, and `null` if the key does not exist. - `get` - Retrieves the `R2ObjectBody` for the given key containing object metadata and the object body as a ReadableStream, if the key exists, and `null` if the key does not exist. - In the event that a precondition specified in options fails, get() returns an R2Object with body undefined. - `put` - Stores the given value and metadata under the associated key. Once the write succeeds, returns an `R2Object` containing metadata about the stored Object. - In the event that a precondition specified in options fails, put() returns `null`, and the object will not be stored. - R2 writes are strongly consistent. Once the Promise resolves, all subsequent read operations will see this key value pair globally. - `delete` - Deletes the given values and metadata under the associated keys. Once the delete succeeds, returns void. - R2 deletes are strongly consistent. Once the Promise resolves, all subsequent read operations will no longer see the provided key value pairs globally. - Up to 1000 keys may be deleted per call. - `list` * Returns an R2Objects containing a list of R2Object contained within the bucket. * The returned list of objects is ordered lexicographically. * Returns up to 1000 entries, but may return less in order to minimize memory pressure within the Worker. * To explicitly set the number of objects to list, provide an [R2ListOptions](/r2/api/workers/workers-api-reference/#r2listoptions) object with the `limit` property set. * `createMultipartUpload` - Creates a multipart upload. - Returns Promise which resolves to an `R2MultipartUpload` object representing the newly created multipart upload. Once the multipart upload has been created, the multipart upload can be immediately interacted with globally, either through the Workers API, or through the S3 API. - `resumeMultipartUpload` - Returns an object representing a multipart upload with the given key and uploadId. - The resumeMultipartUpload operation does not perform any checks to ensure the validity of the uploadId, nor does it verify the existence of a corresponding active multipart upload. This is done to minimize latency before being able to call subsequent operations on the `R2MultipartUpload` object. ## `R2Object` definition `R2Object` is created when you `PUT` an object into an R2 bucket. `R2Object` represents the metadata of an object based on the information provided by the uploader. Every object that you `PUT` into an R2 bucket will have an `R2Object` created. - `key` - The object's key. - `version` - Random unique string associated with a specific upload of a key. - `size` - Size of the object in bytes. - `etag` :::note Cloudflare recommends using the `httpEtag` field when returning an etag in a response header. This ensures the etag is quoted and conforms to [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110#section-8.8.3). ::: - The etag associated with the object upload. - `httpEtag` - The object's etag, in quotes so as to be returned as a header. - `uploaded` - A Date object representing the time the object was uploaded. - `httpMetadata` - Various HTTP headers associated with the object. Refer to [HTTP Metadata](#http-metadata). - `customMetadata` - A map of custom, user-defined metadata associated with the object. - `range` - A `R2Range` object containing the returned range of the object. - `checksums` - A `R2Checksums` object containing the stored checksums of the object. Refer to [checksums](#checksums). - `writeHttpMetadata` - Retrieves the `httpMetadata` from the `R2Object` and applies their corresponding HTTP headers to the `Headers` input object. Refer to [HTTP Metadata](#http-metadata). - `storageClass` - The storage class associated with the object. Refer to [Storage Classes](#storage-class). - `ssecKeyMd5` - Hex-encoded MD5 hash of the [SSE-C](/r2/examples/ssec) key used for encryption (if one was provided). Hash can be used to identify which key is needed to decrypt object. ## `R2ObjectBody` definition `R2ObjectBody` represents an object's metadata combined with its body. It is returned when you `GET` an object from an R2 bucket. The full list of keys for `R2ObjectBody` includes the list below and all keys inherited from [`R2Object`](#r2object-definition). - `body` - The object's value. - `bodyUsed` - Whether the object's value has been consumed or not. - `arrayBuffer` - Returns a Promise that resolves to an `ArrayBuffer` containing the object's value. - `text` - Returns a Promise that resolves to an string containing the object's value. - `json` - Returns a Promise that resolves to the given object containing the object's value. - `blob` - Returns a Promise that resolves to a binary Blob containing the object's value. ## `R2MultipartUpload` definition An `R2MultipartUpload` object is created when you call `createMultipartUpload` or `resumeMultipartUpload`. `R2MultipartUpload` is a representation of an ongoing multipart upload. Uncompleted multipart uploads will be automatically aborted after 7 days. :::note An `R2MultipartUpload` object does not guarantee that there is an active underlying multipart upload corresponding to that object. A multipart upload can be completed or aborted at any time, either through the S3 API, or by a parallel invocation of your Worker. Therefore it is important to add the necessary error handling code around each operation on a `R2MultipartUpload` object in case the underlying multipart upload no longer exists. ::: - `key` - The `key` for the multipart upload. - `uploadId` - The `uploadId` for the multipart upload. - `uploadPart` - Uploads a single part with the specified part number to this multipart upload. Each part must be uniform in size with an exception for the final part which can be smaller. - Returns an `R2UploadedPart` object containing the `etag` and `partNumber`. These `R2UploadedPart` objects are required when completing the multipart upload. - `abort` - Aborts the multipart upload. Returns a Promise that resolves when the upload has been successfully aborted. - `complete` - Completes the multipart upload with the given parts. - Returns a Promise that resolves when the complete operation has finished. Once this happens, the object is immediately accessible globally by any subsequent read operation. ## Method-specific types ### R2GetOptions - `onlyIf` - Specifies that the object should only be returned given satisfaction of certain conditions in the `R2Conditional` or in the conditional Headers. Refer to [Conditional operations](#conditional-operations). - `range` - Specifies that only a specific length (from an optional offset) or suffix of bytes from the object should be returned. Refer to [Ranged reads](#ranged-reads). - `ssecKey` - Specifies a key to be used for [SSE-C](/r2/examples/ssec). Key must be 32 bytes in length, in the form of a hex-encoded string or an ArrayBuffer. #### Ranged reads `R2GetOptions` accepts a `range` parameter, which can be used to restrict the data returned in `body`. There are 3 variations of arguments that can be used in a range: - An offset with an optional length. - An optional offset with a length. - A suffix. - `offset` - The byte to begin returning data from, inclusive. - `length` - The number of bytes to return. If more bytes are requested than exist in the object, fewer bytes than this number may be returned. - `suffix` - The number of bytes to return from the end of the file, starting from the last byte. If more bytes are requested than exist in the object, fewer bytes than this number may be returned. ### R2PutOptions - `onlyIf` - Specifies that the object should only be stored given satisfaction of certain conditions in the `R2Conditional`. Refer to [Conditional operations](#conditional-operations). - `httpMetadata` - Various HTTP headers associated with the object. Refer to [HTTP Metadata](#http-metadata). - `customMetadata` - A map of custom, user-defined metadata that will be stored with the object. :::note Only a single hashing algorithm can be specified at once. ::: - `md5` - A md5 hash to use to check the received object's integrity. - `sha1` - A SHA-1 hash to use to check the received object's integrity. - `sha256` - A SHA-256 hash to use to check the received object's integrity. - `sha384` - A SHA-384 hash to use to check the received object's integrity. - `sha512` - A SHA-512 hash to use to check the received object's integrity. - `storageClass` - Sets the storage class of the object if provided. Otherwise, the object will be stored in the default storage class associated with the bucket. Refer to [Storage Classes](#storage-class). - `ssecKey` - Specifies a key to be used for [SSE-C](/r2/examples/ssec). Key must be 32 bytes in length, in the form of a hex-encoded string or an ArrayBuffer. ### R2MultipartOptions - `httpMetadata` - Various HTTP headers associated with the object. Refer to [HTTP Metadata](#http-metadata). - `customMetadata` - A map of custom, user-defined metadata that will be stored with the object. - `storageClass` - Sets the storage class of the object if provided. Otherwise, the object will be stored in the default storage class associated with the bucket. Refer to [Storage Classes](#storage-class). - `ssecKey` - Specifies a key to be used for [SSE-C](/r2/examples/ssec). Key must be 32 bytes in length, in the form of a hex-encoded string or an ArrayBuffer. ### R2ListOptions - `limit` - The number of results to return. Defaults to `1000`, with a maximum of `1000`. - If `include` is set, you may receive fewer than `limit` results in your response to accommodate metadata. - `prefix` - The prefix to match keys against. Keys will only be returned if they start with given prefix. - `cursor` - An opaque token that indicates where to continue listing objects from. A cursor can be retrieved from a previous list operation. - `delimiter` - The character to use when grouping keys. - `include` - Can include `httpMetadata` and/or `customMetadata`. If included, items returned by the list will include the specified metadata. - Note that there is a limit on the total amount of data that a single `list` operation can return. If you request data, you may receive fewer than `limit` results in your response to accommodate metadata. - The [compatibility date](/workers/configuration/compatibility-dates/) must be set to `2022-08-04` or later in your Wrangler file. If not, then the `r2_list_honor_include` compatibility flag must be set. Otherwise it is treated as `include: ['httpMetadata', 'customMetadata']` regardless of what the `include` option provided actually is. This means applications must be careful to avoid comparing the amount of returned objects against your `limit`. Instead, use the `truncated` property to determine if the `list` request has more data to be returned. ```js const options = { limit: 500, include: ["customMetadata"], }; const listed = await env.MY_BUCKET.list(options); let truncated = listed.truncated; let cursor = truncated ? listed.cursor : undefined; // ❌ - if your limit can't fit into a single response or your // bucket has less objects than the limit, it will get stuck here. while (listed.objects.length < options.limit) { // ... } // ✅ - use the truncated property to check if there are more // objects to be returned while (truncated) { const next = await env.MY_BUCKET.list({ ...options, cursor: cursor, }); listed.objects.push(...next.objects); truncated = next.truncated; cursor = next.cursor; } ``` ### R2Objects An object containing an `R2Object` array, returned by `BUCKET_BINDING.list()`. - `objects` - An array of objects matching the `list` request. - `truncated` boolean - If true, indicates there are more results to be retrieved for the current `list` request. - `cursor` - A token that can be passed to future `list` calls to resume listing from that point. Only present if truncated is true. - `delimitedPrefixes` - If a delimiter has been specified, contains all prefixes between the specified prefix and the next occurrence of the delimiter. - For example, if no prefix is provided and the delimiter is '/', `foo/bar/baz` would return `foo` as a delimited prefix. If `foo/` was passed as a prefix with the same structure and delimiter, `foo/bar` would be returned as a delimited prefix. ### Conditional operations You can pass an `R2Conditional` object to `R2GetOptions` and `R2PutOptions`. If the condition check for `get()` fails, the body will not be returned. This will make `get()` have lower latency. If the condition check for `put()` fails, `null` will be returned instead of the `R2Object`. - `etagMatches` - Performs the operation if the object's etag matches the given string. - `etagDoesNotMatch` - Performs the operation if the object's etag does not match the given string. - `uploadedBefore` - Performs the operation if the object was uploaded before the given date. - `uploadedAfter` - Performs the operation if the object was uploaded after the given date. Alternatively, you can pass a `Headers` object containing conditional headers to `R2GetOptions` and `R2PutOptions`. For information on these conditional headers, refer to [the MDN docs on conditional requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests#conditional_headers). All conditional headers aside from `If-Range` are supported. For more specific information about conditional requests, refer to [RFC 7232](https://datatracker.ietf.org/doc/html/rfc7232). ### HTTP Metadata Generally, these fields match the HTTP metadata passed when the object was created. They can be overridden when issuing `GET` requests, in which case, the given values will be echoed back in the response. - `contentType` - `contentLanguage` - `contentDisposition` - `contentEncoding` - `cacheControl` - `cacheExpiry` ### Checksums If a checksum was provided when using the `put()` binding, it will be available on the returned object under the `checksums` property. The MD5 checksum will be included by default for non-multipart objects. - `md5` - The MD5 checksum of the object. - `sha1` - The SHA-1 checksum of the object. - `sha256` - The SHA-256 checksum of the object. - `sha384` - The SHA-384 checksum of the object. - `sha512` - The SHA-512 checksum of the object. ### `R2UploadedPart` An `R2UploadedPart` object represents a part that has been uploaded. `R2UploadedPart` objects are returned from `uploadPart` operations and must be passed to `completeMultipartUpload` operations. - `partNumber` - The number of the part. - `etag` - The `etag` of the part. ### Storage Class The storage class where an `R2Object` is stored. The available storage classes are `Standard` and `InfrequentAccess`. Refer to [Storage classes](/r2/buckets/storage-classes/) for more information. --- # Use R2 from Workers URL: https://developers.cloudflare.com/r2/api/workers/workers-api-usage/ import { Render, PackageManagers, WranglerConfig } from "~/components"; ## 1. Create a new application with C3 C3 (`create-cloudflare-cli`) is a command-line tool designed to help you set up and deploy Workers & Pages applications to Cloudflare as fast as possible. To get started, open a terminal window and run: Then, move into your newly created directory: ```sh cd r2-worker ``` ## 2. Create your bucket Create your bucket by running: ```sh npx wrangler r2 bucket create ``` To check that your bucket was created, run: ```sh npx wrangler r2 bucket list ``` After running the `list` command, you will see all bucket names, including the one you have just created. ## 3. Bind your bucket to a Worker You will need to bind your bucket to a Worker. :::note[Bindings] A binding is how your Worker interacts with external resources such as [KV Namespaces](/kv/concepts/kv-namespaces/), [Durable Objects](/durable-objects/), or [R2 Buckets](/r2/buckets/). A binding is a runtime variable that the Workers runtime provides to your code. You can declare a variable name in your Wrangler file that will be bound to these resources at runtime, and interact with them through this variable. Every binding's variable name and behavior is determined by you when deploying the Worker. Refer to the [Environment Variables](/workers/configuration/environment-variables/) documentation for more information. A binding is defined in the Wrangler file of your Worker project's directory. ::: To bind your R2 bucket to your Worker, add the following to your Wrangler file. Update the `binding` property to a valid JavaScript variable identifier and `bucket_name` to the `` you used to create your bucket in [step 2](#2-create-your-bucket): ```toml [[r2_buckets]] binding = 'MY_BUCKET' # <~ valid JavaScript variable name bucket_name = '' ``` Find more detailed information on configuring your Worker in the [Wrangler Configuration documentation](/workers/wrangler/configuration/). ## 4. Access your R2 bucket from your Worker Within your Worker code, your bucket is now available under the `MY_BUCKET` variable and you can begin interacting with it. :::caution[Local Development mode in Wrangler] By default `wrangler dev` runs in local development mode. In this mode, all operations performed by your local worker will operate against local storage on your machine. Use `wrangler dev --remote` if you want R2 operations made during development to be performed against a real R2 bucket. ::: An R2 bucket is able to READ, LIST, WRITE, and DELETE objects. You can see an example of all operations below using the Module Worker syntax. Add the following snippet into your project's `index.js` file: ```js export default { async fetch(request, env) { const url = new URL(request.url); const key = url.pathname.slice(1); switch (request.method) { case "PUT": await env.MY_BUCKET.put(key, request.body); return new Response(`Put ${key} successfully!`); case "GET": const object = await env.MY_BUCKET.get(key); if (object === null) { return new Response("Object Not Found", { status: 404 }); } const headers = new Headers(); object.writeHttpMetadata(headers); headers.set("etag", object.httpEtag); return new Response(object.body, { headers, }); case "DELETE": await env.MY_BUCKET.delete(key); return new Response("Deleted!"); default: return new Response("Method Not Allowed", { status: 405, headers: { Allow: "PUT, GET, DELETE", }, }); } }, }; ``` :::caution[Prevent potential errors when accessing request.body] The body of a [Request](https://developer.mozilla.org/en-US/docs/Web/API/Request) can only be accessed once. If you previously used `request.formData()` in the same request, you may encounter a TypeError when attempting to access `request.body`.

To avoid errors, create a clone of the Request object with `request.clone()` for each subsequent attempt to access a Request's body. Keep in mind that Workers have a [memory limit of 128MB per Worker](https://developers.cloudflare.com/workers/platform/limits#worker-limits) and loading particularly large files into a Worker's memory multiple times may reach this limit. To ensure memory usage does not reach this limit, consider using [Streams](https://developers.cloudflare.com/workers/runtime-apis/streams/). ::: ## 5. Bucket access and privacy With the above code added to your Worker, every incoming request has the ability to interact with your bucket. This means your bucket is publicly exposed and its contents can be accessed and modified by undesired actors. You must now define authorization logic to determine who can perform what actions to your bucket. This logic lives within your Worker's code, as it is your application's job to determine user privileges. The following is a short list of resources related to access and authorization practices: 1. [Basic Authentication](/workers/examples/basic-auth/): Shows how to restrict access using the HTTP Basic schema. 2. [Using Custom Headers](/workers/examples/auth-with-headers/): Allow or deny a request based on a known pre-shared key in a header. {/* */} Continuing with your newly created bucket and Worker, you will need to protect all bucket operations. For `PUT` and `DELETE` requests, you will make use of a new `AUTH_KEY_SECRET` environment variable, which you will define later as a Wrangler secret. For `GET` requests, you will ensure that only a specific file can be requested. All of this custom logic occurs inside of an `authorizeRequest` function, with the `hasValidHeader` function handling the custom header logic. If all validation passes, then the operation is allowed. ```js const ALLOW_LIST = ["cat-pic.jpg"]; // Check requests for a pre-shared secret const hasValidHeader = (request, env) => { return request.headers.get("X-Custom-Auth-Key") === env.AUTH_KEY_SECRET; }; function authorizeRequest(request, env, key) { switch (request.method) { case "PUT": case "DELETE": return hasValidHeader(request, env); case "GET": return ALLOW_LIST.includes(key); default: return false; } } export default { async fetch(request, env, ctx) { const url = new URL(request.url); const key = url.pathname.slice(1); if (!authorizeRequest(request, env, key)) { return new Response("Forbidden", { status: 403 }); } // ... }, }; ``` For this to work, you need to create a secret via Wrangler: ```sh npx wrangler secret put AUTH_KEY_SECRET ``` This command will prompt you to enter a secret in your terminal: ```sh npx wrangler secret put AUTH_KEY_SECRET ``` ```sh output Enter the secret text you'd like assigned to the variable AUTH_KEY_SECRET on the script named : ********* 🌀 Creating the secret for script name ✨ Success! Uploaded secret AUTH_KEY_SECRET. ``` This secret is now available as `AUTH_KEY_SECRET` on the `env` parameter in your Worker. ## 6. Deploy your bucket With your Worker and bucket set up, run the `npx wrangler deploy` [command](/workers/wrangler/commands/#deploy) to deploy to Cloudflare's global network: ```sh npx wrangler deploy ``` You can verify your authorization logic is working through the following commands, using your deployed Worker endpoint: :::caution When uploading files to R2 via `curl`, ensure you use **[`--data-binary`](https://everything.curl.dev/http/post/binary)** instead of `--data` or `-d`. Files will otherwise be truncated. ::: ```sh # Attempt to write an object without providing the "X-Custom-Auth-Key" header curl https://your-worker.dev/cat-pic.jpg -X PUT --data-binary 'test' #=> Forbidden # Expected because header was missing # Attempt to write an object with the wrong "X-Custom-Auth-Key" header value curl https://your-worker.dev/cat-pic.jpg -X PUT --header "X-Custom-Auth-Key: hotdog" --data-binary 'test' #=> Forbidden # Expected because header value did not match the AUTH_KEY_SECRET value # Attempt to write an object with the correct "X-Custom-Auth-Key" header value # Note: Assume that "*********" is the value of your AUTH_KEY_SECRET Wrangler secret curl https://your-worker.dev/cat-pic.jpg -X PUT --header "X-Custom-Auth-Key: *********" --data-binary 'test' #=> Put cat-pic.jpg successfully! # Attempt to read object called "foo" curl https://your-worker.dev/foo #=> Forbidden # Expected because "foo" is not in the ALLOW_LIST # Attempt to read an object called "cat-pic.jpg" curl https://your-worker.dev/cat-pic.jpg #=> test # Note: This is the value that was successfully PUT above ``` By completing this guide, you have successfully installed Wrangler and deployed your R2 bucket to Cloudflare. ## Related resources 1. [Workers Tutorials](/workers/tutorials/) 2. [Workers Examples](/workers/examples/) --- # Use the R2 multipart API from Workers URL: https://developers.cloudflare.com/r2/api/workers/workers-multipart-usage/ By following this guide, you will create a Worker through which your applications can perform multipart uploads. This example worker could serve as a basis for your own use case where you can add authentication to the worker, or even add extra validation logic when uploading each part. This guide also contains an example Python application that uploads files to this worker. This guide assumes you have set up the [R2 binding](/workers/runtime-apis/bindings/) for your Worker. Refer to [Use R2 from Workers](/r2/api/workers/workers-api-usage) for instructions on setting up an R2 binding. ## An example Worker using the multipart API The following example Worker exposes an HTTP API which enables applications to use the multipart API through the Worker. In this example, each request is routed based on the HTTP method and the action request parameter. As your Worker becomes more complicated, consider utilizing a serverless web framework such as [Hono](https://honojs.dev/) to handle the routing for you. The following example Worker includes any new information about the state of the multipart upload in the response to each request. For the request which creates the multipart upload, the `uploadId` is returned. For requests uploading a part, the part number and `etag` are returned. In turn, the client keeps track of this state, and includes the uploadId in subsequent requests, and the `etag` and part number of each part when completing a multipart upload. Add the following code to your project's `index.js` file and replace `MY_BUCKET` with your bucket's name: ```js interface Env { MY_BUCKET: R2Bucket; } export default { async fetch( request, env, ctx ): Promise { const bucket = env.MY_BUCKET; const url = new URL(request.url); const key = url.pathname.slice(1); const action = url.searchParams.get("action"); if (action === null) { return new Response("Missing action type", { status: 400 }); } // Route the request based on the HTTP method and action type switch (request.method) { case "POST": switch (action) { case "mpu-create": { const multipartUpload = await bucket.createMultipartUpload(key); return new Response( JSON.stringify({ key: multipartUpload.key, uploadId: multipartUpload.uploadId, }) ); } case "mpu-complete": { const uploadId = url.searchParams.get("uploadId"); if (uploadId === null) { return new Response("Missing uploadId", { status: 400 }); } const multipartUpload = env.MY_BUCKET.resumeMultipartUpload( key, uploadId ); interface completeBody { parts: R2UploadedPart[]; } const completeBody: completeBody = await request.json(); if (completeBody === null) { return new Response("Missing or incomplete body", { status: 400, }); } // Error handling in case the multipart upload does not exist anymore try { const object = await multipartUpload.complete(completeBody.parts); return new Response(null, { headers: { etag: object.httpEtag, }, }); } catch (error: any) { return new Response(error.message, { status: 400 }); } } default: return new Response(`Unknown action ${action} for POST`, { status: 400, }); } case "PUT": switch (action) { case "mpu-uploadpart": { const uploadId = url.searchParams.get("uploadId"); const partNumberString = url.searchParams.get("partNumber"); if (partNumberString === null || uploadId === null) { return new Response("Missing partNumber or uploadId", { status: 400, }); } if (request.body === null) { return new Response("Missing request body", { status: 400 }); } const partNumber = parseInt(partNumberString); const multipartUpload = env.MY_BUCKET.resumeMultipartUpload( key, uploadId ); try { const uploadedPart: R2UploadedPart = await multipartUpload.uploadPart(partNumber, request.body); return new Response(JSON.stringify(uploadedPart)); } catch (error: any) { return new Response(error.message, { status: 400 }); } } default: return new Response(`Unknown action ${action} for PUT`, { status: 400, }); } case "GET": if (action !== "get") { return new Response(`Unknown action ${action} for GET`, { status: 400, }); } const object = await env.MY_BUCKET.get(key); if (object === null) { return new Response("Object Not Found", { status: 404 }); } const headers = new Headers(); object.writeHttpMetadata(headers); headers.set("etag", object.httpEtag); return new Response(object.body, { headers }); case "DELETE": switch (action) { case "mpu-abort": { const uploadId = url.searchParams.get("uploadId"); if (uploadId === null) { return new Response("Missing uploadId", { status: 400 }); } const multipartUpload = env.MY_BUCKET.resumeMultipartUpload( key, uploadId ); try { multipartUpload.abort(); } catch (error: any) { return new Response(error.message, { status: 400 }); } return new Response(null, { status: 204 }); } case "delete": { await env.MY_BUCKET.delete(key); return new Response(null, { status: 204 }); } default: return new Response(`Unknown action ${action} for DELETE`, { status: 400, }); } default: return new Response("Method Not Allowed", { status: 405, headers: { Allow: "PUT, POST, GET, DELETE" }, }); } }, } satisfies ExportedHandler; ``` After you have updated your Worker with the above code, run `npx wrangler deploy`. You can now use this Worker to perform multipart uploads. You can either send requests from your existing application to this Worker to perform uploads or use a script to upload files through this Worker. The next section is optional and shows an example of a Python script which uploads a chosen file on your machine to your Worker. ## Perform a multipart upload with your Worker (optional) This example application uploads a local file to the Worker in multiple parts. It uses Python's built-in `ThreadPoolExecutor` to parallelize the uploading of parts to the Worker, which increases upload speeds. HTTP requests to the Worker are made with the [requests](https://pypi.org/project/requests/) library. Utilizing the multipart API in this way also allows you to use your Worker to upload files larger than the [Workers request body size limit](/workers/platform/limits#request-limits). The uploading of individual parts is still subject to this limit. Save the following code in a file named `mpuscript.py` on your local machine. Change the `worker_endpoint variable` to where your worker is deployed. Pass the file you want to upload as an argument when running this script: `python3 mpuscript.py myfile`. This will upload the file `myfile` from your machine to your bucket through the Worker. ```python import math import os import requests from requests.adapters import HTTPAdapter, Retry import sys import concurrent.futures # Take the file to upload as an argument filename = sys.argv[1] # The endpoint for our worker, change this to wherever you deploy your worker worker_endpoint = "https://myworker.myzone.workers.dev/" # Configure the part size to be 10MB. 5MB is the minimum part size, except for the last part partsize = 10 * 1024 * 1024 def upload_file(worker_endpoint, filename, partsize): url = f"{worker_endpoint}{filename}" # Create the multipart upload uploadId = requests.post(url, params={"action": "mpu-create"}).json()["uploadId"] part_count = math.ceil(os.stat(filename).st_size / partsize) # Create an executor for up to 25 concurrent uploads. executor = concurrent.futures.ThreadPoolExecutor(25) # Submit a task to the executor to upload each part futures = [ executor.submit(upload_part, filename, partsize, url, uploadId, index) for index in range(part_count) ] concurrent.futures.wait(futures) # get the parts from the futures uploaded_parts = [future.result() for future in futures] # complete the multipart upload response = requests.post( url, params={"action": "mpu-complete", "uploadId": uploadId}, json={"parts": uploaded_parts}, ) if response.status_code == 200: print("🎉 successfully completed multipart upload") else: print(response.text) def upload_part(filename, partsize, url, uploadId, index): # Open the file in rb mode, which treats it as raw bytes rather than attempting to parse utf-8 with open(filename, "rb") as file: file.seek(partsize * index) part = file.read(partsize) # Retry policy for when uploading a part fails s = requests.Session() retries = Retry(total=3, status_forcelist=[400, 500, 502, 503, 504]) s.mount("https://", HTTPAdapter(max_retries=retries)) return s.put( url, params={ "action": "mpu-uploadpart", "uploadId": uploadId, "partNumber": str(index + 1), }, data=part, ).json() upload_file(worker_endpoint, filename, partsize) ``` ## State management The stateful nature of multipart uploads does not easily map to the usage model of Workers, which are inherently stateless. In a normal multipart upload, the multipart upload is usually performed in one continuous execution of the client application. This is different from multipart uploads in a Worker, which will often be completed over multiple invocations of that Worker. This makes state management more challenging. To overcome this, the state associated with a multipart upload, namely the `uploadId` and which parts have been uploaded, needs to be kept track of somewhere outside of the Worker. In the example Worker and Python application described in this guide, the state of the multipart upload is tracked in the client application which sends requests to the Worker, with the necessary state contained in each request. Keeping track of the multipart state in the client application enables maximal flexibility and allows for parallel and unordered uploads of each part. When keeping track of this state in the client is impossible, alternative designs can be considered. For example, you could track the `uploadId` and which parts have been uploaded in a Durable Object or other database. --- # aws CLI URL: https://developers.cloudflare.com/r2/examples/aws/aws-cli/ import { Render } from "~/components";
With the [`aws`](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) CLI installed, you may run [`aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config) to configure a new profile. You will be prompted with a series of questions for the new profile's details. :::note[Compatibility] Client versions `2.23.0` and `1.37.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `2.22.35` or `1.36.40`, or alternatively, add the CRC32 checksum flag to the cli command: ```sh aws s3api put-object --bucket sdk-example --key sdk.png --body file/path --checksum-algorithm CRC32 ``` ::: ```shell aws configure ``` ```sh output AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [None]: auto Default output format [None]: json ``` You may then use the `aws` CLI for any of your normal workflows. ```sh aws s3api list-buckets --endpoint-url https://.r2.cloudflarestorage.com # { # "Buckets": [ # { # "Name": "sdk-example", # "CreationDate": "2022-05-18T17:19:59.645000+00:00" # } # ], # "Owner": { # "DisplayName": "134a5a2c0ba47b38eada4b9c8ead10b6", # "ID": "134a5a2c0ba47b38eada4b9c8ead10b6" # } # } aws s3api list-objects-v2 --endpoint-url https://.r2.cloudflarestorage.com --bucket sdk-example # { # "Contents": [ # { # "Key": "ferriswasm.png", # "LastModified": "2022-05-18T17:20:21.670000+00:00", # "ETag": "\"eb2b891dc67b81755d2b726d9110af16\"", # "Size": 87671, # "StorageClass": "STANDARD" # } # ] # } ``` ## Generate presigned URLs You can also generate presigned links which allow you to share public access to a file temporarily. ```sh # You can pass the --expires-in flag to determine how long the presigned link is valid. $ aws s3 presign --endpoint-url https://.r2.cloudflarestorage.com s3://sdk-example/ferriswasm.png --expires-in 3600 # https://.r2.cloudflarestorage.com/sdk-example/ferriswasm.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature= aws s3 presign --endpoint-url https://.r2.cloudflarestorage.com s3://sdk-example/ferriswasm.png --expires-in 3600 # https://.r2.cloudflarestorage.com/sdk-example/ferriswasm.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature= ``` --- # aws-sdk-go URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-go/ import { Render } from "~/components";
This example uses version 2 of the [aws-sdk-go](https://github.com/aws/aws-sdk-go-v2) package. You must pass in the R2 configuration credentials when instantiating your `S3` service client: :::note[Compatibility] Client version `1.73.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `1.72.3` or add the following to their config: ```go config.WithRequestChecksumCalculation(0) config.WithResponseChecksumValidation(0) ``` ::: ```go package main import ( "context" "encoding/json" "fmt" "github.com/aws/aws-sdk-go-v2/aws" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/credentials" "github.com/aws/aws-sdk-go-v2/service/s3" "log" ) func main() { var bucketName = "sdk-example" var accountId = "" var accessKeyId = "" var accessKeySecret = "" cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKeyId, accessKeySecret, "")), config.WithRegion("auto"), ) if err != nil { log.Fatal(err) } client := s3.NewFromConfig(cfg, func(o *s3.Options) { o.BaseEndpoint = aws.String(fmt.Sprintf("https://%s.r2.cloudflarestorage.com", accountId)) }) listObjectsOutput, err := client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{ Bucket: &bucketName, }) if err != nil { log.Fatal(err) } for _, object := range listObjectsOutput.Contents { obj, _ := json.MarshalIndent(object, "", "\t") fmt.Println(string(obj)) } // { // "ChecksumAlgorithm": null, // "ETag": "\"eb2b891dc67b81755d2b726d9110af16\"", // "Key": "ferriswasm.png", // "LastModified": "2022-05-18T17:20:21.67Z", // "Owner": null, // "Size": 87671, // "StorageClass": "STANDARD" // } listBucketsOutput, err := client.ListBuckets(context.TODO(), &s3.ListBucketsInput{}) if err != nil { log.Fatal(err) } for _, object := range listBucketsOutput.Buckets { obj, _ := json.MarshalIndent(object, "", "\t") fmt.Println(string(obj)) } // { // "CreationDate": "2022-05-18T17:19:59.645Z", // "Name": "sdk-example" // } } ``` ## Generate presigned URLs You can also generate presigned links that can be used to temporarily share public write access to a bucket. ```go presignClient := s3.NewPresignClient(client) presignResult, err := presignClient.PresignPutObject(context.TODO(), &s3.PutObjectInput{ Bucket: aws.String(bucketName), Key: aws.String("example.txt"), }) if err != nil { panic("Couldn't get presigned URL for PutObject") } fmt.Printf("Presigned URL For object: %s\n", presignResult.URL) ``` --- # aws-sdk-java URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-java/ import { Render } from "~/components";
This example uses version 2 of the [aws-sdk-java](https://github.com/aws/aws-sdk-java-v2/#using-the-sdk) package. You must pass in the R2 configuration credentials when instantiating your `S3` service client: :::note[Compatibility] Client version `2.30.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `2.29.52` or add the following to their S3Config: ```java this.requestChecksumCalculation = "when_required", this.responseChecksumValidation = "when_required" ``` ::: ```java import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.*; import software.amazon.awssdk.services.s3.S3Configuration; import java.net.URI; import java.util.List; /** * Client for interacting with Cloudflare R2 Storage using AWS SDK S3 compatibility */ public class CloudflareR2Client { private final S3Client s3Client; /** * Creates a new CloudflareR2Client with the provided configuration */ public CloudflareR2Client(S3Config config) { this.s3Client = buildS3Client(config); } /** * Configuration class for R2 credentials and endpoint */ public static class S3Config { private final String accountId; private final String accessKey; private final String secretKey; private final String endpoint; public S3Config(String accountId, String accessKey, String secretKey) { this.accountId = accountId; this.accessKey = accessKey; this.secretKey = secretKey; this.endpoint = String.format("https://%s.r2.cloudflarestorage.com", accountId); } public String getAccessKey() { return accessKey; } public String getSecretKey() { return secretKey; } public String getEndpoint() { return endpoint; } } /** * Builds and configures the S3 client with R2-specific settings */ private static S3Client buildS3Client(S3Config config) { AwsBasicCredentials credentials = AwsBasicCredentials.create( config.getAccessKey(), config.getSecretKey() ); S3Configuration serviceConfiguration = S3Configuration.builder() .pathStyleAccessEnabled(true) .build(); return S3Client.builder() .endpointOverride(URI.create(config.getEndpoint())) .credentialsProvider(StaticCredentialsProvider.create(credentials)) .region(Region.of("auto")) .serviceConfiguration(serviceConfiguration) .build(); } /** * Lists all buckets in the R2 storage */ public List listBuckets() { try { return s3Client.listBuckets().buckets(); } catch (S3Exception e) { throw new RuntimeException("Failed to list buckets: " + e.getMessage(), e); } } /** * Lists all objects in the specified bucket */ public List listObjects(String bucketName) { try { ListObjectsV2Request request = ListObjectsV2Request.builder() .bucket(bucketName) .build(); return s3Client.listObjectsV2(request).contents(); } catch (S3Exception e) { throw new RuntimeException("Failed to list objects in bucket " + bucketName + ": " + e.getMessage(), e); } } public static void main(String[] args) { S3Config config = new S3Config( "your_account_id", "your_access_key", "your_secret_key" ); CloudflareR2Client r2Client = new CloudflareR2Client(config); // List buckets System.out.println("Available buckets:"); r2Client.listBuckets().forEach(bucket -> System.out.println("* " + bucket.name()) ); // List objects in a specific bucket String bucketName = "demos"; System.out.println("\nObjects in bucket '" + bucketName + "':"); r2Client.listObjects(bucketName).forEach(object -> System.out.printf("* %s (size: %d bytes, modified: %s)%n", object.key(), object.size(), object.lastModified()) ); } } ``` ## Generate presigned URLs You can also generate presigned links that can be used to temporarily share public write access to a bucket. ```java // import required packages for presigning // Rest of the packages are same as above import software.amazon.awssdk.services.s3.presigner.S3Presigner; import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest; import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest; import java.time.Duration; public class CloudflareR2Client { private final S3Client s3Client; private final S3Presigner presigner; /** * Creates a new CloudflareR2Client with the provided configuration */ public CloudflareR2Client(S3Config config) { this.s3Client = buildS3Client(config); this.presigner = buildS3Presigner(config); } /** * Builds and configures the S3 presigner with R2-specific settings */ private static S3Presigner buildS3Presigner(S3Config config) { AwsBasicCredentials credentials = AwsBasicCredentials.create( config.getAccessKey(), config.getSecretKey() ); return S3Presigner.builder() .endpointOverride(URI.create(config.getEndpoint())) .credentialsProvider(StaticCredentialsProvider.create(credentials)) .region(Region.of("auto")) .serviceConfiguration(S3Configuration.builder() .pathStyleAccessEnabled(true) .build()) .build(); } public String generatePresignedUploadUrl(String bucketName, String objectKey, Duration expiration) { PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder() .signatureDuration(expiration) .putObjectRequest(builder -> builder .bucket(bucketName) .key(objectKey) .build()) .build(); PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest); return presignedRequest.url().toString(); } // Rest of the methods remains the same public static void main(String[] args) { // config the client as before // Generate a pre-signed upload URL valid for 15 minutes String uploadUrl = r2Client.generatePresignedUploadUrl( "demos", "README.md", Duration.ofMinutes(15) ); System.out.println("Pre-signed Upload URL (valid for 15 minutes):"); System.out.println(uploadUrl); } } ``` --- # aws-sdk-js-v3 URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-js-v3/ import { Render } from "~/components";
JavaScript or TypeScript users may continue to use the [`@aws-sdk/client-s3`](https://www.npmjs.com/package/@aws-sdk/client-s3) npm package as per normal. You must pass in the R2 configuration credentials when instantiating your `S3` service client: :::note[Compatibility] Client version `3.729.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `3.726.1` or add the following to their S3Client config: ```ts requestChecksumCalculation: "WHEN_REQUIRED", responseChecksumValidation: "WHEN_REQUIRED", ``` ::: ```ts import { S3Client, ListBucketsCommand, ListObjectsV2Command, GetObjectCommand, PutObjectCommand, } from "@aws-sdk/client-s3"; const S3 = new S3Client({ region: "auto", endpoint: `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`, credentials: { accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET_ACCESS_KEY, }, }); console.log(await S3.send(new ListBucketsCommand({}))); // { // '$metadata': { // httpStatusCode: 200, // requestId: undefined, // extendedRequestId: undefined, // cfId: undefined, // attempts: 1, // totalRetryDelay: 0 // }, // Buckets: [ // { Name: 'user-uploads', CreationDate: 2022-04-13T21:23:47.102Z }, // { Name: 'my-bucket-name', CreationDate: 2022-05-07T02:46:49.218Z } // ], // Owner: { // DisplayName: '...', // ID: '...' // } // } console.log( await S3.send(new ListObjectsV2Command({ Bucket: "my-bucket-name" })), ); // { // '$metadata': { // httpStatusCode: 200, // requestId: undefined, // extendedRequestId: undefined, // cfId: undefined, // attempts: 1, // totalRetryDelay: 0 // }, // CommonPrefixes: undefined, // Contents: [ // { // Key: 'cat.png', // LastModified: 2022-05-07T02:50:45.616Z, // ETag: '"c4da329b38467509049e615c11b0c48a"', // ChecksumAlgorithm: undefined, // Size: 751832, // StorageClass: 'STANDARD', // Owner: undefined // }, // { // Key: 'todos.txt', // LastModified: 2022-05-07T21:37:17.150Z, // ETag: '"29d911f495d1ba7cb3a4d7d15e63236a"', // ChecksumAlgorithm: undefined, // Size: 279, // StorageClass: 'STANDARD', // Owner: undefined // } // ], // ContinuationToken: undefined, // Delimiter: undefined, // EncodingType: undefined, // IsTruncated: false, // KeyCount: 8, // MaxKeys: 1000, // Name: 'my-bucket-name', // NextContinuationToken: undefined, // Prefix: undefined, // StartAfter: undefined // } ``` ## Generate presigned URLs You can also generate presigned links that can be used to share public read or write access to a bucket temporarily. ```ts import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; // Use the expiresIn property to determine how long the presigned link is valid. console.log( await getSignedUrl( S3, new GetObjectCommand({ Bucket: "my-bucket-name", Key: "dog.png" }), { expiresIn: 3600 }, ), ); // https://my-bucket-name..r2.cloudflarestorage.com/dog.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-Signature=&X-Amz-SignedHeaders=host&x-id=GetObject // You can also create links for operations such as putObject to allow temporary write access to a specific key. console.log( await getSignedUrl( S3, new PutObjectCommand({ Bucket: "my-bucket-name", Key: "dog.png" }), { expiresIn: 3600 }, ), ); ``` You can use the link generated by the `putObject` example to upload to the specified bucket and key, until the presigned link expires. ```sh curl -X PUT https://my-bucket-name..r2.cloudflarestorage.com/dog.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-Signature=&X-Amz-SignedHeaders=host&x-id=PutObject -F "data=@dog.png" ``` --- # aws-sdk-js URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-js/ import { Render } from "~/components";
If you are interested in the newer version of the AWS JavaScript SDK visit this [dedicated aws-sdk-js-v3 example page](/r2/examples/aws/aws-sdk-js-v3/). JavaScript or TypeScript users may continue to use the [`aws-sdk`](https://www.npmjs.com/package/aws-sdk) npm package as per normal. You must pass in the R2 configuration credentials when instantiating your `S3` service client: ```ts import S3 from "aws-sdk/clients/s3.js"; const s3 = new S3({ endpoint: `https://${accountid}.r2.cloudflarestorage.com`, accessKeyId: `${access_key_id}`, secretAccessKey: `${access_key_secret}`, signatureVersion: "v4", }); console.log(await s3.listBuckets().promise()); //=> { //=> Buckets: [ //=> { Name: 'user-uploads', CreationDate: 2022-04-13T21:23:47.102Z }, //=> { Name: 'my-bucket-name', CreationDate: 2022-05-07T02:46:49.218Z } //=> ], //=> Owner: { //=> DisplayName: '...', //=> ID: '...' //=> } //=> } console.log(await s3.listObjects({ Bucket: "my-bucket-name" }).promise()); //=> { //=> IsTruncated: false, //=> Name: 'my-bucket-name', //=> CommonPrefixes: [], //=> MaxKeys: 1000, //=> Contents: [ //=> { //=> Key: 'cat.png', //=> LastModified: 2022-05-07T02:50:45.616Z, //=> ETag: '"c4da329b38467509049e615c11b0c48a"', //=> ChecksumAlgorithm: [], //=> Size: 751832, //=> Owner: [Object] //=> }, //=> { //=> Key: 'todos.txt', //=> LastModified: 2022-05-07T21:37:17.150Z, //=> ETag: '"29d911f495d1ba7cb3a4d7d15e63236a"', //=> ChecksumAlgorithm: [], //=> Size: 279, //=> Owner: [Object] //=> } //=> ] //=> } ``` ## Generate presigned URLs You can also generate presigned links that can be used to share public read or write access to a bucket temporarily. ```ts // Use the expires property to determine how long the presigned link is valid. console.log( await s3.getSignedUrlPromise("getObject", { Bucket: "my-bucket-name", Key: "dog.png", Expires: 3600, }), ); // https://my-bucket-name..r2.cloudflarestorage.com/dog.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-Signature=&X-Amz-SignedHeaders=host // You can also create links for operations such as putObject to allow temporary write access to a specific key. console.log( await s3.getSignedUrlPromise("putObject", { Bucket: "my-bucket-name", Key: "dog.png", Expires: 3600, }), ); ``` You can use the link generated by the `putObject` example to upload to the specified bucket and key, until the presigned link expires. ```sh curl -X PUT https://my-bucket-name..r2.cloudflarestorage.com/dog.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-Signature=&X-Amz-SignedHeaders=host --data-binary @dog.png ``` --- # aws-sdk-net URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-net/ import { Render } from "~/components";
This example uses version 3 of the [aws-sdk-net](https://www.nuget.org/packages/AWSSDK.S3) package. You must pass in the R2 configuration credentials when instantiating your `S3` service client: ## Client setup In this example, you will pass credentials explicitly to the `IAmazonS3` initialization. If you wish, use a shared AWS credentials file or the SDK store in-line with other AWS SDKs. Refer to [Configure AWS credentials](https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/net-dg-config-creds.html) for more details. :::note[Compatibility] Client version `3.7.963.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `3.7.962.0` or add the following to their AmazonS3Config: ```csharp RequestChecksumCalculation = "WHEN_REQUIRED", ResponseChecksumValidation = "WHEN_REQUIRED" ``` ::: ```csharp private static IAmazonS3 s3Client; public static void Main(string[] args) { var accessKey = ""; var secretKey = ""; var credentials = new BasicAWSCredentials(accessKey, secretKey); s3Client = new AmazonS3Client(credentials, new AmazonS3Config { ServiceURL = "https://.r2.cloudflarestorage.com", }); } ``` ## List buckets and objects The [ListBucketsAsync](https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MIS3ListBucketsAsyncListBucketsRequestCancellationToken.html) and [ListObjectsAsync](https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MIS3ListObjectsV2AsyncListObjectsV2RequestCancellationToken.html) methods can be used to list buckets under your account and the contents of those buckets respectively. ```csharp static async Task ListBuckets() { var response = await s3Client.ListBucketsAsync(); foreach (var s3Bucket in response.Buckets) { Console.WriteLine("{0}", s3Bucket.BucketName); } } // sdk-example // my-bucket-name ``` ```csharp static async Task ListObjectsV2() { var request = new ListObjectsV2Request { BucketName = "sdk-example" }; var response = await s3Client.ListObjectsV2Async(request); foreach (var s3Object in response.S3Objects) { Console.WriteLine("{0}", s3Object.Key); } } // dog.png // cat.png ``` ## Upload and retrieve objects The [PutObjectAsync](https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MIS3PutObjectAsyncPutObjectRequestCancellationToken.html) and [GetObjectAsync](https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MIS3GetObjectAsyncStringStringCancellationToken.html) methods can be used to upload objects and download objects from an R2 bucket respectively. :::caution `DisablePayloadSigning = true` must be passed as Cloudflare R2 does not currently support the Streaming SigV4 implementation used by AWSSDK.S3. ::: ```csharp static async Task PutObject() { var request = new PutObjectRequest { FilePath = @"/path/file.txt", BucketName = "sdk-example", DisablePayloadSigning = true }; var response = await s3Client.PutObjectAsync(request); Console.WriteLine("ETag: {0}", response.ETag); } // ETag: "186a71ee365d9686c3b98b6976e1f196" ``` ```csharp static async Task GetObject() { var bucket = "sdk-example"; var key = "file.txt" var response = await s3Client.GetObjectAsync(bucket, key); Console.WriteLine("ETag: {0}", response.ETag); } // ETag: "186a71ee365d9686c3b98b6976e1f196" ``` ## Generate presigned URLs The [GetPreSignedURL](https://docs.aws.amazon.com/sdkfornet/v3/apidocs/items/S3/MIS3GetPreSignedURLGetPreSignedUrlRequest.html) method allows you to sign ahead of time, giving temporary access to a specific operation. In this case, presigning a `PutObject` request for `sdk-example/file.txt`. ```csharp static string? GeneratePresignedUrl() { AWSConfigsS3.UseSignatureVersion4 = true; var presign = new GetPreSignedUrlRequest { BucketName = "sdk-example", Key = "file.txt", Verb = HttpVerb.GET, Expires = DateTime.Now.AddDays(7), }; var presignedUrl = s3Client.GetPreSignedURL(presign); Console.WriteLine(presignedUrl); return presignedUrl; } // URL: https://.r2.cloudflarestorage.com/sdk-example/file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature= ``` --- # aws-sdk-php URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-php/ import { Render } from "~/components";
This example uses version 3 of the [aws-sdk-php](https://packagist.org/packages/aws/aws-sdk-php) package. You must pass in the R2 configuration credentials when instantiating your `S3` service client: :::note[Compatibility] Client version `3.337.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `3.336.15` or add the following to their $options: ```php 'request_checksum_calculation' => 'when_required', 'response_checksum_validation' => 'when_required' ``` ::: ```php "; $access_key_id = ""; $access_key_secret = ""; $credentials = new Aws\Credentials\Credentials($access_key_id, $access_key_secret); $options = [ 'region' => 'auto', 'endpoint' => "https://$account_id.r2.cloudflarestorage.com", 'version' => 'latest', 'credentials' => $credentials ]; $s3_client = new Aws\S3\S3Client($options); $contents = $s3_client->listObjectsV2([ 'Bucket' => $bucket_name ]); var_dump($contents['Contents']); // array(1) { // [0]=> // array(5) { // ["Key"]=> // string(14) "ferriswasm.png" // ["LastModified"]=> // object(Aws\Api\DateTimeResult)#187 (3) { // ["date"]=> // string(26) "2022-05-18 17:20:21.670000" // ["timezone_type"]=> // int(2) // ["timezone"]=> // string(1) "Z" // } // ["ETag"]=> // string(34) ""eb2b891dc67b81755d2b726d9110af16"" // ["Size"]=> // string(5) "87671" // ["StorageClass"]=> // string(8) "STANDARD" // } // } $buckets = $s3_client->listBuckets(); var_dump($buckets['Buckets']); // array(1) { // [0]=> // array(2) { // ["Name"]=> // string(11) "sdk-example" // ["CreationDate"]=> // object(Aws\Api\DateTimeResult)#212 (3) { // ["date"]=> // string(26) "2022-05-18 17:19:59.645000" // ["timezone_type"]=> // int(2) // ["timezone"]=> // string(1) "Z" // } // } // } ?> ``` ## Generate presigned URLs You can also generate presigned links that can be used to share public read or write access to a bucket temporarily. ```php $cmd = $s3_client->getCommand('GetObject', [ 'Bucket' => $bucket_name, 'Key' => 'ferriswasm.png' ]); // The second parameter allows you to determine how long the presigned link is valid. $request = $s3_client->createPresignedRequest($cmd, '+1 hour'); print_r((string)$request->getUri()) // https://sdk-example..r2.cloudflarestorage.com/ferriswasm.png?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Signature= // You can also create links for operations such as putObject to allow temporary write access to a specific key. $cmd = $s3_client->getCommand('PutObject', [ 'Bucket' => $bucket_name, 'Key' => 'ferriswasm.png' ]); $request = $s3_client->createPresignedRequest($cmd, '+1 hour'); print_r((string)$request->getUri()) ``` You can use the link generated by the `putObject` example to upload to the specified bucket and key, until the presigned link expires. ```sh curl -X PUT https://sdk-example..r2.cloudflarestorage.com/ferriswasm.png?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-Date=&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Signature= --data-binary @ferriswasm.png ``` --- # aws-sdk-ruby URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-ruby/ import { Render } from "~/components";
Many Ruby projects also store these credentials in environment variables instead. :::note[Compatibility] Client version `1.178.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `1.177.0` or add the following to their s3 client instantiation: ```ruby request_checksum_calculation: "when_required", response_checksum_validation: "when_required" ``` ::: Add the following dependency to your `Gemfile`: ```ruby gem "aws-sdk-s3" ``` Then you can use Ruby to operate on R2 buckets: ```ruby require "aws-sdk-s3" @r2 = Aws::S3::Client.new( access_key_id: "#{access_key_id}", secret_access_key: "#{secret_access_key}", endpoint: "https://#{cloudflare_account_id}.r2.cloudflarestorage.com", region: "auto", ) # List all buckets on your account puts @r2.list_buckets #=> { #=> :buckets => [{ #=> :name => "your-bucket", #=> :creation_date => "…", #=> }], #=> :owner => { #=> :display_name => "…", #=> :id => "…" #=> } #=> } # List the first 20 items in a bucket puts @r2.list_objects(bucket:"your-bucket", max_keys:20) #=> { #=> :is_truncated => false, #=> :marker => nil, #=> :next_marker => nil, #=> :name => "your-bucket", #=> :prefix => nil, #=> :delimiter =>nil, #=> :max_keys => 20, #=> :common_prefixes => [], #=> :encoding_type => nil #=> :contents => [ #=> …, #=> …, #=> …, #=> ] #=> } ``` --- # aws-sdk-rust URL: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-rust/ import { Render } from "~/components";
This example uses the [aws-sdk-s3](https://crates.io/crates/aws-sdk-s3) crate from the [AWS SDK for Rust](https://github.com/awslabs/aws-sdk-rust). You must pass in the R2 configuration credentials when instantiating your `S3` client: :::note[Compatibility] Client versions may introduce modifications to the default checksum behavior that could be incompatible with R2 APIs. If you encounter checksum-related errors, add the following to your config: ```rust .request_checksum_calculation(RequestChecksumCalculation::WhenRequired) .response_checksum_validation(ResponseChecksumValidation::WhenRequired) ``` ::: ## Basic Usage ```rust use aws_sdk_s3 as s3; use aws_smithy_types::date_time::Format::DateTime; #[tokio::main] async fn main() -> Result<(), s3::Error> { let bucket_name = "sdk-example"; let account_id = ""; let access_key_id = ""; let access_key_secret = ""; // Configure the client let config = aws_config::from_env() .endpoint_url(format!("https://{}.r2.cloudflarestorage.com", account_id)) .credentials_provider(aws_sdk_s3::config::Credentials::new( access_key_id, access_key_secret, None, // session token is not used with R2 None, "R2", )) .region("auto") .load() .await; let client = s3::Client::new(&config); // List buckets let list_buckets_output = client.list_buckets().send().await?; println!("Buckets:"); for bucket in list_buckets_output.buckets() { println!(" - {}: {}", bucket.name().unwrap_or_default(), bucket.creation_date().map_or_else( || "Unknown creation date".to_string(), |date| date.fmt(DateTime).unwrap() ) ); } // List objects in a specific bucket let list_objects_output = client .list_objects_v2() .bucket(bucket_name) .send() .await?; println!("\nObjects in {}:", bucket_name); for object in list_objects_output.contents() { println!(" - {}: {} bytes, last modified: {}", object.key().unwrap_or_default(), object.size().unwrap_or_default(), object.last_modified().map_or_else( || "Unknown".to_string(), |date| date.fmt(DateTime).unwrap() ) ); } Ok(()) } ``` ## Upload Objects To upload an object to R2: ```rust use aws_sdk_s3::primitives::ByteStream; use std::path::Path; async fn upload_object( client: &s3::Client, bucket: &str, key: &str, file_path: &str, ) -> Result<(), s3::Error> { let body = ByteStream::from_path(Path::new(file_path)).await.unwrap(); client .put_object() .bucket(bucket) .key(key) .body(body) .send() .await?; println!("Uploaded {} to {}/{}", file_path, bucket, key); Ok(()) } ``` ## Download Objects To download an object from R2: ```rust use std::fs; use std::io::Write; async fn download_object( client: &s3::Client, bucket: &str, key: &str, output_path: &str, ) -> Result<(), Box> { let resp = client .get_object() .bucket(bucket) .key(key) .send() .await?; let data = resp.body.collect().await?; let bytes = data.into_bytes(); let mut file = fs::File::create(output_path)?; file.write_all(&bytes)?; println!("Downloaded {}/{} to {}", bucket, key, output_path); Ok(()) } ``` ## Generate Presigned URLs You can also generate presigned links that can be used to temporarily share public read or write access to a bucket. ```rust use aws_sdk_s3::presigning::PresigningConfig; use std::time::Duration; async fn generate_get_presigned_url( client: &s3::Client, bucket: &str, key: &str, expires_in: Duration, ) -> Result { let presigning_config = PresigningConfig::expires_in(expires_in)?; // Generate a presigned URL for GET (download) let presigned_get_request = client .get_object() .bucket(bucket) .key(key) .presigned(presigning_config) .await?; Ok(presigned_get_request.uri().to_string()) } async fn generate_upload_presigned_url( client: &s3::Client, bucket: &str, key: &str, expires_in: Duration, ) -> Result { let presigning_config = PresigningConfig::expires_in(expires_in)?; // Generate a presigned URL for PUT (upload) let presigned_put_request = client .put_object() .bucket(bucket) .key(key) .presigned(presigning_config) .await?; Ok(presigned_put_request.uri().to_string()) } ``` You can use these presigned URLs with any HTTP client. For example, to upload a file using the PUT URL: ```bash curl -X PUT "https://" -H "Content-Type: application/octet-stream" --data-binary "@local-file.txt" ``` To download a file using the GET URL: ```bash curl -X GET "https://" -o downloaded-file.txt ``` --- # aws4fetch URL: https://developers.cloudflare.com/r2/examples/aws/aws4fetch/ import { Render } from "~/components";
JavaScript or TypeScript users may continue to use the [`aws4fetch`](https://www.npmjs.com/package/aws4fetch) npm package as per normal. This package uses the `fetch` and `SubtleCrypto` APIs which you will be familiar with when working in browsers or with Cloudflare Workers. You must pass in the R2 configuration credentials when instantiating your `S3` service client: ```ts import { AwsClient } from "aws4fetch"; const R2_URL = `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`; const client = new AwsClient({ accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET_ACCESS_KEY, }); const ListBucketsResult = await client.fetch(R2_URL); console.log(await ListBucketsResult.text()); // // // // 2022-04-13T21:23:47.102Z // user-uploads // // // 2022-05-07T02:46:49.218Z // my-bucket-name // // // // ... // ... // // const ListObjectsV2Result = await client.fetch( `${R2_URL}/my-bucket-name?list-type=2`, ); console.log(await ListObjectsV2Result.text()); // // my-bucket-name // // cat.png // 751832 // 2022-05-07T02:50:45.616Z // "c4da329b38467509049e615c11b0c48a" // STANDARD // // // todos.txt // 278 // 2022-05-07T21:37:17.150Z // "29d911f495d1ba7cb3a4d7d15e63236a" // STANDARD // // false // 1000 // 2 // ``` ## Generate presigned URLs You can also generate presigned links that can be used to share public read or write access to a bucket temporarily. ```ts import { AwsClient } from "aws4fetch"; const client = new AwsClient({ service: "s3", region: "auto", accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET_ACCESS_KEY, }); const R2_URL = `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`; // Use the `X-Amz-Expires` query param to determine how long the presigned link is valid. console.log( ( await client.sign( new Request(`${R2_URL}/my-bucket-name/dog.png?X-Amz-Expires=${3600}`), { aws: { signQuery: true }, }, ) ).url.toString(), ); // https://.r2.cloudflarestorage.com/my-bucket-name/dog.png?X-Amz-Expires=3600&X-Amz-Date=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-SignedHeaders=host&X-Amz-Signature= // You can also create links for operations such as PutObject to allow temporary write access to a specific key. console.log( ( await client.sign( new Request(`${R2_URL}/my-bucket-name/dog.png?X-Amz-Expires=${3600}`, { method: "PUT", }), { aws: { signQuery: true }, }, ) ).url.toString(), ); ``` You can use the link generated by the `PutObject` example to upload to the specified bucket and key, until the presigned link expires. ```sh curl -X PUT "https://.r2.cloudflarestorage.com/my-bucket-name/dog.png?X-Amz-Expires=3600&X-Amz-Date=&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=&X-Amz-SignedHeaders=host&X-Amz-Signature=" -F "data=@dog.png" ``` --- # boto3 URL: https://developers.cloudflare.com/r2/examples/aws/boto3/ import { Render } from "~/components";
You must configure [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) to use a preconstructed `endpoint_url` value. This can be done through any `boto3` usage that accepts connection arguments; for example: :::note[Compatibility] Client version `1.36.0` introduced a modification to the default checksum behavior from the client that is currently incompatible with R2 APIs. To mitigate, users can use `1.35.99` or add the following to their s3 resource config: ```python request_checksum_calculation = 'WHEN_REQUIRED', response_checksum_validation = 'WHEN_REQUIRED' ``` ::: ```python import boto3 s3 = boto3.resource('s3', endpoint_url = 'https://.r2.cloudflarestorage.com', aws_access_key_id = '', aws_secret_access_key = '' ) ``` You may, however, omit the `aws_access_key_id` and `aws_secret_access_key ` arguments and allow `boto3` to rely on the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` [environment variables](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#using-environment-variables) instead. An example script may look like the following: ```python import boto3 s3 = boto3.client( service_name ="s3", endpoint_url = 'https://.r2.cloudflarestorage.com', aws_access_key_id = '', aws_secret_access_key = '', region_name="", # Must be one of: wnam, enam, weur, eeur, apac, auto ) # Get object information object_information = s3.head_object(Bucket=, Key=) # Upload/Update single file s3.upload_fileobj(io.BytesIO(file_content), , ) # Delete object s3.delete_object(Bucket=, Key=) ``` ```sh python main.py ``` ```sh output Buckets: - user-uploads - my-bucket-name Objects: - cat.png - todos.txt ``` --- # Configure custom headers URL: https://developers.cloudflare.com/r2/examples/aws/custom-header/ Some of R2's [extensions](/r2/api/s3/extensions/) require setting a specific header when using them in the S3 compatible API. For some functionality you may want to set a request header on an entire category of requests. Other times you may want to configure a different header for each individual request. This page contains some examples on how to do so with `boto3` and with `aws-sdk-js-v3`. ## Setting a custom header on all requests When using certain functionality, like the `cf-create-bucket-if-missing` header, you may want to set a constant header for all `PutObject` requests you're making. ### Set a header for all requests with `boto3` `Boto3` has an event system which allows you to modify requests. Here we register a function into the event system which adds our header to every `PutObject` request being made. ```python import boto3 client = boto3.resource('s3', endpoint_url = 'https://.r2.cloudflarestorage.com', aws_access_key_id = '', aws_secret_access_key = '' ) event_system = client.meta.events # Define function responsible for adding the header def add_custom_header(params, **kwargs): params["headers"]['cf-create-bucket-if-missing'] = 'true' event_system.register('before-call.s3.PutObject', add_custom_header) response = client.put_object(Bucket="my_bucket", Key="my_file", Body="file_contents") print(response) ``` ### Set a header for all requests with `aws-sdk-js-v3` `aws-sdk-js-v3` allows the customization of request behavior through the use of its [middleware stack](https://aws.amazon.com/blogs/developer/middleware-stack-modular-aws-sdk-js/). This example adds a middleware to the client which adds a header to every `PutObject` request being made. ```ts import { PutObjectCommand, S3Client, } from "@aws-sdk/client-s3"; const client = new S3Client({ region: "auto", endpoint: `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`, credentials: { accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET_ACCESS_KEY, }, }); client.middlewareStack.add( (next, context) => async (args) => { const r = args.request as RequestInit r.headers["cf-create-bucket-if-missing"] = "true"; return await next(args) }, { step: 'build', name: 'customHeaders' }, ) const command = new PutObjectCommand({ Bucket: "my_bucket", Key: "my_key", Body: "my_data" }); const response = await client.send(command); console.log(response); ``` ## Set a different header on each request Certain extensions that R2 has provided in the S3 compatible api may require setting a different header on each request. For example, you may want to only want to overwrite an object if its etag matches a certain expected value. This value will likely be different for each object that is being overwritten, which requires the `If-Match` header to be different with each request you make. This section shows examples of how to accomplish that. ### Set a header per request in `boto3` To enable us to pass custom headers as an extra argument into the call to `client.put_object()` we need to register 2 functions into `boto3`'s event system. This is necessary because `boto3` performs a parameter validation step which rejects extra method arguments. Since this parameter validation occurs before we can set headers on the request, we first need to move the custom argument into the request context before the parameter validation happens. In a subsequent step we can now actually set the headers based on the information we put in the request context. ```python import boto3 client = boto3.resource('s3', endpoint_url = 'https://.r2.cloudflarestorage.com', aws_access_key_id = '', aws_secret_access_key = '' ) event_system = client.meta.events # Moves the custom headers from the parameters to the request context def process_custom_arguments(params, context, **kwargs): if (custom_headers := params.pop("custom_headers", None)): context["custom_headers"] = custom_headers # Here we extract the headers from the request context and actually set them def add_custom_headers(params, context, **kwargs): if (custom_headers := context.get("custom_headers")): params["headers"].update(custom_headers) event_system.register('before-parameter-build.s3.PutObject', process_custom_arguments) event_system.register('before-call.s3.PutObject', add_custom_headers) custom_headers = {'If-Match' : '"29d911f495d1ba7cb3a4d7d15e63236a"'} # Note that boto3 will throw an exception if the precondition failed. Catch this exception if necessary response = client.put_object(Bucket="my_bucket", Key="my_key", Body="file_contents", custom_headers=custom_headers) print(response) ``` ### Set a header per request in `aws-sdk-js-v3` Here we again configure the header we would like to set by creating a middleware, but this time we add the middleware to the request itself instead of to the whole client. ```ts import { PutObjectCommand, S3Client, } from "@aws-sdk/client-s3"; const client = new S3Client({ region: "auto", endpoint: `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`, credentials: { accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET_ACCESS_KEY, }, }); const command = new PutObjectCommand({ Bucket: "my_bucket", Key: "my_key", Body: "my_data" }); const headers = { 'If-Match': '"29d911f495d1ba7cb3a4d7d15e63236a"' } command.middlewareStack.add( (next) => (args) => { const r = args.request as RequestInit Object.entries(headers).forEach( ([k, v]: [key: string, value: string]): void => { r.headers[k] = v }, ) return next(args) }, { step: 'build', name: 'customHeaders' }, ) const response = await client.send(command); console.log(response); ``` --- # S3 SDKs URL: https://developers.cloudflare.com/r2/examples/aws/ import { DirectoryListing } from "~/components"; --- # Snowflake URL: https://developers.cloudflare.com/r2/reference/partners/snowflake-regions/ import { Render } from "~/components"; This page details which R2 location or jurisdiction is recommended based on your Snowflake region. You have the following inputs to control the physical location where objects in your R2 buckets are stored (for more information refer to [data location](/r2/reference/data-location/)): - [**Location hints**](/r2/reference/data-location/#location-hints): Specify a geophrical area (for example, Asia-Pacific or Western Europe). R2 makes a best effort to place your bucket in or near that location to optimize performance. You can confirm bucket placement after creation by navigating to the **Settings** tab of your bucket and referring to the **Bucket details** section. - [**Jurisdictions**](/r2/reference/data-location/#jurisdictional-restrictions): Enforce that data is both stored and processed within a specific jurisdiction (for example, the EU or FedRAMP environment). Use jurisdictions when you need to ensure data is stored and processed within a jurisdiction to meet data residency requirements, including local regulations such as the [GDPR](https://gdpr-info.eu/) or [FedRAMP](https://blog.cloudflare.com/cloudflare-achieves-fedramp-authorization/). ## North and South America (Commercial) | Snowflake region name | Cloud | Region ID | Recommended R2 location | | ------------------------- | ----- | ---------------- | ----------------------- | | Canada (Central) | AWS | `ca-central-1` | Location hint: `enam` | | South America (Sao Paulo) | AWS | `sa-east-1` | Location hint: `enam` | | US West (Oregon) | AWS | `us-west-2` | Location hint: `wnam` | | US East (Ohio) | AWS | `us-east-2` | Location hint: `enam` | | US East (N. Virginia) | AWS | `us-east-1` | Location hint: `enam` | | US Central1 (Iowa) | GCP | `us-central1` | Location hint: `enam` | | US East4 (N. Virginia) | GCP | `us-east4` | Location hint: `enam` | | Canada Central (Toronto) | Azure | `canadacentral` | Location hint: `enam` | | Central US (Iowa) | Azure | `centralus` | Location hint: `enam` | | East US 2 (Virginia) | Azure | `eastus2` | Location hint: `enam` | | South Central US (Texas) | Azure | `southcentralus` | Location hint: `enam` | | West US 2 (Washington) | Azure | `westus2` | Location hint: `wnam` | ## U.S. Government | Snowflake region name | Cloud | Region ID | Recommended R2 location | | --------------------- | ----- | --------------- | ----------------------- | | US Gov East 1 | AWS | `us-gov-east-1` | Jurisdiction: `fedramp` | | US Gov West 1 | AWS | `us-gov-west-1` | Jurisdiction: `fedramp` | | US Gov Virginia | Azure | `usgovvirginia` | Jurisdiction: `fedramp` | :::note Cloudflare Enterprise customers may contact their account team or [Cloudflare Support](/support/contacting-cloudflare-support/) to get access to the FedRAMP jurisdiction. ::: ## Europe and Middle East | Snowflake region name | Cloud | Region ID | Recommended R2 location | | ----------------------------- | ----- | ------------------ | ----------------------------------------- | | EU (Frankfurt) | AWS | `eu-central-1` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | EU (Zurich) | AWS | `eu-central-2` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | EU (Stockholm) | AWS | `eu-north-1` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | EU (Ireland) | AWS | `eu-west-1` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | Europe (London) | AWS | `eu-west-2` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | EU (Paris) | AWS | `eu-west-3` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | Middle East Central2 (Dammam) | GCP | `me-central2` | Location hint: `weur`/`eeur` | | Europe West2 (London) | GCP | `europe-west-2` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | Europe West3 (Frankfurt) | GCP | `europe-west-3` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | Europe West4 (Netherlands) | GCP | `europe-west-4` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | North Europe (Ireland) | Azure | `northeurope` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | Switzerland North (Zurich) | Azure | `switzerlandnorth` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | West Europe (Netherlands) | Azure | `westeurope` | Jurisdiction: `eu` or hint: `weur`/`eeur` | | UAE North (Dubai) | Azure | `uaenorth` | Location hint: `weur`/`eeur` | | UK South (London) | Azure | `uksouth` | Jurisdiction: `eu` or hint: `weur`/`eeur` | ## Asia Pacific and China | Snowflake region name | Cloud | Region ID | Recommended R2 location | | -------------------------------- | ----- | ---------------- | ----------------------- | | Asia Pacific (Tokyo) | AWS | `ap-northeast-1` | Location hint: `apac` | | Asia Pacific (Seoul) | AWS | `ap-northeast-2` | Location hint: `apac` | | Asia Pacific (Osaka) | AWS | `ap-northeast-3` | Location hint: `apac` | | Asia Pacific (Mumbai) | AWS | `ap-south-1` | Location hint: `apac` | | Asia Pacific (Singapore) | AWS | `ap-southeast-1` | Location hint: `apac` | | Asia Pacific (Sydney) | AWS | `ap-southeast-2` | Location hint: `oc` | | Asia Pacific (Jakarta) | AWS | `ap-southeast-3` | Location hint: `apac` | | China (Ningxia) | AWS | `cn-northwest-1` | Location hint: `apac` | | Australia East (New South Wales) | Azure | `australiaeast` | Location hint: `oc` | | Central India (Pune) | Azure | `centralindia` | Location hint: `apac` | | Japan East (Tokyo) | Azure | `japaneast` | Location hint: `apac` | | Southeast Asia (Singapore) | Azure | `southeastasia` | Location hint: `apac` | --- # First Live Stream with OBS URL: https://developers.cloudflare.com/stream/examples/obs-from-scratch/ ## Overview Stream empowers customers and their end-users to broadcast a live stream quickly and at scale. The player can be embedded in sites and applications easily, but not everyone knows how to make a live stream because it happens in a separate application. This walkthrough will demonstrate how to start your first live stream using OBS Studio, a free live streaming application used by thousands of Stream customers. There are five required steps; you should be able to complete this walkthrough in less than 15 minutes. ### Before you start To go live on Stream, you will need any of the following: - A paid Stream subscription - A Pro or Business zone plan — these include 100 minutes of video storage and 10,000 minutes of video delivery - An enterprise contract with Stream enabled Also, you will also need to be able to install the application on your computer. If your computer and network connection are good enough for video calling, you should at least be able to stream something basic. ## 1. Set up a [Live Input](/stream/stream-live/start-stream-live/) You need a Live Input on Stream. Follow the [Start a live stream](/stream/stream-live/start-stream-live/) guide. Make note of three things: - **RTMPS URL**, which will most likely be `rtmps://live.cloudflare.com:443/live/` - **RTMPS Key**, which is specific to the new live input - Whether you selected the beta "Low-Latency HLS Support" or not. For your first test, leave this _disabled._ ([What's that?](https://blog.cloudflare.com/cloudflare-stream-low-latency-hls-open-beta)) ## 2. Install OBS Download [OBS Studio](https://obsproject.com/) for Windows, macOS, or Linux. The OBS Knowledge Base includes several [installation guides](https://obsproject.com/kb/category/1), but installer defaults are generally acceptable. ## 3. First Launch OBS Configuration When you first launch OBS, the Auto-Configuration Wizard will ask a few questions and offer recommended settings. See their [Quick Start Guide](https://obsproject.com/kb/quick-start-guide) for more details. For a quick start with Stream, use these settings: - **Step 1: "Usage Information"** - Select "Optimize for streaming, recording is secondary." - **Step 2: "Video Settings"** - **Base (Canvas) Resolution:** 1920x1080 - **FPS:** "Either 60 or 30, but prefer 60 when possible" - **Step 3: "Stream Information"** - **Service:** "Custom" - For **Server**, enter the RTMPS URL from Stream - For **Stream Key**, enter the RTMPS Key from Stream - If available, select both **"Prefer hardware encoding"** and **"Estimate bitrate with a bandwidth test."** ## 4. Set up a Stage Add some test content to the stage in OBS. In this example, I have added a background image, a web browser (to show [time.is](https://time.is)), and an overlay of my webcam: ![OBS Stage](~/assets/images/stream/examples/obs-from-scratch/obs-stage.png) OBS offers many different audio, video, still, and generated sources to set up your broadcast content. Use the "+" button in the "Sources" panel to add content. Check out the [OBS Sources Guide](https://obsproject.com/kb/sources-guide) for more information. For an initial test, use a source that will show some motion: try a webcam ("Video Capture Device"), a screen share ("Display Capture"), or a browser with a site that has moving content. ## 5. Go Live Click the "Start Streaming" button on the bottom right panel under "Controls" to start a stream with default settings. Return to the Live Input page on Stream Dash. Under "Input Status," you should see "🟢 Connected" and some connection metrics. Further down the page, you will see a test player and an embed code. For more ways to watch and embed your Live Stream, see [Watch a live stream](/stream/stream-live/watch-live-stream/). ## 6. (Optional) Optimize Settings Tweaking some settings in OBS can improve quality, glass-to-glass latency, or stability of the stream playback. This is particularly important if you selected the "Low-Latency HLS" beta option. Return to OBS, click "Stop Streaming." Then click "Settings" and open the "Output" section: ![OBS Output Settings - Simple Mode](~/assets/images/stream/examples/obs-from-scratch/obs-output-settings-1.png) - Change **Output Mode** to "Advanced" ![OBS Output Settings - Advanced Mode](~/assets/images/stream/examples/obs-from-scratch/obs-output-settings-2.png) _Your available options in the "Video Encoder" menu, as well as the resulting "Encoder Settings," may look slightly different than these because the options vary by hardware._ - **Video Encoder:** may have several options. Start with the default selected, which was "x264" in this example. Other options to try, which will leverage improved hardware acceleration when possible, include "QuickSync H.264" or "NVIDIA NVENC." See OBS's guide to Hardware Encoding for more information. H.264 is the required output codec. - **Rate Control:** confirm "CBR" (constant bitrate) is selected. - **Bitrate:** depending on the content of your stream, a bitrate between 3000 Kbps and 8000 Kbps should be sufficient. Lower bitrate is more tolerant to network congestion and is suitable for content with less detail or less motion (speaker, slides, etc.) where a higher bitrate requires a more stable network connection and is best for content with lots of motion or details (events, moving cameras, video games, screen share, higher framerates). - **Keyframe Interval**, sometimes referred to as _GOP Size_: - If you did _not_ select Low-Latency HLS Beta, set this to 4 seconds. Raise it to 8 if your stream has stuttering or freezing. - If you _did_ select the Low-Latency HLS Beta, set this to 2 seconds. Raise it to 4 if your stream has stuttering or freezing. Lower it to 1 if your stream has smooth playback. - In general, higher keyframe intervals make more efficient use of bandwidth and CPU for encoding, at the expense of higher glass-to-glass latency. Lower keyframe intervals reduce latency, but are more resource intensive and less tolerant to network disruptions and congestion. - **Profile** and **Tuning** can be left at their default settings. - **B Frames** (available only for some encoders) should be set to 0 for LL-HLS Beta streams. For more information about these settings and our recommendations for Live, see the "[Recommendations, requirements and limitations](/stream/stream-live/start-stream-live/#recommendations-requirements-and-limitations)" section of [Start a live stream](/stream/stream-live/start-stream-live/). ## What's Next With these steps, you have created a Live Input on Stream, broadcast a test from OBS, and you saw it played back in via the Stream built-in player in Dash. Up next, consider trying: - Embedding your live stream into a website - Find and replay the recording of your live stream --- # Android URL: https://developers.cloudflare.com/stream/viewing-videos/using-own-player/android/ import { Render } from "~/components" You can stream both on-demand and live video to native Android apps using [ExoPlayer](https://exoplayer.dev/). ## Example Apps * [Android](/stream/examples/android/) ## Using ExoPlayer Play a video from Cloudflare Stream using ExoPlayer: --- # Use your own player URL: https://developers.cloudflare.com/stream/viewing-videos/using-own-player/ import { Render, Badge } from "~/components" Cloudflare Stream is compatible with all video players that support HLS and DASH, which are standard formats for streaming media with broad support across all web browsers, mobile operating systems and media streaming devices. Platform-specific guides: * [Web](/stream/viewing-videos/using-own-player/web/) * [iOS (AVPlayer)](/stream/viewing-videos/using-own-player/ios/) * [Android (ExoPlayer)](/stream/viewing-videos/using-own-player/android/) ## Fetch HLS and Dash manifests ### URL Each video and live stream has its own unique HLS and DASH manifest. You can access the manifest by replacing `` with the UID of your video or live input, and replacing `

` with your unique customer code, in the URLs below:

```txt title="HLS"
https://customer-.cloudflarestream.com//manifest/video.m3u8
```

```txt title="DASH"
https://customer-.cloudflarestream.com//manifest/video.mpd
```

#### LL-HLS playback 

If a Live Inputs is enabled for the Low-Latency HLS beta, add the query string `?protocol=llhls` to the HLS manifest URL to test the low latency manifest in a custom player. Refer to [Start a Live Stream](/stream/stream-live/start-stream-live/#use-the-api) to enable this option.

```txt title="HLS"
https://customer-.cloudflarestream.com//manifest/video.m3u8?protocol=llhls
```

### Dashboard

1. Log into the [Stream Dashboard](https://dash.cloudflare.com/?to=/:account/stream).
2. From the list of videos, locate your video and select it.
3. From the **Settings** tab, locate the **HLS Manifest URL** and **Dash Manifest URL**.
4. Select **Click to copy** under the option you want to use.

### API

Refer to the [Stream video details API documentation](/api/resources/stream/methods/get/) to learn how to fetch the manifest URLs using the Cloudflare API.

## Customize manifests by specifying available client bandwidth

Each HLS and DASH manifest provides multiple resolutions of your video or live stream. Your player contains adaptive bitrate logic to estimate the viewer's available bandwidth, and select the optimal resolution to play. Each player has different logic that makes this decision, and most have configuration options to allow you to customize or override either bandwidth or resolution.

If your player lacks such configuration options or you need to override them, you can add the `clientBandwidthHint` query param to the request to fetch the manifest file. This should be used only as a last resort — we recommend first using customization options provided by your player. Remember that while you may be developing your website or app on a fast Internet connection, and be tempted to use this setting to force high quality playback, many of your viewers are likely connecting over slower mobile networks.



* `clientBandwidthHint` float
  * Return only the video representation closest to the provided bandwidth value (in Mbps). This can be used to enforce a specific quality level. If you specify a value that would cause an invalid or empty manifest to be served, the hint is ignored.



Refer to the example below to display only the video representation with a bitrate closest to 1.8 Mbps.

```txt title="Example"
https://customer-f33zs165nr7gyfy4.cloudflarestream.com/6b9e68b07dfee8cc2d116e4c51d6a957/manifest/video.m3u8?clientBandwidthHint=1.8
```

## Play live video in native apps with less than 1 second latency

If you need ultra low latency, and your users view live video in native apps, you can stream live video with [**glass-to-glass latency of less than 1 second**](https://blog.cloudflare.com/magic-hdmi-cable/), by using SRT or RTMPS for playback.

![Diagram showing SRT and RTMPS playback via the Cloudflare Network](~/assets/images/stream/stream-rtmps-srt-playback-magic-hdmi-cable.png)

SRT and RTMPS playback is built into [ffmpeg](https://ffmpeg.org/). You will need to integrate ffmpeg with your own video player —  neither [AVPlayer (iOS)](/stream/viewing-videos/using-own-player/ios/) nor [ExoPlayer (Android)](/stream/viewing-videos/using-own-player/android/) natively support SRT or RTMPS playback.



We recommend using [ffmpeg-kit](https://github.com/arthenica/ffmpeg-kit) as a cross-platform wrapper for ffmpeg.

### Examples

* [RTMPS Playback with ffplay](/stream/examples/rtmps_playback/)
* [SRT playback with ffplay](/stream/examples/srt_playback/)

---

# iOS

URL: https://developers.cloudflare.com/stream/viewing-videos/using-own-player/ios/

import { Render } from "~/components"

You can stream both on-demand and live video to native iOS, tvOS and macOS apps using [AVPlayer](https://developer.apple.com/documentation/avfoundation/avplayer).



## Example Apps

* [iOS](/stream/examples/ios/)

## Using AVPlayer

Play a video from Cloudflare Stream using AVPlayer:



---

# Web

URL: https://developers.cloudflare.com/stream/viewing-videos/using-own-player/web/

import { Render } from "~/components"

Cloudflare Stream works with all web video players that support HLS and DASH.



## Examples

* [Video.js](/stream/examples/video-js/)
* [HLS reference player (hls.js)](/stream/examples/hls-js/)
* [DASH reference player (dash.js)](/stream/examples/dash-js/)
* [Vidstack](/stream/examples/vidstack/)

---

# Use the Stream Player

URL: https://developers.cloudflare.com/stream/viewing-videos/using-the-stream-player/

import { InlineBadge } from "~/components"

Cloudflare provides a customizable web player that can play both on-demand and live video, and requires zero additional engineering work.


  
    
  


To add the Stream Player to a web page, you can either:

* Generate an embed code in the [Stream Dashboard](https://dash.cloudflare.com/?to=/:account/stream) for a specific video or live input.
* Use the code example below, replacing `` with the video UID (or [signed token](/stream/viewing-videos/securing-your-stream/) and `` with the your unique customer code, which can be found in the [Stream Dashboard](https://dash.cloudflare.com/?to=/:account/stream).

```html

```

Stream player is also available as a [React](https://www.npmjs.com/package/@cloudflare/stream-react) or [Angular](https://www.npmjs.com/package/@cloudflare/stream-angular) component.

## Player Size

### Fixed Dimensions

Changing the `height` and `width` attributes on the `iframe` will change the pixel value dimensions of the iframe displayed on the host page.

```html

```

### Responsive

To make an iframe responsive, it needs styles to enforce an aspect ratio by setting the `iframe` to `position: absolute;` and having it fill a container that uses a calculated `padding-top` percentage.

```html


  

```

## Basic Options

Player options are configured with querystring parameters in the iframe's `src` attribute. For example:

`https://customer-.cloudflarestream.com//iframe?autoplay=true&muted=true`

* `autoplay` default: `false`

  * If the autoplay flag is included as a querystring parameter, the player will attempt to autoplay the video. If you don't want the video to autoplay, don't include the autoplay flag at all (instead of setting it to `autoplay=false`.) Note that mobile browsers generally do not support this attribute, the user must tap the screen to begin video playback. Please consider mobile users or users with Internet usage limits as some users don't have unlimited Internet access before using this attribute.

    :::caution

    Some browsers now prevent videos with audio from playing automatically. You may set `muted` to `true` to allow your videos to autoplay. For more information, refer to [New `

Welcome to Remix

Sales enquiry

My Blog

← Go back

← Go back

My Blog

Posts

{post.title}

{post.title}

Example Site

Awesome Cat Blog! 😺

Note transcript

Note recordings

Create note

No notes created

Post Processing

Generate types that match your Worker's configuration

Migrating from `@cloudflare/workers-types` to `wrangler types`

QR Generator

QR Generator

Todos

Todos

Todos