JoeCode

Using curl with protected endpoints example

Mar 16, 2023

Have you ever found yourself wanting to extract and download the JSON data used by a web application you subscribe to? Most online applications are reluctant to expose a “download” all button, but most of them rely on a RESTful service to power their frontend experience. Using your web browsers development tools you can discover your target applications endpoints and request parameters.

Login to your target application using your favorite web browser
Open your browsers developer tools console and navigate to the network inspection tab
Browse to a resource your interested in
Watch for “fetch” requests within your network inspector
Inspect the Headers used to execute the data request
Note the Host, Path, Request-Type, Content-Type, Cookie, and Request Data

Assume we captured the following data:

Host: https://app.foobar.com
Path: /services/random/api/endpoint/goes/here
Request-Type: POST
Content-Type: application/json
Cookie: really-long-string-of-cookie-text-goes-here
Request Data: {“param1”:value1,”param2”:value2,”foo”:””,”bar”:[],”page”:1,”size”:10}

Open up a text editor and craft your curl request so that it looks like the following:

curl --globoff -d '{"param1":value1,"param2":value2,"foo":"","bar":[],"page":1,"size":10}' \
-H "Content-Type: application/json" \
-H "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1" \
-H "Cookie: really-long-string-of-cookie-text-goes-here" \
-X POST https://app.foobar.com/services/random/api/endpoint/goes/here

After you have the base query working, manipulate the data request parameters to see if you can get more data per request (“size” looks promising…).

Pay attention to any “X-FOOBAR” Header elements. You’ll probably need to add them to the curl query as well.

JoeCode

Using curl with protected endpoints example

Resources