Using curl with protected endpoints example

Mar 16, 2023

Have you ever found yourself wanting to extract and download the JSON data used by a web application you subscribe to? Most online applications are reluctant to expose a “download” all button, but most of them rely on a RESTful service to power their frontend experience. Using your web browsers development tools you can discover your target applications endpoints and request parameters.

  1. Login to your target application using your favorite web browser
  2. Open your browsers developer tools console and navigate to the network inspection tab
  3. Browse to a resource your interested in
  4. Watch for “fetch” requests within your network inspector
  5. Inspect the Headers used to execute the data request
  6. Note the Host, Path, Request-Type, Content-Type, Cookie, and Request Data

Assume we captured the following data:

Open up a text editor and craft your curl request so that it looks like the following:

curl --globoff -d '{"param1":value1,"param2":value2,"foo":"","bar":[],"page":1,"size":10}' \
-H "Content-Type: application/json" \
-H "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1" \
-H "Cookie: really-long-string-of-cookie-text-goes-here" \

After you have the base query working, manipulate the data request parameters to see if you can get more data per request (“size” looks promising…).

Pay attention to any “X-FOOBAR” Header elements. You’ll probably need to add them to the curl query as well.