7

For the last couple of years we have been able to download a FedACH file from Federal Reserve website (frbservices.org) using the wget utility. The link below provided the original solution:

wget how to download file from a web page that prompts you to click on "Agree" condition

  1. get the cookies:

    wget --post-data="agreementValue=Agree" https://frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies cookie.txt --keep-session-cookies --delete-after

  2. download the file:

    wget --load-cookies=cookie.txt 'https://frbservices.org/EPaymentsDirectory/FedACHdir.txt?AgreementSessionObject=Agree'

Using this wget commands just downloads https://www.frbservices.org/EPaymentsDirectory/download.html because the website is requesting that a popup form be filled in after clicking the “Agree” button now. Is there some way to just close the form using wget?

In IE and Chrome, the steps below were executed to download the desired file:

  1. Go to URL: https://www.frbservices.org/EPaymentsDirectory/FedACHdir.txt

  2. Click Agree button.

  3. Form is displayed requesting organization information.

  4. Click Close button on form (I did not fill in any information).

  5. Go to URL: https://www.frbservices.org/EPaymentsDirectory/FedACHdir.txt

  6. File downloads successfully.

dessert
  • 40,956
GMP
  • 71
  • 3

1 Answers1

6

The website requires you to have two cookies set: A valid JSESSIONID and the abaDataCaptureCookie called agreement cookie. The first can be obtained with your first wget command, the second one needs to be added manually.

  1. Obtain a valid JSESSIONID and save it to cookies.txt:

    wget --post-data="agreementValue=Agree" frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies cookies.txt --keep-session-cookies --delete-after

  2. Add the abaDataCaptureCookie line:

    echo -e "frbservices.org\tFALSE\t/EPaymentsDirectory/\tFALSE\t0\tabaDataCaptureCookie\tabaDataCaptureCookie" >>cookies.txt

  3. Download the file:

    wget --load-cookies cookies.txt frbservices.org/EPaymentsDirectory/FedACHdir.txt

I'm not quite sure how long the first cookie stays valid, but I assume you won't download the list multiple times per hour, so obtaining a new one on every run seems OK to me – that's the failsafe approach after all.

Script version of the above steps, using a tempfile as the cookie file:

#!/bin/bash
cookiefile=$(mktemp)
wget --post-data="agreementValue=Agree" frbservices.org/EPaymentsDirectory/submitAgreement --save-cookies $cookiefile --keep-session-cookies --delete-after
echo -e "frbservices.org\tFALSE\t/EPaymentsDirectory/\tFALSE\t0\tabaDataCaptureCookie\tabaDataCaptureCookie" >>$cookiefile
wget --load-cookies $cookiefile frbservices.org/EPaymentsDirectory/FedACHdir.txt

Running this script will save the current FedACHdir.txt to the current directory without leaving a cookies.txt lying around, if there's already a file with this name wget adds a number and saves it as e.g. FedACHdir.txt.1.

On resolving this issue I found great help here: Format of cookies when using wget? · U&L

dessert
  • 40,956