I've been in on-and-off contact with Michael Morisy over at MuckRock, and one of the things that's come up is the API script that they're working on the export the FOIA requests and responses. Unfortunately, the script crashes after processing a few dozen, but not consistently - likely the result of character encoding issues. He's given me permission to share the in-progress script. https://github.com/MuckRock/API-examples/blob/master/export_all_requests_as_... If anyone is able to look at the code and offer any ideas about solutions (or even pinpointing the problem), it'd be a great public service. I'm not code savvy enough to help, but I'm hoping someone on the list can bring in some fresh eyes and help MuckRock get the script working. --Mike
On 11/10/15, Michael Best <themikebest@gmail.com> wrote:
I've been in on-and-off contact with Michael Morisy over at MuckRock, and one of the things that's come up is the API script that they're working on the export the FOIA requests and responses.
neat!
Unfortunately, the script crashes after processing a few dozen, but not consistently - likely the result of character encoding issues. He's given me permission to share the in-progress script.
going over it now. will also fix the insecure HTTPS configuration. also ill; progress will be slow! also noticed fixed page sizes, will adjust this based on API responses as well. (no hardcode at 20 per page) and a few things with dir and file creation i might shore up (?) will advise... best regards,
On 11/10/15, coderman <coderman@gmail.com> wrote:
...
Unfortunately, the script crashes after processing a few dozen, but not consistently...
hey Mike, do you have an example base URI that is close to / causing failures? i'm up over a gig of leeched docs, no crashes yet. ( curious to nail down culprit... :) i may create a UTF-8 and UTF-16 heavy request on MuckRock, for test, if this fails.
participants (2)
-
coderman -
Michael Best