Download failures

[WEricBr]

I am running into a couple of issues when running the script...

A.  Running the script, the script reports Found "10024 books in total" which is not even close to the number of books I have.

B. I get about 3 different errors (so far):
    1.  Error:  No valid download URL Found
         Failed to download [Book Title]: No valid download URL found

        These are not surprising as they are very old,
        and when I try downloading them manually from my "Digital Content" page I get the error:
            You do not have any compatible devices registered for this content

    2.  Error:  None displayed, just 0/1 and a blank "progress bar"
        [Book Title] | 0/1
        
       These are downloadable when I try to download them manually from my "Digital Content" page

    3.  Error:  404 Not Found
        Failed to download [Book Title]: Failed to fetch (404 Not Found) https://www.amazon.com/hz/mycd/ajax
        
        The "Book Title" for these had weird characters at the beginning "阿阿阿", 
        and when I try downloading them manually from my "Digital Content" page I get the error:
            Something went wrong
            There was an error processing your request. 
            Please try again.<br/>If you have recently removed this item, please refresh the page.

    4.  Error:  503 Service unavailable
        Failed to download [Book Title]: Failed to fetch (503 Service Unavailable) https://www.amazon.com/hz/mycd/ajax

        This one starts at around Processing batch 34/1000 (Books 331-340) with 1 or 2 of these errors,
            then around bach 31-38, it has 3-5 of these errors,
            then around batch 39-47, it has around 6-7 of these errors,
            then around batch 48-90 it gets 7-8 of these errors,
            then around batch 91 no more books are downloaded and I just get are these errors until I stop the script (Ctl-C).

        These books CAN be downloaded successfully when I try downloading them manually from my "Digital Content" page.

        My thought is that the service gets overwhelmed or determines that I am downloading too many books too fast, 
        but I am not sure.

    5.  Some books are downloaded with a different name using the script vs 
         downloading them manually from my "Digital Content" page.

            Book Title:             Wrath: A Novel About The End of Days! (THE WRATH TRILOGY Book 1)
            Amazon Download Title:  Wrath_ The Tribulation Has Begun, And The Church Is Still Here! (THE WRATH TRILOGY Book 1).azw3
            Script Download Title:  Wrath A Novel About The End of Days! (THE WRATH TRILOGY Book 1).azw3

        The file sizes are the same.

Sorry for formatting this as "code", but otherwise the format get messed up.

Any ideas on this these?

Thanks,

Eric

[treetrum]

Thanks for the extensive write up!

Off the top of my head:

A. The 10024 limit seems to be a limit from Amazon, I've had other users report the same. I haven't had any feedback, but I'd suggest pulling the latest code and trying out the startFromOffset flag on subsequent runs to capture the full limit. See this issue for the thread where I'm trying to troubleshoot that issue: #181

B. Errors you've mentioned:

1. I think you're on the money, these books are just not downloadable unfortunately.
2. No idea off the top of my head — will need to look into
3. If you can't download from the UI, I don't think the tool will be able to help you.
4. 503 certainly sounds like rate limiting to me. If you pull the latest code, I've implemented skipping of already downloaded files. This should allow you restart the script and skip what you've already grabbed (and therefore hopefully not get rate limited)
5. The naming differences are just a quirk of the tool — I'm doing some sanitizing of the file names before writing them to disk to try and alleviate some issues. You will notice after pulling the latest code there will be even more differences (i'm now including a unique ASIN in the file name to prevent false-positives for duplicate detection).

[WEricBr]

Thanks.

I have an idea I am going to try in order to hopefully get around the limit. If it works I'll let you know and send you the modifications (I am not accustomed to using GitHub, so I am not sure how to do that in here).

Thanks,

Eric