Unknown field localFile on type S3Image

[naheller]

@DSchau Hello and thank you for making this plugin!

I am having an issue with the image processing query which uses gatsby-transformer-sharp and gatsby-plugin-sharp. These packages are installed, and the below query gives me back id, Key, and Url successfully.

However the file part throws the following error: Unknown field localFile on type S3Image.

allS3Image {
    edges {
        node {
            id
            Key
            Url
            file: localFile {
                image: childImageSharp {
                    sizes(maxWidth: 800, maxHeight: 800) {
                        ...GatsbyImageSharpSizes
                    }
                }
            }
        }
    }
}

I'm still getting the hang of Gatsby, GraphQL, and S3, so any advice would be appreciated!

[DSchau]

My guess is that there aren't any images in the s3 bucket.

If you open up graphiql, e.g. npm run develop in your gatsby project, and then go to localhost:8000/___graphql if you query allS3Image, e.g.

query {
  allS3Image {
    edges {
      node {
        id
      }
    }
  }
}

do you see any results?

[DSchau]

Also if it helps, here's a working example of this plugin. Admittedly, I haven't tested it much with Gatsby v2!

https://github.com/DSchau/photos.maggieanddustin.com

[naheller]

Thanks for your quick response!

Running the above query works fine, and in fact I can display unprocessed S3 images via the Url field. However it seems like the localFile field is not available on S3Image.

Could this be related to the way gatsby-transformer-sharp or gatsby-plugin-sharp are setup? I can make childImageSharp queries to local files via gatsby-source-filesystem, so I know that part is working.

I'll take a look at the working example you shared 👍

[naheller]

If you're able to check, do you see localFile listed in the arguments in your localhost:8000/___graphql docs section?

This is mine:

[DSchau]

Would this regular expression match the types of images in your S3 bucket?

[naheller]

@DSchau I think so - the following query retrieved node ids successfully. Should I be checking against something other than Key here?

query IndexQuery {
    allS3Image(
        filter:{Key:{regex:"/.(jpe?g|png|webp|tiff?)/"}}
      ) {
      edges {
        node {
          id
        }
      }
    }
  }

If you don't mind pointing it out - where is the image processing part hooked into the plugin? Is it via the downloadImageFile function?

[DSchau]

Yup! I believe that creates that localFile graphql resolver.

Most helpful here would be a repo reproducing the issue!

[naheller]

Here's an example repo.

It's got read-only AWS creds in gatsby-config and a few photos in the bucket, so you should be able to start it up without any additional work.

The index page shows the Key, Url, and raw image for each S3 asset. Everything runs as is, but If you query localFile on the node field here, you should get the following error in your console:

Unknown field localFile on type S3Image

Let me know if this helps! Thanks

[DSchau]

It will! Thanks so much! I'll take a look at this in a few!

[naheller]

Hey @DSchau, just checking in.

I wrote above that I could display unprocessed S3 images via the Url field. This is actually not the case. In your gatsby-node file here, you set the Url to begin with https://s3.amazonaws.com/.

Attempting to view an image via this link returns the error:

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

In order to source images directly from my S3 bucket, I had to use a region-specific base url here (https://s3-us-west-2.amazonaws.com). The resulting image can then be sourced successfully.

Could my issue be related to this? Or is the availability of the localFile field on S3Image not reliant on the Url?

[DSchau]

Hey Nathan!

I actually started at Gatsby on Tuesday, so I've been a little busy with that!

I'm not 100% sure what the issue is yet, but I appreciate you debugging! I can try and take a look at it this weekend, sorry for the delay :(

Thanks for your patience!

[naheller]

Congratulations! 🎉

Ok, I'll keep poking around and share any new discoveries. Thanks for keeping in touch.

[jonathanbell]

+1 I've come across this issue pretty much exactly as @naheller has described. So far, my dependencies look like this:

"dependencies": {
    "gatsby": "^1.9.277",
    "gatsby-image": "^1.0.55",
    "gatsby-link": "^1.6.46",
    "gatsby-plugin-react-helmet": "^2.0.11",
    "gatsby-plugin-sharp": "^1.6.48",
    "gatsby-source-s3": "0.0.0",
    "gatsby-transformer-sharp": "^1.6.27",
    "react-helmet": "^5.2.0"
  }

If anyone can point me in the right direction (I'm new to Gatsby) that would be very helpful and appreciated! I would like to use the images in folders inside an AWS bucket and pipe them to the sharp plugin where (I guess) they would be stored locally(??) as cached images. Like @naheller I can display images from the bucket if I string replace the region-specific URL to the bucket. Then I can at least display them on the page.

[DSchau]

@jonathanbell still tracking on this, but see my comment above for more info!

#8 (comment)

[naheller]

I just downgraded to Gatsby 1.9.247 (the same that's used in the photos.maggieanddustin.com repo linked above). The same error is thrown: Unknown field localFile on type S3Image

@jonathanbell Also appears to be experiencing this issue with pre v2 Gatsby, so I suppose we can rule that out...

[naheller]

@DSchau

Ok! After some poking around I fixed this issue, and it is indeed related to region-specific S3 bucket URLs.

I cloned/linked this plugin and found that the localFile field wasn't being added to the imageNode because this conditional never executed.

There are two ways I found to trigger that conditional and make the localFile field available:

1. Change in user behavior: use an S3 bucket based in North Virginia. This is the only AWS region that is accessible via the base URL used by this plugin: https://s3.amazonaws.com.

2. Change in plugin behavior: generate a region-specific base URL in gatsby-node.js. Since my bucket is based in Oregon, I changed the base URL to https://s3-us-west-2.amazonaws.com.

Either of these actions will expose the localFile field and its childImageSharp values as intended!

Since S3 bucket regions will vary between users, I feel some region-dependent URL construction in this plugin would be helpful. I'm happy to submit a PR for this if you'd like 👍

More info on path-style S3 URLs can be found here.

[DSchau]

Awesome job! A PR would be so welcome, and I’ll approve when I can! Ideally a test would be nice too, but this seems fairly innocuous

[jonathanbell]

@naheller Thank you! I changed my AWS bucket to N. Virginia and sure enough it worked. @DSchau I have noticed one issue. If there are spaces inside a folder name inside the S3 AWS bucket, the value of the file property on the node objects will be null when querying allS3Image.

Just thought I would point out that bug. I've changed my folder names to use hyphens instead of spaces and I can get a value (not null) from data.images.edges[0].node.file returned from my allS3Image query, for example. Hopefully that makes sense..

[DSchau]

I think there’s some work for us to do on this 😅

Help is appreciated! Sounds like there are some edge cases here that could be ironed out! I’ll try and get to them soon, but until then, would love any and all PRs!

[DSchau]

Actually with my role at Gatsby, we can pair program so maybe this would be something cool to work on!

Any interest?

[trsvax]

I ran across this problem also. My bucket is accessible via https://s3.amazonaws.com but it's not public. Since the plugin can authenticate it shows the files but createRemoteFileNode is not able to fetch them via just the url

[drewbietron]

I've tried both of the solutions that @naheller posted above and still no luck. I was a little stumped at how the AWS availability zone plays a role in this -- since I also confirmed that localFile is just never added to the node in download-image-file. I tried pushing up a PR to fix this but was getting hung up on stubbing out Gatsby's Redux store when the function tries to call createRemoteFileNode from the gatsby-source-filesystem.

Any other help on getting this to work would be appreciated!

[drewbietron]

Did anyone ever find a solution to this?

[DSchau]

Hi Drew -

I never did figure it out because it works for me and it's hard to pin down issues with other setups, unfortunately.

If you could grant read access to me somehow I could probably take a look.

Also once more, I would happily take a pairing session request so that we can work on this together!

[DSchau]

@drewbietron took me up on that pairing request 🎉

@drewbietron could you post that working snippet?

[drewbietron]

Was able to get this working with the following query. Thanks for pairing on this @DSchau

images: allS3Image {
  edges {
    node {
      id
      localFile {
        childImageSharp {
          fluid(maxWidth: 1200, quality: 90) {
            ...GatsbyImageSharpFluid_withWebp
          }
        }
      }
    }
  }
}

Was also able to run a normal filter -> regex to only return certain files. Which looks like

images: allS3Image(filter: {Url: {regex: "/file-or-folder-name/"}})

One thing that held me up for a while is that my cache'd version of my graphQl schema was out of sync, and I believe it got into that state while I was trying to create a localFile node for an S3 asset that was not public. Basically if localFile isnt returning within the S3Image node in graphql, you know something is up.

To debug possible S3 issues, make sure that imageNodeId is a thing in download-image-file.js

[drewbietron]

I actually ran into the problem again, but was able to confirm that its not due to this plugin.

Originally I was trying to run this plugin against a folder of images off of S3 (about ~150 images, all 10-20mb a piece) and it was not returning a local file for each image. Later on, when I was pairing with @DSchau, we tested it against just one image, and it worked fine (and still does!).

The problem resurfaced when I began running the plugin against a folder of hundred + images and after further digging, the best I can gather is that it's due to gatsy-source-filesystem -> create_remote_file_node.js. More details here.

The stock Gatsby API for creating a local file isn't waiting for an entire image file to download before returning that image back to this S3 plugin, therefore a localFile node is either not created, or created but with an empty object. 🤷‍♂️

[Vacilando]

Same problem.

This shows names of the jpgs in the bucket:

query {
  allS3Image {
    edges {
      node {
        id
      }
    }
  }
}

However, localFile gives an error, and it's not present in the ___graphql explorer either.

Our bucket in in Ireland, so may need to use https://s3-eu-west-1.amazonaws.com instead of https://s3.amazonaws.com

How can we override the value that is hard-coded in this plugin?

[DSchau]

@Vacilando there's nothing hard coded in the plugin.

It uses aws-sdk internally, and the region can be passed to this plugin. https://github.com/DSchau/gatsby-source-s3/blob/master/src/gatsby-node.js#L26

[DSchau]

If you need a separate URL -- you'd want to augment this piece: https://github.com/DSchau/gatsby-source-s3/blob/master/src/gatsby-node.js#L36

PRs welcome!

[Vacilando]

I tried adding
region: 'eu-west-1',
as an additional aws option in gatsby-config.js, but it did not help.

The solution will probably be in what @naheller said above on 9/9/2018:

2. Change in plugin behavior: generate a region-specific base URL in gatsby-node.js. Since my bucket is based in Oregon, I changed the base URL to https://s3-us-west-2.amazonaws.com.
   Since S3 bucket regions will vary between users, I feel some region-dependent URL construction in this plugin would be helpful. I'm happy to submit a PR for this if you'd like 👍

This is I guess what you also mean by saying:

If you need a separate URL -- you'd want to augment this piece: https://github.com/DSchau/gatsby-source-s3/blob/master/src/gatsby-node.js#L36

Guess the easiest would be to take the 'region' value (if submitted) from gatsby-config.js and construct the url in gatsby-node.js by using:
's3-' + $region + '.amazonaws.com' and defaulting to 's3.amazonaws.com'
It is beyond me to code this and submit a PR, but I am sure somebody with experience in writing Gatsby plugins could do this in a few minutes - any volunteers?

[Vacilando]

I confirm things work (localFile is available) when using a bucket in the US East (N. Virginia) region. Thanks for the detective work @naheller and @jonathanbell.

We need to make the URL configurable by region so that this works for all buckets.

[robinmetral]

there's nothing hard coded in the plugin.

It uses aws-sdk internally, and the region can be passed to this plugin. https://github.com/DSchau/gatsby-source-s3/blob/master/src/gatsby-node.js#L26

It is hardcoded, because the region needs to be part of the URL to avoid a PermanentRedirect error.

Here's the endpoint that needs to be hit according to the AWS docs :

protocol://service-code.region-code.amazonaws.com

In our case we simply need to change this:

gatsby-source-s3/src/gatsby-node.js

Line 36 in cb32cc2

Url: `https://s3.amazonaws.com/${rest.Name}/${Key}`,

...into this:

 Url: `https://s3.${awsConfig.region ? `${awsConfig.region}.` : ''}amazonaws.com/${rest.Name}/${Key}`, 

This is exactly what #13 does, but it's been open for six months now...

[iambottle]

Hello Guys,

How did you install the master code. since it is not published in npm yet.

need help here.

[DSchau]

Hello!

It’s Christmas so no not published yet.

Maybe next week!

[good3n]

Any update on this? Can't use with gatsby-image until this is resolved.

[robinmetral]

FYI, I ended up making my own version of gatsby-source-s3 in Typescript at https://github.com/robinmetral/gatsby-source-s3.

Feel free to use it if it helps! Any contribution over there is more than welcome, including bug reports 🙂

I'll support it at least as long as the maintenance status for this plugin and gatsby-source-s3-image is unclear.