Unable to bind service to create a secret #16141

jeff-phillips-18 · 2017-09-05T13:33:22Z

Attempting to create the binding for a service is failing.

Error:
Error injecting binding results for Binding "myproject/mongodb-persistent-kqwxg-cjxwq": Unexpected error in response: secrets "mongodb-persistent-kqwxg-7ieki" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion servicecatalog.k8s.io/v1alpha1 Kind Binding: allowed by openshift authorizer, no matches for servicecatalog.k8s.io/, Kind=Binding

Version

$ oc version
oc v1.5.0-alpha.2+631de37-4492
kubernetes v1.7.0+695f48a16f
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v3.7.0-alpha.1+631de37-125
kubernetes v1.7.0+695f48a16f

jeff-phillips-18 · 2017-09-05T13:34:02Z

@deads2k

deads2k · 2017-09-05T13:41:13Z

Please include the client requests being made. Normally --loglevel=8

spadgett · 2017-09-05T17:45:44Z

@pmorie Can you help? It's the service catalog creating this secret, so we aren't able to get the request for @deads2k. Binding is currently broken.

pmorie · 2017-09-06T17:36:27Z

@jeff-phillips-18 how are you deploying catalog? oc cluster up?

pmorie · 2017-09-06T18:19:25Z

@jpeeler is going to take a look at this.

spadgett · 2017-09-06T18:20:50Z

@pmorie @jpeeler Jeff and I both see this with cluster up.

$ oc cluster up --version=latest --service-catalog

pmorie · 2017-09-06T18:26:39Z

@spadgett can you bump the loglevel in the template to get @deads2k the info he requested?

jpeeler · 2017-09-07T15:34:04Z

Here the controller output at loglevel 8 with the issue reproduced:
controller-manager.txt
Additional logging:
output2-truncated.txt

jpeeler · 2017-09-07T15:55:13Z

I0907 15:29:42.919771       1 controller_binding.go:448] Creating/updating Secret projectname/mongodb-persistent-r4pmf-credentials-6ip7m
I0907 15:29:42.919997       1 round_trippers.go:383] GET https://172.30.0.1:443/api/v1/namespaces/projectname/secrets/mongodb-persistent-r4pmf-credentials-6ip7m
I0907 15:29:42.920036       1 round_trippers.go:390] Request Headers:
I0907 15:29:42.920055       1 round_trippers.go:393]     Accept: application/json, */*
I0907 15:29:42.920072       1 round_trippers.go:393]     User-Agent: controller-manager/v1.7.0+$Format:%h$ (linux/amd64) kubernetes/$Format/service-catalog-controller-manager
I0907 15:29:42.920088       1 round_trippers.go:393]     Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXNlcnZpY2UtY2F0YWxvZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlci10b2tlbi0zY2xkbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImIxMzM4ZThhLTkzZTAtMTFlNy1iODEzLTUyNTQwMDI0N2I3NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXNlcnZpY2UtY2F0YWxvZzpzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlciJ9.Q7RyoNCXORX3oIm1tU3zRDfpfSdLRPEj33LoZ2Gt7POGoTG2RtBU5To9mKZ1l6ZpHOLxs72ecujQDnH5T0Qlnfq-pQB7jT0cQ2Y7S55iB1daaw8QPLhrZjZQOddPNP322vy5EWzEAvHT9HEMTVL7ItjDhzoNd0BeSTngY--xMCaQHJBEyybrKIWxl3-cTYai-jjrfOzVfgpJWftcrqveyIUp7wUnRC_U6UTfCtArg85g3ecDzZ7qNC9wUq-U5O2WjeSjokXYE0znjrH6TlA4WUtb5qoGl8ljj3bytnqfQCLWNxFkN2AuVJN8DE9y6kRCJGk2MBYU4QstHtzyVqS0gA
I0907 15:29:42.929911       1 round_trippers.go:408] Response Status: 404 Not Found in 9 milliseconds
I0907 15:29:42.929935       1 round_trippers.go:411] Response Headers:
I0907 15:29:42.929942       1 round_trippers.go:414]     Content-Type: application/json
I0907 15:29:42.929948       1 round_trippers.go:414]     Content-Length: 258
I0907 15:29:42.929952       1 round_trippers.go:414]     Date: Thu, 07 Sep 2017 15:29:42 GMT
I0907 15:29:42.929957       1 round_trippers.go:414]     Cache-Control: no-store
I0907 15:29:42.929991       1 request.go:991] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"mongodb-persistent-r4pmf-credentials-6ip7m\" not found","reason":"NotFound","details":{"name":"mongodb-persistent-r4pmf-credentials-6ip7m","kind":"secrets"},"code":404}
I0907 15:29:42.930189       1 request.go:991] Request Body: {"kind":"Secret","apiVersion":"v1","metadata":{"name":"mongodb-persistent-r4pmf-credentials-6ip7m","namespace":"projectname","creationTimestamp":null,"ownerReferences":[{"apiVersion":"servicecatalog.k8s.io/v1alpha1","kind":"Binding","name":"mongodb-persistent-r4pmf-d7swv","uid":"1356dab8-93e1-11e7-a794-0242ac110006","controller":true,"blockOwnerDeletion":true}]},"data":{"admin_password":"aExRd3hoUHZiRXRmd2hTVw==","database_name":"c2FtcGxlZGI=","password":"SllrVVd2V1c2V25vUmdEMA==","uri":"bW9uZ29kYjovLzE3Mi4zMC4yLjE2MjoyNzAxNw==","username":"dXNlclRGWQ=="}}
I0907 15:29:42.930243       1 round_trippers.go:383] POST https://172.30.0.1:443/api/v1/namespaces/projectname/secrets
I0907 15:29:42.930255       1 round_trippers.go:390] Request Headers:
I0907 15:29:42.930263       1 round_trippers.go:393]     Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXNlcnZpY2UtY2F0YWxvZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlci10b2tlbi0zY2xkbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImIxMzM4ZThhLTkzZTAtMTFlNy1iODEzLTUyNTQwMDI0N2I3NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXNlcnZpY2UtY2F0YWxvZzpzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlciJ9.Q7RyoNCXORX3oIm1tU3zRDfpfSdLRPEj33LoZ2Gt7POGoTG2RtBU5To9mKZ1l6ZpHOLxs72ecujQDnH5T0Qlnfq-pQB7jT0cQ2Y7S55iB1daaw8QPLhrZjZQOddPNP322vy5EWzEAvHT9HEMTVL7ItjDhzoNd0BeSTngY--xMCaQHJBEyybrKIWxl3-cTYai-jjrfOzVfgpJWftcrqveyIUp7wUnRC_U6UTfCtArg85g3ecDzZ7qNC9wUq-U5O2WjeSjokXYE0znjrH6TlA4WUtb5qoGl8ljj3bytnqfQCLWNxFkN2AuVJN8DE9y6kRCJGk2MBYU4QstHtzyVqS0gA
I0907 15:29:42.930272       1 round_trippers.go:393]     Accept: application/json, */*
I0907 15:29:42.930280       1 round_trippers.go:393]     Content-Type: application/json
I0907 15:29:42.930287       1 round_trippers.go:393]     User-Agent: controller-manager/v1.7.0+$Format:%h$ (linux/amd64) kubernetes/$Format/service-catalog-controller-manager
I0907 15:29:42.940554       1 round_trippers.go:408] Response Status: 403 Forbidden in 10 milliseconds
I0907 15:29:42.940580       1 round_trippers.go:411] Response Headers:
I0907 15:29:42.940588       1 round_trippers.go:414]     Cache-Control: no-store
I0907 15:29:42.940593       1 round_trippers.go:414]     Content-Type: application/json
I0907 15:29:42.940598       1 round_trippers.go:414]     Content-Length: 483
I0907 15:29:42.940604       1 round_trippers.go:414]     Date: Thu, 07 Sep 2017 15:29:42 GMT
I0907 15:29:42.940652       1 request.go:991] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"mongodb-persistent-r4pmf-credentials-6ip7m\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion servicecatalog.k8s.io/v1alpha1 Kind Binding: allowed by openshift authorizer, no matches for servicecatalog.k8s.io/, Kind=Binding","reason":"Forbidden","details":{"name":"mongodb-persistent-r4pmf-credentials-6ip7m","kind":"secrets"},"code":403}
W0907 15:29:42.940833       1 controller_binding.go:311] Error injecting binding results for Binding "projectname/mongodb-persistent-r4pmf-d7swv": Unexpected error in response: secrets "mongodb-persistent-r4pmf-credentials-6ip7m" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion servicecatalog.k8s.io/v1alpha1 Kind Binding: allowed by openshift authorizer, no matches for servicecatalog.k8s.io/, Kind=Binding
I0907 15:29:42.940859       1 controller_binding.go:545] Setting Binding 'projectname/mongodb-persistent-r4pmf-d7swv' condition "Ready" to False
I0907 15:29:42.940888       1 controller_binding.go:584] Updating status for Binding projectname/mongodb-persistent-r4pmf-d7swv
I0907 15:29:42.940989       1 request.go:991] Request Body: {"kind":"Binding","apiVersion":"servicecatalog.k8s.io/v1alpha1","metadata":{"name":"mongodb-persistent-r4pmf-d7swv","generateName":"mongodb-persistent-r4pmf-","namespace":"projectname","selfLink":"/apis/servicecatalog.k8s.io/v1alpha1/namespaces/projectname/bindings/mongodb-persistent-r4pmf-d7swv","uid":"1356dab8-93e1-11e7-a794-0242ac110006","resourceVersion":"96","creationTimestamp":"2017-09-07T15:27:35Z","finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"instanceRef":{"name":"mongodb-persistent-r4pmf"},"parametersFrom":[{"secretKeyRef":{"name":"mongodb-persistent-r4pmf-bind-parameters-d3xvt","key":"parameters"}}],"secretName":"mongodb-persistent-r4pmf-credentials-6ip7m","externalID":"274a04e7-4262-489d-910b-11482fd36d89"},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2017-09-07T15:27:36Z","reason":"ErrorInjectingBindResult","message":"Error injecting bind result Error injecting binding results for Binding \"projectname/mongodb-persistent-r4pmf-d7swv\": Unexpected error in response: secrets \"mongodb-persistent-r4pmf-credentials-6ip7m\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion servicecatalog.k8s.io/v1alpha1 Kind Binding: allowed by openshift authorizer, no matches for servicecatalog.k8s.io/, Kind=Binding"}]}}

djzager · 2017-09-07T15:58:35Z

Is this because they are now called ServiceInstanceCredential? https://github.com/openshift/service-catalog/blob/master/contrib/examples/apiserver/binding.yaml

pmorie · 2017-09-07T15:59:04Z

@djzager no - that change hasn't landed in origin yet.

djzager · 2017-09-07T16:00:12Z

@pmorie Origin does not use the openshift fork of the service-catalog?

Just when I thought I had the connections made.

pmorie · 2017-09-07T16:00:40Z

@djzager see #16150

pmorie · 2017-09-07T17:50:08Z

I believe the issue here is that the garbage collector doesn't know about aggregated APIs until 1.8

ironcladlou · 2017-09-07T18:10:44Z

Looks like the wired rest mapper only knows about compiled types. Pretty sure you would need to use a dynamic mapper backed by a discovery client to map custom resource instances.

deads2k · 2017-09-07T18:14:58Z

Looks like the wired rest mapper only knows about compiled types. Pretty sure you would need to use a dynamic mapper backed by a discovery client to map custom resource instances.

Yeah, I'm looking for that pull to pick now

deads2k · 2017-09-07T19:00:12Z

Opened #16215 . It won't make GC work for you, but you'll at least be able to create resources.

jpeeler · 2017-09-07T21:30:52Z

I can confirm that the rest mapping problem goes away and the error changes slightly:

I0907 21:16:32.551730       1 request.go:991] Response Body: {"kind":"Binding","apiVersion":"servicecatalog.k8s.io/v1alpha1","metadata":{"name":"mongodb-persistent-c6ztb-p2tsh","generateName":"mongodb-persistent-c6ztb-","namespace":"meproject","selfLink":"/apis/servicecatalog.k8s.io/v1alpha1/namespaces/meproject/bindings/mongodb-persistent-c6ztb-p2tsh/status","uid":"42693c12-9411-11e7-bb2d-0242ac110002","resourceVersion":"65","creationTimestamp":"2017-09-07T21:12:30Z","finalizers":["kubernetes-incubator/service-catalog"]},"spec":{"instanceRef":{"name":"mongodb-persistent-c6ztb"},"parametersFrom":[{"secretKeyRef":{"name":"mongodb-persistent-c6ztb-bind-parameters-3jtfg","key":"parameters"}}],"secretName":"mongodb-persistent-c6ztb-credentials-vp2y8","externalID":"6f3544d5-80b0-4835-a798-2aa488afe453"},"status":{"conditions":[{"type":"Ready","status":"False","lastTransitionTime":"2017-09-07T21:12:30Z","reason":"ErrorInjectingBindResult","message":"Error injecting bind result Error injecting binding results for Binding \"meproject/mongodb-persistent-c6ztb-p2tsh\": Unexpected error in response: secrets \"mongodb-persistent-c6ztb-credentials-vp2y8\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: User \"system:serviceaccount:kube-service-catalog:service-catalog-controller\" cannot update bindings/finalizers.servicecatalog.k8s.io in project \"meproject\", \u003cnil\u003e"}]}}
I0907 21:16:32.551988       1 controller.go:193] Error syncing Binding meproject/mongodb-persistent-c6ztb-p2tsh: Unexpected error in response: secrets "mongodb-persistent-c6ztb-credentials-vp2y8" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot update bindings/finalizers.servicecatalog.k8s.io in project "meproject", <nil>
I0907 21:16:32.552101       1 event.go:218] Event(v1.ObjectReference{Kind:"Binding", Namespace:"meproject", Name:"mongodb-persistent-c6ztb-p2tsh", UID:"42693c12-9411-11e7-bb2d-0242ac110002", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"65", FieldPath:""}): type: 'Warning' reason: 'ErrorInjectingBindResult' Error injecting binding results for Binding "meproject/mongodb-persistent-c6ztb-p2tsh": Unexpected error in response: secrets "mongodb-persistent-c6ztb-credentials-vp2y8" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot update bindings/finalizers.servicecatalog.k8s.io in project "meproject", <nil>

deads2k · 2017-09-07T22:30:51Z

I can confirm that the rest mapping problem goes away and the error changes slightly:

That's a permissions problem. You cannot set blockownerdeletion unless you have the power to update the referee/finalizers subresource. That's actually a kube admission protection present since 1.7. Must be that you haven't turned it on before.

Fixes this error: secrets "mongodb-persistent-c6ztb-credentials-vp2y8" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: User "system:serviceaccount:kube-service-catalog:service-catalog-controller" cannot update bindings/finalizers.servicecatalog.k8s.io in project "meproject" (openshift#16141)

jpeeler · 2017-09-08T17:30:02Z

@deads2k To be clear, "referee" is referring to bindings, and not some other k8s type I don't know about right? (Edit: my understanding is correct.)

pmorie · 2017-09-10T04:48:13Z

I believe that @jpeeler has established that this is solved by a combination of #16215 and #16253

djzager · 2017-09-11T19:06:40Z

After merging the 2 PRs (#16215 and #16253), I was able to create bindings:


I0911 19:04:19.608288      64 controller_binding.go:126] Processing Binding foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p
--
  | I0911 19:04:19.611587      64 controller.go:357] Creating client for Broker ansible-service-broker, URL: https://asb-1338-ansible-service-broker.172.17.0.1.nip.io
  | I0911 19:04:19.611638      64 controller_binding.go:199] Adding/Updating Binding foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p
  | I0911 19:04:19.643404      64 controller_binding.go:448] Creating/updating Secret foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-credentials-pvicf
  | I0911 19:04:19.652611      64 controller_binding.go:545] Setting Binding 'foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p' condition "Ready" to True
  | I0911 19:04:19.652646      64 controller_binding.go:565] Found status change for Binding "foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2017-09-11 19:04:19.652601094 +0000 UTC
  | I0911 19:04:19.652671      64 controller_binding.go:584] Updating status for Binding foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p
  | I0911 19:04:19.759611      64 request.go:638] Throttling request took 188.569612ms, request: PUT:https://172.30.0.1:443/apis/servicecatalog.k8s.io/v1alpha1/serviceclasses/dancer-mysql-persistent
  | I0911 19:04:19.767963      64 controller_broker.go:259] Reconciled serviceClass dancer-mysql-persistent (broker template-service-broker)
  | I0911 19:04:19.767996      64 controller_broker.go:244] Reconciling serviceClass mysql-persistent (broker template-service-broker)
  | I0911 19:04:19.768015      64 controller_broker.go:363] Found existing serviceClass mysql-persistent; updating
  | I0911 19:04:19.959478      64 request.go:638] Throttling request took 306.594786ms, request: PUT:https://172.30.0.1:443/apis/servicecatalog.k8s.io/v1alpha1/namespaces/foo-project/bindings/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p/status
  | I0911 19:04:19.970083      64 controller_binding.go:333] Successfully bound to Instance foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m of ServiceClass dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates at Broker ansible-service-broker
  | I0911 19:04:19.970166      64 event.go:218] Event(v1.ObjectReference{Kind:"Binding", Namespace:"foo-project", Name:"dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p", UID:"f5b02965-9723-11e7-b228-0242ac110004", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"322", FieldPath:""}): type: 'Normal' reason: 'InjectedBindResult' Injected bind result
  | I0911 19:04:19.980554      64 controller_binding.go:121] Not processing event for Binding foo-project/dh-ansibleplaybookbundle-rhscl-postgresql-apb-lates-zvs9m-sc90p because checksum showed there is no work to do

Automatic merge from submit-queue add dynamic rest mapper to the admission plugin initializer fixes #16141 picks the dynamic memcache discovery for the RESTMapper and wires it for admission. There's a post-start hook that starts it refreshing.

spadgett added kind/bug Categorizes issue or PR as related to a bug. priority/P1 labels Sep 5, 2017

pmorie mentioned this issue Sep 6, 2017

Rebase service catalog to v0.0.17 #16150

Merged

pmorie assigned jpeeler Sep 6, 2017

deads2k mentioned this issue Sep 7, 2017

add dynamic rest mapper to the admission plugin initializer #16215

Merged

jpeeler mentioned this issue Sep 8, 2017

catalog: add update permission for bindings/finalizers #16253

Merged

pweil- added the component/kubernetes label Sep 11, 2017

openshift-merge-robot closed this as completed in #16215 Sep 12, 2017

magaldima mentioned this issue Jul 27, 2018

pod failed to be created - cannot set an ownerRef on a resource you can't delete argoproj/argo-workflows#922

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to bind service to create a secret #16141

Unable to bind service to create a secret #16141

jeff-phillips-18 commented Sep 5, 2017

jeff-phillips-18 commented Sep 5, 2017

deads2k commented Sep 5, 2017

spadgett commented Sep 5, 2017

pmorie commented Sep 6, 2017 •

edited

Loading

pmorie commented Sep 6, 2017

spadgett commented Sep 6, 2017

pmorie commented Sep 6, 2017

jpeeler commented Sep 7, 2017 •

edited

Loading

jpeeler commented Sep 7, 2017 •

edited by pmorie

Loading

djzager commented Sep 7, 2017

pmorie commented Sep 7, 2017

djzager commented Sep 7, 2017

pmorie commented Sep 7, 2017

pmorie commented Sep 7, 2017

ironcladlou commented Sep 7, 2017

deads2k commented Sep 7, 2017

deads2k commented Sep 7, 2017

jpeeler commented Sep 7, 2017 •

edited by pmorie

Loading

deads2k commented Sep 7, 2017 •

edited

Loading

jpeeler commented Sep 8, 2017 •

edited

Loading

pmorie commented Sep 10, 2017

djzager commented Sep 11, 2017

Unable to bind service to create a secret #16141

Unable to bind service to create a secret #16141

Comments

jeff-phillips-18 commented Sep 5, 2017

Version

jeff-phillips-18 commented Sep 5, 2017

deads2k commented Sep 5, 2017

spadgett commented Sep 5, 2017

pmorie commented Sep 6, 2017 • edited Loading

pmorie commented Sep 6, 2017

spadgett commented Sep 6, 2017

pmorie commented Sep 6, 2017

jpeeler commented Sep 7, 2017 • edited Loading

jpeeler commented Sep 7, 2017 • edited by pmorie Loading

djzager commented Sep 7, 2017

pmorie commented Sep 7, 2017

djzager commented Sep 7, 2017

pmorie commented Sep 7, 2017

pmorie commented Sep 7, 2017

ironcladlou commented Sep 7, 2017

deads2k commented Sep 7, 2017

deads2k commented Sep 7, 2017

jpeeler commented Sep 7, 2017 • edited by pmorie Loading

deads2k commented Sep 7, 2017 • edited Loading

jpeeler commented Sep 8, 2017 • edited Loading

pmorie commented Sep 10, 2017

djzager commented Sep 11, 2017

pmorie commented Sep 6, 2017 •

edited

Loading

jpeeler commented Sep 7, 2017 •

edited

Loading

jpeeler commented Sep 7, 2017 •

edited by pmorie

Loading

jpeeler commented Sep 7, 2017 •

edited by pmorie

Loading

deads2k commented Sep 7, 2017 •

edited

Loading

jpeeler commented Sep 8, 2017 •

edited

Loading