X-Forwarded-For not working in SaaS

[jjbayer]

Steps to reproduce:

Sending the following in my local dev environment:

curl -v http://localhost:8000/api/2/envelope/ \
     -H X-forwarded-for:4.4.4.4 \
    --data-binary '{"dsn":"$SENTRY_DSN"}\n{"type": "event"}\n{"user": {"ip_address": "{{auto}}"}}'

Results in

However, the same does not work for SaaS (ignores the X-Forwarded-For header):

curl -v https://o123.ingest.sentry.io/api/2/envelope/ \
     -H X-forwarded-for:4.4.4.4 \
    --data-binary '{"dsn":"$SENTRY_DSN"}\n{"type": "event"}\n{"user": {"ip_address": "{{auto}}"}}'

[jjbayer]

This was an attempt to emulate a tunnel setup, as discussed here: getsentry/sentry-javascript#5798 (reply in thread)

[jan-auer]

@beezz could we get some help on this one, please? I vaguely remember that we do not pass this header through for a reason, but I cannot find documentation on that.

Note: We need to apply the same policy to x-vercel-forwarded-for

[AbhiPrasad]

Synced offline - created an OPs ticket here for the SRE folks to investigate further. https://getsentry.atlassian.net/browse/OPS-3823

[iker-barriocanal]

We'll track this in the OPS ticket above.

[AaronFriel]

@iker-barriocanal sorry, can you keep this open until your internal tracker resolves the issue?

I am unable to subscribe to notifications on your internal tracker, and it appears this is closed as resolved.

[IanVS]

Does that mean this hasn't been fixed yet? I was just about to test it out.

[AbhiPrasad]

Let's re-open - I'll take care of updating with the OPs tickets.

[AbhiPrasad]

Hey, thanks for your patience everyone. We've made the changes in our SaaS infrastructure to now support X-Forwarded-For header. It should correctly forward the ip_address now.

We've also added support for using a proprietary header X-Sentry-Forwarded-For in cases where you do not want to set X-Forwarded-For. I will update our docs to reflect this.

[szwabodev]

Hello, sorry for bringing back an old issue, but I am having trouble with getting real user IP to save in my tunneled Sentry errors/messages while using Cloudflare as a proxy for my NextJS application. The problem is that Cloudflare replaces the "X-Forwarded-For" header with its own server IP and the real user IP is under "CF-Connecting-IP" header. I have tried using a middleware for my Sentry tunnel endpoint to replace the request's and response's "X-Forwarded-For" and "X-Vercel-Forwarder-For" as well as "X-Sentry-Forwarded-For" with the value of real user IP, however Sentry still keeps storing the cloudflare IP instead. Would it be possible to add this "CF-Connecting-IP" as a priority header so that Sentry can automatically grab its value instead of the others if it exists? I will keep trying to replace it somehow in the middleware in the meantime, but it would be much easier if Sentry did it automatically.

[AbhiPrasad]

@szwabodev please open a new GH issue - we can take a look!

[Niaro]

@szwabodev have u opened a GH issue? could u please share the link? Im also experiencing proxy IP instead of client IP in sentry

[szwabodev]

@szwabodev have u opened a GH issue? could u please share the link? Im also experiencing proxy IP instead of client IP in sentry

Hey, yes it was reported here: #3493

Looks like it should be handled already, but I just checked 2 of my latest errors from 7.113.0 SDK version and it still has cloudflare IP in user context instead of real one. I will comment on the linked issue about it.

[reyoucat]

This still looks the same.
I guess I should just put it in as a custom tag. or useX-Sentry-Forwarded-For

[AbhiPrasad]

@reyoucat A lot has changed since we made this fix, there could be a different root cause here. Could you open a new GitHub issue in the Sentry repo that describes your setup and why the forwarded for header is not working? https://github.com/getsentry/sentry.