Syntax error in BODY. Unexpected token: ')'

[TheSuunny]

Describe the bug

Hello. I have an old email with a reminder for a certain date sent using Gmail, once there was a normal email with an HTML body, but now for some reason it has disappeared and only the attached files remain.

When trying to execute folder.FetchAsync with fetchRequest(Message Summary Items.Full | Message Summary Items.UniqueId | MessageSummaryItems.PreviewText) an exception occurs (MessageSummaryItems.PreviewText).

C: A00000007 FETCH 56 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (REFERENCES)])
S: * 56 FETCH (UID 683 FLAGS (\Seen $NotJunk NotJunk) INTERNALDATE "22-Oct-2020 11:42:46 +0000" RFC822.SIZE 3285 ENVELOPE ("Thu, 22 Oct 2020 14:42:46 +0300" "Accepted: Invitation: =?utf-8?Q?=D0=9F=D0=BE=D0=B2=D1=82=D0=BE=D1=80=D0=?=@ Wed Jan 13, 2021 15:00 - 16:00 ([email protected])" ((NIL NIL "test" "test.space")) ((NIL NIL "test" "test.space")) ((NIL NIL "test" "test.space")) ((NIL NIL "test" "gmail.com")) NIL NIL NIL "<4054e02d-184d-48ea-a42c-1a7ce1afd22a@Spark>") BODYSTRUCTURE ((("text" "plain" ("charset" "utf-8") NIL NIL "7bit" 0 1 NIL ("inline" ("filename" "")) NIL NIL)(("text" "html" ("charset" "utf-8") NIL NIL "quoted-printable" 0 1 NIL ("inline" ("filename" "")) NIL NIL)("text" "calendar" ("charset" "utf-8") "<B4A0A860187D44DA8891F5C7997A690A>" NIL "quoted-printable" 716 20 NIL ("inline" ("filename" "")) NIL NIL) "related" ("boundary" "5f91703b_238e1f29_1e8") NIL NIL NIL) "alternative" ("boundary" "5f91703b_46e87ccd_1e8") NIL NIL NIL)("application" "ics" NIL NIL NIL "base64" 772 NIL ("attachment" ("filename" "reply.ics")) NIL NIL) "mixed" ("boundary" "5f91703b_3d1b58ba_1e8") NIL NIL NIL) BODY[HEADER.FIELDS (REFERENCES)] {2}
S: 
S: )
S: A00000007 OK FETCH Completed.
C: A00000008 UID FETCH 683 (BODY.PEEK[1.1]<0.512>)
S: * 56 FETCH (UID 683 BODY[1.1]<0> )
S: A00000008 OK UID FETCH Completed.

Exception

Unhandled exception. MailKit.Net.Imap.ImapProtocolException: Syntax error in BODY. Unexpected token: ')'
   at MailKit.Net.Imap.ImapFolder.FetchStreamAsync(ImapEngine engine, ImapCommand ic, Int32 index, Boolean doAsync)
   at MailKit.Net.Imap.ImapEngine.ProcessUntaggedResponseAsync(Boolean doAsync, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapCommand.StepAsync(Boolean doAsync)
   at MailKit.Net.Imap.ImapEngine.IterateAsync(Boolean doAsync)
   at MailKit.Net.Imap.ImapEngine.RunAsync(ImapCommand ic, Boolean doAsync)
   at MailKit.Net.Imap.ImapFolder.FetchPreviewTextAsync(FetchSummaryContext sctx, Dictionary`2 bodies, Int32 octets, Boolean doAsync, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapFolder.GetPreviewTextAsync(FetchSummaryContext sctx, Boolean doAsync, CancellationToken cancellationToken)
   at MailKit.Net.Imap.ImapFolder.FetchAsync(Int32 min, Int32 max, IFetchRequest request, Boolean doAsync, CancellationToken cancellationToken)
   at Test.Imap.Clients.BaseClient.Run() in /root/BaseClient.cs:line 57
   at Test.Imap.Services.InboxService.Start(ImapOptions options) in /root/Services/test.cs:line 29
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

Platform (please complete the following information):

• OS: MacOS
• .NET Runtime: CoreCLR
• .NET Framework: net6.0
• MailKit Version: 3.4.0

[jstedfast]

Okay, so it looks like this is the problematic response from the server:

C: A00000008 UID FETCH 683 (BODY.PEEK[1.1]<0.512>)
S: * 56 FETCH (UID 683 BODY[1.1]<0> )

Notice how after the BODY[1.1]<0> token, there is a space (spaces are used to separate tokens) and then a ).

MailKit's ImapClient is reading that ) and expecting to find the first 512 bytes of the message body as either a quoted string, a NIL token or a literal string (e.g. {512}\r\n....)

In other words, the IMAP protocol uses key/value pairs in the responses and in this case, the BODY[1.1]<0> is a key and it is missing the value that should be paired with it.

This is technically a server bug since it is skipping the value token but I'll try to think of a way to better handle this type of scenario.

FWIW, one of the things I am considering is moving toward a Telemetry-based system that provides me info about these types of situations so that I can figure out work-arounds for these server bugs and then have the code handle them by skipping until the end of the line and just dropping that particular response and proceeding normally (i.e. not throwing an ImapProtocolException).

I'm not sure how the MimeKit/MailKit developer base (and their respective user-bases) feels about that, though, and it would also mean I'd have to start taking dependencies on other NuGet packages. I'm also not sure if I'd be able to report telemetry that would provide enough context w/o having to push user-sensitive data (most people, when they report ImapProtocolExceptions like you did, will manually scrub the log data of any sensitive info).

Anyway, just some of stuff I've been thinking about...

[TheSuunny]

@jstedfast Can you tell me if it is possible to ignore such emails? So that FetchAsync doesn't break completely, and receives the rest of the emails, except for the current one?

I found another similar letter:

C: A00000006 FETCH 661 (UID FLAGS INTERNALDATE RFC822.SIZE ENVELOPE BODY)
S: * 661 FETCH (UID 2055 FLAGS (\Seen encrypted) INTERNALDATE "04-May-2021 13:37:26 +0000" RFC822.SIZE 69415 ENVELOPE ("Tue, 04 May 2021 13:36:24 +0000" "=?utf-8?b?0KHQutC40LTQutCwIDIwJSDQvdCwINCy0YHRkSDQtNC+IDA3LjA1?= =?utf-8?q?_23=3A59?=" (("=?utf-8?b?0JDRgNC10L3QtNCwINC40LPRgNGD0YjQtdC6IFRPWVJFTlQgVG95cmVudA==?=" NIL "test" "app16010.mail-carrotquest.io")) ((NIL NIL "test=app16010.mail-carrotquest.io" "mail-carrotquest.io")) (("=?utf-8?b?0JDRgNC10L3QtNCwINC40LPRgNGD0YjQtdC6IFRPWVJFTlQgVG95cmVudA==?=" NIL "test" "app16010.mail-carrotquest.io")) ((NIL NIL "test" "test.space")) NIL NIL NIL "<16010-906198814229007608-d47ab3da-b2a7-48d2-bbf1-c4de2b1cde86@mail-carrotquest.io>") BODY (("text" "plain" ("charset" "ascii") NIL NIL "7bit" 0 1)("text" "html" ("charset" "utf-8") NIL NIL "quoted-printable" 66367 1111) "alternative"))
S: A00000006 OK FETCH Completed.

I don't know why, but they lost the body of the message, although these are the yandex servers.

[jstedfast]

Can you tell me if it is possible to ignore such emails?

Unfortunately, it's not currently possible (hence my thoughts about using Telemetry and making MailKit just drop non-parsable responses).

I've heard about a lot of Yandex bugs unfortunately :-(

The original bug was with GMail, though, right?

Either way I'll try to find a solution, but one of the things I do is add a comment about which server(s) had the bug and/or implement a "Quirks Mode" for that server (see the ImapQuirksMode enum in ImapEngine.cs).

[TheSuunny]

Can you tell me if it is possible to ignore such emails?

Unfortunately, it's not currently possible (hence my thoughts about using Telemetry and making MailKit just drop non-parsable responses).

I've heard about a lot of Yandex bugs unfortunately :-(

The original bug was with GMail, though, right?

Either way I'll try to find a solution, but one of the things I do is add a comment about which server(s) had the bug and/or implement a "Quirks Mode" for that server (see the ImapQuirksMode enum in ImapEngine.cs).

Yes - Gmail

[jstedfast]

I just implemented a potential fix for this.

It doesn't work around the broken response, but it does avoid requesting preview text if the body is 0 bytes long which is what I think is causing GMail to send that broken response.

[jstedfast]

Try the latest build from https://www.myget.org/feed/mimekit/package/nuget/MailKit and see if that works

[TheSuunny]

@jstedfast Yes, everything works fine. Thank you :)

[jstedfast]

You're welcome!

[sommmen]

FWIW, one of the things I am considering is moving toward a Telemetry-based system that provides me info about these types of situations so that I can figure out work-arounds for these server bugs

Was reading the release notes and came across this, just my 2 cents, i'd be against telemetry in the main package, it seems intrusive. A whole opt-in seperate package would be a-ok, i wouldn't personally install it before having issues though.

[jstedfast]

@sommmen

A whole opt-in seperate package would be a-ok, i wouldn't personally install it before having issues though.

This is why I probably won't bother. The problem with having telemetry be opt-in and have it in a second package is that that's what everyone would do - no one would opt-in until/unless they hit a problem.

At that point, they can just send me the logs. The ProtocolLogger already redacts credentials.

This also avoids having to have a cloud database which avoids incurring extra costs to myself.

[sq5]

Hello, still having this Problem with some letters:

Syntax error in BODY. Unexpected token: ')' at MailKit.Net.Imap.ImapFolder.FetchStreamAsync(ImapEngine engine, ImapCommand ic, Int32 index, Boolean doAsync)

Mailkit Version 4.3.0, mail server - Yandex

[jstedfast]

@sq5 please file a new issue and include a protocol log, otherwise there's nothing I can do.