Search Support

Avoid support scams. We will never ask you to call or text a phone number or share personal information. Please report suspicious activity using the “Report Abuse” option.

Learn More

Mail Folder Gets Corrupted with Thunderbird Daily

  • 27 replies
  • 2 have this problem
  • 44 views
  • Last reply by bsfinkel

more options

I am running Thunderbird Daily. I had a problem recently where a number of my mail folders got corrupted; mail messages were missing. So, what I did was to restore from a good backup before the corruption the folder and folder.msf files to a new directory. One corrupt folder is my Sent folder. I then copied the restored Sent and Sent.msf to my TB Local Folders directory as Sent-restored and sent-restored.msf (while TB was not running). I started TB and looked at the sent-restored folder. The messages are there, and they are intact. I then select ~2400 mail messages from Sent-restored and copy into Sent. When I look at the Sent directory, I see that all of these messages have been copied, and the directory seems OK - all the correct dates, recipients, and subjects. But ALL of the messages I copied now have the same text - the text of one (I believe) of the messages.

I have tried this twice, with the same results. What can I do to debug/correct this? Thanks.

Modified by Wayne Mery

All Replies (7)

more options

I am having something similar happen also, but I am using the beta's. I was on Beta 5 and at least one of my folders with a few thousand messages somehow seems to have corrupted. It was fine one day, now, a few days ago I went in and I have a BUNCH of messages that are duplicates with weird dates, subjects and from fields and, about 90% of the messages are now missing.

Deleting the index doesn't help any.

If I look into the regular folder file (The one I am looking at is for Amazon), the file has a normal size and reads fine and looks fine in Notepad++. I can see the individual messages and all the correct info.

I have upgraded to 95.0b5 and the issue is still there.

I am by far not a programmer nor am I a DBA, but, if corruption happens, isn't there some sort of a script that can be run on the folder files to check and repair? (In my past, I did some work on an application that used a SQLite DB and our dev team had a script that just went through the DB and cleaned and repaired it magically). To me, this would be gold to have.

Helpful?

more options

The mail file is in mbox format. In this format, each mail message begins with a line that begins with "From " starting in column 1. Any line that begins with those characters is treated as the start of a new message. That is why, when the body of a mail message has a line that begins with those characters, a ">" character is pre-pended to that line to ensure that that line is not treated as the beginning of a new mail message. If your mail file is not too large (I once had to extract a few mail messages from an mbox file that was over 2 Gb in size), you can check to see if it is in proper format. Try

     grep "^From " mboxfile

to see all the lines that signify the beginning of a new mail message In one of my files I see

    From - Sat Jan 11 21:01:54 2020

This line gives a timestamp for the message. In mbox format, following this initial line is the RFC 5322 mail header piece. Then the first null line separates the header from the body of the mail message. I would check to see if your file is in proper mbox file. I did verify that if the .msf file is missing, Thunderbird will rebuild it when that folder is first re-accessed. I am now back on the non-daily build (91.3.2), and I have not had a chance to test my message-copy problem. I will try to find a few minutes this afternoon to test.

The mbox file is a straight character-based file. It may contain characters in a non-ASCII character set, if a mail message is encoded therein, but the file is NOT a binary file. Any text editor should be able to open the file, assuming that it is not too large for the editor.

Note that in Windows 10 I have GOW (GNU On Windows) installed; it gives me the GNU Unix utilities - grep, sed, awk, et alia.

Helpful?

more options

I just ran another test. I copied six messages from a sent-211118 folder to my Sent folder. The first copied correctly, but the other five did NOT. The .msf file showed the date, time, sender, subject of the mail message copied, but the content of the message was from another message. I now have to determine from where that message came. I looked at one message via cntl-u, and it is not a complete message; it does NOT begin with a "From " line. I cannot believe that my Sent folder or the sent-211118 folder are corrupted.

Helpful?

more options

I did some testing, and the results are in a .txt file. I cannot attach it, so I will include it in multiple replies.

Thunderbird Sent Folder Corruption d:\computer\tb-sent-corruption.txt 05Dec21 0422PM Barry Finkel bsfinkel@att.net

01) At some point many of my Thunderbird folders got corrupted;

   mail messages from 3/2020 through 08/2021 were missing.
   I have no idea what happened.

02) I "restored" some of the folders from a weekly Monday

   backup before the corruption to a new directory.

03) One corrupted file was "Sent", so I copied the restored

   Sent file to the Thunderbird pop.att.yahoo.com directory
   as Sent-restored.

03) I then copied one mail message from Sent-restored to Sent.

   Thunderbird automatically built a Sent-restored.msf file.
   The one mail message copied fine; it was intact, had the
   correct text, and the index information displayed (sender,
   recipient(s), date, time, subject) were all OK.

04) I then copied a bunch of mail messages from Sent-restored

   to Sent.  The firs one copied was fine, but all the rest
   were corrupted.  The displayed index was fine, but the text
   of the mail messages was all the same.  I have tried this
   a number of times, all with the same result.

05) I finally took my current Sent file and extracted (via grep,

   head, and tail) all of the mail messages I had sent since the
   restored backup (sent-restored) file.

06) I combined the Sent-restored file with the new Sent messages

   mbox file, and I created a new Sent file.  This was done with
   Thunderbird shut down.  Thunderbird then re-created a new
   Sent.msf file.  This new Sent folder looks OK; I can search
   it to find messages.   This was all with Thunderbird Daily.

07) I regressed from Daily to 91.3.2 on 11/30/2021, as someone on.

   the Thunderbird Forum said that the Daily build had some
   known corruption problems.

08) I tried coping one mail message from Sent-restored to Sent,

   and it copied fine.  I then deleted the now-duplicate message
   in the Sent folder.

09) When I tried to copy two messages at the same time, The first

   copied OK, but the second had the same corruption I saw with the
   Daily build.

10) I now am trying to determine from where the text of the corrupted

   mail message came.  The beginning text of the message in the mbox
   file (seen via CNTL-U) does not begin with a "From: " line.
   It does contain five complete messages, with the "From: " line
   "escaped" with the standard ">" character.  The last piece of the
   message is an incomplete mail message; the end of a base64-encoded
   JPEG attachment is missing.  There are 1,355 lines in the message.

11) I the corrupted text coming from the source file (Sent-restored)

   or from the destination file (Sent)?

12) Right now I cannot believe that the two mbox files nor the

   corresponding .msf files are corrupt.

13) The first line of the corrupted message has this text:

   you like your email forwarded?
   And this text line does NOT appear in the source file.
   It appears three times in the destination file.
   This implies that the messages being corrupted during the copy
   operation are getting the text from the destination file.

14) The last line of the corrupted message has this text:

   naM/LxzTbi9+0ZRV2r3FQCQLjjOKAHOrO4d2JY+lNKFVOM1bgD3cyrjAFbN1p8AiVUQh8DOa
   It appears ONCE in the source file and FOUR times in the dest file.

15) The 1,355-line message appears in the destination file in three

   places, starting with these lines:
        16627687
        16629577
        16631296  This is the last message in the destination file.

16) tail +16630933 Sent > \temp\sent-16630933

   The resulting file has
        a) a message I sent Tue, 16 Nov 2021 14:26:46 GMT
           two null lines
           the 1,355 lines that comprise the corrupted message
        b) three messages I sent today (12/05/2021)

17) In my Sent file, line 16627300 is the beginning of a message

   I sent Fri, 03 Dec 2021 20:39:56 GMT
   tail +16627300 Sent > \temp\sent-16627300
   The resulting file has 5,487 lines and contains:
   a) the message I sent: From - Fri, 03 Dec 2021 20:39:56 GMT
   b) a null line
   c) the message I sent: From - Tue, 16 Nov 2021 14:26:46 GMT
   d) two null lines
   e) "you like your e-mail forwarded?"  [line  388]
      "naM/..."                          [line 1742]
      "ou like your mail forwarded"      [line 1744]
      "u like your e-mail forwarded?"    [line 1779]
      " like your e-mail forwarded?"     [line 1804]
      "like your e-mail forwarded?"      [line 1865]
      "you like your e-mail forwarded?"  [line 2278]
      "naM/..."                          [line 3632]
      "you like your e-mail forwarded?"  [line 3997]
      "naM/..."                          [line 5351]
   f) the three messages I sent today: From - Sun, 05 Dec 2021 15:08:46 GMT [line 5353]
   There is only one line with a non-ASCII character - in a
   message I sent today.

18) The index displayed for the Sent folder in Thunderbird has

   the message I sent Dec 03 followed by the three messages I sent
   today (Dec 05).   The displayed index looks fine, and each of
   the four messages looks fine.  So, what in the Sent mbox file
   is causing the corruption?


D:\computer>cat tb-sent-corruption.txt Thunderbird Sent Folder Corruption d:\computer\tb-sent-corruption.txt 05Dec21 0236PM Barry Finkel bsfinkel@att.net

01) At some point many of my Thunderbird folders got corrupted;

   mail messages from 3/2020 through 08/2021 were missing.
   I have no idea what happened.

02) I "restored" some of the folders from a weekly Monday

   backup before the corruption to a new directory.

03) One corrupted file was "Sent", so I copied the restored

   Sent file to the Thunderbird pop.att.yahoo.com directory
   as Sent-restored.

03) I then copied one mail message from Sent-restored to Sent.

   Thunderbird automatically built a Sent-restored.msf file.
   The one mail message copied fine; it was intact, had the
   correct text, and the index information displayed (sender,
   recipient(s), date, time, subject) were all OK.

04) I then copied a bunch of mail messages from Sent-restored

   to Sent.  The firs one copied was fine, but all the rest
   were corrupted.  The displayed index was fine, but the text
   of the mail messages was all the same.  I have tried this
   a number of times, all with the same result.

05) I finally took my current Sent file and extracted (via grep,

   head, and tail) all of the mail messages I had sent since the
   restored backup (sent-restored) file.

06) I combined the Sent-restored file with the new Sent messages

   mbox file, and I created a new Sent file.  This was done with
   Thunderbird shut down.  Thunderbird then re-created a new
   Sent.msf file.  This new Sent folder looks OK; I can search
   it to find messages.   This was all with Thunderbird Daily.

07) I regressed from Daily to 91.3.2 on 11/30/2021, as someone on.

   the Thunderbird Forum said that the Daily build had some
   known corruption problems.

08) I tried coping one mail message from Sent-restored to Sent,

   and it copied fine.  I then deleted the now-duplicate message
   in the Sent folder.

09) When I tried to copy two messages at the same time, The first

   copied OK, but the second had the same corruption I saw with the
   Daily build.

10) I now am trying to determine from where the text of the corrupted

   mail message came.  The beginning text of the message in the mbox
   file (seen via CNTL-U) does not begin with a "From: " line.
   It does contain five complete messages, with the "From: " line
   "escaped" with the standard ">" character.  The last piece of the
   message is an incomplete mail message; the end of a base64-encoded
   JPEG attachment is missing.  There are 1,355 lines in the message.

11) I the corrupted text coming from the source file (Sent-restored)

   or from the destination file (Sent)?

12) Right now I cannot believe that the two mbox files nor the

   corresponding .msf files are corrupt.

13) The first line of the corrupted message has this text:

   you like your email forwarded?
   And this text line does NOT appear in the source file.
   It appears three times in the destination file.
   This implies that the messages being corrupted during the copy
   operation are getting the text from the destination file.

14) The last line of the corrupted message has this text:

   naM/LxzTbi9+0ZRV2r3FQCQLjjOKAHOrO4d2JY+lNKFVOM1bgD3cyrjAFbN1p8AiVUQh8DOa
   It appears ONCE in the source file and FOUR times in the dest file.

15) The 1,355-line message appears in the destination file in three

   places, starting with these lines:
        16627687
        16629577
        16631296  This is the last message in the destination file.

Helpful?

more options

Sorry for the mis-formatting/posting. Here are items 19-end.

19) I again did

       grep -n "^From " Sent > \temp\sent-from-211205-1500
   and compared it with the sent-from-211205-0822 file I
   created this morning.  The only differences are the
   three (now four) messages I sent today after 08:22.
   So, the "^From " lines in the mbox file seem OK.

20) But ...

        c:\temp>tail sent-from-211205-0822
        16625167:From - Fri, 03 Dec 2021 01:55:27 GMT
        16625201:From - Fri, 03 Dec 2021 03:02:55 GMT
        16625427:From - Fri, 03 Dec 2021 14:47:04 GMT
        16627237:From - Fri, 03 Dec 2021 14:53:48 GMT
        16627257:From - Fri, 03 Dec 2021 17:20:58 GMT
        16627280:From - Fri, 03 Dec 2021 17:23:51 GMT
        16627300:From - Fri, 03 Dec 2021 20:39:56 GMT
        16627324:From - Tue, 16 Nov 2021 14:26:46 GMT
        16629214:From - Tue, 16 Nov 2021 14:26:46 GMT
        16630933:From - Tue, 16 Nov 2021 14:26:46 GMT
   This shows that there are three messages at the end of
   the Sent file from 16 Nov.  These contain the corrupted
   mail message.

21) Where are the "^From " lines at the end of the Sent file?

        tail +16627300 sent > \temp\sent-16627300
        grep -n "^From " sent-16627300
        1:From - Fri, 03 Dec 2021 20:39:56 GMT
        25:From - Tue, 16 Nov 2021 14:26:46 GMT
        1915:From - Tue, 16 Nov 2021 14:26:46 GMT
        3634:From - Tue, 16 Nov 2021 14:26:46 GMT
        5353:From - Sun, 05 Dec 2021 15:08:46 GMT
        5400:From - Sun, 05 Dec 2021 15:23:16 GMT
        5434:From - Sun, 05 Dec 2021 16:01:40 GMT
        5488:From - Sun, 05 Dec 2021 20:42:06 GMT
   So, at the end of the 03 Dec message, Thunderbird
   is not recognizing the three "16 Nov" "From" lines
   and treating them as the starts of three mail messages.

22) Where are the null lines in this "end-of-Sent" file?

        c:\temp>grep -n  "^$" sent-16627300
        16:
        20:
        22:
        24:
        45:
        50:
        53:
        55:
        61:
        386:
        387:
        393:
        409:
        424:
        426:
        428:
        435:
        450:
        452:
        454:
        464:
        485:
        491:
        495:
        501:
        516:
        518:
        520:
        523:
        538:
        544:
        546:
        547:
        564:
        582:
        598:
        608:
        613:
        630:
        635:
        640:
        645:
        652:
        1743:
        1749:
        1765:
        1778:
        1784:
        1800:
        1803:
        1809:
        1825:
        1840:
        1842:
        1844:
        1851:
        1864:
        1870:
        1886:
        1901:
        1903:
        1905:
        1912:
        1914:
        1935:
        1940:
        1943:
        1945:
        1951:
        2276:
        2277:
        2283:
        2299:
        2314:
        2316:
        2318:
        2325:
        2340:
        2342:
        2344:
        2354:
        2375:
        2381:
        2385:
        2391:
        2406:
        2408:
        2410:
        2413:
        2428:
        2434:
        2436:
        2437:
        2454:
        2472:
        2488:
        2498:
        2503:
        2520:
        2525:
        2530:
        2535:
        2542:
        3633:
        3654:
        3659:
        3662:
        3664:
        3670:
        3995:
        3996:
        4002:
        4018:
        4033:
        4035:
        4037:
        4044:
        4059:
        4061:
        4063:
        4073:
        4094:
        4100:
        4104:
        4110:
        4125:
        4127:
        4129:
        4132:
        4147:
        4153:
        4155:
        4156:
        4173:
        4191:
        4207:
        4217:
        4222:
        4239:
        4244:
        4249:
        4254:
        4261:
        5352:
        5368:
        5370:
        5381:
        5393:
        5397:
        5399:
        5418:
        5420:
        5431:
        5433:
        5451:
        5460:
        5467:
        5471:
        5479:
        5482:
        5485:
        5487:
        5505:
        5507:
        5509:
        5514:
        5516:
        5517:
        5518:
        5559:
        5560:
  There are many null lines in this file, and there is a null
  line preceding each of the "^From" lines.  So, I contend
  that the end of this mbox file is in proper mbox format.
  There is a null line before each "^From " line.

23) The only thing I see that could be causing the problem is

   that the three 16 Nov messages are not in date sequence.
   I just looked at RFC 4155 (mbox format), and I do not see
   anything that says that the mail messages have to be in date
   order.  Is Thunderbird getting confused?  It does not show
   these messages.  But I do see four messages from 16 Nov
   in the Sent folder as displayed by Thunderbird.
   The Subjects displayed are:
        Where our Members Live
        Where our Members Live
        Re: Review of Recent Activity
        Re: Review of Recent Activity
   There are different "^Subject" linies in this file, as
   there are multiple messages in the last three 16 Nov
   mail messages.  If there are multiple "Subject:" lines
   in a mail message, which one does Thunderbird use?
   I asume the first one.   But I see in the file only
   one "Where Our Members Live" subject line and no
   "Review" subject lines.  Thus, I conclude that maybe
   the Sent.msf file is corrupt.

25) Shut down Thunderbird, rename the Sent.msf file, and

   restart Thunderbird.  The Sent folder now displays
   messages from 16 Nov properly.

26) Copy two 15 Nov messages from Sent-211118 to Sent.

   Both messages appear fine in the Sent-211118 Folder
   when I view them in Thunderbird.
   The first copied properly, but the second has the
   same corruption as before.  Now I have no idea what
   may be causing the corruption.

Helpful?

more options

Helpful?

more options

It seems likely. Is there anything I can do to give more info for the developers? My case involves copying/moving messages from one folder to another; the cases in the bug report seem to be different. For now, if I have to copy a few messages, i will copy one-at-a-time. For larger blocks of messages I will wait for more info on the bug. Thanks.

Helpful?

  1. 1
  2. 2
Ask a question

You must log in to your account to reply to posts. Please start a new question, if you do not have an account yet.