Tuxera Coupon For Ntfs- 3g

Joined: Thu Jan 11, 2007 21:34
Posts: 77

Few problems after heavily using ntfs-3g

I used NTFS-3g for e.g. filling an 800 GB partition with data for a few months now and ancountered following problems:
1. Sometimes, NTFS-3g seems to allocate extern space for the security descriptors instead of saving them in the MFT (a changing of the security setting for all files in Vista solved that)
2. Even then, it seems that the place in which the SDs was saved belongs to 'System' according both Diskkeeper and O&O defrag, which makes defragment nearly impossible.
3. The 'Analysing' both with chkdsk, Diskeeper and O&O is much slower if the files are saved with NTFS-3g than with Vista, I hear much more seeks on the HDD, too. That has nothing to do with MFT and/or data fragmentation, MFT is in 2-3 pieces, only 500 files from 400k are fragmented, but analysing is only at a speed about 200 files/second.
4. It can depend on the other problems, but while the allocator of NTFS-3g seems to be a lot better than Vistas in view of fragmentation, it seems that it likes to spreed the files over the whole HDD, which is very bad if I linearly copy 700 GB fairly big files (mostly 5 MB-20 GB) to a 800 GB partition. It was extreme on a 50 GB partition filled with 20 GB little files (average ca. 20-50 kb) - it was like a chessboard on the blockview on O&O^^

Btw, I'm using Ubuntu 8.10 and 9.04, so I should have had always fairly current versions (e.g. now I've 2009.2.1).

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

Hi,
1. What do you mean by 'extern space'? What security setting did you change in Vista?
At the moment Security Descriptors always need extra space in the MFT record. The Advanced NTFS-3G driver solves this because it uses the Security Indexes. I think it would be very nice if the Stable NTFS-3G driver would do the same as the next step for merging.
2. Why do you think they are Security Descriptors and not something else? Defrag is certainly not impossible, only Windows doesn't support it.
3. How much is the 'much more'? All of them use the Microsoft NTFS driver and if it's indeed very inefficient then we could release a fast and efficient NTFS defragmenter.
4. The strategy is very good for copying large files. But not so much for many small files. At the moment this how we minimize file fragmentation. Extreme file fragmentation would significantly slow down things.
Regards, Szaka

1.1 External space - 'normal' space, not in the MFT - that can be prooved because when I changed the SDs I got a few MB freed.
1.2 security settings - I meant the permission rights
2. Because on these areas there where the SDs (seen that wifh O&O defrag) before I changed all security rights which I saw at locked files on O&O - I know that there can be mft-records outside the 'main' $MFT-file, right? Perhaps Vista deleted only the SDs in this 'extra mft spaces' (don't know how there are called officially), but not the 'extra mft spaces' itself?
3. much more - Analysing files wich chkdsk/O&O created with Vista are at speed at ca. 5000 files/sec, created with NTFS-3f at only 100..200 files/sec
Btw, I'm on freenode, my name there is mifritscher, too, if you have further questions.

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

1.1 The NTFS-3G created SD is 80 bytes. This easily fits into the 1024 bytes large MFT record. There is a very-very minimal chance that NTFS-3G ever pushes out an SD from the MFT record.
What I think is that Vista does this. It can be confirmed if you send the ntfsclone metadata image to szaka@ntfs-3g.org following the instructions at http://www.linux-ntfs.org/doku.php?id=n .. s_metadata and tell me which files are the 'problematic'.
2. There are no MFT records outside the $MFT file.
3. How do you run chkdsk so you can see such stats as 5000 files/s, 200 files/s?

chkdsk /f x:
under Vista, it shows e.g. x/x files are worked or so, I calced this speeds by viewing them, so there aren't that accourate^^
Ok, I'll send you the metadata soon - but I already changed all permissions :-/

hmm:
michi@michis-ibm:/tmp$ sudo ntfsclone -f --ignore-fs-check --metadata --output metadata.img /dev/mapper/truecrypt2
ntfsclone v2.0.0 (libntfs 10:0:0)
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 53686956032 bytes (53687 MB)
Current device size: 53686960128 bytes (53687 MB)
Scanning volume ..
ERROR: Cluster 1638402 referenced twice!
You didn't shutdown your Windowsproperly?
Strange, because I actually let shutdown Vista probably and run a chkdsk this day..
Also it is strage that the option ignore-fs-check is ignored..
Trying another volume (this time the 850 GB one)

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

The 'Cluster 1638402 referenced twice!' is a fairly bad error message. It means two files use the same data block which must never happen.
I really don't know how much ntfsprogs 2.0.0 can be trusted. It definitely has many problems though I'm not aware ntfsclone would have this kind of bug.
You may try ntfsprogs 1.13.1 and if the error is the same then chkdsk should fix the problem.
But first please run 'ntfscluster -fc 1638402 /dev/mapper/truecrypt2 2> /dev/null'. It should tell which files overlap.

ok, ntfscluster does only find one file..
I'm looking for a 1.13.1 version of ntfsprogs now.

Interesting: oth 1.13 and 2.0 ntfscluster show only one file, but both ntfsclone have the same behavior..

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

Please try this one http://ntfs-3g.org/download/ntfs-3g.clone-1.5388.tgz
then you can do
ntfs-3g.clone -mso - DEVICE | bzip2 > DEVICE-meta.img.bz2
This is much faster than ntfsclone.

ok, this version worked for the 50 GB volume I've sent you the link.
the 'big' volume is still beeing worked on ;)

now, also the big volume is sent.

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

Two files (017.part.met and 027.part) indeed overlap:

Both are related to eMule and one of them was updated by Windows the other by Linux. It's impossible to tell which one made the error but we do know that Vista has reliability problems with sparse files. Afair, eMule (or some torrent client?) recommends using preallocated files because of this. I don't know how this can be configured.
I'm also sorry to say that apparently you have 12 corrupt file/directory. Hopefully this is only an nfts-3g.clone bug which is still in development. Please do a 'find >/dev/null' at the root directory then check the log files for error. I can not access four directories at the top directory.

running :-)
Ah - I remember now - I got 2 BSODs shorttime after starting amule (page fault in nonpages area)
I runned emule first under Vista, then under Linux. Hmpf, I didn't linked the crashs with NTFS-problems :-/
Additionally, a find over the truecypt2-volume showed no error..

Hmm, and the truecrypt4 volume was defragmented by PerfectDisk - but was remounted after the first crash, but not used, so I think this volume should be ok beause of the journaling.
Perhaps ntfs-3g shouldn't simply omit the journaling-data?
The next thing I do is going to Vista again and do a full chkdsk.

following errors were reported on truecrypt2:
CHKDSK überprüft Dateien (Phase 1 von 3)..
Beschädigter Attributeintrag (128, ') wird Datensätzen verarbeitet)
vom Datensatzsegment 67050 gelöscht.
75 Prozent abgeschlossen. (252192 von 259472 Indexeinträgen verarbeitet)
Indexeintrag 017.part.met in Index $I30 der Datei 66979 wird gelöscht.
75 Prozent abgeschlossen. (252245 von 259472 Indexeinträgen verarbeitet)
259472 Indexeinträge verarbeitet.
Indexüberprüfung beendet.
CHKDSK stellt verlorene Dateien wieder her.
1 nicht indizierte Dateien verarbeitet.
Verwaiste Datei 017.part.met (7845) wird in Verzeichnisdatei 66979 wiederhergest
ellt.
Verwaiste Datei 017PAR~1.MET (7845) wird in Verzeichnisdatei 66979 wiederhergest
ellt.
99 Prozent abgeschlossen. (250429 von 251456 Beschreibungen verarbeitet)
99 Prozent abgeschlossen. (251099 von 251456 Beschreibungen verarbeitet)
251456 Sicherheitsbeschreibungen verarbeitet.
2 unbenutzte Indexeinträge werden aus dem Index $SII der Datei 9 gelöscht.
2 unbenutzte Indexeinträge werden aus dem Index $SDH der Datei 9 gelöscht.
2 nicht verwendete Sicherheitsbeschreibungen werden aufgeräumt.
Spiegelkopie des Datenstroms der Sicherheitsbeschreibungen wird berichtigt.
Überprüfung der Sicherheitsbeschreibungen beendet.
Attribut DATA wird in Datei 67050 eingefügt.
4009 Datendateien verarbeitet.
Fehler im Attribut BITMAP der Masterdateitabelle (MFT) werden berichtigt.
CHKDSK hat freien Speicher gefunden, der in der Volumebitmap als
zugeordnet gekennzeichnet ist.
Windows hat Probleme im Dateisystem behoben.
So only a few errors :-))

truecrypt4 was ok (despite of the extreme slowness of chkdsk)

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

> CHKDSK überprüft Dateien (Phase 1 von 3)..
> Beschädigter Attributeintrag (128, ') wird Datensätzen verarbeitet)
> vom Datensatzsegment 67050 gelöscht.
This is the 027.part file I noted above and which overlaps with the below
017.part.met file.
There is absoluely nothing wrong with the NTFS metadata of the file.
Chkdsk removed it because one of its clusters (blocks) was already
referenced by the 017.part.met file.
> Indexeintrag 017.part.met in Index $I30 der Datei 66979 wird gelöscht.
This is the other file I noted above which overlaps with 027.part. It has
another problem. The directory index was created by Linux and it was
correct but Windows (defrag?) moved the file into a different MFT record
but it didn't update the index. It still points into the old, valid MFT
record which got marked unused.
This seems to be either another Windows NTFS driver bug or you had a system
crash during defragmentation.
> CHKDSK stellt verlorene Dateien wieder her.
> 1 nicht indizierte Dateien verarbeitet.
Refers to 017.part.met.
> Verwaiste Datei 017.part.met (7845) wird in Verzeichnisdatei 66979
> wiederhergest ellt. Verwaiste Datei 017PAR~1.MET (7845) wird in
> Verzeichnisdatei 66979 wiederhergest ellt.
Windows finishes moving the file and updates the directory index which
didn't happen earlier.
> 2 unbenutzte Indexeinträge werden aus dem Index $SII der Datei 9
> gelöscht. 2 unbenutzte Indexeinträge werden aus dem Index $SDH der
> Datei 9 gelöscht. 2 nicht verwendete Sicherheitsbeschreibungen werden
> aufgeräumt.
http://ntfs-3g.org/support.html#cleaningup
> Attribut DATA wird in Datei 67050 eingefügt.
Here CHKDSK replaced a good file with an empty one.
> Fehler im Attribut BITMAP der Masterdateitabelle (MFT) werden berichtigt.
> CHKDSK hat freien Speicher gefunden, der in der Volumebitmap als
> zugeordnet gekennzeichnet ist.
http://ntfs-3g.org/support.html#usedfreespace
Overall the only problem was the one what ntfsclone also found.

yes, this problem wasn't so harmfull and almost expected after I realized that the Vista-crashs had something todo with NTFS.
The main questions for me are: Why were the SDs saved externally, and why are this places still shown as 'locked'?

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

I couldn't find yet any external SD. So far all of them were in the base MFT record.

yes, there all moved to the MFT by changing the permissions.
I'll try to create a image with external SDs.
Hmm, I experimented with the 'advanced' version of the ntfs-3g driver for a few weeks, could it be that this driver created them?

Joined: Tue Nov 21, 2006 23:15
Posts: 1648

Re: Few problems after heavily using ntfs-3g

SDs are always in the $MFT. Maybe you mean non-base MFT records, i.e. extended MFT record.
Truecrypt2 has 485 SDs, all in base MFT records. Truecrypt4 has 11 SDs, all in base MFT records.

It shouldn't do that but maybe that has something to do with the issue you experience.

Joined: Tue Sep 04, 2007 17:22
Posts: 1286

Re: Few problems after heavily using ntfs-3g

Hi,

The Advanced ntfs-3g store the SD (security descriptors) like Windows does : they are grouped in the special file $Secure and a reference is put in the base record of the file, while the standard ntfs-3g creates a special entry, generally in the base record of the file. Some defragmenters have already been mentioned as disturbed by these special entries (see viewtopic.php?f=2&t=864), but you should not experience that with Advanced ntfs-3g (unless some unknown problem occurred).
With Advanced ntfs-3g, like with Windows, a single entry is created in $Secure for all the files which have the same owner and protection, but when an entry is not needed any more (because the protections of all the files which were using it have been changed), the entry is not freed and can be used for a subsequent file using the said owner and protection. This is why you get the following warnings :

$Secure is file 9 and $SII and $SDH are two indexes for fast access to a SD, and you get two unused entries, which is quite normal. chkdsk has reclaimed their space, and they will be recreated if they are needed again.

Due to the grouping of the SD, such a big count is very unlikely created by Advanced ntfs-3g or Windows. For a single user, using all the values for chmod from 0000 to 07777, at most 4096 SD can be created in $Secure, and 251456 means at least 62 user+group pairs.
What exactly is the remaining problem with the SD ?
Regards
Jean-Pierre

Joined: Tue Sep 04, 2007 17:22
Posts: 1286

Re: Few problems after heavily using ntfs-3g

Hi,
It just occurred to me that currently the cache size for the SD is limited to 262144 entries in $Secure. This is not the limit on the number of SDs, if there are more SD, they just cannot be cached.
Is it possible you really have 251456 entries in $Secure, not counting the entries created by standard ntfs-3g (which cannot be cached). This would mean a lot of users.. or data corruption.
Regards
Jean-Pierre