Question
Question Posted 05/21/13:
I am attempting to download all Study Data from ADNI@LONI.
In the Download Study Data page, I have clicked on ALL and checked both "Select ALL tabular data (csv format)" and "Select ALL documents and zip files [30.0 GB]". Downloading starts after clicking on "Donwload>>".
After more than a day of downloading, I get a "download.zip" file that wheighs 30 GB. I am unable to unzip this file without errors on a Linux (Ubuntu 12.04) machine. The data unzipped from the 30 GB "download.zip" wheighs only 7GB. It looks like the genotype results are lost in the process.
Do you have any clue that could help us? Has this issue ever been reported?
$ ls -lh download.zip
-rw-rw-r-- 1 dp165978 dp165978 31G mai 18 14:51 download.zip
$
$ unzip download.zip
Archive: download.zip
warning [download.zip]: 30064771072 extra bytes at beginning or within zipfile
(attempting to process anyway)
file #1: bad zipfile offset (local header sig): 30064771072
(attempting to re-compensate)
inflating: CONVERSIONS_DICT.csv
[...]
inflating: adni_gwas_v2_set3.zip
file #84: bad zipfile offset (local header sig): 1624369043
(attempting to re-compensate)
file #84: bad zipfile offset (local header sig): 1624369043
[...]
file #99: bad zipfile offset (lseek): 33929445376
inflating: ADNIGO2_GWAS_Set_9.zip
[...]
inflating: PTDEMOG.csv
$
$ rm -f download.zip
$
$ du -sh .
7,1G .
$
I am attempting to download all Study Data from ADNI@LONI.
In the Download Study Data page, I have clicked on ALL and checked both "Select ALL tabular data (csv format)" and "Select ALL documents and zip files [30.0 GB]". Downloading starts after clicking on "Donwload>>".
After more than a day of downloading, I get a "download.zip" file that wheighs 30 GB. I am unable to unzip this file without errors on a Linux (Ubuntu 12.04) machine. The data unzipped from the 30 GB "download.zip" wheighs only 7GB. It looks like the genotype results are lost in the process.
Do you have any clue that could help us? Has this issue ever been reported?
$ ls -lh download.zip
-rw-rw-r-- 1 dp165978 dp165978 31G mai 18 14:51 download.zip
$
$ unzip download.zip
Archive: download.zip
warning [download.zip]: 30064771072 extra bytes at beginning or within zipfile
(attempting to process anyway)
file #1: bad zipfile offset (local header sig): 30064771072
(attempting to re-compensate)
inflating: CONVERSIONS_DICT.csv
[...]
inflating: adni_gwas_v2_set3.zip
file #84: bad zipfile offset (local header sig): 1624369043
(attempting to re-compensate)
file #84: bad zipfile offset (local header sig): 1624369043
[...]
file #99: bad zipfile offset (lseek): 33929445376
inflating: ADNIGO2_GWAS_Set_9.zip
[...]
inflating: PTDEMOG.csv
$
$ rm -f download.zip
$
$ du -sh .
7,1G .
$
Response posted 05/21/13 by michael weiner:
I cant answer this by Diana Truran "Truran, Diana Truran" or karen crawford can help you and tell them i suggested them
mike
mike
Response posted 05/21/13 by Dimitri Papadopoulos:
I'm not certain how to contact Diana Truran or Karen Crawford, I don't work at LONI.
Anyway, I've isolated the bug and found a workaround:
The ZIP file has probably been generated by Java software since most LONI software is written in Java. It looks like ZIP files created by Java programs may be uncompatible with unzip.
I was able to unzip the file using Java's "jar" command:
jar xf download.zip
instead of using unzip:
unzip download.zip
After reading a few posts here and there on the Web, it looks like the bug is on the Java side and not the unzip side. You might want to fix the code that creates the ZIP files o the fly on the LONI Web server.
http://www.info-zip.org/phpBB3/viewtopic.php?t=367
http://www.info-zip.org/phpBB3/viewtopic.php?t=385
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=661956
Anyway, I've isolated the bug and found a workaround:
The ZIP file has probably been generated by Java software since most LONI software is written in Java. It looks like ZIP files created by Java programs may be uncompatible with unzip.
I was able to unzip the file using Java's "jar" command:
jar xf download.zip
instead of using unzip:
unzip download.zip
After reading a few posts here and there on the Web, it looks like the bug is on the Java side and not the unzip side. You might want to fix the code that creates the ZIP files o the fly on the LONI Web server.
http://www.info-zip.org/phpBB3/viewtopic.php?t=367
http://www.info-zip.org/phpBB3/viewtopic.php?t=385
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=661956
