The download tool fails to download the data from vresion 630

[HomingHamster]

Readme or version number outdated. Files are renamed/reorganised in a way that is not obviously renamed in the current public dataset.

(.venv) ubuntu@lightbulb:~/PycharmProjects/FlyBrainV2/codex$ ./scripts/make_data.sh
Loading data for version 630..
 loading data from static/data/630...
 downloading raw data file neurons.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/neurons.csv.gz</Details></Error>
WARNING: Raw data file neurons.csv.gz for version 630 could not be downloaded
 downloading raw data file classification.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/classification.csv.gz</Details></Error>
WARNING: Raw data file classification.csv.gz for version 630 could not be downloaded
 downloading raw data file cell_stats.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/cell_stats.csv.gz</Details></Error>
WARNING: Raw data file cell_stats.csv.gz for version 630 could not be downloaded
 downloading raw data file connections.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/connections.csv.gz</Details></Error>
WARNING: Raw data file connections.csv.gz for version 630 could not be downloaded
 downloading raw data file labels.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/labels.csv.gz</Details></Error>
WARNING: Raw data file labels.csv.gz for version 630 could not be downloaded
 downloading raw data file coordinates.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/coordinates.csv.gz</Details></Error>
WARNING: Raw data file coordinates.csv.gz for version 630 could not be downloaded
 downloading raw data file nblast.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/nblast.csv.gz</Details></Error>
WARNING: Raw data file nblast.csv.gz for version 630 could not be downloaded
 downloading raw data file connectivity_tags.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/connectivity_tags.csv.gz</Details></Error>
WARNING: Raw data file connectivity_tags.csv.gz for version 630 could not be downloaded
 downloading raw data file morphology_clusters.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/morphology_clusters.csv.gz</Details></Error>
WARNING: Raw data file morphology_clusters.csv.gz for version 630 could not be downloaded
 downloading raw data file connectivity_clusters.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/connectivity_clusters.csv.gz</Details></Error>
WARNING: Raw data file connectivity_clusters.csv.gz for version 630 could not be downloaded
 downloading raw data file svd.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/svd.csv.gz</Details></Error>
WARNING: Raw data file svd.csv.gz for version 630 could not be downloaded
 downloading raw data file lr_matching.csv.gz for version 630..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/630/lr_matching.csv.gz</Details></Error>
WARNING: Raw data file lr_matching.csv.gz for version 630 could not be downloaded
 loading data from static/data/630:
   0 neuron rows
   0 connection rows
   0 label rows (?)
   0 coordinate rows
   0 nblast rows

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py", line 229, in <module>
    load_and_pickle_neuron_db_versions(versions=[DEFAULT_DATA_SNAPSHOT_VERSION])
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py", line 177, in load_and_pickle_neuron_db_versions
    db = load_neuron_db(version=v, data_root_path=data_root_path)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py", line 104, in load_neuron_db
    neuron_db = initialize_neuron_data(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/neuron_data_initializer.py", line 135, in initialize_neuron_data
    assert neuron_file_rows[0] == get_neurons_file_columns()
           ~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

[HomingHamster]

in codex/data/local_data_loader.py:

GCS_PICKLE_URL_TEMPLATE = "https://storage.googleapis.com/flywire-data/codex/data/fafb/{version}/neuron_db.pickle.gz"
GCS_RAW_DATA_URL_TEMPLATE = (
    "https://storage.googleapis.com/flywire-data/codex/data/fafb/{version}/{filename}"
)

adding /fafb seems to solve this for version 630

still no access to version 783:

(.venv) ubuntu@lightbulb:~/PycharmProjects/FlyBrainV2/codex$ ./scripts/make_data.sh
Loading data for version 783..
 loading data from static/data/783...
 downloading raw data file neurons.csv.gz for version 783..
 downloading raw data file classification.csv.gz for version 783..
 downloading raw data file cell_stats.csv.gz for version 783..
 downloading raw data file connections.csv.gz for version 783..
 downloading raw data file labels.csv.gz for version 783..
/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py:70: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
  return rows, datetime.utcfromtimestamp(
 downloading raw data file coordinates.csv.gz for version 783..
 downloading raw data file nblast.csv.gz for version 783..
 downloading raw data file connectivity_tags.csv.gz for version 783..
 downloading raw data file morphology_clusters.csv.gz for version 783..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/fafb/783/morphology_clusters.csv.gz</Details></Error>
WARNING: Raw data file morphology_clusters.csv.gz for version 783 could not be downloaded
 downloading raw data file connectivity_clusters.csv.gz for version 783..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/fafb/783/connectivity_clusters.csv.gz</Details></Error>
WARNING: Raw data file connectivity_clusters.csv.gz for version 783 could not be downloaded
 downloading raw data file svd.csv.gz for version 783..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/fafb/783/svd.csv.gz</Details></Error>
WARNING: Raw data file svd.csv.gz for version 783 could not be downloaded
 downloading raw data file lr_matching.csv.gz for version 783..
Download failed: status code 404
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: flywire-data/codex/data/fafb/783/lr_matching.csv.gz</Details></Error>
WARNING: Raw data file lr_matching.csv.gz for version 783 could not be downloaded
 loading data from static/data/783:
   139256 neuron rows
   3869879 connection rows
   160045 label rows (2025-05-31)
   238910 coordinate rows
   133683 nblast rows

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py", line 229, in <module>
    load_and_pickle_neuron_db_versions(versions=[DEFAULT_DATA_SNAPSHOT_VERSION])
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py", line 177, in load_and_pickle_neuron_db_versions
    db = load_neuron_db(version=v, data_root_path=data_root_path)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/local_data_loader.py", line 104, in load_neuron_db
    neuron_db = initialize_neuron_data(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/PycharmProjects/FlyBrainV2/codex/codex/data/neuron_data_initializer.py", line 165, in initialize_neuron_data
    assert classification_rows[0] == get_classification_file_columns()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

[arie-matsliah]

hi @HomingHamster , thanks for pointing this out. I have updated the data version to latest release and seems like links to data files are fixed. Please give it a try and let me know if you spot any issues.

[HomingHamster]

can confirm, it does work now, thanks!

P.S. Is there any way to get the Neuropils visualization tile running offline also? I don't mind compiling my own visualizations if its possible with a home GPU setup. Can I create a feature request for this?

[arie-matsliah]

Great, thanks for reporting. Re offline viz, I believe this is doable but non-trivial amount of work. You would need to download the meshes and then hack neuroglancer (the 3D vis tool) to fetch them from your local storage. Not sure if this has been done before.