-
Notifications
You must be signed in to change notification settings - Fork 62
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Update IDC dataset with new views and
v6 version (#266)
* feat: New views for v1-v6 * feat: bootstrap idc_v6 dataset * fix: Add back impersonating account * fix: Regenerate DAG with v6 dataset * fix: Trailing semi-colon and CURRENT_VERSION env var
- Loading branch information
1 parent
445577c
commit 02cae2b
Showing
35 changed files
with
1,093 additions
and
23 deletions.
There are no files selected for viewing
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/analysis_results_metadata.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.analysis_results_metadata` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.analysis_results_metadata` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/auxiliary_metadata.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.auxiliary_metadata` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.auxiliary_metadata` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/dicom_all.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.dicom_all` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.dicom_all` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/dicom_metadata.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.dicom_metadata` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.dicom_metadata` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/dicom_metadata_curated.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.dicom_metadata_curated` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.dicom_metadata_curated` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/measurement_groups.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.measurement_groups` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.measurement_groups` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/nlst_canc.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.nlst_canc` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.nlst_canc` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/nlst_ctab.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.nlst_ctab` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.nlst_ctab` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/nlst_ctabc.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.nlst_ctabc` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.nlst_ctabc` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/nlst_prsn.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.nlst_prsn` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.nlst_prsn` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/nlst_screen.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.nlst_screen` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.nlst_screen` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/original_collections_metadata.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.original_collections_metadata` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.original_collections_metadata` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/qualitative_measurements.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.qualitative_measurements` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.qualitative_measurements` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/quantitative_measurements.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.quantitative_measurements` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.quantitative_measurements` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/segmentations.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.segmentations` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.segmentations` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/tcga_biospecimen_rel9.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.tcga_biospecimen_rel9` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.tcga_biospecimen_rel9` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/tcga_clinical_rel9.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.tcga_clinical_rel9` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.tcga_clinical_rel9` |
2 changes: 1 addition & 1 deletion
2
datasets/idc/_images/generate_bq_views/queries/current/version_metadata.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| select * from `PROJECT.idc_v5.version_metadata` | ||
| select * from `PROJECT.idc_CURRENT_VERSION.version_metadata` |
54 changes: 54 additions & 0 deletions
54
datasets/idc/_images/generate_bq_views/queries/v1/dicom_pivot_v1.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| SELECT | ||
| pivot.PatientID, | ||
| pivot.BodyPartExamined, | ||
| pivot.SeriesInstanceUID, | ||
| pivot.SliceThickness, | ||
| pivot.SeriesNumber, | ||
| pivot.SeriesDescription, | ||
| pivot.StudyInstanceUID, | ||
| pivot.StudyDescription, | ||
| pivot.StudyDate, | ||
| pivot.SOPInstanceUID, | ||
| pivot.Modality, | ||
| pivot.SOPClassUID, | ||
| pivot.collection_id, | ||
| Internal_structure, | ||
| Sphericity, | ||
| Calcification, | ||
| Lobular_Pattern, | ||
| Spiculation, | ||
| Margin, | ||
| Texture, | ||
| Subtlety_score, | ||
| Malignancy, | ||
| SUVbw, | ||
| Volume, | ||
| Diameter, | ||
| Surface_area_of_mesh, Total_Lesion_Glycolysis, | ||
| Standardized_Added_Metabolic_Activity, | ||
| Percent_Within_First_Quarter_of_Intensity_Range, | ||
| Percent_Within_Third_Quarter_of_Intensity_Range, | ||
| Percent_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Percent_Within_Second_Quarter_of_Intensity_Range, | ||
| Standardized_Added_Metabolic_Activity_Background, | ||
| Glycolysis_Within_First_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Third_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Second_Quarter_of_Intensity_Range, | ||
| pivot.AnatomicRegionSequence, | ||
| SegmentedPropertyCategoryCodeSequence, | ||
| SegmentedPropertyTypeCodeSequence, | ||
| pivot.FrameOfReferenceUID, | ||
| SegmentNumber, | ||
| SegmentAlgorithmType, | ||
| pivot.crdc_study_uuid, | ||
| pivot.crdc_series_uuid, | ||
| pivot.crdc_instance_uuid, | ||
| Program, | ||
| pivot.tcia_tumorLocation, | ||
| pivot.source_DOI, | ||
| gcs_url, | ||
| pivot.tcia_species | ||
| FROM `PROJECT.DATASET.dicom_derived_all` pivot | ||
| JOIN `PROJECT.DATASET.dicom_all` dicom_all | ||
| ON pivot.SOPInstanceUID = dicom_all.SOPInstanceUID |
70 changes: 70 additions & 0 deletions
70
datasets/idc/_images/generate_bq_views/queries/v2/dicom_pivot_v2.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| SELECT | ||
| pivot.PatientID, | ||
| pivot.BodyPartExamined, | ||
| pivot.SeriesInstanceUID, | ||
| pivot.SliceThickness, | ||
| pivot.SeriesNumber, | ||
| pivot.SeriesDescription, | ||
| pivot.StudyInstanceUID, | ||
| pivot.StudyDescription, | ||
| pivot.StudyDate, | ||
| pivot.SOPInstanceUID, | ||
| pivot.Modality, | ||
| pivot.SOPClassUID, | ||
| pivot.collection_id, | ||
| Apparent_Diffusion_Coefficient, | ||
| Internal_structure, | ||
| Sphericity, | ||
| Calcification, | ||
| Lobular_Pattern, | ||
| Spiculation, | ||
| Margin, | ||
| Texture, | ||
| Subtlety_score, | ||
| Malignancy, | ||
| SUVbw, | ||
| Volume, | ||
| Diameter, | ||
| Surface_area_of_mesh, | ||
| Total_Lesion_Glycolysis, | ||
| Standardized_Added_Metabolic_Activity, | ||
| Percent_Within_First_Quarter_of_Intensity_Range, | ||
| Percent_Within_Third_Quarter_of_Intensity_Range, | ||
| Percent_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Percent_Within_Second_Quarter_of_Intensity_Range, | ||
| Standardized_Added_Metabolic_Activity_Background, | ||
| Glycolysis_Within_First_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Third_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Second_Quarter_of_Intensity_Range, | ||
| pivot.AnatomicRegionSequence, | ||
| SegmentedPropertyCategoryCodeSequence, | ||
| SegmentedPropertyTypeCodeSequence, | ||
| pivot.FrameOfReferenceUID, | ||
| SegmentNumber, | ||
| SegmentAlgorithmType, | ||
| pivot.crdc_study_uuid, | ||
| pivot.crdc_series_uuid, | ||
| pivot.crdc_instance_uuid, | ||
| Program, | ||
| pivot.tcia_tumorLocation, | ||
| pivot.source_DOI, | ||
| gcs_url, | ||
| AdditionalPatientHistory, | ||
| Allergies, ImageType, | ||
| LastMenstrualDate, | ||
| MedicalAlerts, | ||
| EthnicGroup, | ||
| Occupation, | ||
| PatientAge, | ||
| PatientComments, | ||
| PatientSize, | ||
| PatientWeight, | ||
| PregnancyStatus, | ||
| ReasonForStudy, | ||
| RequestedProcedureComments, | ||
| SmokingStatus, | ||
| pivot.tcia_species | ||
| FROM `PROJECT.DATASET.dicom_derived_all` pivot | ||
| JOIN `PROJECT.DATASET.dicom_all` dicom_all | ||
| ON pivot.SOPInstanceUID = dicom_all.SOPInstanceUID |
70 changes: 70 additions & 0 deletions
70
datasets/idc/_images/generate_bq_views/queries/v3/dicom_pivot_v3.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| SELECT | ||
| pivot.PatientID, | ||
| pivot.BodyPartExamined, | ||
| pivot.SeriesInstanceUID, | ||
| pivot.SliceThickness, | ||
| pivot.SeriesNumber, | ||
| pivot.SeriesDescription, | ||
| pivot.StudyInstanceUID, | ||
| pivot.StudyDescription, | ||
| pivot.StudyDate, | ||
| pivot.SOPInstanceUID, | ||
| pivot.Modality, | ||
| pivot.SOPClassUID, | ||
| pivot.collection_id, | ||
| Apparent_Diffusion_Coefficient, | ||
| Internal_structure, | ||
| Sphericity, | ||
| Calcification, | ||
| Lobular_Pattern, | ||
| Spiculation, | ||
| Margin, | ||
| Texture, | ||
| Subtlety_score, | ||
| Malignancy, | ||
| SUVbw, | ||
| Volume, | ||
| Diameter, | ||
| Surface_area_of_mesh, | ||
| Total_Lesion_Glycolysis, | ||
| Standardized_Added_Metabolic_Activity, | ||
| Percent_Within_First_Quarter_of_Intensity_Range, | ||
| Percent_Within_Third_Quarter_of_Intensity_Range, | ||
| Percent_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Percent_Within_Second_Quarter_of_Intensity_Range, | ||
| Standardized_Added_Metabolic_Activity_Background, | ||
| Glycolysis_Within_First_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Third_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Second_Quarter_of_Intensity_Range, | ||
| pivot.AnatomicRegionSequence, | ||
| SegmentedPropertyCategoryCodeSequence, | ||
| SegmentedPropertyTypeCodeSequence, | ||
| pivot.FrameOfReferenceUID, | ||
| SegmentNumber, | ||
| SegmentAlgorithmType, | ||
| pivot.crdc_study_uuid, | ||
| pivot.crdc_series_uuid, | ||
| pivot.crdc_instance_uuid, | ||
| Program, | ||
| pivot.tcia_tumorLocation, | ||
| pivot.source_DOI, | ||
| gcs_url, | ||
| AdditionalPatientHistory, | ||
| Allergies, ImageType, | ||
| LastMenstrualDate, | ||
| MedicalAlerts, | ||
| EthnicGroup, | ||
| Occupation, | ||
| PatientAge, | ||
| PatientComments, | ||
| PatientSize, | ||
| PatientWeight, | ||
| PregnancyStatus, | ||
| ReasonForStudy, | ||
| RequestedProcedureComments, | ||
| SmokingStatus, | ||
| pivot.tcia_species | ||
| FROM `PROJECT.DATASET.dicom_derived_all` pivot | ||
| JOIN `PROJECT.DATASET.dicom_all` dicom_all | ||
| ON pivot.SOPInstanceUID = dicom_all.SOPInstanceUID |
74 changes: 74 additions & 0 deletions
74
datasets/idc/_images/generate_bq_views/queries/v4/dicom_pivot_v4.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| SELECT | ||
| pivot.PatientID, | ||
| pivot.BodyPartExamined, | ||
| pivot.SeriesInstanceUID, | ||
| pivot.SliceThickness, | ||
| pivot.SeriesNumber, | ||
| pivot.SeriesDescription, | ||
| pivot.StudyInstanceUID, | ||
| pivot.StudyDescription, | ||
| pivot.StudyDate, | ||
| pivot.SOPInstanceUID, | ||
| pivot.Modality, | ||
| pivot.SOPClassUID, | ||
| pivot.collection_id, | ||
| pivot.AnatomicRegionSequence, | ||
| pivot.FrameOfReferenceUID, | ||
| pivot.crdc_study_uuid, | ||
| pivot.crdc_series_uuid, | ||
| pivot.crdc_instance_uuid, | ||
| pivot.program, | ||
| pivot.tcia_tumorLocation, | ||
| pivot.source_DOI, | ||
| pivot.tcia_species, | ||
| pivot.license_short_name, | ||
| pivot.gcs_url, | ||
| pivot.Manufacturer, | ||
| pivot.ManufacturerModelName, | ||
| Apparent_Diffusion_Coefficient, | ||
| Internal_structure, | ||
| Sphericity, | ||
| Calcification, | ||
| Lobular_Pattern, | ||
| Spiculation, | ||
| Margin, | ||
| Texture, | ||
| Subtlety_score, | ||
| Malignancy, | ||
| SUVbw, | ||
| Volume, | ||
| Diameter, | ||
| Surface_area_of_mesh, | ||
| Total_Lesion_Glycolysis, | ||
| Standardized_Added_Metabolic_Activity, | ||
| Percent_Within_First_Quarter_of_Intensity_Range, | ||
| Percent_Within_Third_Quarter_of_Intensity_Range, | ||
| Percent_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Percent_Within_Second_Quarter_of_Intensity_Range, | ||
| Standardized_Added_Metabolic_Activity_Background, | ||
| Glycolysis_Within_First_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Third_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Fourth_Quarter_of_Intensity_Range, | ||
| Glycolysis_Within_Second_Quarter_of_Intensity_Range, | ||
| SegmentedPropertyCategoryCodeSequence, | ||
| SegmentedPropertyTypeCodeSequence, | ||
| SegmentNumber, | ||
| SegmentAlgorithmType, | ||
| AdditionalPatientHistory, | ||
| Allergies, | ||
| ImageType, | ||
| LastMenstrualDate, | ||
| MedicalAlerts, | ||
| EthnicGroup, | ||
| Occupation, | ||
| PatientAge, | ||
| PatientComments, | ||
| PatientSize, | ||
| PatientWeight, | ||
| PregnancyStatus, | ||
| ReasonForStudy, | ||
| RequestedProcedureComments, | ||
| SmokingStatus | ||
| FROM `PROJECT.DATASET.dicom_derived_all` pivot | ||
| JOIN `PROJECT.DATASET.dicom_all` dicom_all | ||
| ON pivot.SOPInstanceUID = dicom_all.SOPInstanceUID |
Oops, something went wrong.