Merge pull request #1208 from resource-watch/cit_002

update outdated dataset

Author: weiqi-tori

check_if_nrt_datasets_current/contents/src/__init__.py

@@ -132,9 +132,9 @@ def main():
         # Tropical Cyclones
         elif 'dis.015' in r['data']['attributes']['name']:
             allowed_time = datetime.timedelta(days=50)
-        # oil spills data set doesn't always have events that occur every 30 days
+        # oil spills data set doesn't always have events that occur every 50 days
         elif 'ene.008' in r['data']['attributes']['name']:
-            allowed_time = datetime.timedelta(days=30)
+            allowed_time = datetime.timedelta(days=50)
         # Vegetation Health Index
         elif 'foo.024' in r['data']['attributes']['name']:
             allowed_time = datetime.timedelta(days=12)

check_if_nrt_datasets_current/outdated_nrt_scripts.csv

@@ -1,18 +1,18 @@
 WRI ID,Public Title,Date Added to Sheet,Last Check,Known Reason?,Script in pause
-cit.003a.rw0.nrt,Air Quality: Fine Particulate Matter (PM2.5) Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cit.003b.rw0.nrt,Air Quality: Particulate Matter (PM10) Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cit.003c.rw0.nrt,Air Quality: Sulfur Dioxide (SO₂) Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cit.003d.rw0.nrt,Air Quality: Nitrogen Dioxide (NO₂) Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cit.003e.rw0.nrt,Air Quality: Ozone (O₃) Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cit.003f.rw0.nrt,Air Quality: Carbon Monoxide (CO) Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cit.003g.rw0.nrt,Air Quality: Black Carbon Station Measurements,11/9/2023,07/10/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
-cli.039.rw1.nrt,Standardised Precipitation-Evapotranspiration Index,11/21/2023,07/10/2024,Source upgraded and it needs an account for the latest data,TRUE
-cli.040.rw0.nrt.dash,Sea Level,5/25/2023,07/10/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
-cli.041.rw0.nrt.dash,Land Ice: Antarctica,5/19/2023,07/10/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
-cli.042.rw0.nrt.dash,Land Ice: Greenland,5/19/2023,07/10/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
-cli.043.rw0.nrt.dash,Minimum Annual Arctic Sea Ice Extent,7/4/2023,07/10/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
-com.024.rw0.nrt,Oil Palm Concessions,1/12/2021,07/10/2024,the dataset is now only available as vector layer tiles,
-com.025.rw0.nrt,Wood Fiber Concessions,1/12/2021,07/10/2024,the dataset is now only available as vector layer tiles,
-foo.003.rw1.nrt,Current and Projected Food Insecurity,7/4/2023,07/10/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
-ocn.002.rw0.nrt,Wave Height,8/19/2021,07/10/2024,the version we use is no longer updated by the data source,TRUE
-soc.062c.rw1.nrt,"Recent Internal Displacement from Conflict, Natural Disasters, and Development",8/10/2022,07/10/2024,the data source has not been updated since 6/30/2022,TRUE
+cit.003a.rw0.nrt,Air Quality: Fine Particulate Matter (PM2.5) Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cit.003b.rw0.nrt,Air Quality: Particulate Matter (PM10) Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cit.003c.rw0.nrt,Air Quality: Sulfur Dioxide (SO₂) Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cit.003d.rw0.nrt,Air Quality: Nitrogen Dioxide (NO₂) Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cit.003e.rw0.nrt,Air Quality: Ozone (O₃) Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cit.003f.rw0.nrt,Air Quality: Carbon Monoxide (CO) Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cit.003g.rw0.nrt,Air Quality: Black Carbon Station Measurements,11/9/2023,08/02/2024,OpenAQ updated its API and data fetching options. The script cannot directly get data from their S3 bucket anymore. Need to register an account and use their API instead.,TRUE
+cli.039.rw1.nrt,Standardised Precipitation-Evapotranspiration Index,11/21/2023,08/02/2024,Source upgraded and it needs an account for the latest data,TRUE
+cli.040.rw0.nrt.dash,Sea Level,5/25/2023,08/02/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
+cli.041.rw0.nrt.dash,Land Ice: Antarctica,5/19/2023,08/02/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
+cli.042.rw0.nrt.dash,Land Ice: Greenland,5/19/2023,08/02/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
+cli.043.rw0.nrt.dash,Minimum Annual Arctic Sea Ice Extent,7/4/2023,08/02/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
+com.024.rw0.nrt,Oil Palm Concessions,1/12/2021,08/02/2024,the dataset is now only available as vector layer tiles,
+com.025.rw0.nrt,Wood Fiber Concessions,1/12/2021,08/02/2024,the dataset is now only available as vector layer tiles,
+foo.003.rw1.nrt,Current and Projected Food Insecurity,7/4/2023,08/02/2024,"source site has major update on their data storage, we paused on updating it since this dataset may not be kept in the data catelog ",TRUE
+ocn.002.rw0.nrt,Wave Height,8/19/2021,08/02/2024,the version we use is no longer updated by the data source,TRUE
+soc.062c.rw1.nrt,"Recent Internal Displacement from Conflict, Natural Disasters, and Development",8/10/2022,08/02/2024,the data source has not been updated since 6/30/2022,TRUE

dis_015a_tropical_cyclones/contents/src/__init__.py

@@ -45,7 +45,7 @@
 
 # url for cyclone track data for latest 3 years
 url_a = 'https://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/'
-url_b = 'v04r00/access/shapefile/IBTrACS.last3years.list.v04r00.lines.zip'
+url_b = 'v04r01/access/shapefile/IBTrACS.last3years.list.v04r01.lines.zip'
 SOURCE_URL = url_a + url_b
 
 # maximum number of attempts that will be made to download the data