Visualising data in NetCDF format

Daniel Lee, Software & Data Format Engineer at EUMETSAT guides you through a variety of free and open software, for you to visualize NetCDF format files. This video is divided in 4 different Chapters, feel free to skip to the following timestamps:
CHAPTER 1: Viewing Data with Panoply - 4:31
CHAPTER 2: Making Maps with QGIS - 9:40
CHAPTER 3: Troubleshooting coordinate systems - 12:45
CHAPTER 4: Workflow automation with Python - 23:15
You can find all the links mentioned by Daniel in this video below:
CMEMS: marine.copernic...
CAMS: atmosphere.cop...
Example notebook: gist.github.co...
GDAL: www.gdal.org/
Jupyter: jupyter.org/
Scipy stack, including matplotlib and numpy: www.scipy.org/
Info on netCDF: www.unidata.uc...
netCDF tools download: www.unidata.uc...
QGIS: www.qgis.org/e...
Xarray: xarray.pydata.o...
Scitools: scitools.org.uk/
CF Conventions: xarray.pydata.o...
ACDD: wiki.esipfed.or...
Natural Earth: www.naturaleart...

[@richasay9077]

This is probably the very best software tutorial video I have ever watched, regardless of topic, period. I hope you guys will keep bringing scientific content to the world! These packages are so broad and deep in their capabilities that mastering any one is a huge challenge, let alone having to use many... Thank you!

[@EUMETSAT1]

We're very happy to know you are finding them useful :)

[@mikecmw8492]

Kudos. Absolutely the best. Ucar should be embarrassed by this demo. Its is orders of magnitude better. On a geodesic scale of course

[@uelectron1513]

Lot of clarity and concise explanation , love to have a training series from you guys ... keep rocking

[@lonewolf_0948]

I'm so excited to such a tutorial videos on data visualisation and techniques.. Thanks for this initiative. We researchers expecting more. Love from India 🇮🇳

[@NamanRawal-l9j]

I have to echo what some others have said here. This really is a very well done demonstration and presentation. Regardless of the content of the video, the video itself is very well put together.

[@EUMETSAT1]

Thanks very much for the feedback, we're happy you like the video :)

[@felixjose2323]

Excellent presentation of various tools that can be used for data NetCDf data manipulation!

[@EUMETSAT1]

Thanks for the kind feedback!

[@miskaknapek]

Excellent - wonderfully explained. Many thanks!

[@edhartnett]

NetCDF is my favorite format! ;-) I'm so excited that Copernicus has chosen to use netCDF-4 for all this exciting and wonderful data. Please let me know if there is anything I can do to help with your use of netCDF.

[@suzilharold9334]

is it possible to convert NETCDF to CSV File and query using SQL ?

[@edhartnett]

@@suzilharold9334 see www.unidata.ucar.edu/software/rosetta/

[@mishkatulsaba2207]

Kindly let me the full code or thy way to merge trmm daily datasets having 4830 files to single file and then to extract at a paticular point

[@sakibshahriar3974]

This is great work. Thanks for sharing

[@RevyakinArtemy]

Thank you, Daniel.It's very interesting and useful video tutorial. It wil be interesting for me something about hdf4 and hdf5 manipulation and visualazing process.

[@임규호-k6p]

Really appreciate your explanation.

[@vijayvishwakarma6384]

Respected Daniel sir, thank you for producing this amazing video. It is very informative and useful. I am a PhD student and my research interest is related to atmospheric science. Atmospheric science is all about data and its meaningful analysis. I have recently started my research work and planned to use Python for my research purpose. I am working on observation, reanalysis and model data. Please make some video series on working with the mentioned data type and their meteorological analysis(e.g. plotting stream line on a level 850 hpa, etc.) using Python. If possible kindly share with me some useful documents/weblinks, it will be an immense help. Thank you.

[@EUMETSAT1]

Hello, thanks very much for your valuable feedback - much appreciated! We'll definitely take it on board - keep an eye on this channel for future videos and don't hesitate to get in touch if you have any other questions.

[@vijayvishwakarma6384]

Thank you for your positive response.

[@yahampathmarambe8879]

Very clear explanation, QGIS 3 (at least my installation) behave differently when netCDF file browsing

[@purityrima1366]

Thank you! You are a good teacher!

[@MrsMollyify]

Thank You so much!! This was amazing!

[@EUMETSAT1]

We're happy you liked it!

[@lachlanmcleod5294]

Thanks a lot Daniel, this is great!

[@ramshaharis2020]

nice video , plz do post video related to Radar Altimetry with details

[@angelosbirmpakos8488]

This is a great tutorial! Although I am facing a problem. My netcdf file has the extension .nc4 and cannot be opened by QGIS and generally this extension gets me in trouble very often when I'm trying to do something. All tutorials on youtube and documentations on google use .nc files. Any suggestions? Thanks in advance.

[@edhartnett]

The extension is not actually significant to the netcdf libraries - they don't care what the file is called. But nc4 would indicate a netCDF-4/HDF5 data format. (As opposed to netCDF classic format.) But netCDF-4/HDF5 has been common for the last decade, so it's not clear why your tools can't read the file.

[@ehtashamnaseer7380]

This is an excellent explanation. Can you please provide a tutorial on CAMS products retrieval and data assimilation in ECMWF.

[@gabriel_iglesias]

Awesome, thank you!

[@alistair2687]

Great tutorial, thank you so much

[@singhranjit5001]

Thanks, appreciate your explanation the Tutorial very easily

[@mrtnsnp]

Neat, thanks for this video. I've been looking for sample code to replace basemap, not realising that is seems to be part of matplotlib itself. Now on to figuring out why xarray doesn't fully understand Sentinel 5 P level 2 (swath) data. If seems the coordiates are found, but not recognised for what they are. Need to find time to look into the details.

[@hayleyevers-king7120]

I would highly recommend looking in to Cartopy :)

[@vuongpham88]

Great content. Thanks for your sharing.

[@willhadley9145]

This is a god sent. Thank you so much

[@ValleyOfTheKens]

At 19:18, if the SST file is projected correctly using a transverse mercator projection in panoply, why don't we add the data into QGIS with a local UTM projection? Seem easier than using GDAL warp. Is it because we want the SST to have consistent projection with the rest of the NetCDF?

[@EUMETSAT1]

Hi there, thanks for getting in touch. QGIS has on-the-fly reprojection but it makes things slow, and sometimes GDAL handles the metadata better than QGIS does so it makes things easier to reproject beforehand. Hope that helps answer your question.

[@jaderamos7061]

Wonderful tool, thank you! It helped a lot in my work.

[@EUMETSAT1]

You're very welcome! We're glad you found it useful.

[@hafezahmad3747]

Thank you!

[@parvanehhonarmand9238]

Thank you for the video. Any tutorial to extract local data from Netcdf global data?

[@EUMETSAT1]

Hello, thanks for getting in touch :) Please contact our user helpdesk who might be able to give you more info on that, here: [e-mail]

[@navsiv11]

Hi.. really informative video, I'm working in R&D project in bathymetry data, how to analyse the bathymetry data, please do one video on bathymetry and what are the tools we can use plz help

[@EUMETSAT1]

Hi Chinnu. EUMETSAT doesn't distribute bathymetry data, I'm afraid, but you could try taking a look at the following websites from UNESCO's GEBCO initiative (www.gebco.net/data_and_products/gridded_bathymetry_data/) or this site, provided by our partner CNES (French national space agency) www.altimetry.info/thematic-use-cases/geodesy-geophysics/bathymetry-estimate-from-altimetry/

[@EUMETSAT1]

In addition, you might find the following resources useful - ArcGIS has a Bathymetry module: www.esri.com/en-us/arcgis/products/arcgis-for-maritime-bathymetry/overview and Grass GIS might be useful too: grass.osgeo.org/grass76/manuals/raster3dintro.html

[@navsiv11]

thank you so much for the reply... here in my company i joined as a data scientist and i dont have any knowledge on bathymetry. my company project is on bathymetry data and its analysis... i got bathymetry data from gebco and bodc. those conventions and terminology are hard to grasp it will be very helpful if i have a video on this in general language in data scientist perspective lt will be very helpful... (familiar with jupyter notebook and netcdf python package and the code but terminology like 1d grid,2d grid, other features are hard to grasp). plz help .

[@EUMETSAT1]

@@navsiv11 This article explains how bathymetry can be derived from Altimetry: ftp.space.dtu.dk/pub/DTU15/DOCUMENTS/MSS/DTU15MSS+LAT.pdf

[@hikikomorihachiman7491]

Thank you

[@cottoncandy1886]

Great, thank you!

[@asifnaseer9356]

Thank you that was really helpful to understand

[@user-ky4yg8kf5p]

i downloaded era5 hourly data in 3 year range : (1) 1959-1979, (2) 1980-2000, and (3) 2001-2021 for different hPa levels. Theres this one file i didnt rename and i am not sure which year range it is in. How can i check the years?

[@EUMETSAT1]

Hello, apologies for the late reply. Please contact ECMWF for support regarding the era5 products. If your question is related to the use of a piece of code you've written, you could also share to StackOverflow. Hope this helps!

[@MandeepSingh-cu8xx]

Hi Daniel, to troubleshoot the coordinates issue for multiple multi-dimensional NetCDF files, how can we use gdalwarp (within QGIS) under the batch processing while identifying the layer we want in the .nc files?

[@EUMETSAT1]

Hello, thanks for getting in touch. Please contact our user helpdesk at: [e-mail] who might be able to offer you some assistance.

[@suthagargovindaraj5476]

Hai very nice presentation... I would like to know how to download ocean current data? If avail the data are real time ?

[@EUMETSAT1]

Hello, please get in touch with: [e-mail].

[@monocilindro3655]

Great tutorial. Just a question. How can one visualize multitemporal netdcf datasets? I can make it in Panoply but I cannot in Qgis. I have tried the Crayfish plugin but it doesn't work with CMES dataset (or maybe I am doing it wrong)

[@EUMETSAT1]

Visualising multitemporal datasets is no problem, but of course it depends on what you want to do with them. If you're wanting to make an animation using the tools shown in the video try matplotlib.animation (matplotlib.org/3.1.0/api/animation_api.html). Of course, you might want to tackle the problem using subplots - there's a cool demo for that here: matplotlib.org/3.1.0/gallery/subplots_axes_and_figures/subplot.htmlIn the end it depends on what you want to do specifically with the data. Due to the limitations on how QGIS interprets the CMEMS data you might need to look at alternatives, like those shown in chapter 4.We hope this helps.

[@ramshaharis2020]

Hi Daniel, Can you guide me how to calculate sea surface current while algorithms??

[@mohamedhatem5289]

I want to compare between two Datasets which have a large number of netcdf files by calculating standard deviation which software should I use Thanks in advance

[@EUMETSAT1]

Hello, thanks for your question. In general, if working with a large number of data files, we would recommend writing a short script in a programming language with suitable tools for opening the files and conducting the statistical analysis (Python, R, etc.). Hope that helps!

[@bakyayita]

Great work

[@alexyshr]

Hi Daniel. I want to see the same exercise you did in Python, but in R. Do you have something similar in R?. I need to extract tocal column over a point and do the time series plot. In additon I need to analize four netcdf files at the same time.. do you recommend me to join files befor to work (which tool?) or it can be join inside Python or R?. Thank you very much, Daniel

[@EUMETSAT1]

Thanks for the input! We’ve noted the interest in R. Concerning whether to join files beforehand, or to use Python or R, this depends on what works best for you. It sounds like you’re working with R already, so it’d probably make sense to load the netCDF files into R and merge them there, then extract the values at the positions you need. Wish you much success!

[@albertnkwasa2648]

Thanks for this. But if you wanted to save your point data from the dataarray into a csv file as timeseries. how would you go about it?

[@EUMETSAT1]

Hi, thanks for your question and apologies for the late response. Our suggestion would be to use Python to convert the relevant variable within the data array of your interest. There's a section of the video devoted to those tools here: kzbin.info/www/bejne/jqLSlqevobZ0f7s The corresponding example code is linked in the video description. It does not do precisely what you might want but it is a good starting point, you should be able to produce a CSV using that approach very quickly. Hope that helps and don't hesitate to get in touch again.

[@ronaldyndabu7265]

Thanks a lot this the best tutorial video!

[@EUMETSAT1]

You're very welcome, glad you found it useful.

[@diplo4235]

How can I pass multiple values from one sample into a nc file? Because scipy.netcdf is not letting me add date(yyyy-mm-dd) into the nc file

[@EUMETSAT1]

Hello, it would be a good idea to consult with either the SciPy community or the NetCDF documentation. Hope you find a solution.

[@BafanaGweba]

Thank you so much for the Video. Can you please make a video about downloading a Sea state parameters (Hs, Tp, Dir) satellite data, and how to process that data in python? I am struggling in downloading the Satellite data in your website (EUMESAT). I need satellite data so that I can compare it with model and observed data along the South African coast.

[@stephenyabo2115]

Excellent

[@jehanne5865]

How would you project the map in the WSG84 EPSG4326 CRS to center it in the 'usual' way?

[@EUMETSAT1]

Hi Jehanne, thanks for your query. So that we can assist you properly, please contact our user help desk directly at this address: [e-mail]

[@ziffification]

Tried really hard to register to get the data described in this video but I hit a dead end when trying to register.

[@kalthamismail695]

Is it possible to crop an image in panoply? could you please assist in this regard?

[@EUMETSAT1]

Hello, thanks for writing. You can crop an image in Panoply but the way you do it is by setting limits to the map extent. Panoply has a lot of options concerning how to map things so you can change what’s shown by changing the projection of the map, and the parameters of that projection. It should be noted that doing this doesn’t change the underlying data, it only changes the visualisation of that data. Here’s the NASA Panoply homepage: www.giss.nasa.gov/tools/panoply/ Hope that's useful - please feel free to write again if you have more questions.

[@kalthamismail695]

@@EUMETSAT1 Thank you so much for your reply and explanation. Truly appreciate it.

[@lekshmionair216]

Sir, while i do file, open a location where my. Nc file get saved,it doesn't get open why this sir kindly reply sir

[@EUMETSAT1]

Hello, thanks for getting in touch. Please contact the service helpdesk here: [e-mail]

[@rajeshkumarsahu2646]

How can I get lifted index spatial plot by using netcdf time series file Can I get the code for that

[@nothing21797]

Thank you for such an informative video session. May I know how can I download the data in detail?

[@EUMETSAT1]

What data would you be interested in downloading Rahul?

[@nothing21797]

@@EUMETSAT1 I want to download data which is related wind behavior ( velocity, direction, intensity) in the Indian ocean region. Can you please help me with it?

[@EUMETSAT1]

@@nothing21797 Hi Rahul - Sorry for the delay replying to you. Our Ocean and Sea Ice SAF has a number of wind products that might be of use to you: www.osi-saf.org/?q=content/wind-products

[@nsikan-georgeemana6524]

Hallo pls im completely new to Panolpy and data manipulation with data viewing and analysis software I have been trying to plot a netcdf dataset from CRU in Panolpy and displaying my AOI with the köppen climatic classification . I need the AOI to be displayed in such a way that its location is detectable by a Köppen classification Please i need help because im writing my research paper but now stuck Please could you do a video tutorial on that please Secondly i downloaded precipitation dataets for 20 years, unzipped and extracted the file and wanted Panoply to display the 20 year datsets as one single dtasets with the time window Rather it gave me an option of displaying the pre dataset year by year instead of bi- decadal

[@ivancabreraperez2550]

Thank you for such an informative video session. May I know how can I download the data in detail? I want to download data which is related SO2 and CO2 in the Atlantic ocean.

[@federicofierli6811]

Hi Ivan, which CO2 and SO2 data you would like to work with ? CO2 and SO2 analysis (not pure satellite data) are available with the CAMS service. My advice would be to go to atmosphere.ecmwf.int and browse there - Cheers - Federico Fierli - Amtospheric Science expert, EUMETSAT

[@EUMETSAT1]

Dear Ivan, Regarding your recent enquiry, if you are referring to IASI L2 data, then partially it is AC SAF. The AC SAF owns IASI L2 SO2, but not CO2. Nevertheless, both parameters are available for download through the EDC as a monoblock, as all IASI Sounding Products are. Try this link: eoportal.eumetsat.int/userMgmt/protected/dataCentre.faces?acronym=IASSND02&id=EO:EUM:DAT:METOP:MXI-N-SO2 We hope you find this information helpful - please let us know if you have any other questions. EUMETSAT User Services

[@datafakts]

Great Test data or where to get data that works with the Jupyter notebook would be nice. While I downloaded No2 data from CAMS, it refuses to index by lat lon. isel works sel not [7000000 values with dtype=float32] Coordinates: * longitude (longitude) float32 335.05 335.15 335.25 ... 44.849976 44.94998 * latitude (latitude) float32 69.95 69.85 69.75 ... 30.149998 30.049995 * level (level) float32 0.0 * time (time) timedelta64[ns] 00:00:00 01:00:00 ... 1 days 00:00:00 Attributes: species: Nitrogen Dioxide units: µg/m3 value: hourly values standard_name: mass_concentration_of_nitrogen_dioxide_in_air sel on anything in latitude or longitude fails with a key error. ie: 335.15 or 69.85 no2.sel(latitude=69.85,longitude=335.15) macc-raq-op.meteo.fr/index.php?calculation-model=ENSEMBLE&category=ensemble&date=LAST&level=SFC&offset=000&species=o3&subensemble=hourly_ensemble select species NO2 no2[0,0,1,1] etc gives me exactly the values above. This works funnily enough no2.sel(latitude=no2.latitude[0],longitude=no2.longitude[0]) type(no2.latitude[0]) => xarray.core.dataarray.DataArray type(69.85) => float even this works darmstadt = no2.sel(latitude=no2.latitude[0].latitude, longitude=no2.longitude[0].longitude) your data set has dim 3 the one from cams seems to have 4 but if I do no2.sel(latitude='69.85',longitude='335.15') it complains it wants a float. value0 = float(69.95) value1 = float(335.05) no2.sel(latitude=value0, longitude=value1) doesnt' work either Key Error Very confusing

[@EUMETSAT1]

Hi biofacts and sorry for the delay replying to you. We're trying to get an answer from Daniel on this point. We'll be back to you as soon as we can.

[@Alejvm97]

Is there a part two of this video? I'm so interested in 23:05 ...

[@EUMETSAT1]

Hello, thanks for getting in touch. There's no follow-up video planned as yet, but we'll definitely take your feedback on board!

[@ha2709]

Thanks for your video. How can i get the .nc file

[@EUMETSAT1]

Hello, please get in touch with: [e-mail] for assistance. Thanks!

[@ZPsychoMan.]

Nice video, but how I create Netcdf files?

[@EUMETSAT1]

You can find out more about the NetCDF format on the UCAR Unidata website, Matheus: www.unidata.ucar.edu/software/netcdf/

[@suhaibmalik6062]

This is a good tutorial! I have a quick query regarding (s5phub.copernicus.eu/dhus/#/home) NO2 data. though the file which was downloaded is in Netcdf, but I am unable to view properly in panoply. Though file is open but how can I find the latitude and longitutude? the X-axis contain ground pixel while Y axis contain scale line. and how can i be able to export into ArcMAP as a raster file? Thanks.

[@bajitborlanglchyne960]

How about displaying only a my particular country??

[@EUMETSAT1]

Hello, please get in touch with: [e-mail] so we can help you with your query. Thanks!

[@imranahmed6232]

Can anyone help me to convert data from netCDF format to CSV format using python?

[@EUMETSAT1]

Hello, thanks for your question. There is no straightforward way to convert between netCDF and CSV as the two formats are very different - CSV implies a tabular structure, whereas netCDF files can be composed of arrays with different lengths. Also, variables in netCDF files can be associated with each other, while this is not necessarily the case in CSV. We would advise that you either search for examples on the Internet where other users have done similar things, because they might show more of what you're looking for, or to use the netCDF directly in Python (this might be the better solution). Otherwise there’s not really a general way to convert between these formats. We hope this helps - please don't hesitate to get back in touch!

[@anrob4]

You have a typo in the link for Panoply. It should be "giss" instead "gisst". www.giss.nasa.gov/tools/panoply/

[@ollydix]

What a mess. Wouldn't a simple CSV be a much better format to use?

[@princess4sadia]

nice one Daniel. Do you have a girlfriend?