Convert netCDF into CSV using python

Question

I am new to Python and trying to parse the netCDF file into .CSV in columanar format, so I can load that data into RDBMS for other reporting purpose. Please refer the details below.

Sanpshot of my netCDF file:

dimensions:
time = UNLIMITED ; // (36 currently)
grid_latitude = 548 ;
grid_longitude = 421 ;
time_0 = UNLIMITED ; // (3 currently)
pressure = 3 ;
time_1 = UNLIMITED ; // (3 currently)
bnds = 2 ;
pressure_0 = 2 ;
pressure_1 = 3 ;
dim0 = UNLIMITED ; // (3 currently)
grid_longitude_0 = 421 ;
grid_latitude_0 = 547 ;
time_3 = UNLIMITED ; // (3 currently)
variables:
float stratiform_snowfall_rate(time, grid_latitude, grid_longitude) ;
stratiform_snowfall_rate:_FillValue = -1.073742e+09f ;
string stratiform_snowfall_rate:long_name = "stratiform_snowfall_rate" ;
string stratiform_snowfall_rate:units = "kg m-2 s-1" ;
string stratiform_snowfall_rate:um_stash_source = "m01s04i204" ;
string stratiform_snowfall_rate:grid_mapping = "rotated_latitude_longitude" ;string stratiform_snowfall_rate:coordinates = "forecast_period forecast_reference_time" ;int rotated_latitude_longitude ;

My Code:

from netCDF4 import Dataset, num2date
filename ='prods_op_mogreps-uk_20140717_03_11_015.nc'
nc = Dataset(filename, 'r', Format='NETCDF4')
 ncv = nc.variables
 lats = nc.variables['grid_latitude'][:]
 lons = nc.variables['grid_longitude'][:]
 sfc= nc.variables['stratiform_snowfall_rate'][:]
 times = nc.variables['time'][:]
 units = nc.variables['time'].units
 dates = num2date (times[:], units=units, calendar='365_day')
 header = ['Latitude', 'Longitude']
 for d in dates:
    header.append(d)
import csv
with open('output.csv', 'wb') as csvFile:
    outputwriter = csv.writer(csvFile, delimiter=',')
    for time_index, time in enumerate(times): # pull the dates out for the header
         t = num2date(time, units = units, calendar='365_day')
         header.append(t)
    outputwriter.writerow(header)  
    for lat_index, lat in enumerate(lats):
        content = lat
        #print lat_index
        for lon_index, lon in enumerate(lons):
            content.append(lon)
            #print lon_index    
            for time_index, time in enumerate(times): # for a date
                # pull out the data 
                data = sfc[time_index,lat_index,lon_index]
                content.append(data)
                outputwriter.writerow(content)
csvFile.close()
nc.close()

I am getting below error:

TypeError                                 Traceback (most recent call last)
<ipython-input-41-b4b3b888999f> in <module>
      4          t = num2date(time, units = units, calendar='365_day')
      5          header.append(t)
----> 6     outputwriter.writerow(header)
      7     for lat_index, lat in enumerate(lats):
      8         content = lat

TypeError: a bytes-like object is required, not 'str'

Please help me on this code. Thanks

SpghttCd · Accepted Answer · 2019-05-02 06:36:38Z

1

You open your output file in binary mode by choosing 'wb'.
Therefore the file writing function expects binary data, i.e. a bytes object.

However, as you're requesting help for writing a csv-file, I assume you want to write plain text data, so you simply should remove the b for binary here:

with open('output.csv', 'w') as csvFile:

answered May 2, 2019 at 6:36

SpghttCd

10.9k2 gold badges23 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

B. Go · Accepted Answer · 2020-06-12 16:33:33Z

1

The easiest method for this would be to use xarray and pandas.

import xarray as xr
import pandas as pd

You will first need to read in the data using xarray:

data = xr.open_dataset(filename)

This will then need to be converted to a pandas dataset, with the index reset:

data_df = data.to_dataframe().reset_index()

Finally, you would need to save this as a csv:

data_df.to_csv(outfile)

edited Jun 12, 2020 at 16:33

B. Go

1,4244 gold badges18 silver badges23 bronze badges

answered Jun 12, 2020 at 15:40

Robert Wilson

3,44714 silver badges22 bronze badges

Collectives™ on Stack Overflow

Convert netCDF into CSV using python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related