Notebook Issues - Read CSV

Help
1

Hi All,

I’m struggling to get anything out of my notebooks. I’ve tried lots of different variations and examples from the forums but can’t seem to find one which works for me.

I have setup a simple input. 60 minutes of data into data.csv

I have setup a simple Notebook file. Code is:

import os
import pandas as pd

input_dir = os.path.join(os.environ[‘INPUT_DIR’])

dataset = pd.read_csv(input_dir, ‘data.csv’)

–

My notebook debug is:

–

[NbConvertApp] Converting notebook /tmp/input.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] ERROR | Error while converting ‘/tmp/input.ipynb’
Traceback (most recent call last):
File “/opt/conda/lib/python3.7/site-packages/nbconvert/nbconvertapp.py”, line 407, in export_single_notebook
output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/exporter.py”, line 178, in from_filename
return self.from_file(f, resources=resources, **kw)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/exporter.py”, line 196, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/notebook.py”, line 32, in from_notebook_node
nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/exporter.py”, line 138, in from_notebook_node
nb_copy, resources = self._preprocess(nb_copy, resources)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/exporters/exporter.py”, line 315, in _preprocess
nbc, resc = preprocessor(nbc, resc)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/preprocessors/base.py”, line 47, in call
return self.preprocess(nb, resources)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py”, line 354, in preprocess
nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/preprocessors/base.py”, line 69, in preprocess
nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
File “/opt/conda/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py”, line 378, in preprocess_cell
raise CellExecutionError.from_cell_and_msg(cell, out)
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:

import os
import pandas as pd

input_dir = os.path.join(os.environ[‘INPUT_DIR’])

dataset = pd.read_csv(input_dir, ‘data.csv’)

IsADirectoryError Traceback (most recent call last)
in
4 input_dir = os.path.join(os.environ[‘INPUT_DIR’])
5
----> 6 dataset = pd.read_csv(input_dir, ‘data.csv’)

/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
683 )
684
–> 685 return _read(filepath_or_buffer, kwds)
686
687 parser_f.name = name

/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
455
456 # Create the parser.
–> 457 parser = TextFileReader(fp_or_buf, **kwds)
458
459 if chunksize or iterator:

/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py in init(self, f, engine, **kwds)
893 self.options[“has_index_names”] = kwds[“has_index_names”]
894
–> 895 self._make_engine(self.engine)
896
897 def close(self):

/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py in _make_engine(self, engine)
1145 ’ “python-fwf”)’.format(engine=engine)
1146 )
-> 1147 self._engine = klass(self.f, **self.options)
1148
1149 def _failover_to_python(self):

/opt/conda/lib/python3.7/site-packages/pandas/io/parsers.py in init(self, f, **kwds)
2280 encoding=self.encoding,
2281 compression=self.compression,
-> 2282 memory_map=self.memory_map,
2283 )
2284 self.handles.extend(handles)

/opt/conda/lib/python3.7/site-packages/pandas/io/common.py in _get_handle(path_or_buf, mode, encoding, compression, memory_map, is_text)
398 elif is_text:
399 # No explicit encoding
–> 400 f = open(path_or_buf, mode, errors=“replace”, newline="")
401 else:
402 # Binary mode

IsADirectoryError: [Errno 21] Is a directory: ‘/tmp/input/’
IsADirectoryError: [Errno 21] Is a directory: ‘/tmp/input/’

Jupyter exited with status code: 1

–

Any advice would be appreciated.

I have managed to get some other notebooks working on my local machine. However porting to Losant is my issue.

Many thanks,
Chris

3

Hi @ChrisGM,

We here at the Losant Forums are happy to help with Losant Notebook problems surrounding accessing data and performing analysis. Our expertise lies in the Losant Platform, so we recommend checking out the Jupyter Blog, the Jupyter Live Chat, or posting on the Jupyter Discourse Forum for challenging problems within the Jupyter Notebook and help writing Notebook Code. In this case, it looks as though this snippet will cause some issues:

input_dir = os.path.join(os.environ[‘INPUT_DIR’])
dataset = pd.read_csv(input_dir, ‘data.csv’)

In this first line, you are not joining any values, which is likely resulting in this error; I will update the documentation to fix this snippet. I would recommend rewriting this to:

dataset = pd.read_csv(os.path.join(os.environ['INPUT_DIR'], 'data.csv'))

Please let me know if this does not provide a resolution,
Julia

1 Like