I have a notebook execute node in a workflow. In the notebook, main routine calls a function that returns a value. In the notebook output property if a have something like notebookResponse, I get only the following and not the return value.
notebookResponse
{
“executionId”: “62b0ac80902a5283cd8beba0”,
“success”: true
}
Notebooks run asynchronously, because depending on how complex the notebook is, it can take up to 60 minutes to complete.
This node only schedules the notebook to execute. It then returns whether it successfully scheduled the execution and a unique ID for the execution.
To obtain the results of a notebook, you’ll need to use the Notebook Trigger, which will fire a workflow whenever a notebook execution completes.
You can find more about this in the Notebook: Execute Node documentation, specifically this section:
NOTE this node does not wait for a notebook execution to complete. It only requests the execution of a notebook. If you want to react to the completion of a notebook in a workflow, you will want to use the Notebook Trigger.
I tried to run the way you suggested. Thanks
However, my notebook run keeps failing as in the attached result.log. The notebook perfectly runs outside in a standalone Jupyter notebook.
results.log
[NbConvertApp] Converting notebook /tmp/input.ipynb to notebook
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-nbconvert", line 11, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
app.start()
File "/opt/conda/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 361, in start
self.convert_notebooks()
File "/opt/conda/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 533, in convert_notebooks
self.convert_single_notebook(notebook_filename)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 498, in convert_single_notebook
output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/nbconvertapp.py", line 427, in export_single_notebook
output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 186, in from_filename
return self.from_file(f, resources=resources, **kw)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 204, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
nb_copy, resources = self._preprocess(nb_copy, resources)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/exporters/exporter.py", line 323, in _preprocess
nbc, resc = preprocessor(nbc, resc)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
return self.preprocess(nb, resources)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 84, in preprocess
self.preprocess_cell(cell, resources, index)
File "/opt/conda/lib/python3.9/site-packages/nbconvert/preprocessors/execute.py", line 105, in preprocess_cell
cell = self.execute_cell(cell, index, store_history=True)
File "/opt/conda/lib/python3.9/site-packages/nbclient/util.py", line 84, in wrapped
return just_run(coro(*args, **kwargs))
File "/opt/conda/lib/python3.9/site-packages/nbclient/util.py", line 62, in just_run
return loop.run_until_complete(coro)
File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/opt/conda/lib/python3.9/site-packages/nbclient/client.py", line 854, in async_execute_cell
self._check_raise_for_error(cell, exec_reply)
File "/opt/conda/lib/python3.9/site-packages/nbclient/client.py", line 756, in _check_raise_for_error
raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
#!/usr/bin/env python
import requests
import json
class MahaDiscom(object):
""" Class to get det
ails about Mahadiscom """
def __init__(self, cn, bun, ct):
""" __init__ """
self.serverurl = 'https://wss.mahadiscom.in/wss/'
self.con_details = {
'ConsumerNo': cn,
'BuNumber': bun,
'consumerType': ct,
}
def is_consumer_valid(self):
"""
Check if consumer is valid or not
returns : True if consumer is valid
False if consumer is not valid
"""
actionurl = 'wss?uiActionName=validateConsumerNumberHTLT&IsAjax=true'
url = self.serverurl + actionurl
response = requests.post(url, data=self.con_details)
if response.status_code != 200 or response.text == 'false':
return False
elif response.text == 'true':
return True
def get_bill_details(self):
"""
Get bill details
returns : a dictionary with consumer's bill details
"""
billdetails = {}
actionurl = "wss?uiActionName=viewImpExpInformation&IsAjax=true"
url = self.serverurl + actionurl
response = requests.post(url, data=self.con_details)
try:
billdetails = json.loads(response.text)
except ValueError as err:
print("Unable to parse json response " + str(err))
return {}
if response.status_code == 200:
return billdetails
print("ERROR: Return code is Non-2xx : %d" % response.status_code)
return {}
def main():
mahadiscom = MahaDiscom(cn='266513040164',bun='4016', ct='LT')
billdetails = mahadiscom.get_bill_details()
return billdetails
#i = 0
#print (len(billdetails))
#while i < len(billdetails):
#jo=json.loads(billdetails[i])
#print(billdetails[i]["BILL_MONTH"]+" "+str(billdetails[i]["BILL_YEAR"]))
#print("EXP " + str(billdetails[i]["EXP_UNITS_KWH_TOTAL"]))
#print("IMP "+ str(billdetails[i]["IMP_UNITS_KWH_TOTAL"]))
#print("GEN "+str(billdetails[i]["GEN_UNITS_KWH_TOTAL"]))
#print("----------")
#i += 1
if __name__ == "__main__":
#data = {}
#data = main()
#return data
main()
------------------
---------------------------------------------------------------------------
gaierror Traceback (most recent call last)
/opt/conda/lib/python3.9/site-packages/urllib3/connection.py in _new_conn(self)
173 try:
--> 174 conn = connection.create_connection(
175 (self._dns_host, self.port), self.timeout, **extra_kw
/opt/conda/lib/python3.9/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
72
---> 73 for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
74 af, socktype, proto, canonname, sa = res
/opt/conda/lib/python3.9/socket.py in getaddrinfo(host, port, family, type, proto, flags)
953 addrlist = []
--> 954 for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
955 af, socktype, proto, canonname, sa = res
gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
NewConnectionError Traceback (most recent call last)
/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
698 # Make the request on the httplib connection object.
--> 699 httplib_response = self._make_request(
700 conn,
/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
381 try:
--> 382 self._validate_conn(conn)
383 except (SocketTimeout, BaseSSLError) as e:
/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
1009 if not getattr(conn, "sock", None): # AppEngine might not have `.sock`
-> 1010 conn.connect()
1011
/opt/conda/lib/python3.9/site-packages/urllib3/connection.py in connect(self)
357 # Add certificate verification
--> 358 conn = self._new_conn()
359 hostname = self.host
/opt/conda/lib/python3.9/site-packages/urllib3/connection.py in _new_conn(self)
185 except SocketError as e:
--> 186 raise NewConnectionError(
187 self, "Failed to establish a new connection: %s" % e
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f872f59ab20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
MaxRetryError Traceback (most recent call last)
/opt/conda/lib/python3.9/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
438 if not chunked:
--> 439 resp = conn.urlopen(
440 method=request.method,
/opt/conda/lib/python3.9/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
754
--> 755 retries = retries.increment(
756 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
/opt/conda/lib/python3.9/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
573 if new_retry.is_exhausted():
--> 574 raise MaxRetryError(_pool, url, error or ResponseError(cause))
575
MaxRetryError: HTTPSConnectionPool(host='wss.mahadiscom.in', port=443): Max retries exceeded with url: /wss/wss?uiActionName=viewImpExpInformation&IsAjax=true (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f872f59ab20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
ConnectionError Traceback (most recent call last)
/tmp/ipykernel_15/2811413364.py in <module>
71 #data = main()
72 #return data
---> 73 main()
74
75
/tmp/ipykernel_15/2811413364.py in main()
53 def main():
54 mahadiscom = MahaDiscom(cn='266513040164',bun='4016', ct='LT')
---> 55 billdetails = mahadiscom.get_bill_details()
56 return billdetails
57 #i = 0
/tmp/ipykernel_15/2811413364.py in get_bill_details(self)
38 actionurl = "wss?uiActionName=viewImpExpInformation&IsAjax=true"
39 url = self.serverurl + actionurl
---> 40 response = requests.post(url, data=self.con_details)
41 try:
42 billdetails = json.loads(response.text)
/opt/conda/lib/python3.9/site-packages/requests/api.py in post(url, data, json, **kwargs)
115 """
116
--> 117 return request('post', url, data=data, json=json, **kwargs)
118
119
/opt/conda/lib/python3.9/site-packages/requests/api.py in request(method, url, **kwargs)
59 # cases, and look like a memory leak in others.
60 with sessions.Session() as session:
---> 61 return session.request(method=method, url=url, **kwargs)
62
63
/opt/conda/lib/python3.9/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
540 }
541 send_kwargs.update(settings)
--> 542 resp = self.send(prep, **send_kwargs)
543
544 return resp
/opt/conda/lib/python3.9/site-packages/requests/sessions.py in send(self, request, **kwargs)
653
654 # Send the request
--> 655 r = adapter.send(request, **kwargs)
656
657 # Total elapsed time of the request (approximately)
/opt/conda/lib/python3.9/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
514 raise SSLError(e, request=request)
515
--> 516 raise ConnectionError(e, request=request)
517
518 except ClosedPoolError as e:
ConnectionError: HTTPSConnectionPool(host='wss.mahadiscom.in', port=443): Max retries exceeded with url: /wss/wss?uiActionName=viewImpExpInformation&IsAjax=true (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f872f59ab20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
ConnectionError: HTTPSConnectionPool(host='wss.mahadiscom.in', port=443): Max retries exceeded with url: /wss/wss?uiActionName=viewImpExpInformation&IsAjax=true (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f872f59ab20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
Jupyter exited with status code: 1
It looks as though your notebook is attempting to access resources from the internet. The Notebook Execution Environment does not have network access. This is due to the security concerns of running arbitrary Python code within our cloud infrastructure. All required input data must be passed to the notebook via Inputs.
Does this mean I cannot make any HTTP requests within the notebook?
Correct, HTTP requests within a notebook are not supported.
Any alternative you suggest to do this. I do not have a REST API to do this
It looks like you might be accessing a WebSocket endpoint. If that’s the case, I do not have any easy workarounds. I would probably recommend shifting this code to an AWS Lambda Function, Google Function, or Azure Function.
These environments do allow network access and can be directly invoked using a workflow.
Thanks, will look into