Packages and Libraries#
Author: Mike Wood
Learning Objectives: By the end of this notebook, you should be able to:
Install a new conda library on your machine
Generate a list of all modules available in your conda environment
Develop your own Python library
Implement a custom Python library in your conda environment
Installing Conda Packages#
In this course, we are using miniconda to manage our installation of Python as well as the packages we installed for this course. You may remember from our first lecture that we installed a package so that we can use this Jupyter notebook with the following command in our terminal:
conda install jupyter
To remind ourselves of this process we will install two more packages which will be used later on in this class: pandas and scipy
The pandas library will provide a convenient way to work with numeric data. The scipy library will give us a wide range of computational tools.
If you’d like to get a list of all the modules available in your conda installation, you can use the help('modules') feature. Note that may have received some UserWarnings in the when using this function - this results from Python checking the import of each individual module. Developers will place a UserWarning in a module when they suspect it will be depricated later. If you’d like to disable these statements, fear not! There is a module for that:
# import the warnings module
import warnings
# use the warnings module to filer out the UserWarnings
warnings.simplefilter("ignore", UserWarning)
# use the help method to generate a list of all modules
help('modules')
Show code cell output
Please wait a moment while I gather a list of all available modules...
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
Cell In[2], line 2
1 # use the help method to generate a list of all modules
----> 2 help('modules')
File ~/opt/anaconda3/envs/cs122/lib/python3.10/_sitebuiltins.py:103, in _Helper.__call__(self, *args, **kwds)
101 def __call__(self, *args, **kwds):
102 import pydoc
--> 103 return pydoc.help(*args, **kwds)
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pydoc.py:2006, in Helper.__call__(self, request)
2004 def __call__(self, request=_GoInteractive):
2005 if request is not self._GoInteractive:
-> 2006 self.help(request)
2007 else:
2008 self.intro()
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pydoc.py:2053, in Helper.help(self, request)
2051 elif request == 'symbols': self.listsymbols()
2052 elif request == 'topics': self.listtopics()
-> 2053 elif request == 'modules': self.listmodules()
2054 elif request[:8] == 'modules ':
2055 self.listmodules(request.split()[1])
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pydoc.py:2205, in Helper.listmodules(self, key)
2203 def onerror(modname):
2204 callback(None, modname, None)
-> 2205 ModuleScanner().run(callback, onerror=onerror)
2206 self.list(modules.keys())
2207 self.output.write('''
2208 Enter any module name to get more help. Or, type "modules spam" to search
2209 for modules whose name or summary contain the string "spam".
2210 ''')
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pydoc.py:2234, in ModuleScanner.run(self, callback, key, completer, onerror)
2231 if name.lower().find(key) >= 0:
2232 callback(None, modname, desc)
-> 2234 for importer, modname, ispkg in pkgutil.walk_packages(onerror=onerror):
2235 if self.quit:
2236 break
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pkgutil.py:107, in walk_packages(path, prefix, onerror)
104 # don't traverse path items we've seen before
105 path = [p for p in path if not seen(p)]
--> 107 yield from walk_packages(path, info.name+'.', onerror)
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pkgutil.py:107, in walk_packages(path, prefix, onerror)
104 # don't traverse path items we've seen before
105 path = [p for p in path if not seen(p)]
--> 107 yield from walk_packages(path, info.name+'.', onerror)
File ~/opt/anaconda3/envs/cs122/lib/python3.10/pkgutil.py:92, in walk_packages(path, prefix, onerror)
90 if info.ispkg:
91 try:
---> 92 __import__(info.name)
93 except ImportError:
94 if onerror is not None:
File ~/opt/anaconda3/envs/cs122/lib/python3.10/site-packages/jupyter_cache/cli/commands/__init__.py:3
1 """The jupyter-cache CLI."""
----> 3 from .cmd_cache import * # noqa: F401,F403,E402
4 from .cmd_notebook import * # noqa: F401,F403,E402
5 from .cmd_project import * # noqa: F401,F403,E402
File ~/opt/anaconda3/envs/cs122/lib/python3.10/site-packages/jupyter_cache/cli/commands/cmd_cache.py:3
1 import click
----> 3 from jupyter_cache.cli import arguments, options, pass_cache
4 from jupyter_cache.cli.commands.cmd_main import jcache
5 from jupyter_cache.utils import tabulate_cache_records
File ~/opt/anaconda3/envs/cs122/lib/python3.10/site-packages/jupyter_cache/cli/options.py:7
4 import click
6 from jupyter_cache.entry_points import ENTRY_POINT_GROUP_EXEC, list_group_names
----> 7 from jupyter_cache.readers import list_readers
10 def callback_autocomplete(ctx, param, value):
11 if value and not ctx.resilient_parsing:
File ~/opt/anaconda3/envs/cs122/lib/python3.10/site-packages/jupyter_cache/readers.py:5
1 """Module for handling different functions to read "notebook-like" files."""
3 from typing import Any, Callable, Dict, Set
----> 5 import nbformat as nbf
7 from .entry_points import ENTRY_POINT_GROUP_READER, get_entry_point, list_group_names
9 DEFAULT_READ_DATA = (("name", "nbformat"), ("type", "plugin"))
File ~/opt/anaconda3/envs/cs122/lib/python3.10/site-packages/nbformat/__init__.py:11
6 # Copyright (c) IPython Development Team.
7 # Distributed under the terms of the Modified BSD License.
9 from traitlets.log import get_logger
---> 11 from . import v1, v2, v3, v4
12 from ._version import __version__, version_info
13 from .sentinel import Sentinel
File ~/opt/anaconda3/envs/cs122/lib/python3.10/site-packages/nbformat/v3/__init__.py:57
55 from .nbjson import writes as write_json
56 from .nbjson import writes as writes_json
---> 57 from .nbpy import reads as read_py
58 from .nbpy import reads as reads_py
59 from .nbpy import to_notebook as to_notebook_py
File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)
File <frozen importlib._bootstrap>:1006, in _find_and_load_unlocked(name, import_)
File <frozen importlib._bootstrap>:688, in _load_unlocked(spec)
File <frozen importlib._bootstrap_external>:879, in exec_module(self, module)
File <frozen importlib._bootstrap_external>:975, in get_code(self, fullname)
File <frozen importlib._bootstrap_external>:1074, in get_data(self, path)
KeyboardInterrupt:
We can see in the above block that we have modules from the standard installation (e.g. abc) and we also have modules which we installed previously (e.g. jupyter).
Creating a Package#
So far, we have created modules that contain functions that used within Jupyter notebooks. Today, we’ll create several modules and group them together into a package.
Let’s begin by creating a module called general_notes that contains a simple function module_import_note with a print statement that reminds us about importing modules. When complete, run the following code block to ensure your package works as expected:
# import the general_notes module
import general_notes as gn
# test the module_import_note function
gn.module_import_note()
A module can be imported in a Jupyter notebook
if it is in the same directory as the notebook
Next, let’s create a separate module called conda_notes that specifically stores notes about conda. Build a function with a simple function called
# import the conda_notes module
import conda_notes as cn
# test the conda_environment_setup function
cn.conda_environment_setup()
The command to make a new conda environment is conda create --name my_env
Now, consider if we could bundle these scripts into a package that we could import together.
To create a package, first create a directory structure for your package and move everything into the directories. For this example, we will create a directory called python_notes with a subdirectory called conda. Put the general_notes module into the top level directory and the conda_notes modules into the subdirectory.
To give your directories a package structure, we need to add an __init__.py module to each subdirectory. The __init__.py module doesn’t need to contain any code - but you can add code if you like. This is typically where developers will add UserWarnings as we saw above. Add the __init__.py modules and try import your modules from the package and running the functions we defined above:
# import the python_notes package
import python_notes.general_notes as gn
# run the module_import_note function from the general_notes modules
gn.module_import_note()
A module can be imported in a Jupyter notebook
if it is in the same directory as the notebook
Using your package from a different directory#
If you want to use your package from a different directory, then one option you have is to tell Python where you want it to look. One way to do this is to use the sys module to modify your system path:
# import the sys module
import sys
# insert the path to the directory of the module in another directory
sys.path.insert(1,'/Users/mike/Documents/Python/')
Now, the module can be accessed, imported, and utilized:
# import the python_notes package
import python_notes.conda.conda_notes as cn
# test the conda_environment_setup function
cn.conda_environment_setup()
The command to make a new conda environment is conda create --name my_env
Installing Your Package in Your Conda Environment#
After you’ve created a package, you may be interested in having your package available in other contexts. We can see above how to explicitly provide a path to a package. How can we install our package into our conda environment? The first thing we will need is a new file called setup.py and fill it with the following contents:
Now, using your terminal, run the command from the directory where your setup.py file is located (be sure your conda environment is activated):
pip install .
If the installation is successful, your should receive a nice note about it was successfully installed. Let’s check that we now see it in our list:
# use the help method to generate a list of all modules
help('modules')
Show code cell output
Please wait a moment while I gather a list of all available modules...
AppKit babel ipykernel random
Cocoa backcall ipykernel_launcher ratelim
CoreFoundation backports ipython_genutils re
Foundation base64 ipywidgets readline
IPython bdb isoduration referencing
OpenSSL binascii itertools reprlib
PIL binhex itsdangerous requests
__future__ bisect jedi requests_oauthlib
_abc bleach jinja2 resource
_aix_support blinker joblib retrying
_argon2_cffi_bindings branca json rfc3339_validator
_ast brotli json5 rfc3986_validator
_asyncio bs4 jsonpointer rlcompleter
_bisect builtins jsonschema rpds
_blake2 bz2 jsonschema_specifications rsa
_bootsubprocess cProfile jupyter runpy
_brotli cached_property jupyter_client sched
_bz2 cachetools jupyter_console scipy
_cffi_backend calendar jupyter_core secrets
_codecs certifi jupyter_events select
_codecs_cn cffi jupyter_lsp selectors
_codecs_hk cgi jupyter_server send2trash
_codecs_iso2022 cgitb jupyter_server_terminals setuptools
_codecs_jp charset_normalizer jupyterlab shelve
_codecs_kr chunk jupyterlab_plotly shlex
_codecs_tw click jupyterlab_pygments shutil
_collections cmath jupyterlab_server signal
_collections_abc cmd jupyterlab_widgets site
_compat_pickle code jwt six
_compression codecs keyword sklearn
_contextvars codeop kiwisolver smtpd
_crypt collections lib2to3 smtplib
_csv colorama libfuturize sndhdr
_ctypes colorsys libpasteurize sniffio
_ctypes_test colour linecache socket
_curses comm locale socketserver
_curses_panel compileall logging socks
_datetime concurrent lxml sockshandler
_dbm conda_notes lzma soupsieve
_decimal configparser mailbox sqlite3
_distutils_hack contextlib mailcap sqlparse
_elementtree contextvars markupsafe sre_compile
_functools contourpy marshal sre_constants
_hashlib copy math sre_parse
_heapq copyreg matplotlib ssl
_imp crypt matplotlib_inline stack_data
_io cryptography mimetypes stat
_json cs122notes mistune statistics
_locale csv mmap string
_lsprof ctypes modulefinder stringprep
_lzma curses moviepy struct
_markupbase cycler multidict subprocess
_md5 dash multiprocessing sunau
_multibytecodec dash_core_components multitasking symtable
_multiprocessing dash_html_components munkres sys
_opcode dash_table nbclient sysconfig
_operator dataclasses nbconvert syslog
_osx_support datetime nbformat tabnanny
_pickle dateutil nest_asyncio tarfile
_plotly_future_ dbm netrc telnetlib
_plotly_utils debugpy nis tempfile
_posixshmem decimal nntplib tenacity
_posixsubprocess decorator notebook terminado
_py_abc defusedxml notebook_shim termios
_pydecimal difflib ntpath test
_pyio dis nturl2path test_module
_queue distutils numbers textwrap
_random django numpy this
_scproxy doctest oauthlib threading
_sha1 email objc threadpoolctl
_sha256 encodings opcode time
_sha3 ensurepip openpyxl timeit
_sha512 entrypoints operator timezonefinder
_signal enum optparse tinycss2
_sitebuiltins errno os tkinter
_socket et_xmlfile overrides tkmacosx
_sqlite3 exceptiongroup packaging token
_sre executing pandas tokenize
_ssl fastjsonschema pandocfilters tomli
_stat faulthandler parso tornado
_statistics fcntl past tqdm
_string filecmp pathlib trace
_strptime fileinput pdb traceback
_struct flask peewee tracemalloc
_symtable flask_cors pexpect traitlets
_sysconfigdata__darwin_darwin fnmatch pickle tty
_sysconfigdata_x86_64_apple_darwin13_4_0 folium pickleshare turtle
_testbuffer fontTools pickletools turtledemo
_testcapi fqdn pip types
_testclinic fractions pipes typing
_testimportmultiple frozendict pkg_resources typing_extensions
_testinternalcapi frozenlist pkgutil typing_utils
_testmultiphase ftplib pkgutil_resolve_name tzdata
_thread functools platform unicodedata
_threading_local future platformdirs unicodedata2
_tkinter gc playhouse unittest
_tracemalloc general_notes plistlib uri_template
_uuid genericpath plotly uritemplate
_warnings geocoder poplib urllib
_weakref geographiclib posix urllib3
_weakrefset geopy posixpath uu
_xxsubinterpreters getopt pprint uuid
_xxtestfuzz getpass profile venv
_yaml gettext proglog warnings
_zoneinfo glob prometheus_client wave
abc google_auth_httplib2 prompt_toolkit wcwidth
aifc google_auth_oauthlib pstats weakref
aiohttp googleapiclient psutil webbrowser
aiosignal graphlib pty webcolors
ansi2html grp ptyprocess webencodings
antigravity gzip pure_eval websocket
anyio h3 pwd werkzeug
apiclient hashlib pwiz wheel
appdirs heapq py_compile widgetsnbextension
appnope hmac pyasn1 wsgiref
argon2 html pyasn1_modules xdrlib
argparse html5lib pyclbr xml
array http pycparser xmlrpc
arrow httplib2 pydoc xxlimited
asgiref idlelib pydoc_data xxlimited_35
ast idna pyexpat xxsubtype
astrology imageio pygments xyzservices
asttokens imageio_ffmpeg pylab yaml
async_lru imaplib pyparsing yarl
async_timeout imghdr python_notes yfinance
asynchat imp pythonjsonlogger zipapp
asyncio importlib pytz zipfile
asyncore importlib_metadata pyu2f zipimport
atexit importlib_resources qtconsole zipp
attr inspect qtpy zlib
attrs io queue zmq
audioop ipaddress quopri zoneinfo
Enter any module name to get more help. Or, type "modules spam" to search
for modules whose name or summary contain the string "spam".
Now that our module is installed in our environment, we can check that the module can be imported, even if we move it out of directory:
# import the python_notes package
import python_notes.conda.conda_notes as cn
# test the conda_environment_setup function
cn.conda_environment_setup()
The command to make a new conda environment is conda create --name my_env
Modifying an installed package#
After creating an installing your package, its likely that you would like to add more contends to your module.
Add a new function to your python_notes modules. Try running your new function:
# run the module_import_note function from the general_notes modules
gn.package_installation()
To install a package using pip and a setup file, use the pip install . command