Python's multi-platform availability is awesome, I develop on Mac and execute on Windows or Linux. What isn't awesome is dependency management, every time I touch a machine I go through a loop...
- Run the script.. it fails
- Pip install whatever I've missed.. I typically code my self reminders
- Try again, and keep looping until it works.
One option might be ansible, script the dependencies before running the script, but ansible+linux, ansible+windows & ansible+osx are not interchangeable; even in linux you have to write your playbooks to deal with APT & YUM differences, out the gate I was spending too much time on the playbook and not what I was trying to do. I have also explored virtualenv as an option and whereas it separates the host-os from script dependency it doesn't solve portability issue, I'd still end up going thru the loop.
Having read this, the cool kids are screaming CONTAINERS at the screen. After reading a few tutorials it's quite easy to get a webserver like NGINX up and running but it takes an extra penny to drop to see how that applies to python scripts and that penny will probably only drop if you have to port your stuff onto different systems.
So that should explain the why?!?! And here are my examples to help you out.
Example 1: Containerised Python Requests.
Python requests is a common dependency for me, if you need your script to talk to "the outside world" the chances are you need requests. Requests is really easy to install, so this is like a Hello World on running a python script in docker.
First up you need a Docker file and a base image, I like to start with the official python+alpine container; the pip install on top of that, here's my example...
FROM python:alpine
LABEL maintainer="Nick <linickx.com>"
LABEL version="0.1"
RUN pip install requests
WORKDIR /app
ENTRYPOINT ["python"]
A couple of points, WORKDIR is where in the container your command is executed, and ENTRYPOINT is the default executable before any COMMAND that might be run.
If you're read the getting started on docker, you might be wondering how this has made things simpler, building a container for each system then running it is just as much pain as ansible/virtualenv/another-script... Well, this is where docker-compose comes in, here's my example...
version: '2'
services:
py_local:
build:
context: ./
dockerfile: Dockerfile
command: test.py
volumes:
- ./:/app/
Here we build the container (with our dependencies), mount the local directory as /app/ and then execute test.py.. command happens after entrypoint so what happens inside the container is something like cd $WORKDIR;python test.py... all with ONE COMMAND -> docker-compose up
mbp:$ docker-compose up
Building py_local
Step 1/6 : FROM python:alpine
---> 29b5ce58cfbc
Step 2/6 : LABEL maintainer="Nick <linickx.com>"
---> Running in fcf3f52dbf31
Removing intermediate container fcf3f52dbf31
---> 45564d24bf07
Step 3/6 : LABEL version="0.1"
---> Running in 73280d93d36b
Removing intermediate container 73280d93d36b
---> ec67a43f288b
Step 4/6 : RUN pip install requests
---> Running in 329180a50263
Collecting requests
Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB)
Collecting certifi>=2017.4.17 (from requests)
Downloading https://files.pythonhosted.org/packages/56/9d/1d02dd80bc4cd955f98980f28c5ee2200e1209292d5f9e9cc8d030d18655/certifi-2018.10.15-py2.py3-none-any.whl (146kB)
Collecting urllib3<1.25,>=1.21.1 (from requests)
Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting idna<2.8,>=2.5 (from requests)
Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests)
Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Installing collected packages: certifi, urllib3, idna, chardet, requests
Successfully installed certifi-2018.10.15 chardet-3.0.4 idna-2.7 requests-2.20.0 urllib3-1.24.1
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 329180a50263
---> b8549d493202
Step 5/6 : WORKDIR /app
---> Running in 62475957059e
Removing intermediate container 62475957059e
---> 24091216b63b
Step 6/6 : ENTRYPOINT ["python"]
---> Running in 4072f961b795
Removing intermediate container 4072f961b795
---> 2e1680284ec1
Successfully built 2e1680284ec1
Successfully tagged docker-python-alpine-requests_py_local:latest
WARNING: Image for service py_local was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating docker-python-alpine-requests_py_local_1 ... done
Attaching to docker-python-alpine-requests_py_local_1
py_local_1 | [INFO] 2018-11-04 10:57:46,652 Requests Works!
docker-python-alpine-requests_py_local_1 exited with code 0
mbp:$
So, [INFO] 2018-11-04 10:57:46,652 Requests Works! is the output of the script after it ran, and since we mounted ./ as /app/ if your script wrote out a file it would appear locally on your system (and not in the container)
Example 2: Pandas
Building requests in a container as a single pip command might not seem that much of a time save, however something a bit more tricky like python's pandas is a different story; panda's has a couple of OS dependencies plus a pip dependency.
I've posted a Dockerfile on github, and also a public image, as you can see from the pull below there's a bit more under the hood; the output below is from this docker-compose file.
mbp:$ docker-compose up
Pulling py_pandas (linickx/python-alpine-pandas:)...
latest: Pulling from linickx/python-alpine-pandas
4fe2ade4980c: Already exists
7cf6a1d62200: Already exists
3be976674be6: Already exists
c52373c891c4: Already exists
afe67e449426: Already exists
b000252b6aec: Pull complete
dd3345e1f782: Pull complete
ac49affbd3aa: Pull complete
3be8701c00d1: Pull complete
7f48fbfd73c2: Pull complete
319ea22ffca9: Pull complete
fed0e39d2eaf: Pull complete
53c49977267c: Pull complete
Digest: sha256:22d5162f5cfd01b6665d2952c42606a69a57d7bba5861b61ca88724884e1380b
Status: Downloaded newer image for linickx/python-alpine-pandas:latest
Creating docker-python-alpine-pandas_py_pandas_1 ... done
Attaching to docker-python-alpine-pandas_py_pandas_1
py_pandas_1 | [INFO] 2018-10-28 21:18:44,458 Pandas Works!
docker-python-alpine-pandas_py_pandas_1 exited with code 0
mbp:$
A quick note on Security
I have posted my public images for testing, the sources of which can be found on github, however no-one should trust a docker container from a random on the Internet, you have no control over what goes in and the container or it's dependencies could change without your knowledge, find a trusted base like the official alpine and build your local Dockerfile & repositories from there :)