Pushbullet setup for monitoring...

I recently setup a home server and wanted to use pushbullet for notifications. I wittily thought that pbull would be a good name for my personal module (which sounds a bit like pit bull, the dog not the rapper). Thus the class called Bark... woof.

It turned out to be tremendously useful, so I share it here. Let me know if you have any questions. 

The only two parts that need to be updated are the nickname of you personal device where you want to receive the notifications and the API key from your account. 

from pushbullet.pushbullet import Pushbullet

class bark:
	def __init__(self):
		self.pb = Pushbullet(api_key=key)
		self.iden = None

	def get_my_phone(self, nickname="[ENTER YOUR PHONE NAME]"):
		print self.pb.list_devices()
		devices = self.pb.list_devices()['devices']
		for device in devices:
				if device['nickname'] == nickname:
					self.iden  = device['iden']
				print "no device"

	def message(self, title, body):
		print self.iden
		self.pb.bullet_note(self.iden, title, body)

Once you have the class built, you can create a command line version pretty easily. The code can go in your site-packages for easy importing into any project. 

Python and SAP: Part 3 - Write your queries in SQL

This is the third part of my SAP and Python series (see part 1 and part 2 first!)

After you complete part 1 and part 2 - you can connect to SAP and query data. However, the RFC call parameters are clunky and the whole thing feels a bit complicated for daily use (to me at least). 

To make my life easier,  I wrote a little SQL parser. It's not special and it's not even all that efficient, but the it does the job.

In part 2, I wrote the following (without comments for a quicker read):

s = main() 

fields = ['MATNR', 'EAN11']
table = 'MEAN'
where = ['MATNR <> 0']
maxrows = 10
fromrow = 0

results, headers = s.qry(fields, table, where, maxrows, fromrow)

print headers
print results

Hardly pretty... However, with my SQL parser, I write this instead:

s = main() 

query = "select matnr, ean11 from mean where matnr <> 0"
maxrows = 10
fromrow = 0

results, headers = s.sql_query(query, maxrows, fromrow)

print headers
print results

Much cleaner. I just write SQL and suddenly, writing a little program do some mundane query is a breeze!

The two functions for this lovely toy is below. Let me know if you find a better method... 

Good luck with your SQL adventures :)

def split_where(self, seg):
    # This magical function splits by spaces when not enclosed in quotes..
    where = seg.split(' ')
    where = [x.replace('@', ' ') for x in where]
    return where

def select_parse(self, statement):
    statement = " ".join([x.strip('\t') for x in statement.upper().split('\n')])

    if 'WHERE' not in statement:
        statement = statement + ' WHERE '

    regex = re.compile("SELECT(.*)FROM(.*)WHERE(.*)")

    parts = regex.findall(statement)
    parts = parts[0]
    select = [x.strip() for x in parts[0].split(',')]
    frm = parts[1].strip()
    where = parts[2].strip()

    # splits by spaces but ignores quoted string with ''
    PATTERN = re.compile(r"""((?:[^ '"]|'[^']*'|"[^"]*")+)""")
    where = PATTERN.split(where)[1::2]

    cleaned = [select, frm, where]
    return cleaned


*** Update *** As requested by Florian, here is the code for sql_query. I've removed the headers option (as I prefer to get the rows into a dictionary, which provides the headers). 

def sql_query(self, statement, MaxRows=0, FromRow=0, to_dict=False):
    statement = self.select_parse(statement)

    results = self.qry(statement[0], statement[1], statement[2], MaxRows, FromRow)
    if to_dict:
        headers = statement[0]
        results2 = []
        for line in results:
            new_line = OrderedDict()
            header_counter = 0
            for field in line:
                    new_line[headers[header_counter]] = field.strip()
                    header_counter += 1
                except Exception as e:
                    new_line[headers[header_counter-1]] = new_line[headers[header_counter-1]]+ " " + " ".join(line[header_counter:])

        results = results2
    return results

Python and SAP: Part 2 - Getting data from SAP

So now you're connected to SAP (see part 1 here), but what do you do with it?

Well, as the name implies, PyRFC allows you to perform Remote Function Calls from Python. In theory, any valid Function Module in SAP should work. Use Transaction SE37 to view your function module and check the inputs. 

In my case, I wanted to read data from SAP. Therefore, the function module that interested me was RFC_READ_TABLE. In SE37, you can see the inputs for this function module (but I've done the work for you below).

Also, worth noting: You can do a search and see the general fear this solution produces. People get quite alarmist about anything that "queries" SAP. However, let's remember that there are serious controls that you can put in place (see this link). With a good basis team, calling functions and reading data from SAP can be locked down tight. It's clearly a concern, but completely manageable. 

On to the fun part! Let's create a little class for getting data out of SAP. We're going to use a class, rather than a script, so that we can build on it in the next lesson.

The connection class

First, we need to initialize our connection again. 

from pyrfc import Connection
import re

class main():
    def __init__(self):
        self.conn = Connection(ashost=ASHOST, sysnr=SYSNR, client=CLIENT, user=USER, passwd=PASSWD)

Ok - pretty simple When we initialize the class, we connect to SAP. You could pass the login parameters to the class - might even be a good idea.

Now we want to create a function to call RFC_READ_TABLE. To do this, we need to pass:

  • Fields: the fields we want to return
  • SQLTable: The table we want to query
  • Where: Any WHERE conditions
  • MaxRows: The maximum number of rows to return
  • FromRow: The starting row of the result set to use

Therefore, our query function should look like this:

    def qry(self, Fields, SQLTable, Where = '', MaxRows=50, FromRow=0):
        """A function to query SAP with RFC_READ_TABLE"""

        # By default, if you send a blank value for fields, you get all of them
        # Therefore, we add a select all option, to better mimic SQL.
        if Fields[0] == '*':
            Fields = ''
            Fields = [{'FIELDNAME':x} for x in Fields] # Notice the format

        # the WHERE part of the query is called "options"
        options = [{'TEXT': x} for x in Where] # again, notice the format

        # we set a maximum number of rows to return, because it's easy to do and
        # greatly speeds up testing queries.
        rowcount = MaxRows

        # Here is the call to SAP's RFC_READ_TABLE
        tables = self.conn.call("RFC_READ_TABLE", QUERY_TABLE=SQLTable, DELIMITER='|', FIELDS = Fields, \ 
                                OPTIONS=options, ROWCOUNT = MaxRows, ROWSKIPS=FromRow)

        # We split out fields and fields_name to hold the data and the column names
        fields = []
        fields_name = []

        data_fields = tables["DATA"] # pull the data part of the result set
        data_names = tables["FIELDS"] # pull the field name part of the result set

        headers = [x['FIELDNAME'] for x in data_names] # headers extraction
        long_fields = len(data_fields) # data extraction
        long_names = len(data_names) # full headers extraction if you want it

        # now parse the data fields into a list
        for line in range(0, long_fields):

        # for each line, split the list by the '|' separator
        fields = [x.strip().split('|') for x in fields ]

        # return the 2D list and the headers
        return fields, headers

Easy right? The comments should give you an idea what is happening in each step. The result from SAP is in text, so much of the function is just parsing the data.

So if you wanted to use your new class, you could do something like this:

# Init the class and connect
# I find this can be very slow to do... 
s = main() 

# Choose your fields and table
fields = ['MATNR', 'EAN11']
table = 'MEAN'
# you need to put a where condition in there... could be anything
where = ['MATNR <> 0']

# max number of rows to return
maxrows = 10

# starting row to return
fromrow = 0

# query SAP
results, headers = s.qry(fields, table, where, maxrows, fromrow)

print headers
print results

You can download the full class from here. I suggest you modify it and learn what each portion does before you do any testing on your own system. I've left out error checking and assertions for simplicity.

That's it for now. In the next segment, I'll show you how to write standard SQL and have it converted to the RFC format. In the meantime, ping me if you have questions in the comments.




Python and SAP: Part 1 - Connecting to SAP

First things first - you want to use Python with SAP, you'll need to install some stuff.

You'll need PyRFC - a module to call RFC_READ_TABLE from SAP (the function module you need to use to read data from SAP tables)

From the PyRFC website (https://github.com/SAP/PyRFC):

The pyrfc Python package provides Python bindings for SAP NetWeaver RFC Library, for a comfortable way of calling ABAP modules from Python and Python modules from ABAP, via SAP Remote Function Call (RFC) protocol.

Here are some instructions on getting this done. I had a terrible time with this. 

Key points from my install:

  • Make sure that you install the CYTHON distribution to match the version of pyRFC that you are using (in my case it was 19.2). You can search for the distribution by modifying the following link: 

  • SAPCAR.exe is a program that you'll need to download from the SAP service place. Put it in a folder for later.

  • Download the SAR file from http://service.sap.com/rfc-library and put it in the same directory as your SAPCAR.exe

  • I had to install the GCC binaries to compile PyRFC. You may not need to do this, but this link made for the most simple install: 

  • Once you extract everything, move the NWRFSDK folder to your C path to keep things simple and then:

    • Set SAPNWRFC_HOME=%PATH%;C:\nwrfcsdk\lib

    • move the contents of C:\nwrfcsdk\include to C:\nwrfcsdk\lib\include

Once you have everything installed, the test script provided in the link above is pretty straightforward. 

import pyrfc


conn = pyrfc.Connection(ashost=ASHOST, sysnr=SYSNR, client=CLIENT, user=USER, passwd=PASSWD)

If you get errors, check that you've followed everything in the link above to the letter AND make sure you have set your path variable correctly. 

The setup process is a complete pain, but totally worth it. Post a comment below if you have any questions. I'll try to help where I can.

- ab

Python Urllib2 from behind a proxy

The number of times I searched for this before understanding...

from urllib2 import ProxyHandler, build_opener, install_opener
  proxy = ProxyHandler({'http': 'http://domain\\user:password@proxy_address:port'})
  opener = build_opener(proxy)
  print "Proxy Connected\n"
  print "Proxy not Found\n"

This little snippet of code will let you use a python application behind a proxy to access external resources. It's very useful for using 3rd party APIs.





Python: Running more than one process at once

Running many scripts at the same time may not come up on a daily basis and may not even be a good idea, but here is a great little code snippet. I had a few scripts that could run in parallel to save time without too much of a performance hit (the scripts grab data from external sources).

from subprocess import Popen
files = ['file1.py',

threads = []
for file in files:
    t = Popen(file, shell=True)

[x.wait() for x in threads]


This code runs five files at the same time and waits for all five to finish before continuing... Lovely.

Enjoy !

Using a Redirect Manager is like Hosting a Guestbook for Despicable people

I maintain several websites for my work at shift8 and I recently put in place a piece of software to record redirects on my site. Each time someone typed in a url like "http://shift8solutions.com/[REQUEST]" my redirect manager would record the attempt and the IP of the user. The goal was to find broken links and maybe, just maybe, understand if people expected a page to exist that I hadn't created.

How naive am I?

The true result was a log of all the bungled attempts to gain entry to my site. This made me happy because 1. I love data, 2. they did not succeed and 3. I learned a ton. Here is a breakdown of my wannabe hackers by country (top ten offenders):


In general, the goal seemed to be direct access to phpmyadmin and various RSS feeds (that do not exist). There were also some very specific references to products that I do not own - most likely references to security issues in other software platforms. I recognized elements from Wordpress, Drupal, and Joomla. In general, it was fishing, not really hacking and it wasn't very sophisticated. I did learn a bit about what is visible to outsiders about my software choices.

For fun, using google tables, I made a heat map of the above data by country - Notice that most of these attempts were made from the US, but Argentina and most of Europe had activity as well.


It's bizarre to look through these logs - there is far more information tracked, but it wouldn't be nice to post it. I guess I should say thanks to these guys - it made for an interesting bit of research and my site is stronger for it.

Thanks goons!


Network Diagrams and Python Web Crawlers

Output 27k 600px 1000zoom invert I'm fascinated by networks and data visualization. I've always wanted to try my hand at making some of the inspiring images I see on blogs like flowing data. This network diagram is my first amateur attempt.

The code

I started by writing a rather simple web crawler in Python. The logic for the bot was:

1. Open a page

2. Create a list of all the links on that page (capture the total number of links)

3. For each link, create a new bot to follow the link and start the whole process again.

This was a great chance to use the Threading module in Python. I am not an expert in threading or multiprocessing. However, threading allowed me to create a new bot for each link I wanted to follow.

Here is the code for my spider class:


Created on Jun 13, 2012

@author: Alex Baker



import urllib2,BeautifulSoup,time

from threading import Thread


    def scan(self,url,mem, f):

        # Get the url

        usock = urllib2.urlopen(url)

        # Your current URL is now your "old" url and

        # all the new ones come from the page

        old_url = url

        # Read the data to a variable

        data = usock.read()


        # Create a Beautiful Soup object to parse the contents

        soup = BeautifulSoup.BeautifulSoup(data)

        # Get the title

        title = soup.title.string

        # Get the total number of links

        count = len(soup.findAll('a'))

        # For each link, create a new bot and follow it.

        for link in soup.findAll('a'):


                # Cleaning up the url

                url = link.get('href').strip()

                # Avoid some types of link like # and javascript

                if url[:1] in ['#', '/','','?','j']:


                # Also, avoid following the same link

                elif url == old_url:



                    # Get the domain - not interested in other links

                    url_domain = url.split('/')[2]

                    # Build a domain link for our bot to follow

                    url = "http://%s/" % (url_domain)

                    # Make sure that you have not gone to this domain already

                    if self.check_mem(url, mem)==0:


                            # Create your string to write to file

                            text = "%s,%s,%s\n" % (old_url, url, count)

                            # Write to your file object


                            print text

                            # Add the domain to the "memory" to avoid it going forward


                            # Spawn a new bot to follow the link

                            spawn = spider1()

                            # Set it loose!

                            Thread(target=spawn.scan, args=(url, mem, f)).start()

                        except Exception, errtxt:

                            # For Threading errors print the error.

                            print errtxt


                            # For any other type of error, give the url.

                            print 'error with url %s' % (url)


                # Just keep going - avoids allowing the thread to end in error.


    def check_mem(self, url,mem):

        # Quick function to check in the "member" if the domain has already been visited.



            return 1


            return 0

As you can see, the code is simplistic - it only considers the domain/sub-domain rather than each individual link. Also, because it checks to make sure that no domain is used twice

To run the class, I used something like this:

mem = []

f = open('output.txt', 'w')

url = 'http://justanasterisk.com'# write the url here

s = spider1()

s.scan(url, mem, f)

Once started, it doesn't stop - so kill it after a while (or build that in). Running this on my MacBook, I recorded 27,000 links in about 10 minutes.

The data

The number of data points is small in comparison to some of the sets I've explored using BigQuery or Amazon SimpleDB. However, I wanted to make a visualization and I realized that the number of pixels would really define how many data point were useful. I figured that 10 minutes would give me the structure that I wanted. I used my blog justanasterisk.com as the starting point. I won't attach the data (you can create that yourself) but suffice to say that each line was:

source, destination, # of links on source page

The visualization

Here is where I was out of my element. I browsed a few different tools and the best (read: easiest) solution for my needs was Cytoscape. It is simple to use and has several presets included to make you feel like you've done some serious analysis. For the image above, I used one of the built in layouts (modified slightly) and a custom visual style.


Screen Shot 2012 06 18 at 09 38 PM

Screen Shot 2012 06 18 at 09 39 PM

Screen Shot 2012 06 18 at 10 40 AM

I won't underwhelm you with further details, but shoot me an email if you want more. I'll probably add a few more images to this post when I get them rendered.



SMS Workflow Madness: Twilio to PHP to Python to Dropbox to Autohotkey to Conquer The World

Recently, I've been trying to trigger some python code using a text message. It has been a complicated little journey, so I thought I'd write it up for you. If you don't want to read through it all, the summary is - twilio to PHP to launch Python to put a file in Dropbox, autohotkey to monitor dropbox and run a python script. Away we go... First, Twilio is a great service if you want to develop anything with text messages. At first, I built a quick fix using If This Then That (which you should check out either way). However, I soon realized that the benefit of a text message is that it is nearly instant. IFTTT only checks tasks every 15 minutes and in a crunch, I would want a response back before then...

So I signed up for Twilio and created my application. The applications can be very complex, but for my purposes, I just needed a few lines of PHP to receive the text from the SMS and then use that information. Here is my test script:

header("content-type: text/xml");
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n";
$body = $_REQUEST['Body'];
$from = $_REQUEST['From'];
$path = "/var/www/cgi-bin/addfile.py";
$command = "python ".$path." '$body'";
$command = escapeshellcmd($command);
echo "<Response>
<Sms>Thanks for the message:".$body." your num:".$from." </Sms>

There's a lot going on there, but here is the gist. The first two lines format the document as XML for Twilio to understand what should be done. No surprises here.

header("content-type: text/xml");
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n";

The next part pulls the data from the text message into body and from and then passes these values to a python script I wrote to interact with dropbox.

$body = $_REQUEST['Body'];
$from = $_REQUEST['From'];
$path = "/var/www/cgi-bin/addfile.py";
$command = "python ".$path." '$body'";
$command = escapeshellcmd($command);

The final part is the xml. This I pulled straight from the Twilio getting started guide

echo "<Response>
<Sms>Thanks for the message:".$body." your num:".$from." </Sms>

Ok so now we have a file for Twilio to interact with. Next we need to put some content in that python file. Before you try this out, you'll need to install the dropbox api libraries. I used the command

easy_install dropbox

but you might have to do that differently based on your operating system.


# Include the Dropbox SDK libraries
from dropbox import client, rest, session
import sys

name = sys.argv[1]

# Get your app key and secret from the Dropbox developer website
APP_KEY = 'xxxxxxxxxxxxxxx'
APP_SECRET = 'xxxxxxxxxxxxxxx'

# Access type will be defined in your dropbox settings
ACCESS_TYPE = 'app_folder'
sess = session.DropboxSession(APP_KEY, APP_SECRET, ACCESS_TYPE)

# I removed this section after obtaining my access_token
# and access_token_secret, but you'll need to do it once.
# The return value will be a string that you can parse.
#request_token = sess.obtain_request_token()
#url = sess.build_authorize_url(request_token)
#print "url:", url
#print "Please visit this website and press the 'Allow' button.

access_token = "xxxxxxxxxxxxxxx"
access_token_secret= "xxxxxxxxxxxxxxx"

sess.set_token(access_token, access_token_secret)

client = client.DropboxClient(sess)
print "linked account:", client.account_info()

#create the file if it doesn't exist
#f = open('file.txt', "w")

#open it for reading only...
f = open('file.txt')
# put the file to the app_folder in dropbox
response = client.put_file('/'+name+'.txt', f)
# this is the response passed back to PHP for debugging.
print "uploaded:", response

The file above is a bit of a mess but the idea is simple, take an argument as the command, authenticate with dropbox and put a file in dropbox with that name. I've tried a few different ways to do this a Dropbox PHP class or two... The python script turned out to be much easier for me - perhaps you have had better luck?

So now, with all that lovely code above, when I send a text message to my twilio account number, the php file takes the SMS message as a command and launches the python dropbox script, putting a file with that command name in my folder. The last part is an autohotkey script that I have to monitor the app_folder (it's actually sitting in the app folder for simplicity). Here is that file:

setTimer check_file,1000

IfExist, command.txt
 filemove command.txt, %A_ScriptDir%\processed\command%A_Now%.txt
 run myprogram.py command

This script checks my folder for a file called "command.txt" and then if it finds it, runs a script and moves the file to a processed folder with a time stamp. It's not perfect, as it requires a separate "look" for each command that you want to run, but it was perfect for my needs.

So that's my system. It's not pretty and it has a few more steps than I'd like for efficiency and safety, but it does work. Fast. In fact, a text message can trigger a program on my remote machine within 10 seconds. That is not bad...

Let me know if you've tried something similar or have suggestions on improvements. I'd love to hear it.



shift8solutions.com -> Check. It. Out.

For a while now, I've been working on a warehouse management system (WMS) on my business website shift8solutions.com. As I'm finally done, I thought I'd share the good news with those who visit this site. Please take a look at shift8WMS and share with your friends! Shift8WMS is a user friendly, web based warehousing system for small businesses. The goal is to provide inexpensive but excellent products to small business owners. I think visiting shift8 will give you a more complete understanding so check it out!

All the very best,


SAPeedie... for free

I've decided to give away my portable SAPeedie app for SAP testing. It's a simple enough windows app, but below is the extract from the help file. I haven't spent much time on it recently, so please be aware that I'm not going to provide active support. However, I'll post answers to questions if enough people have issues, so feel free to share your experience... Here is the link (V2.2)

Sapeedie is a tool I developed to aid in SAP testing. The program simplifies actions that I found tedious and repetitive and includes the standard windows keyboard shortcuts that are missing in SAP.

Quick Shortcuts: Control + N - open a new session Control + A - select all (some text fields do not allow you to select all) ` - sends you to the command box directly (keeping your hands on the keyboard and avoiding the built in shortcut which is awkward) Windows + S - Sets the current window to stay "on top". All other windows will remain behind the tacked window(s). The same keys to remove this property. Alt + Right Button - a bit more convenient than that silly Ctrl + Y selection shortcut.

Advanced Features (the cool stuff): Control + Right Mouse Click - Opens a special menu that you can customize as needed. You can put favorite transactions in your txns.txt file and have quick access to them from anywhere in SAP. The same menu also provides access to the DocLog (more details on that in a second). Each file you click in the DocLog is transferred to your clipboard.

Control + Shift + Right Click - Captures the message on the screen from SAP. For example, if you create a PO, the message might be "PO 12345 created". When you control + shift + right click, the message is saved into your doclog.txt file, with the name of the transaction you were running, and the date/time. This will allow you to keep track of your testing - every time you press those keys (even if there isn't a number) the text will be saved. It's a sort of "this is what I did" log. Just remember to click!

Copy Cliboard to your DocLog - it's in the menu, but if you want to capture a large quantity of text, with a reference to where you got it, you can use this feature... (There's room for improvement but this allows you to bring ANYTHING into your doclog. Just copy it first.)

Mapping (made-up) Shipping Data

I'm reading Ben Fry's book Visualizing Data and having some serious fun with mapping. I invented some shipping data (from my home state of Ohio to the other 50) and then put it on a map of the US using some of Ben's example code. I didn't manage to get the animated version to work on the site, but it's still pretty... If you're interested, let me know and I'll pass along the code. Below is an example of the output. Love to hear your thoughts.


Shipping Map


Spotify control script for AHK

I found it rather annoying to reopen Spotify whenever I wanted to change tracks or increase the volume... Using Autohotkey, it's no longer a problem (I found some great examples via Google). Enjoy


; ; Spotify volume and track controls. ;


^!Space:: { DetectHiddenWindows, On ControlSend, ahk_parent, {Space}, ahk_class SpotifyMainWindow DetectHiddenWindows, Off

} return

^!Right:: { DetectHiddenWindows, On ControlSend, ahk_parent, ^{Right}, ahk_class SpotifyMainWindow DetectHiddenWindows, Off

} return

^!Left:: { DetectHiddenWindows, On ControlSend, ahk_parent, ^{Left}, ahk_class SpotifyMainWindow DetectHiddenWindows, Off } return

^!Down:: { DetectHiddenWindows, On ControlSend, ahk_parent, ^{Down}, ahk_class SpotifyMainWindow DetectHiddenWindows, Off } return

^!Up:: { DetectHiddenWindows, On ControlSend, ahk_parent, ^{Up}, ahk_class SpotifyMainWindow DetectHiddenWindows, Off } return

Stock widget...

I’ve been thinking a lot about dashboards recently and wanted to use processing to create a “stock tool” – something to grab real-time stock data for a company and display it in an interesting way. Take a look here

New processing project: Clock

Another Processing project - this one is kinda fun. It's a clock, created with code and some basic shapes. I think the coolest part is that I used "processing.js", a library that renders the result in an HTML5 format - you can see the animation on your iPad! (or iPhone) Check it out here

Here is an image of the output.


This applet uses random numbers (within some boundaries) to create the stems and petals.  If you press any key while it runs, it will stop. Each time will be different, so enjoy! Take a look at the java applet herehtml5 animation here and below is a sample of the output.