Azure Event hubs with Python

In this blog I will demonstrate how to use Azure Event hubs to send and receive messages using python.

Event hubs in the words of Microsoft are:

“Event Hubs is a fully managed, real-time data ingestion service that’s simple, trusted and scalable. Stream millions of events per second from any source to build dynamic data pipelines and immediately respond to business challenges. Keep processing data during emergencies using the geo-disaster recovery and geo-replication features.”

So to get started you’ll need a Azure subscription or you can sign up for a free account here

Once you’ve created your Azure subscription we’ll need to start by creating an Event hubs namespace, so firstly click on the Event Hubs icon or search for this in the Azure search bar.

AZURE SETUP

Once in the Event hubs click add and this will open the Create Namespace pane , in here enter the below:

Name: Anything, but this needs to be unique to Azure so test is not going to work

Pricing: Basic level is more than enough

Subscription:  This will be pre-populated with the subscription your logged in under

Resource Group: These are excellent for grouping together items which are related so you’re able to remove these all in one group

Location: Uk South, but you can pick whichever one is closest to you

Once this resource has been provisioned you’ll need to click into the namespace and select Event hubs

Then select to add a new event hub by clicking on the cross at the top and the below Create Event Hub pane will appear.

Simply enter a name for this event hub

Once this is provisioned click to go back into the event hub you’ve just created and select Shared access polices in the left pane under settings

Click Add and on the Add SAS Policy pane that appears add a name and click send and listen  and click create

Once this has been provisioned click into the Policy and click to copy the Connection string, paste this somewhere for later.

We’ve now completed the setup from an azure point of view, so onto the Python client and receiver part.

PYTHON SETUP

I’ve supplied these two python scripts in my github repo at the following link.

First we need to open the install the relevant python libraries so you’ll need to issue the below pip command in whatever command tool you use, bash or cmd Prompt

Next we need to open and edit the EventHubClient.py &  EventHubReceiver.py file and edit the following values in both files like below

Using the connection string you copied earlier, here is mine as an example to help you derive whats needed

Endpoint=sb://datadooreventns.servicebus.windows.net/;SharedAccessKeyName=AppEventHubAccess;SharedAccessKey=DP1lDaWxxcuWUZcNal78DTD50JQc0WNQU3lMz5j6/5I=;EntityPath=ddeventhub

ADDRESS = This is concatenation of the endpoint part of the string and the EntityPath

USER = Is the SharedAccessKeyName

KEY = Is the SharedAccessKey

Save and close both files and open a CMD prompt window from the directory where these files are located, and execute a following commands in two different CMD Prompts in this order.

BRINGING IT ALL TOGETHER

Then

If successful, you should start to see the data flowing between the two using the Eventhubs as its streaming queue.

Input

Output

You can then use this as an output for Stream Analytics, blob storage….the world is your data oyster 🙂

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s