Category Archives: programming

Program License – GPL

I mentioned before that I would post my code, which I will indeed post, but the more I thought on it, the more I realized I should probably choose a license model for the program now, before actually posting any code. 

To this end, I have decided on the GNU GPL Version 3.  I am 100% for sharing and making sure it stays free for anyone who thinks its useful and wants to use it.  If it ends up being used by a company with money, I would sure appreciate funds to continue development and get a few research stations going.  I'm also very much open to adding pretty much any sensor to the units, if you send me a few for testing. 

I'm going to work on the Installer and install scripts for the next few days, so that people can easily get this up and running to try out.  Of course the hardware is still needed to be effective, but I'll be sure to include a section in the Help Menu of the PC program, that will help you make your own sensor unit. Ultimately I would like to give people the ability to digitize the world around them easily, to find out all kinds of Cool stuff.  You never know what you'll find after visualizing the data in different ways.  

I'm thinking I might have something usable by the beginning of September, 2018.  That doesn't mean bug free or pretty, but it will be usable! 

I have already spent a bit of time on the PC installer.  I have been updating my hard coded locations in the program to system variables, so it can be installed and run from any windows computer (Will have to double check if the system locations work in Linux too).  It has just passed being usable on a different computer after running the installer.  The installer puts a folder of Python 3.7 in the app directory and the shortcuts link directly to it, running the script.  The sensor installers are already working fairly well, but I would like to customize it to ask for certain variables upon running it, so its a one stop shop to configure and install.  

I'll post the update log in a later post.  

Enhancements / Plans / ChangeLog

Since I do lots of testing on the fly with multiple sensors, I thought it was about time to add some functionality to the program around sensor control.  

Up until now, I have used SSH for 90% of my sensor interaction, but there is really only 2 things I do a lot of… Update the programs and reboot the sensor.  So to that end, I have added a "Sensor Commands" section to allow just that and more.  Essentially, you check the sensor('s) you want to do something with in the main window, than click a command button like upgrade, reboot or shutdown.  It will send the command to all selected sensors.  

I added a SMB update (Windows shares) and a Online HTTP update (allows upgrading remote sensors with internet access).  I added a checkbox for the shutdown, so I don't accidently click it during quick testing.

I have also created a Windows Installer for the program, which includes a install of python 3.7 with the required libraries.  I originally just created a single .exe using pyinstaller, but Plotly didn't seem to like working with it, so I thought the next best thing was to just make the install easy.  It also runs much faster this way!  

Some of the things I hope to add in the near future include.

  • Put phpSysInfo on Sensors, using lighttpd server for system diagnostic Information
  • Update output files to local user desktop
  • Improve Graph outputs and Options at generation
  • Have update script terminate programs at the end, so the updated versions can start with improved crontab autostart
  • Improve Install scripts for ease of use
    • Ask and auto set Hostname, IP, Wifi, etc
  • Have program write config to file, and load it on boot to Include
    • IP List for Sensors
    • Default Date ranges in Graphs

I spent some time going over the back end code, and thought it best to merge a few things.  So here's yet another changelog.

ChangeLog Aug 15th, 2018

  • Merged 2 .py programs into one for socket communication
    • Merged the live sensor data send with general socket commands
  • Improved logging
  • Added Time corrections to Motion Graph (timezone)
  • added Sensor command "Terminate Sensor Programs"
  • Changed Auto Start method from rc.local to crontab scripts.  
    • Auto runs script every 1 min
    • The script checks to see if the program is running
      • If not running, start it
      • If running, do nothing
      • Allows crash recovery and applying remote updates without restarting the unit
  • Updated E-Ink Controller with Static IP 
    • Updated E-Ink & Sensor update scripts to match new IP
  • Updated Motion Database to save IP for better graph Sensor Identification 
  • (35% done) Started creating a Database "Splitter / Merger / Updater" program to allow
    • Modifications of old databases to newer models or merging with additional fields 
    • Split databases for Later analysis or archiving (X timeframe is in Y location, take just that timeframe and store in new database for later access). 
  • Updated 'Update scripts' 
    • Improved Speed (less network copies)

Changes and Improvements

​Probably no big surprise to anyone, I have changed the Sensor programs some more!  Enough to deserve a couple of new screen shots. 

 

 

There are a fair amount of both Major and Minor changes, both front facing to the user and on the back end.  

Major Changes

  • Put Graphing into its own Window + made the time selection more accurate
  • Separated Sensor checks for data retrieval vs checking online
    • Faster Online Checks
    • Allows Different functions on same socket connection
  • Added SQL database Sizes to Sensor Information HTML page
  • Updated SQL queries to handle more data points for graphing
  • Added Motion detection recording to Sensors (Separate database, checking 4 times every second (0.25 seconds), recording if its change more then 0.05 since the last recording (If any axis changes that much, it records all data))
  • Added Graphing option for the Motion Database
    • Motion graphing bypasses the "Data Points To Skip"

Minor Changes

  • Updated code to auto check selected sensors before trying to get data (database, current readings)
  • Updated and cleaned up Log code (tracking progress through program)
    • Before writing new log data, remove all white space before and after text in the log window
    • Add "Relay" functions in order to return better log details from custom imports
  • Updated some Variable and Function names to better match their actual function
  • Broke up code into new def statements and used for statements to simplify code and make more readable. 
  • About 5 or so other changes I can't quite remember…  

I spent a great deal of time working with the SQL queries, especially because I had to convert the time text from the program date selection, into something I could compare to from the database.  I could of saved a lot of time by making sure it printed out the 2 comparisons from the start all the time, but since I didn't… that was a good 10+ hours spent!  Its all sorted out now and it should be much easier to spot future errors in input calculations. 

While dealing with the SQL queries them selves, I quickly found out, that too many data points in a SQL database, will make getting, holding and manipulating the data in a single grab, a no go (Goes slow, if too many points, just doesn't do the graph).  Thanks to a misaligned threshold setting on my motion detection by 0.02, It was recording every few seconds…. sometimes more often.  It quickly added up to a 6MB database.  This gave me the chance to test my Graphing with large data sets, and it successfully graphed just shy of 160,000 data sets PER Time/X/Y/Z.  That's about 640,000 data entries!   Even on my Ryzen system, it took about 10 seconds to process that.  However, this is an unusual amount of data and most of my readings will never reach this level.  Its still comforting to know I can process that much, as it will be useful for reviewing long term data.  It may also become necessary to compare multiple sensors in a single graph, which would multiply the data entries by however many sensors you needed to compare.  

Here is how smokey it looks out side my window… I wish I could say that's clouds or mist, but its smoke at about 11AM in the morning on a Sunday…. Terrible terrible smoke…

Super Smoke

Tweaking Town

So I have not quite got the motivation to do some soldering for the Air Quality sensor.  So instead I have been tweaking the program and thinking about future endeavors with it.  To that end I am figuring out how I may utilize the already built in accelerometer to track things like tremors or earth quakes.  looking at the readings, it doesn't get very accurate (goes to like 9 decimal points, but sitting still makes variation in as much as 0.01). 

The seemingly not so accurate readings may be literal movement (cars, construction, etc)  in the area I am in, but I won't know for sure, until I take it somewhere more… stable and less populated.  Depending on what I hope to track, I'll change how many decimal points I record (hoping that it won't record anything until something happens, saving resources).  This may come in handy later, if I for instance, put a sensor in a tree with battery and solar.  Should a squirrel decide to touch, nibble or throw the sensor down, It could record the event.  With a Internet connection of some kind, I could be notified, so the sensor doesn't stay offline or in the wrong place for too long. 

In either case, that will be done later, as I need to see what kind of resources and space these readings will take. I tried a few things that didn't work due to SQL stuff last night, but the rest of the code is basically done.  I need to figure how, why, when and where I'll record the data.  Due to the constant writes, I may create a whole new database, so database downloads don't take too long over time.  

Other small changes done include.

  • Making the RGB values/lines on the graph Red / Green / Blue
  • Added TextBox for Custom Sensor name in Graph (makes it easier to look over, as "Kootenay Lake North" then "Sensor2")
  • Changed graphing from 2 columns to 1 (larger area for the timescale)  and put RGB beneath Lumens (compare raw light value to colour spectrum).  

I also uploaded a HTML page of my "Back Yard Shed" sensor to my site You can check it out here.  It's about 8 days worth of sensor data.  

http://dragonwarz.net/temp-plot.html

 

More updates!

So I mentioned last post I was probaby going to use a pre-made HTML generator, but turns out I can get my own to work without too much trouble.  I now have it building the HTML file on the fly, by only adding sensors that check as online.  

Here's a few screen shots of the changes.

This first one shows the HTML page dynamically made in the background.  It groups them in 3 per row and basically unlimited columns.  I re-designed the PC app, in order to allow more sensors, and let the HTML page + Graphs handle the data.  When you "Check Online Status" of sensors, it auto unchecks any that are found to be offline, so you don't have to wait a long time for all the sensors to timeout in later operations.  I have also set the timeout to 2 seconds when checking sensors.  

I'm using plotly for plotting now, and its SOO nice!  As you can see above, you can select the timeframe and how many data recordings to skip before plotting (It asks to open a sqlite3 database file, which you download off the senors).  This allows for flexible viewing for both details and timeframes.  Thanks to plotly, this all looks super nice too!  Through plotly, the graph is actually a HTML page, of which you can do other tasks like save it as a pic, upload to their online editor and more!  

There are some other things updated too, but this is the main stuff.  I'll probably prioritize attaching a air quality sensor to the hardware now, as its been REALLY smokey out here the past 3 years or so due to soo many forest fires.  I hope to see just how bad the air is during these times and others.  

 

 

Updates & Changes

So, I was going to use Django… until I realized I can actually just use a HTML template and have python itself edit the variables, and spit out a new copy.  So that's what I did.  I will probably still go to some kind of auto HTML generator, so I can add sensors on the fly more seamlessly, but this is a good start to get the rest of the back end up to date.  I'm also changing a few things to be more efficent, like using 'for' and 'while' loops where I used to have mutliple code for each sensor.  

Here are a pic of the progress.  

GUI App beside the HTML output file.  Its opening the html doc locally, but it will be on the Sensor "Controller", accessed through a http server.  

App & HTML output

I'm pretty happy with the progress.  I'm now waiting on some hardware to arrive, so I can get started on making the Sensors self sustaining for power.  

I have also done a lot of other updates since my last post.  Here's a quick point view.

  • Updated storage method from text file to sqlite3 database
  • Updated *Most code to reflect new storage method
  • Created install script for new sensors -> One command you run in terminal and in the end, you just change a IP and your done!  New sensor ready to go. 
    • Still need to enable or set ic2 / hostname / wifi country
    • Optional: Enable SSH / Set boot to CLI autologin 
  • Updated Sensor "Controller" to enable dual button modes
    • If the same button is pressed again within 5 seconds of the command finishing after the original pressing of the button, it will do something different
    • Button Options 
      • 1a) Display Local Host Info + Age of Update Files on Controller
      • 1b) Update Local programs and Sensor upgrades off Local NAS 
      • 2a) Display Sensor Online & UpTime OR Offline
      • 2b) Download Sensor's Database to local Controller
      • 3a) Upgrade Sensors with progs from Local Controller
      • 3b) Reboot Sensors
      • 4) Shutdown Local Controller
  • Updated script for Controller to install OR update the unit (Can auto detect)

That's it for the most part.  I have also been hanging out on the Raspberry Pi IRC channel, and getting some decent ideas from there.  

Until next time!  

Converting my Code to a WebApp

I'm converting my program to a webpage!  Using Python + Django.

Since I'm a Network Admin, I know keeping programs up to date can be annoying.  A Web app on the other hand, can be updated once and its everywhere!  Not to mention the compatibility of just needing a new'ish web browswer to run the thing.  

Well, here it is after about 4 days, learning what I needed, how to use what I needed and actually making the program.  Here's a screen shot.

WebApp1

I have it displaying multiple sensors, which the readings get updated every time you goto the page. Most of it is self explainitory, although the funny dashes and colour after "Colour: ", is the actual colour representation of the RGB light sensor.  

Now I have to bring over the other stuff like updating sensors, rebooting devices and downloading the longer term sensor logs.  

 

 

 

Using Python3 to collect and manipulate SenseHAT Sensor data

So, as I said in my last post, I want to digitize the world around me, starting with the sensors on my Raspberry Pi 3B+ and SenseHAT attachment.  

After much thinking, writing down ideas and generally trying things out, I finally got things working the way I wanted for the most part.  Here's what I have so far.

1.  Sensor Units: These are the Pi's with the SenseHAT and have the following programs running to enable the features I want (Only the HTTP server is not done by me, I just installed one). 

  • Sensor Readings to File: A simple program that reads "all" the sensors and writes the results as a string, each value separated by a ",". It creates one line each time its run.  It auto runs every 5 min through cron (Linux scheduler).  You could also do something similar with Windows "Task Scheduler".
  • HTTP Server:  The HTTP server just lists and allows downloads of the sensor data files. 
  • SenseHAT Joystick Controls: A small program auto runs with the pi and allows you to use the 5 "button" joystick to run commands and display info on the 8×8 LED grid.  It currently does the following, which helps a great deal when there is nothing attached but the SenseHAT and power.  I also have VNC / SSH enabled on the Pi's to connect remotely.  
    • 1. Display IP
    • 2. Display CPU Temp
    • 3. Display SenseHAT Temp
    • 4. Shutdown
  • Sensor Data over WiFi: Another small program runs automatically and listens on the network for my client app to request data.  

2.  Portable Sensor Data Display and File Downloader:  This is a PI with a 2.7" E-Ink display with 4 buttons.  It simply runs one program on boot to do the following.

  • Show it's own system information, including IP
  • Display Current Sensor Readings
  • Download Sensor Data Files
  • Shutdown

3.  PC: I made a more in depth GUI app for the PC, that can connect remotely to the units to gather information.  It has a lot more options, and is a bit easier to show a picture. 

Its still a work in progress, but as you can see, I can already compare Sensor readings over time.  I can also specify how many entries to skip when graphing the data files, as there should be about 288 entries for a day, after which, the program creates a new file for that day.  Plotting 288 points can not only take a bit of resources, but also isn't needed for most things being looked at. So I set it at 12 by default, so it shows a reading every hour.  

All these separate systems, apps and scripts connect through a mini router I got for $35 that's powered by USB.  This enables me to take the E-Ink pi and the mini router, power them both up off a USB battery Bank and wait about 2 min for the Units to boot and the Sensor Units to auto connect to WiFi, and I can then pull data remotely from the units, up to however far the router's and pi's range are (50-100 meters?).  Great for putting Sensors on tree's, in water, down a hole, etc, etc.  

I also enabled a OpenVPN server on the router itself, allowing me to connect through the internet and grab the sensor data with my PC program.  This can be done, even from remote sites through 3G or 4G connectivity (AKA cell phone service). Simply add a USB dongle for 3G / 4G to the router, and BAM, I can suddenly connect a lot of units, within a decent range to the internet for remote monitoring and maintenance!  

That's pretty much were I am at now.  I'm also working on other aspects like Pi Cases for weather proofing or even water and underwater deployment.  High altitude balloons are also in the future.  

I think that about does it for this post.  I'll eventually post my Python code too, but I wanted to clean it up and add some comments to explain things first (being my first Python project and all). 

 

 

A Raspberry Pi 3B+ Python Project!

I now have a use for this domain 🙂  My Python + Raspberry Pi development. 

So, about a month ago, I got a few raspberry pi 3B+ with 2x senseHAT's.  Long story short, the senseHAT is an attachment that is loaded with sensors, a 5 'button' joystick and 8×8 LED grid.  By making your own programs, you can access the hardware and get readings or have the LED's light up in any way you like, including animations.  

The raspberry pi peaked my interest for a few reasons.  

1.  They are pretty cheap at about $50 Canadian.

2.  They are small and power efficient.  AKA I can power them for a long time (few days) off a USB battery bank.  

3.  Decent support to help learn programming.  Check out their website www.raspberrypi.org

However, I was more interested in what kind of environmental data I could digitize through the sensors.  AKA more accurate local weather type information.  Before I could start however, I needed to think of how I could get and store information from the sensors and ways I could manipulate that data to find cool patterns (like when its the coldest time of day, and how and when that changes over days, months, years, etc).  

I have tried a few times to learn programing (C++, Java), but ultimately failed the self taught way, due to a lack of direction to assist with learning (AKA motivation).  But now, for about $150, I can get the pi + senseHAT to do a lot! 

I have actually come pretty far already considering I only got them a month ago, but I'll post about that later.  

This site will be active! 

w00t!