​Probably no big surprise to anyone, I have changed the Sensor programs some more!  Enough to deserve a couple of new screen shots. 

Primary App, showing its Log, Graph window and Sensor details in the background HTML page.

Graph demo with days worth of data and a gap when it was not recording.

There are a fair amount of both Major and Minor changes, both front facing to the user and on the back end.  

Major Changes

• Put Graphing into its own Window + made the time selection more accurate
• Separated Sensor checks for data retrieval vs checking online
  • Faster Online Checks
  • Allows Different functions on same socket connection
• Added SQL database Sizes to Sensor Information HTML page
• Updated SQL queries to handle more data points for graphing
• Added Motion detection recording to Sensors (Separate database, checking 4 times every second (0.25 seconds), recording if its change more then 0.05 since the last recording (If any axis changes that much, it records all data))
• Added Graphing option for the Motion Database
  • Motion graphing bypasses the "Data Points To Skip"

Minor Changes

• Updated code to auto check selected sensors before trying to get data (database, current readings)
• Updated and cleaned up Log code (tracking progress through program)
  • Before writing new log data, remove all white space before and after text in the log window
  • Add "Relay" functions in order to return better log details from custom imports
• Updated some Variable and Function names to better match their actual function
• Broke up code into new def statements and used for statements to simplify code and make more readable. 
• About 5 or so other changes I can't quite remember…  

I spent a great deal of time working with the SQL queries, especially because I had to convert the time text from the program date selection, into something I could compare to from the database.  I could of saved a lot of time by making sure it printed out the 2 comparisons from the start all the time, but since I didn't… that was a good 10+ hours spent!  Its all sorted out now and it should be much easier to spot future errors in input calculations. 

While dealing with the SQL queries them selves, I quickly found out, that too many data points in a SQL database, will make getting, holding and manipulating the data in a single grab, a no go (Goes slow, if too many points, just doesn't do the graph).  Thanks to a misaligned threshold setting on my motion detection by 0.02, It was recording every few seconds…. sometimes more often.  It quickly added up to a 6MB database.  This gave me the chance to test my Graphing with large data sets, and it successfully graphed just shy of 160,000 data sets PER Time/X/Y/Z.  That's about 640,000 data entries!   Even on my Ryzen system, it took about 10 seconds to process that.  However, this is an unusual amount of data and most of my readings will never reach this level.  Its still comforting to know I can process that much, as it will be useful for reviewing long term data.  It may also become necessary to compare multiple sensors in a single graph, which would multiply the data entries by however many sensors you needed to compare.  

Here is how smokey it looks out side my window… I wish I could say that's clouds or mist, but its smoke at about 11AM in the morning on a Sunday…. Terrible terrible smoke…

Super Smoke

So I have not quite got the motivation to do some soldering for the Air Quality sensor.  So instead I have been tweaking the program and thinking about future endeavors with it.  To that end I am figuring out how I may utilize the already built in accelerometer to track things like tremors or earth quakes.  looking at the readings, it doesn't get very accurate (goes to like 9 decimal points, but sitting still makes variation in as much as 0.01). 

The seemingly not so accurate readings may be literal movement (cars, construction, etc)  in the area I am in, but I won't know for sure, until I take it somewhere more… stable and less populated.  Depending on what I hope to track, I'll change how many decimal points I record (hoping that it won't record anything until something happens, saving resources).  This may come in handy later, if I for instance, put a sensor in a tree with battery and solar.  Should a squirrel decide to touch, nibble or throw the sensor down, It could record the event.  With a Internet connection of some kind, I could be notified, so the sensor doesn't stay offline or in the wrong place for too long. 

In either case, that will be done later, as I need to see what kind of resources and space these readings will take. I tried a few things that didn't work due to SQL stuff last night, but the rest of the code is basically done.  I need to figure how, why, when and where I'll record the data.  Due to the constant writes, I may create a whole new database, so database downloads don't take too long over time.  

Other small changes done include.

• Making the RGB values/lines on the graph Red / Green / Blue
• Added TextBox for Custom Sensor name in Graph (makes it easier to look over, as "Kootenay Lake North" then "Sensor2")
• Changed graphing from 2 columns to 1 (larger area for the timescale)  and put RGB beneath Lumens (compare raw light value to colour spectrum).  

I also uploaded a HTML page of my "Back Yard Shed" sensor to my site You can check it out here.  It's about 8 days worth of sensor data.  

http://dragonwarz.net/temp-plot.html

So I mentioned last post I was probaby going to use a pre-made HTML generator, but turns out I can get my own to work without too much trouble.  I now have it building the HTML file on the fly, by only adding sensors that check as online.  

Here's a few screen shots of the changes.

This first one shows the HTML page dynamically made in the background.  It groups them in 3 per row and basically unlimited columns.  I re-designed the PC app, in order to allow more sensors, and let the HTML page + Graphs handle the data.  When you "Check Online Status" of sensors, it auto unchecks any that are found to be offline, so you don't have to wait a long time for all the sensors to timeout in later operations.  I have also set the timeout to 2 seconds when checking sensors.  

I'm using plotly for plotting now, and its SOO nice!  As you can see above, you can select the timeframe and how many data recordings to skip before plotting (It asks to open a sqlite3 database file, which you download off the senors).  This allows for flexible viewing for both details and timeframes.  Thanks to plotly, this all looks super nice too!  Through plotly, the graph is actually a HTML page, of which you can do other tasks like save it as a pic, upload to their online editor and more!  

There are some other things updated too, but this is the main stuff.  I'll probably prioritize attaching a air quality sensor to the hardware now, as its been REALLY smokey out here the past 3 years or so due to soo many forest fires.  I hope to see just how bad the air is during these times and others.  

So, I was going to use Django… until I realized I can actually just use a HTML template and have python itself edit the variables, and spit out a new copy.  So that's what I did.  I will probably still go to some kind of auto HTML generator, so I can add sensors on the fly more seamlessly, but this is a good start to get the rest of the back end up to date.  I'm also changing a few things to be more efficent, like using 'for' and 'while' loops where I used to have mutliple code for each sensor.  

Here are a pic of the progress.  

GUI App beside the HTML output file.  Its opening the html doc locally, but it will be on the Sensor "Controller", accessed through a http server.  

I'm pretty happy with the progress.  I'm now waiting on some hardware to arrive, so I can get started on making the Sensors self sustaining for power.  

I have also done a lot of other updates since my last post.  Here's a quick point view.

• Updated storage method from text file to sqlite3 database
• Updated *Most code to reflect new storage method
• Created install script for new sensors -> One command you run in terminal and in the end, you just change a IP and your done!  New sensor ready to go. 
  • Still need to enable or set ic2 / hostname / wifi country
  • Optional: Enable SSH / Set boot to CLI autologin 
• Updated Sensor "Controller" to enable dual button modes
  • If the same button is pressed again within 5 seconds of the command finishing after the original pressing of the button, it will do something different
  • Button Options 
    • 1a) Display Local Host Info + Age of Update Files on Controller
    • 1b) Update Local programs and Sensor upgrades off Local NAS 
    • 2a) Display Sensor Online & UpTime OR Offline
    • 2b) Download Sensor's Database to local Controller
    • 3a) Upgrade Sensors with progs from Local Controller
    • 3b) Reboot Sensors
    • 4) Shutdown Local Controller
• Updated script for Controller to install OR update the unit (Can auto detect)

That's it for the most part.  I have also been hanging out on the Raspberry Pi IRC channel, and getting some decent ideas from there.  

Until next time!