Monthly Archives: September 2018

Reworking write to DB Code (again)

So I had the Trigger database write code all working with variance triggers for the Magnetometer, Accelerometer and Gyroscope.  However, I noticed the datetime for writing was only accurate to 1 second.  Triggers are done in ms not sec.  It took me a bit to figure out how to format the strftime in SQLite3 properly (compounded by another error when I changed how the DB is checked). 
Long story short, it now records trigger events down to 3 decimal points of a second.  This will be perfect, since I usually do things no lower than 100ms.   

I have made both the Trigger DB AND the Interval DB into continually running programs instead of one off runs, so it doesn’t have to load up all the code every time it writes to the database.  This can have a big impact on how long it takes to get the sensor readings, which is important for the time sensitive Trigger code.

I was thinking of getting the datestamp for writing into the database, after the sensor readings are returned from the hardware sensor, but then I have to use python to find and convert the timezone off the Sensor itself into the nice generic UTC 0… But i’m not sure how to Dynamically get the timezone off the System to do the conversion (Not to mention, that alone may add more delay between getting the readings and the timestamp).  So i’m letting SQLite3 itself insert the time when it writes to the DB, as it always writes in UTC 0 if you don’t specify. 
This isn’t super accurate, since it has to go through a few functions before it can write, but it is a relatively small difference for when the DB is written vs when it got the sensor readings. I tested the time difference by printing the system time to the screen, just after each Trigger reading, and comparing it to what it is in the database.  It seems like the accuracy is within 200ms on a Raspberry Pi 3B+.  

In other news, I found out how to disable Resizing the window using GUIZero and created prompts for when actions are complete, such as downloading Databases and sending commands to the sensors.  This especially helps when the network is not so great and you get longer wait times and or network timeouts.  

“Trigger” database code

So I spent a good portion of today, re-working my database code.  When I started, I realized my sensor modules ALL get loaded, regardless if they are used or even installed.  For my code, it’s pretty small, but the sensor’s imported code will add up, especially as more sensors are supported.  So I found a way to load them on demand AFTER checking to see which sensors are installed.  

To make this happen, I turned each sensor module, into a class and moved the import into the class initializer. AKA, it only imports when a instance of the class is made.  This also makes it a bit easier to follow the code, as each instance has all the needed functions.  Due to the new simplicity, I was also able to remove 2 other .py modules.  I’m also finding that a lot of my modules are getting smaller and smaller, most are no more than 200 lines of code now, with a lot of it being the license at the top and setting up the logger.  So that’s helping with code management. 

The Trigger code has all the basics EXCEPT the actual variance checks (AKA the triggers).  I’ll probably have that done tomorrow, but in the meantime, I wanted to build up some database readings, so I made it write all the trigger readings to the database every 5 min and updated my sensors with it.  

It seems like every time I get something working, I have another 3 things I want to add!  My todo list is getting pretty long, and much of it, is no simple task to do (relatively speaking … probably take days or weeks).  If I’m not adding a new feature, I’m refactoring the code!  In fact, I seem to be refactoring more then adding functions now, but that’s mostly because I’m learning as I go, and start implementing better ways of doing things as I learn them.  A few examples include the new logger, using classes, making better logic loops and naming things to make sense, based on what they do or hold.  

As always, I’m happy with the progress that’s being made.  If not in the program, in my learning how to program ^_^

I still like looking back at my old code and seeing how its changed.  I should really remember to take screenshots of the program’s progress, to keep a visual record of its development. I found a pic of the program, near its initial development, and I remember how excited I was at the time, that I was able to make something like that.  The program(s) have come a long way since then (2 months ago), but because I’m excited about the progress, not the product, my excitement and motivation level is remaining fairly high (when I have had enough sleep).  

I think that’s enough programing for today.  Until next time!

Side thought of usefulness

With the recent changes to modularize the sensors, you don’t NEED to add a sensor on the Raspberry Pi to get started.  Choose the Raspberry Pi as your only sensor in the sensor config file, and you can keep track of System up time and CPU Temperature over time.

This will make testing the program as a whole, a LOT easier and a bit cheaper. This could also be handy for tracking the stability of a Raspberry Pi. If you have a Pi that’s rebooting or freezing up, this would help you track down the real culprit, by giving you a timeframe to narrow the search.  Combine that with a CPU Temperature reading accompanying the timeframe, and you can see if the unit is having overheating issues! You can also view multiple Pi’s real time stats in a html table for easy comparison.

This gives me another idea for a Option in the program.  I could add a entry in the Sensor config file, to allow “Extra System Information Recordings”, which would record things like CPU, RAM and disk usage.  By enabling this, you could literally use the Sensor Programs to truly monitor the health and stability of your Raspberry Pi’s.  It can even be enhanced to allow other systems fairly easily, like Windows and Mac.    

Of course I never designed them to be used for this scenario. I would have to think about security around the socket connections if there’s a good chance the Sensors are on a open network.

On another related topic. While talking with someone about the Sensors, they suggested I add a realtime graphing feature, to be able to constantly monitor things like Humidity for gardeners.  I think I’m about ready to take that on.  I’m not 100% sure how to pull it off just yet … What I should really do, is get a plotly dash app working on the sensor itself, so then I just have to pull up a URL based on the IP of the sensor.  But then you can only do one at a time, and that’s not very effective for bigger operations… I’ll have to think on this one more.

In other news, I’m feeling pretty good with the state of the Control Center App.  I have improved a bunch of the error checking and logging, so it’s harder to make the program “mess up”.  I should probably work on notifying the user when things go wrong though.  The program adds a error log message when things go wrong, but nothing extra; AKA, if a graph fails for example, it would look like it didn’t do anything.  Putting up a GUI OK button message with the error would be good, I just need to be careful to only show it once, so it doesn’t pop up a few thousand times for things like a invalid SQL entry, or maybe create a counter, then at the end, it pop’s up a single message if the counter > 0, and list the count in the message, so you know how many errors were encountered.  That could be useful… A few things to consider.  

New release Alpha.16.1

I have been working on polishing up the small things for this Alpha release, both in the Raspberry Pi sensor code and the control center code.

I have improved the pi sensor installer and upgrade scripts, both of which seem to be working well now.  I have tested it out a few times on my main sensor, from scratch and through upgrades.  After that, I upgraded all the units I have running right now, which there are 9. Things seem to be running smoothly, so I have decided to push it to the master branch.  I have toned down a lot of the logging messages from info to debug, so only errors show for the most part, and that looks pretty nice.  I’m debating putting sensor Online checks into debug if they are in fact online and only log offlines.  

Anywho, this release is looking good!  Shortly after merging, I changed a few more things to help out, such as saving the IP’s to the config file and loading on startup.  I also found and fixed a bug in the graphing datetime conversions, where it needed to apply the inverse conversion for getting the SQL data, then a normal conversion for what it SHOWS on the Graph.  It now lines up the proper time perfectly!  I also realized I was only graphing by date, not datetime, so I changed that to be specific to the second.

I probably won’t merge the new changes to master, until I get the trigger Database recording AND graphing working.  I have done a bunch of ground work, so there’s not a lot more I need to do to get the Sensor recording part working, but I will have to put a bit more time into the graphing from SQL … 

I have updated the installers, database demo’s and even posted install instructions for making a sensor (very basic).  So feel free to try things out!  Keeping in mind, it’s a Alpha release, so things break between releases and there will probably be a few bugs.  That being said, it seems pretty stable and usable to me!  

I hope to next release or the one after will be a Beta release.  Once the Trigger Database is working, I’ll be pretty happy with the feature set to release a first version, trying to work more on bugs for a bit.  Then I can start adding more features, but I’ll probably need new types of sensors to get really creative.  Not to say there is not a rather large list of features I could add, I just want to get a stable foundation before adding more on top. 

That’s it for now.   Slowly getting closer to a stable release! 

Tweaking & RP Sensor Installer

I have spent the past few days just going over the code and tweaking it.  Fixed a few bugs, refactored a bunch o code and even started on the “Help” section for building a sensor!  Of course at that point, I realized I needed to work on my installers for the RP Sensor code itself.  So that’s what I did. 

Any additional runs AFTER install, simply opens the config files, so you don’t get duplicated network code from the installer.  This also makes it easy to re-configure your sensor, simply by re-running the install code.  I have also copied both the Upgrade and config_edit scripts into the pi user’s home folder, so when you login with SSH or open a terminal, you can go right to upgrading or editing config.  

The Help section for building your own sensor is… Basic at best right now, but I wanted at least SOMETHING there to get started.  As I was making it, I went over the shell scripts for installing the Sensor code.  I managed to make it, so you only have to run a single command in terminal, and then it starts.  This command includes the download itself, and how it works, is this.  The command tells wget to get the install script off my http server, then tells bash to execute that script, after adding the execute permission to it.  This install script downloads all the other needed files off my http server and puts them where they should be, after which, it updates the Raspbian system itself, and installs all necessary dependencies through pip3.  During the process, it auto fills and opens up necessary config files to edit (Installed Sensors, IP, Wireless), with instructions built in as comments.  After its done, it automatically reboots the Raspberry Pi.  When the Pi boots back up, it should auto connect to whatever wifi was entered and start recording sensor data every 5 minutes to a SQL database.  By putting your computer on the correct IP subnet, you can also connect to the sensor with the KootNet Sensors – PC Control Center program.

I will be merging these updates and changes with the master branch and creating new installers and demo stuff soon.  I’m taking a bit more time to test things out, find and fix a few bugs and improve log output. 

Graphing for Interval Database’s has been Upgraded!

Yay!  Upgraded the Graphing abilities & completely Redid the back end code. It now allows the user to choose what sensors to graph through a series of checkboxes.  Creating Data Classes REALLY helped with this. 

This should set the stage for future additions of sensors, as I can add on more, without having to worry about a lot of edits in other functions.  Thanks to the new Data Classes, it should also be fairly easy to integrate new graph types (pie, bar, trace, etc).  So that will be coming later. I’m pretty happy about this, since I have been delaying this upgrade for awhile, due to all the other code stuff being learned & applied.

I suppose this free’s me up to do one or more of the following. 
A) Upgrading Logging on other aspects, such as the HW Sensor Software
B) Upgrade and Enhance the other HW Sensor Software in General
C) OverHaul the Trigger Database recording code
D) Work on the Sensor HardWare itself

Improving the Sensor Hardware has been another thing I want to work on, specifically the getting them to be self sustaining for power, so I can leave them in the wild to collect data.  I have mostly been waiting on batteries to use with a power system + Solar.  I have yet to find a way to connect my bigger USB Battery banks up to it, so it can charge AND use it.  I’m kinda stuck with these much smaller Lithium Ion batteries (Should be here this week).  I’ll have to see how long they can run for, and how good the solar has to be to keep them going… and think of a backup for when the sun don’t shine.

Sometime after getting the batteries and solar working, I’ll need to figure out some protective cases, since there are some pretty good storms that come through, not to mention animals and rodents doing who knows what.  I found a site that uses more common items, like random pipe and plastic containers. I’ll probably attempt that when I know the dimensions of the sensor’s and their components. 

The Python Class

Ahh, the Python Class, and I don’t mean a Classroom full of students.  

I found a EXCELLENT YouTube channel for explaining aspects of Python.  In fact, it’s the same place I found the logger videos.  I didn’t realize it was the same person at first, but once I did, I bookmarked it!  

His name is Corey Schafer’s and his videos are located at the following link. 
https://www.youtube.com/user/schafer5/playlists

Now that I have learned the basics of a Class, I’m starting to use it already in my overhaul of the graph section of my Sensor Program.  It should clean up my code very nicely, because I can forward one class object to the graph function, and it will hold ALL the needed data!  Right now, I have been sending each needed variable to each function, which had up to 6 variables!  It wasn’t pretty, that’s for sure.  However, with a data class, I can store everything that’s needed, pass the one class instance, and BAM!  Nice looking code, pretty much unlimited variables, and I can create as many instances as I want, with very little code!  It was even nicer when I realized I can add new variables (or function) to it, without having to adjust every function that uses it.  All very good things ^_^  

I have to admit, I have not actually added any new functionality to the program at all, but I have been learning a lot on how to improve the existing code and set the foundation of future enhancements.  Add on to this, GitHub and PyCharm, and the management of the code has also greatly improved!  

So although not adding thing to the program seems to be bothering me, I know it’s for a good cause and will help much more in the long run.  So I guess overall, I’m feeling pretty good with the progress.  

On another side note, my “step dad” / “Friend” / “something?” … not sure what to call him, but he’s awesome and knows a lot around chemistry and science.  Anyway, I asked for a bit of input from him about what sensors I might add and what I might be able to figure out with long or short term readings of said sensors.  He’s going to give it some thought and get back to me.  So I’m excited about that!  I’m hoping it will lead to practical use of my project to find out something new or cool!  

Logger change Done!

Yay, updated to use the standard Python logger in all the Control Center py files EXCEPT Graphing, which I need to do a big overhaul on anyway, so I’ll do that at the same time.  

I might update the sensor code for the hardware next, to give it a decent log.  I also still need to update the trigger code in a multitude of ways too.  That or maybe I’ll do more on the Graphing, as I have been meaning to do that for a few weeks now.  

Just thought I would post about the excitement of logging!  Being able to adjust the logging level is super nice too!  I can definitely get away without making a test program for a bit.  At least until I work through a bit of my todo list and break apart a few of my rather long functions.  I think that about does it for programing today, time to relax a bit.  

On a unrelated note, the smoke is basically gone now.  Had a few good rain falls and its very clear skies!  

Logging first, test apps later

Since I have been replacing the logger in my app away from my home grown one, it’s looking like it will function WAY better between different py files then mine.  I was doing some crazy return messages from other py files to the main, where the log was kept.  I realized this was not sustainable pretty darn quick, and only had a few function use it outside the primary py file.  Now with the built in Python logger, I think I can get a lot of good info delivered for troubleshooting purposes.  

Since I want to get started on creating new abilities and function, I’m going to do the logging overhaul now, and then continue functionality development.  I’ll delay implementing the test programs, to space out the “boring” work.  This should aid in keeping Motivation higher. 

Chances are, because I’m using this project to learn how to program initially, I would end up creating and tearing down the test programs every time I learn better ways of doing things.  So I think logging is a better way to go for now, until I have more firm foundation of creating stable classes and functions…. of which I have not actually created any class yet, pretty much using separate py files as a “class” (mimic the organization of  class).  AKA I still have to read up on python classes to see when and why to implement them vs a new py file.  

Work and Family matters have picked up recently, so I have less time to work on the program, but I’m trying to at least poke at it once a day (the program itself or learning about programing).  I’m also still figuring out PyCharm and Git both of which I’m liking more and more.  

Testing & Logging

The next big thing I have to work on, is creating test programs I can run, to auto test my main programs.  Considering all the functions and variables, this will probably take a bit of time to complete.

Right now, I just manually test the program aspects that get changed, such as creating graphs when the graph functions change.  This takes up a lot of time, and it’s only going to take more time as things progress further.  However, I don’t want to rush my test code, since its going to be used a LOT!  So I’ll set aside some time to plan it out a bit.  What it does, how and why.

In other news, I have taken a sensor on 2 trips, recording the whole time.  For one, this gave me a 9MB Motion database file 🙂  This should help greatly in stress testing some of the graph features.  Since motion can record up to 4 times per second, I need to find a way to filter that data, so it doesn’t have over 14,000 graph points in a hour’s time or less.  

I was thinking, if a data point is within 2 Seconds of the last one AND the difference is less than variance (say 0.1), skip it.  I can also set the time and variance as variables at the top, so they can be changed. That should cut down on a lot of the noise, yet leave flexibility to adjust for whatever the need may be.  Or I could just use one variable, so if it’s over X variance, it records for say 15 seconds, and resets that timmer if X variance happens again during the time.  This would give a balance of fairly accurate recordings during an event, while recording nothing if its been still after 15 seconds.  

Come to think of it, I should probably add a config file on the sensor units themselves for variables that might be useful to change, depending on the situation.

In other news, I found out there is a included module in Python for logging.  I found it when I was going through PyCharm EDU lessons.  So that should clean up some of my code and allow a more universal experience.  

I have already swapped the logger in my main py file, thanks to word replacement in Notepad++.  PyCharm wouldn’t let me do it through refactoring though, because I was changing it from a function (message_print_log) to logger.info(), and PyCharm really didn’t want to change from no period thingy to something with a period in it.  

I do like the logger so far, it seems simple yet flexible.  I expect to be fully using it, instead of my home grown one in a few days time with minimal effort.  I do have to change how my logging window functions though… In fact I’ll probably just open up the log file itself in a text editor to “open” the log.  

Until next time.

Edit: I forgot to mention, I have all my code on GitHub now!  (All 4 programs since I started coding in June 2018).  

https://github.com/chad-ermacora