GUI Refactoring!

I finally did it!  I have split up the GUI code into 9 files, one file per GUI Window for the most part. The GUI was in a single file over 1800 lines long, after breaking it up, most files are under 200 lines, but the Graph and IP Selection code is about 500 each.  Still, a lot nicer to deal with, then the 1800 line monster.  

It took me about 3 days to get that done, and there are still some little things to do, like clean up the shared configurations, since I had to mess around, trying to share config between the modular windows.  It appears to all be working now, but it will take a few more days to clear out old code and update other parts of the program to function more seamlessly with the new code.  I’ll also review the 500 line graph and IP selection modules to see about optimizing that.  

While I was breaking up the GUI, I also streamlined a few other things, like the logger, which has its own module now, so other windows can import and use that. There is MUCH less code in each module file for logging now.

Since there are no updates to the function of the program, I won’t be releasing a new version, but I did silently release ver. Alpha.19.2 last week, which includes a few new features I did before the GUI overhaul, like the Live Graph, Live Sensor Readings Report and multi threading for Sensor checks and downloads.

I want to make an overview video of the Control Center, but that will have to wait, since I don’t know much about Video production, and my program keeps changing 🙂  

Guess that’s it for today.  

More features!

It’s been a bit since my last update.  I’m happy to say, there are a few nice new features to the PC Control Center.  

I have now added the ability to graph a sensor’s reading in real time with a customizable duration between new readings.  I was originally going to pull up a graph for each sensor IP & selected sensor types, but matplotlib isn’t thread safe, and I’ll have to look into multiprocessing to get that working.  You can open multiple copies of the program itself, to view multiple live sensors at the same time.  

I also made a new Report, for viewing a snapshot of all sensor readings from the selected sensor IP’s.  Its a good way to see if you have the sensor setup right remotely, or to quickly test multiple sensors, making sure things are working as expected.  

I’m feeling pretty good about the advancements, but the primary GUI code is getting pretty big, at over 2000 lines of code!  The more that builds up, the less I feel like figuring out how to break it up, but it will be done at some point, and the longer I wait, the more I’ll of learned about coding, which should help with re-designing.  The main problem I’m having, is wrapping my head around how to interact with all the GUI components between software modules.  I’m sure the solution is in Object Oriented Programing, but I’m still working on OOP concepts, so I’m not sure what my timeframe is going to be for re-designing the GUI. 

O cool side note, I have 4 Raspberry Pi’s that have been up and running continuously for more than 60 days!  That’s pretty good for reliability, considering 2 of them are outside and I test my programs on them all the time. 

I almost forgot, I spent time making sure the PC Control Center works properly under Linux, specifically Raspbian.  I had to add some customizations in the program for the GNU/linux OS, especially in the GUI window sizes, but it works very well now!  I’m going to put the PC Control Center into the Raspberry Pi Sensor installer, so you can manage multiple sensors, from the sensors themselves through VNC.  This may help with not so great remote connections, as the local units should have a better connection to interact with the local sensors and prevent network timeouts.  

On another side note, I’m kinda excited thinking of what I might do with the sensors, using these programs as a interface to get readings remotely.  I was thinking of making other programs, that interact with the sensor network server, to get readings and combine a bunch to automate certain things, like opening doors or windows, turning on and off lights, adjusting light levels based on current light levels, etc.   
With halloween coming up, one could add a thermal or motion detector with a distance sensor to perfectly time spooky whatevers. Imagine a skeleton that does something different depending on the distance of the person or objects, for example at 6 or more meters, it plays something to get them over, like a nice lady voice offering candy, then at 3 meters, it starts to laugh, if the person moves away, it stops laughing, and tries to call them back, otherwise if they keep coming, it laughs until 1 meter, then it does some scary thing with lights and all.
Loads of cool stuff that can be done!  

I’m sure I’m forgetting to mention a bunch of other changes …. like how I multithreaded the database downloads, but that’s good enough for this update.  I plan on doing a bit more testing and refactoring before putting out another version.  

Until next time!

Updates

It’s been a few days, mostly because a bit more work came my way.  This didn’t stop me from getting a few more things done on the programs.

For starters, I now have it so you can get the Sensors configuration settings in a HTML table, much like the System Details.  I have also made it so you can push those settings on multiple units at once from the PC Control Center.  

I changed my “Home Grown” crontab auto start to standard systemd Services.  Its much faster for program restarts and I can completely disable things on a per program basis (Which is done when you disable SQL recording on the sensor configuration).  I also have network sensitive things wait for the network before starting up.  Then of course it just seems nicer to integrate through the standard System’s Services ^_^

I’m also happy to report, that this version is backwards compatible with the previous version!  Normally this wouldn’t be a big deal, but it means my code is getting more stable and adding functionality is not breaking previous things.  Who knows, maybe a Beta release is coming?

Since the software is coming along nicely, and you can pretty much fully re-configure a sensor remotely with the PC Control Center, I’m thinking I should focus on some Hardware Design to get more Sensors out there and self sustaining.  I’m also wanting to start on a TriCorder too ^_^

Version Alpha.17.1 Released

I have added some new features, refactored a lot of code and tested things out.  I think it’s good to go!

Some features to mention include
– Save IP list to config and auto reload it on start
– Added Operating System Upgrade to Sensor Commands window
– Added help menu for Program Website Link, DIY Sensor Unit, Sensor Unit Help, PC Control Center Help (WIP)
– GUI, Text and Layout Changes
– Enhanced error checking
– Toned down Logging amount to almost ONLY errors and warnings
– Bug fixes

KootNet Sensors – PC Control Center Ver. Alpha.17.1

Star Trek Tricorder!

OMG I just realized I have what I need to make a real tricorder now!  What’s more, I can use my existing Sensor Project with it!  

What I was thinking is this.  Create a remote unit similar to my other remote sensor unit, that allows wireless interactions with the sensors, but have local sensors as well, so you can swap between the tricorder itself OR any sensor in wireless range and get real time readings on a small LED / LCD screen.  

Add to this the Garden Breakout Board, where you can swap sensors, this should be a pretty awesome unit!  

I can’t tell you how long I have wanted a REAL tricorder to digitize the world around me!  It’s why I started my original Sensor Project.  It just dawned on me today, that I don’t need to make the sensor board from scratch to make a Tricorder, I can use pre-made ones with the Garden Board (allows up to 6 sensors at a time) to choose whatever sensor is needed for the job!  … well within reason of what’s available for it.  

I’m pretty sure I have all the needed hardware to get started, just no case yet.  But I suppose I should get the unit working before I make a case anyway, in ‘case’ the design dimensions change as I go. 

Soo excited now!  

Reworking write to DB Code (again)

So I had the Trigger database write code all working with variance triggers for the Magnetometer, Accelerometer and Gyroscope.  However, I noticed the datetime for writing was only accurate to 1 second.  Triggers are done in ms not sec.  It took me a bit to figure out how to format the strftime in SQLite3 properly (compounded by another error when I changed how the DB is checked). 
Long story short, it now records trigger events down to 3 decimal points of a second.  This will be perfect, since I usually do things no lower than 100ms.   

I have made both the Trigger DB AND the Interval DB into continually running programs instead of one off runs, so it doesn’t have to load up all the code every time it writes to the database.  This can have a big impact on how long it takes to get the sensor readings, which is important for the time sensitive Trigger code.

I was thinking of getting the datestamp for writing into the database, after the sensor readings are returned from the hardware sensor, but then I have to use python to find and convert the timezone off the Sensor itself into the nice generic UTC 0… But i’m not sure how to Dynamically get the timezone off the System to do the conversion (Not to mention, that alone may add more delay between getting the readings and the timestamp).  So i’m letting SQLite3 itself insert the time when it writes to the DB, as it always writes in UTC 0 if you don’t specify. 
This isn’t super accurate, since it has to go through a few functions before it can write, but it is a relatively small difference for when the DB is written vs when it got the sensor readings. I tested the time difference by printing the system time to the screen, just after each Trigger reading, and comparing it to what it is in the database.  It seems like the accuracy is within 200ms on a Raspberry Pi 3B+.  

In other news, I found out how to disable Resizing the window using GUIZero and created prompts for when actions are complete, such as downloading Databases and sending commands to the sensors.  This especially helps when the network is not so great and you get longer wait times and or network timeouts.  

“Trigger” database code

So I spent a good portion of today, re-working my database code.  When I started, I realized my sensor modules ALL get loaded, regardless if they are used or even installed.  For my code, it’s pretty small, but the sensor’s imported code will add up, especially as more sensors are supported.  So I found a way to load them on demand AFTER checking to see which sensors are installed.  

To make this happen, I turned each sensor module, into a class and moved the import into the class initializer. AKA, it only imports when a instance of the class is made.  This also makes it a bit easier to follow the code, as each instance has all the needed functions.  Due to the new simplicity, I was also able to remove 2 other .py modules.  I’m also finding that a lot of my modules are getting smaller and smaller, most are no more than 200 lines of code now, with a lot of it being the license at the top and setting up the logger.  So that’s helping with code management. 

The Trigger code has all the basics EXCEPT the actual variance checks (AKA the triggers).  I’ll probably have that done tomorrow, but in the meantime, I wanted to build up some database readings, so I made it write all the trigger readings to the database every 5 min and updated my sensors with it.  

It seems like every time I get something working, I have another 3 things I want to add!  My todo list is getting pretty long, and much of it, is no simple task to do (relatively speaking … probably take days or weeks).  If I’m not adding a new feature, I’m refactoring the code!  In fact, I seem to be refactoring more then adding functions now, but that’s mostly because I’m learning as I go, and start implementing better ways of doing things as I learn them.  A few examples include the new logger, using classes, making better logic loops and naming things to make sense, based on what they do or hold.  

As always, I’m happy with the progress that’s being made.  If not in the program, in my learning how to program ^_^

I still like looking back at my old code and seeing how its changed.  I should really remember to take screenshots of the program’s progress, to keep a visual record of its development. I found a pic of the program, near its initial development, and I remember how excited I was at the time, that I was able to make something like that.  The program(s) have come a long way since then (2 months ago), but because I’m excited about the progress, not the product, my excitement and motivation level is remaining fairly high (when I have had enough sleep).  

I think that’s enough programing for today.  Until next time!

Side thought of usefulness

With the recent changes to modularize the sensors, you don’t NEED to add a sensor on the Raspberry Pi to get started.  Choose the Raspberry Pi as your only sensor in the sensor config file, and you can keep track of System up time and CPU Temperature over time.

This will make testing the program as a whole, a LOT easier and a bit cheaper. This could also be handy for tracking the stability of a Raspberry Pi. If you have a Pi that’s rebooting or freezing up, this would help you track down the real culprit, by giving you a timeframe to narrow the search.  Combine that with a CPU Temperature reading accompanying the timeframe, and you can see if the unit is having overheating issues! You can also view multiple Pi’s real time stats in a html table for easy comparison.

This gives me another idea for a Option in the program.  I could add a entry in the Sensor config file, to allow “Extra System Information Recordings”, which would record things like CPU, RAM and disk usage.  By enabling this, you could literally use the Sensor Programs to truly monitor the health and stability of your Raspberry Pi’s.  It can even be enhanced to allow other systems fairly easily, like Windows and Mac.    

Of course I never designed them to be used for this scenario. I would have to think about security around the socket connections if there’s a good chance the Sensors are on a open network.

On another related topic. While talking with someone about the Sensors, they suggested I add a realtime graphing feature, to be able to constantly monitor things like Humidity for gardeners.  I think I’m about ready to take that on.  I’m not 100% sure how to pull it off just yet … What I should really do, is get a plotly dash app working on the sensor itself, so then I just have to pull up a URL based on the IP of the sensor.  But then you can only do one at a time, and that’s not very effective for bigger operations… I’ll have to think on this one more.

In other news, I’m feeling pretty good with the state of the Control Center App.  I have improved a bunch of the error checking and logging, so it’s harder to make the program “mess up”.  I should probably work on notifying the user when things go wrong though.  The program adds a error log message when things go wrong, but nothing extra; AKA, if a graph fails for example, it would look like it didn’t do anything.  Putting up a GUI OK button message with the error would be good, I just need to be careful to only show it once, so it doesn’t pop up a few thousand times for things like a invalid SQL entry, or maybe create a counter, then at the end, it pop’s up a single message if the counter > 0, and list the count in the message, so you know how many errors were encountered.  That could be useful… A few things to consider.  

New release Alpha.16.1

I have been working on polishing up the small things for this Alpha release, both in the Raspberry Pi sensor code and the control center code.

I have improved the pi sensor installer and upgrade scripts, both of which seem to be working well now.  I have tested it out a few times on my main sensor, from scratch and through upgrades.  After that, I upgraded all the units I have running right now, which there are 9. Things seem to be running smoothly, so I have decided to push it to the master branch.  I have toned down a lot of the logging messages from info to debug, so only errors show for the most part, and that looks pretty nice.  I’m debating putting sensor Online checks into debug if they are in fact online and only log offlines.  

Anywho, this release is looking good!  Shortly after merging, I changed a few more things to help out, such as saving the IP’s to the config file and loading on startup.  I also found and fixed a bug in the graphing datetime conversions, where it needed to apply the inverse conversion for getting the SQL data, then a normal conversion for what it SHOWS on the Graph.  It now lines up the proper time perfectly!  I also realized I was only graphing by date, not datetime, so I changed that to be specific to the second.

I probably won’t merge the new changes to master, until I get the trigger Database recording AND graphing working.  I have done a bunch of ground work, so there’s not a lot more I need to do to get the Sensor recording part working, but I will have to put a bit more time into the graphing from SQL … 

I have updated the installers, database demo’s and even posted install instructions for making a sensor (very basic).  So feel free to try things out!  Keeping in mind, it’s a Alpha release, so things break between releases and there will probably be a few bugs.  That being said, it seems pretty stable and usable to me!  

I hope to next release or the one after will be a Beta release.  Once the Trigger Database is working, I’ll be pretty happy with the feature set to release a first version, trying to work more on bugs for a bit.  Then I can start adding more features, but I’ll probably need new types of sensors to get really creative.  Not to say there is not a rather large list of features I could add, I just want to get a stable foundation before adding more on top. 

That’s it for now.   Slowly getting closer to a stable release! 

Tweaking & RP Sensor Installer

I have spent the past few days just going over the code and tweaking it.  Fixed a few bugs, refactored a bunch o code and even started on the “Help” section for building a sensor!  Of course at that point, I realized I needed to work on my installers for the RP Sensor code itself.  So that’s what I did. 

Any additional runs AFTER install, simply opens the config files, so you don’t get duplicated network code from the installer.  This also makes it easy to re-configure your sensor, simply by re-running the install code.  I have also copied both the Upgrade and config_edit scripts into the pi user’s home folder, so when you login with SSH or open a terminal, you can go right to upgrading or editing config.  

The Help section for building your own sensor is… Basic at best right now, but I wanted at least SOMETHING there to get started.  As I was making it, I went over the shell scripts for installing the Sensor code.  I managed to make it, so you only have to run a single command in terminal, and then it starts.  This command includes the download itself, and how it works, is this.  The command tells wget to get the install script off my http server, then tells bash to execute that script, after adding the execute permission to it.  This install script downloads all the other needed files off my http server and puts them where they should be, after which, it updates the Raspbian system itself, and installs all necessary dependencies through pip3.  During the process, it auto fills and opens up necessary config files to edit (Installed Sensors, IP, Wireless), with instructions built in as comments.  After its done, it automatically reboots the Raspberry Pi.  When the Pi boots back up, it should auto connect to whatever wifi was entered and start recording sensor data every 5 minutes to a SQL database.  By putting your computer on the correct IP subnet, you can also connect to the sensor with the KootNet Sensors – PC Control Center program.

I will be merging these updates and changes with the master branch and creating new installers and demo stuff soon.  I’m taking a bit more time to test things out, find and fix a few bugs and improve log output.