Author Archives: chad.e - Page 2

Remote Log Viewer & more Threading!

I got a remote sensor log viewer up and running, so you can live view the tail end of a log for individual sensors, or download all logs off all selected sensors and review them later.

I have also added threading to all HTTP downloads (logs + Databases) and Reports.  Reports generate a lot faster now!  I also made a minor change to HTTP downloading, which places the end IP oct at the start of the file for easier sensor identification on files.

Many parts of the programs are looking better and better, but there are some, like the network commands code, that just looks… lacking.  I have off and on been thinking about how to re-work my 30 or so ‘if + elif’ statements to check what command to run.  There must be a way to objectify the code, so it doesn’t look like a run on sentence… nothings coming to mind though, but then again, I haven’t eaten this morning yet… so maybe I’ll do that and re-think it over. 

I should really start working on the help file for the Control Center, since its starting to stabilize and there are enough core features I can document.  I would like to find a decent template file to use for my help files, since my versions are… also lacking.  That being said, they do get needed information across, in a non overwhelming way, they could just use a bit more “zing”.  

On the learning side of things, I’m pretty happy with progress.  I have learned a LOT of programing concepts and python libraries over a decently short amount of time.  As I go, and re-use some of those library and programing concepts, things just seem more “obvious” in how it all works together. When I first start on something new, I often pause on how to work it into my existing code, but once it’s there, it seems soo much easier re integrate into other sections.  After the 3rd or so time, I then start seeing how to refactor the code to be shorter, more obvious in function or more effective.  

Alpha.20.2 Released!

This release has a lot more nice touches, than huge features.

The reports section has been updated to include the following.
– Local Computer time when the report was generated
– Currently installed version on each sensor
– Last updated per sensor, with how it was updated at the time

When the “Advanced” features are enabled on the Sensor commands, it now turn the upgrade options into “Clean” upgrade options, which essentially deletes everything but data and config, and re-installs. 

The configuration window now has the live graph refresh time and a label above the DateTime entries to show how to set it (YYYY-MM-DD HH:MM:SS). 

I have also done a lot of changes on the RP Sensor install script, such as including Control Center and installing its dependencies.  I have made the HTTP upgrade method download a zip of the files and extract it, instead of getting each individual program file.  I also tested the install script until it worked flawlessly on a RP3B+ with a SenseHAT, a RP Zero W with a Garden HAT & a RP Zero W with a EnviroPHAT.  They all worked well, but I would NOT recommend using the Control Center on the Pi Zero, due to its limited resources (AKA its slow…).  

I have also updated some website links, pictures and downloads.  So it should be easy to find more useful information about the programs, especially with the new “Pictures & Screenshots” section.  

That’s it for now. 

Getting Raspbian working – KootNet Control Center

So I have been spending most of the day, getting the Control Center program to work on Raspbian, and installed with the normal Sensor installer script (AKA, you can use the Pi as a sensor AND controller by default).  

The dependencies are annoying, and matplotlib kept on failing to plot live graphs… 4 hours later I found out it’s some glitch in the version of numpy that was installed.  Long story short, getting the latest version fixed it.
python3 -m pip install -U numpy
It takes a pretty long time to update it this way on the Raspberry Pi Zero W, but I would rather that, then making the install process more complicated for the user.  

I’m doing final testing now, to make sure the single command line to install is working with no additional steps.  Once I get this working, I’ll probably release another version.

*Few hours later*
So the Raspbian installer script appears to work fairly well!  It installs both the sensor programs and the control center, then all dependencies and rebuilds the latest numpy.  It also puts a shortcut to the control center on the desktop of the pi user, for easy opening.  I set the first IP as the localhost 127.0.0.1 by default and checked it.  This makes it so you can start testing things right away on the local unit.  I tested the install script on a Raspberry Pi 3B+ with a Sense HAT and a Raspberry Pi Zero W with a EnviroPHAT.  The 3B+ went MUCH faster, and functioned pretty well … things worked just fine on the Pi Zero… but slow… just opening the Control Center takes about 30 seconds.  Still, it does function decently once its loaded, so it will make a good fall back way to test things. 

To give a bit of a timeframe to the slowness, I’m pretty sure it took about 2 hours on the pi zero one time, just to finish the install, most spent rebuilding the python3 numpy module.  I’m hoping to fix this easier in the future, by creating a virtual pip environment with specific versions.

Since I changed the install script to download a zip of all the files, then unzip it locally, its WAY FASTER!  It also seems less prone to network glitches.  Besides being faster, it also helps with sending less data, since the zip file uses compression.  

Lots more to enhance, but things are coming along nicely.  

I hope everyone has a great weekend!

Making programs to test my programs!

Another milestone reached!  I have now looked into and created a unittest program for testing the control center program.  It’s just a basic one right now, and it mostly tests the configuration sections.  As time goes on, I’ll add more tests and probably re-structure some of my other functions to be more test friendly.  

With tests coming along and the GUI broken up using OOP concepts, I think that covers 2 of the bigger things I had to figure out.  Now it’s just optimizing and expanding them… or another overhaul, if I find other new programing concepts worth implementing. 

I might work on adding the control center into the Raspberry Pi Sensor installer next … and creating menu shortcuts would be nice too.  That way, you can fully test out the sensor programs, simply by setting up a Raspberry Pi.  The sensors also responds a lot nicer, when it’s using 127.0.0.1 ^_^

So most likely I’ll be doing touch up work next.  I have a fairly decent list of smaller changes to be made, like making live graphs work with the Trigger sensor types, adding a “Last updated” entry on my reports and implementing more unit tests. 

Until next time.

GUI Refactoring!

I finally did it!  I have split up the GUI code into 9 files, one file per GUI Window for the most part. The GUI was in a single file over 1800 lines long, after breaking it up, most files are under 200 lines, but the Graph and IP Selection code is about 500 each.  Still, a lot nicer to deal with, then the 1800 line monster.  

It took me about 3 days to get that done, and there are still some little things to do, like clean up the shared configurations, since I had to mess around, trying to share config between the modular windows.  It appears to all be working now, but it will take a few more days to clear out old code and update other parts of the program to function more seamlessly with the new code.  I’ll also review the 500 line graph and IP selection modules to see about optimizing that.  

While I was breaking up the GUI, I also streamlined a few other things, like the logger, which has its own module now, so other windows can import and use that. There is MUCH less code in each module file for logging now.

Since there are no updates to the function of the program, I won’t be releasing a new version, but I did silently release ver. Alpha.19.2 last week, which includes a few new features I did before the GUI overhaul, like the Live Graph, Live Sensor Readings Report and multi threading for Sensor checks and downloads.

I want to make an overview video of the Control Center, but that will have to wait, since I don’t know much about Video production, and my program keeps changing 🙂  

Guess that’s it for today.  

More features!

It’s been a bit since my last update.  I’m happy to say, there are a few nice new features to the PC Control Center.  

I have now added the ability to graph a sensor’s reading in real time with a customizable duration between new readings.  I was originally going to pull up a graph for each sensor IP & selected sensor types, but matplotlib isn’t thread safe, and I’ll have to look into multiprocessing to get that working.  You can open multiple copies of the program itself, to view multiple live sensors at the same time.  

I also made a new Report, for viewing a snapshot of all sensor readings from the selected sensor IP’s.  Its a good way to see if you have the sensor setup right remotely, or to quickly test multiple sensors, making sure things are working as expected.  

I’m feeling pretty good about the advancements, but the primary GUI code is getting pretty big, at over 2000 lines of code!  The more that builds up, the less I feel like figuring out how to break it up, but it will be done at some point, and the longer I wait, the more I’ll of learned about coding, which should help with re-designing.  The main problem I’m having, is wrapping my head around how to interact with all the GUI components between software modules.  I’m sure the solution is in Object Oriented Programing, but I’m still working on OOP concepts, so I’m not sure what my timeframe is going to be for re-designing the GUI. 

O cool side note, I have 4 Raspberry Pi’s that have been up and running continuously for more than 60 days!  That’s pretty good for reliability, considering 2 of them are outside and I test my programs on them all the time. 

I almost forgot, I spent time making sure the PC Control Center works properly under Linux, specifically Raspbian.  I had to add some customizations in the program for the GNU/linux OS, especially in the GUI window sizes, but it works very well now!  I’m going to put the PC Control Center into the Raspberry Pi Sensor installer, so you can manage multiple sensors, from the sensors themselves through VNC.  This may help with not so great remote connections, as the local units should have a better connection to interact with the local sensors and prevent network timeouts.  

On another side note, I’m kinda excited thinking of what I might do with the sensors, using these programs as a interface to get readings remotely.  I was thinking of making other programs, that interact with the sensor network server, to get readings and combine a bunch to automate certain things, like opening doors or windows, turning on and off lights, adjusting light levels based on current light levels, etc.   
With halloween coming up, one could add a thermal or motion detector with a distance sensor to perfectly time spooky whatevers. Imagine a skeleton that does something different depending on the distance of the person or objects, for example at 6 or more meters, it plays something to get them over, like a nice lady voice offering candy, then at 3 meters, it starts to laugh, if the person moves away, it stops laughing, and tries to call them back, otherwise if they keep coming, it laughs until 1 meter, then it does some scary thing with lights and all.
Loads of cool stuff that can be done!  

I’m sure I’m forgetting to mention a bunch of other changes …. like how I multithreaded the database downloads, but that’s good enough for this update.  I plan on doing a bit more testing and refactoring before putting out another version.  

Until next time!

Updates

It’s been a few days, mostly because a bit more work came my way.  This didn’t stop me from getting a few more things done on the programs.

For starters, I now have it so you can get the Sensors configuration settings in a HTML table, much like the System Details.  I have also made it so you can push those settings on multiple units at once from the PC Control Center.  

I changed my “Home Grown” crontab auto start to standard systemd Services.  Its much faster for program restarts and I can completely disable things on a per program basis (Which is done when you disable SQL recording on the sensor configuration).  I also have network sensitive things wait for the network before starting up.  Then of course it just seems nicer to integrate through the standard System’s Services ^_^

I’m also happy to report, that this version is backwards compatible with the previous version!  Normally this wouldn’t be a big deal, but it means my code is getting more stable and adding functionality is not breaking previous things.  Who knows, maybe a Beta release is coming?

Since the software is coming along nicely, and you can pretty much fully re-configure a sensor remotely with the PC Control Center, I’m thinking I should focus on some Hardware Design to get more Sensors out there and self sustaining.  I’m also wanting to start on a TriCorder too ^_^

Version Alpha.17.1 Released

I have added some new features, refactored a lot of code and tested things out.  I think it’s good to go!

Some features to mention include
– Save IP list to config and auto reload it on start
– Added Operating System Upgrade to Sensor Commands window
– Added help menu for Program Website Link, DIY Sensor Unit, Sensor Unit Help, PC Control Center Help (WIP)
– GUI, Text and Layout Changes
– Enhanced error checking
– Toned down Logging amount to almost ONLY errors and warnings
– Bug fixes

KootNet Sensors – PC Control Center Ver. Alpha.17.1

Star Trek Tricorder!

OMG I just realized I have what I need to make a real tricorder now!  What’s more, I can use my existing Sensor Project with it!  

What I was thinking is this.  Create a remote unit similar to my other remote sensor unit, that allows wireless interactions with the sensors, but have local sensors as well, so you can swap between the tricorder itself OR any sensor in wireless range and get real time readings on a small LED / LCD screen.  

Add to this the Garden Breakout Board, where you can swap sensors, this should be a pretty awesome unit!  

I can’t tell you how long I have wanted a REAL tricorder to digitize the world around me!  It’s why I started my original Sensor Project.  It just dawned on me today, that I don’t need to make the sensor board from scratch to make a Tricorder, I can use pre-made ones with the Garden Board (allows up to 6 sensors at a time) to choose whatever sensor is needed for the job!  … well within reason of what’s available for it.  

I’m pretty sure I have all the needed hardware to get started, just no case yet.  But I suppose I should get the unit working before I make a case anyway, in ‘case’ the design dimensions change as I go. 

Soo excited now!  

Reworking write to DB Code (again)

So I had the Trigger database write code all working with variance triggers for the Magnetometer, Accelerometer and Gyroscope.  However, I noticed the datetime for writing was only accurate to 1 second.  Triggers are done in ms not sec.  It took me a bit to figure out how to format the strftime in SQLite3 properly (compounded by another error when I changed how the DB is checked). 
Long story short, it now records trigger events down to 3 decimal points of a second.  This will be perfect, since I usually do things no lower than 100ms.   

I have made both the Trigger DB AND the Interval DB into continually running programs instead of one off runs, so it doesn’t have to load up all the code every time it writes to the database.  This can have a big impact on how long it takes to get the sensor readings, which is important for the time sensitive Trigger code.

I was thinking of getting the datestamp for writing into the database, after the sensor readings are returned from the hardware sensor, but then I have to use python to find and convert the timezone off the Sensor itself into the nice generic UTC 0… But i’m not sure how to Dynamically get the timezone off the System to do the conversion (Not to mention, that alone may add more delay between getting the readings and the timestamp).  So i’m letting SQLite3 itself insert the time when it writes to the DB, as it always writes in UTC 0 if you don’t specify. 
This isn’t super accurate, since it has to go through a few functions before it can write, but it is a relatively small difference for when the DB is written vs when it got the sensor readings. I tested the time difference by printing the system time to the screen, just after each Trigger reading, and comparing it to what it is in the database.  It seems like the accuracy is within 200ms on a Raspberry Pi 3B+.  

In other news, I found out how to disable Resizing the window using GUIZero and created prompts for when actions are complete, such as downloading Databases and sending commands to the sensors.  This especially helps when the network is not so great and you get longer wait times and or network timeouts.