Monthly Archives: September 2018

Graphing for Interval Database’s has been Upgraded!

Yay!  Upgraded the Graphing abilities & completely Redid the back end code. It now allows the user to choose what sensors to graph through a series of checkboxes.  Creating Data Classes REALLY helped with this. 

This should set the stage for future additions of sensors, as I can add on more, without having to worry about a lot of edits in other functions.  Thanks to the new Data Classes, it should also be fairly easy to integrate new graph types (pie, bar, trace, etc).  So that will be coming later. I’m pretty happy about this, since I have been delaying this upgrade for awhile, due to all the other code stuff being learned & applied.

I suppose this free’s me up to do one or more of the following. 
A) Upgrading Logging on other aspects, such as the HW Sensor Software
B) Upgrade and Enhance the other HW Sensor Software in General
C) OverHaul the Trigger Database recording code
D) Work on the Sensor HardWare itself

Improving the Sensor Hardware has been another thing I want to work on, specifically the getting them to be self sustaining for power, so I can leave them in the wild to collect data.  I have mostly been waiting on batteries to use with a power system + Solar.  I have yet to find a way to connect my bigger USB Battery banks up to it, so it can charge AND use it.  I’m kinda stuck with these much smaller Lithium Ion batteries (Should be here this week).  I’ll have to see how long they can run for, and how good the solar has to be to keep them going… and think of a backup for when the sun don’t shine.

Sometime after getting the batteries and solar working, I’ll need to figure out some protective cases, since there are some pretty good storms that come through, not to mention animals and rodents doing who knows what.  I found a site that uses more common items, like random pipe and plastic containers. I’ll probably attempt that when I know the dimensions of the sensor’s and their components. 

The Python Class

Ahh, the Python Class, and I don’t mean a Classroom full of students.  

I found a EXCELLENT YouTube channel for explaining aspects of Python.  In fact, it’s the same place I found the logger videos.  I didn’t realize it was the same person at first, but once I did, I bookmarked it!  

His name is Corey Schafer’s and his videos are located at the following link.

Now that I have learned the basics of a Class, I’m starting to use it already in my overhaul of the graph section of my Sensor Program.  It should clean up my code very nicely, because I can forward one class object to the graph function, and it will hold ALL the needed data!  Right now, I have been sending each needed variable to each function, which had up to 6 variables!  It wasn’t pretty, that’s for sure.  However, with a data class, I can store everything that’s needed, pass the one class instance, and BAM!  Nice looking code, pretty much unlimited variables, and I can create as many instances as I want, with very little code!  It was even nicer when I realized I can add new variables (or function) to it, without having to adjust every function that uses it.  All very good things ^_^  

I have to admit, I have not actually added any new functionality to the program at all, but I have been learning a lot on how to improve the existing code and set the foundation of future enhancements.  Add on to this, GitHub and PyCharm, and the management of the code has also greatly improved!  

So although not adding thing to the program seems to be bothering me, I know it’s for a good cause and will help much more in the long run.  So I guess overall, I’m feeling pretty good with the progress.  

On another side note, my “step dad” / “Friend” / “something?” … not sure what to call him, but he’s awesome and knows a lot around chemistry and science.  Anyway, I asked for a bit of input from him about what sensors I might add and what I might be able to figure out with long or short term readings of said sensors.  He’s going to give it some thought and get back to me.  So I’m excited about that!  I’m hoping it will lead to practical use of my project to find out something new or cool!  

Logger change Done!

Yay, updated to use the standard Python logger in all the Control Center py files EXCEPT Graphing, which I need to do a big overhaul on anyway, so I’ll do that at the same time.  

I might update the sensor code for the hardware next, to give it a decent log.  I also still need to update the trigger code in a multitude of ways too.  That or maybe I’ll do more on the Graphing, as I have been meaning to do that for a few weeks now.  

Just thought I would post about the excitement of logging!  Being able to adjust the logging level is super nice too!  I can definitely get away without making a test program for a bit.  At least until I work through a bit of my todo list and break apart a few of my rather long functions.  I think that about does it for programing today, time to relax a bit.  

On a unrelated note, the smoke is basically gone now.  Had a few good rain falls and its very clear skies!  

Logging first, test apps later

Since I have been replacing the logger in my app away from my home grown one, it’s looking like it will function WAY better between different py files then mine.  I was doing some crazy return messages from other py files to the main, where the log was kept.  I realized this was not sustainable pretty darn quick, and only had a few function use it outside the primary py file.  Now with the built in Python logger, I think I can get a lot of good info delivered for troubleshooting purposes.  

Since I want to get started on creating new abilities and function, I’m going to do the logging overhaul now, and then continue functionality development.  I’ll delay implementing the test programs, to space out the “boring” work.  This should aid in keeping Motivation higher. 

Chances are, because I’m using this project to learn how to program initially, I would end up creating and tearing down the test programs every time I learn better ways of doing things.  So I think logging is a better way to go for now, until I have more firm foundation of creating stable classes and functions…. of which I have not actually created any class yet, pretty much using separate py files as a “class” (mimic the organization of  class).  AKA I still have to read up on python classes to see when and why to implement them vs a new py file.  

Work and Family matters have picked up recently, so I have less time to work on the program, but I’m trying to at least poke at it once a day (the program itself or learning about programing).  I’m also still figuring out PyCharm and Git both of which I’m liking more and more.  

Testing & Logging

The next big thing I have to work on, is creating test programs I can run, to auto test my main programs.  Considering all the functions and variables, this will probably take a bit of time to complete.

Right now, I just manually test the program aspects that get changed, such as creating graphs when the graph functions change.  This takes up a lot of time, and it’s only going to take more time as things progress further.  However, I don’t want to rush my test code, since its going to be used a LOT!  So I’ll set aside some time to plan it out a bit.  What it does, how and why.

In other news, I have taken a sensor on 2 trips, recording the whole time.  For one, this gave me a 9MB Motion database file 🙂  This should help greatly in stress testing some of the graph features.  Since motion can record up to 4 times per second, I need to find a way to filter that data, so it doesn’t have over 14,000 graph points in a hour’s time or less.  

I was thinking, if a data point is within 2 Seconds of the last one AND the difference is less than variance (say 0.1), skip it.  I can also set the time and variance as variables at the top, so they can be changed. That should cut down on a lot of the noise, yet leave flexibility to adjust for whatever the need may be.  Or I could just use one variable, so if it’s over X variance, it records for say 15 seconds, and resets that timmer if X variance happens again during the time.  This would give a balance of fairly accurate recordings during an event, while recording nothing if its been still after 15 seconds.  

Come to think of it, I should probably add a config file on the sensor units themselves for variables that might be useful to change, depending on the situation.

In other news, I found out there is a included module in Python for logging.  I found it when I was going through PyCharm EDU lessons.  So that should clean up some of my code and allow a more universal experience.  

I have already swapped the logger in my main py file, thanks to word replacement in Notepad++.  PyCharm wouldn’t let me do it through refactoring though, because I was changing it from a function (message_print_log) to, and PyCharm really didn’t want to change from no period thingy to something with a period in it.  

I do like the logger so far, it seems simple yet flexible.  I expect to be fully using it, instead of my home grown one in a few days time with minimal effort.  I do have to change how my logging window functions though… In fact I’ll probably just open up the log file itself in a text editor to “open” the log.  

Until next time.

Edit: I forgot to mention, I have all my code on GitHub now!  (All 4 programs since I started coding in June 2018).

GitHub & PyCharm

I finally got around to trying out GitHub (Hello-World setup) and shortly after uploaded my “Network Testers” python program, which essentially turns 2x raspberry pi’s with a E-Ink display into a Ethernet cable / line tester.  The process went well and I like how GitHub tracks changes with side by side comparisons.   You can check out my GitHub account here.

After doing GitHub, I went over to PyCharm and started watching the setup and basic operational video’s they posted on YouTube.  That also went quite well, and I like how it’s automatically helping me with a multitude of things! It seems to apply something similar to pycodestyle automatically in the code, which is fabulous!  It also organizes the multitude of files on the side, so it’s easy to go between different .py files in a project.  Those files can be opened in a split window, then grouped in tabs per split for multiple open files.  I only started going through PyCharm about an hour ago, so still lots to learn, but it’s already looking pretty sweet.  O did I mention it has a dark theme?  I LOVE dark themes, due to the long nights of coding in dark rooms.  

I don’t think it will be long before I post my Sensor Project into GitHub, since PyCharm can integrate with it.  I just need to figure out and test the integration with the hello world repository I created on GitHub, before I move my KootNet Sensors over, and start using PyCharm with it full time.  

I’m excited to be working on the future stability and accessibility of my projects.  

Until next time. 

Sensors Tested!

I have now tested and fixed bugs for all sensors, minus the Trigger ones (still need to re-write the code for that one).  I’m happy to say, that it looks good!  They all passed sensor get and database write tests.  I also added a “round_decimal_to” variable at the top of each sensor that can use it, so I have a consistent round to, and the ability to change per sensor.  One of the light sensors, recorded the Colour values for Red, Green, Blue to like 14 decimal points!  Would these decimal placements make a difference? I’m guessing not, since it was like 34.00000000000001 when they did show, but usually showed whole numbers like 34.  Anywho, the round_decimal_to is set at 3 for now, which should be a decent balance between useful information and background noise. 

I already like my new system for selecting installed sensors.  Its very nice to just open the config file and change a few 0’s to 1’s and 1’s to 0’s as I test different sensors on the Breakout Garden sensor board.  

I think that’s it for today.  Time to relax, then goto bed. 

Soo many modifications….

OK, just finished a big overhaul of the code running on the sensors themselves and tested it on the Enivro PHAT sensor (current primary sensor).  After a bit of fiddling, it works very well! It’s also much easier to figure out what’s going on in the code.  

The Installer script has been updated to set and work with all the sensors I have, and seamlessly works with the sensor to database code.  

That took the past 3 days with moderate poking.  However, I still have to convert the Motion database code, which I will be renaming to “Trigger” Database, as its much more accurate to what it captures.  Updating the trigger database code will probably take longer because I have to add new types of sensors to it.  Right now, I’m only capturing acceleration, but I have to add the following sensors (to start) GAS resistance, gyroscope and Distance.  Not to mention I need to figure out what variance to set for the triggers themselves.  

Not much progress on actual new features, but I have started to make a features to do list, which can be accessed here.  It helps remind me of where the project is going, in case I get caught up in the smaller details. 

I think I’m going to continue on my “Clean Code” book, as it is really setting the stage for easier management BEFORE my code gets too large to want to clean.  I also have a surprising amount of files to get this all working, so I’ll be working towards putting the code on GitHub, to more easily manage things when I need to change different sections that might affect others.  

I suppose that’s it for now.  It’s still smokey here, but not AS bad as before.  Hopefully things will clear up in a few weeks.  

Adding Sensor Support

I decided to get more hardware support for different Sensors going.  In the process, I’m cleaning up and enhancing the sensor and database code.  This will allow recording of ONLY what sensors are present, so there’s no big list of “N/A” for things not there, just null columns.  

I think I have added / modified all the compatible sensors that I have, which include the Raspberry Pi Sense hat, and the following, that are all Pimoroni brand Sensors;  bh1745 (Light), BME680 (Temp, Pressure, Humidity and gas resistance), Enviro hat (temp, pressure, light, magnetometer, accelerometer), LSM303D (magnetometer, accelerometer), VL53L1X (Distance sensor /w lasers). 

Now to modify the SQL writer to work with them.  Although, I’m going to take a little time to rethink how it determines which sensors are available, as I would like to cut down on the amount of code needed to identify said available sensors… Not to mention make it simpler to understand and modify. 

30 min later …. 

I think I got it!  create a new line per sensor in the “Sensors installed” file, have a number as the first char, such as 0 = not installed, 1 = installed.  Then after that, put the name of the sensor, as it would only look at the line’s first char to check.

# The first char is 0 for disabled and 1 for enabled
0 = Pimoroni BME680
1 = Pimoroni Enviro
0 = Pimoroni bh1745
0 = etc.

This way, it’s easy to modify if you open the config up, as the names right there with simple instructions.  hmm… yeah I think that will do nicely. Time to work that into the installer script.