I finally got around to trying out GitHub (Hello-World setup) and shortly after uploaded my “Network Testers” python program, which essentially turns 2x raspberry pi’s with a E-Ink display into a Ethernet cable / line tester.  The process went well and I like how GitHub tracks changes with side by side comparisons.   You can check out my GitHub account here. 
https://github.com/chad-ermacora

After doing GitHub, I went over to PyCharm and started watching the setup and basic operational video’s they posted on YouTube.  That also went quite well, and I like how it’s automatically helping me with a multitude of things! It seems to apply something similar to pycodestyle automatically in the code, which is fabulous!  It also organizes the multitude of files on the side, so it’s easy to go between different .py files in a project.  Those files can be opened in a split window, then grouped in tabs per split for multiple open files.  I only started going through PyCharm about an hour ago, so still lots to learn, but it’s already looking pretty sweet.  O did I mention it has a dark theme?  I LOVE dark themes, due to the long nights of coding in dark rooms.  

I don’t think it will be long before I post my Sensor Project into GitHub, since PyCharm can integrate with it.  I just need to figure out and test the integration with the hello world repository I created on GitHub, before I move my KootNet Sensors over, and start using PyCharm with it full time.  

I’m excited to be working on the future stability and accessibility of my projects.  

Until next time. 

I have now tested and fixed bugs for all sensors, minus the Trigger ones (still need to re-write the code for that one).  I’m happy to say, that it looks good!  They all passed sensor get and database write tests.  I also added a “round_decimal_to” variable at the top of each sensor that can use it, so I have a consistent round to, and the ability to change per sensor.  One of the light sensors, recorded the Colour values for Red, Green, Blue to like 14 decimal points!  Would these decimal placements make a difference? I’m guessing not, since it was like 34.00000000000001 when they did show, but usually showed whole numbers like 34.  Anywho, the round_decimal_to is set at 3 for now, which should be a decent balance between useful information and background noise. 

I already like my new system for selecting installed sensors.  Its very nice to just open the config file and change a few 0’s to 1’s and 1’s to 0’s as I test different sensors on the Breakout Garden sensor board.  

I think that’s it for today.  Time to relax, then goto bed. 

OK, just finished a big overhaul of the code running on the sensors themselves and tested it on the Enivro PHAT sensor (current primary sensor).  After a bit of fiddling, it works very well! It’s also much easier to figure out what’s going on in the code.  

The Installer script has been updated to set and work with all the sensors I have, and seamlessly works with the sensor to database code.  

That took the past 3 days with moderate poking.  However, I still have to convert the Motion database code, which I will be renaming to “Trigger” Database, as its much more accurate to what it captures.  Updating the trigger database code will probably take longer because I have to add new types of sensors to it.  Right now, I’m only capturing acceleration, but I have to add the following sensors (to start) GAS resistance, gyroscope and Distance.  Not to mention I need to figure out what variance to set for the triggers themselves.  

Not much progress on actual new features, but I have started to make a features to do list, which can be accessed here.  It helps remind me of where the project is going, in case I get caught up in the smaller details. 

I think I’m going to continue on my “Clean Code” book, as it is really setting the stage for easier management BEFORE my code gets too large to want to clean.  I also have a surprising amount of files to get this all working, so I’ll be working towards putting the code on GitHub, to more easily manage things when I need to change different sections that might affect others.  

I suppose that’s it for now.  It’s still smokey here, but not AS bad as before.  Hopefully things will clear up in a few weeks.  

I decided to get more hardware support for different Sensors going.  In the process, I’m cleaning up and enhancing the sensor and database code.  This will allow recording of ONLY what sensors are present, so there’s no big list of “N/A” for things not there, just null columns.  

I think I have added / modified all the compatible sensors that I have, which include the Raspberry Pi Sense hat, and the following, that are all Pimoroni brand Sensors;  bh1745 (Light), BME680 (Temp, Pressure, Humidity and gas resistance), Enviro hat (temp, pressure, light, magnetometer, accelerometer), LSM303D (magnetometer, accelerometer), VL53L1X (Distance sensor /w lasers). 

Now to modify the SQL writer to work with them.  Although, I’m going to take a little time to rethink how it determines which sensors are available, as I would like to cut down on the amount of code needed to identify said available sensors… Not to mention make it simpler to understand and modify. 

30 min later …. 

I think I got it!  create a new line per sensor in the “Sensors installed” file, have a number as the first char, such as 0 = not installed, 1 = installed.  Then after that, put the name of the sensor, as it would only look at the line’s first char to check.

Example.
# The first char is 0 for disabled and 1 for enabled
0 = Pimoroni BME680
1 = Pimoroni Enviro
0 = Pimoroni bh1745
0 = etc.

This way, it’s easy to modify if you open the config up, as the names right there with simple instructions.  hmm… yeah I think that will do nicely. Time to work that into the installer script.

I have been reading the Clean Code book I got today, and it is exactly what I was looking for!  Keeping functions small, preferably do one thing only, have descriptive names, don’t pass too many variables to a function.  These are just a few of the insights helping me clean up my code.  It’s also a good point in development to do this I think, as there are a lot of areas I can improve, and what better way to learn then through lots of practice! 

I took my one big Sensor Commands function and put it in it’s own .py file, then broke it up into many smaller functions.  MUCH nicer, but still needs work.  I’m pretty happy overall, since I do actually want to share my code and have it be useful to others.  Being easy to read and follow seem like important steps in that direction.  

Guess I’ll be spending lots of time working on the code, but making zero visual and functional changes ^_^  It actually sounds like it would be one of those really boring things, and I suppose it would be if that’s ALL I was doing.  However, just like organizing your office to perfection, there’s a real satisfaction from cleaning up programing code… of course I wouldn’t go as far as to say “To Perfection”, since I’m still very much learning!  But a desire to learn, and do better are very important steps on the path towards Perfection (Never obtained, always getting closer). 

I took a bit of time to update the websites.  This should better relay some information about the project.  You’ll notice a few more links in the menus.

I have also been testing the new PC application features a fair bit, and think they are ready to be released.  I’ll upload new installers and test Databases by next week. Old databases won’t work with the new version, due to new columns added to take magnetic readings (Graphing looks for those new columns).

Old databases can be updated fairly easy, by just having the updated sensor units record into the database (auto adds missing columns).  This will allow use in the new program.  I’m also working on my SQL Database modifier so this can be done locally, but that’s still a ways off.  

I used PyCodeStyle on my primary application, and its in good shape now according to that (LOTS of minor changes made).  Although I still have to look into making my ‘try: except:’ statements more accurate, as the except catches all right now.  I’ll have to find a walkthrough or video explaining how to use the except properly and specifically.  

I also started to read the new book I got for “Clean Code”, which most of it so far makes sense, as it’s a lot of attention to detail, continual re-evaluations and putting the time in learning and practicing better ways.  Sounds like what I’m doing already, so that’s good to hear in the first chapter. I’m looking forward to getting into the details in later chapters!  

Until next update! 

As my Project gets bigger (well over 1000 lines of code for the PC prog alone), its becoming more and more obvious that I need to set aside time to learn how to write better code to make it easier to read.  Adding better built in error checking and well documented functions would help too. 

I’m also fairly concerned about the feedback I’ll get when posting my code for the first time, as I know from experience, a lot of people online are better at pointing out faults then actually helping you (Part of the reason I have not posted my code directly yet)

After a fair amount of google searching, using different searches to find relevant information, I came across a good link for python that had more links and book references to help.  So I think I’m in a good place to start learning about cleaning up my code.  

The first and more direct helper is “pycodestyle” (was pep8), which is a python module that can be installed with pip.  This will be HUGE in the initially clean up stage.  I also got a book called “Clean Code: A Handbook of Agile Software Craftsmanship (Robert C. Martin Series)”.  The reviews were all good, except for the physical quality of the book (formatting off for print size and pages falling out?).  So I got a kindle version and will start to read through that.  

I’m also happy to say that the smoke has cleared up a LOT!  So I have been going for my morning walks again, and feeling much better.  

There have been some decent additions to the PC control Center program, both in function and organization of data.  So I may post a updated version soon, or at least update the pictures.  

In other news, I have opened up a GitHub account, and it looks fairly straightforward in how it works and what it offers (At least to get started).  So once I actually clean up my code, it’s going there for public review and access!  

Guess that’s it for this post.  Lots to do and learn, but enjoying the process.  I especially like looking back at older versions of my program and seeing how much progress has been made.  

I’m thinking of how I might enhance the commands and update system.  

It would be nice if one could queue up commands to run, and have the sensor run them whenever they get online to check.  

To this end, I had a few ideas.  The sensors could check a folder through a http URL every x min, and if it finds a file there, with its host name as the filename, it would check the date stamp inside.  If the date stamp is newer than its local “Last Ran” date stamp, it would run the commands inside and then put the current time as its local “Last Run” date stamp.  

This could be further enhanced by having the sensor itself connect back to a “Server” where the Control Center software is running to say it ran the commands or even remain connected for future communications.  The URL to connect to would be included in the http file the sensor initially looks at. 

This has a few advantages.  By changing the URL or IP in the http file, sensors would start connecting to you in minutes, also no ports need to be open to have the sensors run commands, and the sensors only need to be online long enough to run said commands. If you do want to control them in real time, ports only have to be setup / forwarded to the Control Center PC.  Once setup, just turn it on, load the KootNet Sensors – PC Control Center and wait for sensors to check in. Once they have, control them as normal through the program.  

This could really be useful in areas with spotty internet connections.  Or a way to que up a bunch of tasks for mostly offline sensors.  A few things that could be useful include, sending the database to a online storage location, changing how or what is being recorded or getting it to update.  Get them online for 10 min or however long the commands will take (Plus the x time it waits to checkin), and it’s done.  

This will DEFINITELY be an undertaking…. I might make this a whole separate program, just for reverse connection management… that way there are multiple ways to manage the units, depending on your needs, and multiple ways to try and connect, if something goes wrong.  I’ll see once I get started and figure out how its all going to fit together.  

I have now updated the Sensor Install to be modular for the Sensor its being installed onto.  You only have to select the right sensor on install.  Everything else is taken care of.  I have also merged the Install & NAS Update script into one.  If it’s the first run, it does a install, asking for sensor type, installs python modules and updates the system in general.  If it’s not the first install, it updates the sensor programs and crontab (no user interaction required for updates).

I have also updated the file structure on the sensors, so it’s a bit easier to understand what does what.  3 folders have been added

1) auto_start = autostart scripts to make sure programs are running and start them if they are not running. 

2) sensor_modules = supported sensor modules to get sensor data (Imported into main program)

3) upgrade = Install and update scripts (NAS / Online / E-Ink). 

I think I updated all other pointers in programs and scripts to match the new locations, but I have yet to test them all.  

The folder /home/sensors/sensor_modules/ on the sensors themselves, has a .py programs for each different Sensor that is supported.  Adding new Sensors is fairly easy now, as you just add another .py file, that has a def, which returns the new sensor readings. Look at one of the other modules to use as an example, since absent sensor readings still need to be returned, simply set them manually as “N/A”.  

This took pretty much all day, but I’m glad it’s done, as it will make things SOO much easier in the future adding different sensors.  

I think I’ll work on fixes next, as there are a few things I need to tweak, like actually showing the hostname on the graphs and applying the timezone to motion plots.  

In other news, it’s still super smokey and is off the Canadian Air Quality scale (has been for weeks, with a few dips to 8 out of 10). I have also run my Air Quality sensor inside and out.  Inside it read about 150,000 Ohms @ 33% Humidity.  Outside it’s more like 300,000 Ohms @ 40% Humidity.  The air quality equation still eludes me (Have not made the time to go over it), but having double the resistance… seems bad, especially since inside is not exactly fresh healthy air either.  

All right!  New sensors are in and I’m testing them out.  Got demo code for most of them working, although I should of read the fine print on a few of the Sensor boards, as they don’t have working code for Python yet… But that’s only the Thermal Cameras and maybe the motion one.. but I’m pretty sure I can get the motion one to work.  

I have been learning about the Air Quality Index (AQI) as well as how this sensor works in approximating it.  For starters, it seems like its more about getting relative changes after a “Burn in” test.  Which is OK to spot differences over time, but I would like to get a more … absolute value as well, so I know how bad it is already!  Guess I’ll just have to compare indoor vs. outdoor for smokey air issues…. not that inside is super smoke free… But it’s a start!  

I’m not recording any of this to a database yet, as I more wanted to play and learn about the sensors before recording.  So as I type this, I have the demo code showing me the approx Air quality inside.  It takes a 5 min burn in period before it starts to show the AQI in the form of a % out of 100.  Inside its been running for approx 5-10 min and its showing a 96%.  I’ll put it outside shortly and see how it drops (I have it running on a battery for moving around without shutting down).  

That’s it for this update.  New sensors will be supported soon AND I’ll be working on the Sensor installer (not the PC app), so there is less fuss getting different sensors working.